Solved

DateTimeConverter cant autodetect

  • 30 April 2018
  • 8 replies
  • 30 views
jorge_vidinha
Rank
Badge +1

I have the following datetime format built from python datetime like bellow.

datetime.datetime.today()

2018-04-29 12:20:36.975031

Im reading a csv dataset and trying to load it to an elasticsearch writer setting the writer field with the pythondate to a datatype 'date' but it always fail with following error 

2018-04-30 16:35:25|   2.0|  0.0|ERROR |ELASTICSEARCH writer: {"took":0,"errors":true,"items":[{"index":{"_index":"imob","_type":"imov","_id":"pzIxF2MB3029rqSru8Tt","status":400,"error":{"type":"mapper_parsing_exception","reason":"failed to parse [date]","caused_by":{"type":"illegal_field_value_exception","reason":"Cannot parse \"0000/00/00 00:00:00.000\": Value 0 for monthOfYear must be in the range [1,12]"}}}}]}

 Seems odd cause the date beeing sent is beeing checked with a logger just before the ES writer and the  value is 

2018-04-30 16:40:42|   1.7|  0.0|INFORM|Attribute(string)                 : `date' has value `2018-04-29 12:20:36.975031'

So how is it beeing converted to \"0000/00/00 00:00:00.000\" as the ES 400 code replys back ?

Also tried to convert this format any other formats but it looks like FME does not recognizes the python created format . 

Anyone dribbled similar topic before ?

Regards 

0684Q00000ArCmBQAV.png

icon

Best answer by jakemolnar 30 April 2018, 19:34

View original

8 replies

jakemolnar
Badge

Hi @jorge_vidinha,

The FME Elasticsearch Writer requires datetime attributes to be in FME datetime format before writing to Elasticsearch (http://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_ReadersWriters/!Transformer_Parameters/standard_fme_date_time_format.htm).

This will be converted to Elasticsearch datetime format as part of the translation from FME features to Elasticsearch documents. Just a side note: right now, the writer only supports a few Elasticsearch datetime formats on output (though we support almost all of them when *reading* from Elasticsearch).

takashi
Rank
Userlevel 2
Badge +17

Regarding the question on the DateTimeConverter, you have set 'ISO' to the Input Format parameter, whereas the input date value is not an ISO datetime value. Check the ISO 8601 format.

david_r
Rank
Userlevel 4

Rather than using the datetime.today() function, try this:

datetime.datetime.now().strftime('%Y%m%d%H%M%S')

This will render the timestamp in the format required by FME, e.g.

'20180501091536'

In other words, no need using the DateTimeConverter.

O

Hi @jorge_vidinha,

The FME Elasticsearch Writer requires datetime attributes to be in FME datetime format before writing to Elasticsearch (http://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_ReadersWriters/!Transformer_Parameters/standard_fme_date_time_format.htm).

This will be converted to Elasticsearch datetime format as part of the translation from FME features to Elasticsearch documents. Just a side note: right now, the writer only supports a few Elasticsearch datetime formats on output (though we support almost all of them when *reading* from Elasticsearch).

I cannot get ES to recognise any date formats at all. On the writer I'm using dynamic attribute definition, and have tried various date and datetime formats. All of it gets recognised only as string values.

 

 

takashi
Rank
Userlevel 2
Badge +17

Hi @jorge_vidinha,

The FME Elasticsearch Writer requires datetime attributes to be in FME datetime format before writing to Elasticsearch (http://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_ReadersWriters/!Transformer_Parameters/standard_fme_date_time_format.htm).

This will be converted to Elasticsearch datetime format as part of the translation from FME features to Elasticsearch documents. Just a side note: right now, the writer only supports a few Elasticsearch datetime formats on output (though we support almost all of them when *reading* from Elasticsearch).

Hi @ottadini, I guess the issue is where how you have configured the dynamic writer feature type, rather than the datetime format. With a dynamic schema configuration, the data type of each destination attribute would be determined based on the schema definition you derived from somewhere. If the data type of an attribute in the schema definition was a string type, the destination attribute would also be a string type even if the value has been formatted with a datetime format.

 

  • From where did you derive the schema definition?
  • What data type does the attribute have in the schema definition?
O
Hi @ottadini, I guess the issue is where how you have configured the dynamic writer feature type, rather than the datetime format. With a dynamic schema configuration, the data type of each destination attribute would be determined based on the schema definition you derived from somewhere. If the data type of an attribute in the schema definition was a string type, the destination attribute would also be a string type even if the value has been formatted with a datetime format.

 

  • From where did you derive the schema definition?
  • What data type does the attribute have in the schema definition?
Hi @takashi, thanks for the help.

 

I retrieve the datetime as a file property from a Directory and File pathnames reader. I transform the datetime to an ISO date using a DateTimeConverter. The remainder of my feature attributes come from a FeatureReader, but they are not constant, so I need a dynamic schema.

 

Is there a way I can mix dynamic schema definition with some manually defined schema?

 

I have a SchemaSetter as well, however I removed this and it made no difference.

 

 

takashi
Rank
Userlevel 2
Badge +17
Hi @ottadini, I guess the issue is where how you have configured the dynamic writer feature type, rather than the datetime format. With a dynamic schema configuration, the data type of each destination attribute would be determined based on the schema definition you derived from somewhere. If the data type of an attribute in the schema definition was a string type, the destination attribute would also be a string type even if the value has been formatted with a datetime format.

 

  • From where did you derive the schema definition?
  • What data type does the attribute have in the schema definition?
  • In a translation process, FME treats datetime values as character strings. The SchemaSetter won't determine if an attribute type is datetime.
  • If necessary, you can add user attributes common to every destination feature type through the User Attributes tab in a dynamic writer feature type.
  • As @jakemolnar and @david_r suggested, you have to format datetime values with Standared FME Date/Time Format before writing features into the writer feature type. It's a general rule of FME regardless of what the destination data format is. At run time, FME writers automatically convert the datetime format from the FME standard one to the proper one for the destination data format.

 

jorge_vidinha
Rank
Badge +1

Hi @jorge_vidinha,

The FME Elasticsearch Writer requires datetime attributes to be in FME datetime format before writing to Elasticsearch (http://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_ReadersWriters/!Transformer_Parameters/standard_fme_date_time_format.htm).

This will be converted to Elasticsearch datetime format as part of the translation from FME features to Elasticsearch documents. Just a side note: right now, the writer only supports a few Elasticsearch datetime formats on output (though we support almost all of them when *reading* from Elasticsearch).

 

Ok clear this is now working fine , i had to preprocess my datetime format according to %Y-%m-%dT%H:%M:%S$ and now able to use DateTimeConverter and set output format to FME datetime format . This way ElasticSearch does recognizes well the date data type.

 

Thanks all,

Reply


Community Stats

28,662
Posts
109,324
Replies
36,528
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.