Solved

ElasticSearch writer cannot parse dates

O

I'm using FME 2018.0.

I originally thought this was a problem with the dynamic attribute definition of the writer, but for this test I have used a Manual schema. 

I don't understand why, but the writer is trying to parse a date that I am not seeing in the inspector. The value for Modified_date in the inspector is: 

2011-05-27

The response I get from the error is:

{
    "took": 3,
    "errors": true,
    "items": [
        {
            "index": {
                "_index": "testing_2",
                "_type": "drawings",
                "_id": "3mN9KWMBLz2D_kWNISIW",
                "status": 400,
                "error": {
                    "type": "mapper_parsing_exception",
                    "reason": "failed to parse [Modified_date]",
                    "caused_by": {
                        "type": "illegal_field_value_exception",
                        "reason": "Cannot parse \"0000/00/00 00:00:00.000\": Value 0 for monthOfYear must be in the range [1,12]"
                    }
                }
            }
        },
        {
            "index": {
                "_index": "testing_2",
                "_type": "drawings",
                "_id": "32N9KWMBLz2D_kWNISIW",
                "status": 400,
                "error": {
                    "type": "mapper_parsing_exception",
                    "reason": "failed to parse [Modified_date]",
                    "caused_by": {
                        "type": "illegal_field_value_exception",
                        "reason": "Cannot parse \"0000/00/00 00:00:00.000\": Value 0 for monthOfYear must be in the range [1,12]"
                    }
                }
            }
        }
    ]
}

icon

Best answer by takashi 7 May 2018, 06:44

View original

4 replies

O
What is the simplest workspace that can create a date, and write that to an ES index using dynamic field attributes? I've attached a MWE by way of example - maybe there's something fundamentally wrong with how I'm doing it. Can anyone help? Some ( @takashi, @jakemolnar) have previously said that dates must be FME format, which I've done, but I cannot get them parsed as dates by ES.

 

 

mwe-dates-to-es.fmw
takashi
Rank
Userlevel 2
Badge +17

As I have answered already in a relevant thread, FME handles date/time values as character strings while translating and the SchemaSetter won't consider any attribute value as a date/time value even if it was formatted with Standard FME Date/Time format.

If you actually need to dynamically configure destination schema, you will have to implement a process to define the destination schema (i.e. create "attribute{}" list). See these articles to learn more.

Or, if basically destination schema can be derived from the source dataset dynamically but you just need to add an identical date field always, you could also statically add it to every destination schema through the User Attributes tab in the dynamic writer feature type.

O

As I have answered already in a relevant thread, FME handles date/time values as character strings while translating and the SchemaSetter won't consider any attribute value as a date/time value even if it was formatted with Standard FME Date/Time format.

If you actually need to dynamically configure destination schema, you will have to implement a process to define the destination schema (i.e. create "attribute{}" list). See these articles to learn more.

Or, if basically destination schema can be derived from the source dataset dynamically but you just need to add an identical date field always, you could also statically add it to every destination schema through the User Attributes tab in the dynamic writer feature type.

Or, if basically destination schema can be derived from the source dataset dynamically but you just need to add an identical date field always, you could also statically add it to every destination schema through the User Attributes tab in the dynamic writer feature type.

 

Thanks Takashi, this was what I needed to do.

 

The ES Writer doesn't seem able to give a data type when in dynamic mode. All attributes get written as type text.

 

takashi
Rank
Userlevel 2
Badge +17

As I have answered already in a relevant thread, FME handles date/time values as character strings while translating and the SchemaSetter won't consider any attribute value as a date/time value even if it was formatted with Standard FME Date/Time format.

If you actually need to dynamically configure destination schema, you will have to implement a process to define the destination schema (i.e. create "attribute{}" list). See these articles to learn more.

Or, if basically destination schema can be derived from the source dataset dynamically but you just need to add an identical date field always, you could also statically add it to every destination schema through the User Attributes tab in the dynamic writer feature type.

Good to hear.

 

I believe that the ES writer can configure destination schema dynamically, but you need to define a schema definition correctly. Again, the SchemaSetter is useless in your case. See the articles I have linked above to learn how you can create a schema definition.

 

 

Reply


Community Stats

28,667
Posts
109,348
Replies
36,535
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.