Skip to main content
  • EN

I am trying to query elasticsearch using fme to find the most recent date in a document. when I run it in kibana console as :

POST officeearthsearch/earthsearch/_search
{ "_source": _"CHANGED_DATE"],
  "query": {
    "match_all": {}
  },
  "size": 1,
  "sort": Â
    {
      "CHANGED_DATE": {
        "order": "desc"
      }
    }
  ]
}

it works and returns a single record containing the most recent changed date.

In the fme elasticsearch reader I input into the JSON query:

{
  "query": {
    "match_all": {}
  },
  "size": 1,
  "sort": p
    {
      "CHANGED_DATE": {
        "order": "desc"
      }
    }
  ]
}

and get the following error:

FME API version of module 'ELASTICSEARCH' matches current internal version (3.8 20170315)

ELASTICSEARCH reader: Directive '_HOST' has value 'search-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com:80'

ELASTICSEARCH reader: Directive '_RAW_JSON' has value 'No'

ELASTICSEARCH reader: Directive '_NUM_FEATURES_PER_REQUEST' has value '1'

ELASTICSEARCH reader: Directive '_FME_JSON_ATTRIBUTE_FLATTENING_MODE' has value 'ALL_LEVELS'

ELASTICSEARCH reader: Opening index 'search-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com:80/officeearthsearch'

ELASTICSEARCH reader: Feature Type Parameter 'elasticsearch_query_document' has value '{

  "query": {

    "match_all": {}

  },

  "size": 1,

  "sort": h

    {

      "CHANGED_DATE": {

        "order": "desc"

      }

    }

  ]

}' for feature type 'earthsearch'

ELASTICSEARCH reader: HTTP Error 400-599: HTTP error code : 403

ELASTICSEARCH reader: {"message":"When Content-Type:application/x-www-form-urlencoded, URL cannot include query-string parameters (after '?'): '/officeearthsearch/earthsearch/_search?scroll=1m&size;=1'"}

A fatal error has occurred. Check the logfile above for details

FME Session Duration: 2.4 seconds. (CPU: 0.3s user, 0.2s system)

END - ProcessID: 10964, peak process memory usage: 67032 kB, current process memory usage: 48728 kB

 

 

Can you please suggest what I am doing wrong.

Many Thanks

Oliver

Hi @olivermorris,

I tried a stripped down version of your query and it seemed to work. If you can let me know which version of elasticsearch you are using and which version of FME I can find more details on syntax, but for now could you try the following syntax and use the " Features Per Request" parameter in the reader navigation -->Advanced parameters to limit the search to 1.

{

"query": { "match_all": {} },

"sort": { "CHANGED_DATE": { "order": "desc" } }

}

I am trying with 17.1.1.0 and couldnt get this to work, still the same error. Running elasticsearch 5.1.1.

ELASTICSEARCH reader: HTTP Error 400-599: HTTP error code : 403

ELASTICSEARCH reader: {"message":"When Content-Type:application/x-www-form-urlencoded, URL cannot include query-string parameters (after '?'): '/officeearthsearch/earthsearch/_search?scroll=1m&size;=1'"}

A fatal error has occurred. Check the logfile above for details

Will try with FME b17717

sorry, still no luck even with the parameter changed. workbench is here https://we.tl/TEAr5KcqFP

It looks like this issue is caused by Amazon Web Services acting as a proxy and changing the content type of the message which causes any query to fail. Were looking to see if there is anything FME can do to workaround this issue.

I am having similar issues. I am using FME beta 2018.1 and used the Elasticsearch reader. The Elasticsearch (version 6.3) inside Amazon and is configured not to use any authentication. It works perfectly when I don't insert any JSON Query but when I put a simple query like

{"query":{"term":{"CountryOrTerritory":"Belgium"}}}

I get the following error.

2018-06-22 14:40:32| 0.6| 0.0|ERROR |ELASTICSEARCH reader: HTTP Error 400-599: HTTP error code : 406

2018-06-22 14:40:32| 0.6| 0.0|ERROR |ELASTICSEARCH reader: {"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}

2018-06-22 14:40:32| 0.6| 0.0|ERROR |A fatal error has occurred. Check the logfile above for details

If I use an HTTPCaller and write a GET statement it works as expected.

/ieda/_search?q=CountryOrTerritory=Belgium

Another error I found is in the Elastic search reader and writer is that it doesn't like HTTP:// syntax. The document says you can use it but it doesn't work when you have something like

http://13.54.7.126:9201, It only works when you type 13.54.7.126:9201

Jan

@janb

 

I just used the http caller in the end as well. I am glad it isnt just me!

 

Hi @olivermorris,

I am facing the same issue with both loading and reading data to and fro from elasticsearch in FME. Could you tell me what is the fix? I appreciate any help.

Hi @olivermorris,

I am facing the same issue with both loading and reading data to and fro from elasticsearch in FME. Could you tell me what is the fix? I appreciate any help.

Hi @vkambham,

We made changes in the reader in build 18236 and you should no longer have the content issue. Please let me know if you are still having issue in the latest version.

 

Richard

Reply


Community Stats

30,108
Posts
114,295
Replies
37,744
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.