Elastic Reader Issues

It looks to be Version 7.8 of Elastic.

Do you know when the reader is going to read newer versions?

It just so happens that I'm working on it right now, as we speak.

It's not my role to make any promises about the schedule, but it would be reasonable to expect support in a Beta release in the near future.

@jakemolnar can you let me know when the update is released on the stable version? Many thanks

Seb

@sebkingsley I'll ask our Community team to update you, and you can always check the status of https://knowledge.safe.com/content/idea/101746/add-support-for-elasticsearch-v7.html , since they will update it when I complete the new feature.

Hi, do you have any updates on this issue?

I've seen in the change log of fme desktop 2021 beta (Build 21214) that elastic 7+ is supported. So i've dowloaded it and tried to connect to an elasticsearch 7.9.1, but it doesn't work. Do you have any information of when it will be fixed and if it will work for all 7+ versions (7.9.1 included) ?

@miay did you add a new Elasticsearch Reader?

In the beta there should be a old Elasticsearch Reader (deprecated, supports versions 6.8 and lower) and a new Elasticsearch Reader (supports version 7+).

If it doesn't work, then we would definitely want to know. Do you have any logs or workspaces you can share?

@miay​ I had the same issue and logged a support case for it.

I was hoping the new reader might come in 2020.2

Yes, i used the new Elasticsearch reader. We use Elasticsearch 7.9. 1. I can create a workspace and send you the logs.

@miay​  Sounds great, that would be very helpful. Any information about your server configuration would also be helpful (eg. Do you have authentication set up? Are you accessing through a proxy? Do you host the server yourself, or is it on AWS for example? etc. etc.)

Hi,

We host the elastic stack.

The fme desktop 2020 beta is installed in a virtual pc within our network.

I can access elastic/kibana from chrome without problems.

I get this error message when i click on the button at the right of "Feature types to read":

The text that appears in the log is: Creating reader for format: Elastic Elasticsearch

Trying to find a DYNAMIC plugin for reader named `ELASTICSEARCH_CLUSTER'

Loaded module 'ELASTICSEARCH_CLUSTER' from file 'C:\\Program Files\\FME2021\\plugins/gdal/GDAL.dll'

FME API version of module 'ELASTICSEARCH_CLUSTER' matches current internal version (3.8 20200922)

Creating reader for format: Elastic Elasticsearch

Trying to find a DYNAMIC plugin for reader named `ELASTICSEARCH_CLUSTER'

FME API version of module 'ELASTICSEARCH_CLUSTER' matches current internal version (3.8 20200922)

Creating reader for format: Elastic Elasticsearch

Trying to find a DYNAMIC plugin for reader named `ELASTICSEARCH_CLUSTER'

FME API version of module 'ELASTICSEARCH_CLUSTER' matches current internal version (3.8 20200922)

@miay​ is that the entire log?

• If not, can you attach the entire thing?
• If so, are there any lines that were red in color? If this is the entire log, then it indicates there may a problem with the configuration of the new Elasticsearch plugin in the beta; unlicensed, missing binaries, something like that.

Hi, thanks for your answer.

Sorry, i might missed out to explain that i experience the problem when i set up the Elastic reader or featurereader, and click on the button at the right of "features type to read". The reader cannot access the indices in Elastic. So i actually don't have anything to run, and therefore no contents in the log.

Regarding the licensing, i've connected to our license server.

Regarding proxy, we don't need to go through a proxy.

This was the exact same issue I had.

The log file stated it couldnt read / find it

Interesting.. @miay​ and @bubblebeb​, you both also tried the old Elasticsearch reader, right?

Anyway, in a web browser could you go to:

• [your-elasticsearch-server]/[an-index-you-expected-to-find]/_settings
• [your-elasticsearch-server]/_cluster/settings
• [your-elasticsearch-server]/

and copy all non-secret results here?

Then I can try to replicate your setup, see what is different with our local testing setup, and account for the differences.

Hi @jakemolnar​ 

{"focalpoint.projects":{"settings":{"index":{"creation_date":"1588326007516","number_of_shards":"1","number_of_replicas":"1","uuid":"PQijcry5SJSccvWyXHjw-A","version":{"created":"7060299","upgraded":"7090099"},"provided_name":"focalpoint.projects"}}}}

{"persistent":{"xpack":{"monitoring":{"collection":{"enabled":"true"}}}},"transient":{}}

{

"name" : "bad-7-node1",

"cluster_name" : "fg-bad-7",

"cluster_uuid" : "RH6Cl87uQQyYl_q2a7SKwQ",

"version" : {

"number" : "7.9.0",

"build_flavor" : "default",

"build_type" : "rpm",

"build_hash" : "a479a2a7fce0389512d6a9361301708b92dff667",

"build_date" : "2020-08-11T21:36:48.204330Z",

"build_snapshot" : false,

"lucene_version" : "8.6.0",

"minimum_wire_compatibility_version" : "6.8.0",

"minimum_index_compatibility_version" : "6.0.0-beta1"

},

"tagline" : "You Know, for Search"

}

Hope that helps.

Thanks @bubblebeb​, that will be really helpful for me. I'll be able to replicate your setup and see what's going wrong.

Hi @jakemolnar​ ,

Here are my results:

{"logstash-sodgeopuat-202009":{"settings":{"index":{"lifecycle":{"name":"mmogc-test-ilm-policy"},"refresh_interval":"5s","number_of_shards":"2","provided_name":"logstash-sodgeopuat-202009","creation_date":"1598933246996","priority":"100","number_of_replicas":"0","uuid":"nnhWBroQQzOpySGSYpDSrQ","version":{"created":"7090099","upgraded":"7090199"}}}}}

{"persistent":{},"transient":{}}

{

 "name" : "myservername",

 "cluster_name" : "elasticsearch",

 "cluster_uuid" : "lymWD6HIS-OLaB2VRiuWIw",

 "version" : {

  "number" : "7.9.1",

  "build_flavor" : "default",

  "build_type" : "rpm",

  "build_hash" : "083627f112ba94dffc1232e8b42b73492789ef91",

  "build_date" : "2020-09-01T21:22:21.964974Z",

  "build_snapshot" : false,

  "lucene_version" : "8.6.2",

  "minimum_wire_compatibility_version" : "6.8.0",

  "minimum_index_compatibility_version" : "6.0.0-beta1"

 },

 "tagline" : "You Know, for Search"

}

Thanks @miay​, I think I've probably found the problem with your help (and that of @bubblebeb​).

Just to follow up on some probable lines of inquiry: are your Elasticsearch deployments username/password protected?

Hi @jakemolnar​ ,

Thanks for your efforts!

Yes, our Elastic deployments are password protected.

Hi @jakemolnar​ 

Yes ours are Username and Password protected.

@miay​ and @bubblebeb​ that's great; that lines up with the evidence I've observed on my end. That means I've likely fixed the issue that was giving you trouble, and I imagine the fix will make it into a beta soon. You can keep an eye out for a change that mentions Elasticsearch in https://downloads.safe.com/fme/beta/whatsnew_2021_0.txt

Thanks @jakemolnar​ !

I'll keep an eye on "what's new".

Hi @jakemolnar​ 

I have installed Build b21221 and tried to connect to both our elastic stacks (AWS and internal) and get the same error message as before.

ELASTICSEARCH_CLUSTER reader: Failed to open the dataset 'ES:'. Please ensure source data is valid and correct reader is selected

UniversalReader -- readSchema resulted in 0 schema features being returned

Reader Parameter(0) = >https://xxxxxxx.eu-west-2.aws.cloud.es.io:9243/<

Reader Parameter(1) = >USE_AUTHENTICATION<

Reader Parameter(1) = >YES<

Reader Parameter(3) = >BASIC_AUTH_USERNAME<

Reader Parameter(3) = >gis<

Reader Parameter(5) = >BASIC_AUTH_PASSWORD<

Reader Parameter(5) = >********<

Reader Parameter(7) = >RETRIEVE_ALL_TABLE_NAMES<

Reader Parameter(7) = >YES<

Reader Directive(0) = >USE_AUTHENTICATION<

Reader Directive(1) = >YES<

Reader Directive(2) = >BASIC_AUTH_USERNAME<

Reader Directive(3) = >xxxxxxx<

Reader Directive(4) = >BASIC_AUTH_PASSWORD<

Reader Directive(5) = >********<

Reader Directive(6) = >RETRIEVE_ALL_TABLE_NAMES<

Reader Directive(7) = >YES<

UniversalReader -- readSchema resulted in 0 schema features being returned

ELASTICSEARCH_CLUSTER reader: A fatal error has occurred. Check the logfile above for details

Any tips on what might be the cause of this?

Thanks

Seb

@bubblebeb​ that's strange; our QA department verified that we are able to connect to an AWS-hosted and password-protected cluster now.

I've seen this kind of error when the cluster is completely empty; do you have any indices created?

Have you made sure the deployment is reachable from your machine? ie. trying to reach https://xxxxxxx.eu-west-2.aws.cloud.es.io:9243/

in a browser or with curl like `curl --user 'username:password' -XGET https://xxxxxxx.eu-west-2.aws.cloud.es.io:9243/`