ElasticSearch | Community

Hi,

I am trying to connect to my AWS elasticache service

https://search-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com/

I can go here on a web page and it returns:

{"name": "dF9fcow","cluster_name": "929479087525:dev-elasticsearch","cluster_uuid": "Owj46HL6TpSWiGnOeYeCaw","version": {"number": "5.1.1","build_hash": "5395e21","build_date": "2016-12-15T22:47:19.049Z","build_snapshot": false,"lucene_version": "6.3.0"},"tagline": "You Know, for Search"}

But for some reason FME cannot connect, it appears to be trying over port 9200.

This is the error I get:

ELASTICSEARCH writer: CURL error was: Failed to connect to search-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com port 9200: Connection refused

ELASTICSEARCH writer: Unable to locate or open server 'https://search-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com'.  Please ensure the server exists and is accessible.  Note that the ELASTICSEARCH writer does not have the ability to create a new server.  However, creating the server manually in advance allows for subsequent data insertion

A fatal error has occurred. Check the logfile above for details

Has anyone got any suggestions of how I can connect?

Many Thanks,

Oliver

Also below is the log output:

Creating reader for format: Elastic Elasticsearch

Trying to find a DYNAMIC plugin for reader named `ELASTICSEARCH'

FME API version of module 'ELASTICSEARCH' matches current internal version (3.8 20170808)

Creating reader for format: Elastic Elasticsearch

Trying to find a DYNAMIC plugin for reader named `ELASTICSEARCH'

FME API version of module 'ELASTICSEARCH' matches current internal version (3.8 20170808)

ELASTICSEARCH reader: Directive '_HOST' has value 'https://search-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com/'

ELASTICSEARCH reader: CURL error was: Could not resolve host: https%3A%2F%2Fsearch-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com%2F

ELASTICSEARCH reader: Failed to detect server version

ELASTICSEARCH reader: Directive '_RAW_JSON' has not been set.  Using default value 'NO'

ELASTICSEARCH reader: Directive '_NUM_FEATURES_PER_REQUEST' has not been set.  Using default value '100'

ELASTICSEARCH reader: Directive '_FME_JSON_ATTRIBUTE_FLATTENING_MODE' has not been set.  Using default value 'ALL_LEVELS'

ELASTICSEARCH reader: Opening index 'https://search-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com//index_list'

ELASTICSEARCH reader: CURL error was: Could not resolve host: https%3A%2F%2Fsearch-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com%2F

ELASTICSEARCH reader: Could not resolve host: https%3A%2F%2Fsearch-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com%2F

ELASTICSEARCH reader: Failed to open the dataset 'ES:'. Please ensure source data is valid and correct reader is selected

A fatal error has occurred. Check the logfile above for details

Failed to obtain any schemas from reader 'ELASTICSEARCH' from 1 datasets. This may be due to invalid datasets or format accessibility issues due to licensing, dependencies, or module loading. See logfile for more information

UniversalReader -- readSchema resulted in 0 schema features being returned

Reader Parameter(0) = >INDEX_LIST<

Reader Parameter(1) = >HOST<

Reader Parameter(2) = >https:<solidus><solidus>search-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com<solidus><

Reader Parameter(3) = >SETTINGS_DYNAMIC_SELECT_QUERY<

Reader Parameter(4) = >QUERY_INDICES<

Reader Parameter(5) = >FME_READ_SCHEMA<

Reader Parameter(6) = >YES<

Reader Parameter(7) = >_MERGE_SCHEMAS<

Reader Parameter(8) = >YES<

Reader Directive(0) = >RUNTIME_MACROS<

Reader Directive(1) = >HOST,https:<solidus><solidus>search-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com<solidus>,SETTINGS_DYNAMIC_SELECT_QUERY,QUERY_INDICES,FME_READ_SCHEMA,YES,_MERGE_SCHEMAS,YES<

Configuration object contents:

FME Temporary Folder: C:\Users\H159842\AppData\Local\Temp

mapping file Line 0: `APPLICATION' `FME_WORKBENCH'

mapping file Line 1: `AUTOREG' `YES'

mapping file Line 2: `LOG_ENCODING' `UTF-8'

mapping file Line 3: `LOG_MAX_FEATURES' `-1'

mapping file Line 4: `LOG_APPEND' `NO'

mapping file Line 5: `FME_HOME' `C:\Program Files\FME\'

mapping file Line 6: `FME_SESSION_USED' `TRUE'

mapping file Line 7: `R_4_METAFILE' `ELASTICSEARCH'

mapping file Line 8: `R_4_DATASET' `INDEX_LIST'

mapping file Line 9: `R_4_HOST' `https:<solidus><solidus>search-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com<solidus>'

mapping file Line 10: `R_4_SETTINGS_DYNAMIC_SELECT_QUERY' `QUERY_INDICES'

mapping file Line 11: `R_4_FME_READ_SCHEMA' `YES'

mapping file Line 12: `R_4__MERGE_SCHEMAS' `YES'

mapping file Line 13: `R_4_HOST' `https:<solidus><solidus>search-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com<solidus>'

mapping file Line 14: `R_4_SETTINGS_DYNAMIC_SELECT_QUERY' `QUERY_INDICES'

mapping file Line 15: `R_4_FME_READ_SCHEMA' `YES'

mapping file Line 16: `R_4__MERGE_SCHEMAS' `YES'

mapping file Line 17: `R_4_MERGE_SCHEMAS' `YES'

mapping file Line 18: `R_4_FME_JSON_ATTRIBUTE_FLATTENING_MODE' `'

mapping file Line 19: `SCHEMA_TIME' `YES'

mapping file Keyword: `LOG_ENCODING'                           occurs 1 time(s)

mapping file Keyword: `R_4_HOST'                               occurs 2 time(s)

mapping file Keyword: `R_4_FME_READ_SCHEMA'                    occurs 2 time(s)

mapping file Keyword: `APPLICATION'                            occurs 1 time(s)

mapping file Keyword: `FME_HOME'                               occurs 1 time(s)

mapping file Keyword: `R_4_SETTINGS_DYNAMIC_SELECT_QUERY'      occurs 2 time(s)

mapping file Keyword: `R_4_MERGE_SCHEMAS'                      occurs 1 time(s)

mapping file Keyword: `R_4_DATASET'                            occurs 1 time(s)

mapping file Keyword: `FME_SESSION_USED'                       occurs 1 time(s)

mapping file Keyword: `R_4_METAFILE'                           occurs 1 time(s)

mapping file Keyword: `LOG_MAX_FEATURES'                       occurs 1 time(s)

mapping file Keyword: `LOG_APPEND'                             occurs 1 time(s)

mapping file Keyword: `R_4_FME_JSON_ATTRIBUTE_FLATTENING_MODE' occurs 1 time(s)

mapping file Keyword: `R_4__MERGE_SCHEMAS'                     occurs 2 time(s)

mapping file Keyword: `AUTOREG'                                occurs 1 time(s)

mapping file Keyword: `SCHEMA_TIME'                            occurs 1 time(s)

UniversalReader -- readSchema resulted in 0 schema features being returned

Failed to obtain any schemas from reader 'ELASTICSEARCH' from 1 datasets. This may be due to invalid datasets or format accessibility issues due to licensing, dependencies, or module loading. See logfile for more information

Page 1 / 1

Also tried using cURL on the machine and requests to the service worked absolutely fine?!?

I dont think the issue lies with proxies/networking etc

Hello @olivermorris

I suspect that the server connection string you are using is not quite right. From our documentation, the reader is expecting a server string in the following format: ' <server>:<port>'. with the port defaulting to 9200 if it is not specified.

Are you able to confirm that the URL you are inputting: 'https://search-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com/'

is, in fact, the hostname/IP Address of the ElasticSearch server?

I tried to append :443 at the end as it is going over https but no luck. I can reach it through a browser, cURL and Sense. Just not FME?

Below is the error log:

Creating reader for format: Elastic Elasticsearch

Trying to find a DYNAMIC plugin for reader named `ELASTICSEARCH'

FME API version of module 'ELASTICSEARCH' matches current internal version (3.8 20170808)

Creating reader for format: Elastic Elasticsearch

Trying to find a DYNAMIC plugin for reader named `ELASTICSEARCH'

FME API version of module 'ELASTICSEARCH' matches current internal version (3.8 20170808)

ELASTICSEARCH reader: Directive '_HOST' has value 'https://search-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com:443'

ELASTICSEARCH reader: CURL error was: Could not resolve host: https%3A%2F%2Fsearch-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com

ELASTICSEARCH reader: Failed to detect server version

ELASTICSEARCH reader: Directive '_RAW_JSON' has not been set.  Using default value 'NO'

ELASTICSEARCH reader: Directive '_NUM_FEATURES_PER_REQUEST' has not been set.  Using default value '100'

ELASTICSEARCH reader: Directive '_FME_JSON_ATTRIBUTE_FLATTENING_MODE' has not been set.  Using default value 'ALL_LEVELS'

ELASTICSEARCH reader: Opening index 'https://search-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com:443/index_list'

ELASTICSEARCH reader: CURL error was: Could not resolve host: https%3A%2F%2Fsearch-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com

ELASTICSEARCH reader: Could not resolve host: https%3A%2F%2Fsearch-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com

ELASTICSEARCH reader: Failed to open the dataset 'ES:'. Please ensure source data is valid and correct reader is selected

A fatal error has occurred. Check the logfile above for details

Failed to obtain any schemas from reader 'ELASTICSEARCH' from 1 datasets. This may be due to invalid datasets or format accessibility issues due to licensing, dependencies, or module loading. See logfile for more information

UniversalReader -- readSchema resulted in 0 schema features being returned

Reader Parameter(0) = >INDEX_LIST<

Reader Parameter(1) = >HOST<

Reader Parameter(2) = >https:<solidus><solidus>search-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com:443<

Reader Parameter(3) = >SETTINGS_DYNAMIC_SELECT_QUERY<

Reader Parameter(4) = >QUERY_INDICES<

Reader Parameter(5) = >FME_READ_SCHEMA<

Reader Parameter(6) = >YES<

Reader Parameter(7) = >_MERGE_SCHEMAS<

Reader Parameter(8) = >YES<

Reader Directive(0) = >RUNTIME_MACROS<

Reader Directive(1) = >HOST,https:<solidus><solidus>search-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com:443,SETTINGS_DYNAMIC_SELECT_QUERY,QUERY_INDICES,FME_READ_SCHEMA,YES,_MERGE_SCHEMAS,YES<

Configuration object contents:

FME Temporary Folder: C:\Users\oliverm\AppData\Local\Temp

mapping file Line 0: `APPLICATION' `FME_WORKBENCH'

mapping file Line 1: `AUTOREG' `YES'

mapping file Line 2: `LOG_ENCODING' `UTF-8'

mapping file Line 3: `LOG_MAX_FEATURES' `-1'

mapping file Line 4: `LOG_APPEND' `NO'

mapping file Line 5: `FME_HOME' `C:\Program Files\FME\'

mapping file Line 6: `FME_SESSION_USED' `TRUE'

mapping file Line 7: `R_11_METAFILE' `ELASTICSEARCH'

mapping file Line 8: `R_11_DATASET' `INDEX_LIST'

mapping file Line 9: `R_11_HOST' `https:<solidus><solidus>search-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com:443'

mapping file Line 10: `R_11_SETTINGS_DYNAMIC_SELECT_QUERY' `QUERY_INDICES'

mapping file Line 11: `R_11_FME_READ_SCHEMA' `YES'

mapping file Line 12: `R_11__MERGE_SCHEMAS' `YES'

mapping file Line 13: `R_11_HOST' `https:<solidus><solidus>search-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com:443'

mapping file Line 14: `R_11_SETTINGS_DYNAMIC_SELECT_QUERY' `QUERY_INDICES'

mapping file Line 15: `R_11_FME_READ_SCHEMA' `YES'

mapping file Line 16: `R_11__MERGE_SCHEMAS' `YES'

mapping file Line 17: `R_11_MERGE_SCHEMAS' `YES'

mapping file Line 18: `R_11_FME_JSON_ATTRIBUTE_FLATTENING_MODE' `'

mapping file Line 19: `SCHEMA_TIME' `YES'

mapping file Keyword: `SCHEMA_TIME'                            occurs 1 time(s)

mapping file Keyword: `LOG_ENCODING'                           occurs 1 time(s)

mapping file Keyword: `APPLICATION'                            occurs 1 time(s)

mapping file Keyword: `FME_HOME'                               occurs 1 time(s)

mapping file Keyword: `R_11_HOST'                              occurs 2 time(s)

mapping file Keyword: `FME_SESSION_USED'                       occurs 1 time(s)

mapping file Keyword: `R_11_FME_JSON_ATTRIBUTE_FLATTENING_MODE' occurs 1 time(s)

mapping file Keyword: `R_11_MERGE_SCHEMAS'                     occurs 1 time(s)

mapping file Keyword: `LOG_MAX_FEATURES'                       occurs 1 time(s)

mapping file Keyword: `LOG_APPEND'                             occurs 1 time(s)

mapping file Keyword: `R_11__MERGE_SCHEMAS'                    occurs 2 time(s)

mapping file Keyword: `R_11_DATASET'                           occurs 1 time(s)

mapping file Keyword: `R_11_FME_READ_SCHEMA'                   occurs 2 time(s)

mapping file Keyword: `R_11_SETTINGS_DYNAMIC_SELECT_QUERY'     occurs 2 time(s)

mapping file Keyword: `AUTOREG'                                occurs 1 time(s)

mapping file Keyword: `R_11_METAFILE'                          occurs 1 time(s)

UniversalReader -- readSchema resulted in 0 schema features being returned

Failed to obtain any schemas from reader 'ELASTICSEARCH' from 1 datasets. This may be due to invalid datasets or format accessibility issues due to licensing, dependencies, or module loading. See logfile for more information

Hello @olivermorris

I talked with development and we believe this is a PR for the ElasticSearch format. There are currently two possibilities that we foresee.

1. It appears we are requesting anonymous access and we do not have the options for credentials to be used for authentication.

2. The server may not be configured for anonymous access, might want to double check this. The default config does appear to be fairly permissive though, so this may not be the case. https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html

With that in mind, the first option seems to be the most plausible. We suspect AWS might need more info than we provide it as a client in the connection, in this case likely a signed request. I will file a PR for this, and link this posting to the PR.

Are you able to provide the following information:

- FME Version : Year and Build number

- Elastic Search version

These items along with the two logs you posted above will help the development team determine the cause

I believe @trentatsafe has the right of it.

Try adding:

:80

to the end of your dataset (ie.

search-dev-elasticsearch-z4b7vwlsqswv4uial343rdjccq.us-east-1.es.amazonaws.com:80)

This will tell the reader you are trying to connect over the default HTTP port (port 80), rather than the default Elasticsearch port (port 9200)

yes this did the trick, even though it is over port 443 (https) port 80 works for this. Thank you

Reply

Community Stats

Reply

Community Stats

Sign up

An FME Account is required to contribute

Login to the community

An FME Account is required to contribute

Scanning file for viruses.

This file cannot be downloaded