Question

Help with FME Server, HTTP Caller, and CKAN

  • 25 September 2019
  • 7 replies
  • 18 views
andrewspencer
Badge

Since updating to FME Server 2019 a number of workspaces reading or writing via the web seem to be problematic.

In this particular instance, I have a workspace that writes to CKAN at data.gov.au. Desktop works fine, but since updating our server it no longer works.

My workspace is modeled on this one (with a few minor changes to reflect some updates to the api) :

https://github.com/datagovau/ckan-api-examples/blob/master/FME/CKANUpdate.fmw

The first issue is that my HTTP Caller in desktop has the response body encoding set to 'auto-detect'. I found that Server fails with this, and after trial and error found it works when set to unicode 8-bit.

However the more serious problem is that I can't get it to transfer my file. The error looks like this:

63HTTPCaller_2__3 (HTTPFactory): HTTP/FTP transfer error: 'Failed sending data to the peer'64HTTPCaller_2__3 (HTTPFactory): Please ensure that your network connection is properly set up65HTTPCaller_2__3 (HTTPFactory): Please ensure the following proxy information is correct. The current proxy is: 'myProxy:8080'

 

It would appear that the proxy is correct as the workspace is able to download the json file only two lines earlier:

 

HTTPCaller (HTTPFactory): HTTP transfer summary - status code: 200, download size: '8009 bytes', DNS lookup time: '0.00102 seconds', total transfer time: '0.180459 seconds'

 

If anyone has any suggestions, they would be greatly appreciated....

Thanks

Andrew

7 replies

virtualcitymatt
Rank
Userlevel 4
Badge +26

One tip I have is to turn on FME Debug logging, it's pretty helpful I've found with the HTTPCaller - Especially when a proxy is in use.

 

It's possible that a bug/issue has been fixed meaning that now data is trying to go through the configured proxy whereas before it didn't. It is strange that the issues are only with FME Server. Could it be a firewall or CORS issue?
andrewspencer
Badge

Early days, but I think I may have found a solution. It looks like a bug with libcurl 7.65.2.0 which is installed with Server. Our Desktop is using libcurl 7.61.1.0. I replaced the Server version with the Desktop version and all appears to be well. This might be something Safe might want to have a look into maybe....

rylanatsafe
Rank
Badge +11

Early days, but I think I may have found a solution. It looks like a bug with libcurl 7.65.2.0 which is installed with Server. Our Desktop is using libcurl 7.61.1.0. I replaced the Server version with the Desktop version and all appears to be well. This might be something Safe might want to have a look into maybe....

Hi @andrewspencer, this issue was recently brought to my attention – have there been any problems since replacing libcurl in FME Server?

And can you please let us know the build numbers of FME Server and FME Desktop that you are using? (build numbers are represented with five digits – for example, 19642)

 

Looking through the commit logs, we upgraded to 7.65.2 for approximately Build 19617 and Newer, (and this should have been) across all platforms and Desktop/Server.

The workspace you linked (GitHub) appears to be authored with FME 2015 (Build 15244). Did you use that workspace as a starting template? If so, I am wondering if all of the transformers were upgraded to the latest versions in 2019? (the HTTPCaller in particular has an upgrade available..)

 

If you can share the offending workspace with us that might help us narrow down the issue. You can submit to our Support Team if it contains sensitive information.

I'm hopeful that we can evaluate if this library upgrade will impact any other workflows. I'm sorry for the inconvenience it has caused you! (Great troubleshooting to find and replace that component, by the way!)

andrewspencer
Badge

Hi @andrewspencer, this issue was recently brought to my attention – have there been any problems since replacing libcurl in FME Server?

And can you please let us know the build numbers of FME Server and FME Desktop that you are using? (build numbers are represented with five digits – for example, 19642)

 

Looking through the commit logs, we upgraded to 7.65.2 for approximately Build 19617 and Newer, (and this should have been) across all platforms and Desktop/Server.

The workspace you linked (GitHub) appears to be authored with FME 2015 (Build 15244). Did you use that workspace as a starting template? If so, I am wondering if all of the transformers were upgraded to the latest versions in 2019? (the HTTPCaller in particular has an upgrade available..)

 

If you can share the offending workspace with us that might help us narrow down the issue. You can submit to our Support Team if it contains sensitive information.

I'm hopeful that we can evaluate if this library upgrade will impact any other workflows. I'm sorry for the inconvenience it has caused you! (Great troubleshooting to find and replace that component, by the way!)

Hi @rylanatsafe, thanks for your reply. I've attached a workspace (we have 6 of these running on a daily schedule). Our API key and source file location have been removed, but the end result of this is here: https://data.gov.au/data/organization/nationalnativetitletribunal. The workspace on GitHub is a little old, and we need to do a little more work to flatten the JSON than we did in 2015. All transformers in our workspaces have been have been updated.

Our Desktop build is 19608, our server is 19617.

Since replacing libcurl I've had no issues, and it also resolved the response unicode/autodetect problem.

On the surface, it seemed that Server was trying to switch the traffic to http/2 before failing, whilst Desktop was happy with http/1.1. Thanks again for your interest, and let me know if you need any more info.

NTDA_Schedule_to_Data_Gov_Au_copy.fmw.

rylanatsafe
Rank
Badge +11

Hi @rylanatsafe, thanks for your reply. I've attached a workspace (we have 6 of these running on a daily schedule). Our API key and source file location have been removed, but the end result of this is here: https://data.gov.au/data/organization/nationalnativetitletribunal. The workspace on GitHub is a little old, and we need to do a little more work to flatten the JSON than we did in 2015. All transformers in our workspaces have been have been updated.

Our Desktop build is 19608, our server is 19617.

Since replacing libcurl I've had no issues, and it also resolved the response unicode/autodetect problem.

On the surface, it seemed that Server was trying to switch the traffic to http/2 before failing, whilst Desktop was happy with http/1.1. Thanks again for your interest, and let me know if you need any more info.

NTDA_Schedule_to_Data_Gov_Au_copy.fmw.

Thanks @andrewspencer, I'll take a closer look at the workspace...

In the meantime, are you able to try in Build 19617 of FME Desktop? I'm curious if the issue is unique to FME Server or to the Build. I expect that you will run into the same issue with a newer version of FME Desktop, but would like to confirm that suspicion.

We have 19642 currently available on our Downloads Page, but if you need 19617 (Win64) you can grab that from our FTP here: ftp://ftp.safe.com/fmebuilds/fme-desktop-b19617-win-x64.msi (please note that this link is temporary)

andrewspencer
Badge

Thanks @andrewspencer, I'll take a closer look at the workspace...

In the meantime, are you able to try in Build 19617 of FME Desktop? I'm curious if the issue is unique to FME Server or to the Build. I expect that you will run into the same issue with a newer version of FME Desktop, but would like to confirm that suspicion.

We have 19642 currently available on our Downloads Page, but if you need 19617 (Win64) you can grab that from our FTP here: ftp://ftp.safe.com/fmebuilds/fme-desktop-b19617-win-x64.msi (please note that this link is temporary)

Hi @rylanatsafe. Luckily we had another machine with Desktop build 19617 so I gave it a run there.

User-Agent: FME/2019.7.37.19617 libcurl/7.65.2 Schannel zlib/1.2.11 WinIDN libssh2/1.9.0 nghttp2/1.38.0

Our workspace worked fine! Having said that, it does appear to stick with HTTP/1.1 the whole way through.

Edited to add more info:

So I reverted Server back to libcurl 7.65.2 just so I could verify that I can reproduce the error and save some logs (which I hadn't done).

So this is from Server:

User-Agent: FME/2019.7.37.19617  libcurl/7.65.2 Schannel zlib/1.2.11 WinIDN libssh2/1.9.0 nghttp2/1.38.0

Looks identical to Desktop doesn't it? But then it goes on to do things a bit differently:

2019-10-17 09:47:27|   0.9|  0.0|INFORM|HTTPCaller (HTTPFactory): HTTP info: schannel: ALPN, offering h2
2019-10-17 09:47:27|   0.9|  0.0|INFORM|HTTPCaller (HTTPFactory): HTTP info: schannel: ALPN, offering http/1.1
2019-10-17 09:47:27|   0.9|  0.0|INFORM|HTTPCaller (HTTPFactory): HTTP info: CONNECT phase completed!
2019-10-17 09:47:27|   0.9|  0.0|INFORM|HTTPCaller (HTTPFactory): HTTP info: CONNECT phase completed!
2019-10-17 09:47:27|   0.9|  0.0|INFORM|HTTPCaller (HTTPFactory): HTTP info: schannel: ALPN, server accepted to use h2
2019-10-17 09:47:27|   0.9|  0.0|INFORM|HTTPCaller (HTTPFactory): HTTP info: Using HTTP2, server supports multi-use
2019-10-17 09:47:27|   0.9|  0.0|INFORM|HTTPCaller (HTTPFactory): HTTP info: Connection state changed (HTTP/2 confirmed)

This then fails with:

2019-10-17 09:47:27|   0.9|  0.0|WARN  |JSONFlattener_2 (JSONQueryFactory): The JSON data is incomplete: Unexpectedly encountered the end of JSON data
2019-10-17 09:47:27|   0.9|  0.0|WARN  |JSONFlattener_2 (JSONQueryFactory): The data does not contain any JSON text

Setting the HTTPCaller to expect UTF-8 and republishing gets us past this point, but then the file upload fails with:

2019-10-17 09:49:26|   1.0|  0.0|INFORM|HTTPCaller_KMZ_last_modified (HTTPFactory): HTTP/FTP Transfer: Uploading data to 'https://data.gov.au/api/3/action/resource_update'
2019-10-17 09:49:26|   1.1|  0.1|INFORM|HTTPCaller_KMZ_last_modified (HTTPFactory): HTTP info: Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
2019-10-17 09:49:26|   1.1|  0.0|INFORM|HTTPCaller_KMZ_last_modified (HTTPFactory): HTTP info: schannel: timed out sending data (bytes sent: 0)
2019-10-17 09:49:26|   1.1|  0.0|INFORM|HTTPCaller_KMZ_last_modified (HTTPFactory): HTTP info: Failed sending HTTP2 data
2019-10-17 09:49:26|   1.1|  0.0|INFORM|HTTPCaller_KMZ_last_modified (HTTPFactory): HTTP info: schannel: timed out sending data (bytes sent: 0)
2019-10-17 09:49:26|   1.1|  0.0|INFORM|HTTPCaller_KMZ_last_modified (HTTPFactory): HTTP info: Failed sending HTTP2 data
2019-10-17 09:49:26|   1.1|  0.0|INFORM|HTTPCaller_KMZ_last_modified (HTTPFactory): HTTP info: Connection #0 to host MyProxy left intact
2019-10-17 09:49:26|   1.1|  0.0|ERROR |HTTPCaller_KMZ_last_modified (HTTPFactory): HTTP/FTP transfer error: 'Failed sending data to the peer'
2019-10-17 09:49:26|   1.1|  0.0|ERROR |HTTPCaller_KMZ_last_modified (HTTPFactory): Please ensure that your network connection is properly set up

FME Server re-downgraded to libcurl 7.61.1.0 and everything is working again...

rylanatsafe
Rank
Badge +11

Hi @rylanatsafe. Luckily we had another machine with Desktop build 19617 so I gave it a run there.

User-Agent: FME/2019.7.37.19617 libcurl/7.65.2 Schannel zlib/1.2.11 WinIDN libssh2/1.9.0 nghttp2/1.38.0

Our workspace worked fine! Having said that, it does appear to stick with HTTP/1.1 the whole way through.

Edited to add more info:

So I reverted Server back to libcurl 7.65.2 just so I could verify that I can reproduce the error and save some logs (which I hadn't done).

So this is from Server:

User-Agent: FME/2019.7.37.19617  libcurl/7.65.2 Schannel zlib/1.2.11 WinIDN libssh2/1.9.0 nghttp2/1.38.0

Looks identical to Desktop doesn't it? But then it goes on to do things a bit differently:

2019-10-17 09:47:27|   0.9|  0.0|INFORM|HTTPCaller (HTTPFactory): HTTP info: schannel: ALPN, offering h2
2019-10-17 09:47:27|   0.9|  0.0|INFORM|HTTPCaller (HTTPFactory): HTTP info: schannel: ALPN, offering http/1.1
2019-10-17 09:47:27|   0.9|  0.0|INFORM|HTTPCaller (HTTPFactory): HTTP info: CONNECT phase completed!
2019-10-17 09:47:27|   0.9|  0.0|INFORM|HTTPCaller (HTTPFactory): HTTP info: CONNECT phase completed!
2019-10-17 09:47:27|   0.9|  0.0|INFORM|HTTPCaller (HTTPFactory): HTTP info: schannel: ALPN, server accepted to use h2
2019-10-17 09:47:27|   0.9|  0.0|INFORM|HTTPCaller (HTTPFactory): HTTP info: Using HTTP2, server supports multi-use
2019-10-17 09:47:27|   0.9|  0.0|INFORM|HTTPCaller (HTTPFactory): HTTP info: Connection state changed (HTTP/2 confirmed)

This then fails with:

2019-10-17 09:47:27|   0.9|  0.0|WARN  |JSONFlattener_2 (JSONQueryFactory): The JSON data is incomplete: Unexpectedly encountered the end of JSON data
2019-10-17 09:47:27|   0.9|  0.0|WARN  |JSONFlattener_2 (JSONQueryFactory): The data does not contain any JSON text

Setting the HTTPCaller to expect UTF-8 and republishing gets us past this point, but then the file upload fails with:

2019-10-17 09:49:26|   1.0|  0.0|INFORM|HTTPCaller_KMZ_last_modified (HTTPFactory): HTTP/FTP Transfer: Uploading data to 'https://data.gov.au/api/3/action/resource_update'
2019-10-17 09:49:26|   1.1|  0.1|INFORM|HTTPCaller_KMZ_last_modified (HTTPFactory): HTTP info: Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
2019-10-17 09:49:26|   1.1|  0.0|INFORM|HTTPCaller_KMZ_last_modified (HTTPFactory): HTTP info: schannel: timed out sending data (bytes sent: 0)
2019-10-17 09:49:26|   1.1|  0.0|INFORM|HTTPCaller_KMZ_last_modified (HTTPFactory): HTTP info: Failed sending HTTP2 data
2019-10-17 09:49:26|   1.1|  0.0|INFORM|HTTPCaller_KMZ_last_modified (HTTPFactory): HTTP info: schannel: timed out sending data (bytes sent: 0)
2019-10-17 09:49:26|   1.1|  0.0|INFORM|HTTPCaller_KMZ_last_modified (HTTPFactory): HTTP info: Failed sending HTTP2 data
2019-10-17 09:49:26|   1.1|  0.0|INFORM|HTTPCaller_KMZ_last_modified (HTTPFactory): HTTP info: Connection #0 to host MyProxy left intact
2019-10-17 09:49:26|   1.1|  0.0|ERROR |HTTPCaller_KMZ_last_modified (HTTPFactory): HTTP/FTP transfer error: 'Failed sending data to the peer'
2019-10-17 09:49:26|   1.1|  0.0|ERROR |HTTPCaller_KMZ_last_modified (HTTPFactory): Please ensure that your network connection is properly set up

FME Server re-downgraded to libcurl 7.61.1.0 and everything is working again...

Wow, this is all very strange @andrewspencer, but I am glad you have a workaround for the time being! 

I have opened C149150 for you, so that a member of our Technical Experts Team can continue to look into this further.

Thank you very much for including the extra information from logs, and for going through the effort to test on Desktop 19617.

Reply


Community Stats

28,688
Posts
109,430
Replies
36,542
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.