Question

How to download Netcdf files from a Proxy Server that gets data from a Thredds server. Using FME 2020.

  • 26 April 2021
  • 7 replies
  • 6 views
monicarm
Rank
Badge

I am using

  1. Httpcaller to read the catalog. Getting the response body.
  2. then using a HTMLExtractor but the links are not coming as a http they are like this: <a href='catalog.html?dataset=xxxx/xxxxx/Archive/name1.nc'><tt>name1.nc</tt></a></td>
  3. then using a ListExploder to get just the links like: catalog.html?dataset=xxxx/xxxxx/Archive/name1.nc
  4. then I was creating a URL attribute to concatenate the URL( http:////xxxx.yyyyyy.xx:8083/thredds/catalog/xxxxx/xxxxx/Archive/catalog.html? )and the list ( catalog.html?dataset=xxxx/xxxxx/Archive/name1.nc) to have an URL to downlaod the data as: http://xxxx.yyyyyy.xx:8083/thredds/catalog/xxxxx/xxxxx/Archive/catalog.html?dataset=xxxxx/folder/Archive/name1.nc
  5. But I donæt manage to download it.I get the following error:

HTTP/FTP Transfer: Failed to create file 'R:\\xxxx\\Downloaded\\yyyy\\' for resource download 'http://xxxxxdocker02.dddddd.no:8083/thredds/catalog/xxxxe/xxxxxx/xxxx/catalog.html?dataset=dddddd/dddddd/Archive/name.nc'

 

I have around 65 netcdf files to download. I am also attaching a picture of the fmw.

 

Can someone help me with some hints? I will be very thankful. Or if there is any way of downloading directly from the Thredds server?

Capture_fmw 

 

7 replies

david_r
Rank
Userlevel 4

You've tagged it with FME Server, so I'm going to assume that the error occurs there. Have you verified that the FME Server Engine Service is running with an AD account, and not Local System? The default Local System account does not have access to the network, even over mapped network drives.

See also http://docs.safe.com/fme/html/FME_Server_Documentation/AdminGuide/Running_FME_Server_Engines_Under_Different_Account.htm

monicarm
Rank
Badge

You've tagged it with FME Server, so I'm going to assume that the error occurs there. Have you verified that the FME Server Engine Service is running with an AD account, and not Local System? The default Local System account does not have access to the network, even over mapped network drives.

See also http://docs.safe.com/fme/html/FME_Server_Documentation/AdminGuide/Running_FME_Server_Engines_Under_Different_Account.htm

Sorry, I didn't saw thay I was tagging FME server. I am running locally the FME. We don't have FME server. We are using a proxy server to resemble the catalog from the Thredds server.

david_r
Rank
Userlevel 4

Sorry, I didn't saw thay I was tagging FME server. I am running locally the FME. We don't have FME server. We are using a proxy server to resemble the catalog from the Thredds server.

Have you tried to manually create the folder that FME is unable to create? I'm assuming that FME is being executed interactively, and not through e.g. the Windows task manager.

monicarm
Rank
Badge

Sorry, I didn't saw thay I was tagging FME server. I am running locally the FME. We don't have FME server. We are using a proxy server to resemble the catalog from the Thredds server.

My problem is that FME doesn't recognize the URL to download the files. I have concatened two values to make the URL for downloading. If I just copy for example this to my browser, it downloads the file but it doesn't work using FME.

http://xxxxxdocker02.dddddd.no:8083/thredds/catalog/xxxxe/xxxxxx/xxxx/catalog.html?dataset=dddddd/dddddd/Archive/name1.nc

I have 65 files to download so wanted to use FME for the task. But I am stuck in the process.

I am not sure if it is ok to use the feature Reader or another HttpCaller after the stringconcatenator. I was trying also to use a SubstringExtractor to get the name of the file and using a HttpCaller but it doesn't work.

 

I found this old post, and I was trying to do more or less the same, but didn't manage the download.

Download multiple files using HTTPCaller from URLs stored in CSV (safe.com)

 

nampreetatsafe
Rank
Userlevel 1
Badge +10

Sorry, I didn't saw thay I was tagging FME server. I am running locally the FME. We don't have FME server.  We are using a proxy server to resemble the catalog from the Thredds server.

1. As @david_r​ mentions, the error you got seems to indicate an issue with creating the file or directory.  Can you confirm that you have access to that path and that you have permissions to create directories there? there.

 

2. Otherwise the error you mentioned in your description above is trying to hit the file named "name.nc" rather than "name1.nc", is it possible that the URL isn't constructed correctly?  Is name.nc a valid file name? HTTP/FTP Transfer: Failed to create file 

HTTP/FTP Transfer: Failed to create file 'R:\xxxx\Downloaded\yyyy\' for resource download 'http://xxxxxdocker02.dddddd.no:8083/thredds/catalog/xxxxe/xxxxxx/xxxx/catalog.html?dataset=dddddd/dddddd/Archive/name.nc'

3. The way I'd troubleshoot this, if you already haven't done so,  is to work backwards.  Try using an HTTPCaller with the Request URL parameter hardcoded with the URL that works in your browser (http://xxxx.yyyyyy.xx:8083/thredds/catalog/xxxxx/xxxxx/Archive/catalog.html?dataset=xxxxx/folder/Archive/name1.nc), along with hardcoding the Output Filename.  I've attached a workspace that hits a public URL for a netcdf file as a quick example.  

  • If this works with your URL, try the Output filename construct that you were trying with the hardcoded Request URL.
monicarm
Rank
Badge

Sorry, I didn't saw thay I was tagging FME server. I am running locally the FME. We don't have FME server. We are using a proxy server to resemble the catalog from the Thredds server.

Thanks for your reply @nampreetatsafe​ 

I have to update my FME to try to open the workspace. Could you just post a picture of your workspace meanwhile I fix the updatings.

 

I manage to download the URL but I can't give the files their names. As you can see here, they are dowloded as "http_download_1619597740790_9280.nc"

 

HTTPCaller_2 (HTTPFactory): HTTP/FTP Transfer: Downloading resource 'http://xxxxxdocker02.dddddd.no:8083/thredds/catalog/xxxx/xxxxxx/xxxx/catalog.html?dataset=senorge/seNorge_2018/Archive/seNorge2018_2021.nc' to local file 'R:\\foldername\\subfolder\\SeNorge\\http_download_1619597740790_9280'

 

_results has the URL Links (65 links)

http://xxxxxdocker02.dddddd.no:8083/thredds/catalog/xxxx/xxxxxx/xxxx/catalog.html?dataset=senorge/seNorge_2018/Archive/seNorge2018_2021.nc

http://xxxxxdocker02.dddddd.no:8083/thredds/catalog/xxxx/xxxxxx/xxxx/catalog.html?dataset=senorge/seNorge_2018/Archive/seNorge2018_2020.nc

http://xxxxxdocker02.dddddd.no:8083/thredds/catalog/xxxx/xxxxxx/xxxx/catalog.html?dataset=senorge/seNorge_2018/Archive/seNorge2018_2019.nc

This is how the httpcaller looks now:

httpcallerThis is how I have the model now:

download_nc 

 

Thanks for the help,

 

monicarm
Rank
Badge

Sorry, I didn't saw thay I was tagging FME server. I am running locally the FME. We don't have FME server. We are using a proxy server to resemble the catalog from the Thredds server.

Hi @nampreetatsafe​ 

I have managed to update my FME version to 2021 now and I opened the example you posted. My problem is that I have 65 nc files to download. And I can't manage to download with their names. I was also trying saving as a zip file but it seems that I can't download all the nc files as a zip.

 

Any other idea of how to solve this issue?

 

 

Reply


Community Stats

28,723
Posts
109,511
Replies
36,559
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.