I will like to automate my weekly ftp download.
Mapinfo reader example url (as zip with multiple mapinfo files):
ftp:// ftp.data.com/Geodata_202103159999 .zip
'Geodata_' is always there,
'20210315' is the data that changes (every monday when data is downloaded)
'9999.zip' is always there
In my logik the easiest would be ftp://ftp.data.com/Geodata_*.zip
But it doesnt work. 'Geodata_' must be there can be multiple zip files on the server.
Any suggestions on this problem?
Best answer by chrisatsafe
When it comes to reading from an ftp, you need to specify the complete url so wildcards will not work. However, there are ways to work around this. I tend to do this by getting a list of the filenames from the FTP then feeding the url into the Feature Reader (see attached). You can build the wildcard logic into the Tester before the FeatureReader so that only the files you are interested are read into the workspace.
This example, uses a generic port on the feature reader so I've specified the attribute I wanted to have exposed in the FeatureReader's Attribute & Geometry Handling > Generic Port > Attributes to Expose parameter.
Others have been successful with the approach listed on this post as well.
+2When it comes to reading from an ftp, you need to specify the complete url so wildcards will not work. However, there are ways to work around this. I tend to do this by getting a list of the filenames from the FTP then feeding the url into the Feature Reader (see attached). You can build the wildcard logic into the Tester before the FeatureReader so that only the files you are interested are read into the workspace.
This example, uses a generic port on the feature reader so I've specified the attribute I wanted to have exposed in the FeatureReader's Attribute & Geometry Handling > Generic Port > Attributes to Expose parameter.
Others have been successful with the approach listed on this post as well.
+1@chrisatsafe But what if the first point of entry on your ftp is just a folder, that changes name every week, and you need to read a specific zip file within that folder? Basically I need to go one step deeper to read my zipped shapefile.
So do I do the list trick once more and can I do that from the first Tester?
I'm also not completely sure I understand what the @GetWord value does?
+2@chrisatsafe But what if the first point of entry on your ftp is just a folder, that changes name every week, and you need to read a specific zip file within that folder? Basically I need to go one step deeper to read my zipped shapefile.
So do I do the list trick once more and can I do that from the first Tester?
I'm also not completely sure I understand what the @GetWord value does?
Yes exactly. The example in that workspace is just going for a file but if you need to list the folder and then list the files for that folder you can essentially do the same logic twice. If you set the first FTPCaller to ftp://ftp2.census.gov/acs2002/ you'll be provided with a list of everything in that directory which you can then filter for in the Tester (for example URL = Core_Tables).
Then In the second FTPCaller, you can then set the URL to ftp://ftp2.census.gov/acs2002/@Value(URL)/ and it will return the list of files in that directory
+2Yes exactly. The example in that workspace is just going for a file but if you need to list the folder and then list the files for that folder you can essentially do the same logic twice. If you set the first FTPCaller to ftp://ftp2.census.gov/acs2002/ you'll be provided with a list of everything in that directory which you can then filter for in the Tester (for example URL = Core_Tables).
Then In the second FTPCaller, you can then set the URL to ftp://ftp2.census.gov/acs2002/@Value(URL)/ and it will return the list of files in that directory
With regards to the GetWord function:
"GetWord(string,index)
Returns the indexth word in string . If indexis a negative integer, GetWord() returns the indexth word from the end of string. If there is no word at index, an empty string is returned. Words in string must be delineated by blank spaces (space, tab, return carriage, and others)."
The reason it is invalid in your case is because the attribute is now called URL so it needs to be @GetWord(@Value(URL),-1) rather than @Value(_list)
+1Yes exactly. The example in that workspace is just going for a file but if you need to list the folder and then list the files for that folder you can essentially do the same logic twice. If you set the first FTPCaller to ftp://ftp2.census.gov/acs2002/ you'll be provided with a list of everything in that directory which you can then filter for in the Tester (for example URL = Core_Tables).
Then In the second FTPCaller, you can then set the URL to ftp://ftp2.census.gov/acs2002/@Value(URL)/ and it will return the list of files in that directory
Thank you so much Chris, this solves a huge problem for me :-)
When it comes to reading from an ftp, you need to specify the complete url so wildcards will not work. However, there are ways to work around this. I tend to do this by getting a list of the filenames from the FTP then feeding the url into the Feature Reader (see attached). You can build the wildcard logic into the Tester before the FeatureReader so that only the files you are interested are read into the workspace.
This example, uses a generic port on the feature reader so I've specified the attribute I wanted to have exposed in the FeatureReader's Attribute & Geometry Handling > Generic Port > Attributes to Expose parameter.
Others have been successful with the approach listed on this post as well.
HI Chris, Is there a difference between doing this in FME workbench vs. setting this up in FME Server?
I got this running smooth on my local workbench, but it is timing-out when I run it on the Server. I am getting this error:
"FTP/FTPS/SFTP transfer error: Timeout was reached"
"Termination Message: 'FTPCaller output a <Rejected> feature."
+2HI Chris, Is there a difference between doing this in FME workbench vs. setting this up in FME Server?
I got this running smooth on my local workbench, but it is timing-out when I run it on the Server. I am getting this error:
"FTP/FTPS/SFTP transfer error: Timeout was reached"
"Termination Message: 'FTPCaller output a <Rejected> feature."
Hi @ohyacinthe ,
Hmm, it should be the same. Do you have workbench installed locally on the server machine? Could you try from there?
Only thing that comes to mind for me would be any network related differences preventing the connection (proxy, firewall, other).
HI Chris, Is there a difference between doing this in FME workbench vs. setting this up in FME Server?
I got this running smooth on my local workbench, but it is timing-out when I run it on the Server. I am getting this error:
"FTP/FTPS/SFTP transfer error: Timeout was reached"
"Termination Message: 'FTPCaller output a <Rejected> feature."
Thanks Chris!
Unfortunately y workbench is definitely NOT installed on my server machine. From previous versions of my team’s FME Server, I remember that developing workflows from the server machine’s workbench makes publishing workspaces to the server a lot easier.
1. I’ll look to configure known office proxy settings in my local workspace and try it again on the server.
2. I’m going to ask my team if there is a workbench instance installed on our FME server and start from scratch if I’m still stuck.
Thanks,
Keye
When it comes to reading from an ftp, you need to specify the complete url so wildcards will not work. However, there are ways to work around this. I tend to do this by getting a list of the filenames from the FTP then feeding the url into the Feature Reader (see attached). You can build the wildcard logic into the Tester before the FeatureReader so that only the files you are interested are read into the workspace.
This example, uses a generic port on the feature reader so I've specified the attribute I wanted to have exposed in the FeatureReader's Attribute & Geometry Handling > Generic Port > Attributes to Expose parameter.
Others have been successful with the approach listed on this post as well.
Hi Chris. Hi everyone.
Thanks for the workbench. I have a similar issue. On the ftp, 2 directories. One for the exchange, another for the results.
Url are something like : ftp://.../ftpprivate/user/echange and ftp://.../ftpprivate/user/resultat
How may I list files from the both directories ? How may I connect the ftpcaller dfirectly to echange ? I got a bunch of error when I indicate ftp://.../ftpprivate/user/echange in the url of the ftp caller.
Thanks in advance.
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
