Solved

How can i use wildcard in reader, when reading from a FTP server

L

I will like to automate my weekly ftp download.

Mapinfo reader example url (as zip with multiple mapinfo files):

ftp:// ftp.data.com/Geodata_202103159999 .zip

 

'Geodata_' is always there,

'20210315' is the data that changes (every monday when data is downloaded)

'9999.zip' is always there

 

In my logik the easiest would be ftp://ftp.data.com/Geodata_*.zip

But it doesnt work. 'Geodata_' must be there can be multiple zip files on the server.

 

Any suggestions on this problem?

icon

Best answer by chrisatsafe 17 March 2021, 19:57

View original

10 replies

chrisatsafe
Rank
Badge +2

When it comes to reading from an ftp, you need to specify the complete url so wildcards will not work. However, there are ways to work around this. I tend to do this by getting a list of the filenames from the FTP then feeding the url into the Feature Reader (see attached). You can build the wildcard logic into the Tester before the FeatureReader so that only the files you are interested are read into the workspace.

2021-03-17_11-54-50This example, uses a generic port on the feature reader so I've specified the attribute I wanted to have exposed in the FeatureReader's Attribute & Geometry Handling > Generic Port > Attributes to Expose parameter.

 

Others have been successful with the approach listed on this post as well.

L

Perfect, that works! Thanks!

L
Badge +1

@chrisatsafe​ ​ But what if the first point of entry on your ftp is just a folder, that changes name every week, and you need to read a specific zip file within that folder? Basically I need to go one step deeper to read my zipped shapefile.

So do I do the list trick once more and can I do that from the first Tester?

I'm also not completely sure I understand what the @GetWord value does?image

chrisatsafe
Rank
Badge +2

@chrisatsafe​ ​ But what if the first point of entry on your ftp is just a folder, that changes name every week, and you need to read a specific zip file within that folder? Basically I need to go one step deeper to read my zipped shapefile.

So do I do the list trick once more and can I do that from the first Tester?

I'm also not completely sure I understand what the @GetWord value does?image

Yes exactly. The example in that workspace is just going for a file but if you need to list the folder and then list the files for that folder you can essentially do the same logic twice. If you set the first FTPCaller to ftp://ftp2.census.gov/acs2002/ you'll be provided with a list of everything in that directory which you can then filter for in the Tester (for example URL = Core_Tables).

Then In the second FTPCaller, you can then set the URL to ftp://ftp2.census.gov/acs2002/@Value(URL)/ and it will return the list of files in that directoryimage

chrisatsafe
Rank
Badge +2

Yes exactly. The example in that workspace is just going for a file but if you need to list the folder and then list the files for that folder you can essentially do the same logic twice. If you set the first FTPCaller to ftp://ftp2.census.gov/acs2002/ you'll be provided with a list of everything in that directory which you can then filter for in the Tester (for example URL = Core_Tables).

Then In the second FTPCaller, you can then set the URL to ftp://ftp2.census.gov/acs2002/@Value(URL)/ and it will return the list of files in that directoryimage

With regards to the GetWord function:

"GetWord(string,index)

Returns the indexth word in string . If indexis a negative integer, GetWord() returns the indexth word from the end of string. If there is no word at index, an empty string is returned. Words in string must be delineated by blank spaces (space, tab, return carriage, and others)."

 

The reason it is invalid in your case is because the attribute is now called URL so it needs to be @GetWord(@Value(URL),-1) rather than @Value(_list)

image

L
Badge +1

Yes exactly. The example in that workspace is just going for a file but if you need to list the folder and then list the files for that folder you can essentially do the same logic twice. If you set the first FTPCaller to ftp://ftp2.census.gov/acs2002/ you'll be provided with a list of everything in that directory which you can then filter for in the Tester (for example URL = Core_Tables).

Then In the second FTPCaller, you can then set the URL to ftp://ftp2.census.gov/acs2002/@Value(URL)/ and it will return the list of files in that directoryimage

Thank you so much Chris, this solves a huge problem for me :-)

O

When it comes to reading from an ftp, you need to specify the complete url so wildcards will not work. However, there are ways to work around this. I tend to do this by getting a list of the filenames from the FTP then feeding the url into the Feature Reader (see attached). You can build the wildcard logic into the Tester before the FeatureReader so that only the files you are interested are read into the workspace.

2021-03-17_11-54-50This example, uses a generic port on the feature reader so I've specified the attribute I wanted to have exposed in the FeatureReader's Attribute & Geometry Handling > Generic Port > Attributes to Expose parameter.

 

Others have been successful with the approach listed on this post as well.

HI Chris, Is there a difference between doing this in FME workbench vs. setting this up in FME Server?

 

I got this running smooth on my local workbench, but it is timing-out when I run it on the Server. I am getting this error:

"FTP/FTPS/SFTP transfer error: Timeout was reached"

"Termination Message: 'FTPCaller output a <Rejected> feature."

chrisatsafe
Rank
Badge +2

HI Chris, Is there a difference between doing this in FME workbench vs. setting this up in FME Server?

 

I got this running smooth on my local workbench, but it is timing-out when I run it on the Server. I am getting this error:

"FTP/FTPS/SFTP transfer error: Timeout was reached"

"Termination Message: 'FTPCaller output a <Rejected> feature."

Hi @ohyacinthe​ ,

Hmm, it should be the same. Do you have workbench installed locally on the server machine? Could you try from there?

 

Only thing that comes to mind for me would be any network related differences preventing the connection (proxy, firewall, other).

O

HI Chris, Is there a difference between doing this in FME workbench vs. setting this up in FME Server?

 

I got this running smooth on my local workbench, but it is timing-out when I run it on the Server. I am getting this error:

"FTP/FTPS/SFTP transfer error: Timeout was reached"

"Termination Message: 'FTPCaller output a <Rejected> feature."

Thanks Chris!

Unfortunately y workbench is definitely NOT installed on my server machine. From previous versions of my team’s FME Server, I remember that developing workflows from the server machine’s workbench makes publishing workspaces to the server a lot easier.

1. I’ll look to configure known office proxy settings in my local workspace and try it again on the server.

2. I’m going to ask my team if there is a workbench instance installed on our FME server and start from scratch if I’m still stuck.

Thanks,

Keye

plm

When it comes to reading from an ftp, you need to specify the complete url so wildcards will not work. However, there are ways to work around this. I tend to do this by getting a list of the filenames from the FTP then feeding the url into the Feature Reader (see attached). You can build the wildcard logic into the Tester before the FeatureReader so that only the files you are interested are read into the workspace.

2021-03-17_11-54-50This example, uses a generic port on the feature reader so I've specified the attribute I wanted to have exposed in the FeatureReader's Attribute & Geometry Handling > Generic Port > Attributes to Expose parameter.

 

Others have been successful with the approach listed on this post as well.

Hi Chris. Hi everyone.

Thanks for the workbench. I have a similar issue. On the ftp, 2 directories. One for the exchange, another for the results.

Url are something like : ftp://.../ftpprivate/user/echange and ftp://.../ftpprivate/user/resultat

How may I list files from the both directories ? How may I connect the ftpcaller dfirectly to echange ? I got a bunch of error when I indicate ftp://.../ftpprivate/user/echange in the url of the ftp caller.

 

Thanks in advance.

 

Reply


Community Stats

28,723
Posts
109,511
Replies
36,560
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.