Skip to main content
Solved

Use PythonCaller to get list of files in directory on FTP site


Did this help you find an answer to your question?
Show first post

44 replies

david_r
Evangelist
  • January 9, 2017
david_r wrote:
According to the documentation: "This method returns None" :-)

 

 

So basically, you cannot use this method as a replacement for nlst() without any further 

 

changes. 

 

 

You can, however, specify a callback function for dir() which will then be called once for every line returned by the ftp server. There's a short example here. You will then have to split up each line yourself to get the filenames, dates etc.

 

Rather than using ftp.nlst() you can do something like:

 

lines = []
ftp.retrlines('LIST', lines.append)
You will then have a Python list containing all the text returned by dir() which you can then parse into filename, filesize and timestamp and sort accordingly before deciding exactly which file to download.

 

 

Just be aware that the exact formatting of these lines can vary quite a bit between different ftp servers. Hope this helps.

 


tim_wood
Contributor
Forum|alt.badge.img+8
  • Author
  • Contributor
  • January 9, 2017
james_rutter wrote:
I've implemented the Python code from @david_r which works fine. However, if I exchange ftplib.nlst() for ftplib.dir() to give me a full directory listing (I need to file modified date) FME gives me the error in the screenshot below. Spent all afternoon trying to get around this without success. Any ideas??

 

Thanks

 

James

 

I went to the university of Google and studied "python ftp nlst vs dir" and found this...

 

http://stackoverflow.com/questions/111954/using-pythons-ftplib-to-get-a-directory-listing-portably

 

So I changed the line that reads "CSVlist = ftp.nlst()" to "ftp.dir(CSVlist.append)" and bingo.

 

After ftp.quit(), I run the code shown below (with proper indentations of course). So if you use this, you'll get a list of features like this...

 

0 20170109152813 -rw-rw-rw- 1 user group 31591 Jul 9 2016 filename.csv

 

 

# if CSVlist is empty

 

if not CSVlist:

 

# set the value of FMEFTPError to 'No CSV files to download'

 

feature.setAttribute('FMEFTPError', 'No CSV files to download')

 

# write that feature back to FME (there'll be one feature output from the PythonCaller)

 

self.pyoutput(feature)

 

else:

 

# loop through each item in the CSVlist array

 

for v in CSVlist:

 

# write out a feature for each item in the array and put the value of the array item in the FTPCSVFileName attribute

 

feature.setAttribute('FTPCSVFileName', v)

 

self.pyoutput(feature)

dbklingdom
Contributor
Forum|alt.badge.img+7
  • Contributor
  • November 20, 2017

HEy gang I'm getting an error too. Would defining feature clear this up? What would the definition be? feature = feature. what??


david_r
Evangelist
  • November 21, 2017
dbklingdom wrote:

HEy gang I'm getting an error too. Would defining feature clear this up? What would the definition be? feature = feature. what??

In this context, 'feature' is an instance of the fmeobjects.FMEFeature class.

 

Have a look at the documentation:

 

http://docs.safe.com/fme/html/FME_Objects_Python_API/index.html

tim_wood
Contributor
Forum|alt.badge.img+8
  • Author
  • Contributor
  • October 11, 2019

I'm resurrecting this question because of Python 2.x deprecation. The Python I'm currently using doesn't work if I set the Python compatibility to 3.x. I'm trying to figure out how to update my code but any suggestions would be appreciated...

Or can the FTPCaller now download a whole folder as per PR#69804?

https://knowledge.safe.com/questions/32041/download-and-upload-folder-from-ftp.html


david_r
Evangelist
  • October 11, 2019
tim_wood wrote:

I'm resurrecting this question because of Python 2.x deprecation. The Python I'm currently using doesn't work if I set the Python compatibility to 3.x. I'm trying to figure out how to update my code but any suggestions would be appreciated...

Or can the FTPCaller now download a whole folder as per PR#69804?

https://knowledge.safe.com/questions/32041/download-and-upload-folder-from-ftp.html

In most cases you can use the "2to3" Python command line tool to automatically migrate your code. Simply save your PythonCaller code to a text file and pass it though the utility and paste the result back into the PythonCaller.

https://docs.python.org/2/library/2to3.html


tim_wood
Contributor
Forum|alt.badge.img+8
  • Author
  • Contributor
  • October 11, 2019
david_r wrote:

In most cases you can use the "2to3" Python command line tool to automatically migrate your code. Simply save your PythonCaller code to a text file and pass it though the utility and paste the result back into the PythonCaller.

https://docs.python.org/2/library/2to3.html

Thanks. That doesn't seem to be installed with FME 2018 or 2019 - perhaps that's something Safe could include in future releases...? But I've found it in another Python install we've got. However, it doesn't work in either the command line or IDLE. Both complain about a syntax error, and the command line says "SyntaxError: unexpected character after line continuation character". I've copied and pasted the code out of the FTPCaller and saved it as a PY file. The indentation looks right. Do I need to add something to define a function or something? In the FTPCaller there's a separate section for "Class or Function to Process Features".

As you can tell, I'm not a Python expert!

This is the code from the FTPCaller I've saved as a .PY (the forum software has removed the indentations):

import fme

 

import fmeobjects

 

import ftplib

class GetListOfCSVsFromFTP(object):

 

def input(self,feature):

 

CSVlist = []

 

ftp = ftplib.FTP("ftpsite")

 

ftp.login("user", "password")

 

ftp.cwd(feature.getAttribute('FTPSiteFolder'))

 

try:

 

CSVlist = ftp.nlst()

 

except ftplib.error_perm, resp:

 

if str(resp) == "550 No files found":

 

feature.setAttribute('FMEFTPError', 'File not found or no access')

 

self.pyoutput(feature)

 

else:

 

raise

 

ftp.quit()

 

if not CSVlist:

 

feature.setAttribute('FMEFTPError', 'No CSV files to download')

 

self.pyoutput(feature)

 

else:

 

for v in CSVlist:

 

feature.setAttribute('FTPCSVFileName', v)

 

self.pyoutput(feature)

david_r
Evangelist
  • October 11, 2019
tim_wood wrote:

Thanks. That doesn't seem to be installed with FME 2018 or 2019 - perhaps that's something Safe could include in future releases...? But I've found it in another Python install we've got. However, it doesn't work in either the command line or IDLE. Both complain about a syntax error, and the command line says "SyntaxError: unexpected character after line continuation character". I've copied and pasted the code out of the FTPCaller and saved it as a PY file. The indentation looks right. Do I need to add something to define a function or something? In the FTPCaller there's a separate section for "Class or Function to Process Features".

As you can tell, I'm not a Python expert!

This is the code from the FTPCaller I've saved as a .PY (the forum software has removed the indentations):

import fme

 

import fmeobjects

 

import ftplib

class GetListOfCSVsFromFTP(object):

 

def input(self,feature):

 

CSVlist = []

 

ftp = ftplib.FTP("ftpsite")

 

ftp.login("user", "password")

 

ftp.cwd(feature.getAttribute('FTPSiteFolder'))

 

try:

 

CSVlist = ftp.nlst()

 

except ftplib.error_perm, resp:

 

if str(resp) == "550 No files found":

 

feature.setAttribute('FMEFTPError', 'File not found or no access')

 

self.pyoutput(feature)

 

else:

 

raise

 

ftp.quit()

 

if not CSVlist:

 

feature.setAttribute('FMEFTPError', 'No CSV files to download')

 

self.pyoutput(feature)

 

else:

 

for v in CSVlist:

 

feature.setAttribute('FTPCSVFileName', v)

 

self.pyoutput(feature)

In Python indentation has meaning, so it's almost impossible to debug code where the indents have been stripped. Can you please repost your code using "Code" formatting option.

 


tim_wood
Contributor
Forum|alt.badge.img+8
  • Author
  • Contributor
  • October 11, 2019
david_r wrote:

In Python indentation has meaning, so it's almost impossible to debug code where the indents have been stripped. Can you please repost your code using "Code" formatting option.

0684Q00000ArNPVQA3.png

 

I did that but the forum puts it all on one line.  So maybe it's got the wrong type of CR/LF when pasting from the FTPCaller...

import fmeimport
fmeobjects
import ftplib
class GetListOfCSVsFromFTP(object):
    def input(self,feature):
        CSVlist = []
        ftp = ftplib.FTP("ftpsite")
        ftp.login("user", "password")
        ftp.cwd(feature.getAttribute('FTPSiteFolder'))
        try:
            CSVlist = ftp.nlst()
        except ftplib.error_perm, resp:
            if str(resp) == "550 No files found":
                feature.setAttribute('FMEFTPError', 'File not found or no access')
                self.pyoutput(feature)
            else:
                raise
        ftp.quit()
        if not CSVlist:
            feature.setAttribute('FMEFTPError', 'No CSV files to download')
            self.pyoutput(feature)
        else:
            for v in CSVlist:
                feature.setAttribute('FTPCSVFileName', v)
                self.pyoutput(feature)

david_r
Evangelist
  • October 11, 2019
tim_wood wrote:
I did that but the forum puts it all on one line.  So maybe it's got the wrong type of CR/LF when pasting from the FTPCaller...

import fmeimport
fmeobjects
import ftplib
class GetListOfCSVsFromFTP(object):
    def input(self,feature):
        CSVlist = []
        ftp = ftplib.FTP("ftpsite")
        ftp.login("user", "password")
        ftp.cwd(feature.getAttribute('FTPSiteFolder'))
        try:
            CSVlist = ftp.nlst()
        except ftplib.error_perm, resp:
            if str(resp) == "550 No files found":
                feature.setAttribute('FMEFTPError', 'File not found or no access')
                self.pyoutput(feature)
            else:
                raise
        ftp.quit()
        if not CSVlist:
            feature.setAttribute('FMEFTPError', 'No CSV files to download')
            self.pyoutput(feature)
        else:
            for v in CSVlist:
                feature.setAttribute('FTPCSVFileName', v)
                self.pyoutput(feature)

Yeah, the forum is really bad at handling code insertion...

Try changing this line:

except ftplib.error_perm, resp:

To

except ftplib.error_perm as resp:

If that doesn't work in Python 3, please post your code and the complete error message as a separate question, that way you'll get more attention as well.


ebygomm
Influencer
Forum|alt.badge.img+32
  • Influencer
  • October 11, 2019
david_r wrote:

Yeah, the forum is really bad at handling code insertion...

Try changing this line:

except ftplib.error_perm, resp:

To

except ftplib.error_perm as resp:

If that doesn't work in Python 3, please post your code and the complete error message as a separate question, that way you'll get more attention as well.

That's the only change I've had to make to get the above code to work


tim_wood
Contributor
Forum|alt.badge.img+8
  • Author
  • Contributor
  • October 11, 2019
david_r wrote:

Yeah, the forum is really bad at handling code insertion...

Try changing this line:

except ftplib.error_perm, resp:

To

except ftplib.error_perm as resp:

If that doesn't work in Python 3, please post your code and the complete error message as a separate question, that way you'll get more attention as well.

@david_r Awesome! I am now Python 3.6+ compatible.  Thank you :-)


david_r
Evangelist
  • October 11, 2019
tim_wood wrote:

@david_r Awesome! I am now Python 3.6+ compatible. Thank you :-)

Excellent, you're welcome.


richardt
Contributor
Forum|alt.badge.img+3
  • Contributor
  • March 15, 2021

Is there a way I can set "No Proxy" when using this PythonCaller method to list files in an FTP folder?

 

The system I am now using can only get access to HTTP files via the system proxy, so I have had to set FME Tools > Options > Network Proxy = "use system proxy settings" for the rest of my workbench suite to work. However, this network proxy does not work with FTP (and its configuration is beyond my control). Not a problem with FTPcaller (I can just set it to use "No Proxy"). However, I can't find an obvious way to modify this PythonCaller script. Any suggestions?

 

P.S. otherwise this script has proved invaluable - thanks all!


david_r
Evangelist
  • March 15, 2021
richardt wrote:

Is there a way I can set "No Proxy" when using this PythonCaller method to list files in an FTP folder?

 

The system I am now using can only get access to HTTP files via the system proxy, so I have had to set FME Tools > Options > Network Proxy = "use system proxy settings" for the rest of my workbench suite to work. However, this network proxy does not work with FTP (and its configuration is beyond my control). Not a problem with FTPcaller (I can just set it to use "No Proxy"). However, I can't find an obvious way to modify this PythonCaller script. Any suggestions?

 

P.S. otherwise this script has proved invaluable - thanks all!

You cannot modify the FME proxy settings using a PythonCaller. Also, Python scripts in your workspace does not consider the FME proxy settings at all, you will have to define the proxy settings in your Python code.


chrisatsafe
Contributor
Forum|alt.badge.img+2
  • Contributor
  • March 19, 2021
richardt wrote:

Is there a way I can set "No Proxy" when using this PythonCaller method to list files in an FTP folder?

 

The system I am now using can only get access to HTTP files via the system proxy, so I have had to set FME Tools > Options > Network Proxy = "use system proxy settings" for the rest of my workbench suite to work. However, this network proxy does not work with FTP (and its configuration is beyond my control). Not a problem with FTPcaller (I can just set it to use "No Proxy"). However, I can't find an obvious way to modify this PythonCaller script. Any suggestions?

 

P.S. otherwise this script has proved invaluable - thanks all!

Hi @richardt​ ,

Not a python example, but if you are just after a list of files you can also do this with the FTPCaller, my answer on this thread might help since it doesn't require a proxy.

 

Hope that helps.


richardt
Contributor
Forum|alt.badge.img+3
  • Contributor
  • March 22, 2021
david_r wrote:

You cannot modify the FME proxy settings using a PythonCaller. Also, Python scripts in your workspace does not consider the FME proxy settings at all, you will have to define the proxy settings in your Python code.

Thanks for your feedback David - that clarification could be quite helpful. However, it seems that my problem may be nothing to do with the proxy, but that the machine I am trying to run it from on a customer's premises has a very flaky FTP connection which now always fails whatever the settings... but I have only got to test it further today as their RDP connection hasn't worked for 7 days. Painful!

(I'll respond to the thread when I find out if proxy is an issue at all..)


richardt
Contributor
Forum|alt.badge.img+3
  • Contributor
  • March 22, 2021
chrisatsafe wrote:

Hi @richardt​ ,

Not a python example, but if you are just after a list of files you can also do this with the FTPCaller, my answer on this thread might help since it doesn't require a proxy.

 

Hope that helps.

Hi Chris. That's a really helpful alternative to using the PythonCaller for extracting file names - I hadn't realised that you could get FTPCaller to return a directory listing (albeit with all the extra guff which you neatly then strip out). Works well for me in FME2020.1.3. It also means that I can switch the FTPCaller FME Network proxy settings if need be.


harmen_kampinga
Forum|alt.badge.img+2
david_r wrote:

I took the liberty of "fixing up" your code based on the suggestions here. This works for me using FME 2016:

import fmeobjects
import ftplib

class GetCSVs(object):
    def input(self, feature):
        CSVlist = []
        ftp = ftplib.FTP("REDACTED")
        ftp.login("REDACTED""REDACTED")
        ftp.cwd("/pub")
        try:
            CSVlist = ftp.nlst()
        except ftplib.error_perm, resp:
            if str(resp) == "550 No files found":
                print "No files found"
            else:
                raise
        ftp.quit()
        for i, v in enumerate(CSVlist):
            feature.setAttribute('_list_CSVs{%d}' % i, v)
        self.pyoutput(feature)

0684Q00000ArKsDQAV.png

Hi @david_r​ , tried to get the list from my sFTP environment, but this script doesn't work for me. Does the ftplib support sftp? Or is it something with the Python version? I'm using FME Desktop 2021 with Python version 2.7 or 3

 


Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings