Use PythonCaller to get list of files in directory on FTP site

OK I give up - help me!

This is my first attempt to use the PythonCaller (since FTPCaller can't yet download directories or accept wildcards). I've cobbled together some Python to retrieve a directory listing from an FTP site which I want to output (probably as a list) so I can generate the name of the file I want to download and pass it to FTPCaller.

I've done a whole lot of Googling but I can't fix this error on the feature.setAttribute line:

2016-10-12 17:46:48| 1.9| 0.1|WARN |Python Exception <TypeError>: not all arguments converted during string formatting

2016-10-12 17:46:48| 1.9| 0.0|WARN |Traceback (most recent call last):

File "<string>", line 30, in input

TypeError: not all arguments converted during string formatting

2016-10-12 17:46:48| 1.9| 0.0|ERROR |Error encountered while calling method `input'

2016-10-12 17:46:48| 1.9| 0.0|FATAL |f_32(PythonFactory): PythonFactory failed to process feature

import fme
import fmeobjects
import ftplib
# Template Function interface:
# When using this function, make sure its name is set as the value of
# the 'Class or Function to Process Features' transformer parameter
def processFeature(feature):
    pass
# Template Class Interface:
# When using this class, make sure its name is set as the value of
# the 'Class or Function to Process Features' transformer parameter
class GetCSVs(object):
    def __init__(self):
        CSVlist = �]
    def input(self,feature):
        ftp = ftplib.FTP("REDACTED")
        ftp.login("REDACTED", "REDACTED")
        ftp.cwd("REDACTED")
        try:
            CSVlist = ftp.nlst()
        except ftplib.error_perm, resp:
            if str(resp) == "550 No files found":
                print "No files found"
            else:
                raise
        ftp.quit()
        for i in enumerate(CSVlist):
            feature.setAttribute('_list_CSVs{%d}' % i)
    def close(self):
        self.pyoutput(feature)

Page 2 / 2

According to the documentation: "This method returns None" :-)

So basically, you cannot use this method as a replacement for nlst() without any further

changes.

You can, however, specify a callback function for dir() which will then be called once for every line returned by the ftp server. There's a short example here. You will then have to split up each line yourself to get the filenames, dates etc.

Rather than using ftp.nlst() you can do something like:

lines = h]
ftp.retrlines('LIST', lines.append)

You will then have a Python list containing all the text returned by dir() which you can then parse into filename, filesize and timestamp and sort accordingly before deciding exactly which file to download.

Just be aware that the exact formatting of these lines can vary quite a bit between different ftp servers. Hope this helps.

I've implemented the Python code from @david_r which works fine. However, if I exchange ftplib.nlst() for ftplib.dir() to give me a full directory listing (I need to file modified date) FME gives me the error in the screenshot below. Spent all afternoon trying to get around this without success. Any ideas??

Thanks

James

I went to the university of Google and studied "python ftp nlst vs dir" and found this...

http://stackoverflow.com/questions/111954/using-pythons-ftplib-to-get-a-directory-listing-portably

So I changed the line that reads "CSVlist = ftp.nlst()" to "ftp.dir(CSVlist.append)" and bingo.

After ftp.quit(), I run the code shown below (with proper indentations of course). So if you use this, you'll get a list of features like this...

0 20170109152813 -rw-rw-rw- 1 user group 31591 Jul 9 2016 filename.csv

# if CSVlist is empty

if not CSVlist:

# set the value of FMEFTPError to 'No CSV files to download'

feature.setAttribute('FMEFTPError', 'No CSV files to download')

# write that feature back to FME (there'll be one feature output from the PythonCaller)

self.pyoutput(feature)

else:

# loop through each item in the CSVlist array

for v in CSVlist:

# write out a feature for each item in the array and put the value of the array item in the FTPCSVFileName attribute

feature.setAttribute('FTPCSVFileName', v)

self.pyoutput(feature)

HEy gang I'm getting an error too. Would defining feature clear this up? What would the definition be? feature = feature. what??

In this context, 'feature' is an instance of the fmeobjects.FMEFeature class.

Have a look at the documentation:

http://docs.safe.com/fme/html/FME_Objects_Python_API/index.html

I'm resurrecting this question because of Python 2.x deprecation. The Python I'm currently using doesn't work if I set the Python compatibility to 3.x. I'm trying to figure out how to update my code but any suggestions would be appreciated...

Or can the FTPCaller now download a whole folder as per PR#69804?

https://knowledge.safe.com/questions/32041/download-and-upload-folder-from-ftp.html

Or can the FTPCaller now download a whole folder as per PR#69804?

https://knowledge.safe.com/questions/32041/download-and-upload-folder-from-ftp.html

In most cases you can use the "2to3" Python command line tool to automatically migrate your code. Simply save your PythonCaller code to a text file and pass it though the utility and paste the result back into the PythonCaller.

https://docs.python.org/2/library/2to3.html

Thanks. That doesn't seem to be installed with FME 2018 or 2019 - perhaps that's something Safe could include in future releases...? But I've found it in another Python install we've got. However, it doesn't work in either the command line or IDLE. Both complain about a syntax error, and the command line says "SyntaxError: unexpected character after line continuation character". I've copied and pasted the code out of the FTPCaller and saved it as a PY file. The indentation looks right. Do I need to add something to define a function or something? In the FTPCaller there's a separate section for "Class or Function to Process Features".

As you can tell, I'm not a Python expert!

This is the code from the FTPCaller I've saved as a .PY (the forum software has removed the indentations):

import fme

import fmeobjects

import ftplib

class GetListOfCSVsFromFTP(object):

def input(self,feature):

CSVlist = t]

ftp = ftplib.FTP("ftpsite")

ftp.login("user", "password")

ftp.cwd(feature.getAttribute('FTPSiteFolder'))

try:

CSVlist = ftp.nlst()

except ftplib.error_perm, resp:

if str(resp) == "550 No files found":

feature.setAttribute('FMEFTPError', 'File not found or no access')

self.pyoutput(feature)

else:

raise

ftp.quit()

if not CSVlist:

feature.setAttribute('FMEFTPError', 'No CSV files to download')

self.pyoutput(feature)

else:

for v in CSVlist:

feature.setAttribute('FTPCSVFileName', v)

self.pyoutput(feature)

As you can tell, I'm not a Python expert!

This is the code from the FTPCaller I've saved as a .PY (the forum software has removed the indentations):

import fme

import fmeobjects

import ftplib

class GetListOfCSVsFromFTP(object):

def input(self,feature):

CSVlist = t]

ftp = ftplib.FTP("ftpsite")

ftp.login("user", "password")

ftp.cwd(feature.getAttribute('FTPSiteFolder'))

try:

CSVlist = ftp.nlst()

except ftplib.error_perm, resp:

if str(resp) == "550 No files found":

feature.setAttribute('FMEFTPError', 'File not found or no access')

self.pyoutput(feature)

else:

raise

ftp.quit()

if not CSVlist:

feature.setAttribute('FMEFTPError', 'No CSV files to download')

self.pyoutput(feature)

else:

for v in CSVlist:

feature.setAttribute('FTPCSVFileName', v)

self.pyoutput(feature)

In Python indentation has meaning, so it's almost impossible to debug code where the indents have been stripped. Can you please repost your code using "Code" formatting option.

I did that but the forum puts it all on one line.  So maybe it's got the wrong type of CR/LF when pasting from the FTPCaller...

import fmeimport
fmeobjects
import ftplib
class GetListOfCSVsFromFTP(object):
    def input(self,feature):
        CSVlist = b]
        ftp = ftplib.FTP("ftpsite")
        ftp.login("user", "password")
        ftp.cwd(feature.getAttribute('FTPSiteFolder'))
        try:
            CSVlist = ftp.nlst()
        except ftplib.error_perm, resp:
            if str(resp) == "550 No files found":
                feature.setAttribute('FMEFTPError', 'File not found or no access')
                self.pyoutput(feature)
            else:
                raise
        ftp.quit()
        if not CSVlist:
            feature.setAttribute('FMEFTPError', 'No CSV files to download')
            self.pyoutput(feature)
        else:
            for v in CSVlist:
                feature.setAttribute('FTPCSVFileName', v)
                self.pyoutput(feature)

I did that but the forum puts it all on one line.  So maybe it's got the wrong type of CR/LF when pasting from the FTPCaller...

import fmeimport
fmeobjects
import ftplib
class GetListOfCSVsFromFTP(object):
    def input(self,feature):
        CSVlist = ,]
        ftp = ftplib.FTP("ftpsite")
        ftp.login("user", "password")
        ftp.cwd(feature.getAttribute('FTPSiteFolder'))
        try:
            CSVlist = ftp.nlst()
        except ftplib.error_perm, resp:
            if str(resp) == "550 No files found":
                feature.setAttribute('FMEFTPError', 'File not found or no access')
                self.pyoutput(feature)
            else:
                raise
        ftp.quit()
        if not CSVlist:
            feature.setAttribute('FMEFTPError', 'No CSV files to download')
            self.pyoutput(feature)
        else:
            for v in CSVlist:
                feature.setAttribute('FTPCSVFileName', v)
                self.pyoutput(feature)

Yeah, the forum is really bad at handling code insertion...

Try changing this line:

except ftplib.error_perm, resp:

except ftplib.error_perm as resp:

If that doesn't work in Python 3, please post your code and the complete error message as a separate question, that way you'll get more attention as well.

Yeah, the forum is really bad at handling code insertion...

Try changing this line:

except ftplib.error_perm, resp:

except ftplib.error_perm as resp:

If that doesn't work in Python 3, please post your code and the complete error message as a separate question, that way you'll get more attention as well.

That's the only change I've had to make to get the above code to work

Yeah, the forum is really bad at handling code insertion...

Try changing this line:

except ftplib.error_perm, resp:

except ftplib.error_perm as resp:

If that doesn't work in Python 3, please post your code and the complete error message as a separate question, that way you'll get more attention as well.

@david_r Awesome! I am now Python 3.6+ compatible. Thank you :-)

Excellent, you're welcome.

Is there a way I can set "No Proxy" when using this PythonCaller method to list files in an FTP folder?

The system I am now using can only get access to HTTP files via the system proxy, so I have had to set FME Tools > Options > Network Proxy = "use system proxy settings" for the rest of my workbench suite to work. However, this network proxy does not work with FTP (and its configuration is beyond my control). Not a problem with FTPcaller (I can just set it to use "No Proxy"). However, I can't find an obvious way to modify this PythonCaller script. Any suggestions?

P.S. otherwise this script has proved invaluable - thanks all!

Is there a way I can set "No Proxy" when using this PythonCaller method to list files in an FTP folder?

P.S. otherwise this script has proved invaluable - thanks all!

You cannot modify the FME proxy settings using a PythonCaller. Also, Python scripts in your workspace does not consider the FME proxy settings at all, you will have to define the proxy settings in your Python code.

Is there a way I can set "No Proxy" when using this PythonCaller method to list files in an FTP folder?

P.S. otherwise this script has proved invaluable - thanks all!

Hi @richardt ,

Not a python example, but if you are just after a list of files you can also do this with the FTPCaller, my answer on this thread might help since it doesn't require a proxy.

Hope that helps.

Thanks for your feedback David - that clarification could be quite helpful. However, it seems that my problem may be nothing to do with the proxy, but that the machine I am trying to run it from on a customer's premises has a very flaky FTP connection which now always fails whatever the settings... but I have only got to test it further today as their RDP connection hasn't worked for 7 days. Painful!

(I'll respond to the thread when I find out if proxy is an issue at all..)

Hi @richardt ,

Not a python example, but if you are just after a list of files you can also do this with the FTPCaller, my answer on this thread might help since it doesn't require a proxy.

Hope that helps.

Hi Chris. That's a really helpful alternative to using the PythonCaller for extracting file names - I hadn't realised that you could get FTPCaller to return a directory listing (albeit with all the extra guff which you neatly then strip out). Works well for me in FME2020.1.3. It also means that I can switch the FTPCaller FME Network proxy settings if need be.

I took the liberty of "fixing up" your code based on the suggestions here. This works for me using FME 2016:

import fmeobjects
import ftplib

class GetCSVs(object):
    def input(self, feature):
        CSVlist = u]
        ftp = ftplib.FTP("REDACTED")
        ftp.login("REDACTED", "REDACTED")
        ftp.cwd("/pub")
        try:
            CSVlist = ftp.nlst()
        except ftplib.error_perm, resp:
            if str(resp) == "550 No files found":
                print "No files found"
            else:
                raise
        ftp.quit()
        for i, v in enumerate(CSVlist):
            feature.setAttribute('_list_CSVs{%d}' % i, v)
        self.pyoutput(feature)

Hi @david_r , tried to get the list from my sFTP environment, but this script doesn't work for me. Does the ftplib support sftp? Or is it something with the Python version? I'm using FME Desktop 2021 with Python version 2.7 or 3

Reply

Community Stats

Reply

Community Stats

Sign up

An FME Account is required to contribute

Login to the community

An FME Account is required to contribute

Scanning file for viruses.

This file cannot be downloaded