Solved

How do I use SFTP ssh keys to download a file?


I tried FTPCaller using the full path to the file I want but I am unsure of authentication as it seems undocumented. I am unable to use username and password and must use SSH key files. Setting to ssh-agent seemed like the way to go but how do you choose your key file? I am publishing to Cloud anyways so I thought I would look at what was there and seems there is upload capability for sftp using a subscriber and ftp watcher (no sftp). Am I missing something?

icon

Best answer by debbiatsafe 17 March 2018, 01:54

View original

15 replies

Userlevel 2
Badge +17

Hi @clow

My apologies for the lack of information in the documentation. You are on the right path setting SSH-Agent/Pageant as an authentication type within the FTPCaller.

If you are on the Windows platform, you would have to use a program called Pageant. This is not installed or running on Windows by default so the user would need to install it. The following link should contain more details and instructions: https://www.digitalocean.com/community/tutorials/how-to-use-pageant-to-streamline-ssh-key-authentication-with-putty. Basically, you would need to add your key to Pageant.

If you are on the Linux platform, you would have to use ssh-agent ( https://www.ssh.com/ssh/agent) and generally, this is running by default on Linux.

There is an existing PR (PR81956) that addresses FTPCaller documentation. Once this PR has been resolved, the documentation should become clearer.

Hope this helps,

Debbi

Badge

Hi @clow,

Currently, there is no SFTP watcher available on FME Server:

 

https://knowledge.safe.com/idea/47929/sftp-watcher.html

FME Cloud is running on Linux so you would need to add your key to the FME Cloud instance. It might be possible to add the ssh key following the instructions @DebbiAtSafe shared with you using the SystemCaller transformer but I am afraid that you might need root permissions to either add the file or to change the permission on the file. I will investigate and follow up with you once I know more.

Thanks @GerhardAtSafe and @DebbiAtSafe - I will try on the cloud instance. I look forward to hearing back from you though.

Badge

Thanks @GerhardAtSafe and @DebbiAtSafe - I will try on the cloud instance. I look forward to hearing back from you though.

Hi @clow,

 

unfortunately, it is currently not possible to use the FTPCaller with ssh key files on FME Cloud. The FME Engine service has only limited access to the filesystem of the FME Cloud instance which is not sufficient to add ssh key files that can be used in the FTPCaller. I will raise this with development and will update here once there is news on this.

 

 

A workaround for know could be to use python with the pysftp module. You will need to add the pysftp and the paramiko modules to your FME Cloud instance following these steps:

 

https://knowledge.safe.com/questions/4164/python-libraries-on-fme-cloud.html

 

 

I hope this helps!

 

Badge

Hi @clow

My apologies for the lack of information in the documentation. You are on the right path setting SSH-Agent/Pageant as an authentication type within the FTPCaller.

If you are on the Windows platform, you would have to use a program called Pageant. This is not installed or running on Windows by default so the user would need to install it. The following link should contain more details and instructions: https://www.digitalocean.com/community/tutorials/how-to-use-pageant-to-streamline-ssh-key-authentication-with-putty. Basically, you would need to add your key to Pageant.

If you are on the Linux platform, you would have to use ssh-agent ( https://www.ssh.com/ssh/agent) and generally, this is running by default on Linux.

There is an existing PR (PR81956) that addresses FTPCaller documentation. Once this PR has been resolved, the documentation should become clearer.

Hope this helps,

Debbi

Thank you for this information @DebbiAtSafe. This worked perfectly. And thank you @clow for asking the question. You just saved me a lot of time!

 

Badge

Hi @clow

My apologies for the lack of information in the documentation. You are on the right path setting SSH-Agent/Pageant as an authentication type within the FTPCaller.

If you are on the Windows platform, you would have to use a program called Pageant. This is not installed or running on Windows by default so the user would need to install it. The following link should contain more details and instructions: https://www.digitalocean.com/community/tutorials/how-to-use-pageant-to-streamline-ssh-key-authentication-with-putty. Basically, you would need to add your key to Pageant.

If you are on the Linux platform, you would have to use ssh-agent ( https://www.ssh.com/ssh/agent) and generally, this is running by default on Linux.

There is an existing PR (PR81956) that addresses FTPCaller documentation. Once this PR has been resolved, the documentation should become clearer.

Hope this helps,

Debbi

I'm having trouble getting this working on FME server. I have a system caller that calls pageant, and loads the key. It then uses FTPCaller to connect to the sftp server and download the file. It works great in desktop, but when I put it on server. the system caller just hangs. Any advice getting sftp working on FME Server

 

 

Userlevel 2
Badge +17
I'm having trouble getting this working on FME server. I have a system caller that calls pageant, and loads the key. It then uses FTPCaller to connect to the sftp server and download the file. It works great in desktop, but when I put it on server. the system caller just hangs. Any advice getting sftp working on FME Server

 

 

Hi @adubs

 

Would you be able to provide log files? It may contain more details about it is not working.

Good Day,

I am trying to get the connection working with sftp, but where do you add the SshHostKeyFingerprint, in pageant, i can only see add key, in the format of *.ppk

Badge +6
I'm having trouble getting this working on FME server. I have a system caller that calls pageant, and loads the key. It then uses FTPCaller to connect to the sftp server and download the file. It works great in desktop, but when I put it on server. the system caller just hangs. Any advice getting sftp working on FME Server

 

 

Hi @adubs, I'm experiencing exactly the same issue as you described. I was able to run it in desktop but not on FME server. However the SystemCaller doesn't hang, but FTPCaller shows these errors instead.

 

FTP info: Failure connecting to agent

FTP info: Authentication failure

 

Were you able to resolve this issue? If so, how? Thanks!

Badge +11

Hi @adubs, I'm experiencing exactly the same issue as you described. I was able to run it in desktop but not on FME server. However the SystemCaller doesn't hang, but FTPCaller shows these errors instead.

 

FTP info: Failure connecting to agent

FTP info: Authentication failure

 

Were you able to resolve this issue? If so, how? Thanks!

Hi @fmeuser_gc,

This thread looks to be rather old and I'm sure there's been a lot of changes since FME 2018 that this problem might be rather different today. Would you be able to post a new question in the Forums for better visibility? It sounds like your issue is more to do with the FTP connection, so more details on your specific issue would be great. Thanks!

Badge +6

Hi @fmeuser_gc,

This thread looks to be rather old and I'm sure there's been a lot of changes since FME 2018 that this problem might be rather different today. Would you be able to post a new question in the Forums for better visibility? It sounds like your issue is more to do with the FTP connection, so more details on your specific issue would be great. Thanks!

Hi @jovitaatsafe, actually this issue was resolved on a separate thread. But it was related to system caller, not an issue with FTP connection though.

Badge

Hi there,

 

I see this is an old thread, but it looks as though the FTP Caller is still extremely light on documentation.

In particular, for this same query regarding the SSH Keys.

 

How is the SSH key accessed from within the FTP Caller, when the option only exists to specify a username?

Is there something you need to run upstream before using the FTP Caller, in order for the FTP Authentication to work?

 

Cheers,

Kieran

Userlevel 2
Badge +17

Hi there,

 

I see this is an old thread, but it looks as though the FTP Caller is still extremely light on documentation.

In particular, for this same query regarding the SSH Keys.

 

How is the SSH key accessed from within the FTP Caller, when the option only exists to specify a username?

Is there something you need to run upstream before using the FTP Caller, in order for the FTP Authentication to work?

 

Cheers,

Kieran

Hi @kieran.odonnell​ 

This article contains some instructions on how to use SSH keys with FTPCaller on Windows.

In short, ensure Pageant is running and the private key is loaded in Pageant before running a workspace containing the FTPCaller. Starting/loading the private key in Pageant can be done either manually or via a SystemCaller.

 

I hope this information helps.

Badge +4

I have not tried this on FMECloud, but I do think its straight foreward. First you need to install som pip libraries to one of your shared data folder. Add this folder to your sys.path in python, run som code in your PythonCreator and voila download data from sftp. I can assist with examples I think.

Badge +4

Challange accepted:

I would try something like this:

Use with caution, code needs to be updated to match user spesifics.

Place inside a PythonCreator and call the class SftpReader

import fmeobjects
import os
# NB!!! This is written without any testing or running at all, so bugs might appear!!!
"""
# Step 1. Create a folde under your FME_SHAREDRESOURCE_DATA called "site-packages" on your FME- Server/Cloud
# run this initial setup:
# NB!!! This is written without any testing or running at all, so bugs might appear!!!
 
import os
import subprocess as sp
import shlex
 
fme = os.path.join(FME_MacroValues['FME_HOME_UNIX'], 'fme') # FME_HOME on Windows Server
target = os.path.join(FME_MacroValues['FME_SHAREDRESOURCE_DATA'], 'site-packages')
command = f'{fme} python -m pip install asyncssh -t {target}')
sp.Popen(shlex.split(command))
 
# Watch your FME_SHAREDRESOURCE_DATA, site-packages folder and see the dependencies appear
 
 # Written by Paal Pedersen
"""
# NB!!! This is written without any testing or running at all, so bugs might appear!!!
import sys
sys.path = [os.path.join(FME_MacroValues['FME_SHAREDRESOURCE_DATA'], 'site-packages')] + sys.path
import asyncssh
import asyncio
 
import pathlib
 
async def download_file(self, sftp, file: str, localdir: str = '.'):
    localpath=f"{localdir}{file}"
    if not os.path.exists(localpath):
        os.makedirs(localpath)
        
    self.feature.setAttribute('path', localpath)
    
    localpath, _ = os.path.split(localpath)
    
    self.pyoutput(self.feature)
 
    await sftp.get(file, 
    localpath=localpath, 
    recurse=True,
    preserve=True,
    block_size=16384,
    max_requests=100) 
 
async def run_client(self, host=None, username=None, password=None, known_hosts=None):
    async with asyncssh.connect(host=host, username=username, password=password, known_hosts=known_hosts) as conn:
        async with conn.start_sftp_client() as sftp:
            # Currently only supports downloading full folder
            # These must be changed 
            folders = (
                "/folder/a", "/folder/b", "/folder/c"
                )
 
            def tasks():
                for folder in folders:
                    yield download_file(self, sftp, folder)
 
            await asyncio.gather(*tasks())
 
 
class ConstantOverrideError(Exception):
    pass
 
 
class Constants:
    # You need to change these values:
    HOST = 'sftp.host.com'
    USERNAME = 'username'
    PASSWORD = 'password'
    KNOWN_HOSTS = "~/.ssh/known_hosts"
    def __init__(self):
        config = pathlib.Path(self.KNOWN_HOSTS)
        config.parent.mkdir(parents=True, exist_ok=True)
        config.write_text('sftp.host.com, ip.add.re.ss ssh-rsa AS/SEEN/INSIDE/YOUR/.SSH/FOLDER/AFTER/SAVING/A/SSH/CONNECTION/TO/KNOWN/HOSTS')
 
    def __setattr__(self, name, value):
        raise ConstantOverrideError("It's illegal to override constants")
 
    def __str__(self):
        name = self.__class__.__name__
        attributes = self.__class__.__dict__
        attributes = '\n\t'.join(f'{k}={v}' for k,v in filter(lambda x: x[0] == x[0].upper(), attributes.items()))
        return f'''class {name}:\n\t{attributes}'''
    
    def __repr__(self):
        name = self.__class__.__name__
        return f'<class {name}>'
 
 
class SftpReader(object):
    def __init__(self):
        self.feature = fmeobjects.FMEFeature()
    def input(self,feature):
        c = Constants()
        try:
            asyncio.get_event_loop().run_until_complete(
                run_client(self,
                    host=c.HOST, username=c.USERNAME, 
                    password=c.PASSWORD, known_hosts=c.KNOWN_HOSTS))
        except (OSError, asyncssh.Error) as exc:
            fmeobjects.FMELogFile().logMessageString('SFTP operation failed: ' + str(exc))
        
    def close(self):
        pass

 

 

Reply