How do I use SFTP ssh keys to download a file?

Hi @clow,

Currently, there is no SFTP watcher available on FME Server:

FME Cloud is running on Linux so you would need to add your key to the FME Cloud instance. It might be possible to add the ssh key following the instructions @DebbiAtSafe shared with you using the SystemCaller transformer but I am afraid that you might need root permissions to either add the file or to change the permission on the file. I will investigate and follow up with you once I know more.

Thanks @GerhardAtSafe and @DebbiAtSafe - I will try on the cloud instance. I look forward to hearing back from you though.

Good Day,

I am trying to get the connection working with sftp, but where do you add the SshHostKeyFingerprint, in pageant, i can only see add key, in the format of *.ppk

Hi @adubs, I'm experiencing exactly the same issue as you described. I was able to run it in desktop but not on FME server. However the SystemCaller doesn't hang, but FTPCaller shows these errors instead.

FTP info: Failure connecting to agent

FTP info: Authentication failure

Were you able to resolve this issue? If so, how? Thanks!

Hi @fmeuser_gc,

This thread looks to be rather old and I'm sure there's been a lot of changes since FME 2018 that this problem might be rather different today. Would you be able to post a new question in the Forums for better visibility? It sounds like your issue is more to do with the FTP connection, so more details on your specific issue would be great. Thanks!

Hi @jovitaatsafe, actually this issue was resolved on a separate thread. But it was related to system caller, not an issue with FTP connection though.

Hi there,

I see this is an old thread, but it looks as though the FTP Caller is still extremely light on documentation.

In particular, for this same query regarding the SSH Keys.

How is the SSH key accessed from within the FTP Caller, when the option only exists to specify a username?

Is there something you need to run upstream before using the FTP Caller, in order for the FTP Authentication to work?

Cheers,

Kieran

Hi @kieran.odonnell​ 

This article contains some instructions on how to use SSH keys with FTPCaller on Windows.

In short, ensure Pageant is running and the private key is loaded in Pageant before running a workspace containing the FTPCaller. Starting/loading the private key in Pageant can be done either manually or via a SystemCaller.

I hope this information helps.

I have not tried this on FMECloud, but I do think its straight foreward. First you need to install som pip libraries to one of your shared data folder. Add this folder to your sys.path in python, run som code in your PythonCreator and voila download data from sftp. I can assist with examples I think.

Challange accepted:

I would try something like this:

Use with caution, code needs to be updated to match user spesifics.

Place inside a PythonCreator and call the class SftpReader

import fmeobjects
import os
# NB!!! This is written without any testing or running at all, so bugs might appear!!!
"""
# Step 1. Create a folde under your FME_SHAREDRESOURCE_DATA called "site-packages" on your FME- Server/Cloud
# run this initial setup:
# NB!!! This is written without any testing or running at all, so bugs might appear!!!
 
import os
import subprocess as sp
import shlex
 
fme = os.path.join(FME_MacroValues['FME_HOME_UNIX'], 'fme') # FME_HOME on Windows Server
target = os.path.join(FME_MacroValues['FME_SHAREDRESOURCE_DATA'], 'site-packages')
command = f'{fme} python -m pip install asyncssh -t {target}')
sp.Popen(shlex.split(command))
 
# Watch your FME_SHAREDRESOURCE_DATA, site-packages folder and see the dependencies appear
 
 # Written by Paal Pedersen
"""
# NB!!! This is written without any testing or running at all, so bugs might appear!!!
import sys
sys.path = [os.path.join(FME_MacroValues['FME_SHAREDRESOURCE_DATA'], 'site-packages')] + sys.path
import asyncssh
import asyncio
 
import pathlib
 
async def download_file(self, sftp, file: str, localdir: str = '.'):
    localpath=f"{localdir}{file}"
    if not os.path.exists(localpath):
        os.makedirs(localpath)
        
    self.feature.setAttribute('path', localpath)
    
    localpath, _ = os.path.split(localpath)
    
    self.pyoutput(self.feature)
 
    await sftp.get(file, 
    localpath=localpath, 
    recurse=True,
    preserve=True,
    block_size=16384,
    max_requests=100) 
 
async def run_client(self, host=None, username=None, password=None, known_hosts=None):
    async with asyncssh.connect(host=host, username=username, password=password, known_hosts=known_hosts) as conn:
        async with conn.start_sftp_client() as sftp:
            # Currently only supports downloading full folder
            # These must be changed 
            folders = (
                "/folder/a", "/folder/b", "/folder/c"
                )
 
            def tasks():
                for folder in folders:
                    yield download_file(self, sftp, folder)
 
            await asyncio.gather(*tasks())
 
 
class ConstantOverrideError(Exception):
    pass
 
 
class Constants:
    # You need to change these values:
    HOST = 'sftp.host.com'
    USERNAME = 'username'
    PASSWORD = 'password'
    KNOWN_HOSTS = "~/.ssh/known_hosts"
    def __init__(self):
        config = pathlib.Path(self.KNOWN_HOSTS)
        config.parent.mkdir(parents=True, exist_ok=True)
        config.write_text('sftp.host.com, ip.add.re.ss ssh-rsa AS/SEEN/INSIDE/YOUR/.SSH/FOLDER/AFTER/SAVING/A/SSH/CONNECTION/TO/KNOWN/HOSTS')
 
    def __setattr__(self, name, value):
        raise ConstantOverrideError("It's illegal to override constants")
 
    def __str__(self):
        name = self.__class__.__name__
        attributes = self.__class__.__dict__
        attributes = '\n\t'.join(f'{k}={v}' for k,v in filter(lambda x: x[0] == x[0].upper(), attributes.items()))
        return f'''class {name}:\n\t{attributes}'''
    
    def __repr__(self):
        name = self.__class__.__name__
        return f'<class {name}>'
 
 
class SftpReader(object):
    def __init__(self):
        self.feature = fmeobjects.FMEFeature()
    def input(self,feature):
        c = Constants()
        try:
            asyncio.get_event_loop().run_until_complete(
                run_client(self,
                    host=c.HOST, username=c.USERNAME, 
                    password=c.PASSWORD, known_hosts=c.KNOWN_HOSTS))
        except (OSError, asyncssh.Error) as exc:
            fmeobjects.FMELogFile().logMessageString('SFTP operation failed: ' + str(exc))
        
    def close(self):
        pass