Skip to main content
Archived

Synchronize directories - local, share, sftp

Related products:Transformers
  • July 12, 2023
  • 1 reply
  • 16 views
danilo_fme
mattwilkie
  • danilo_fme
    danilo_fme
  • mattwilkie
    mattwilkie

mattwilkie
Supporter

It would be nice to use FME to synchronize contents of directories, where said directories could be any combination of Local (`D:datastuff`), Share (`\\serverdata hings`), and S/FTP (`sftp://example.ca/opendata/lizardness`).


Currently we're doing this by using Windows Scheduled Tasks and a variety of scripts utilizing some combination of python, WinSCP, FreeFileSync, and robocopy (depending on what year said script was written). Each of these programs and scripts work well enough for their specific task but I'd like to manage them from the same place we manage our other ETL tasks: FME Server.


We could add a sub-process transformer using PythonCaller to any of the above in a workbench but there's no insight into whatever messages the process might emit or ability to respond appropriately to situation X.


Alternatively we could recursively use FTPCaller/FileCopy/... for each level of the tree, enumerate contents, and build the parameters from those, but that's a lot of small incremental busy work with meagre gains in each step.


Note: I said 'synchronize', but we're only interested in mirroring, making one tree an exact replica of another. A natural extention would be to seek bidirectional synchronization, but that's a much harder problem and I think it's fine/better to add that as a separate idea.


A version of this desirable feature was talked about in 2017*, with indication there's an associated Idea, but I couldn't find it, thus this new one.

1 reply

LizAtSafe
Safer
Forum|alt.badge.img+15
  • Safer
  • April 5, 2025
OpenArchived

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings