Question

Mulitple files to process with one workspace

  • 18 February 2022
  • 6 replies
  • 5 views
gisbradokla
Rank
Badge +9

I have multiple files landing in a folder that is watch to trigger a wksp.

2 files go together but don't always have exactly the same name. (they will be similar.) There will also be multiple folders where these files can land and be processed. I need to know several things.

1 idea for parsing the file names of separate file types to get them matched into the same wksp. like maybe they land at the same time?

2 is it possible to use 1 automation to watch and file both file types and use them to kick off 1 wksp? image 1image13 all the jobs coming in will be moved to 1 of 3 folders when process is complete using a file mover. Do i need three qutomations in my wlrkspace to tell the file mover where to move the package (both files) or can this be done with the dataset value (being parsed in the file mover wksp? see image 2 image2IT is also possible that files will land in both incoming locations at the same time. the file types would be the same but the file name would be different

 

I found some information about the filecopy but really need help with the first part of this!

https://community.safe.com/s/question/0D54Q000080hHKOSA2/am-i-using-filecopy-correctly

 

  •  If you have multiple output folders you can set
  •  
  • Destination File Copy Folder : NotUsed 
  • filecopy_source_dataset = server 1
  •  
  • filecopy_dest_dataset = server2\\subfolder (where subfolder is different for each feature)

 

6 replies

S

in regards to a workspace that runs on multiple files, we had to accomplish something like this for one of our workflows. We have a process that reads in anywhere from 1 to 4 files ata given time.

 

to get around this, our automation uses a S3 watcher which is the only big difference but the logic SHOULD be mostly the same. We built two custom transformers, one that lists all the files in a given directory or directories that were modified / uploaded on a given date. And then another that counts all of the files and groups them by file name and makes sure that each group has a specific amount of files. We accomplish this with one automation and one workspace.

 

An example; John smith dental sends us 2 files with the goal of generating a list of patients with insurance. The automation runs whenever either of those files are dropped, we get JSD_Insurance and JSD_Patients, the transformers would get the list of files from the S3 bucket. IF both files are there then the workspace continues using the files it gathered from the list. if it doesn't it will error out - but I suppose you could configure this to do something else.

 

I'll include a screenshot of the file lister and the custom transformer file for the second transformer that filters based on that list. I can't include the file for the first one because it has some proprietary info in it.

 

image 

hopefully this helps

gisbradokla
Rank
Badge +9

in regards to a workspace that runs on multiple files, we had to accomplish something like this for one of our workflows. We have a process that reads in anywhere from 1 to 4 files ata given time.

 

to get around this, our automation uses a S3 watcher which is the only big difference but the logic SHOULD be mostly the same. We built two custom transformers, one that lists all the files in a given directory or directories that were modified / uploaded on a given date. And then another that counts all of the files and groups them by file name and makes sure that each group has a specific amount of files. We accomplish this with one automation and one workspace.

 

An example; John smith dental sends us 2 files with the goal of generating a list of patients with insurance. The automation runs whenever either of those files are dropped, we get JSD_Insurance and JSD_Patients, the transformers would get the list of files from the S3 bucket. IF both files are there then the workspace continues using the files it gathered from the list. if it doesn't it will error out - but I suppose you could configure this to do something else.

 

I'll include a screenshot of the file lister and the custom transformer file for the second transformer that filters based on that list. I can't include the file for the first one because it has some proprietary info in it.

 

image 

hopefully this helps

Thank you

i did find a solution to this.

used a filter for the passed and routed the fail out to another filter for a second file type.

S

Thank you

i did find a solution to this.

used a filter for the passed and routed the fail out to another filter for a second file type.

makes sense! I'm glad you found a solution :)

S

Thank you

i did find a solution to this.

used a filter for the passed and routed the fail out to another filter for a second file type.

also - just in case someone stumbles across this in the future - you may want to mark your reply as the answer if possible

gisbradokla
Rank
Badge +9

Thank you

i did find a solution to this.

used a filter for the passed and routed the fail out to another filter for a second file type.

R

Has anyone successfully set up a two-way communication system with SMS in FME? I'm interested in receiving responses and updating my workflow accordingly.

Reply


Community Stats

28,707
Posts
109,478
Replies
36,551
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.