Solved

Pick up Latest File from Folder Location

  • 1 November 2019
  • 9 replies
  • 116 views
F
Badge

Hi,

Newbie type question. I am currently loading in files to a output on a weekly basis. Issue has been I have to edit the file in the workbench with the new filename.

The file I upload changes on a daily basis with its name. How can FME pick up the latest file that has been updated without having to change the file path on a daily basis?

I have tried to use this https://knowledge.safe.com/questions/65203/read-only-the-newest-file-from-a-directory.html method but unfortunately I am not getting it to work. I also do not need to generate any further data to my existing table, just was interested in the function where it can automatically pick up the latest file at folder level by date modified.

 

Thank You

icon

Best answer by danullen 1 November 2019, 12:20

View original

9 replies

D
Badge

1. As your source, use a Directory and File Pathnames reader.

 

2. A Sorter that sorts descending on path_modified_date.

 

3. A Sampler that only lets the first feature pass.

 

4. A FeatureReader where Dataset is set to path_windows.
nielsgerrits
Rank
Userlevel 6
Badge +33

1. As your source, use a Directory and File Pathnames reader.

 

2. A Sorter that sorts descending on path_modified_date.

 

3. A Sampler that only lets the first feature pass.

 

4. A FeatureReader where Dataset is set to path_windows.

This is why I believe the FeatureReader and FeatureWriter are the biggest game changing transformers in a long time.

F
Badge

1. As your source, use a Directory and File Pathnames reader.

 

2. A Sorter that sorts descending on path_modified_date.

 

3. A Sampler that only lets the first feature pass.

 

4. A FeatureReader where Dataset is set to path_windows.

path2none.fmwThanks Dan - I have put this together, is this correct in the way its working. In terms of then adding the rest of my workbench in terms of filtering out records, does this get attached towards the schema.

path2none.fmw

D
Badge

path2none.fmwThanks Dan - I have put this together, is this correct in the way its working. In terms of then adding the rest of my workbench in terms of filtering out records, does this get attached towards the schema.

path2none.fmw

Almost perfect I think. :)

You just need to change from ascending to descending in the Sorter, to get the newest file instead of the oldest.

F
Badge

Almost perfect I think. :)

You just need to change from ascending to descending in the Sorter, to get the newest file instead of the oldest.

Thanks Dan, i got this working now. In regards to seeing data after the featurereader, is there any specific changes that need to be made? At the featurereader I only get information in regards to the file it is will pick up. Thanks again.

D
Badge

Thanks Dan, i got this working now. In regards to seeing data after the featurereader, is there any specific changes that need to be made? At the featurereader I only get information in regards to the file it is will pick up. Thanks again.

Have you set the Format in the FeatureReader so that FME knows what it's looking at? If the Format is set correctly, and the Dataset points to a valid file (through the path_windows attribute) FME should be able to read the file and extract the features. If it still doesn't work, please post a screen dump of the FeatureReader settings or the .fmw.

F
Badge

Have you set the Format in the FeatureReader so that FME knows what it's looking at? If the Format is set correctly, and the Dataset points to a valid file (through the path_windows attribute) FME should be able to read the file and extract the features. If it still doesn't work, please post a screen dump of the FeatureReader settings or the .fmw.

Hi Dan, apologise for delay in response. I am getting the records to come through it seems as the file count of the file is 227. But I am getting a empty table with no attributes. It is a excel file. Capture.JPG path2none.fmw

D
Badge

Hi Dan, apologise for delay in response. I am getting the records to come through it seems as the file count of the file is 227. But I am getting a empty table with no attributes. It is a excel file. Capture.JPG path2none.fmw

At the bottom of the FeatureReader parameters dialog box you've got "<Generic> Port". You might need to go in there and expose your attributes to get them to show.

F
Badge

At the bottom of the FeatureReader parameters dialog box you've got "<Generic> Port". You might need to go in there and expose your attributes to get them to show.

That seems to be the issue. I have manually setup the attributes, so this now has picked up the attributes. This process is good as long as the source file does not change.

Reply


Community Stats

28,731
Posts
109,530
Replies
36,565
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.