Skip to main content
Solved

Pick up Latest File from Folder Location

  • November 1, 2019
  • 9 replies
  • 445 views
F
Forum|alt.badge.img

Hi,

Newbie type question. I am currently loading in files to a output on a weekly basis. Issue has been I have to edit the file in the workbench with the new filename.

The file I upload changes on a daily basis with its name. How can FME pick up the latest file that has been updated without having to change the file path on a daily basis?

I have tried to use this https://knowledge.safe.com/questions/65203/read-only-the-newest-file-from-a-directory.html method but unfortunately I am not getting it to work. I also do not need to generate any further data to my existing table, just was interested in the function where it can automatically pick up the latest file at folder level by date modified.

 

Thank You

Best answer by danullen

1. As your source, use a Directory and File Pathnames reader.

 

2. A Sorter that sorts descending on path_modified_date.

 

3. A Sampler that only lets the first feature pass.

 

4. A FeatureReader where Dataset is set to path_windows.
View original
Did this help you find an answer to your question?

9 replies

D
Forum|alt.badge.img
  • danullen
  • Best Answer
  • November 1, 2019

1. As your source, use a Directory and File Pathnames reader.

 

2. A Sorter that sorts descending on path_modified_date.

 

3. A Sampler that only lets the first feature pass.

 

4. A FeatureReader where Dataset is set to path_windows.
nielsgerrits
VIP
danullen wrote:

1. As your source, use a Directory and File Pathnames reader.

 

2. A Sorter that sorts descending on path_modified_date.

 

3. A Sampler that only lets the first feature pass.

 

4. A FeatureReader where Dataset is set to path_windows.

This is why I believe the FeatureReader and FeatureWriter are the biggest game changing transformers in a long time.

F
Forum|alt.badge.img
  • fazsnatch
  • Author
  • November 1, 2019
danullen wrote:

1. As your source, use a Directory and File Pathnames reader.

 

2. A Sorter that sorts descending on path_modified_date.

 

3. A Sampler that only lets the first feature pass.

 

4. A FeatureReader where Dataset is set to path_windows.

path2none.fmwThanks Dan - I have put this together, is this correct in the way its working. In terms of then adding the rest of my workbench in terms of filtering out records, does this get attached towards the schema.

path2none.fmw

D
Forum|alt.badge.img
  • danullen
  • November 1, 2019
fazsnatch wrote:

path2none.fmwThanks Dan - I have put this together, is this correct in the way its working. In terms of then adding the rest of my workbench in terms of filtering out records, does this get attached towards the schema.

path2none.fmw

Almost perfect I think. :)

You just need to change from ascending to descending in the Sorter, to get the newest file instead of the oldest.

F
Forum|alt.badge.img
  • fazsnatch
  • Author
  • November 1, 2019
danullen wrote:

Almost perfect I think. :)

You just need to change from ascending to descending in the Sorter, to get the newest file instead of the oldest.

Thanks Dan, i got this working now. In regards to seeing data after the featurereader, is there any specific changes that need to be made? At the featurereader I only get information in regards to the file it is will pick up. Thanks again.

D
Forum|alt.badge.img
  • danullen
  • November 1, 2019
fazsnatch wrote:

Thanks Dan, i got this working now. In regards to seeing data after the featurereader, is there any specific changes that need to be made? At the featurereader I only get information in regards to the file it is will pick up. Thanks again.

Have you set the Format in the FeatureReader so that FME knows what it's looking at? If the Format is set correctly, and the Dataset points to a valid file (through the path_windows attribute) FME should be able to read the file and extract the features. If it still doesn't work, please post a screen dump of the FeatureReader settings or the .fmw.

F
Forum|alt.badge.img
  • fazsnatch
  • Author
  • November 4, 2019
danullen wrote:

Have you set the Format in the FeatureReader so that FME knows what it's looking at? If the Format is set correctly, and the Dataset points to a valid file (through the path_windows attribute) FME should be able to read the file and extract the features. If it still doesn't work, please post a screen dump of the FeatureReader settings or the .fmw.

Hi Dan, apologise for delay in response. I am getting the records to come through it seems as the file count of the file is 227. But I am getting a empty table with no attributes. It is a excel file. Capture.JPG path2none.fmw

D
Forum|alt.badge.img
  • danullen
  • November 4, 2019
fazsnatch wrote:

Hi Dan, apologise for delay in response. I am getting the records to come through it seems as the file count of the file is 227. But I am getting a empty table with no attributes. It is a excel file. Capture.JPG path2none.fmw

At the bottom of the FeatureReader parameters dialog box you've got "<Generic> Port". You might need to go in there and expose your attributes to get them to show.

F
Forum|alt.badge.img
  • fazsnatch
  • Author
  • November 4, 2019
danullen wrote:

At the bottom of the FeatureReader parameters dialog box you've got "<Generic> Port". You might need to go in there and expose your attributes to get them to show.

That seems to be the issue. I have manually setup the attributes, so this now has picked up the attributes. This process is good as long as the source file does not change.

Reply


Helpful Members This Week

Community Stats

30,909
Posts
117,110
Replies
38,708
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.