Skip to main content
Solved

Batch load files of different types perform different action depending upon type

  • November 25, 2015
  • 11 replies
  • 31 views
T
Forum|alt.badge.img
Hello I am brand new FME this week and am intrigued by it's non spatial data integration possibilities. I have a variety of csv files to process. I know explicitly how the header of each file is setup so would like to be able to test for certain conditions i.e. 1st two rows blank and (row 3 column 1 = 'Datetime') is Filetype A if successful this will go forward to relevant workspace runner for that filetype so I can do the required processing. Is it possible to test the structure of CSV before determining how to process it? Thanks and kind regards

Best answer by tomcrumbleholme

Thanks everyone for your pointers. I have got it done I don't know if it is the most elegant but it works.

I have a series of test for each file type if they pass I let one Feature through and create an attribute 'Filetype' and give it the relevant number. This is then appended to a csv file detailing filename and filetype. I am really enjoying using this tool goodness knows how long this would have taken in VBA ;-)
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

11 replies

david_r
Celebrity
  • david_rCelebrity
  • November 25, 2015
Hi You could analyze the contents of the files using the Text File reader, which will return one feature per text line. You could use a Counter to keep track of which line whatever text appears on. David
gio
Contributor
Forum|alt.badge.img+15
  • gioContributor
  • Contributor
  • November 25, 2015
You can analyse and parse any texfile. Adn then decide what to do with it. Useing textfile-reader you get ,amongst others, attributes text_line and line_number. Main tools would be of course regular expressions, variableSetter and Retriever. Possibly some stringoperations. A csv is, afaik, already ordered though, all you would need to do is read it. Set settings in reader parameter.
T
Forum|alt.badge.img
Hi You could analyze the contents of the files using the Text File reader, which will return one feature per text line. You could use a Counter to keep track of which line whatever text appears on. David

Thanks @david_r I will give it a try and report back

T
Forum|alt.badge.img
You can analyse and parse any texfile. Adn then decide what to do with it. Useing textfile-reader you get ,amongst others, attributes text_line and line_number. Main tools would be of course regular expressions, variableSetter and Retriever. Possibly some stringoperations. A csv is, afaik, already ordered though, all you would need to do is read it. Set settings in reader parameter.

Thanks @gio I will also give this a try. I will only need to analyse the first 12 lines of any file max so could do with any hints to limiting how much of the file it parses. Thanks again

jdh
Contributor
Forum|alt.badge.img+40
  • jdhContributor
  • Contributor
  • November 25, 2015
On the CSV reader you can set the Max Features to read to 12, as there is one feature per line.
T
Forum|alt.badge.img
  • tomcrumbleholme
  • Author
  • Best Answer
  • November 26, 2015
Thanks everyone for your pointers. I have got it done I don't know if it is the most elegant but it works.

I have a series of test for each file type if they pass I let one Feature through and create an attribute 'Filetype' and give it the relevant number. This is then appended to a csv file detailing filename and filetype. I am really enjoying using this tool goodness knows how long this would have taken in VBA ;-)
fmelizard
Safer
Forum|alt.badge.img+21
  • fmelizardSafer
  • Safer
  • November 26, 2015
Thanks everyone for your pointers. I have got it done I don't know if it is the most elegant but it works.

I have a series of test for each file type if they pass I let one Feature through and create an attribute 'Filetype' and give it the relevant number. This is then appended to a csv file detailing filename and filetype. I am really enjoying using this tool goodness knows how long this would have taken in VBA ;-)

Good job @tomcrumbleholme -- my only suggestion would be to consider the TestFilter instead of the cascading Testers -- that might clean it up a bit. Glad you're enjoying FME. Warning -- it can be addictive!

T
Forum|alt.badge.img

Good job @tomcrumbleholme -- my only suggestion would be to consider the TestFilter instead of the cascading Testers -- that might clean it up a bit. Glad you're enjoying FME. Warning -- it can be addictive!

Thanks Dale, so I can just have series of else if statements in the filter?

mark2atsafe
Safer
Forum|alt.badge.img+59
  • mark2atsafeSafer
  • Safer
  • November 26, 2015

Thanks Dale, so I can just have series of else if statements in the filter?

Yes, each line in the TestFilter is equivalent to an else if statement. For more info on the Tester vs TestFilter, check out: https://blog.safe.com/2013/03/fmeevangelist113/ and https://blog.safe.com/2015/05/fmeevangelist133/

FME Evangelist to the Rich and Famous!!
T
Forum|alt.badge.img
Update: Even more elegant use AttributeCreator then set to conditional value. Will upload cleaned up workbench later
mark2atsafe
Safer
Forum|alt.badge.img+59
  • mark2atsafeSafer
  • Safer
  • November 26, 2015
Update: Even more elegant use AttributeCreator then set to conditional value. Will upload cleaned up workbench later

Well of course! Don't know why we didn't think of that ourselves :-)

FME Evangelist to the Rich and Famous!!

Community Stats

32,154
Posts
122,043
Replies
39,812
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.