Skip to main content
Solved

Batch load files of different types perform different action depending upon type

T
Forum|alt.badge.img
Hello I am brand new FME this week and am intrigued by it's non spatial data integration possibilities. I have a variety of csv files to process. I know explicitly how the header of each file is setup so would like to be able to test for certain conditions i.e. 1st two rows blank and (row 3 column 1 = 'Datetime') is Filetype A if successful this will go forward to relevant workspace runner for that filetype so I can do the required processing. Is it possible to test the structure of CSV before determining how to process it? Thanks and kind regards

Best answer by tomcrumbleholme

Thanks everyone for your pointers. I have got it done I don't know if it is the most elegant but it works.

I have a series of test for each file type if they pass I let one Feature through and create an attribute 'Filetype' and give it the relevant number. This is then appended to a csv file detailing filename and filetype. I am really enjoying using this tool goodness knows how long this would have taken in VBA ;-)
View original
Did this help you find an answer to your question?
This post is closed to further activity.
It may be a question with a best answer, an implemented idea, or just a post needing no comment.
If you have a follow-up or related question, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

11 replies

david_r
Celebrity
  • david_rCelebrity
  • November 25, 2015
Hi You could analyze the contents of the files using the Text File reader, which will return one feature per text line. You could use a Counter to keep track of which line whatever text appears on. David
gio
Contributor
Forum|alt.badge.img+15
  • gioContributor
  • Contributor
  • November 25, 2015
You can analyse and parse any texfile. Adn then decide what to do with it. Useing textfile-reader you get ,amongst others, attributes text_line and line_number. Main tools would be of course regular expressions, variableSetter and Retriever. Possibly some stringoperations. A csv is, afaik, already ordered though, all you would need to do is read it. Set settings in reader parameter.
T
Forum|alt.badge.img
david_r wrote:
Hi You could analyze the contents of the files using the Text File reader, which will return one feature per text line. You could use a Counter to keep track of which line whatever text appears on. David

Thanks @david_r I will give it a try and report back

T
Forum|alt.badge.img
gio wrote:
You can analyse and parse any texfile. Adn then decide what to do with it. Useing textfile-reader you get ,amongst others, attributes text_line and line_number. Main tools would be of course regular expressions, variableSetter and Retriever. Possibly some stringoperations. A csv is, afaik, already ordered though, all you would need to do is read it. Set settings in reader parameter.

Thanks @gio I will also give this a try. I will only need to analyse the first 12 lines of any file max so could do with any hints to limiting how much of the file it parses. Thanks again

jdh
Contributor
Forum|alt.badge.img+28
  • jdhContributor
  • Contributor
  • November 25, 2015
On the CSV reader you can set the Max Features to read to 12, as there is one feature per line.
T
Forum|alt.badge.img
Thanks everyone for your pointers. I have got it done I don't know if it is the most elegant but it works.

I have a series of test for each file type if they pass I let one Feature through and create an attribute 'Filetype' and give it the relevant number. This is then appended to a csv file detailing filename and filetype. I am really enjoying using this tool goodness knows how long this would have taken in VBA ;-)
fmelizard
Safer
Forum|alt.badge.img+19
  • fmelizardSafer
  • Safer
  • November 26, 2015
tomcrumbleholme wrote:
Thanks everyone for your pointers. I have got it done I don't know if it is the most elegant but it works.

I have a series of test for each file type if they pass I let one Feature through and create an attribute 'Filetype' and give it the relevant number. This is then appended to a csv file detailing filename and filetype. I am really enjoying using this tool goodness knows how long this would have taken in VBA ;-)

Good job @tomcrumbleholme -- my only suggestion would be to consider the TestFilter instead of the cascading Testers -- that might clean it up a bit. Glad you're enjoying FME. Warning -- it can be addictive!

T
Forum|alt.badge.img
fmelizard wrote:

Good job @tomcrumbleholme -- my only suggestion would be to consider the TestFilter instead of the cascading Testers -- that might clean it up a bit. Glad you're enjoying FME. Warning -- it can be addictive!

Thanks Dale, so I can just have series of else if statements in the filter?

mark2atsafe
Safer
Forum|alt.badge.img+46
  • mark2atsafeSafer
  • Safer
  • November 26, 2015
tomcrumbleholme wrote:

Thanks Dale, so I can just have series of else if statements in the filter?

Yes, each line in the TestFilter is equivalent to an else if statement. For more info on the Tester vs TestFilter, check out: https://blog.safe.com/2013/03/fmeevangelist113/ and https://blog.safe.com/2015/05/fmeevangelist133/

T
Forum|alt.badge.img
Update: Even more elegant use AttributeCreator then set to conditional value. Will upload cleaned up workbench later
mark2atsafe
Safer
Forum|alt.badge.img+46
  • mark2atsafeSafer
  • Safer
  • November 26, 2015
tomcrumbleholme wrote:
Update: Even more elegant use AttributeCreator then set to conditional value. Will upload cleaned up workbench later

Well of course! Don't know why we didn't think of that ourselves :-)

Helpful Members This Week

Community Stats

31,504
Posts
119,437
Replies
39,128
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.