Skip to main content
Question

Read files with no line endings i.e. read 100 byte chunks at a time


tim_wood
Contributor
Forum|alt.badge.img+8

A colleague wants to read files that have no line ending (either Windows-style CR+LF or Unix-style LF). They have fixed record lengths (100 bytes). Can FME read this type of file (I'm betting it can!)? It needs to process them in 100 byte chunks.

If FME can't do it directly, is there something on the Hub e.g. this?

https://hub.safe.com/transformers/filerawdatafetcher

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

6 replies

david_r
Celebrity
  • March 29, 2019

There may be other solutions, but it's fairly easy to accomplish using a PythonCaller, something like:

1class FeatureCreator(object):
2    def __init__(self):
3        pass
4    def input(self,feature):
5        f = open("input_file.txt", "r")
6        chunk = 100
7        while True:
8            data = f.read(chunk)
9            if not data:
10                break
11            newFeature = fmeobjects.FMEFeature()
12            newFeature.setAttribute('chunk' data)
13            self.pyoutput(newFeature)
14    def close(self):
15        pass

Will return one feature with an attribute chunk for each chunk of 100 bytes in the file input_file.txt


bruceharold
Supporter
Forum|alt.badge.img+18
  • Supporter
  • March 29, 2019

i would attempt this with first getting the file into an attribute with AttributeFileReader then making a list from the record layout with AttributeSplitter.


bruceharold
Supporter
Forum|alt.badge.img+18
  • Supporter
  • March 29, 2019
david_r wrote:

There may be other solutions, but it's fairly easy to accomplish using a PythonCaller, something like:

1class FeatureCreator(object):
2    def __init__(self):
3        pass
4    def input(self,feature):
5        f = open("input_file.txt", "r")
6        chunk = 100
7        while True:
8            data = f.read(chunk)
9            if not data:
10                break
11            newFeature = fmeobjects.FMEFeature()
12            newFeature.setAttribute('chunk' data)
13            self.pyoutput(newFeature)
14    def close(self):
15        pass

Will return one feature with an attribute chunk for each chunk of 100 bytes in the file input_file.txt

Here i was avoiding mentioning Python and David has no qualms!  The struct module is a candidate too ;-)


david_r
Celebrity
  • March 29, 2019
bruceharold wrote:

Here i was avoiding mentioning Python and David has no qualms! The struct module is a candidate too ;-)

It's late Friday, my qualms have officially left the building ;-)


david_r
Celebrity
  • March 29, 2019
bruceharold wrote:

i would attempt this with first getting the file into an attribute with AttributeFileReader then making a list from the record layout with AttributeSplitter.

This is a good solution, given that the input file isn't very (very) large.


tim_wood
Contributor
Forum|alt.badge.img+8
  • Author
  • Contributor
  • March 29, 2019

Thanks both for your suggestions. This was partly exploratory to see if FME could be used for this particular job. I'll try make sure I come back and post which solution was used and accept the relevant answer.


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings