Skip to main content
Question

Read files with no line endings i.e. read 100 byte chunks at a time


tim_wood
Contributor
Forum|alt.badge.img+8

A colleague wants to read files that have no line ending (either Windows-style CR+LF or Unix-style LF). They have fixed record lengths (100 bytes). Can FME read this type of file (I'm betting it can!)? It needs to process them in 100 byte chunks.

If FME can't do it directly, is there something on the Hub e.g. this?

https://hub.safe.com/transformers/filerawdatafetcher

6 replies

david_r
Evangelist
  • March 29, 2019

There may be other solutions, but it's fairly easy to accomplish using a PythonCaller, something like:

class FeatureCreator(object):
    def __init__(self):
        pass
    def input(self,feature):
        f = open("input_file.txt""r")
        chunk = 100
        while True:
            data = f.read(chunk)
            if not data:
                break
            newFeature = fmeobjects.FMEFeature()
            newFeature.setAttribute('chunk' data)
            self.pyoutput(newFeature)
    def close(self):
        pass

Will return one feature with an attribute chunk for each chunk of 100 bytes in the file input_file.txt


bruceharold
Contributor
Forum|alt.badge.img+17
  • Contributor
  • March 29, 2019

i would attempt this with first getting the file into an attribute with AttributeFileReader then making a list from the record layout with AttributeSplitter.


bruceharold
Contributor
Forum|alt.badge.img+17
  • Contributor
  • March 29, 2019
david_r wrote:

There may be other solutions, but it's fairly easy to accomplish using a PythonCaller, something like:

class FeatureCreator(object):
    def __init__(self):
        pass
    def input(self,feature):
        f = open("input_file.txt""r")
        chunk = 100
        while True:
            data = f.read(chunk)
            if not data:
                break
            newFeature = fmeobjects.FMEFeature()
            newFeature.setAttribute('chunk' data)
            self.pyoutput(newFeature)
    def close(self):
        pass

Will return one feature with an attribute chunk for each chunk of 100 bytes in the file input_file.txt

Here i was avoiding mentioning Python and David has no qualms!  The struct module is a candidate too ;-)


david_r
Evangelist
  • March 29, 2019
bruceharold wrote:

Here i was avoiding mentioning Python and David has no qualms! The struct module is a candidate too ;-)

It's late Friday, my qualms have officially left the building ;-)


david_r
Evangelist
  • March 29, 2019
bruceharold wrote:

i would attempt this with first getting the file into an attribute with AttributeFileReader then making a list from the record layout with AttributeSplitter.

This is a good solution, given that the input file isn't very (very) large.


tim_wood
Contributor
Forum|alt.badge.img+8
  • Author
  • Contributor
  • March 29, 2019

Thanks both for your suggestions. This was partly exploratory to see if FME could be used for this particular job. I'll try make sure I come back and post which solution was used and accept the relevant answer.


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings