Skip to main content

I have a 2GB .ebc file that I need to pull in for transformation. The file has COMP-3 packing set for multiple (but not all) fields.

 

I've tried pulling this file in as a raw text file with default encoding, which results in one extremely long string record that crashes Inspector due to lack of memory.

 

When I try pulling it in via the included EBCDIC US (ibm037) encoding, it sees 100,000+ records but I again run into memory issues on the first record so I cannot see if the reader was truly successful.

 

Is there some trick to reading EBCDIC COMP-3 packed data that I'm missing? Or is this something that requires extra external processing work prior to ingesting the data into FME?

Hmmm. It's quite rare to see this encoding so I don't have much experience with it in FME. What you could try is using the EBCDIC encoding but also setting the reader parameter (in the Navigator window, usually on the left-hand side) called Features to Read > Max Features to Read to 1. Then it will only try reading the first record. That might help at least show if FME is reading the data correctly. I'm hoping it shouldn't fail with memory issues on the very first record (also make sure that the parameter Read Whole File at Once = No).

 

You're probably using the Text File format reader. You could try a couple of other things: there's a Data File format, which might make a difference. Also you could try using an AttributeFileReader transformer, to read the file into an attribute. From there an AttributeSplitter/ListExploder combo would split the data into separate records.

 

Finally, I'd be interested in what error messages are occuring. Does FME crash? Or does it log an error in the log window? Also, is the log showing any performance issues (like a message stating it is reorganising memory use)?

 

That's about all I have, I'm afraid. If you're still having problems, it might be worth logging a support case. I don't see why FME couldn't read an EBCDIC file (in fact I see specific examples that tells me that it should) so it would be useful to let our support team examine the file and see if something is wrong with it (or maybe FME is having problems with the packing).


Reply