Skip to main content
Question

Is there a file size limit when using the AttributeFileReader?


I have it reading some JSON files and some go through and others dont with an error of "INVALID_FILE_INACCESSIBLE". The file sizes range from 500Mb to 8Gb in size.

Forum|alt.badge.img+2
  • February 9, 2023

@markb​ the size of an attribute value should only be limited by the available memory. If you are reading many, very large files at the same time then that might cause an issue. There may be some places where FME still has hard limit on the variable value size which would be a bug - we'll investigate and fix those if we can.


  • February 10, 2023
markatsafe wrote:

@markb​ the size of an attribute value should only be limited by the available memory. If you are reading many, very large files at the same time then that might cause an issue. There may be some places where FME still has hard limit on the variable value size which would be a bug - we'll investigate and fix those if we can.

@Mark Stoakes​ Thank you for the reply, I could see that it was rejecting the larger files straight away so I don't think it was reading it into memory but I could be wrong.

I have now changed it to use a text file FeatureReader which is working well.


mferwerda3
Contributor
Forum|alt.badge.img+5

I am seeing similar, it seems to fail immediately if the file size is above 2 gb


DanAtSafe
Safer
Forum|alt.badge.img+18

Yes, there appears to be a 2GB limit.  I’ve filed FMEENGINE-82398 about this.


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings