Question

Is there a file size limit when using the AttributeFileReader?

  • 8 February 2023
  • 4 replies
  • 38 views
M

I have it reading some JSON files and some go through and others dont with an error of "INVALID_FILE_INACCESSIBLE". The file sizes range from 500Mb to 8Gb in size.

4 replies

M
Badge +2

@markb​ the size of an attribute value should only be limited by the available memory. If you are reading many, very large files at the same time then that might cause an issue. There may be some places where FME still has hard limit on the variable value size which would be a bug - we'll investigate and fix those if we can.

M

@markb​ the size of an attribute value should only be limited by the available memory. If you are reading many, very large files at the same time then that might cause an issue. There may be some places where FME still has hard limit on the variable value size which would be a bug - we'll investigate and fix those if we can.

@Mark Stoakes​ Thank you for the reply, I could see that it was rejecting the larger files straight away so I don't think it was reading it into memory but I could be wrong.

I have now changed it to use a text file FeatureReader which is working well.

mferwerda3
Rank
Badge +4

I am seeing similar, it seems to fail immediately if the file size is above 2 gb

DanAtSafe
Rank
Userlevel 2
Badge +11

Yes, there appears to be a 2GB limit.  I’ve filed FMEENGINE-82398 about this.

Reply


Community Stats

28,732
Posts
109,538
Replies
36,570
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.