HI @patrick.gress,

That sounds more like a limitation of your system resources than a FME issue.

Are you by any chance running the translation with feature caching on? (that can be quite memory intensive)

Hi @itay,

I would not say that this is a resource limitation. The files are in the magnitude of some terabyte, something the database servers can handle, but not my local machine. The usual workflow would normally be like: take X rows from file and put them into the database. This would not take large amounts of memory, both ram and physical.

FME tries to read all data and then begins to put it into the databases, which is quite slow and takes large amounts of memory.

Feature caching is disabled, that was my first guess to.

Just to clarify by "your system resources" I did actually mean your local machine resources.

Could it be that the FME TEMP is not configured for optimal efficiency? https://knowledge.safe.com/articles/176/fme-temp-environment-variable.html

My local machine is not configured to handle terabytes of temporary files, that is correct.

If I would do the job with an external programming language like python I could do what I want with constant ram and physical memory, however, since the read-write pipeline has always the same size. I wonder if this could be done in FME, as well. I don't see the point of storing tons of data in order to write them in the future, when the data can be written and discarded at once.