Hello everyone,
I am relatively new to FME and have put together a conversion of 4 tables from a sql server database into a couple of embedded collections with a $lookup link in MongoDB. I've had a couple of issues getting the data directly into Mongo, but with a subset I have tested transforming it into a JSON and then loading the JSON into Mongo and that works fine.
My issue is seems to be with memory when i try to run the full dataset through. I have been stripping the translation back into smaller chunks so that my machine isn't trying to run the full process at ones, but now I'm at the stage where I cannot strip anything away. The first bit of translation I am trying to do is taking the 6.7M row table and creating unique MongoID's for each row and then saving it as a JSON, which is pretty straight forward. However, each time I have tried to run this with the full dataset I get to approximately 3.5M rows loaded and then the JSON writer closes due to memory issues. Below is a screenshot of the process and the error message:
2018-09-23 17:20:50|13322.1| 4.8|ERROR |Microsoft SQL Server Spatial Reader: Query failed, possibly due to a malformed statement. Query Text `[no query]'. Provider error `(-2147467259) Shared Memory Provider: No process is on the other end of the pipe.'
2018-09-23 17:20:51|13322.1| 0.0|INFORM|Aborting the JSON writer
2018-09-23 17:20:59|13325.7| 3.5|INFORM|Closing the JSON writer
Basically, it loads the data, add an ID, renames the and places it at the start of the features and then writes to JSON with the ID as the main featuretype.
I've tried changing the JSON writer to a featurewriter as I had read that some people saw improved performance from them, but to no avail. I've tried to run it as a batch process but it fails at the same point. I have also opened the header and added the FME_ENGINE_MEMORY_REDLINE clause with as low as 0.1 and am still having issues.
When i read up on the error it appeared that there may have been a problem with the data that I am loading, but i have run checks against the database which brought up no problems and have tried simply loading all the data into FME without running it through any processes and it worked fine.
Is there something that anybody can think of that I might have missed as I am still new to FME and at my wits end. Hopefully somebody recognises this as something basic, but any advice you guys might have is welcome.
Thanks in advance for your help.
Danny
