It took 2h to reproject the first one. now we are at # 32 and it's taking around 4h. Is there a way of doing a memory dump or starting fresh after each raster when doing batch reprojection?
Thank you.
Dominic
Solved
FME getting slower when working with lots of Rasters.
I am reprojecting Rasters, 63 of them. They are all pretty much the same size 1.1GB.
Best answer by davideagle
Consider a master and slave workspace approach. The master prepares the data and the slave does the work...as any good slave should! The master consists of a Directory and File Pathnames Reader, this reads where the raster data is, not the data itself, puts the file path into an attribute and then that value is passed to a WorkspaceRunner. 63 features are sent to the WorkspaceRunner and runs the slave workspace 63 times each time with a different input raster. Between each run the fme.exe is started and stopped and your memory usage is dumped/refreshed. If you have FMEServer you can replace the WorkspaceRunner with the FMEServerJobSubmitter and the process will be quicker, especially if you have more than one engine.
View originalThis post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
+22- Safer
Hi Dominic,
FME tends to save all the data into ffs files in the FME temp location.
You can break up the translation into individual batch processes handling 10 to 15 rasters per batch. Deploy them all via the batch deploy and edit 1 .bat file to contain them all. Run a SQL or DOS command to delete the ffs files from the FME temp location, before each batch.
I remebmer doing something like this when reading LIDAR, before FME introduced the point cloud reader.
Hope this helps,
Itay
+21- Contributor
- Best Answer
Consider a master and slave workspace approach. The master prepares the data and the slave does the work...as any good slave should! The master consists of a Directory and File Pathnames Reader, this reads where the raster data is, not the data itself, puts the file path into an attribute and then that value is passed to a WorkspaceRunner. 63 features are sent to the WorkspaceRunner and runs the slave workspace 63 times each time with a different input raster. Between each run the fme.exe is started and stopped and your memory usage is dumped/refreshed. If you have FMEServer you can replace the WorkspaceRunner with the FMEServerJobSubmitter and the process will be quicker, especially if you have more than one engine.
+25- Supporter
Just FYI, not sure if you are running a very old version of FME? Make sure you use FME after build 12161 atleast, so you have the raster-enhancement mentioned in this post: https://groups.google.com/forum/?fromgroups=#!searchin/fmetalk/raster$20jpeg$20/fmetalk/2couqDKlwcQ/3UAXp1TcpHEJ
@1Spatial The only limitation I have found with the WorkspaceRunner or FMEServerJobSubmitter is the length of attributes. Ie you can not add an attribute for instance _geom in FMEBinary (or other) with a lot of data (I guess thats a URL or DOS-thingy)
+22- Safer
Hi,
Another tip is to remove/disable any visualizers in the ws, this will prevent FME from writing any temp files.
Hope this helps,
Itay