It took 2h to reproject the first one. now we are at # 32 and it's taking around 4h. Is there a way of doing a memory dump or starting fresh after each raster when doing batch reprojection?
Thank you.
Dominic
Best answer by davideagle
View original
FME getting slower when working with lots of Rasters.
I am reprojecting Rasters, 63 of them. They are all pretty much the same size 1.1GB.
Userlevel 4
+13
- Contributor
- 3725 replies
-
19 November 2012
Hi Dominic,
FME tends to save all the data into ffs files in the FME temp location.
You can break up the translation into individual batch processes handling 10 to 15 rasters per batch. Deploy them all via the batch deploy and edit 1 .bat file to contain them all. Run a SQL or DOS command to delete the ffs files from the FME temp location, before each batch.
I remebmer doing something like this when reading LIDAR, before FME introduced the point cloud reader.
Hope this helps,
Itay
+14
- Contributor
- 577 replies
-
19 November 2012
- Best Answer
Consider a master and slave workspace approach. The master prepares the data and the slave does the work...as any good slave should! The master consists of a Directory and File Pathnames Reader, this reads where the raster data is, not the data itself, puts the file path into an attribute and then that value is passed to a WorkspaceRunner. 63 features are sent to the WorkspaceRunner and runs the slave workspace 63 times each time with a different input raster. Between each run the fme.exe is started and stopped and your memory usage is dumped/refreshed. If you have FMEServer you can replace the WorkspaceRunner with the FMEServerJobSubmitter and the process will be quicker, especially if you have more than one engine.
+21
- Contributor
- 955 replies
-
20 November 2012
Just FYI, not sure if you are running a very old version of FME? Make sure you use FME after build 12161 atleast, so you have the raster-enhancement mentioned in this post: https://groups.google.com/forum/?fromgroups=#!searchin/fmetalk/raster$20jpeg$20/fmetalk/2couqDKlwcQ/3UAXp1TcpHEJ
@1Spatial The only limitation I have found with the WorkspaceRunner or FMEServerJobSubmitter is the length of attributes. Ie you can not add an attribute for instance _geom in FMEBinary (or other) with a lot of data (I guess thats a URL or DOS-thingy)
Userlevel 4
+13
- Contributor
- 3725 replies
-
21 November 2012
Hi,
Another tip is to remove/disable any visualizers in the ws, this will prevent FME from writing any temp files.
Hope this helps,
Itay
Reply
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.