Hi - Performance tuning your FME workspaces can sometimes take a while as there's so much to consider. The following articles should give you some tips of where to start:
https://knowledge.safe.com/articles/44349/writers-and-performance.html
Depending on which dataset you read first, this can impact some Transformers that can perform quicker is a certain set of features arrive first. Take a look at:
http://fmepedia.safe.com/articles/FAQ/Group-Based-Transformers
Overall though this can help:
http://fmepedia.safe.com/articles/How_To/Performance-Tuning-FME
General rules to consider are:
- If you don't need it, for the process or at the end, get rid of it ASAP. That goes for attributes and geometry.
- Read data from 1 disk and write to another disk to avoid I/O problems... and make sure your issue is not compounded by a slow network connection.
- Ensure that you have assigned some virtual memory on your machine, this gives you some flexibilty should you run out of physical RAM. Things will slow down, but at least they might have a chance of finishing.
- Consider splitting your FME process into more than one part. Process 1 does some basic filtering of data to keep just what is necessary and loads to FFS format. Process 2 does the first geoprocessing tasks and loads to FFS. Process 3 finishes the tasks and loads to the target format. All called with 1 batch file.
Hopefully these tips may also help:
http://www.youtube.com/watch?v=ecR74ozMQdM
Best of luck, Dave
Hello Dave,
Thanks for answering my question in such a detailed and clear way.
The writer order is quite useful, I thought the opposite order would be more efficient , but I guess it makes sense.
I'm currently running only one reader so that can't be the problem.
I currently run with 36GB initial ram with the option to have 100GB of ram of neccesary. In my view, this shouldn't give any problems, but I do wonder if FME actually makes use of the total RAM or just a portion. The log file gives me the following:
2013-01-28 14:11:41| 0.5| 0.0|INFORM|FME Configuration: Temporary directory is `C:\\Users\\BOYEN~1.VAN\\AppData\\Local\\Temp' 2013-01-28 14:11:41| 0.5| 0.0|INFORM|FME Configuration: FME_HOME is `C:\\Program Files (x86)\\FME 2012\\' 2013-01-28 14:11:41| 0.5| 0.0|INFORM|FME Configuration: FME_BASE is 'no' 2013-01-28 14:11:41| 0.5| 0.0|INFORM|FME Configuration: FME_MF_NAME is 'database_writerv2.fmw' 2013-01-28 14:11:41| 0.5| 0.0|INFORM|FME Configuration: FME_PRODUCT_NAME is 'FME(R) 2012 SP4' 2013-01-28 14:11:41| 0.5| 0.0|INFORM|System Status: 404320MB of disk space available in the FME temporary directory (C:\\Users\\BOYEN~1.VAN\\AppData\\Local\\Temp\\) 2013-01-28 14:11:41| 0.5| 0.0|INFORM|System Status: 3922MB of virtual memory available
It seems like that could be the issue, FME only reads 4GB while 36GB is available. The disk space can't be limited I think, considering the file size is about 1.5gig total.
The crash occurs during writing of TIF files, when ~100 files of the 11000 files are written. With a smaller dataset it doesn't crash.
Ah well, you are certainly "tooled up" when it comes to RAM!!! Note though that the log snippet is highlighting the Virtual Memory available not the physical memory. That too will be in the log but to know what you can use we need to know the FME bit version and the OS you are using.
Ok, so the general rules are as follows:
- 32Bit FME on a 32Bit OS can utilise 2GB of system RAM (and whatever VM you have assigned).
- Unless on an OS like WinXP or similar you have turned on the /3G switch. In this case FME can get access to 3GB of addressable memory. This is a boot.ini hack. See here:
http://fmepedia.safe.com/articles/FAQ/Using-the-3GB-Switch-with-FME
- 32Bit FME on a 64Bit OS can utilise 4GB of RAM... so we are getting better, but in your case with your heavy duty and large file processing you might consider...
- 64Bit FME on a 64Bit OS, as you'll then be able to use all the RAM you have access to.
64bit FME is a separate download, though you can install side by side and use with the same license... available here:
http://www.safe.com/support/support-resources/fme-downloads/64-bit/
Beware though, 9 times out of 10 the best option is currently 32Bit FME on a 64Bit OS, because the 64Bit FME does not support all of the formats. Check here first to ensure your formats are supported by the 64Bit version:
http://www.safe.com/fme/format-search/
...sort by Platform on the page.
I'm using FME2012 32 bit on a 64bit machine so that's where the RAM restriction comes from. Makes sense. I'll get the 64 bit version. Thanks!
I am assuming that this should be possible with a 32bit version if I tune my workspace by splitting up the processes like you said, I will look into that as well. I might even be able to do batch deploy if that works out. I was always under the impression that FME optimized the virtual memory usage by itself.
Certainly without seeing your data I'd imagine it would be fine if a) you tune the process and b) you look at a batch approach.
On that front, the FME Batch Deploy function can be used but personally I have more success with the WorkspaceRunner. Here you can have a master and slave scenario where Workspace1 tells Workspace2 what to do... for this though you'll need to setup some published parameters. An example would be Workspace1 creates 100 string values each is a SQL Query string to select out just a small area of data... These are sent 1 at a time to Workspace2 (calling it 100 times), the same process then is actioned on different data each time.
Regarding virtual memory, FME does optimise memory useage but it depends if your VM is set to System Managed on the OS I believe. I find hard setting this more effective and allocating a good chunk of disk space as VM... after all disk space is usually pretty readily available.
Hi,
I think looking at this link may be usefull for you!
https://groups.google.com/forum/?fromgroups#!topic/fmetalk/YlOXBKDhgZ4
The "FME_ENGINE_MEMORY_REDLINE" still works and I am using it in all my workbenches.
Just put it in the first row of the header of your workbench.
FME_ENGINE_MEMORY_REDLINE 0.8
#! START_WB_HEADER
READER_TYPE MULTI_READER
MULTI_READER_TYPE{0} GEODATABASE_MDB
MULTI_READER_KEYWORD{0} GEODATABASE_MDB_1
You will have to check out the right value for your data.
Using FME 32 bit on a 64 bit windows
some log of my worckbenches look like this:
2013-01-28 17:53:31| 0.1| 0.0|INFORM|FME_ENGINE_MEMORY_REDLINE is set to 80%
2013-01-28 17:53:31| 0.1| 0.0|INFORM|FME Configuration: Process limit is 4095 MB (out of 24574 MB physical memory and 4095 MB address space)
2013-01-28 17:53:31| 0.1| 0.0|INFORM|FME Configuration: Start freeing memory when process usage exceeds 3615 MB of memory or 3855 MB of address space
2013-01-28 17:53:31| 0.1| 0.0|INFORM|FME Configuration: Stop freeing memory when process usage is below 2711 MB of memory and 2891 MB of address space
2013-01-28 17:53:31| 0.1| 0.0|INFORM|FME Configuration: Autodetermining optimal maximum number of objects in memory
Hi,
looking at this link may be usefull for you!
https://groups.google.com/forum/?fromgroups#!topic/fmetalk/YlOXBKDhgZ4
The "FME_ENGINE_MEMORY_REDLINE" still works.
I am using it in all of my workbenches.
Just put it on top of your header.
FME_ENGINE_MEMORY_REDLINE 0.8
#! START_WB_HEADER
READER_TYPE MULTI_READER
MULTI_READER_TYPE{0} GEODATABASE_MDB
You will have to find the right parameter for your data.
Some log of my workbenches looks like this:
2013-01-28 17:53:31| 0.1| 0.0|INFORM|FME_ENGINE_MEMORY_REDLINE is set to 80%
2013-01-28 17:53:31| 0.1| 0.0|INFORM|FME Configuration: Process limit is 4095 MB (out of 24574 MB physical memory and 4095 MB address space)
2013-01-28 17:53:31| 0.1| 0.0|INFORM|FME Configuration: Start freeing memory when process usage exceeds 3615 MB of memory or 3855 MB of address space
2013-01-28 17:53:31| 0.1| 0.0|INFORM|FME Configuration: Stop freeing memory when process usage is below 2711 MB of memory and 2891 MB of address space
2013-01-28 17:53:31| 0.1| 0.0|INFORM|FME Configuration: Autodetermining optimal maximum number of objects in memory
I did a combination of the things mentioned and so far - the process is running smoother and hasn't crashed yet. The ETA is about 3 days so I can't be sure until it is finished but it looks like it is working now.
So just for people that come across the same problem what I've done so far:
- Streamline the process make sure you get rid of attributes you don't need ASAP.
-Read and write from/to different disks.
-Prioritize writers
-Use the 64bit version on a 64 bit machine
-Assign extra virtual memory
-Split the process into dedicated processes
-Put FME_ENGINE_MEMORY_REDLINE on 0.2, just to be safe
To do - would certainly work but too difficult for me as a novice:
-Batch
-Workspacerunner
Thanks a lot for the help, I appreciate it!
Boyen - It would be great for completeness to hear the results when the 3 or so days are up. This is a topic that every FME user bumps into as soon as their data volume starts to increase. Fingers crossed...
Hey Dave -
Just to report back. My FME script completed its task and all worked perfectly. Learned a lot about optimizing the workspace here, thanks a lot!
