this is in relation to https://knowledge.safe.com/questions/19565/tiler-c...,
I have ran it overnight and it still continues. I used 100 x 100 horizontal/Vertical.
I am not sure how long this would it take.
Here is my build, it has only outputted roughly about 10 files. I am not sure if its going to Tile it first and put the data into cache and split them. but basically takes forever.
Thanks!
Jubert
Best answer by takashi
I cannot find any problem in the data flow. However, looks like FME has performed memory optimization many times. This could take long time, and I guess it is caused by that many destination files will be created because of the "fanout". On the other hand, larger number of tiles takes longer time for tiling processing, naturally. This could also be a reason.
I think you will have to find the "bottleneck" where most time has been consumed. Firstly check the full log to find the bottleneck. Also I would try running the workspace in some test cases. e.g. without "fanout", and/or after reducing number of tiles.
Takashi
I cannot find any problem in the data flow. However, looks like FME has performed memory optimization many times. This could take long time, and I guess it is caused by that many destination files will be created because of the "fanout". On the other hand, larger number of tiles takes longer time for tiling processing, naturally. This could also be a reason.
I think you will have to find the "bottleneck" where most time has been consumed. Firstly check the full log to find the bottleneck. Also I would try running the workspace in some test cases. e.g. without "fanout", and/or after reducing number of tiles.
Takashi
2015-11-05 15:58:13|81134.4| 0.0|INFORM|Translation was SUCCESSFUL with 7 warning(s) (3310 feature(s) output)
2015-11-05 15:58:13|81134.4| 0.0|INFORM|FME Session Duration: 22 hours 21 minutes 59.3 seconds. (CPU: 57578.4s user, 23556.0s system)
2015-11-05 15:58:13|81134.4| 0.0|INFORM|END - ProcessID: 4496, peak process memory usage: 3342216 kB, current process memory usage: 453560 kB
Translation was SUCCESSFUL
+59
But like Takashi said, there isn't really anything wrong with the workflow. It's just the large dataset and using a fanout. The only thing I can think of to improve performance is to check where the temporary directory is (it will be mentioned in the log) and ensure it has plenty of space and is a fast disk. Often a different disk to what the operating system uses can help.
If you need to run this translation on a regular basis, feel free to send a copy of the data and workspace to the support team (safe.com/support) and we can run the process at Safe to see if it is taking longer than we would expect.
Thanks! the file size is at 378,236 KB not even on the GB range.
I will drop by safe.com/support to drop the file and see if it can be further optimized or it is what it is..
thanks!
As @markireland says - the 23556 seconds would have been disk thrashing.
In my experience the problem with fanout is that it caches the *entire* output before it then writes it. So it caches it, sorts it, and then writes it.
2015-11-05 15:58:13|81134.4| 0.0|INFORM|Translation was SUCCESSFUL with 7 warning(s) (3310 feature(s) output)
2015-11-05 15:58:13|81134.4| 0.0|INFORM|FME Session Duration: 22 hours 21 minutes 59.3 seconds. (CPU: 57578.4s user, 23556.0s system)
2015-11-05 15:58:13|81134.4| 0.0|INFORM|END - ProcessID: 4496, peak process memory usage: 3342216 kB, current process memory usage: 453560 kB
Translation was SUCCESSFUL
For what it's worth, we have some old workspaces that can take almost a week to complete, so 22 hours really isn't that bad ;-)
But if FME reports a lot of "memory optimization" that often means that it is short on memory and is starting to swap to disk. If you can, consider using 64-bit FME. Also check how much physical RAM you have on your machine. 32-bit FME can only use about 2GB of RAM, whereas 64-bit FME can use about 12GB of RAM (theoretically much more, but strange things happen after 12GB...). Thirdly, don't write to a network share or an external harddrive if performance is important to you, it can be a huge bottleneck. Writing to an SSD drive can also make for a very nice improvement.
David
