Hi Jubert,

 

 

I cannot find any problem in the data flow. However, looks like FME has performed memory optimization many times. This could take long time, and I guess it is caused by that many destination files will be created because of the "fanout". On the other hand, larger number of tiles takes longer time for tiling processing, naturally. This could also be a reason.

 

I think you will have to find the "bottleneck" where most time has been consumed. Firstly check the full log to find the bottleneck. Also I would try running the workspace in some test cases. e.g. without "fanout", and/or after reducing number of tiles.

 

 

Takashi

---

It completed... After 22hours.. I guess patience is a virtue... @takashi thanks for confirming that the process is correct!

 

 

2015-11-05 15:58:13|81134.4| 0.0|INFORM|Translation was SUCCESSFUL with 7 warning(s) (3310 feature(s) output)

 

2015-11-05 15:58:13|81134.4| 0.0|INFORM|FME Session Duration: 22 hours 21 minutes 59.3 seconds. (CPU: 57578.4s user, 23556.0s system)

 

2015-11-05 15:58:13|81134.4| 0.0|INFORM|END - ProcessID: 4496, peak process memory usage: 3342216 kB, current process memory usage: 453560 kB

 

Translation was SUCCESSFUL

---

That data must have a very large number of vertices, given the small number of features and the huge amount of memory it took (peaking at 3,342,216kb = 3.3GB). The 23556 seconds of system time was probably caching back and forth to disk. What filesize is the original dataset?

 

 

But like Takashi said, there isn't really anything wrong with the workflow. It's just the large dataset and using a fanout. The only thing I can think of to improve performance is to check where the temporary directory is (it will be mentioned in the log) and ensure it has plenty of space and is a fast disk. Often a different disk to what the operating system uses can help.

 

 

If you need to run this translation on a regular basis, feel free to send a copy of the data and workspace to the support team (safe.com/support) and we can run the process at Safe to see if it is taking longer than we would expect.

 

---

Hi @markireland,

 

 

Thanks! the file size is at 378,236 KB not even on the GB range.

 

 

I will drop by safe.com/support to drop the file and see if it can be further optimized or it is what it is..

 

 

thanks!

---

I suspect the bottleneck was the fanout.

 

As @markireland says - the 23556 seconds would have been disk thrashing.

 

 

In
my experience the problem with fanout is that it caches the *entire*
output before it then writes it. So it caches it, sorts it, and then
writes it.

---

It completed... After 22hours.. I guess patience is a virtue... @takashi thanks for confirming that the process is correct!

 

 

2015-11-05 15:58:13|81134.4| 0.0|INFORM|Translation was SUCCESSFUL with 7 warning(s) (3310 feature(s) output)

 

2015-11-05 15:58:13|81134.4| 0.0|INFORM|FME Session Duration: 22 hours 21 minutes 59.3 seconds. (CPU: 57578.4s user, 23556.0s system)

 

2015-11-05 15:58:13|81134.4| 0.0|INFORM|END - ProcessID: 4496, peak process memory usage: 3342216 kB, current process memory usage: 453560 kB

 

Translation was SUCCESSFUL

For what it's worth, we have some old workspaces that can take almost a week to complete, so 22 hours really isn't that bad ;-)

But if FME reports a lot of "memory optimization" that often means that it is short on memory and is starting to swap to disk. If you can, consider using 64-bit FME. Also check how much physical RAM you have on your machine. 32-bit FME can only use about 2GB of RAM, whereas 64-bit FME can use about 12GB of RAM (theoretically much more, but strange things happen after 12GB...). Thirdly, don't write to a network share or an external harddrive if performance is important to you, it can be a huge bottleneck. Writing to an SSD drive can also make for a very nice improvement.

David

---