Question

SSD for FME temporary storage

  • 14 November 2012
  • 5 replies
  • 2 views

Hi all,

 

 

any experiences in terms of how much of the time we could save when running Workspace using SSD for temporary files storage instead of regular disk. Particularly, it would be interesting for following model situation: FME 2012 64bit Desktop, WS ran 20 hours and stored 250GB of temp files on regular disk (but separate one).

 

 

Thanks,

 

Jiri

5 replies

Userlevel 3
Badge +13
I suspect it may depend what the temp space is being used for.

 

In our case, its typically used to dump stuff from RAM when FME has too much stuff in it.

 

Typically FME will top out at 20GB RAM use (of the 32GB on the box) then write about 6GB of that to disk, taking it down to ~14GB before going continuing on.

 

It does this 6GB write all at once in a single continuous write. In this case therefore, there's no theoretical benefit to an SSD - they're not much faster than HDD's at continous reads or writes.

 

 

SSD's really shine when it comes to random accesses. I'm not as sure how FME reads data back in - if its randomly, SSD will be much better, if its continuously, there won't be much difference.
SSD's Random access and continous read/write are way better than a HDD i have no doubt about it. The thing is if FME can really benefit from it. I hope the staff could solve this question, because i hace the same doubt.
Badge +10
Its hard to say as it very much depends on your process and the type of temporary files you are using. Certainly it wouldn't be too difficult to test the theory... In case you are not familiar with the setting of your TEMP space, look here and set FME_TEMP as your defined path http://fmepedia.safe.com/articles/FAQ/FME-TEMP-environment-variable in fact I do this anyway just so I know where my temp space is and can get there quickly. Transformers like the InlineQuerier will actually create a database at this location for read/write and may be inserting and reading back features for the duration of a process so that is likely to be quicker if an SSD is in play.

 

 

Sounds like a lot of temp files though are you processing raster or LiDAR?Either way its very positive to hear that your workspace is completing given that volume.

 

 

For performance though its likely you could see greater gains by looking elsewhere first. If the workspace was originally built some FME versions ago and you are now using FME 2012 its likely some of the heavy lifting Transformers you might have used (Clipper, AreaBuilder, Bufferer, DEMGenerator, DonutBuilder, Intersector, NeighbourFinder, TINGenerator etc) have been overhauled and performance and memory use is often a factor that determines the revamp. Go and check if any Transformers that you have used have been superceded and replace them, see here for more:

 

 

http://fmepedia.safe.com/articles/How_To/Transformer-Versioning

 

http://fmepedia.safe.com/articles/How_To/Transformer-Versions

 

 

Its usually literally a case of deleting the old one and adding a new one from the Transformer gallery in the latest version of FME, check that there are no significant parameter changes though.

 

 

You may also see significant performance gains from FME 2012's multi-processing capability, though this will be more trial and error. Some of those same Transformers listed above now are Multi-threaded... more here: 

 

 

http://fmepedia.safe.com/articles/Samples_and_Demos/Parallel-Processing

 

 

Best of luck
Hi all,

 

 

thanks for your answers. It's understood it's important to gain performance elsewere but this is indeed just about the best hardware setup for temporary FME files. Looks like SSD is not the best choice in this case. What about mirroring the disks in RAID 1 mode for instance? Any experiences. It shouldn't slow down writing but should speed up reading.

 

 

Thanks,

 

Jiri
Badge +10
I typically read from a RAID1 setup over a local network and write to a local disk. It's pretty rapid but I don't think you can fully max out the performance benefits of RAID1 without independent disk controllers, one for each disk.

Reply