Skip to main content
Solved

I have two input files, one of MapInfo Tab format and another one is CSV. Both files have around 180 millions row each. I'm using FeatureMerger to combine both files but it's taking more than 20 days to process it.

  • December 19, 2022
  • 2 replies
  • 13 views

Can someone please suggest something to  improve the performance?

Best answer by david_r

I would try using the InlineQuerier and perhaps also the FeatureJoiner for this. Also make sure that you're using a recent version of FME.

Finally, make sure to switch off both breakpoints and feature caching in FME Workbench, this can have a huge influence on performance when working with such volumes.

It's possible that the best performance will be had by first reading everything into a proper database with lots of RAM, properly indexing all the relevant columns and doing the join in SQL, e.g. using an SQLExecutor or SQLCreator.

View original
Did this help you find an answer to your question?

2 replies

david_r
Celebrity
  • Best Answer
  • December 19, 2022

I would try using the InlineQuerier and perhaps also the FeatureJoiner for this. Also make sure that you're using a recent version of FME.

Finally, make sure to switch off both breakpoints and feature caching in FME Workbench, this can have a huge influence on performance when working with such volumes.

It's possible that the best performance will be had by first reading everything into a proper database with lots of RAM, properly indexing all the relevant columns and doing the join in SQL, e.g. using an SQLExecutor or SQLCreator.


Forum|alt.badge.img+2
  • December 19, 2022

@csj5483​ SpatiaLite is a good option as a staging database - pretty much the same as InlineQuerier. But try FeatureJoiner first.


Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings