Can someone please suggest something to improve the performance?
Solved
I have two input files, one of MapInfo Tab format and another one is CSV. Both files have around 180 millions row each. I'm using FeatureMerger to combine both files but it's taking more than 20 days to process it.
Best answer by david_r
I would try using the InlineQuerier and perhaps also the FeatureJoiner for this. Also make sure that you're using a recent version of FME.
Finally, make sure to switch off both breakpoints and feature caching in FME Workbench, this can have a huge influence on performance when working with such volumes.
It's possible that the best performance will be had by first reading everything into a proper database with lots of RAM, properly indexing all the relevant columns and doing the join in SQL, e.g. using an SQLExecutor or SQLCreator.
- Best Answer
I would try using the InlineQuerier and perhaps also the FeatureJoiner for this. Also make sure that you're using a recent version of FME.
Finally, make sure to switch off both breakpoints and feature caching in FME Workbench, this can have a huge influence on performance when working with such volumes.
It's possible that the best performance will be had by first reading everything into a proper database with lots of RAM, properly indexing all the relevant columns and doing the join in SQL, e.g. using an SQLExecutor or SQLCreator.
Reply
Helpful Members This Week
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.