Skip to main content
Open

Add Parallel Processing Parameters to FeatureReader and FeatureWriter

Related products:Transformers
  • June 30, 2017
  • 4 replies
  • 175 views
davideagle
Contributor
Forum|alt.badge.img+22

The FeatureReader and FeatureWriter are ideal candidates to allow concurrent data reading and writing, something that you can't do with multiple readers and writers right now. Adding these parameters is currently possible if you wrap them up in custom transformers like I explain in this webinar, but ideally you shouldn't have to go to this effort.

https://www.youtube.com/embed/YnCot12FA9U

Looking forward to the performance gains this change would bring. In tests my timings suggest in some examples it reduces the readwrite times to a third or even less.

Cheers, Dave

4 replies

erik_jan
Contributor
Forum|alt.badge.img+26
  • erik_janContributor
  • Contributor
  • June 30, 2017

Anything that can speed up reading and writing will get my vote!

Not sure though how this can apply to file based datasets like DWG, DGN and GML.

P
Forum|alt.badge.img
  • peterx
  • May 24, 2019

I know FME 2019 has pushed parallel processing down into Custom Transformers, but when I'm loading-up two 50K+ datasets to compare, I'd much like to load them in parallel, rather than sequentially.

bruceharold
Influencer
Forum|alt.badge.img+19
  • bruceharoldInfluencer
  • Influencer
  • May 24, 2019

I guess concurrent reading of separate files would be an advance. Ideally each stream in the workspace should run concurrently, up to some process maximum.

davidrasner5
Contributor
Forum|alt.badge.img+4
  • davidrasner5Contributor
  • Contributor
  • June 6, 2023

I had to do a spatial filter on a very large data set and was a bit surprised that the feature reader doesn't do it asynchronously. Gave up after a while and had to solve it in Python. 😞

Community Stats

32,749
Posts
124,193
Replies
41,251
Members
Terms & ConditionsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.