Skip to main content
Question

Write New Geodatabase every x features


jacklonsdale
Contributor
Forum|alt.badge.img+4

Hi all,

I have a set of 8000+ GML files that i wish to batch process into a geodatabase, i have found that putting them a workspace runner is effective but slows down massively past 50 GML files. Therefore, i want a way to write a new geodatabase file every 50 features, to reduce the 8000 GML files to 160 geodatabases to then combine afterwards.

Has anyone any ideas? Apart from a huge conditional value in attribute manager!

Cheers

Jack

12 replies

david_r
Evangelist
  • August 20, 2019

Lots of possible solutions, but the easiest is perhaps to use the Grouper transformer from the FME Hub and use the _group_index as part of the dataset fanout.


virtualcitymatt
Celebrity
Forum|alt.badge.img+34

I would suggest you take a look at the ModuloCounter combined with a path reader. You should be able to use the ModuloCounter to help create Groups (https://www.safe.com/blog/2016/12/parallel-processing-tips-evangelist159/ ). Combine it with the Aggregator to build an attribute containing the 50 filenames going into the WorkspaceRunner.

 

The result should be a WorkspaceRunner which runs 160 times each with 50 GML files as the source.

lars_de_vries
Forum|alt.badge.img+10
david_r wrote:

Lots of possible solutions, but the easiest is perhaps to use the Grouper transformer from the FME Hub and use the _group_index as part of the dataset fanout.

Did you know that...

The Grouper works great, but the GroupCounter works faster and does the same job?


david_r
Evangelist
  • August 20, 2019
lars_de_vries wrote:

Did you know that...

The Grouper works great, but the GroupCounter works faster and does the same job?

I did not! Thanks Lars :-)


lars_de_vries
Forum|alt.badge.img+10
david_r wrote:

I did not! Thanks Lars :-)

Now you do!

I created it once and added it to the hub. Found out half a year later that Hans made a similar custom transformer. So I had to put them to the test. If Hans's would outperform mine, I would have removed mine from the Hub, but that wasn't the case.


david_r
Evangelist
  • August 20, 2019
lars_de_vries wrote:

Now you do!

I created it once and added it to the hub. Found out half a year later that Hans made a similar custom transformer. So I had to put them to the test. If Hans's would outperform mine, I would have removed mine from the Hub, but that wasn't the case.

Haha, sounds like a challenge to Hans ;-) @redgeographics


redgeographics
Celebrity
Forum|alt.badge.img+47
david_r wrote:

Haha, sounds like a challenge to Hans ;-) @redgeographics

It's not like I'm short on challenges lately ;)


virtualcitymatt
Celebrity
Forum|alt.badge.img+34
lars_de_vries wrote:

Did you know that...

The Grouper works great, but the GroupCounter works faster and does the same job?

Oh yeah, this works a treat. Works in BulkMode too. Nice one Lars


lars_de_vries
Forum|alt.badge.img+10
virtualcitymatt wrote:

Oh yeah, this works a treat. Works in BulkMode too. Nice one Lars

You're welcome!


jacklonsdale
Contributor
Forum|alt.badge.img+4
  • Author
  • Contributor
  • August 21, 2019
virtualcitymatt wrote:

I would suggest you take a look at the ModuloCounter combined with a path reader. You should be able to use the ModuloCounter to help create Groups (https://www.safe.com/blog/2016/12/parallel-processing-tips-evangelist159/ ). Combine it with the Aggregator to build an attribute containing the 50 filenames going into the WorkspaceRunner.

 

The result should be a WorkspaceRunner which runs 160 times each with 50 GML files as the source.

This seems like a good solution, but does the WorkspaceRunner need quotation marks and comma separators between filenames? I've tried with and without and I just get warnings and it doesn't run.


jacklonsdale
Contributor
Forum|alt.badge.img+4
  • Author
  • Contributor
  • August 21, 2019
david_r wrote:

Lots of possible solutions, but the easiest is perhaps to use the Grouper transformer from the FME Hub and use the _group_index as part of the dataset fanout.

Hi David, works perfectly, thanks for the solution!


gio
Contributor
Forum|alt.badge.img+15
  • Contributor
  • August 21, 2019

it's as simple as

@ceil(@div(@Value(some_value),@Value(some_modulus)),0))

or floor or combine div, fmod

All can create groups based on some modulus.

And require only 1 line in a tester or attrobutecreator or..

 

 

 


Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings