+2Good day,
I am reading several datasets of different files which I transform. After the transformation I want to insert the data into two different arcgis db. I have approx 200 000 records to insert. I would like to insert in batches of 10 000 then pause for 5-10min and then insert the next 10 000 records. I haven’t found yet an approach that is working. What could be working ?
Best answer by j.botterill
Appears you want to split total feature count into batches prior to writing
ModuleCounter is nice to break 200k into batches, so set the parameter to 20
Best approach might be to look how you can use the advanced parameter options, commit frequency … or explore the transaction type = transaction and find a way to use the count attribute you created from modulo
alternatively either
1. apply and AttributeRangeFilter on the output of moduloCounter and connect each output into a featurewriter as a new input. The featurewriter will have many inputs for the same Esri format writer. Add a feature holder or decelerator
2. you could create a looping custom transformer and include a Decelerator and have a featurewriter transformer
3. python caller approach where you use Group By of the count batch (module)
+60Appears you want to split total feature count into batches prior to writing
ModuleCounter is nice to break 200k into batches, so set the parameter to 20
Best approach might be to look how you can use the advanced parameter options, commit frequency … or explore the transaction type = transaction and find a way to use the count attribute you created from modulo
alternatively either
1. apply and AttributeRangeFilter on the output of moduloCounter and connect each output into a featurewriter as a new input. The featurewriter will have many inputs for the same Esri format writer. Add a feature holder or decelerator
2. you could create a looping custom transformer and include a Decelerator and have a featurewriter transformer
3. python caller approach where you use Group By of the count batch (module)
No account yet? Create an account
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
