For my workbench I have one input dataset from an oracle spatial reader. The dataset has over 2 million features. At the moment workbench takes over 12 hours to run as the features get stuck at a featurejoiner transformer near the end of the workbench.
Solved
How can I set up a workspace runner to run 100,000 records at a time and finish when it reaches the end of the dataset(oracle spatial table)?
+1- Contributor
- 5 replies
Best answer by markatsafe
@buckrogers I think you can probably leverage WorkspaceRunner and a BETWEEN query to your database. In the 'parent' workspace calculate how many records you have (COUNT(*)) and then divide that by your batch size. You should be able to calculate the value1 & value2 for the BETWEEN. Then pass those to WorkspaceRunner which will call the child workspace - that actually does the work. Something along those lines...
I've included a couple of example workspaces and sample data that you can use as a possible starting point.
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
@buckrogers I think you can probably leverage WorkspaceRunner and a BETWEEN query to your database. In the 'parent' workspace calculate how many records you have (COUNT(*)) and then divide that by your batch size. You should be able to calculate the value1 & value2 for the BETWEEN. Then pass those to WorkspaceRunner which will call the child workspace - that actually does the work. Something along those lines...
I've included a couple of example workspaces and sample data that you can use as a possible starting point.
+1- Contributor
- 5 replies
@Mark Stoakes that's working perfectly and has quartered the run time. Thank you so much