Hei,
The SQLExecutor transformer uses sets of 10000 records (as postgis reader uses as default). Is there any way to increase it?
Best answer by redgeographics
View original
Is it possible to control 'Number Of Records To Fetch At A Time' on SQLExecutor Transformer?
Userlevel 5
+25
- Influencer
- 3345 replies
-
3 January 2017
- Best Answer
There is an option to add a reader as a resource, giving you access to the specific PostGIS reader parameters, but the SQLCreator doesn't seem to pick those up. Other than that I don't really see a way of doing that.
Userlevel 2
+16
- Contributor
- 2180 replies
-
3 January 2017
Hi @tono
What are you using the SQLExecutor for?
Could you use the FeatureReader or the SQLCreator instead of the SQLExecutor?
Both allow using the WHERE Clause and can use a limit.
+7
Hi @tono
I would suggest on using a LIMIT.
Just do a count to count the amount of objects you will fetch, define a maximum amount to be fetched at a time and use a cloner + expressionevaluator to update the OFFSET value that you will use in the SQLExecutor.
For example, you will fetch 100 000 objects but want to limit to 10 000 in one go.
nbRequests = @ceil [ count / limit ]
cloner (nbRequest) - also creates attribute _copynum
offset = _copynum * limit
Good luck!
+7
@tono
Is your questioned answered?
Hi @tono
I would suggest on using a LIMIT.
Just do a count to count the amount of objects you will fetch, define a maximum amount to be fetched at a time and use a cloner + expressionevaluator to update the OFFSET value that you will use in the SQLExecutor.
For example, you will fetch 100 000 objects but want to limit to 10 000 in one go.
nbRequests = @ceil [ count / limit ]
cloner (nbRequest) - also creates attribute _copynum
offset = _copynum * limit
Good luck!
Hi @tono
What are you using the SQLExecutor for?
Could you use the FeatureReader or the SQLCreator instead of the SQLExecutor?
Both allow using the WHERE Clause and can use a limit.
+16
- Contributor
- 1425 replies
-
16 January 2017
I would also advise using the sql creator/feature reader but first why would you want to increase the number? It doesn't automatically mean the data will be read faster.
Yes, redgeographics did. the answer is 'no' :)
Isnt it? I thought that fetching more rows reading from PostGIS it would be faster. I thought that this could be a bottleneck in my workspace
+16
- Contributor
- 1425 replies
-
16 January 2017
Isnt it? I thought that fetching more rows reading from PostGIS it would be faster. I thought that this could be a bottleneck in my workspace
Userlevel 2
+16
- Contributor
- 2180 replies
-
16 January 2017
How about creating a view in the PostGis database with the outcome of the SQL query (including the generalization) and using that view in a PostGis reader (with the option to limit the number of records).
Reply
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.