I wrote a (Dutch) blog post to explain how you can download data using the ArcGIS REST API, in particular how to do multiple requests when the number of features you want exceeds the server limit.
@arnovananrooij pointed out to me that the Esri ArcGIS Server Feature Service reader does the job very efficiently. As I wrote the post also for readers that do not use FME Desktop, I wondered which requests FME does 'under the hood'.
As it turns out, FME first does one POST request to get all objectIds (the server limit is not a problem for this request):
Endpoint:
https://www.gdngeoservices.nl/arcgis/rest/services/blk/lks_blk_rd/MapServer/1/query?f=json
Body:
where=1=1&outSR=28992&returnIdsOnly=TrueNext, FME does multiple POST requests to get all the data. For every new request the objectIds in the `where` query parameter are updated in order to get the next batch of 1.000 features:
Endpoint:
https://www.gdngeoservices.nl/arcgis/rest/services/blk/lks_blk_rd/MapServer/1/query?f=json
Body:
objectIds=18005606,18005621,18005632,18005638,18005657,18005664,18005832,18005838,...,17911868,17911893,17911904&outFields=ADMINISTRATOR_NM,GEOMETRY.LEN,GEOMETRY.AREA,NEN3610_ID_SOIL_LOCATION,STATUS_OORD,STATUSVER,WBB_DOSSIER_DBK,NEN3610_ID_SOIL_MANAGEMENT,SIKB_ID,VERVOLG_WBB,LOCATIECODE_BEVOEGD_GEZAG,TYPE_CD,BIS_LOCCODEThe value of the `objectIds` query parameter actually contains 1.000 (!) objectIds for every request (but the last one).
I am curious to know: Why isn't FME doing multiple GET requests like this one?
https://www.gdngeoservices.nl/arcgis/rest/services/blk/lks_blk_rd/MapServer/1/query?f=json&where=WBB_DOSSIER_DBK>=18005606&outFields=*Just to be clear: I'm loving the fact that FME does this job so smoothly and 'll keep on using the ArcGIS Server feature service from now on!
But I'm trying to understand how the ArcGIS REST API works internally and what the best way is to query the API when you don't use FME. I was thinking it may have something to do with performance, but I would like to know for sure.
