Skip to main content
  • EN

I've just started using FME. We have a few feature-services in ArcGIS Online with a lot of features (+/- 150.00). We are creating replica's every night on our server by using a Python script from esri. In essence the script works fine. But the replica's fail at least half the time because of time-outs, or sudden server error's on esri's side.

I want to know if we can replace some of our workflows by using FME. I've created a script that reads the AGOL feature service and checks for changes/new features. This gets written to our Postgres database. The script works fine, but again, I get the same errors.

I've changed the batch size to 10 features, but no luck.

There's an option in the reader to start reading from a certain feature. I thought this might be a solution, but it still starts from 0 (maybe it's not reading / downloading it, but it still runs every 10 features. The corresponding log is: "ArcGIS Online Feature Service Reader: Getting feature results..."

So, my question is: can I limit the requests being send to AGOL by limiting the reader somehow (based on date, objectid, ...) so that it doesn't have to request every feature.

Hi Bram

Here are a couple of posts around incremental synchronization of data to ArcGIS Online and applying filters (in this case spatial, but could be a SQL expression). If you can detect the daily delta with timestamps (for example) writing the changes should be very efficient. Do engage with Esri also on timeouts, depending on your scale you may want to look at a premium data store.

 

https://community.esri.com/community/open-platform-standards-and-interoperability/blog/2019/04/08/synchronize-non-esri-tech-goes-here-to-arcgis-online-or-portal

 

https://community.esri.com/community/open-platform-standards-and-interoperability/blog/2019/05/14/make-feature-service-etl-interact-with-your-map

 

Hi @bruceharold,

 

Thanks for your reply.

Your examples are about syncing to ArcGIS Online and about spatial filtering. Unfortunately spatial filtering is not an option since it's a national database. And it needs to go from AGOL to Postgis (or another format). In the AGOL reader there's an option for a search envelope, but nothing else... Maybe I'm missing something, I've just started using FME. My model is using a filter based on date, but it still needs to read the entire AGOL feature service before it can apply the filter. If you have any ideas how to filter before the reading, that would be great!

 

I'll look in to your comment about the premium data store!

You could use an HttpCaller and create your own API request using the query parameters ResultOffset and RecordCount. for example:

https://geoservices.zuid-holland.nl/arcgis/rest/services/Ruimte/Zonnewijzer/FeatureServer/0/query?where=1=1&outFields=*&returnGeometry=false&orderByFields=OBJECTID&resultOffset=1000&recordCount=1000&f=pjson

Hi @bruceharold,

 

Thanks for your reply.

Your examples are about syncing to ArcGIS Online and about spatial filtering. Unfortunately spatial filtering is not an option since it's a national database. And it needs to go from AGOL to Postgis (or another format). In the AGOL reader there's an option for a search envelope, but nothing else... Maybe I'm missing something, I've just started using FME. My model is using a filter based on date, but it still needs to read the entire AGOL feature service before it can apply the filter. If you have any ideas how to filter before the reading, that would be great!

 

I'll look in to your comment about the premium data store!

Hi Bram, if you look at the feature type parameters you'll see the options for spatial and/or logical filtering on reading.

 

Hi @bruceharold,

 

Thanks for your reply.

Your examples are about syncing to ArcGIS Online and about spatial filtering. Unfortunately spatial filtering is not an option since it's a national database. And it needs to go from AGOL to Postgis (or another format). In the AGOL reader there's an option for a search envelope, but nothing else... Maybe I'm missing something, I've just started using FME. My model is using a filter based on date, but it still needs to read the entire AGOL feature service before it can apply the filter. If you have any ideas how to filter before the reading, that would be great!

 

I'll look in to your comment about the premium data store!

Hi Bruce,

Thanks, that's it! I don't know how I missed that...

You could use an HttpCaller and create your own API request using the query parameters ResultOffset and RecordCount. for example:

https://geoservices.zuid-holland.nl/arcgis/rest/services/Ruimte/Zonnewijzer/FeatureServer/0/query?where=1=1&outFields=*&returnGeometry=false&orderByFields=OBJECTID&resultOffset=1000&recordCount=1000&f=pjson

Thanks for this... Sounds interesting! For the moment I've fixed it with the where clause (that I've completely overlooked..).

Reply


Community Stats

29,729
Posts
112,949
Replies
37,370
Members
Terms & ConditionsCookie settings