Skip to main content
Solved

Is it possible to limit AGOL reader to subset?

B
Forum|alt.badge.img

I've just started using FME. We have a few feature-services in ArcGIS Online with a lot of features (+/- 150.00). We are creating replica's every night on our server by using a Python script from esri. In essence the script works fine. But the replica's fail at least half the time because of time-outs, or sudden server error's on esri's side.

I want to know if we can replace some of our workflows by using FME. I've created a script that reads the AGOL feature service and checks for changes/new features. This gets written to our Postgres database. The script works fine, but again, I get the same errors.

I've changed the batch size to 10 features, but no luck.

There's an option in the reader to start reading from a certain feature. I thought this might be a solution, but it still starts from 0 (maybe it's not reading / downloading it, but it still runs every 10 features. The corresponding log is: "ArcGIS Online Feature Service Reader: Getting feature results..."

So, my question is: can I limit the requests being send to AGOL by limiting the reader somehow (based on date, objectid, ...) so that it doesn't have to request every feature.

Best answer by bram

Hi @bruceharold,

 

Thanks for your reply.

Your examples are about syncing to ArcGIS Online and about spatial filtering. Unfortunately spatial filtering is not an option since it's a national database. And it needs to go from AGOL to Postgis (or another format). In the AGOL reader there's an option for a search envelope, but nothing else... Maybe I'm missing something, I've just started using FME. My model is using a filter based on date, but it still needs to read the entire AGOL feature service before it can apply the filter. If you have any ideas how to filter before the reading, that would be great!

 

I'll look in to your comment about the premium data store!

View original
Did this help you find an answer to your question?

6 replies

bruceharold
Contributor
Forum|alt.badge.img+17
  • bruceharoldContributor
  • Contributor
  • May 24, 2019

Hi Bram

Here are a couple of posts around incremental synchronization of data to ArcGIS Online and applying filters (in this case spatial, but could be a SQL expression). If you can detect the daily delta with timestamps (for example) writing the changes should be very efficient. Do engage with Esri also on timeouts, depending on your scale you may want to look at a premium data store.

 

https://community.esri.com/community/open-platform-standards-and-interoperability/blog/2019/04/08/synchronize-non-esri-tech-goes-here-to-arcgis-online-or-portal

 

https://community.esri.com/community/open-platform-standards-and-interoperability/blog/2019/05/14/make-feature-service-etl-interact-with-your-map

 

B
Forum|alt.badge.img
  • bram
  • Author
  • Best Answer
  • May 27, 2019

Hi @bruceharold,

 

Thanks for your reply.

Your examples are about syncing to ArcGIS Online and about spatial filtering. Unfortunately spatial filtering is not an option since it's a national database. And it needs to go from AGOL to Postgis (or another format). In the AGOL reader there's an option for a search envelope, but nothing else... Maybe I'm missing something, I've just started using FME. My model is using a filter based on date, but it still needs to read the entire AGOL feature service before it can apply the filter. If you have any ideas how to filter before the reading, that would be great!

 

I'll look in to your comment about the premium data store!

friesewoudloper
Forum|alt.badge.img

You could use an HttpCaller and create your own API request using the query parameters ResultOffset and RecordCount. for example:

https://geoservices.zuid-holland.nl/arcgis/rest/services/Ruimte/Zonnewijzer/FeatureServer/0/query?where=1=1&outFields=*&returnGeometry=false&orderByFields=OBJECTID&resultOffset=1000&recordCount=1000&f=pjson

bruceharold
Contributor
Forum|alt.badge.img+17
  • bruceharoldContributor
  • Contributor
  • May 28, 2019
bram wrote:

Hi @bruceharold,

 

Thanks for your reply.

Your examples are about syncing to ArcGIS Online and about spatial filtering. Unfortunately spatial filtering is not an option since it's a national database. And it needs to go from AGOL to Postgis (or another format). In the AGOL reader there's an option for a search envelope, but nothing else... Maybe I'm missing something, I've just started using FME. My model is using a filter based on date, but it still needs to read the entire AGOL feature service before it can apply the filter. If you have any ideas how to filter before the reading, that would be great!

 

I'll look in to your comment about the premium data store!

Hi Bram, if you look at the feature type parameters you'll see the options for spatial and/or logical filtering on reading.

 

arepsherbentear
1 person upvoted this
  • arepsherbentear
    arepsherbentear
B
Forum|alt.badge.img
  • bram
  • Author
  • June 3, 2019
bram wrote:

Hi @bruceharold,

 

Thanks for your reply.

Your examples are about syncing to ArcGIS Online and about spatial filtering. Unfortunately spatial filtering is not an option since it's a national database. And it needs to go from AGOL to Postgis (or another format). In the AGOL reader there's an option for a search envelope, but nothing else... Maybe I'm missing something, I've just started using FME. My model is using a filter based on date, but it still needs to read the entire AGOL feature service before it can apply the filter. If you have any ideas how to filter before the reading, that would be great!

 

I'll look in to your comment about the premium data store!

Hi Bruce,

Thanks, that's it! I don't know how I missed that...

B
Forum|alt.badge.img
  • bram
  • Author
  • June 3, 2019
friesewoudloper wrote:

You could use an HttpCaller and create your own API request using the query parameters ResultOffset and RecordCount. for example:

https://geoservices.zuid-holland.nl/arcgis/rest/services/Ruimte/Zonnewijzer/FeatureServer/0/query?where=1=1&outFields=*&returnGeometry=false&orderByFields=OBJECTID&resultOffset=1000&recordCount=1000&f=pjson

Thanks for this... Sounds interesting! For the moment I've fixed it with the where clause (that I've completely overlooked..).

Reply


Helpful Members This Week

Community Stats

31,341
Posts
118,871
Replies
38,944
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.