Skip to main content
Released

Optimize the ArcGIS Online readers and writers

Related products:FME Form
  • August 10, 2016
  • 11 replies
  • 87 views
ngoorman
Contributor
Forum|alt.badge.img+3

The ArcGIS Online reader and writer are quite slow to use. For every feature type, in every AGO reader and writer, FME currently loops through all accessible groups twice (once to find the feature service, once again to find the layer). When a user only has access to a few groups within his organization, it's merely an annoyance. When a user has access to a large number of groups, waiting for FME to find the correct layer can add minutes to the time it takes to run a workspace.

There is a workaround where the HTTPCaller can be used to directly access a layer, much faster than the AGO readers/writers can, but this method has several other downsides. It would be great if the official AGO readers/writers could be improved to be a lot faster.

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

11 replies

fmelizard
Safer
Forum|alt.badge.img+22
  • fmelizardSafer
  • Safer
  • August 10, 2016

Thanks for pointing this out -- I've passed it on to the team. Does seem like there are opportunities here...

runneals
Enthusiast
Forum|alt.badge.img+29
  • runnealsEnthusiast
  • Enthusiast
  • January 12, 2017
For doing near real-time data services, it REALLY eats into the processing time (especially if you have many groups in your organization). For writing to AGOL with an account that has access to all public groups in our org, it takes 9-10 seconds. I could see it looping through when you add a feature service writer or import a dataset, but when you already have them defined, it doesn't seem like it's needed? Also, if there could be a writer like your ARCGISFEATURES reader, I feel like that would also resolve some of the problems as well since you could hit the service up directly.
runneals
Enthusiast
Forum|alt.badge.img+29
  • runnealsEnthusiast
  • Enthusiast
  • January 12, 2017

For the reader, you can use the ARCGISFEATURES reader and hit up your rest endpoint directly, which won't run through the groups.

carsonlam
Safer
Forum|alt.badge.img+7
  • carsonlamSafer
  • Safer
  • January 12, 2017
I'm working on it. :)
runneals
Enthusiast
Forum|alt.badge.img+29
  • runnealsEnthusiast
  • Enthusiast
  • January 12, 2017
Sweet! 2017 release? ;)

 

Have 2 more things for optimization of the writer that we ran into attempting to use AGOL for near real-time high availability GIS services that would be nice to have integrated into FME (shall I submit as separate suggestions & tag you?)

 

1) Functionality to enable editing (pre-write) and disable editing (post write) for a service that is high availability or one you have editing disabled on. Right now we're using an esri Python script on startup/shutdown.

 

2) A Feature Collection writer either separately or integrated into the AGOL/Portal feature service writers (I don't know how they would be packaged together, but we are relying on a shutdown python script from esri that takes the Feature Service and writes it to a Feature Collection)
carsonlam
Safer
Forum|alt.badge.img+7
  • carsonlamSafer
  • Safer
  • January 13, 2017
I'm aiming for the initial 2017 release, but no promises! I'm aware of the 2 other suggestions you've made. We're tracking them internally, but it's not currently scheduled for 2017.

 

carsonlam
Safer
Forum|alt.badge.img+7
  • carsonlamSafer
  • Safer
  • February 14, 2017

@runneals @ngoorman This has been improved in the FME 2017 betas and release candidate, starting with b17233. Please try it out!

ngoorman
Contributor
Forum|alt.badge.img+3
  • ngoormanContributor
  • Author
  • Contributor
  • February 16, 2017
Very nice work, I'll give it a try!

 

 

runneals
Enthusiast
Forum|alt.badge.img+29
  • runnealsEnthusiast
  • Enthusiast
  • February 17, 2017
From some quick playing around, it looks like the recursive issue is fixed. Sweet, thanks!
F
Forum|alt.badge.img+1

This has been released as part of FME 2017.0 . Visit http://www.safe.com/downloads to download it and give it a spin.

bruceharold
Influencer
Forum|alt.badge.img+19
  • bruceharoldInfluencer
  • Influencer
  • August 6, 2019

FeatureCollection writing (i.e JSON) would take advantage of the asynchronous update capability of feature services for large writes; write JSON to file, launch update...

Community Stats

32,750
Posts
124,202
Replies
41,252
Members
Terms & ConditionsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.