Skip to main content
Question

Batch Processing in FME Desktop working sporadically with AGOL

  • October 8, 2019
  • 10 replies
  • 66 views
veenendaalo
Participant
Forum|alt.badge.img+1

I have a number of workspaces pulled together into a .bat file that runs on a schedule every night. Recently, a few of the workspaces fail to run but if I open the workspace in Desktop and run it is fine. The workpaces are pulling data from AGOL and often get an error "ArcGIS Online Feature Service Reader: Encountered an unexpected error. The error code from the server was '503' and the message was: 'An error occurred.'. Details: ''" and nothing further. The issue may well be with AGOL but it feels a little inconsistent when the individually ran workspaces seem to be fine.

Has anyone else had this issue and if so, what suitable steps did you take to resolve?

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

10 replies

bruceharold
Supporter
Forum|alt.badge.img+19
  • bruceharoldSupporter
  • Supporter
  • October 8, 2019

Hi, you're hitting an issue coming from AGOL which is handled in FME 2019.1 with retry logic. If you can't move to 2019.1 then implement your own retry by refactoring your workspaces to use a FeatureReader in a custom transformer with looping from the Rejected port.

palhermans
Contributor
Forum|alt.badge.img+11
  • palhermansContributor
  • Contributor
  • October 8, 2019

Last week there was an upgrade of AGOL and we have seen quite a few "random errors" the last couple of days (for example randomly not possible to create new accounts, users can not access their accounts etc). I am tempted to suggest AGOL might be the problem.

pH
veenendaalo
Participant
Forum|alt.badge.img+1
  • veenendaaloParticipant
  • Author
  • Participant
  • October 8, 2019

Hi, you're hitting an issue coming from AGOL which is handled in FME 2019.1 with retry logic. If you can't move to 2019.1 then implement your own retry by refactoring your workspaces to use a FeatureReader in a custom transformer with looping from the Rejected port.

Thanks - I'll try 2019.1 and see if that helps FME resolves it for me.

veenendaalo
Participant
Forum|alt.badge.img+1
  • veenendaaloParticipant
  • Author
  • Participant
  • October 8, 2019

There is a bug open on ESRI BUG-000123780 Making 'query' and 'applyEdits' requests to the ArcGIS Online hosted feature service intermittently returns 503 errors.

M
Forum|alt.badge.img+2
  • markatsafe
  • October 15, 2019

Thanks - I'll try 2019.1 and see if that helps FME resolves it for me.

@veenendaalo curious to know if you were able to test with FME 2019.1 or higher and the timeout fixes that @bruceharold mentioned above. FME 2019.1 (build 19610) and higher will now retry three times if HTTP 500/502/503/504 responses are encountered.

runneals
Supporter
Forum|alt.badge.img+29
  • runnealsSupporter
  • Supporter
  • October 15, 2019

Hi, you're hitting an issue coming from AGOL which is handled in FME 2019.1 with retry logic. If you can't move to 2019.1 then implement your own retry by refactoring your workspaces to use a FeatureReader in a custom transformer with looping from the Rejected port.

@bruceharold @markatsafe The errors seem to occur less than they did (which at the height of it for us was way back in February/March). I'm pretty disappointed with esri when I get notified that it still is occurring even after SAFE has implemented their 3 retries and having multiple tickets open since February with esri. Their motto/documentation seem to say "just keep retrying until it works" but expecting applications to keep retrying forever isn't realistic. Just last week I had 35 jobs fail on the 8th, 40 fail on the 7th, 20 fail on the 6th with a 503 (and we've also been getting 500, 502, and 504 errors as well that esri can even reproduce. I could understand a couple a day, but 35+ is ridiculous considering 1 year ago we never really ran into any of these issues. Now that we're back into winter mode and running a lot more real-time FME jobs than we do during summer, it will be interesting to see how it performs over the next few months.

bruceharold
Supporter
Forum|alt.badge.img+19
  • bruceharoldSupporter
  • Supporter
  • October 15, 2019

@bruceharold @markatsafe The errors seem to occur less than they did (which at the height of it for us was way back in February/March). I'm pretty disappointed with esri when I get notified that it still is occurring even after SAFE has implemented their 3 retries and having multiple tickets open since February with esri. Their motto/documentation seem to say "just keep retrying until it works" but expecting applications to keep retrying forever isn't realistic. Just last week I had 35 jobs fail on the 8th, 40 fail on the 7th, 20 fail on the 6th with a 503 (and we've also been getting 500, 502, and 504 errors as well that esri can even reproduce. I could understand a couple a day, but 35+ is ridiculous considering 1 year ago we never really ran into any of these issues. Now that we're back into winter mode and running a lot more real-time FME jobs than we do during summer, it will be interesting to see how it performs over the next few months.

I hear you David, and the issue is getting attention, but in the meantime 2019.1 should solve job failures.

veenendaalo
Participant
Forum|alt.badge.img+1
  • veenendaaloParticipant
  • Author
  • Participant
  • October 16, 2019

Hi, you're hitting an issue coming from AGOL which is handled in FME 2019.1 with retry logic. If you can't move to 2019.1 then implement your own retry by refactoring your workspaces to use a FeatureReader in a custom transformer with looping from the Rejected port.

I have to say that FME 2019.1 has improved the situation. Before I was getting maybe 7-10 out of 50 workspaces failing in my batch run and now its maybe 1 or 2. Still not solved the issue entirely. I did speak with ESRI regarding this and as @runneals mentions they state that a 'retry on failure logic' should be applied to get round this.

@bruceharold - thanks for the suggestions. Do you have an example of your 'use a FeatureReader in a custom transformer with looping from the Rejected port'.?

Is there also a way to increase the failure logic in 2019.1 from 3 to say 5?

bruceharold
Supporter
Forum|alt.badge.img+19
  • bruceharoldSupporter
  • Supporter
  • October 16, 2019

There is a bug open on ESRI BUG-000123780 Making 'query' and 'applyEdits' requests to the ArcGIS Online hosted feature service intermittently returns 503 errors.

See the attached for some retrying readers. The 5 second wait may be a bit extreme.

refreshpropertydata2.fmw

veenendaalo
Participant
Forum|alt.badge.img+1
  • veenendaaloParticipant
  • Author
  • Participant
  • October 17, 2019

See the attached for some retrying readers. The 5 second wait may be a bit extreme.

refreshpropertydata2.fmw

Thank you @bruceharold.

Community Stats

32,207
Posts
122,305
Replies
39,891
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.