I have a number of workspaces pulled together into a .bat file that runs on a schedule every night. Recently, a few of the workspaces fail to run but if I open the workspace in Desktop and run it is fine. The workpaces are pulling data from AGOL and often get an error "ArcGIS Online Feature Service Reader: Encountered an unexpected error. The error code from the server was '503' and the message was: 'An error occurred.'. Details: ''" and nothing further. The issue may well be with AGOL but it feels a little inconsistent when the individually ran workspaces seem to be fine.Has anyone else had this issue and if so, what suitable steps did you take to resolve?

Batch Processing in FME Desktop working sporadically with AGOL

+16

bruceharold
Contributor
321 replies
4 years ago
8 October 2019

Hi, you're hitting an issue coming from AGOL which is handled in FME 2019.1 with retry logic. If you can't move to 2019.1 then implement your own retry by refactoring your workspaces to use a FeatureReader in a custom transformer with looping from the Rejected port.

Last week there was an upgrade of AGOL and we have seen quite a few "random errors" the last couple of days (for example randomly not possible to create new accounts, users can not access their accounts etc). I am tempted to suggest AGOL might be the problem.

V

Hi, you're hitting an issue coming from AGOL which is handled in FME 2019.1 with retry logic. If you can't move to 2019.1 then implement your own retry by refactoring your workspaces to use a FeatureReader in a custom transformer with looping from the Rejected port.

Thanks - I'll try 2019.1 and see if that helps FME resolves it for me.

V

There is a bug open on ESRI BUG-000123780 Making 'query' and 'applyEdits' requests to the ArcGIS Online hosted feature service intermittently returns 503 errors.

M

+2

Thanks - I'll try 2019.1 and see if that helps FME resolves it for me.

@veenendaalo curious to know if you were able to test with FME 2019.1 or higher and the timeout fixes that @bruceharold mentioned above. FME 2019.1 (build 19610) and higher will now retry three times if HTTP 500/502/503/504 responses are encountered.

+8

runneals
Contributor
308 replies
4 years ago
15 October 2019

Hi, you're hitting an issue coming from AGOL which is handled in FME 2019.1 with retry logic. If you can't move to 2019.1 then implement your own retry by refactoring your workspaces to use a FeatureReader in a custom transformer with looping from the Rejected port.

@bruceharold @markatsafe The errors seem to occur less than they did (which at the height of it for us was way back in February/March). I'm pretty disappointed with esri when I get notified that it still is occurring even after SAFE has implemented their 3 retries and having multiple tickets open since February with esri. Their motto/documentation seem to say "just keep retrying until it works" but expecting applications to keep retrying forever isn't realistic. Just last week I had 35 jobs fail on the 8th, 40 fail on the 7th, 20 fail on the 6th with a 503 (and we've also been getting 500, 502, and 504 errors as well that esri can even reproduce. I could understand a couple a day, but 35+ is ridiculous considering 1 year ago we never really ran into any of these issues. Now that we're back into winter mode and running a lot more real-time FME jobs than we do during summer, it will be interesting to see how it performs over the next few months.

+16

bruceharold
Contributor
321 replies
4 years ago
15 October 2019

@bruceharold @markatsafe The errors seem to occur less than they did (which at the height of it for us was way back in February/March). I'm pretty disappointed with esri when I get notified that it still is occurring even after SAFE has implemented their 3 retries and having multiple tickets open since February with esri. Their motto/documentation seem to say "just keep retrying until it works" but expecting applications to keep retrying forever isn't realistic. Just last week I had 35 jobs fail on the 8th, 40 fail on the 7th, 20 fail on the 6th with a 503 (and we've also been getting 500, 502, and 504 errors as well that esri can even reproduce. I could understand a couple a day, but 35+ is ridiculous considering 1 year ago we never really ran into any of these issues. Now that we're back into winter mode and running a lot more real-time FME jobs than we do during summer, it will be interesting to see how it performs over the next few months.

I hear you David, and the issue is getting attention, but in the meantime 2019.1 should solve job failures.

V

Hi, you're hitting an issue coming from AGOL which is handled in FME 2019.1 with retry logic. If you can't move to 2019.1 then implement your own retry by refactoring your workspaces to use a FeatureReader in a custom transformer with looping from the Rejected port.

I have to say that FME 2019.1 has improved the situation. Before I was getting maybe 7-10 out of 50 workspaces failing in my batch run and now its maybe 1 or 2. Still not solved the issue entirely. I did speak with ESRI regarding this and as @runneals mentions they state that a 'retry on failure logic' should be applied to get round this.

@bruceharold - thanks for the suggestions. Do you have an example of your 'use a FeatureReader in a custom transformer with looping from the Rejected port'.?

Is there also a way to increase the failure logic in 2019.1 from 3 to say 5?