Hi, you're hitting an issue coming from AGOL which is handled in FME 2019.1 with retry logic. If you can't move to 2019.1 then implement your own retry by refactoring your workspaces to use a FeatureReader in a custom transformer with looping from the Rejected port.
Last week there was an upgrade of AGOL and we have seen quite a few "random errors" the last couple of days (for example randomly not possible to create new accounts, users can not access their accounts etc). I am tempted to suggest AGOL might be the problem.
Hi, you're hitting an issue coming from AGOL which is handled in FME 2019.1 with retry logic. If you can't move to 2019.1 then implement your own retry by refactoring your workspaces to use a FeatureReader in a custom transformer with looping from the Rejected port.
Thanks - I'll try 2019.1 and see if that helps FME resolves it for me.
Thanks - I'll try 2019.1 and see if that helps FME resolves it for me.
@veenendaalo curious to know if you were able to test with FME 2019.1 or higher and the timeout fixes that @bruceharold mentioned above. FME 2019.1 (build 19610) and higher will now retry three times if HTTP 500/502/503/504 responses are encountered.
Hi, you're hitting an issue coming from AGOL which is handled in FME 2019.1 with retry logic. If you can't move to 2019.1 then implement your own retry by refactoring your workspaces to use a FeatureReader in a custom transformer with looping from the Rejected port.
@bruceharold @markatsafe The errors seem to occur less than they did (which at the height of it for us was way back in February/March). I'm pretty disappointed with esri when I get notified that it still is occurring even after SAFE has implemented their 3 retries and having multiple tickets open since February with esri. Their motto/documentation seem to say "just keep retrying until it works" but expecting applications to keep retrying forever isn't realistic. Just last week I had 35 jobs fail on the 8th, 40 fail on the 7th, 20 fail on the 6th with a 503 (and we've also been getting 500, 502, and 504 errors as well that esri can even reproduce. I could understand a couple a day, but 35+ is ridiculous considering 1 year ago we never really ran into any of these issues. Now that we're back into winter mode and running a lot more real-time FME jobs than we do during summer, it will be interesting to see how it performs over the next few months.
@bruceharold @markatsafe The errors seem to occur less than they did (which at the height of it for us was way back in February/March). I'm pretty disappointed with esri when I get notified that it still is occurring even after SAFE has implemented their 3 retries and having multiple tickets open since February with esri. Their motto/documentation seem to say "just keep retrying until it works" but expecting applications to keep retrying forever isn't realistic. Just last week I had 35 jobs fail on the 8th, 40 fail on the 7th, 20 fail on the 6th with a 503 (and we've also been getting 500, 502, and 504 errors as well that esri can even reproduce. I could understand a couple a day, but 35+ is ridiculous considering 1 year ago we never really ran into any of these issues. Now that we're back into winter mode and running a lot more real-time FME jobs than we do during summer, it will be interesting to see how it performs over the next few months.
I hear you David, and the issue is getting attention, but in the meantime 2019.1 should solve job failures.
Hi, you're hitting an issue coming from AGOL which is handled in FME 2019.1 with retry logic. If you can't move to 2019.1 then implement your own retry by refactoring your workspaces to use a FeatureReader in a custom transformer with looping from the Rejected port.
I have to say that FME 2019.1 has improved the situation. Before I was getting maybe 7-10 out of 50 workspaces failing in my batch run and now its maybe 1 or 2. Still not solved the issue entirely. I did speak with ESRI regarding this and as @runneals mentions they state that a 'retry on failure logic' should be applied to get round this.
@bruceharold - thanks for the suggestions. Do you have an example of your 'use a FeatureReader in a custom transformer with looping from the Rejected port'.?
Is there also a way to increase the failure logic in 2019.1 from 3 to say 5?
See the attached for some retrying readers. The 5 second wait may be a bit extreme.
refreshpropertydata2.fmw
