I’m using an HTTPCaller to applyEdits using ESRI’s REST API. It includes sending along some JSON to add to a table in a feature service. The workspace runs successfully testing with 1 or 2 records in the JSON parameter, which is all sent with a single HTTP call, but fails when testing with 10 records. It looks like FME is truncating the HTTP string, which causes the request to fail. That’s what it looks like in the FME log anyway; the query string appears on two lines (see attached screenshot). If I try applyEdits in a web browser at the REST endpoint and paste the JSON in manually, I can add the data without issue. Is there a parameter or formatting change I need to make to keep FME from truncating the HTTP call? (FME Desktop/Form 2023.2.2.0, build 23764)
I can think of the following advice
- modulo counter to create some grouping and then a test filter to duplicated http callers, effectively distributing your http requests?
- under the httpcaller advanced > turn down the concurrent request from 25 to say 10
- embed the HTTPcaller into a custom transformer with a looping arrangement introducing https://support.safe.com/hc/en-us/articles/25407586812429-Getting-Started-with-API-Pagination make sure the httpcaller only has 1 set for the concurrent requests
The applyEdits endpoint only supports POST operations. You appear to be using GET
https://developers.arcgis.com/rest/services-reference/enterprise/apply-edits-feature-service/
GET requests (which use the query string) make a URL longer. Servers have a limit of how long this incoming url can be. It’s very possible that this url is exceeding that.
You would need to use a POST request to apply your edits. This sends a separate body along with the request which can be much larger.
https://www.w3schools.com/tags/ref_httpmethods.asp
Additionally, have you considered using Writers instead of the HTTPCaller?
I am using POST in my HTTP request (see attached screenshot). Is there something indicating the HTTPCaller is using GET instead?
With the exception of a Portal Feature Service Writer inside a FeatureWriter transformer, we are using HTTPCallers to perform a variety of tasks - purge locks, apply edits, start reading, start editing, stop editing, stop reading, reconciling (and posting), validating UN network topology, and updating the UN network. It all works nicely with one or two records so we were planning to go that route if possible. As noted above, I can paste the JSON in a web browser using ESRI’s REST endpoint but it fails with FME HTTPCaller if there are more than a few records in the JSON payload. I figure it should work in both. Hoping I’m missing something small.
Where are you placing the JSON in the HTTPCaller? Under query string parameters or body?
I place the JSON under query string parameters. That’s how I pass in the parameters for all of the HTTP calls using ESRI’s REST API. (Apologies for the delayed response. Been on vacation.)
You’ve run into a limitation: most systems don’t allow URLs longer than 2048 characters, which means your JSON may be getting cut off.
Have a look at the ESRI documentation on how to send the JSON in the request body rather than in the query string.
I place the JSON under query string parameters. That’s how I pass in the parameters for all of the HTTP calls using ESRI’s REST API. (Apologies for the delayed response. Been on vacation.)
You’ve run into a limitation: most systems don’t allow URLs longer than 2048 characters, which means your JSON may be getting cut off.
Have a look at the ESRI documentation on how to send the JSON in the request body rather than in the query string.
Yea, that will be the problem. Place it in the body.
GET is designed to retrieve information. Parameters (like search terms, bounding boxes, or output formats) are passed in the query string of the URL. GET requests should not have a body.
POST is designed to send data to a service for processing. Because the data can be large or complex, it’s usually passed in the body of the request. Unlike GET, POST can include both a body and query string parameters if needed.
The Esri ArcGIS Connector package supplies the Esri ArcGIS Feature Service format which in addition to the write mode applyEdits supplies goodness like retry logic. No need for HTTPCaller.