I’m using a ChangeDetector to compare a field called ‘notes’ between a Viewer layer and a Collector layer. They basically have the same features, the Viewer layer is more of a catalog with some slight schema changes. The goal is to update the Viewer layer with any changes that are made to the Collector layer. I built the whole workspace in FME Form and tested it repeatedly with many features and everything worked as planned. I identified changes in the Collector layer and wrote them to the Viewer layer. Great.
I go to publish the workspace to FME Flow and run it, and it fails the first time with the message:
“ArcGIS Online Feature Service Writer: 'updateResults' error for a feature in 'my_layer'. The error code from the server was '1000' and the message was: 'Operation failed. The index entry of length 2256 bytes for the index 'notes_Index' exceeds the maximum length of 1700 bytes for nonclustered indexes.'”
I finally discovered that a subset of features (about 3,691) from the Collector layer are identified as updates but FME can’t write the “new” values to the Viewer layer because the character limit is exceeded. These values were NOT identified as updates when I ran it just before on FME Form (on a different virtual machine).
I calculated string lengths and discovered that the “notes” values in these Collector features are 2x as long as their counterpart records in the Viewer layer, but they only actually look different visually if I write them out to a CSV and open it in Notepad; then I can see that every note looks like this:
Instead of “test value”, it shows “t e s t v a l u e”. Spaces in between each letter. Consulting AI (Claude, for those keeping score) told me that this is due to values being incorrectly encoded. Esri uses UTF-16 for some mobile geodatabases, but the values should use UTF-8. Using a Logger, I did confirm that a non-problematic feature was marked as UTF-8 for the “notes” field, and a problematic one was flagged as UTF-16LE.
I tried using the Attribute Encoder in multiple ways to fix the problematic records.
- To “define” the encoding as UTF-16LE (“Destination: UTF-16LE, Use Bytes”) - FAILED
- To actually encode as UTF-16LE (“Destination: UTF-16LE, Honor Encoding”) - FAILED
- To “define” as UTF-16LE and then change encoding to UTF-8 (2 Attribute encoders, combining steps 1 and 2 but switched Destination to UTF-8) - FAILED
I’m thinking of encoding similarly to map projections; if a layer has an unknown projection, it must first be defined before it can be re-projected. With encoding, is it right that I need to define the encoding before I can change it to a different type?
Can anyone tell me first, WHY this is happening for a seemingly random subset of features, and why it’s only getting flagged when I run this on FME Flow? And how to implement a solution that will not only deal with these problem features but be able to handle any new ones that come in down the road would be a real cherry on top.