Question

Writing then overwriting to ArcGIS Portal error

D

Hello,

 

I am having an issue when trying to overwrite a feature layer in ArcGIS Portal. It writes first time fine, but fails to overwrite.

A bit of background of what I am trying to. The workbench reads in a ODBC connection, performs multiple joins, creates vertices based off lat and lng, then writes to ArcGIS Portal. 

 

As mentioned, this is writing fine but I noticed an issue when trying to overwrite the file. The error message is:

 

ArcGIS Portal Feature Service Writer: Encountered an unexpected error. The error code from the server was '500' and the message was: 'JSONObject["globalId"] not found.'. Details: ''

ARCGISPORTALFEATURES writer: A fatal error has occurred. Check the logfile above for details

 

I have had a similar issue before when performing a similar task, and I created a forum post discussing the issue - Error when trying to overwrite an existing ArcGIS Portal feature (safe.com)

 

The issue doesn't actually seem to be anything to do with a Global ID (I am not actually using one). It got mentioned in here that the problem was likely to do with integers, and casting the integers using Attribute Manager was the first thing to do. I think what actually sorted that workbench in the end was using SubstringExtractor, as one of the attributes had a field length issue and using substring extractor sorted it and let the workbench overwrite.

 

However on this new workbench, which is totally different data but still publishing the end product to Portal, I don’t have any integers in my data. I have two date fields then the rest are strings. 

Based on what I learned from the above forum post, I carried out some additional steps including using AttributeRemover to remove some attributes which weren’t really required and could potentially cause issues. But I still had problems so I used AttributeRenamer to ensure all fields were lowercase and didn’t contain spaces, as this had causes issues before, but that wasn’t it.

Another user in the forum post above mentioned they had this issue also, and it was due to the string character count exceeding the size allowed in the feature layer. In a couple of fields I did have some records which had over 1k characters, so in my workbench I used a substring extractor to only allow 100 characters, but that didn’t seem to work. 

 

I am out of ideas now so any suggestions please let me know. The data is sensitive so I don't really want to share the raw data, but attached is an image of the workbench.

workbench pic 

Thanks

 

11 replies

ebygomm
Rank
Userlevel 1
Badge +21

Can you share a screenshot of the writer settings?

rahulsharma
Rank
Badge +8

Hi @dan_mbc

​ I think you are on right track but we might need to use an AttributeValidator first.

Cause

The main reasons for such error messages are a mismatch between incoming data and the feature service's existing schema.

Resolution

This issue has been resolved in FME 2021.0 and newer. The string data get trimmed automatically based on the schema. 

For 2020.x and earlier

For this example, I have 3 attributes with data types of string (length 2), integer, and a DateTime field.

Right-click on ArcGIS Online/Portal writer → Show Summary Annotation

Check that the incoming attribute matches the schema on the writer using AttributeValidator

if the attribute fails the check you should get an error similar to this below 

Attribute 'string1' with value 'yes' fails to check for Maximum Length = 2 

To fix this use AttributeManager, StringExtractor, or DateTimeFormater to fix the data type before writing.

@Evaluate(@int8(@Value(string1)))  -- Convert data to Int8
@Substring(@Value(string2),0,49)   -- Trim string attribute length to 50 characters

 

 

 

 

D

Hi @dan_mbc

​ I think you are on right track but we might need to use an AttributeValidator first.

Cause

The main reasons for such error messages are a mismatch between incoming data and the feature service's existing schema.

Resolution

This issue has been resolved in FME 2021.0 and newer. The string data get trimmed automatically based on the schema. 

For 2020.x and earlier

For this example, I have 3 attributes with data types of string (length 2), integer, and a DateTime field.

Right-click on ArcGIS Online/Portal writer → Show Summary Annotation

Check that the incoming attribute matches the schema on the writer using AttributeValidator

if the attribute fails the check you should get an error similar to this below 

Attribute 'string1' with value 'yes' fails to check for Maximum Length = 2 

To fix this use AttributeManager, StringExtractor, or DateTimeFormater to fix the data type before writing.

@Evaluate(@int8(@Value(string1)))  -- Convert data to Int8
@Substring(@Value(string2),0,49)   -- Trim string attribute length to 50 characters

 

 

 

 

Thank you. 

 

I am currently on FME 2021.0.1.0, but, I think when I originally created this workbench I was on a 2020 version, then a week or two later I downloaded the new version. So this workbench might still be treated as a 2020 workbench? 

 

I am therefore following the workaround:

I used the AttributeValidator to firstly check all the string fields, and they all passed. I then checked the date fields and these failed. So it would appear these are the issue, and looking at it I think I can see why. The format for my OpenDateTime and ClosedDateTime field is as you'd expect, a date and a time, but the FME writer to Portal is just writing it as a date field. I can't see a way to change this in the writer settings, it only seems to offer date? 

I noticed in a previous workbench which also writes to Portal, I get options in writer to select  the types of field as 'esriFieldType', including esriFieldTypeDate, but I don't seem to have them options in this new workbench? Not sure if that would have helped or not? 

 

Anyway back to the workaround, I firstly deleted the feature layer in Portal. I have used the DateTimeConverter to convert them into the correct format and then ran the workbench. This ran as usual with no problems. I noticed still in the writer the datetime fields are still only showing as a date field type in the writer. 

I then re-ran the workbench but this fails with the same error. I think I expected this as there is still a mismatch between FME and Portal with the date field? Do I need to change the DateTimeConverter to output only as a date field (YMD), although the time info is useful?

 

I have attached some screenshots showing the updated section of workbench (including the show summary annotation) and also a screenshot of the DateTimeConverter.

 

ThanksDateTimeConverterworkbench update pic 

 

 

 

 

 

D

Can you share a screenshot of the writer settings?

Hi, attached is the settings. writer_settings2writer_settings

D

Just to add to this, I decided to create a new workbench in version 2021 and repeat much of the existing workbench.

 

I managed to get it working and overwriting fine, however I realised I had removed the two fields with long character counts (details and locationspecifics). But my writer did include the datetime attributes, so the new workbench seemed to sort that issue with the dates. However once I added back in the details and locationspecifics fields, and written then tried to overwrite, it failed.

It mentioned in the warning about how the server was truncating the fields, but it still failed with the same error as above. I decided to use SubstringExtractor to make both them fields only 200 long, and that stopped the warnings but the error still remained.

 

I then used attribute manager (removed SubstringExtractor transformers) to trim the characters to 50 (as advised above) rather than using substringExtractor, using the values: @Substring(@Value(locationsspecs),0,49).

 

This had the desired result, the characters were only 50, but once written to Portal it then failed to overwrite.

 

Feel like I am getting closer, but not quite there.

D

Just to add to this, I decided to create a new workbench in version 2021 and repeat much of the existing workbench.

 

I managed to get it working and overwriting fine, however I realised I had removed the two fields with long character counts (details and locationspecifics). But my writer did include the datetime attributes, so the new workbench seemed to sort that issue with the dates. However once I added back in the details and locationspecifics fields, and written then tried to overwrite, it failed.

It mentioned in the warning about how the server was truncating the fields, but it still failed with the same error as above. I decided to use SubstringExtractor to make both them fields only 200 long, and that stopped the warnings but the error still remained.

 

I then used attribute manager (removed SubstringExtractor transformers) to trim the characters to 50 (as advised above) rather than using substringExtractor, using the values: @Substring(@Value(locationsspecs),0,49).

 

This had the desired result, the characters were only 50, but once written to Portal it then failed to overwrite.

 

Feel like I am getting closer, but not quite there.

Hi @rahulsharma​ , I don't suppose you have had chance to look at this?

 

I feel like this second workbench is closer to working but when I added in the two field with long character counts, it caused the usual error. Wondered if you had any ideas?

rahulsharma
Rank
Badge +8

Hi @rahulsharma​ , I don't suppose you have had chance to look at this?

 

I feel like this second workbench is closer to working but when I added in the two field with long character counts, it caused the usual error. Wondered if you had any ideas?

Hi @dan_mbc​ I have tried some DateTime data but it all works for me. I will escalate to a support ticket. You can provide me some sample data that we can try and possibly find a solution.

rahulsharma
Rank
Badge +8

Conducted a screen share session with @dan_mbc​ and was able to resolve the issue.

BACKGROUND:

One of the attributes (string), contains some text with angle brackets (<, >), which are being interpreted as HTML tags when the translating or writing data to existing Portal Feature services, which causes an error on the server-side, and the features get rejected.

 

RESOLUTION: Replace Angle brackets

Option1 : Use StringReplacer transformer to remove the angle brackets

Option2: Use StringPairReplacer to replace <this> to (this)

 

nielsgerrits
Rank
Userlevel 6
Badge +33

Conducted a screen share session with @dan_mbc​ and was able to resolve the issue.

BACKGROUND:

One of the attributes (string), contains some text with angle brackets (<, >), which are being interpreted as HTML tags when the translating or writing data to existing Portal Feature services, which causes an error on the server-side, and the features get rejected.

 

RESOLUTION: Replace Angle brackets

Option1 : Use StringReplacer transformer to remove the angle brackets

Option2: Use StringPairReplacer to replace <this> to (this)

 

@rahulsharma​ Does a document with known issues like this one exist?

D

Conducted a screen share session with @dan_mbc​ and was able to resolve the issue.

BACKGROUND:

One of the attributes (string), contains some text with angle brackets (<, >), which are being interpreted as HTML tags when the translating or writing data to existing Portal Feature services, which causes an error on the server-side, and the features get rejected.

 

RESOLUTION: Replace Angle brackets

Option1 : Use StringReplacer transformer to remove the angle brackets

Option2: Use StringPairReplacer to replace <this> to (this)

 

Thanks again for your time Rahul, the workbench runs perfectly now.

 

I no doubt will encounter this problem again too, so was great to diagnose the problem.

rahulsharma
Rank
Badge +8

@rahulsharma​ Does a document with known issues like this one exist?

Hi @nielsgerrits​ 

Currently, there is not one but our team is working on this, stay tuned.

Reply


Community Stats

28,729
Posts
109,527
Replies
36,565
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.