Solved

Different flavors of Shape ?

lifalin2016
Rank
Userlevel 1
Badge +22

Hello all.

Last year, I delivered a number of Shape files to a customer, which they had no problems in using.

Now I've adapted another of our systems to create Shape files as a supplement to an email message.

These latter Shape files cannot be read by their ArcGIS Pro. The error message is "Failed to add data: ... A user-supplied component or subscriber raised an exception (Exception from HRESULT: 0x80040208)". I've also tried to apply Geopackages, but they err the same way.

 

Both output routines uses FME. The former used straight up fixed writers, and the latter uses FeatureWriter, but that shouldn't matter, right ?

I cannot see any difference between the two Shape files, and QGIS reads them both without any qualms.

 

Apparently they can read the offending Shape into QGIS, and have it save it as another Shape, that works in their ArcGIS Pro.

I cannot see any difference between the two flavors (in QGIS), except that one fails when input to ArcGIS Pro.

Does anyone have any insights as to why this might be happening ?

icon

Best answer by debbiatsafe 22 January 2022, 02:56

View original
1 Attachment
-- Cheers, Lars I.

10 replies

nielsgerrits
Rank
Userlevel 6
Badge +33

Not sure what the issue is, but I can't open any of the shapefiles my ArcGIS Pro 2.9. (Cannot connect to database.)

I can open the shapefile from the zip "Does not work" in QGIS, but I can't open the shapefile from "Does work" in QGIS 3.16.1 (Invalid Data Source).

I can open both shapefiles with FME 2021.2.0.0

debbiatsafe
Rank
Userlevel 3
Badge +17

Hi @Lars I Nielsen​ ,

I'm also unable to read both datasets with ArcGIS software. Note the shapefile in Does work.zip is missing the required .shx file.

 

Are you able to check if you're using the newer SHAPEFILE or the older deprecated ESRISHAPE format in your workspace?

lifalin2016
Rank
Userlevel 1
Badge +22

Hi @Lars I Nielsen​ ,

I'm also unable to read both datasets with ArcGIS software. Note the shapefile in Does work.zip is missing the required .shx file.

 

Are you able to check if you're using the newer SHAPEFILE or the older deprecated ESRISHAPE format in your workspace?

Oops, sorry about that. I've added it to the new "Does work 2" zip.

1 Attachment
-- Cheers, Lars I.
lifalin2016
Rank
Userlevel 1
Badge +22

Not sure what the issue is, but I can't open any of the shapefiles my ArcGIS Pro 2.9. (Cannot connect to database.)

I can open the shapefile from the zip "Does not work" in QGIS, but I can't open the shapefile from "Does work" in QGIS 3.16.1 (Invalid Data Source).

I can open both shapefiles with FME 2021.2.0.0

Try with the new "Does work" ZIP, in which I've added the missing SHX file, that Debbie pointed out. My bad.

-- Cheers, Lars I.
debbiatsafe
Rank
Userlevel 3
Badge +17

Hi @Lars I Nielsen​ ,

I'm also unable to read both datasets with ArcGIS software. Note the shapefile in Does work.zip is missing the required .shx file.

 

Are you able to check if you're using the newer SHAPEFILE or the older deprecated ESRISHAPE format in your workspace?

UPDATE: In FME 2022.1+, the Shapefile Writer will not allow the first character to be non-alpha and will prefix any attribute names starting with a non-alpha character with a ‘Q_’ string.

 

Hi @Lars I Nielsen​ 

Thanks for adding the .shx file. I suspect the issue may be caused by the attributes in the DBF file. There are a few attributes that start with an underscore. As noted in this Esri article, field names should not start with an underscore or number.

There are also a large number of repeat attributes with a number appended, leading to a large number of attributes (~210) with many of them not containing values. If you delete some of these attributes and rewrite the DBF, ArcGIS Pro is able to read the shapefile. I haven't isolated whether the issue is with the number of attributes or the values within them.

In QGIS, the attributes that start with an underscore are not read/displayed and I suspect their absence is why an export from QGIS works.

lifalin2016
Rank
Userlevel 1
Badge +22

UPDATE: In FME 2022.1+, the Shapefile Writer will not allow the first character to be non-alpha and will prefix any attribute names starting with a non-alpha character with a ‘Q_’ string.

 

Hi @Lars I Nielsen​ 

Thanks for adding the .shx file. I suspect the issue may be caused by the attributes in the DBF file. There are a few attributes that start with an underscore. As noted in this Esri article, field names should not start with an underscore or number.

There are also a large number of repeat attributes with a number appended, leading to a large number of attributes (~210) with many of them not containing values. If you delete some of these attributes and rewrite the DBF, ArcGIS Pro is able to read the shapefile. I haven't isolated whether the issue is with the number of attributes or the values within them.

In QGIS, the attributes that start with an underscore are not read/displayed and I suspect their absence is why an export from QGIS works.

Thanks Debbie.

Removing the attributes starting with an initial underscore did the trick, so problem solved 🙂

However, this prompts two questions:

  1. Why is FME even able to write a Shape file with such attributes in the first place, if they're deprecated and unwanted ?
  2. I'm using schema features output from a FeatureReader, which apparently includes attributes from the input features as well as from the read/output features. All the unwanted attributes comes from the input features only, and I expected the schema features to only contain schema attributes read from the FeatureReader, not the input features prompting the reading. Is this a bug og WAD ?

Cheers.

-- Cheers, Lars I.
debbiatsafe
Rank
Userlevel 3
Badge +17

UPDATE: In FME 2022.1+, the Shapefile Writer will not allow the first character to be non-alpha and will prefix any attribute names starting with a non-alpha character with a ‘Q_’ string.

 

Hi @Lars I Nielsen​ 

Thanks for adding the .shx file. I suspect the issue may be caused by the attributes in the DBF file. There are a few attributes that start with an underscore. As noted in this Esri article, field names should not start with an underscore or number.

There are also a large number of repeat attributes with a number appended, leading to a large number of attributes (~210) with many of them not containing values. If you delete some of these attributes and rewrite the DBF, ArcGIS Pro is able to read the shapefile. I haven't isolated whether the issue is with the number of attributes or the values within them.

In QGIS, the attributes that start with an underscore are not read/displayed and I suspect their absence is why an export from QGIS works.

Hi @Lars I Nielsen​ 

I will file a ticket for this issue as the writer currently allows underscores as the first character, and does not appear to follow the format specifications.

As for your second question, can you clarify if you are seeing the input feature's attributes within the schema list attribute (eg. attribute{}.name=inputFtAttribute)? Or simply the input feature's attributes attached to the schema feature as an attribute?

lifalin2016
Rank
Userlevel 1
Badge +22

UPDATE: In FME 2022.1+, the Shapefile Writer will not allow the first character to be non-alpha and will prefix any attribute names starting with a non-alpha character with a ‘Q_’ string.

 

Hi @Lars I Nielsen​ 

Thanks for adding the .shx file. I suspect the issue may be caused by the attributes in the DBF file. There are a few attributes that start with an underscore. As noted in this Esri article, field names should not start with an underscore or number.

There are also a large number of repeat attributes with a number appended, leading to a large number of attributes (~210) with many of them not containing values. If you delete some of these attributes and rewrite the DBF, ArcGIS Pro is able to read the shapefile. I haven't isolated whether the issue is with the number of attributes or the values within them.

In QGIS, the attributes that start with an underscore are not read/displayed and I suspect their absence is why an export from QGIS works.

Hi Debbie.

This is actually a very good question.

There are 367 entries in the attribute{} list, so I didn't check every single entry beforehand, and when I expanded my filtering of that list to exclude entries with names starting with an underscore, and this solved the problem in the FeatureWriter, I assumed that they were included in that list.

Now I've examined the list more closely, and the "underscore-attributes" are _not_ in the attribute{} list, but only present as regular attributes !?!

So to sum up what happened, my fix didn't really change anything, but somehow miraculously it now works ???

I better fast remove those "underscore-attributes" just for good measure, as to not rely on "miracles".

-------------- My additional 3rd question is then this:

If a FeatureWriter is told to use "Schema from Schema Feature", which I assume means (only) the attribute{} list, which I'm merging onto every single data feature (and removing "fme_schema_handling" to output the first feature too), why did the FW output seem to include the regular attributes too ?

 

Cheers

-- Cheers, Lars I.
debbiatsafe
Rank
Userlevel 3
Badge +17

UPDATE: In FME 2022.1+, the Shapefile Writer will not allow the first character to be non-alpha and will prefix any attribute names starting with a non-alpha character with a ‘Q_’ string.

 

Hi @Lars I Nielsen​ 

Thanks for adding the .shx file. I suspect the issue may be caused by the attributes in the DBF file. There are a few attributes that start with an underscore. As noted in this Esri article, field names should not start with an underscore or number.

There are also a large number of repeat attributes with a number appended, leading to a large number of attributes (~210) with many of them not containing values. If you delete some of these attributes and rewrite the DBF, ArcGIS Pro is able to read the shapefile. I haven't isolated whether the issue is with the number of attributes or the values within them.

In QGIS, the attributes that start with an underscore are not read/displayed and I suspect their absence is why an export from QGIS works.

I suspect you may be seeing this behaviour because of the way the FeatureWriter works.

 

When adding a FeatureWriter, the Attribute Definition is set to Automatic mode by default. This adds all attributes on incoming features to the Attribute Definition dialog.

When you enable Dynamic Schema Definition on the FeatureWriter, the Attribute Definition changes to Dynamic mode but all of the previously added attributes are not removed from the Attribute Definition dialog. This results in all of the attributes on incoming features being output.

lifalin2016
Rank
Userlevel 1
Badge +22

UPDATE: In FME 2022.1+, the Shapefile Writer will not allow the first character to be non-alpha and will prefix any attribute names starting with a non-alpha character with a ‘Q_’ string.

 

Hi @Lars I Nielsen​ 

Thanks for adding the .shx file. I suspect the issue may be caused by the attributes in the DBF file. There are a few attributes that start with an underscore. As noted in this Esri article, field names should not start with an underscore or number.

There are also a large number of repeat attributes with a number appended, leading to a large number of attributes (~210) with many of them not containing values. If you delete some of these attributes and rewrite the DBF, ArcGIS Pro is able to read the shapefile. I haven't isolated whether the issue is with the number of attributes or the values within them.

In QGIS, the attributes that start with an underscore are not read/displayed and I suspect their absence is why an export from QGIS works.

Thanks Debbie.

I did encounter this very problem in the same workspace, but got that answered seperately in this thread.

I just double-checked that all my FeatureWriter user attribute panes are indeed empty now. My sample "does work" shape file must have been made before I got that problem sorted out.

Thanks for clearing that up for me.

-- Cheers, Lars I.

Reply


Community Stats

28,731
Posts
109,530
Replies
36,567
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.