Solved

Attribute definition names truncated in SHAPE file writer

  • 19 April 2020
  • 7 replies
  • 51 views
fmeuser_gc
Rank
Badge +6

Workspace has just ESRI Geodatabase reader and SHAPE file writer (FME 2019 version) as I'm trying to write to a shapefile.

In Shapefile writer, few attribute definition names are being shortened(truncated) but retains the same data type(as reader). Tried to rename those shortened attributes(in writer), but still it reverts to shortened version. Thereby causing difference in schema definition between reader and writer.

This is the case for all different types of Attribute Definitions - Automatic, Manual, Copy from Reader..

I think I've tried possible workarounds, but still same issue.

 

Any thoughts please? Thanks!

icon

Best answer by erik_jan 20 April 2020, 02:43

View original

7 replies

erik_jan
Rank
Userlevel 2
Badge +16

You ran into one of the limitations of the Shape file: Attribute names can be no longer than 10 characters.

That is why I recommend File Geodatabase whenever possible.

If it is not possible then use the Shape file, but to map all attributes automatically, right click the link to the Shape writer feature type and choose "Auto Connect Attributes" from the menu.

This will tell FME which source attribute will map to which destination attribute, and since they were truncated by the writer these will map correctly.

Hope this helps.

B
Badge +3

You ran into one of the limitations of the Shape file: Attribute names can be no longer than 10 characters.

That is why I recommend File Geodatabase whenever possible.

If it is not possible then use the Shape file, but to map all attributes automatically, right click the link to the Shape writer feature type and choose "Auto Connect Attributes" from the menu.

This will tell FME which source attribute will map to which destination attribute, and since they were truncated by the writer these will map correctly.

Hope this helps.

It's amazing how long-lived the SHP file format is, considering all its annoying limitations. Beyond the known 10 character Field Name limit, there are limits on how large the Text Fields can be, how many fields are allowed, they don't work terribly well with large datasets, have flaky NULL and date/time support, are not the most field space efficient and aren't the fastest things to read/write as well............but every application with GIS integration has SHP file format support!

Personally, I agree with @erik_jan, where possible use a more modern data container. FGDBs are pretty good, and I use SpatiaLite a lot since it gives a single, portable compact file that can be quickly and easily opened and read by ArcGIS, QGIS etc. whilst being able to be easily ODBC'ed and SQL'ed natively without having to do anything else to it.

....But that doesn't stop external companies from sending us their data as SHP files. They still keep doing it!

fmeuser_gc
Rank
Badge +6

You ran into one of the limitations of the Shape file: Attribute names can be no longer than 10 characters.

That is why I recommend File Geodatabase whenever possible.

If it is not possible then use the Shape file, but to map all attributes automatically, right click the link to the Shape writer feature type and choose "Auto Connect Attributes" from the menu.

This will tell FME which source attribute will map to which destination attribute, and since they were truncated by the writer these will map correctly.

Hope this helps.

Thanks @erik_jan. Yeah, the attributes were connected but in some cases truncated names in SHP file aren't meaningful as it only shows the 10 characters.

Considering that SHP file reader/writer is most popular format, hope 'safe' considers this enhancement in next releases.

erik_jan
Rank
Userlevel 2
Badge +16

Thanks @erik_jan. Yeah, the attributes were connected but in some cases truncated names in SHP file aren't meaningful as it only shows the 10 characters.

Considering that SHP file reader/writer is most popular format, hope 'safe' considers this enhancement in next releases.

There is nothing Safe can do about it.

I am sure they would have done it a long time ago, if it was possible.

It is a limitation of the Shape format, not of the Shape writer in FME.

fmeuser_gc
Rank
Badge +6

It's amazing how long-lived the SHP file format is, considering all its annoying limitations. Beyond the known 10 character Field Name limit, there are limits on how large the Text Fields can be, how many fields are allowed, they don't work terribly well with large datasets, have flaky NULL and date/time support, are not the most field space efficient and aren't the fastest things to read/write as well............but every application with GIS integration has SHP file format support!

Personally, I agree with @erik_jan, where possible use a more modern data container. FGDBs are pretty good, and I use SpatiaLite a lot since it gives a single, portable compact file that can be quickly and easily opened and read by ArcGIS, QGIS etc. whilst being able to be easily ODBC'ed and SQL'ed natively without having to do anything else to it.

....But that doesn't stop external companies from sending us their data as SHP files. They still keep doing it!

Thanks @bwn. We mainly deal with SDE instead of FGDB and ton of SHP files. I didn't know about the Text field limitation that you mentioned. I've to consider using FGDB.

Hope we get all these sorted out in next releases.

B
Badge +3

There is nothing Safe can do about it.

I am sure they would have done it a long time ago, if it was possible.

It is a limitation of the Shape format, not of the Shape writer in FME.

Yes, it is up to ESRI to change or update the format, but they've been pretty public that even they don't want their users using Shape format anymore and would prefer them to switch to FGDB as the standard file based format. Shape is maintained as a legacy format.

It may be around for a while yet though. Microsoft recently deprecated their support for dBase in Microsoft Access (dBase being the backend non-spatial data store for Shape) as it was an "old, no longer used format". All the GIS community then madly bombarded Microsoft with "we still use it with our Shape files!" so Microsoft re-activated dBase support!

B
Badge +3

Thanks @bwn. We mainly deal with SDE instead of FGDB and ton of SHP files. I didn't know about the Text field limitation that you mentioned. I've to consider using FGDB.

Hope we get all these sorted out in next releases.

FGDB was created from SDE technology which is why they share virtually identical object definitions. A Feature Class, Relationship Class, Geometric Network etc. that exists in SDE can all be directly transposed into an FGDB and vice-versa. None of those classes exist in the SHP file format.

This was ESRI's solution that was meant to end the reliance and usage of SHP but it hasn't quite turned out that way......one problem was the ESRI was very late in releasing a workable FGDB API and then what we have today is still not quite complete......plus they didn't bank on how much people had invested into making solutions exchange data in the SHP format and didn't want to spend time/money converting them to instead exchange data in FGDB or another more modern GIS format.

However, the SHP format is unlikely to get an upgrade. ESRI doesn't even own the dBase technology it uses.

Reply


Community Stats

28,692
Posts
109,433
Replies
36,544
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.