Solved

S-57 ENC Reader : bug in implementation of gdal ?

M

Hello,

I use FME to transform data in S-57 format (the native format of our databases) into more traditional formats, in particular shapefiles.

 

The database export is therefore in S-57 format (extension .000), with the classes and attributes present for the ENCs, but this model is enriched with additional attributes and classes. This can be done by modifying the files s57attributes.csv, s57objectclasses.csv and s57expectedinput.csv in the directory $FME_HOME/s57 (Custom Object Classes File and Custom Attributes File). These files are also useful for correctly displaying additional attributes and classes in QGis (in the directory C:\\Program Files<QGIS>\\share\\gdal). While in QGis an S-57 file with additional classes and attributes is displayed correctly, this is not the case in FME.

  • For additional classes: some are handled correctly, others generate an error when read and you can't go further in the processing (there are also cases where the class seems to be ignored, without crashing the processing). It seems that this is related to the class identifier that appears in the file s57objectclasses.csv
  • For additional attributes: some are correctly handled, others are not read. In the same way, it seems that it is related to the identifier of the attribute which appears in the file s57attributes.csv

 

In both cases, if the values of the class or attribute codes are too high, the classes or attributes are misread.

Both QGis and FME rely on the gdal library to read the S-57 format. It seems that the implementation of gdal in FME is not complete.

 

I am working with the following version of FME: FME(R) 2020.1.1.1 (20200817 - Build 20614 - WIN64)

An upgrade to FME 2022 is planned for my workstation in the near future. However, from what I've read, I don't think there have been any improvements on the Reader S-57 since then.

 

I am attaching the 3 csv files s57attributes.csv, s57objectclasses.csv and s57expectedinput.csv as well as 2 S-57 files for analysis: test_s57_2_classes.000 (1 S-57 class + 1 additional class correctly read) and test_s57_3_classes.000 (the same objects as test_s57_2_classes.000 + 1 additional class incorrectly read: aisatn). The objects of the 2 files contain additional attributes not managed by FME: for example "resume" or "descrp".

 

I have found a solution with the use of the ogr2ogr executable, but this makes processing more complex.

 

Can anyone from safe could analyze my problem ?

 

Regards,

Mélanie

icon

Best answer by markatsafe 10 April 2023, 20:20

View original
1 Attachment

9 replies

M
Badge +2

@melanied​ It looks like FME is  is reading the 'Codes' as int16 (max value 32,767) instead of unit16 (max value 65,535) or int32. This means that the custom codes you have added, i.e. in s57attributes.csv:

"Code","Attribute","Acronym","Attributetype","Class"
61014,List of Lights Publication Identifier,$lolid,E,$
61021,Resume pour PING,resume,S,F

are not being read correctly. If you are able to change the code values to be less than 32000 then I think FME will read your custom features OK.  For example:

"Code","Attribute","Acronym","Attributetype","Class"
21014,List of Lights Publication Identifier,$lolid,E,$
21021,Resume pour PING,resume,S,F

This applies to all three of the configuration files you've modified.  We'll try and get this fixed in FME. 

M

@Mark Stoakes​ 

Thank you for your answer.

Unfortunately, it is not possible to change these codes because they are already used in a lot of data produced, so we would not be able to read "old data" after.

 

In the official documentation about S-57 (https://iho.int/uploads/user/pubs/standards/s-57/31Main.pdf), on page 71 (7.5.1.1 Data dictionary definition field structure), there is this table :

tableThe maximum value for these codes, class or attribute is indeed 65534.

 

I would appreciate (and some colleagues as well) a correction in a further version of FME.

 

 

M

Now I have FME 2022.1 : it seems that this bug is fixed. I appreciate also the possibility to choose custom object classes file and custom attributes file in advanced options of the reader.

A lot of simplifications for my workspaces !

 

M
Badge +2

@melanied​ My apologies for not letting you know that this was addressed in FME 2022.1. FME had a hard limit on the s57 classes which we removed in FME 2022.1

M

Hi again !

So everything works fine with FME 2022.1 when I replace files s57attributes.csv and s57objectclasses.csv in $FME_HOME/s57 directory.

In the reader, I select ENC 3.1 and custom classes and attributes are correctly read.

 

If I keep the original files s57attributes.csv and s57objectclasses.csv in $FME_HOME/s57 directory and I select the custom files in advanced options the files are not correctly read.

Capture d’écran 2023-04-07 111324For custom attributes, it's ok but for custom classes, there are in the generic port.

You can make the test with the example files of my first post.

 

Regards,

 

Mélanie

M
Badge +2

@melanied​ the feature that is read as Generic has OBJL 61003 which not is not in your objectclasses.csv . , so it appears as the Generic feature type. 'fishzn' is listed OBJL 720 and so is seen as the feature type 'fishzn'

I think you should probably make separate CSV files that only have you custom classes - see the attached.

You still need to replace the original s57expectedinput.csv with the revised file.

1 Attachment
M

@Mark Stoakes​ 

Thank you for your answer but I don't agree (and help neither) !

In help, it is written :

Custom Object Classes File

A CSV file containing custom object classes. Object class definitions in this file will override existing definitions with the same code.

Custom Attributes File

A CSV file containing custom attributes. These attributes will be added to all object class definitions. Attribute definitions in this file will override existing definitions with the same code.

 

=> "the feature that is read as Generic has OBJL 61003 which not is not in your objectclasses.csv " : yes it is in my objectclasses.csv, at the last line !

And this file is correct because if I replace in $FME_HOME/s57 directory the files are read correctly.

=> "'fishzn' is listed OBJL 720 and so is seen as the feature type 'fishzn'" : no, this class is not read.

Capture d’écran 2023-04-11 114525The interest of this advanced option is to avoid to do copies of custom files in the installation directory, which I will continue to do for the moment for the objectclasses.csv file ; s57attributes.csv is correctly taken into account with this option : additional attributes are taken into account.

 

 

M
Badge +2

@melanied​ My error I was looking at the wrong file. I am also working with FME 2022.2. I reproduced your issue using the complete files. When I create a custom file with only the new feature classes I get the expected result

2023-04-11_9-16-23I'll see if we can change FME to read a complete file.

M

OK I have this result too with a differential file. I'm quite confused about the result concerning the attributes of FAIRWY class : custom attributes of this class are correctly read although they are not described in the differential objectclasses.csv file neither in the original objectclasses.csv file.

Reply


Community Stats

28,684
Posts
109,411
Replies
36,542
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.