Question

ESRISHAPE readers don't fetch schemas from all source tables in ZIP archives

  • 22 March 2017
  • 5 replies
  • 2 views

Hello!

TLDR; My ESRISHAPE readers don't fetch schemas from all my source tables.

I have a workspace with multiple ESRISHAPE readers, each of which are reading its own separate ZIP archive.

The readers are using Merge Feature Type (*) to allow all Shape files in each archive to be read.

I'm writing all these Shape files to a dynamic MSSQL_SPATIAL writer using "fme_feature_type" as Schema Definition Name. Most Shape files have copies (containing data from separate geographic regions) in each source archive. Their region suffix is ignored so they can be written to the same destination tables.

It goes something like this:

  • Reader1
    • The archive contains the tables:
      • Table_A_1
      • Table_B_1
      • Table_C_1
    • The reader fetches the schemas of all the tables in the first archive, logging:
      • "Reader `ESRISHAPE_1' of type `ESRISHAPE' provided schema for types: Table_A_1,Table_B_1,Table_C_1"
  • Reader2
    • The archive contains the tables:
      • Table_A_2
      • Table_B_2
      • Table_C_2
      • Table_D_2
    • The reader seems to only fetch the schema of the first table:
      • "Reader `ESRISHAPE_2' of type `ESRISHAPE' provided schema for types: Table_A_2"
  • Writer1
    • All the features of Tables A-C from Reader1 are written to the tables:
      • Table_A
      • Table_B
      • Table_C
  • Tables A-C from Reader2 are also written to the same destination tables
  • When features from Table_D_2 are about to be written, Writer1 encounters a fatal error:
    • "Microsoft SQL Server Spatial Writer: A feature has arrived with 'fme_feature_type' of Table_D, but the feature type is not recognized. Halting translation"

If I move Reader2 above Reader1 in the Navigator to make Reader2 go first, then all table schemas from Reader2 are read, and Reader1 can write all of its features to tables A-C.

This leads me to believe that there is some efficiency feature in the ESRISHAPE reader type, or the Multi Reader, which might be possible to turn off?

It's not a solution for me to just change the order of the readers, since in my actual case I have 15 readers and no way to guarantee that any of the archives contain all possible tables, e.g. Table_A to Table_Z.

Any ideas? :)


5 replies

Badge +22

Assuming the processing for each region is the same, why not have a single reader, reading all the zip files?

Badge +22

Assuming the processing for each region is the same, why not have a single reader, reading all the zip files?

I can replicate your issue, which is definetly a bug, and you should report it to SAFE tech support.

 

 

Assuming the processing for each region is the same, why not have a single reader, reading all the zip files?

Thanks for your quick suggestion, using a single reader solves my problem! :)

 

 

The reason I had 15 separate readers is that they actually read from separate URLs, to an FTP server, so unless I want to Python it out it seems the "Source Files" parameter will be quite long.

 

 

Luckily, in my case all the archives are on the same server so I can use the same Network Authentication parameters.

 

 

The readers were going through somewhat different processing, but I can separate the flow from the single reader using FindRegEx on the fme_dataset parameter :)

 

 

I'll report the bug to SAFE Tech support. Thanks again!

 

Userlevel 3
Badge +13

Good news @mattias! PR76790 has now been fixed. This should be working as expected in the latest available FME 2018.0 beta build. Please let us know if this is not the case.

Good news @mattias! PR76790 has now been fixed. This should be working as expected in the latest available FME 2018.0 beta build. Please let us know if this is not the case.

Thank you for continuously improving the product :) I look forward to testing 2018.0.

 

 

Reply