Question

Read zip files wich a user can upload

  • 10 June 2021
  • 9 replies
  • 46 views

Hi,

I would like to create a fme server app where a user can upload multiple ZIP files. For each zipfile I would like to call a FeatureReader for one file within the zip file. How can I accomplish that?

I tried the directory and file path names reader but this gives an error in FME server saying it expects a folder and not a file.

Thank you


9 replies

Userlevel 5
Badge +29

FME ultimately treats a zip file as a folder. When setting up the reader, it needs to be of the format that is in the zip folder. The path to read it is then something like:

"C:\\temp\\folder\\data.zip\\data.csv"

But when I set up a GML reader, for instance, how can I let it know which file from the ZIP to read? This is why I thought I need a feature reader, but I don't know a way how to ask the user for a file in FME Server app without having a reader.

Badge +2

But when I set up a GML reader, for instance, how can I let it know which file from the ZIP to read? This is why I thought I need a feature reader, but I don't know a way how to ask the user for a file in FME Server app without having a reader.

Hi @arnovananrooij​ ,

To be able to upload a file in FME Server all that is required is a Published Parameter of type 'File' so if you would like to use a Feature Reader in your workspace, create a file user parameter and then point the FeatureReader dataset to that user parameter.

 

If your zip file contains multiple files of different formats, then FME will only read in the file that matches that format.

 

For example, if you have a zip file that contains CSV and GML. Your workspace will have two Readers/Features Readers, one pointing to CSV and one pointing to GML format, but they can both point to the same File User Parameter holding the zip filepath value and each reader will only read in the file that's extension matches the specified format.

Userlevel 5
Badge +29

But when I set up a GML reader, for instance, how can I let it know which file from the ZIP to read? This is why I thought I need a feature reader, but I don't know a way how to ask the user for a file in FME Server app without having a reader.

Something you might not be aware of is FME can also use wildcards when its reading data.

A path formatted like "C:\\temp\\folder\\*.zip\\*.csv" would read all csv's in all zips that are in "folder"

See more here (under Path Filter)

https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_ReadersWriters/path/PATH_reader.htm

Something you might not be aware of is FME can also use wildcards when its reading data.

A path formatted like "C:\\temp\\folder\\*.zip\\*.csv" would read all csv's in all zips that are in "folder"

See more here (under Path Filter)

https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_ReadersWriters/path/PATH_reader.htm

Yes, I'm aware of this. But I don't know how to configure it in FME Server. I now have a gml reader and i want to read **\\*.xml from the zip file. I've tried everything but still won't work. It looks like it wants to read the zip file as a gml file and gives an error: invalid byte 'R' at position 2 of a 2-byte sequence'.

Userlevel 5
Badge +29

Something you might not be aware of is FME can also use wildcards when its reading data.

A path formatted like "C:\\temp\\folder\\*.zip\\*.csv" would read all csv's in all zips that are in "folder"

See more here (under Path Filter)

https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_ReadersWriters/path/PATH_reader.htm

In the FeatureReader, the file pathname should be $(ParameterName)\\**\\*.xml - I tend to put a folder wild card in regardless of whether there will be a folder or not, doesn't hurt to have it there

$(ParameteName) will be the value from the parameter you set as type 'File' (from @hollyatsafe​ )

@hkingsbury​ @hollyatsafe​ 

Now my workspace starts with a creator and then a FeatureReader. The featureReader reads the correct file from the zip file, bij combining the parameter of type Multi File with wildcards to something like $(Bestanden)\\**\\GI_Gebiedsinformatielevering*.xml. This works fine in FME Desktop. In FME Server however I get an error: line:1 column:13 message:invalid byte 'R' at position 2 of a 2-byte sequence'. This is the same error I get when I had the workspace with a direct XML reader, so without the FeatureReader.

Userlevel 5
Badge +29

@hkingsbury​ @hollyatsafe​ 

Now my workspace starts with a creator and then a FeatureReader. The featureReader reads the correct file from the zip file, bij combining the parameter of type Multi File with wildcards to something like $(Bestanden)\\**\\GI_Gebiedsinformatielevering*.xml. This works fine in FME Desktop. In FME Server however I get an error: line:1 column:13 message:invalid byte 'R' at position 2 of a 2-byte sequence'. This is the same error I get when I had the workspace with a direct XML reader, so without the FeatureReader.

Could it be to do with the xml file encoding? See here:

https://stackoverflow.com/questions/2421272/invalid-byte-2-of-2-byte-utf-8-sequence

Badge +8

@arnovananrooij​ , I know I'm about two years late, but I just saw this issue. I have exactly the same issue right now and it only occurs with xml files. Solution: read the file as text file with the setting whole file at once. After that you can use all the XML-transformers in the normal way.

Reply