Question

How to know what is inside zip file for processing

  • 19 September 2016
  • 10 replies
  • 31 views
G
Badge

Hi all,

New on the forum but not on FME :) (although not an expert). I come acrross a doubt on how to work with zip files. I know FME reads zip files directly but only if you know what kind of file is inside it. So If I have some URL with zip files but those can have dwg, shp, png...I cant really figure it out how to say FME ok, you shape file go this path, dwg this other path... I know I could use httpcaller and store them, unzip and work with them after unzipping by filtering out the inputs but I want to save disk space and dont really want to unzip.

Any idea will be welcome.

Thanks!

10 replies

takashi
Rank
Userlevel 2
Badge +17

Hi @geodavid76, I think Python ZipFile.namelist() method would be a quick way. See here to learn more: Python documentation | ZipFile.namelist()

itay
Rank
Badge +16

Hi @geodavid76, you could also use the list commend on 7z and read back the contents of the archive into the workspace.

G
Badge

I have tried with the zipfile.namelist() but I dont manage to get data out of it although seems to be working.

from zipfile import ZipFile

 

 

def fmeUnzip(fmeFeature):

 

zipFile = fmeFeature.getAttribute("_response_file_path")

 

z = ZipFile(zipFile)

 

z.namelist()

how can I print in python to a field in FME? This is what Im not doing well now...

takashi
Rank
Userlevel 2
Badge +17

I have tried with the zipfile.namelist() but I dont manage to get data out of it although seems to be working. 

from zipfile import ZipFile

 

 

def fmeUnzip(fmeFeature):

 

  zipFile = fmeFeature.getAttribute("_response_file_path")

 

  z = ZipFile(zipFile)

 

  z.namelist()

how can I print in python to a field in FME? This is what Im not doing well now...

@geodavid76, the namelist method returns a Python list containing archived file names as its elements, so you will have to retrieve each element. For example, a PythonCaller with this script outputs features having an attribute called "_name" that stores a file name for each.

 

# PythonCaller Script Example
from zipfile import ZipFile
class FeatureProcessor(object):
    def input(self, feature):
        path = feature.getAttribute("_response_file_path")
        with ZipFile(path, 'r') as z:
            for name in z.namelist():
                feature.setAttribute('_name', name)
                self.pyoutput(feature)
G
Badge

It works perfectly now!

mark2atsafe
Rank
Userlevel 4
Badge +25
Sadly the File/Directory reader won't read a list of files from a zip archive. It's already been filed as an enhancement request (PR#69077) and if/when we implemented that it would solve this problem without the need for Python.

 

ebygomm
Rank
Userlevel 1
Badge +21
Sadly the File/Directory reader won't read a list of files from a zip archive. It's already been filed as an enhancement request (PR#69077) and if/when we implemented that it would solve this problem without the need for Python.

 

Is this still the case, python is the only option to validate the contents of a zipfile?

mark2atsafe
Rank
Userlevel 4
Badge +25

Is this still the case, python is the only option to validate the contents of a zipfile?

I think so. I can see a number of different requests for this functionality, spread across several issues in our database. So I've pinged the developers to let them know and see if we can get a fix scheduled. fyi the new reference number is FMEENGINE-38193.

But as for now, I think Python is the most reliable 😒

ebygomm
Rank
Userlevel 1
Badge +21

Is this still the case, python is the only option to validate the contents of a zipfile?

I thought i might be able to do something clever by using the schema output on the FeatureReader, but I'd need some sort of schemas first option I think, otherwise I'd have to introduce a fairly convoluted workflow to ensure that features haven't started to be written before confirming that all the files in the zip file are present and correct. It's only a line of python to get the file list...

arnold_bijlsma
Rank
Badge +3
Sadly the File/Directory reader won't read a list of files from a zip archive. It's already been filed as an enhancement request (PR#69077) and if/when we implemented that it would solve this problem without the need for Python.

 

@mark2atsafe​ Hi Mark.

 

Is there any update on this, as I have got exactly the same issue as @geodavid76​ ?

Reply


Community Stats

28,711
Posts
109,488
Replies
36,560
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.