Skip to main content
Solved

Zipping files: libraries, compression levels

fikusas
Contributor
Forum|alt.badge.img+5

I have multiple shapefiles (total size is 8.4 GB). Those need to be zipped. When using FME's ZipArchiver, compressed file size is 4.43 GB. But when using third party app – 7zip, the result file is ~11% smaller (3.94 GB). My guess woul be that ZipArchiver uses different libararies for compression. Maybe it is possible to use 7zip libraries in ZipArchiver, when compressing files?

LZMA compression won't work in this case, because I need to be able to extract archives with built-in Windows extractor.

Best answer by takashi

Hi @fikusas, the ZipArchiver uses the native Python zipfile module to perform archiving, and I don't have any plan to change it to use 7-Zip.

In Python 3.6 and earlier, the zipfile module doesn't have any options to control compression level.

In Python 3.7 or later, the zipfile.ZipFile class has complesslevel parameter. If the workspace is configured to run with Python 3.7+, I think you can modify the script of the PythonCaller within the transformer to control the compression level. See here to learn more:

ZipFile Objects - Python Documentation

Another option, if you prefer to use 7-Zip to compress files, you can use the SystemCaller to execute 7z.exe which is the command line version of 7-Zip. See here to learn more:

Command Line Version User's Guide - 7z.exe

View original
Did this help you find an answer to your question?

3 replies

takashi
Contributor
Forum|alt.badge.img+19
  • takashiContributor
  • Contributor
  • Best Answer
  • April 25, 2020

Hi @fikusas, the ZipArchiver uses the native Python zipfile module to perform archiving, and I don't have any plan to change it to use 7-Zip.

In Python 3.6 and earlier, the zipfile module doesn't have any options to control compression level.

In Python 3.7 or later, the zipfile.ZipFile class has complesslevel parameter. If the workspace is configured to run with Python 3.7+, I think you can modify the script of the PythonCaller within the transformer to control the compression level. See here to learn more:

ZipFile Objects - Python Documentation

Another option, if you prefer to use 7-Zip to compress files, you can use the SystemCaller to execute 7z.exe which is the command line version of 7-Zip. See here to learn more:

Command Line Version User's Guide - 7z.exe

takashi
Contributor
Forum|alt.badge.img+19
  • takashiContributor
  • Contributor
  • April 25, 2020
takashi wrote:

Hi @fikusas, the ZipArchiver uses the native Python zipfile module to perform archiving, and I don't have any plan to change it to use 7-Zip.

In Python 3.6 and earlier, the zipfile module doesn't have any options to control compression level.

In Python 3.7 or later, the zipfile.ZipFile class has complesslevel parameter. If the workspace is configured to run with Python 3.7+, I think you can modify the script of the PythonCaller within the transformer to control the compression level. See here to learn more:

ZipFile Objects - Python Documentation

Another option, if you prefer to use 7-Zip to compress files, you can use the SystemCaller to execute 7z.exe which is the command line version of 7-Zip. See here to learn more:

Command Line Version User's Guide - 7z.exe

I added "Compression Level" parameter to the ZipArchiver. You can download the new one from FME Hub site.

https://hub.safe.com/publishers/pacific-spatial-solutions/transformers/ziparchiver

Just be aware that the parameter works only if the version of Python interpreter is 3.7 or later. If the Python version is 3.6 or earlier, the parameters setting will just be ignored.

----------

Compression Level (Python 3.7+) :

If FME uses Python 3.7+ when running the workspace, you can control the compression levels with this parameter.

  • Level 0 indicates just archiving with no compression.
  • In a level between 1-9, less number indicates faster and less compression, greater number indicates slower and more compression. However, the order of the numbers does not always exactly match the order of actual compression levels. Depending on the conditions, 6 could be more compression than 9 for example.
  • Currently (in Python 3.7), "Default" is equivalent to level 6, but it may be changed in the future.

This parameter works only with Python 3.7 or later. If the version of Python interpreter is 3.6 or earlier, this parameter setting will just be ignored.

fikusas
Contributor
Forum|alt.badge.img+5
  • fikusasContributor
  • Author
  • Contributor
  • April 25, 2020

Thanks for your effort, @takashi! compresslevel of 9 doesn't gives as good compression as using 7zip. SystemCaller worked just fine zipping via 7z.exe.

Reply


Helpful Members This Week

Community Stats

30,942
Posts
117,252
Replies
38,742
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.