Question

Reading ZIP files in FME server from untrusted source that might contain malware/virus

  • 17 June 2019
  • 9 replies
  • 11 views
oscarwillems
Rank
Badge +8

Hello,

For a project we are looking into building several workspaces/token-based apps that can be run on FME Server. Due to strict security requirements we have to make sure that either compressed files are rejected or are handled in a safe manner.

As I understood FME 2018 and older decompress the zip file first to a temp folder, then using the extracted files to read the data (into ffs), this potentially exposes the file system to malicious files from the zip.

Upon being read, the data is extracted and used just as if it were a normal dataset. FME is able to read data stored directly in an archive file, or within sub-folders. archive_files_reading.htm
Notable New Features 2019.0

Automatic Decompression

Read your compressed data files directly in FME without having to decompress them first. Another barrier has been taken down!

FME 2019 New Features

Does this expose the content of the zip to the file system? Somehow the files of the zip have to be stored somewhere. EDIT: Yes it still extracts everything to a temp folder including files not in the filter (somearchive.zip\*.xml)

How does FME deal with potentially harmful zip archives? Store it at an intermediate location and run a virus scanner and then use the validated zip?

9 replies

david_r
Rank
Userlevel 4

You wrote:

FME 2019 decompresses the zip directly to FME's internal ffs format [...]

Have you verified this? It was my understanding that FME would simply unzip the data as-is, without any conversion.

I would not assume that FME does anything at all to detect potentially harmful zip files, as far as I know there are no such mechanisms built into FME Server. My recommendation would be to custom build the zip file deposit functions before passing it on to FME Server.

Operations might include

  • Sanitize zip filename to avoid illegal characters
  • Sanitize/check filenames of zip file contents to avoid illegal characters
  • Check for invalid/malicious zip files
  • Check zip size (to avoid filling disk space, should be done at upload to terminate streaming once size limit has been reached)
  • Verify checksums in zip headers
  • Scan for virus and trojans

Basically, as you mention, allowing "anybody" to upload "anything" is a huge security risk.

oscarwillems
Rank
Badge +8

You wrote:

FME 2019 decompresses the zip directly to FME's internal ffs format [...]

Have you verified this? It was my understanding that FME would simply unzip the data as-is, without any conversion.

I would not assume that FME does anything at all to detect potentially harmful zip files, as far as I know there are no such mechanisms built into FME Server. My recommendation would be to custom build the zip file deposit functions before passing it on to FME Server.

Operations might include

  • Sanitize zip filename to avoid illegal characters
  • Sanitize/check filenames of zip file contents to avoid illegal characters
  • Check for invalid/malicious zip files
  • Check zip size (to avoid filling disk space, should be done at upload to terminate streaming once size limit has been reached)
  • Verify checksums in zip headers
  • Scan for virus and trojans

Basically, as you mention, allowing "anybody" to upload "anything" is a huge security risk.

Notable New FeaturesAutomatic Decompression

Read your compressed data files directly in FME without having to decompress them first. Another barrier has been taken down!

FME 2019 New Features

david_r
Rank
Userlevel 4
Notable New FeaturesAutomatic Decompression

Read your compressed data files directly in FME without having to decompress them first. Another barrier has been taken down!

FME 2019 New Features

Thanks! That's interesting. But does it really mean that the files are automatically converted to FFS, or are they simply streamed into the regular reader from the zip libraries?

oscarwillems
Rank
Badge +8

Thanks! That's interesting. But does it really mean that the files are automatically converted to FFS, or are they simply streamed into the regular reader from the zip libraries?

I did some testing, and it still extracts everything to a temp folder, albeit briefly (only trackable when running with breakpoints) including my "MaliciousFile.ps1" that I sneaked into my zip.... even with the *.xml filter in the path name (.zip\\*.xml)

So zip file and archive support might be something that was already on 2018 but is now officially in 2019

david_r
Rank
Userlevel 4

I did some testing, and it still extracts everything to a temp folder, albeit briefly (only trackable when running with breakpoints) including my "MaliciousFile.ps1" that I sneaked into my zip.... even with the *.xml filter in the path name (.zip\\*.xml)

So zip file and archive support might be something that was already on 2018 but is now officially in 2019

Thats very interesting, thanks for testing and sharing your results!

runneals
Rank
Badge +14

@rylanatsafe This would be something good to take a look at if you haven't already

rylanatsafe
Rank
Badge +11

@rylanatsafe This would be something good to take a look at if you haven't already

Thank you @runneals for the notification – I had not seen this thread yet.

rylanatsafe
Rank
Badge +11

You wrote:

FME 2019 decompresses the zip directly to FME's internal ffs format [...]

Have you verified this? It was my understanding that FME would simply unzip the data as-is, without any conversion.

I would not assume that FME does anything at all to detect potentially harmful zip files, as far as I know there are no such mechanisms built into FME Server. My recommendation would be to custom build the zip file deposit functions before passing it on to FME Server.

Operations might include

  • Sanitize zip filename to avoid illegal characters
  • Sanitize/check filenames of zip file contents to avoid illegal characters
  • Check for invalid/malicious zip files
  • Check zip size (to avoid filling disk space, should be done at upload to terminate streaming once size limit has been reached)
  • Verify checksums in zip headers
  • Scan for virus and trojans

Basically, as you mention, allowing "anybody" to upload "anything" is a huge security risk.

@oscarwillems After reviewing this Q&A posting in its entirety, I think that @david_r provides sound recommendations.

The posting in the "FME 2019 New Features" article is a little ambiguous with its wording... There was no change in functionality for decompressing ZIP files – rather we added additional support for other compression formats (.rar, .tar, .tar.gz, among others).

As observed, FME does extract all contents of the compressed file – and noted that these files are cleaned up after the translation, i.e. only temporarily stored, and they are not manipulated by FME unless configured in the FME Workspace itself.

The filter will not prevent certain files from extracting to your file system, but it will limit what FME reads and will actively perform translations on, such as converting to the internal format FFS.

We do not have plans to implement any virus or security utility with FME Desktop or FME Server for the context of scanning / verifying compressed files used in translations.

If you have any further concerns, please add comment or get in touch with our Support Team.

chk1
Rank
Badge +6

You wrote:

FME 2019 decompresses the zip directly to FME's internal ffs format [...]

Have you verified this? It was my understanding that FME would simply unzip the data as-is, without any conversion.

I would not assume that FME does anything at all to detect potentially harmful zip files, as far as I know there are no such mechanisms built into FME Server. My recommendation would be to custom build the zip file deposit functions before passing it on to FME Server.

Operations might include

  • Sanitize zip filename to avoid illegal characters
  • Sanitize/check filenames of zip file contents to avoid illegal characters
  • Check for invalid/malicious zip files
  • Check zip size (to avoid filling disk space, should be done at upload to terminate streaming once size limit has been reached)
  • Verify checksums in zip headers
  • Scan for virus and trojans

Basically, as you mention, allowing "anybody" to upload "anything" is a huge security risk.

Hello David, you mention "Check zip size", checking the contents should be of concern as well. There is a technique called a "zip bomb" (basically, stuffing giant text files into a tiny zip file, some techniques also involve modifying the zip headers to include this file several times). Just something that has been on my mind for a while, not sure if FME (or the underlying libraries) checks for such kind of files.

Reply


Community Stats

28,729
Posts
109,527
Replies
36,564
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.