Skip to main content
Solved

FME download service seems to discard zipped outputfiles

  • November 26, 2019
  • 9 replies
  • 72 views
thijsknapen
Contributor
Forum|alt.badge.img+10

On FME Server I am not able to obtain a zipped output file within the data download url. In the log it does show that the file is created, but it doesn't return in the download url.

 

So to clarify, say I create a couple of files in my workbench:

1. file1.csv

2. file2.zip (directly containing e.g. file1.csv)

3. file3.txt

 

When I run the workspace locally, all of the files are created and placed in the specified outputfolder. If however I publish the workspace to FME Server I obtain a download link to a zipfile that only contains file1.csv and file3.txt

Any ideas on what I'm doing wrong? Or is it maybe just not possible to use the data download service in combination with zipped output files?

N.b. From browsing the forum, maybe the issue is related to: https://safesoftware.cloud.answerhub.com/questions/93696/fme-datastreaming-not-returning-zip-file.html

 

Best answer by helmoet

Hi @thijsknapen, you could try generating all output using FeatureWriters (or using FMEServerJobSubmitter with your original workspace). After that you could gather the results using a FeatureReader, choosing for Format "Directory and File Pathnames" and for Folder the directory you created the files in. After that connect a FileCopy writer that will be the only writer configured in the DataDownload service. The data download link will contain all files written by the filecopy writer, including zip files.

View original
Did this help you find an answer to your question?

9 replies

jdh
Contributor
Forum|alt.badge.img+28
  • jdhContributor
  • Contributor
  • November 26, 2019

When publishing the workspace to the server, there is a setting on the DataDownloadService about the writers to include in the zip. Do you have all the writers selected?

thijsknapen
Contributor
Forum|alt.badge.img+10
  • thijsknapenContributor
  • Author
  • Contributor
  • November 26, 2019
jdh wrote:

When publishing the workspace to the server, there is a setting on the DataDownloadService about the writers to include in the zip. Do you have all the writers selected?

Hi jdh, Thanks for the reply. In response; Yes, i did. I have three featurewriters, and all of them are selected. Today also discovered the following topic: https://knowledge.safe.com/questions/82459/how-can-i-change-the-folder-structure-returned-in.html, and tried to see whether that could solve the issue (e.g. specifiing "\\file2.zip" or "/file2.zip" or "\\file2.zip\\" after clicking on 'More'), but unfortunately without succes.

H
Forum|alt.badge.img+2

Hi @thijsknapen,

This is a shortcoming of FME Engine and how it unable to write a zip file within a zip file (issue reference: FMEENGINE-48120). In FME Desktop if you were to change the Destination path for any zipped writer so that it was being written into another zipped file then whilst the translation would succeed when you go to view the result you'll find the bottom level file is correctly zipped it is actually being stored in a folder that just has .zip in the name rather than a second zipped file.

When you select the Data Download Service in FME Server the engine has to zip up the result in order to produce the URL, in your scenario it is attempting the same thing outlined above and so, since the top layer actually a folder rather than a file, the Data Download URL cannot be created.

Therefore in order to receive the data download link none of the writers selected to be included can be outputting to a zip file.

helmoet
Forum|alt.badge.img+8
  • helmoet
  • Best Answer
  • November 27, 2019

Hi @thijsknapen, you could try generating all output using FeatureWriters (or using FMEServerJobSubmitter with your original workspace). After that you could gather the results using a FeatureReader, choosing for Format "Directory and File Pathnames" and for Folder the directory you created the files in. After that connect a FileCopy writer that will be the only writer configured in the DataDownload service. The data download link will contain all files written by the filecopy writer, including zip files.

thijsknapen
Contributor
Forum|alt.badge.img+10
  • thijsknapenContributor
  • Author
  • Contributor
  • November 28, 2019
helmoet wrote:

Hi @thijsknapen, you could try generating all output using FeatureWriters (or using FMEServerJobSubmitter with your original workspace). After that you could gather the results using a FeatureReader, choosing for Format "Directory and File Pathnames" and for Folder the directory you created the files in. After that connect a FileCopy writer that will be the only writer configured in the DataDownload service. The data download link will contain all files written by the filecopy writer, including zip files.

Hi @helmoet, Tnx for the suggestion. I already used featurewriters in the 'inflicted workspace', so I'll definitely give it a try!

thijsknapen
Contributor
Forum|alt.badge.img+10
  • thijsknapenContributor
  • Author
  • Contributor
  • November 28, 2019
hollyatsafe wrote:

Hi @thijsknapen,

This is a shortcoming of FME Engine and how it unable to write a zip file within a zip file (issue reference: FMEENGINE-48120). In FME Desktop if you were to change the Destination path for any zipped writer so that it was being written into another zipped file then whilst the translation would succeed when you go to view the result you'll find the bottom level file is correctly zipped it is actually being stored in a folder that just has .zip in the name rather than a second zipped file.

When you select the Data Download Service in FME Server the engine has to zip up the result in order to produce the URL, in your scenario it is attempting the same thing outlined above and so, since the top layer actually a folder rather than a file, the Data Download URL cannot be created.

Therefore in order to receive the data download link none of the writers selected to be included can be outputting to a zip file.

Hi @hollyatsafe, thanks for the feedback. I was already afraid that the root issue would be in respect to nested zips. However, so far I didn't understand why that would be an issue. As i'm always also interested in the how/why of things, I greatly appreciate the given explanation. N.b. Is this issue also documented in some place, i.e. could I have known upon reading some documentation page?

thijsknapen
Contributor
Forum|alt.badge.img+10
  • thijsknapenContributor
  • Author
  • Contributor
  • November 29, 2019
helmoet wrote:

Hi @thijsknapen, you could try generating all output using FeatureWriters (or using FMEServerJobSubmitter with your original workspace). After that you could gather the results using a FeatureReader, choosing for Format "Directory and File Pathnames" and for Folder the directory you created the files in. After that connect a FileCopy writer that will be the only writer configured in the DataDownload service. The data download link will contain all files written by the filecopy writer, including zip files.

Hi @helmoet. I tried to follow your method today, but unfortunately without succes. I was able to use the fileCopy writer and have been able to verify that the files were indeed copied to the newly desired 'copyFolder' (I am able to access the FME Server over the network, and can see that the files are indeed copied to the new directory). However, selecting only the filecopy writer to be outputted in the data download, I obtain an empty zip. This also occurs when I only select e.g. only a .xlsx file to be copied to the new 'copyFolder'... Any ideas on what I'm doing wrong?

thijsknapen
Contributor
Forum|alt.badge.img+10
  • thijsknapenContributor
  • Author
  • Contributor
  • November 30, 2019
helmoet wrote:

Hi @thijsknapen, you could try generating all output using FeatureWriters (or using FMEServerJobSubmitter with your original workspace). After that you could gather the results using a FeatureReader, choosing for Format "Directory and File Pathnames" and for Folder the directory you created the files in. After that connect a FileCopy writer that will be the only writer configured in the DataDownload service. The data download link will contain all files written by the filecopy writer, including zip files.

Hi @helmoet. I managed to make the 'filecopy method' work today. The missing link was that I discovered from the following topic https://knowledge.safe.com/questions/60007/why-cant-filecopy-writer-output-be-included-in-the.html that upon using FME Server you need to use relative file path(s) for the 'filecopy_dest_dataset' attribute in order for its output to return in the data download service.

H
Forum|alt.badge.img+2
thijsknapen wrote:

Hi @hollyatsafe, thanks for the feedback. I was already afraid that the root issue would be in respect to nested zips. However, so far I didn't understand why that would be an issue. As i'm always also interested in the how/why of things, I greatly appreciate the given explanation. N.b. Is this issue also documented in some place, i.e. could I have known upon reading some documentation page?

Hi @thijsknapen,

I apologise for not responding sooner - I somehow missed this reply. I'm happy to say that a fix has gone into FME 2020 beta b20146 (safe.com/beta) to allow for nested zipping and this now means it is possible to use the Data Download Service to write out zipped datasets.

 

 

As for the documentation I'm not aware that this was recorded anywhere, we do not tend to backport issues in our documentation and since this has been fixed for 2020 there is no requirement to add this any longer. Thanks to your post this information is now available on our Community so hopefully if anyone else encounters this problem they will find this post!

- Thanks, Holly

Reply


Helpful Members This Week

Community Stats

30,937
Posts
117,247
Replies
38,739
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.