Solved

FME Server 2020.2 - Filecopy used in Data Download App not using Zip file structure as set in Desktop


Badge

I recently upgraded my FME Server (to 2020.2.4) and the unwanted file hierarchy for FileCopy output datasets in Data Download Zip files have returned! I now have an additional folder (called FILECOPY_1) in the output zip from a Data Download Service on Server. I've updated the Writers in Desktop (2020.2.4), but the edit functionality (Publish --> Edit Register Services --> More) seems to do nothing for the structure of the zip file for FileCopy outputs anymore after publishing (the old way is outlined here: https://community.safe.com/s/question/0D54Q000080hM9QSAU/how-can-i-change-the-folder-structure-returned-in-the-zip-file-from-fme-servers-data-download-service).

 

I this a known bug, and is there a fix please?

 

Many thanks!

icon

Best answer by hollyatsafe 15 November 2022, 19:32

View original

12 replies

Badge +9

Hi @mlufkin​ , I'm sorry to hear you've run into issues with this after your upgrade! I did some testing using FME 2020.2.4 to see if I could replicate this issue, but it worked for me. In my testing, I used a single File Copy writer writing out one file and a second writer. It created the folder structure I wanted (I only removed the FILECOPY_1 parts in the output zip folder). These were my settings: Screen Shot 2021-04-19 at 2.45.11 PMHow are the writer(s) and Data Download output folder structure configured in your case?

Badge

Hi @mlufkin​ , I'm sorry to hear you've run into issues with this after your upgrade! I did some testing using FME 2020.2.4 to see if I could replicate this issue, but it worked for me. In my testing, I used a single File Copy writer writing out one file and a second writer. It created the folder structure I wanted (I only removed the FILECOPY_1 parts in the output zip folder). These were my settings: Screen Shot 2021-04-19 at 2.45.11 PMHow are the writer(s) and Data Download output folder structure configured in your case?

Hi there - thanks for this..

 

My settings are similar, and are shown below:

CaptureAs shown, in my zip, I'm looking for a folder in there called 'GRIDS', and a loose xlsx file (labelled here as 'Quote') - having an empty 'Path in Zip File' worked in 2019.2, but using these settings gives me 2 folders in the Zip output; GRIDS (correct) and FILECOPY_1 (containing the Quote xlsx).

 

Thanks,

 

Badge +9

Hi there - thanks for this..

 

My settings are similar, and are shown below:

CaptureAs shown, in my zip, I'm looking for a folder in there called 'GRIDS', and a loose xlsx file (labelled here as 'Quote') - having an empty 'Path in Zip File' worked in 2019.2, but using these settings gives me 2 folders in the Zip output; GRIDS (correct) and FILECOPY_1 (containing the Quote xlsx).

 

Thanks,

 

Thanks @mlufkin​! I do see the same behaviour too. If I set the path to a blank value, the file is saved to FILECOPY_1 anyway. It also looks like the Path in Zip File isn't being updated properly when re-publishing a workspace. For example, if you were to change GRIDS to something else, that new name would not be used and GRIDS would still be created in the zip folder. We'll investigate this one more on our end. Thanks for raising this issue! I'll post an update here when we know more. Sorry for this trouble!

Badge

Hi there - thanks for this..

 

My settings are similar, and are shown below:

CaptureAs shown, in my zip, I'm looking for a folder in there called 'GRIDS', and a loose xlsx file (labelled here as 'Quote') - having an empty 'Path in Zip File' worked in 2019.2, but using these settings gives me 2 folders in the Zip output; GRIDS (correct) and FILECOPY_1 (containing the Quote xlsx).

 

Thanks,

 

Thanks @lauraatsafe​  - it's probably also worth mentioning that if I update the Writers (in the top left), and THEN change those folders in the Zip path, and then publish, it does use the folder structure as defined. This is irrespective of whether I'm updating the Writers to anything new. This 'trick' doesn't work for the FileCopy Writer, but fixes the others formets you're seeing in your above post.

I have the same issue now, as we migrated to FME(R) 2021.2.5.0 (20220331 - Build 21816 - WIN64) . I was able to resolve this when publishing manually. When I have removed the "Path in Zip File" parameter and have published the workspace with FME Desktop, the generated ZIP file has no subfolders. But we usually publish the workspaces with a python script and the FME server REST api, see https://docs.safe.com/fme/html/FME_REST/apidoc/v3/index.html#!/user

When I transfer the FME files onto another system and publish them with my scripts, the Path to Zip File parameter is back. Are there any hidden parameters in the REST api I could use?

 

bye, Nicole

Badge +9

I have the same issue now, as we migrated to FME(R) 2021.2.5.0 (20220331 - Build 21816 - WIN64) . I was able to resolve this when publishing manually. When I have removed the "Path in Zip File" parameter and have published the workspace with FME Desktop, the generated ZIP file has no subfolders. But we usually publish the workspaces with a python script and the FME server REST api, see https://docs.safe.com/fme/html/FME_REST/apidoc/v3/index.html#!/user

When I transfer the FME files onto another system and publish them with my scripts, the Path to Zip File parameter is back. Are there any hidden parameters in the REST api I could use?

 

bye, Nicole

Hi Nicole,

 

It looks like you may be able to set the Zip File structure from the REST API. You can see the ZipLayout parameter in the picture below:2022-05-20_16-32-34

Hi @siennaatsafe​ 

Thanks a lot for your hint!

We have tried to update the settings with a call in Python code but it does not seem to update.

 

url = '{}/fmerest/v3/repositories/{}/items/{}/detailedservices'.format(fme_server_url, STORME_REPOSITORY_NAME, filename)

payload = "{'detailedServices': {'dataDownload': {'zipLayout': ['']}}}"

r = requests.put(url, data=payload, headers=put_headers_form)

 

We are getting a Http Status OK, but when we are doing a Get-Request there is no change in the settings.

Whats wrong with it?

 

Thanks for your help!

 

kind regards, Nicole

 

Badge +9

Hi @siennaatsafe​ 

Thanks a lot for your hint! 

We have tried to update the settings with a call in Python code but it does not seem to update.

 

url = '{}/fmerest/v3/repositories/{}/items/{}/detailedservices'.format(fme_server_url, STORME_REPOSITORY_NAME, filename)

payload = "{'detailedServices': {'dataDownload': {'zipLayout': ['']}}}"

r = requests.put(url, data=payload, headers=put_headers_form)

 

We are getting a Http Status OK, but when we are doing a Get-Request there is no change in the settings.

Whats wrong with it? 

 

Thanks for your help!

 

kind regards, Nicole

 

Hi @nsulzberger​ ,

 

I've found having this in my request body worked. I've attached a video of testing this out in the REST API documentation.

{
  "jobSubmitter": {
    "reader": "SHAPEFILE_1",
    "enabled": true
  },
  "dataDownload": {
    "reader": "SHAPEFILE_1",
    "zipLayout": {},
    "writers": [
      "GENERIC_1"
    ],
    "enabled": true
  },
  "kmlNetworkLink": {
    "enabled": false
  },
  "dataStreaming": {
    "enabled": false
  }
}

 

 

 

Hi @nsulzberger​ ,

 

I've found having this in my request body worked. I've attached a video of testing this out in the REST API documentation.

{
  "jobSubmitter": {
    "reader": "SHAPEFILE_1",
    "enabled": true
  },
  "dataDownload": {
    "reader": "SHAPEFILE_1",
    "zipLayout": {},
    "writers": [
      "GENERIC_1"
    ],
    "enabled": true
  },
  "kmlNetworkLink": {
    "enabled": false
  },
  "dataStreaming": {
    "enabled": false
  }
}

 

 

 

Hi @siennaatsafe​ , 

Thanks a lot for the help!

 

I'm working with Nicole on the project. We finally managed to update the zipLayout  with the following code: 

            url = '{}/fmerest/v3/repositories/{}/items/{}/detailedservices'.format(fme_server_url, STORME_REPOSITORY_NAME, filename)
            rGet = requests.get(url, headers=get_headers)
            detailedservices = rGet.json()
            detailedservices['dataDownload']['zipLayout']= {"TEXTLINE_1": "Response.txt", "TEXTLINE_2": "Response.txt"}
            payload = json.dumps(detailedservices)
            rPut = requests.put(url, data=payload, headers=put_headers_form)
            logging.info(filename + " zipLayout updated for TEXTLINE_1 and TEXTLINE_2")

It was tricky to figure out what the ZipLayout needed to look like. Our experience is that it is only updated if the name in the ZipLayout is the same as the Writer's name. So we specified it for each writer. I hope this helps if anyone else struggles with the same problem.

 

Best wishes, 

Shari

Badge +2

This issue has been fixed for FME 2023.0.

Badge

This issue has been fixed for FME 2023.0.

Excellent - thanks!

This issue has been fixed for FME 2023.0.

Is there no option to update the data download service in fme2022.1.3? We are in the process of migrating to this version (Desktop and Server) which I thought by now would be well tested but are running into issues like this where capability in previous versions no longer seem to work. BTW, I have found that setting the output folder for a writer to ".." removes the unwanted folders in the zip file.

Reply