• EN
Question

Download URLfrom server app

totobobiasos
Badge

So, I've created an server app that sometimes posts very large datasets and have so far been able to restrict access to the data by just emailing the dataset back as an attachment (user email as parameter)

(its not super safe, but safe enough - the data isn't super sensitive but sensitive enough).

 

I've thus been able to restrict the data to the person running the server app by letting the person enter their email and get the data back that way. But now the datasets is growing kinda fast and the email client isn't putting up with 20-50 megabytes of attachments. So I read that it was possible to get the server app to respond with the download link (again not super safe, but safe enough).

 

So: how do I do that? What parameter carries the data download link from server apps? Or, it doesn't work that way? (in wich case I'd have to do a workaround, and nobody likes workarounds :-)

 

Been searching and trying for several hours without luck. Tried the ServerNotifier and the emailer and both are sending emails on completion, but not with the download link...

 

 

FME server 2019.2 and FME workbench 2019.0.0.1

13 replies

itay
Rank
Badge +16

Hi, basically you need to register the ws of the app with the datadoenload service when publishing to server.

Thats how easy it is...

Hope this helps.

takashi
Rank
Userlevel 3
Badge +17

Agree with Itay.

If you run the workspace as a Data Download service, FME Server automatically creates a download zip file, writes the destination files into the zip file, and also provides the full path of the zip file to the workspace through FME_SERVER_DEST_DIR parameter.

In the workspace, you can extract the zip file name from the parameter value (full path), then build the download URL like this.

{FME Server Web URL}/fmedatadownload/results/{zip file name extracted from FME_SERVER_DEST_DIR}
totobobiasos
Badge

Agree with Itay.

If you run the workspace as a Data Download service, FME Server automatically creates a download zip file, writes the destination files into the zip file, and also provides the full path of the zip file to the workspace through FME_SERVER_DEST_DIR parameter.

In the workspace, you can extract the zip file name from the parameter value (full path), then build the download URL like this.

{FME Server Web URL}/fmedatadownload/results/{zip file name extracted from FME_SERVER_DEST_DIR}

Hmm, seems easy enough

Unfortunately I get an error message: Undefined macro `FME_SERVER_DEST_DIR' dereferenced in file .....fmw

 

Do I have to set that parameter myself in the workspace or is that a system parameter set in the server? 

 

Just to be clear: the feature writers are set to write to c:\temp (because testing the workspace in FME workbench), but that is not the FME_SERVER_DEST_DIR?

totobobiasos
Badge

Hi, basically you need to register the ws of the app with the datadoenload service when publishing to server.

Thats how easy it is...

Hope this helps.

ws = writers, right?

Tha'ts what I did and... nada.

 

Seems like I'm missing something fundamental

takashi
Rank
Userlevel 3
Badge +17

Agree with Itay.

If you run the workspace as a Data Download service, FME Server automatically creates a download zip file, writes the destination files into the zip file, and also provides the full path of the zip file to the workspace through FME_SERVER_DEST_DIR parameter.

In the workspace, you can extract the zip file name from the parameter value (full path), then build the download URL like this.

{FME Server Web URL}/fmedatadownload/results/{zip file name extracted from FME_SERVER_DEST_DIR}

As far as I know, the FME_SERVER_DEST_DIR will be created only if the workspace is run as a Data Download Service with FME Server. In other conditions, referring the parameter causes the error.

totobobiasos
Badge

Ah! Darn.

I had the APP registered as jobsubmitter. Changed that to downloadservice. Now I recieve the email and the download link extracted form FME_SERVER_DEST_DIR

I forgot to mention that I do not want the download link to post on the resulting fme server page after the app is done. I ONLY want the email - because I want to restrict access to running our FME server (we're a municipality with thosands of users - but in my organization just about 100 are allowed to run the app).

 

If I don't include the feature writer in the download no zip is created, even though i get the download parameter. Or am I missing something? I'm currently in the middle of deploying a web map after 18 months of developing so a bit stressed out...

 

In the start of the workspace email is checked if it contains "fastighet" (swedish for property) - as these users are allowed to run the server - and the server should then email the download link to that email. Else just email "sorry, you're not allowed to run this".

But anyone could trick the server by just using say my email and then use our hardware making our server slow - and still get the result through the download page. It is not possible to lock the whole server down because it contains apps that are allowed to run on the whole network as well.

@takashi
takashi
Rank
Userlevel 3
Badge +17

Note: Since the followings are not documented officially, I cannot guaranteed that this works fine in your system configuration and in the future. It's a kind of hack. If you need a guaranteed answer, please contact Safe.

 

In my observation,

{FME Server Web URL}/fmedatadownload/results

points this resource directory in the FME Server system,

Resources > System > temp > engineresults

and you can refer the resource directory with this path string in a workspace

$(FME_SHAREDRESOURCE_SYSTEM)temp/engineresults

 

That is, for example, if you save a file into this location as a result of a JobSubmitter Service,

$(FME_SHAREDRESOURCE_SYSTEM)temp/engineresults/the-unique-filename.ext

you can download the file from this URL, without showing download link on the web app interface.

{FME Server Web URL}/fmedatadownload/results/the-unique-filename.ext

 

Be aware that the filename should be unique.

totobobiasos
Badge

Note: Since the followings are not documented officially, I cannot guaranteed that this works fine in your system configuration and in the future. It's a kind of hack. If you need a guaranteed answer, please contact Safe.

 

In my observation,

{FME Server Web URL}/fmedatadownload/results

points this resource directory in the FME Server system,

Resources > System > temp > engineresults

and you can refer the resource directory with this path string in a workspace

$(FME_SHAREDRESOURCE_SYSTEM)temp/engineresults

 

That is, for example, if you save a file into this location as a result of a JobSubmitter Service,

$(FME_SHAREDRESOURCE_SYSTEM)temp/engineresults/the-unique-filename.ext

you can download the file from this URL, without showing download link on the web app interface.

{FME Server Web URL}/fmedatadownload/results/the-unique-filename.ext

 

Be aware that the filename should be unique.

Ah, I like hacks :-)

Now I have to figure out how to get multiple writers to post in a single zip using the above solution...

 

I thank you for the answer.

takashi
Rank
Userlevel 3
Badge +17

Note: Since the followings are not documented officially, I cannot guaranteed that this works fine in your system configuration and in the future. It's a kind of hack. If you need a guaranteed answer, please contact Safe.

 

In my observation,

{FME Server Web URL}/fmedatadownload/results

points this resource directory in the FME Server system,

Resources > System > temp > engineresults

and you can refer the resource directory with this path string in a workspace

$(FME_SHAREDRESOURCE_SYSTEM)temp/engineresults

 

That is, for example, if you save a file into this location as a result of a JobSubmitter Service,

$(FME_SHAREDRESOURCE_SYSTEM)temp/engineresults/the-unique-filename.ext

you can download the file from this URL, without showing download link on the web app interface.

{FME Server Web URL}/fmedatadownload/results/the-unique-filename.ext

 

Be aware that the filename should be unique.

This screenshot illustrates a possible way to archive destination datasets created by multiple FeatureWriter into a single zip file which can be downloaded.

0684Q00000ArMb2QAF.png

totobobiasos
Badge

This screenshot illustrates a possible way to archive destination datasets created by multiple FeatureWriter into a single zip file which can be downloaded.

I think you just saved me like 2 days of work.

 

Thanks again!

mark2atsafe
Rank
Userlevel 4
Badge +25

Hi @totobobiasos

I found this really interesting, so covered this as my question-of-the-week here: https://knowledge.safe.com/questions/112189/question-of-the-week-server-data-urls.html

Overall, it's not that Takashi's solution wouldn't work, it's more that I think you're looking at it the wrong way. If you could restrict app access, then you wouldn't need to worry about whether users could see the download link.

Essentially you can do that in FME 2020 by requiring authentication in an app. Of course, I notice you're still on 2019. So that post outlines a few other ways of doing what you need, in terms of providing a URL.

The general thought at Safe is that using S3 buckets is the best solution. It really depends on the degree of security you need. But the Data Download results folder is open to all users (it has to be) so it's not as secure as writing to a different service like S3, or a resources folder on Server itself (where you can control the access a bit better).

Anyway, if you've got a working solution you may be happy and I don't blame you for keeping what works. But in the longer term you might find some of the other info on there gives you better options.

Hope this helps.

totobobiasos
Badge

Hi @totobobiasos

I found this really interesting, so covered this as my question-of-the-week here: https://knowledge.safe.com/questions/112189/question-of-the-week-server-data-urls.html

Overall, it's not that Takashi's solution wouldn't work, it's more that I think you're looking at it the wrong way. If you could restrict app access, then you wouldn't need to worry about whether users could see the download link.

Essentially you can do that in FME 2020 by requiring authentication in an app. Of course, I notice you're still on 2019. So that post outlines a few other ways of doing what you need, in terms of providing a URL.

The general thought at Safe is that using S3 buckets is the best solution. It really depends on the degree of security you need. But the Data Download results folder is open to all users (it has to be) so it's not as secure as writing to a different service like S3, or a resources folder on Server itself (where you can control the access a bit better).

Anyway, if you've got a working solution you may be happy and I don't blame you for keeping what works. But in the longer term you might find some of the other info on there gives you better options.

Hope this helps.

Cool!

The app-functionality is truly great. We're using it right and left for a lot of tasks, varying from sending well formatted email to support to building a dwg viewer workspace with background maps in QGIS.

For us the need to restrict the app is as much an issue as to restrict access to the result data, because we only have one engine to work with. And allowing many users would keep the server busy when we'd need it ourselves. The data isn't crucial or extremely sensitive, but enough so that if it were continuosly distributed freely it could indicate something that potentially could be VERY costsome.

If the data was very sensitive I'd have to do a completely different solution (maybe install workbench locally at the user).

Using anything else than our own storage servers is, though, strictly prohibited by the IT department due to unclear classification of data. (that is what S3 is, right?)

 

So my hope is that this somewhat hacky solution works out, because it is what I was looking for... :-)

mark2atsafe
Rank
Userlevel 4
Badge +25
Cool!

The app-functionality is truly great. We're using it right and left for a lot of tasks, varying from sending well formatted email to support to building a dwg viewer workspace with background maps in QGIS.

For us the need to restrict the app is as much an issue as to restrict access to the result data, because we only have one engine to work with. And allowing many users would keep the server busy when we'd need it ourselves. The data isn't crucial or extremely sensitive, but enough so that if it were continuosly distributed freely it could indicate something that potentially could be VERY costsome.

If the data was very sensitive I'd have to do a completely different solution (maybe install workbench locally at the user).

Using anything else than our own storage servers is, though, strictly prohibited by the IT department due to unclear classification of data. (that is what S3 is, right?)

 

So my hope is that this somewhat hacky solution works out, because it is what I was looking for... :-)

Yes, S3 is off-premises, so that probably wouldn't work for you. But your hacky solution sounds OK, so let's see how it goes. Let us know if you have problems.

Reply


Community Stats

28,799
Posts
109,770
Replies
36,617
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.