• EN
Solved

Temp vs System/Temp FME Cloud

  • 11 August 2020
  • 2 replies
  • 8 views
virtualcitymatt
Rank
Userlevel 5
Badge +28

Hi all,

 

I have a question about FME Cloud and the Temp folder.

 

I have found that the results from the DataDownload and also when data is uploaded to FME Server it goes into the System/Temp resource and not into the Temp resource.

 

Is there a specific reason for this?

I've just been looking at this article and have bumped up my temp space. But when looking and the usage on the FME Cloud console it seems it's never really used.

 

By contrast the Primary disk is getting used constantly for tasks like writing out DataDownload results and Upload results.

 

Any thoughts? Should I just reduce my temp again? @gerhardatsafe​ what are your thoughts?

 

Thanks in advance

icon

Best answer by gerhardatsafe 13 August 2020, 00:17

View original

2 replies

G
Badge

Hi @virtualcitymatt​

 Currently FME Server does not utilize the Temporary Disk on FME Cloud for the System Shared Resources. The article you might have had in mind (https://community.safe.com/s/article/fme-cloud-how-to-speed-up-your-workflows-with-the) is focusing on FME Engine processes that read & write from & to disk. The disk is attached to the FME Cloud instance so the FME Engine can read & write directly. The bigger the disk the more IOPS are available so if your FME Workspace is processing lots of data (multiple GBs) or has to store data on the Temp Disk during the translation an increased disk size will speed up your workflows.

 

Obviously the IOPS would also have an affect on uploads and downloads of files in a way, but there is a network interface between the disk and whatever you upload or download and your network connection has usually a way bigger impact on upload download performance then the IOPS of the disk, so I would assume this could be neglected.

 

The Temp Disk is also purged after every restart which could be undesirable in some scenarios as well. If you think these resources should be on the Temp Disk please add an Idea for this. I can definitely see arguments for it, e.g file large uploads to the Temp Disk via FME Server Apps, but it might be a bit unexpected for some that the Data Download results won't persist after restart which is harder than Cleanup Task .

 

I hope this helps!

virtualcitymatt
Rank
Userlevel 5
Badge +28

Hi @virtualcitymatt​

 Currently FME Server does not utilize the Temporary Disk on FME Cloud for the System Shared Resources. The article you might have had in mind (https://community.safe.com/s/article/fme-cloud-how-to-speed-up-your-workflows-with-the) is focusing on FME Engine processes that read & write from & to disk. The disk is attached to the FME Cloud instance so the FME Engine can read & write directly. The bigger the disk the more IOPS are available so if your FME Workspace is processing lots of data (multiple GBs) or has to store data on the Temp Disk during the translation an increased disk size will speed up your workflows.

 

Obviously the IOPS would also have an affect on uploads and downloads of files in a way, but there is a network interface between the disk and whatever you upload or download and your network connection has usually a way bigger impact on upload download performance then the IOPS of the disk, so I would assume this could be neglected.

 

The Temp Disk is also purged after every restart which could be undesirable in some scenarios as well. If you think these resources should be on the Temp Disk please add an Idea for this. I can definitely see arguments for it, e.g file large uploads to the Temp Disk via FME Server Apps, but it might be a bit unexpected for some that the Data Download results won't persist after restart which is harder than Cleanup Task .

 

I hope this helps!

Thanks for the detailed response - OK makes sense that it's used as temp for processing. I can definitely see the pitfalls of having uploads and data download files on a drive which gets wiped every restart. Good to know.

Reply


Community Stats

28,818
Posts
109,858
Replies
36,638
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.