Skip to main content
  • EN

I was wondering about the FME Flow Temp Resource (i.e. '$(FME_SHAREDRESOURCE_TEMP)').

As the name suggests its supposed to be a temporary folder. 

 

First question I have on this,

  • Is it correct that there is no default mechanism (e.g. that there is no default cleanup task) that cleans out this Temp (FME_SHAREDRESOURCE_TEMP) folder, and that if you want to have it be cleaned every X days/weeks, you should create a new System Cleanup Task?

 

It seemed to me that this was the case, and while setting up my own System Cleanup Task to have this Temp (FME_SHAREDRESOURCE_TEMP) folder cleared, I came across some confusing points I would like to address/ask.

 

My second question;

  • How does the 'Remove Files Older Than' parameter set (at tab 'Tasks' of 'System Cleanup') relate to the 'Task Interval' parameter for 'Normal Conditions' (that can be configured at the 'Configuration' tab of 'System Cleanup')?

 

E.g., if I set up a task to clean up the 'Temp Resource' folder once every hour, but the 'Task Interval' for 'Normal Conditions' is set to 1 Day (default), will files in the 'Temp Resource' folder be cleanup up once every hour or once every day?

It seems to me that the latter is the case, and that there is no real point in setting the 'Remove Files Older Than' parameter of a System Cleanup Task, to be more frequent than the 'Task Interval' parameter for 'Normal Conditions'.

If so, I would have appreciated a warning or information window upon configuring a 'Remove Files Older Than' frequency < 'Task Interval' for 'Normal Conditions'. But that may be more something for an Idea on the forum (but before I do so, I would first like to hear from others if they believe my assumption on this second point is correct).

 

Then, a third point of attention.

  • The 'information buttons' in front of the 'Filter Settings' parameters of a Cleanup Task (in front of the parameters 'Filter Type', 'Pattern' and 'Remove Files Older Than') only seem to describe the cleanup of files in the configured source resource (sub)folder.

 

However, it seems to me that also folders and nested files/folders are removed. I'm pretty sure that I noticed this was the case. Although I'm happy with the behavior, I suppose the information window could be a bit more clear on this. I'll consider to also post an Idea about this.

 

Then my last question for now;

  • For the path of resource folder in a Cleanup task (i.e. the parameter 'Source Folder'), does it matter if there is an additional forward slash between the 'system resource parameter' and the subfolder? E.g. can/should I use '$FME_SHAREDRESOURCE_LOG)/myfolder' or can/should I use '$FME_SHAREDRESOURCE_LOG)myfolder'.

 

This last item/question is because I was a bit confused that when I set the new Cleanup Task, and set the path of the resource folder through the FME Flow interface, the path that was set was '$FME_SHAREDRESOURCE_LOG)/myfolder', whereas all the default Cleanup Tasks that are shipped with FME Flow, don't have such a forward slash between the parameter and the subfolder (e.g. the path for the default cleanup task 'Delete_Dashboard_Temp_Files' is '$(FME_SHAREDRESOURCE_DASHBOARD)temp').

 

Feedback regarding to these points would be appreciated 🙂

Hi thijsknapen in Server/Flow under System Cleaning up FME Server temp files (safe.com) you have the default settings and in the aggressive tasks have the rules to delete/keep certain files. I find the cleanup tasks as a %

of remaining disk space, to be the most suitable/scalable.

 

In terms of forward/backslash this is no doubt a linux vs windows bug, where they forgot to update the default for windows. Generally double slash should be resolved by FME, otherwise a string replacer prior to using the Server parameter can be added to the canvas to replace '\\\\' with '\\'

Hi thijsknapen in Server/Flow under System Cleaning up FME Server temp files (safe.com) you have the default settings and in the aggressive tasks have the rules to delete/keep certain files. I find the cleanup tasks as a %

of remaining disk space, to be the most suitable/scalable.

 

In terms of forward/backslash this is no doubt a linux vs windows bug, where they forgot to update the default for windows. Generally double slash should be resolved by FME, otherwise a string replacer prior to using the Server parameter can be added to the canvas to replace '\\\\' with '\\'

Hi @jamesb16otterill​ , Thanks for your reply. I unfortunately don't really see how this is an answer/response to the points I addressed.

 

It seems you are referring to the settings about low disk conditions. My question was not about this. One of my points was about a bit of a confusion on an interval/frequency that you can configure for Cleanup Tasks.

On the one hand, for Cleanup Tasks (in the 'Tasks' tab), you have a 'Remove Files older than' parameter (max age),

whereas on the other hand, in the 'Configuration' tab you have a 'Task interval' parameter (for the Cleanup under Normal conditions (and thus not under Low Disk conditions, where the parameter you mention play a role)). It seems to me that there is no point in having a 'Remove Files older than' that is set to be more frequent than the 'Task interval' for normal conditions, and I suggested that that could use some explanation to prevent confusion to users as myself.

 

Regarding the second point. I didn't mention a forward/backslash situation. I was wondering if it matters whether the source path of a cleanup task equals/starts with '$FME_SHAREDRESOURCE_TEMP)/myfolder' or whether this should be '$FME_SHAREDRESOURCE_TEMP)myfolder'

in FME flow, there are many ways to clean up files. In 2023.0 there are 14 tasks that come with the install. When creating a new cleanup task, if the filter settings are sufficient or need improving, then perhaps create a New FME idea and get it voted up. Thats the best way to get it added to the product. NewCleanupTask_FMETEMP_30days oldThe cleanup Configuration tab runs all those 14+ tasks under these conditions

CleanUpConfig

To your second point, go to resources and navigate to the folder. Select/Tick it and then under Actions > Properties you'll see the system path fileproperties

in FME flow, there are many ways to clean up files. In 2023.0 there are 14 tasks that come with the install. When creating a new cleanup task, if the filter settings are sufficient or need improving, then perhaps create a New FME idea and get it voted up. Thats the best way to get it added to the product. NewCleanupTask_FMETEMP_30days oldThe cleanup Configuration tab runs all those 14+ tasks under these conditions

CleanUpConfig

To your second point, go to resources and navigate to the folder. Select/Tick it and then under Actions > Properties you'll see the system path fileproperties

Hi @jamesb16otterill​, thanks for the feedback. However, again I feel like you're not responding to the points I addressed. It seems you are more or less talking about the same topic and showing the interface of FME Flow regarding System Cleanup.

 

If the points/items in my question are unclear to you, feel free to ask and maybe I can make them more clear. E.g. I could share some similar screenshots like you posted of how it looks to me in the FME Flow GUI (and explain it more visually so to say).

 

In the mean time I'll hope that someone else can respond that is willing to address the points I shared.

 

Update; I edited my question to hopefully make the points a bit more clear.

I looked into some of these points and maybe I can provide a little more info.

 

As far as I can tell by default the FME_SHAREDRESOURCE_TEMP isn't cleaned up by FME Server at all. My understanding for is that the idea of this folder is that an administrator can point it to a location on a network fileshare or disk which is managed externally, however, I have no reference for this. For example the FME_SHAREDRESOURCE_TEMP on FME Cloud is sitting on a disk which gets cleaned up every time the instance gets paused. https://community.safe.com/s/article/fme-cloud-how-to-speed-up-your-workflows-with-the

 

Another difference is that file in the FME_SHAREDRESOURCE_TEMP do not get backed up when performing a backup task.

 

For the second point I think having a daily cleanup but not deleting Items older than an hour is still somewhat valid. In this case you might have a folder which you want to keep pretty empty but maybe there are data in there which are used for a job which takes several minutes to process. This cleanup setting would make sure those data were not deleted during process. It's a pretty edge case though.

 

Mostly though I think the use case is for when you want to keep logs going back a month or two weeks or so and you just don't want them deleted every day. I like your idea of a warning.

 

For the point about the filter on the delete I think it could be a little clearer too.

 

your last point - good finding. seems like it should be consistent but I suspect it doesn't matter otherwise it most likely would already be fixed. I've got one set up with two // so I can report back if it doesn't get deleted.

 

 

 

I looked into some of these points and maybe I can provide a little more info.

 

As far as I can tell by default the FME_SHAREDRESOURCE_TEMP isn't cleaned up by FME Server at all. My understanding for is that the idea of this folder is that an administrator can point it to a location on a network fileshare or disk which is managed externally, however, I have no reference for this. For example the FME_SHAREDRESOURCE_TEMP on FME Cloud is sitting on a disk which gets cleaned up every time the instance gets paused. https://community.safe.com/s/article/fme-cloud-how-to-speed-up-your-workflows-with-the

 

Another difference is that file in the FME_SHAREDRESOURCE_TEMP do not get backed up when performing a backup task.

 

For the second point I think having a daily cleanup but not deleting Items older than an hour is still somewhat valid. In this case you might have a folder which you want to keep pretty empty but maybe there are data in there which are used for a job which takes several minutes to process. This cleanup setting would make sure those data were not deleted during process. It's a pretty edge case though.

 

Mostly though I think the use case is for when you want to keep logs going back a month or two weeks or so and you just don't want them deleted every day. I like your idea of a warning.

 

For the point about the filter on the delete I think it could be a little clearer too.

 

your last point - good finding. seems like it should be consistent but I suspect it doesn't matter otherwise it most likely would already be fixed. I've got one set up with two // so I can report back if it doesn't get deleted.

 

 

 

Thanks @virtualcitymatt​ ! I'd like to add to this that the cleanup of the temp folder on FME Flow Hosted (FME Cloud) only occurs on an instance pause, not on a reboot.

I looked into some of these points and maybe I can provide a little more info.

 

As far as I can tell by default the FME_SHAREDRESOURCE_TEMP isn't cleaned up by FME Server at all. My understanding for is that the idea of this folder is that an administrator can point it to a location on a network fileshare or disk which is managed externally, however, I have no reference for this. For example the FME_SHAREDRESOURCE_TEMP on FME Cloud is sitting on a disk which gets cleaned up every time the instance gets paused. https://community.safe.com/s/article/fme-cloud-how-to-speed-up-your-workflows-with-the

 

Another difference is that file in the FME_SHAREDRESOURCE_TEMP do not get backed up when performing a backup task.

 

For the second point I think having a daily cleanup but not deleting Items older than an hour is still somewhat valid. In this case you might have a folder which you want to keep pretty empty but maybe there are data in there which are used for a job which takes several minutes to process. This cleanup setting would make sure those data were not deleted during process. It's a pretty edge case though.

 

Mostly though I think the use case is for when you want to keep logs going back a month or two weeks or so and you just don't want them deleted every day. I like your idea of a warning.

 

For the point about the filter on the delete I think it could be a little clearer too.

 

your last point - good finding. seems like it should be consistent but I suspect it doesn't matter otherwise it most likely would already be fixed. I've got one set up with two // so I can report back if it doesn't get deleted.

 

 

 

Hi @virtualcitymatt​ .

Thanks, this is the type of response I was hoping for! As a very brief summary it looks to me that in general you are confirming most of my suspicions. The bit of background info (e.g. on FME Flow Hosted) and the nice usecase for the (niche) usecase to the second point is much appreciated!

It took some time, but I now created 2 ideas;

This one to address item 2: Add info icon or warning pop up for cleanup task with 'Remove Files Older Than' more frequent than interval of cleanup executions | Community (safe.com)

And this one to address item 3: Improve explanation at 'Info button' at the 'Filter Settings' of 'System Cleanup Tasks' | Community (safe.com)

Reply


Community Stats

30,354
Posts
115,364
Replies
38,091
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.