Skip to main content
Solved

Why is my Local\\Temp\\FMETable folder so big?


Forum|alt.badge.img

Since upgrading to FME 2018, I've noticed my available hard drive space has decreased considerably. Poking around in my Local/Temp folder, I see there is a folder called FMETable that contains over 35GB of files.

Diving deeper, there are gigantic files in there related to the FeatureJoiner:

So my question: Is this supposed to be happening? Do I need to manually clean this stuff up on regular intervals?

Best answer by debbiatsafe

I'm pleased to announce the Purge Temporary Files option (Tools menu) will delete the FMETable directory in FME's TEMP location after a recent update. Please note that using this option while a translation is running could potentially cause the translation to fail, if it removes files that are needed.

You will find the update in our latest FME 2018.1 beta: safe.com/beta

View original
Did this help you find an answer to your question?

22 replies

erik_jan
Contributor
Forum|alt.badge.img+17
  • Contributor
  • April 18, 2018

FeatureJoiner creates a temporary database to do the joining faster.

But that should be cleaned up after a successful run of the workspace.

Please submit this case for further review to safe.com/support.


Forum|alt.badge.img
  • Author
  • April 18, 2018
erik_jan wrote:

FeatureJoiner creates a temporary database to do the joining faster.

But that should be cleaned up after a successful run of the workspace.

Please submit this case for further review to safe.com/support.

So if an unwitting FME user accidentally achieves a cartesian join in the FeatureJoiner, does this mean the entire hard drive could get filled up? The translation would probably fail as well, so FME wouldn't be able to clean up after itself...?

 

 


erik_jan
Contributor
Forum|alt.badge.img+17
  • Contributor
  • April 18, 2018
erik_jan wrote:

FeatureJoiner creates a temporary database to do the joining faster.

But that should be cleaned up after a successful run of the workspace.

Please submit this case for further review to safe.com/support.

I had to do a test:

 

Two data sources,a FeatureJoiner and Inspectors.

 

I see the file appearing in Temp/FMETable, but it is removed after the successful run.

 


david_r
Celebrity
  • April 18, 2018

Further to what @erik_jan says, could it be that you've had a lot of crashes or failed runs lately, that prevented FME from cleaning up after itself?

Another thing to try, is to execute Tools -> Purge temporary files in FME Workbench and see if the temporary files disappear.


erik_jan
Contributor
Forum|alt.badge.img+17
  • Contributor
  • April 18, 2018
erik_jan wrote:

FeatureJoiner creates a temporary database to do the joining faster.

But that should be cleaned up after a successful run of the workspace.

Please submit this case for further review to safe.com/support.

Even if the workspace fails, the file is cleaned up.

 

But if the FME executable crashes (could be because it is running out of disk space) FME can not clean up the temporary files.

 

As @david_r mentions, the FME Workbench menu has an option to purge temporary files.

 


Forum|alt.badge.img
  • Author
  • April 18, 2018
david_r wrote:

Further to what @erik_jan says, could it be that you've had a lot of crashes or failed runs lately, that prevented FME from cleaning up after itself?

Another thing to try, is to execute Tools -> Purge temporary files in FME Workbench and see if the temporary files disappear.

I've had failed runs, yes. I don't recall any hard crashes. I did try the 'Purge temporary files' functionality in workbench, but it did not clean out any of these files.

 

 


david_r
Celebrity
  • April 18, 2018
jt wrote:
I've had failed runs, yes. I don't recall any hard crashes. I did try the 'Purge temporary files' functionality in workbench, but it did not clean out any of these files.

 

 

In that case I agree that you should submit this as a support case.

Forum|alt.badge.img
  • Author
  • April 18, 2018

Thank you both @erik_jan and @david_r. I have submitted a support case as suggested.


Forum|alt.badge.img
  • Author
  • April 18, 2018
david_r wrote:

Further to what @erik_jan says, could it be that you've had a lot of crashes or failed runs lately, that prevented FME from cleaning up after itself?

Another thing to try, is to execute Tools -> Purge temporary files in FME Workbench and see if the temporary files disappear.

It seems to leave the files behind if I "Stop" the translation while the FeatureJoiner is churning.

 


david_r
Celebrity
  • April 18, 2018
jt wrote:
It seems to leave the files behind if I "Stop" the translation while the FeatureJoiner is churning.

 

I'm guessing you just found a glitch in the matrix. I'm sure Safe would like to know about this.

erik_jan
Contributor
Forum|alt.badge.img+17
  • Contributor
  • April 18, 2018
jt wrote:
It seems to leave the files behind if I "Stop" the translation while the FeatureJoiner is churning.

 

Yes, I can reproduce that.

 

The files remain in the folder when you stop during the execution of the FeatureJoiner.

 

 


ev_robin
Contributor
Forum|alt.badge.img+11
  • Contributor
  • May 3, 2018

I found the exact same issue. But I was able to get the file up to 729 GB! I am in the process of submitting another case.


debbiatsafe
Safer
Forum|alt.badge.img+20
  • Safer
  • Best Answer
  • July 12, 2018

I'm pleased to announce the Purge Temporary Files option (Tools menu) will delete the FMETable directory in FME's TEMP location after a recent update. Please note that using this option while a translation is running could potentially cause the translation to fail, if it removes files that are needed.

You will find the update in our latest FME 2018.1 beta: safe.com/beta


Forum|alt.badge.img+1
david_r wrote:

Further to what @erik_jan says, could it be that you've had a lot of crashes or failed runs lately, that prevented FME from cleaning up after itself?

Another thing to try, is to execute Tools -> Purge temporary files in FME Workbench and see if the temporary files disappear.

I have this same problem. I run a workspace that uses the FeatureJoiner. I'm reading in millions of features and the file size in FMETable folder becomes way too large and causes storage problems. I'll have to ditch the FeatureJoiner and use the FeatureMerger or something similar. This suggests that the FeatureJoiner should only be used on relatively small datasets.


david_r
Celebrity
  • February 20, 2019
spatial_aus wrote:

I have this same problem. I run a workspace that uses the FeatureJoiner. I'm reading in millions of features and the file size in FMETable folder becomes way too large and causes storage problems. I'll have to ditch the FeatureJoiner and use the FeatureMerger or something similar. This suggests that the FeatureJoiner should only be used on relatively small datasets.

I've unfortunately observed the same thing. Not sure why, because the FeatureJoiner should, in theory, be more efficient than the FeatureMerger.


xiaomengatsafe
Safer
Forum|alt.badge.img+3
david_r wrote:

I've unfortunately observed the same thing. Not sure why, because the FeatureJoiner should, in theory, be more efficient than the FeatureMerger.

Hi @spatial_aus and @david_r, I've got some potential explanation for the observed behaviour:

David is correct that FeatureJoiner should be more efficient, but there is a caveat. It only works with features in "Bulk Mode" (some people may know Bulk Mode as our previous internal terminology, "Feature Table").

So, if the incoming features are not in Bulk Mode, FeatureJoiner will cache the incoming features to disk in Bulk Mode. And this is likely the reason for the large Temp file. We are considering creating an article to explain this in more detail.

I should also note, we are working to add Bulk Mode support for most of our transformers and formats. So these issues will hopefully become less common.

However, there is one concerning detail we want to clarify. Temp files created during translation should be cleaned up after the translation stops, even with a translation failure. The only exception would be if FME crashed, then the temp files might be left behind.

So, I'm curious, in your scenarios, did the workbench crash? If not, were the temp files cleaned up afterwards? If there was no crash, and the temp files were left behind after translation stopped, we would like to investigate why.

 


david_r
Celebrity
  • March 2, 2019
xiaomengatsafe wrote:

Hi @spatial_aus and @david_r, I've got some potential explanation for the observed behaviour:

David is correct that FeatureJoiner should be more efficient, but there is a caveat. It only works with features in "Bulk Mode" (some people may know Bulk Mode as our previous internal terminology, "Feature Table").

So, if the incoming features are not in Bulk Mode, FeatureJoiner will cache the incoming features to disk in Bulk Mode. And this is likely the reason for the large Temp file. We are considering creating an article to explain this in more detail.

I should also note, we are working to add Bulk Mode support for most of our transformers and formats. So these issues will hopefully become less common.

However, there is one concerning detail we want to clarify. Temp files created during translation should be cleaned up after the translation stops, even with a translation failure. The only exception would be if FME crashed, then the temp files might be left behind.

So, I'm curious, in your scenarios, did the workbench crash? If not, were the temp files cleaned up afterwards? If there was no crash, and the temp files were left behind after translation stopped, we would like to investigate why.

 

Thanks for the info, that could well explain my observations. It would be fantastic, however, if the FME could warn you (log) that you're using the FeatureJoiner with non-bulk mode features, since that seems to have a big impact on performance and disk usage. If you unwittingly end up using the FeatureJoiner with the "wrong" type of features I have the impression that it could much faster to simply use the FeatureMerger -- is that correct?

Unfortunately I've stopped using the FeatureJoiner entirly because of this, so I cannot really remember the details, but I believe at some point I had several GB of temp files coming from the FeatureJoiner, dating back. This was just after the FeatureJoiner was introduced, so it may well have been improved since then.

 


xiaomengatsafe
Safer
Forum|alt.badge.img+3
david_r wrote:

Thanks for the info, that could well explain my observations. It would be fantastic, however, if the FME could warn you (log) that you're using the FeatureJoiner with non-bulk mode features, since that seems to have a big impact on performance and disk usage. If you unwittingly end up using the FeatureJoiner with the "wrong" type of features I have the impression that it could much faster to simply use the FeatureMerger -- is that correct?

Unfortunately I've stopped using the FeatureJoiner entirly because of this, so I cannot really remember the details, but I believe at some point I had several GB of temp files coming from the FeatureJoiner, dating back. This was just after the FeatureJoiner was introduced, so it may well have been improved since then.

 

Hi @david_r, good idea about logging something to inform users, when features are getting cached. I'll bring that suggestion to the dev team.

Regarding processing speed, in theory, we actually expect caching in bulk mode using FeatureJoiner to be faster than using FeatureMerger. So, we would be really interested in seeing your example, if it demonstrates otherwise.

Would you be willing to share your data and workspace with us through a support case? (if the sample data is too large, you can upload them to our ftp). Thank you very much.


srg
Forum|alt.badge.img+10
  • March 6, 2019
debbiatsafe wrote:

I'm pleased to announce the Purge Temporary Files option (Tools menu) will delete the FMETable directory in FME's TEMP location after a recent update. Please note that using this option while a translation is running could potentially cause the translation to fail, if it removes files that are needed.

You will find the update in our latest FME 2018.1 beta: safe.com/beta

@DebbiAtSafe Is this accidental deletion being fixed in FME 2019?


debbiatsafe
Safer
Forum|alt.badge.img+20
srg wrote:

@DebbiAtSafe Is this accidental deletion being fixed in FME 2019?

Hi @srg

The fix for Purge Temporary Files under Tools menu not deleting .fts files is also in FME 2019. The current behaviour for Purge Temporary Tools is it will delete temporary files left in the designated temp folder. These files are usually left behind if FME does not shutdown 'cleanly' (more information on this here).

I'm not certain I understand what 'accidental deletion' refers to in your question. Could you expand on what you mean?


srg
Forum|alt.badge.img+10
  • March 7, 2019
debbiatsafe wrote:

Hi @srg

The fix for Purge Temporary Files under Tools menu not deleting .fts files is also in FME 2019. The current behaviour for Purge Temporary Tools is it will delete temporary files left in the designated temp folder. These files are usually left behind if FME does not shutdown 'cleanly' (more information on this here).

I'm not certain I understand what 'accidental deletion' refers to in your question. Could you expand on what you mean?

@DebbiAtSafe my meaning of accidental deletion is deletion of temporary files created by a FME process which is still running.

From your response, I understand that it is fixed in FME 2019.

 


debbiatsafe
Safer
Forum|alt.badge.img+20
srg wrote:

@DebbiAtSafe my meaning of accidental deletion is deletion of temporary files created by a FME process which is still running.

From your response, I understand that it is fixed in FME 2019.

 

@srg The fix mentioned was not meant to address accidental deletion of temporary files created by an FME process which is still running. Instead, it was meant to address the incorrect behaviour where .fts files were not deleted when the user selects Purge Temporary Files.

We do not recommend using Purge Temporary Files when a translation is running to avoid potential translation failures.


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings