+3Since upgrading to FME 2018, I've noticed my available hard drive space has decreased considerably. Poking around in my Local/Temp folder, I see there is a folder called FMETable that contains over 35GB of files.
Diving deeper, there are gigantic files in there related to the FeatureJoiner:
So my question: Is this supposed to be happening? Do I need to manually clean this stuff up on regular intervals?
Best answer by debbiatsafe
I'm pleased to announce the Purge Temporary Files option (Tools menu) will delete the FMETable directory in FME's TEMP location after a recent update. Please note that using this option while a translation is running could potentially cause the translation to fail, if it removes files that are needed.
You will find the update in our latest FME 2018.1 beta: safe.com/beta
+22FeatureJoiner creates a temporary database to do the joining faster.
But that should be cleaned up after a successful run of the workspace.
Please submit this case for further review to safe.com/support.
+3FeatureJoiner creates a temporary database to do the joining faster.
But that should be cleaned up after a successful run of the workspace.
Please submit this case for further review to safe.com/support.
+22FeatureJoiner creates a temporary database to do the joining faster.
But that should be cleaned up after a successful run of the workspace.
Please submit this case for further review to safe.com/support.
Two data sources,a FeatureJoiner and Inspectors.
I see the file appearing in Temp/FMETable, but it is removed after the successful run.
Further to what @erik_jan says, could it be that you've had a lot of crashes or failed runs lately, that prevented FME from cleaning up after itself?
Another thing to try, is to execute Tools -> Purge temporary files in FME Workbench and see if the temporary files disappear.
+22FeatureJoiner creates a temporary database to do the joining faster.
But that should be cleaned up after a successful run of the workspace.
Please submit this case for further review to safe.com/support.
But if the FME executable crashes (could be because it is running out of disk space) FME can not clean up the temporary files.
As @david_r mentions, the FME Workbench menu has an option to purge temporary files.
+3Further to what @erik_jan says, could it be that you've had a lot of crashes or failed runs lately, that prevented FME from cleaning up after itself?
Another thing to try, is to execute Tools -> Purge temporary files in FME Workbench and see if the temporary files disappear.
+3Thank you both @erik_jan and @david_r. I have submitted a support case as suggested.
+3Further to what @erik_jan says, could it be that you've had a lot of crashes or failed runs lately, that prevented FME from cleaning up after itself?
Another thing to try, is to execute Tools -> Purge temporary files in FME Workbench and see if the temporary files disappear.
+22
The files remain in the folder when you stop during the execution of the FeatureJoiner.
+12I found the exact same issue. But I was able to get the file up to 729 GB! I am in the process of submitting another case.
+21I'm pleased to announce the Purge Temporary Files option (Tools menu) will delete the FMETable directory in FME's TEMP location after a recent update. Please note that using this option while a translation is running could potentially cause the translation to fail, if it removes files that are needed.
You will find the update in our latest FME 2018.1 beta: safe.com/beta
+1Further to what @erik_jan says, could it be that you've had a lot of crashes or failed runs lately, that prevented FME from cleaning up after itself?
Another thing to try, is to execute Tools -> Purge temporary files in FME Workbench and see if the temporary files disappear.
I have this same problem. I run a workspace that uses the FeatureJoiner. I'm reading in millions of features and the file size in FMETable folder becomes way too large and causes storage problems. I'll have to ditch the FeatureJoiner and use the FeatureMerger or something similar. This suggests that the FeatureJoiner should only be used on relatively small datasets.
I have this same problem. I run a workspace that uses the FeatureJoiner. I'm reading in millions of features and the file size in FMETable folder becomes way too large and causes storage problems. I'll have to ditch the FeatureJoiner and use the FeatureMerger or something similar. This suggests that the FeatureJoiner should only be used on relatively small datasets.
I've unfortunately observed the same thing. Not sure why, because the FeatureJoiner should, in theory, be more efficient than the FeatureMerger.
+11I've unfortunately observed the same thing. Not sure why, because the FeatureJoiner should, in theory, be more efficient than the FeatureMerger.
Hi @spatial_aus and @david_r, I've got some potential explanation for the observed behaviour:
David is correct that FeatureJoiner should be more efficient, but there is a caveat. It only works with features in "Bulk Mode" (some people may know Bulk Mode as our previous internal terminology, "Feature Table").
So, if the incoming features are not in Bulk Mode, FeatureJoiner will cache the incoming features to disk in Bulk Mode. And this is likely the reason for the large Temp file. We are considering creating an article to explain this in more detail.
I should also note, we are working to add Bulk Mode support for most of our transformers and formats. So these issues will hopefully become less common.
However, there is one concerning detail we want to clarify. Temp files created during translation should be cleaned up after the translation stops, even with a translation failure. The only exception would be if FME crashed, then the temp files might be left behind.
So, I'm curious, in your scenarios, did the workbench crash? If not, were the temp files cleaned up afterwards? If there was no crash, and the temp files were left behind after translation stopped, we would like to investigate why.
Hi @spatial_aus and @david_r, I've got some potential explanation for the observed behaviour:
David is correct that FeatureJoiner should be more efficient, but there is a caveat. It only works with features in "Bulk Mode" (some people may know Bulk Mode as our previous internal terminology, "Feature Table").
So, if the incoming features are not in Bulk Mode, FeatureJoiner will cache the incoming features to disk in Bulk Mode. And this is likely the reason for the large Temp file. We are considering creating an article to explain this in more detail.
I should also note, we are working to add Bulk Mode support for most of our transformers and formats. So these issues will hopefully become less common.
However, there is one concerning detail we want to clarify. Temp files created during translation should be cleaned up after the translation stops, even with a translation failure. The only exception would be if FME crashed, then the temp files might be left behind.
So, I'm curious, in your scenarios, did the workbench crash? If not, were the temp files cleaned up afterwards? If there was no crash, and the temp files were left behind after translation stopped, we would like to investigate why.
Thanks for the info, that could well explain my observations. It would be fantastic, however, if the FME could warn you (log) that you're using the FeatureJoiner with non-bulk mode features, since that seems to have a big impact on performance and disk usage. If you unwittingly end up using the FeatureJoiner with the "wrong" type of features I have the impression that it could much faster to simply use the FeatureMerger -- is that correct?
Unfortunately I've stopped using the FeatureJoiner entirly because of this, so I cannot really remember the details, but I believe at some point I had several GB of temp files coming from the FeatureJoiner, dating back. This was just after the FeatureJoiner was introduced, so it may well have been improved since then.
+11Thanks for the info, that could well explain my observations. It would be fantastic, however, if the FME could warn you (log) that you're using the FeatureJoiner with non-bulk mode features, since that seems to have a big impact on performance and disk usage. If you unwittingly end up using the FeatureJoiner with the "wrong" type of features I have the impression that it could much faster to simply use the FeatureMerger -- is that correct?
Unfortunately I've stopped using the FeatureJoiner entirly because of this, so I cannot really remember the details, but I believe at some point I had several GB of temp files coming from the FeatureJoiner, dating back. This was just after the FeatureJoiner was introduced, so it may well have been improved since then.
Hi @david_r, good idea about logging something to inform users, when features are getting cached. I'll bring that suggestion to the dev team.
Regarding processing speed, in theory, we actually expect caching in bulk mode using FeatureJoiner to be faster than using FeatureMerger. So, we would be really interested in seeing your example, if it demonstrates otherwise.
Would you be willing to share your data and workspace with us through a support case? (if the sample data is too large, you can upload them to our ftp). Thank you very much.
+10I'm pleased to announce the Purge Temporary Files option (Tools menu) will delete the FMETable directory in FME's TEMP location after a recent update. Please note that using this option while a translation is running could potentially cause the translation to fail, if it removes files that are needed.
You will find the update in our latest FME 2018.1 beta: safe.com/beta
@DebbiAtSafe Is this accidental deletion being fixed in FME 2019?
+21@DebbiAtSafe Is this accidental deletion being fixed in FME 2019?
Hi @srg
The fix for Purge Temporary Files under Tools menu not deleting .fts files is also in FME 2019. The current behaviour for Purge Temporary Tools is it will delete temporary files left in the designated temp folder. These files are usually left behind if FME does not shutdown 'cleanly' (more information on this here).
I'm not certain I understand what 'accidental deletion' refers to in your question. Could you expand on what you mean?
+10Hi @srg
The fix for Purge Temporary Files under Tools menu not deleting .fts files is also in FME 2019. The current behaviour for Purge Temporary Tools is it will delete temporary files left in the designated temp folder. These files are usually left behind if FME does not shutdown 'cleanly' (more information on this here).
I'm not certain I understand what 'accidental deletion' refers to in your question. Could you expand on what you mean?
@DebbiAtSafe my meaning of accidental deletion is deletion of temporary files created by a FME process which is still running.
From your response, I understand that it is fixed in FME 2019.
+21@DebbiAtSafe my meaning of accidental deletion is deletion of temporary files created by a FME process which is still running.
From your response, I understand that it is fixed in FME 2019.
@srg The fix mentioned was not meant to address accidental deletion of temporary files created by an FME process which is still running. Instead, it was meant to address the incorrect behaviour where .fts files were not deleted when the user selects Purge Temporary Files.
We do not recommend using Purge Temporary Files when a translation is running to avoid potential translation failures.
