FME FLOW Data Cleanup

Right now you have the job logs deletion disabled, so they won’t be deleted at all. If it does get enabled somewhere down the line it’ll only delete logs that are more than 521 weeks (i.e. 10-ish years) old, so I think you’re good.

However, 2 things I’d like to point out:

• How many jobs per year are you expecting to run? Be careful you don’t run out of drive space (because that will trigger more strict cleanup tasks and it may be that job logs are still deleted if that happens)
• How likely do you think it’s going to be that you’ll need to see a job log from such a long time ago?

Unfortunately there’s only one setting for the job log deletion and I wish there would be some more fine-grained control. Like on jobs that are run monthly, I’d like to retain them for a bit longer so if one run fails I can check it against the previous log, from a month ago. 

If you need to keep the logs I’d almost be tempted to move them to some kind of archival storage somewhere - As ​@redgeographics points out the logs can take up quite a bit of space, especially those ones which get filled with millions of warnings.

You could set up a nightly schedule to upload the logs to s3 or some other storage location. 

Also keep in mind that if you upgrade FME Flow then you will need to do something with those logs anyway as they do not get migrated when you update the instance.  The same is true with the job table itself. A new FME Flow instance will restart the job id back to 0.

https://support.safe.com/hc/en-us/articles/25407507766541-FME-Flow-Administration-Backup-Migrations-and-Upgrades

In our own FME Flow I back up the job history - both in case the data gets deleted from FME Flow somehow and so I have a record following updates.  
 

Ah yes, I overlooked the fact that job logs don’t get migrated as part of a backup/restore. If you plan to upgrade your FME Flow somewhere in that 10-year period (😂), do what ​@virtualcitymatt says.

Unfortunately there’s only one setting for the job log deletion and I wish there would be some more fine-grained control. 

Have you created an idea for this? I’d really like to be able to keep the last X amount of job logs for a workspace rather than a defined time period as it’s difficult balancing what to keep when you have workspaces running a few hundred times a day and some running once a month. 

I have now: 

 (vote for me!)

After initially installing Flow, there should be a pop up asking the Admin whether they are happy with the default 1 week job log storage, or if they’d like more. I generally set it for 4-5 weeks