Question of the Week: DateTimes in File Names (01112019)

Hello FME'ers and Happy November!

There's a saying that "things always come in threes" and here's proof. This Question of the Week is not so much one question, as three instances of the same requirement: to put a datetime in a file name.

Question of the Week

Q) I have a workspace that downloads files, renames them and stores them in a ZIP file. Said ZIP should contain the current date in its filename. Can this be done somehow?

Asked in various forms, here, here, and here!

A) Basically the user wants to add the current datetime to an output filename. The obvious use is to stop multiple runs overwriting the same file. I like this question because there are multiple places where this might be done: on a writer, in a transformer, or as the name of a log file. Let's take the writer scenario first.

Writer Timestamps

To get a timestamp I obviously need to use either the DateTimeStamper transformer or the DateTimeNow() function. However, if I enter that function directly into the output parameter, then my folder is literally DateTimeNow()

 

 

As one user found out, the same happens if you used the DateTimeStamper and put the resulting attribute (say @Value(MyDateTimeAttribute)) directly into the writer parameter.

The answer is to use the dataset fanout functionality instead. One user tried that, but crucially I think they tried to put the datetime into the directory name, when it should instead go into the fanout part:

 

 

Simple. Except... it can still be a bit tricky. The DateTimeNow() function is returning times to tiny fractions of a second. Instead of 1 folder, groups of features get written to multiple output folders:

 

 

I honestly didn't expect that. I thought it would use the first datetime as the folder for all outgoing data. Obviously that's not the case, but it's OK because I can also use the DateTimeFormat() function to format the date, like so:

 

 

Giving me...

 

 

Of course, you can use whatever flags you want to generate the right style of datetime for you.

 

 

Here's if you want to try it out.

 

 

The other solution - as given in one question thread - would be to generate the datetime as an attribute in the workspace itself and use that in the fanout component:

 

 

The nice thing there is that you generate a single datetime and attach it to all features. You avoid the issue of having different values as time passes, yet still get fractional seconds support.

 

 

Here's too.Transformer Timestamps

The second use of timestamps like this would be in a transformer like the FTPCaller. One question asked specifically for that:

I'd like to download .zip files to a folder whose name is actually the date on which the download took place. However, when I try and incorporate this into the FTPCaller's 'Target File' path (e.g. C:\\Downloads\\@DateTimeFormat(@DateTimeNow(),%Y_%m_%d)\\@filename) the translation fails.

Admittedly the question is an older one, but the solution was posted this week. As per Dale:

Took a bit longer than we hoped, but for the record, this is fixed as of FME 2019.2 build 19790 and will be in FME 2020 betas with builds later than 20110.

So that scenario should now work too.

Log File Timestamps

The final use is to set the workspace logfile name to include a datetime. Currently it's only possible in a single workspace by using a Scripted Parameter (Python or Tcl). But... you can do this with a two-workspace setup, which is exactly what you'd use for batch processing, and exactly when this technique is most useful.

First, in the helper workspace (the one doing the work) publish the Log File parameter:

Now in your control workspace (the one running the helper) you can create your logfile name - including the datetime - and pass it to the helper workspace:

The result is a separate log file for each process:

That's way better because otherwise each process would overwrite the previous log file.

Here are in case you want to check out the setup.

I hope you find this useful. I love datetime questions, and so combining them into FME parameters really is my idea of fun (hey, winter nights are long and cold here in Manitoba). You can also find much of this information in a tutorial on datetimes in general.

Other Notable Questions

Here are a few other notable questions that I see from this week:

• Adjacent Feature Handling: Adjacent feature attributes help in a wide range of uses, so are useful to know about
• Workspace Completion Report: I like this because the user (@agelfert) has a working prototype, but reached out for help improving the method. I think that's an important lesson for us all there!
• Raster Map Legend Creation: A question that isn't answered and yet I know it must be possible. Has anyone tried this before?
• 3D Pipelines: Just because I love me a 3D buffered line feature!

Don't forget the Monday quiz results (and look out for next Monday's quiz).

And remember, we issued a security update for FME Server last week too.

Plus let me know if there is a question you would like to nominate as Question of the Week.