Question

FME Golf: Round 2 - The Log Files

mark2atsafe
Rank
Userlevel 4
Badge +25

Previously I'd challenged users to a game of FME Golf. The idea was to take an FME workspace and reduce it to the smallest file size possible. The winner was Todd Davis, who took a workspace 75,835 bytes in size and reduced it to 35,229 bytes, while still producing the same result.

So, let's call this new challenge FME Golf Round 2!

I wanted to make this a little different than before, so here is the twist. What I want you to do is not reduce the size of the workspace, but the size of the log file!

If you download and run you get a log file of 42,472 bytes (45,056 bytes on disk) - or that's the size of the log file I get. The challenge is to edit the FME workspace in such a way as to reduce the size of that log file!

The rules:

  • The log file must be produced by running the workspace in Workbench
  • Editing of the log - whether manual or with a script - is not permitted. For example, you can't just run a shutdown script to empty the file!
  • The workspace can be edited in any way you want, and I'm going to say you can use the PythonCaller and/or TclCaller, if you think it will help
  • The workspace must write the exact same CSV files that it does now
  • You must use FME2016; 2016.0, .1, etc. are all OK.
  • You can't turn off any logging of WARN, ERROR, STATS or INFORM under options

I'm going to say you can submit a log; I won't make you submit the workspace or output unless you really want to. I'm going to trust you not to cheat! But you official score will be the size of the file as it appears on my computer.

To be honest, I don't know what to expect here. I can think of a few ways to make the log smaller, but I don't know how well they will work; so I am really looking forward to see what you can do!

And as a quick hint: this workspace comes from an exercise in the advanced training manual. It is one of three versions that produce the same result. Maybe that's a good starting point...

I'll take a look in a couple of week's time, so no immediate rush.

Oh, and of course the winner gets the coveted FME Golf custom badge! Let's not forget that. And I will see if marketing feel like sending out a small piece of swag too.

Have fun!

Mark

11 replies

mark2atsafe
Rank
Userlevel 4
Badge +25
As an extra rule (thanks for pointing it out @bruceharold) you can't turn off any of the message types under Tools > FME Options. I hadn't mentioned that before because I thought it only affected the log window, not the log file. Shows how wrong I can be!

 

itay
Rank
Badge +16
As an extra rule (thanks for pointing it out @bruceharold) you can't turn off any of the message types under Tools > FME Options. I hadn't mentioned that before because I thought it only affected the log window, not the log file. Shows how wrong I can be!

 

And there I was thinking it was that easy.....

 

 

takashi
Rank
Userlevel 2
Badge +17
As an extra rule (thanks for pointing it out @bruceharold) you can't turn off any of the message types under Tools > FME Options. I hadn't mentioned that before because I thought it only affected the log window, not the log file. Shows how wrong I can be!

 

My first inspiration was this startup Python script, but this might also be against the rules.

 

from fmeobjects import FMELogFile
FMELogFile().silent(True)
david_r
Rank
Userlevel 4
My first inspiration was this startup Python script, but this might also be against the rules.

 

from fmeobjects import FMELogFile
FMELogFile().silent(True)
Had the exact same idea ;-)
todd_davis
Rank
Userlevel 1
Badge +12
As an extra rule (thanks for pointing it out @bruceharold) you can't turn off any of the message types under Tools > FME Options. I hadn't mentioned that before because I thought it only affected the log window, not the log file. Shows how wrong I can be!

 

@Mark2AtSafe Also wondering if you need to say that python can't be used for reading inputs or writing outputs (e.g. writing the output csv)? Otherwise you won't be able to tell if anything has really been carried.

 

mark2atsafe
Rank
Userlevel 4
Badge +25
Let's just say that you know when you're cheating - or at least when you're bending the rules beyond what they should be. And turning off logging in any way is not what we're intending here!

 

 

todd_davis
Rank
Userlevel 1
Badge +12

Here is my first go (28,897) cutting out a few "normal" processes. Trying to think of a way of avoid all those details with the multiple writers kicking into action without using python as the writer, or "redirect to inspector" (which doesn't really output anything but makes the log way smaller)...hmmm let me think...

ciarab
Rank
Badge +9

Gave it a go but could only get it down to a determined (32,015). I'll have to have a another think about it! :)

log.zip

jdh
Rank
Badge +22

I got it down to 21,241 bytes, but I did post process the log file to replace my username and pcname. a.zip. The size before stripping personal data was 21,322. No scripting was used to produce the csv files.

todd_davis
Rank
Userlevel 1
Badge +12

Round 2...Well this get the log file down at 18,878 and certainly comes out with the same final output. It also has no scripting but uses the AttributeFileWriter and so you can't actually see the numbers that are been written out and that is why that is small. I have thrown all "best practice" out the door on this one. Not a big fan of it, because you don't even know what has been written.

2.zip

mark2atsafe
Rank
Userlevel 4
Badge +25
Hey folks. Apologies for not getting back to this sooner. You know how it is. I'll take a look at the submissions early next week. So last chance if you want to submit.

 

Reply


Community Stats

28,723
Posts
109,511
Replies
36,559
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.