Solved

Extract (only) summation stats from FME log files ?

lifalin2016
Rank
Userlevel 1
Badge +22

Hi,

I have some FME workspaces, that investigate the recent FME log files from previous translations. This is done to uncover any problems in a rational way.

I want to extract the statistics at the bottom, but the "|STATS |" string is used extensively thru-out the log file, and would completely swamp the output if this was the only search criteria (while reading as a TEXT file).

Do anyone have a good idea how to extract just the summation details ?

Cheers

Lars I

icon

Best answer by takashi 11 May 2017, 16:14

View original
-- Cheers, Lars I.

22 replies

itay
Rank
Badge +16

I would look into searching for the log lines that contain Features Read Summary & Features Written Summary since they are always present.

david_r
Rank
Userlevel 4

All the statistics are available in the shutdown script, it's really easy to e.g. write a part of it to a custom log file: https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_Workbench/Configuration/FME_END_PYTHON.htm

Also lots of good info here:

https://knowledge.safe.com/articles/849/how-to-extract-and-use-log-information-in-workbenc.html

lifalin2016
Rank
Userlevel 1
Badge +22

David_r: Well, I'm investigating multiple log files from other workspace translations, as a separate workspace translation, so a shutdown script in any of these previously run workspaces isn't really an option. And I don't want to be dependent on some logic in these workspaces anyhow.

itay: I've looked at the "Feature Read Summary" and "Feature Written Summary" lines, but I'm really just interested in the numbers below these lines, and how can I get them without all the irrelevant STATS lines in the rest of the log file ?

-- Cheers, Lars I.
takashi
Rank
Userlevel 2
Badge +17

Hi @lifalin2016, interesting challenge. This workflow extracts the text block of consecutive STATS lines in the bottom of a log file.

itay
Rank
Badge +16

David_r: Well, I'm investigating multiple log files from other workspace translations, as a separate workspace translation, so a shutdown script in any of these previously run workspaces isn't really an option. And I don't want to be dependent on some logic in these workspaces anyhow.

itay: I've looked at the "Feature Read Summary" and "Feature Written Summary" lines, but I'm really just interested in the numbers below these lines, and how can I get them without all the irrelevant STATS lines in the rest of the log file ?

I have done in the past something similar to what @takashi proposes, that is one way to do it....

 

f.kemminje
Badge +3

Write a python program and loop thru each log file , go to last line of log file and search for string "stats" and print it or write to another text file.

takashi
Rank
Userlevel 2
Badge +17

Hi @lifalin2016, interesting challenge. This workflow extracts the text block of consecutive STATS lines in the bottom of a log file.

0684Q00000ArKIJQA3.png

Python script version inspired by @fkemminje's post.

 

# PythonCaller Script Example
# Extracts consecutive STATS lines in the bottom of a log file.
# Assume that the character encoding of the log file is UTF-8.
import os, codecs
class LogSummaryExtractor(object):
    def input(self, feature):
        path = feature.getAttribute('_logFile') # FME log file path
        if path and os.path.exists(path):
            with codecs.open(path, mode='r', encoding='utf-8') as f:
                stats = []
                for row in f.readlines()[-1::-1]:
                    if '|STATS' in row:
                        stats.append(row)
                    elif stats:
                        break
                feature.setAttribute('_summary', ''.join(stats[-1::-1]))
        self.pyoutput(feature)
itay
Rank
Badge +16

No need to over complicate things with exotic scripting :), it can all be done with FME...

ftwritten.fmwt

lifalin2016
Rank
Userlevel 1
Badge +22

Hi @lifalin2016, interesting challenge. This workflow extracts the text block of consecutive STATS lines in the bottom of a log file.

Hi itay,

 

Most of the comments here have been helpful, but your solution is the one that taught me a few new tricks. And it works too :-) I do get the last batch of "STATS" lines that I wanted.

 

Using a variable to track reading state, and reading the log files backwards, are tricks I'll keep in mind for future use.

 

I did write a small Python transformer also, but knowing how to do it without coding is more versatile.

 

Cheers

 

Lars I.

 

-- Cheers, Lars I.
G
Badge +3

same way you extract data from any structured text file..

basically regexp searching, which is made easier nowadays due to the vastly improved stringsearcher.

an variablesetter/retriever combination.

I don't use a separate creator for the variablesetter, that's not necessary.

It' is for instance a handy to process your log after runtime to check for warnings and which attributes are involved in the warning etc. Then auto generate a report.

For some good reading in the morning with coffee....;)

A
Badge +5

No need to over complicate things with exotic scripting :), it can all be done with FME...

ftwritten.fmwt

what is the text line number

A
Badge +5

what is the text line number

\\CreateGemWPL.log' does not exist

 

f.kemminje
Badge +3

what is the text line number

@asadamjad​ instead you select your own log file. Any fme log file work here.

f.kemminje
Badge +3
Python script version inspired by @fkemminje's post.

 

# PythonCaller Script Example
# Extracts consecutive STATS lines in the bottom of a log file.
# Assume that the character encoding of the log file is UTF-8.
import os, codecs
class LogSummaryExtractor(object):
    def input(self, feature):
        path = feature.getAttribute('_logFile') # FME log file path
        if path and os.path.exists(path):
            with codecs.open(path, mode='r', encoding='utf-8') as f:
                stats = []
                for row in f.readlines()[-1::-1]:
                    if '|STATS' in row:
                        stats.append(row)
                    elif stats:
                        break
                feature.setAttribute('_summary', ''.join(stats[-1::-1]))
        self.pyoutput(feature)

😍 👍 

f.kemminje
Badge +3

No need to over complicate things with exotic scripting :), it can all be done with FME...

ftwritten.fmwt

👍 Good!

A
Badge +5

what is the text line number

its not working

f.kemminje
Badge +3

what is the text line number

here is the fme workbench

A
Badge +5

what is the text line number

I m getting this

 

image

A
Badge +5

what is the text line number

is there anyway to get summry like this s

 

image

f.kemminje
Badge +3

what is the text line number

open python caller

replace feature.getAttribute('_logfile') from the left hand panel

just double click on "_logfile"

fme will copy attribute from left window.

This is the issue with python caller transformer 2022, when we send the fme to other system feature.getAttribute function not doing its job. we need to double click on attribute and re define the variable.

 

@mark2atsafe​ 

A
Badge +5

what is the text line number

I didnt get it

A
Badge +5

I attached the WS please check

 

Reply


Community Stats

28,670
Posts
109,365
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.