Hi,
I have some FME workspaces, that investigate the recent FME log files from previous translations. This is done to uncover any problems in a rational way.
I want to extract the statistics at the bottom, but the "|STATS |" string is used extensively thru-out the log file, and would completely swamp the output if this was the only search criteria (while reading as a TEXT file).
Do anyone have a good idea how to extract just the summation details ?
Cheers
Lars I
I would look into searching for the log lines that contain Features Read Summary & Features Written Summary since they are always present.
All the statistics are available in the shutdown script, it's really easy to e.g. write a part of it to a custom log file: https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_Workbench/Configuration/FME_END_PYTHON.htm
Also lots of good info here:
https://knowledge.safe.com/articles/849/how-to-extract-and-use-log-information-in-workbenc.html
David_r: Well, I'm investigating multiple log files from other workspace translations, as a separate workspace translation, so a shutdown script in any of these previously run workspaces isn't really an option. And I don't want to be dependent on some logic in these workspaces anyhow.
itay: I've looked at the "Feature Read Summary" and "Feature Written Summary" lines, but I'm really just interested in the numbers below these lines, and how can I get them without all the irrelevant STATS lines in the rest of the log file ?
Hi @lifalin2016, interesting challenge. This workflow extracts the text block of consecutive STATS lines in the bottom of a log file.
David_r: Well, I'm investigating multiple log files from other workspace translations, as a separate workspace translation, so a shutdown script in any of these previously run workspaces isn't really an option. And I don't want to be dependent on some logic in these workspaces anyhow.
itay: I've looked at the "Feature Read Summary" and "Feature Written Summary" lines, but I'm really just interested in the numbers below these lines, and how can I get them without all the irrelevant STATS lines in the rest of the log file ?
Write a python program and loop thru each log file , go to last line of log file and search for string "stats" and print it or write to another text file.
Hi @lifalin2016, interesting challenge. This workflow extracts the text block of consecutive STATS lines in the bottom of a log file.
# PythonCaller Script Example
# Extracts consecutive STATS lines in the bottom of a log file.
# Assume that the character encoding of the log file is UTF-8.
import os, codecs
class LogSummaryExtractor(object):
def input(self, feature):
path = feature.getAttribute('_logFile') # FME log file path
if path and os.path.exists(path):
with codecs.open(path, mode='r', encoding='utf-8') as f:
stats = ]
for row in f.readlines()-1::-1]:
if '|STATS' in row:
stats.append(row)
elif stats:
break
feature.setAttribute('_summary', ''.join(stats-1::-1]))
self.pyoutput(feature)
No need to over complicate things with exotic scripting 
, it can all be done with FME...
Hi @lifalin2016, interesting challenge. This workflow extracts the text block of consecutive STATS lines in the bottom of a log file.
Most of the comments here have been helpful, but your solution is the one that taught me a few new tricks. And it works too
I do get the last batch of "STATS" lines that I wanted.Using a variable to track reading state, and reading the log files backwards, are tricks I'll keep in mind for future use.
I did write a small Python transformer also, but knowing how to do it without coding is more versatile.
Cheers
Lars I.
same way you extract data from any structured text file..
basically regexp searching, which is made easier nowadays due to the vastly improved stringsearcher.
an variablesetter/retriever combination.
I don't use a separate creator for the variablesetter, that's not necessary.
It' is for instance a handy to process your log after runtime to check for warnings and which attributes are involved in the warning etc. Then auto generate a report.
For some good reading in the morning with coffee....;)
No need to over complicate things with exotic scripting , it can all be done with FME...
what is the text line number
what is the text line number
\\CreateGemWPL.log' does not exist
what is the text line number
@asadamjad instead you select your own log file. Any fme log file work here.
# PythonCaller Script Example
# Extracts consecutive STATS lines in the bottom of a log file.
# Assume that the character encoding of the log file is UTF-8.
import os, codecs
class LogSummaryExtractor(object):
def input(self, feature):
path = feature.getAttribute('_logFile') # FME log file path
if path and os.path.exists(path):
with codecs.open(path, mode='r', encoding='utf-8') as f:
stats = o]
for row in f.readlines()a-1::-1]:
if '|STATS' in row:
stats.append(row)
elif stats:
break
feature.setAttribute('_summary', ''.join(stats-1::-1]))
self.pyoutput(feature)
No need to over complicate things with exotic scripting , it can all be done with FME...
👍 Good!
what is the text line number
its not working
what is the text line number
here is the fme workbench
what is the text line number
I m getting this
what is the text line number
is there anyway to get summry like this s
what is the text line number
open python caller
replace feature.getAttribute('_logfile') from the left hand panel
just double click on "_logfile"
fme will copy attribute from left window.
This is the issue with python caller transformer 2022, when we send the fme to other system feature.getAttribute function not doing its job. we need to double click on attribute and re define the variable.
what is the text line number
I didnt get it
I attached the WS please check