+38Hi,
I have some FME workspaces, that investigate the recent FME log files from previous translations. This is done to uncover any problems in a rational way.
I want to extract the statistics at the bottom, but the "|STATS |" string is used extensively thru-out the log file, and would completely swamp the output if this was the only search criteria (while reading as a TEXT file).
Do anyone have a good idea how to extract just the summation details ?
Cheers
Lars I
Best answer by takashi
Hi @lifalin2016, interesting challenge. This workflow extracts the text block of consecutive STATS lines in the bottom of a log file.
+18I would look into searching for the log lines that contain Features Read Summary & Features Written Summary since they are always present.
All the statistics are available in the shutdown script, it's really easy to e.g. write a part of it to a custom log file: https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_Workbench/Configuration/FME_END_PYTHON.htm
Also lots of good info here:
https://knowledge.safe.com/articles/849/how-to-extract-and-use-log-information-in-workbenc.html
+38David_r: Well, I'm investigating multiple log files from other workspace translations, as a separate workspace translation, so a shutdown script in any of these previously run workspaces isn't really an option. And I don't want to be dependent on some logic in these workspaces anyhow.
itay: I've looked at the "Feature Read Summary" and "Feature Written Summary" lines, but I'm really just interested in the numbers below these lines, and how can I get them without all the irrelevant STATS lines in the rest of the log file ?
Hi @lifalin2016, interesting challenge. This workflow extracts the text block of consecutive STATS lines in the bottom of a log file.
+18David_r: Well, I'm investigating multiple log files from other workspace translations, as a separate workspace translation, so a shutdown script in any of these previously run workspaces isn't really an option. And I don't want to be dependent on some logic in these workspaces anyhow.
itay: I've looked at the "Feature Read Summary" and "Feature Written Summary" lines, but I'm really just interested in the numbers below these lines, and how can I get them without all the irrelevant STATS lines in the rest of the log file ?
+11Write a python program and loop thru each log file , go to last line of log file and search for string "stats" and print it or write to another text file.
Hi @lifalin2016, interesting challenge. This workflow extracts the text block of consecutive STATS lines in the bottom of a log file.
# PythonCaller Script Example
# Extracts consecutive STATS lines in the bottom of a log file.
# Assume that the character encoding of the log file is UTF-8.
import os, codecs
class LogSummaryExtractor(object):
def input(self, feature):
path = feature.getAttribute('_logFile') # FME log file path
if path and os.path.exists(path):
with codecs.open(path, mode='r', encoding='utf-8') as f:
stats = []
for row in f.readlines()[-1::-1]:
if '|STATS' in row:
stats.append(row)
elif stats:
break
feature.setAttribute('_summary', ''.join(stats[-1::-1]))
self.pyoutput(feature)
No need to over complicate things with exotic scripting :), it can all be done with FME...
+38Hi @lifalin2016, interesting challenge. This workflow extracts the text block of consecutive STATS lines in the bottom of a log file.
Most of the comments here have been helpful, but your solution is the one that taught me a few new tricks. And it works too :-) I do get the last batch of "STATS" lines that I wanted.
Using a variable to track reading state, and reading the log files backwards, are tricks I'll keep in mind for future use.
I did write a small Python transformer also, but knowing how to do it without coding is more versatile.
Cheers
Lars I.
+15same way you extract data from any structured text file..
basically regexp searching, which is made easier nowadays due to the vastly improved stringsearcher.
an variablesetter/retriever combination.
I don't use a separate creator for the variablesetter, that's not necessary.
It' is for instance a handy to process your log after runtime to check for warnings and which attributes are involved in the warning etc. Then auto generate a report.
For some good reading in the morning with coffee....;)
+5No need to over complicate things with exotic scripting :), it can all be done with FME...
what is the text line number
+11what is the text line number
@asadamjad instead you select your own log file. Any fme log file work here.
+11
# PythonCaller Script Example
# Extracts consecutive STATS lines in the bottom of a log file.
# Assume that the character encoding of the log file is UTF-8.
import os, codecs
class LogSummaryExtractor(object):
def input(self, feature):
path = feature.getAttribute('_logFile') # FME log file path
if path and os.path.exists(path):
with codecs.open(path, mode='r', encoding='utf-8') as f:
stats = []
for row in f.readlines()[-1::-1]:
if '|STATS' in row:
stats.append(row)
elif stats:
break
feature.setAttribute('_summary', ''.join(stats[-1::-1]))
self.pyoutput(feature)
😍 👍
+11No need to over complicate things with exotic scripting :), it can all be done with FME...
👍 Good!
+11what is the text line number
open python caller
replace feature.getAttribute('_logfile') from the left hand panel
just double click on "_logfile"
fme will copy attribute from left window.
This is the issue with python caller transformer 2022, when we send the fme to other system feature.getAttribute function not doing its job. we need to double click on attribute and re define the variable.
