Write logfile "ERROR" and "WARN" messages to a database table

Using fmeobjects, it is possible to define a user function that is called for every line that is output to the FME log. You could take advantage of this to capture and filter the messages based on the severity, then write those lines to the database e.g. using the FMEUniversalWriter object (if it's Oracle it might even be easier to write it directly using the cx_Oracle Python module).

Example startup script:

from fmeobjects import *
def LogSkimmer(severity, text):
    if severity in (FME_WARN, FME_ERROR):
        # Write the 'text' object to the database here...
FMELogFile().setCallBack(LogSkimmer)

David

I suspect, that this may need to be done as a post process where you run a separate workspace to read the .log file using the text file reader and parse out the lines beginning with "ERROR" and "WARN" and then load those to the database. If no one else comes back with a better method let me know and I'll post further details.

In the mean time, also take a look at the FME Store transformer the MessageLogger that I wrote a while back. This is helpful when you'd like to send user configured Errors or Warnings to the log file, rather than just ones created by FME. This maybe handy when features do not conform to your own rules rather than just failing rules defined in say the GeometryValidator.

All the best, Dave

Using fmeobjects, it is possible to define a user function that is called for every line that is output to the FME log. You could take advantage of this to capture and filter the messages based on the severity, then write those lines to the database e.g. using the FMEUniversalWriter object (if it's Oracle it might even be easier to write it directly using the cx_Oracle Python module).

Example startup script:

from fmeobjects import *
def LogSkimmer(severity, text):
    if severity in (FME_WARN, FME_ERROR):
        # Write the 'text' object to the database here...
FMELogFile().setCallBack(LogSkimmer)

David

Using the LogSkimmer function I was able to collect the log file errors and print them in the FME log window but I haven't been able yet to create a feature using the text object (type string) every time that a "WARN" or "ERROR" is found.

I am receiving the following Python error in my FME log window:

(__main__:26: RuntimeWarning: tp_compare didn't return -1 or -2 for exception) 

when I am running the following functions:

from fmeobjects import *

def log_skimmer(severity,text):

    if severity in (FME_WARN,FME_ERROR):

        feature = FMEFeature()
        
	#THIS IS WHERE I AM GETTING THE ERROR AT
	feature.setattribute("MESSAGE_TEXT",text)
    
    return feature
        
###RUN FUNCTIONS###
FMELogFile().setCallBack(log_skimmer)

IMPORTANT UPDATE: This could be well a bug; I have tested other methods of the feature that I am creating using FMEFeature() when the if statement evaluates to "True" including setAttributeNullWithType(), setFeatureType(featureType), and setGeometryType(geomType) and they seem to work fine. It seems that the error is produced when I am using the setAttribute(attrName, attrValue) method.

Hi

I'm not sure you're allowed to create FMEFeature objects inside the callback function, at least that's not how I understand this to work. Also, your usage of the FMEUniversalWriter above does not make much sense to me, it looks very incomplete.

I think might be better off installing and using psycopg2 module to write directly to posgresql and skipping the FMEUniversalWriter altogether.

David

Hi David,

Using the LogSkimmer function I was able to collect the log file errors and print them in the FME log window but I haven't been able yet to create a feature using the text object (type string) every time that a "WARN" or "ERROR" is found.

I am receiving the following Python error in my FME log window:

(__main__:26: RuntimeWarning: tp_compare didn't return -1 or -2 for exception) 

when I am running the following functions:

from fmeobjects import *

def log_skimmer(severity,text):

    if severity in (FME_WARN,FME_ERROR):

        feature = FMEFeature()
        
	#THIS IS WHERE I AM GETTING THE ERROR AT
	feature.setattribute("MESSAGE_TEXT",text)
    
    return feature
        
###RUN FUNCTIONS###
FMELogFile().setCallBack(log_skimmer)

IMPORTANT UPDATE: This could be well a bug; I have tested other methods of the feature that I am creating using FMEFeature() when the if statement evaluates to "True" including setAttributeNullWithType(), setFeatureType(featureType), and setGeometryType(geomType) and they seem to work fine. It seems that the error is produced when I am using the setAttribute(attrName, attrValue) method.

If you're able to use FME 2016, we've got a new LogMessageStreamer transformer for you http://docs.safe.com/fme/2016.0/html/FME_Desktop_Documentation/FME_Transformers/Transformers/LogMessageStreamer.htm

Unfortunately I was not able to write the "WARN" and "ERROR" information skimmed by the "Translation Log" into a database table using the FMEUniversalWriter Class. I managed though to write the log data into a *.csv and a *.txt using the python csv module or the python write method respectively which in the first instance may fit the purpose of what I am trying to achieve.

Code is following:

from fmeobjects import *
import csv

def log_skimmer(severity,text):
   
    if severity in (FME_WARN,FME_ERROR):
        csv_row = [str(text),str(severity)]
        with open('C:\\log.csv','a') as csv_file:
            csv_writer = csv.writer(csv_file,delimiter=',')
            csv_writer.writerow(csv_row)

    return None

FMELogFile().setCallBack(log_skimmer)

The FMEUniversalWriter Class seems very promising though and deserves some additional documentation including commented examples to assist the community better understand it and use it in startup & shutdown scripts.

This answer has been accepted as "best answer" because it describes a more complete method for tackling the problem described in the fist post in this topic; It is only because of @david_r contribution of the log_skimmer() function and the clear explanation of the setCallBack() function that made the answer possible.