Question

Alerts for Error Messages (a sort of Friday challenge)

mark2atsafe
Rank
Userlevel 4
Badge +25

So, I'm looking at this question on StackExchange.

The user has color blindness and has trouble reading error messages in the FME log, because they are colored red.

I've filed an enhancement request and we'll work on adding those to our special color blindness theme.

But, in the meantime I was wondering if we could do something else to help. They don't want to have to copy/paste the log to check for errors. So I was thinking perhaps a shutdown python script that automatically extracted error messages? Or even a script that just popped up a dialog saying "there has been an error"?

Any thoughts?

My Python skills are pretty limited. And yes, I just tweeted recently how I never needed them, because of FME. Ironic (or perhaps moronic) is the word you're looking for!

NB: I'll treat it as a challenge and award extra reputation points for really good solutions.

20 replies

erik_jan
Rank
Userlevel 2
Badge +12

I have used a Python Shutdown script before that took the LOG file and created 4 separate log files based on the type of message (INFORM, STATS, WARN and ERROR).

After that the user only had to check if a ERROR file was present.

david_r
Rank
Userlevel 4

Here's a suggestion using the fmeobjects log file callback and some global variables.

In the startup python script:

from fmeobjects import *
 
# Global dictionary used to stor important log messages
global fme_log_messages
fme_log_messages = {FME_WARN: [], FME_ERROR: [], FME_FATAL: []}
 
# Global integer to keep track of current log line number
global fme_log_message_count
fme_log_message_count = 0
 
def catch_log_messages(severity, message):
    global fme_log_message_count
    global fme_log_messages
    fme_log_message_count += 1
    if severity in (FME_WARN, FME_ERROR, FME_FATAL) and message:
        fme_log_messages[severity].append('%8d : %s' % (fme_log_message_count, message))
 
FMELogFile().setCallBack(catch_log_messages)

In the shutdown python script:

from __future__ import print_function
 
global fme_log_messages
no_messages = [' ' * 15 + 'None']
 
print('=' * 80)
print(' Line number   Log message')
print('-' * 80)
print('WARNINGS')
for warning in fme_log_messages[FME_WARN] or no_messages:
    print(warning)
    
print('ERRORS')
for error in fme_log_messages[FME_ERROR] or no_messages:
    print(error)
    
print('FATAL ERRORS')
for fatal in fme_log_messages[FME_FATAL] or no_messages:
    print(fatal)
 
print('=' * 80)

At the end of the translation the shutdown script will output a synopsis of all the warnings, errors and fatal errors with line numbers (referencing the written log), e.g.:

XpVDAOh 

Notice that the synopsis is output using the python 'print' function and that the text is therefore NOT included in the log file, it is only sent to the translation log console window. This is by design, but the script can of course be extended to also write to the log file, if necessary.

If there are no warnings etc, it looks like this:

eL8YZW0 

Tested with Python 2.7 and 3.5.

Template file: fmelogsummarywriter.fmwt

cwarren
Badge

I'm new to Python as well and based my script on  @david_r 's code from this post: https://knowledge.safe.com/questions/24394/how-to-write-a-python-script-to-log-fme-error-to-t.html

I have it using ctypes to bring up a dialog box.

Successful - Displays total features written.

0684Q00000ArKJmQAN.png

Failure - Displays failed message

Startup Script

from fmeobjects import *
global my_fme_messages


my_fme_messages = []
 
def LogSkimmer(severity, text):
    if severity in (FME_FATAL, FME_ERROR):
        my_fme_messages.append(text)
 
FMELogFile().setCallBack(LogSkimmer)


Shutdown Script

import fme
import fmeobjects
import ctypes
import winsound


global my_fme_messages
runtime = fme.elapsedRunTime / 60
status = fme.status
totalfeatureswritten = fme.totalFeaturesWritten


# Translation failed message. Filters only relevant messages
if status is False:
    for msg in my_fme_messages:
        if 'logfile' not in msg:
            winsound.PlaySound('SystemExit', winsound.SND_ASYNC) # plays windows sound
            ctypes.windll.user32.MessageBoxA(0, 'FME translation failed' + msg, 'Error', 0)
            
      
# Translation successful message. Custom message    
if status is True:
    ctypes.windll.user32.MessageBoxA(0, 'FME translation was successful. Total features written: ' + str(totalfeatureswritten), 'Successful', 0)
    winsound.PlaySound('SystemExit', winsound.SND_ASYNC) # plays windows sound


roland.martin
Badge +2

I think he (I assume it's a he!) is saying that he can't tell the red from the black, not that he can't see the red text. He's presumably struggling to find where the error has occurred.

So a good solution to me would somehow highlight whereabouts in the log file the problem was. @david_r's idea seems a pretty good start to me.

Some kind of analytical tools relating to the log file might be good, rather than just a crude Find - things like:

  • Jump to previous / next error
  • Match the log line to the transformer
  • etc
carmijo
Rank
Badge +10

I know this isn't the most fun solution but it would seem that the 'Find text' option would be enough to get by for the meantime.

danilo_fme
Rank
Userlevel 4
Badge +30

 

That is very insteresting challenge @Mark2AtSafe :)
bruceharold
Rank
Badge +16

How about an FME Option in Appearance for log error font including colour - way simpler than scripting per workspace.

mark2atsafe
Rank
Userlevel 4
Badge +25

How about an FME Option in Appearance for log error font including colour - way simpler than scripting per workspace.

Yes, that will come for sure. But I was thinking of a quick and dirty solution in the meantime.

 

 

mark2atsafe
Rank
Userlevel 4
Badge +25
Thanks for this folks. I'll post links to these solutions on the StackExchange site. And ten points for Gryffindor! Sorry, I mean 50 points to @david_r and to @madwarren for their great Python solutions. And upvotes for all other suggestions too.

 

redgeographics
Rank
Userlevel 4
Badge +25

Here's a suggestion using the fmeobjects log file callback and some global variables.

In the startup python script:

from fmeobjects import *
 
# Global dictionary used to stor important log messages
global fme_log_messages
fme_log_messages = {FME_WARN: [], FME_ERROR: [], FME_FATAL: []}
 
# Global integer to keep track of current log line number
global fme_log_message_count
fme_log_message_count = 0
 
def catch_log_messages(severity, message):
    global fme_log_message_count
    global fme_log_messages
    fme_log_message_count += 1
    if severity in (FME_WARN, FME_ERROR, FME_FATAL) and message:
        fme_log_messages[severity].append('%8d : %s' % (fme_log_message_count, message))
 
FMELogFile().setCallBack(catch_log_messages)

In the shutdown python script:

from __future__ import print_function
 
global fme_log_messages
no_messages = [' ' * 15 + 'None']
 
print('=' * 80)
print(' Line number   Log message')
print('-' * 80)
print('WARNINGS')
for warning in fme_log_messages[FME_WARN] or no_messages:
    print(warning)
    
print('ERRORS')
for error in fme_log_messages[FME_ERROR] or no_messages:
    print(error)
    
print('FATAL ERRORS')
for fatal in fme_log_messages[FME_FATAL] or no_messages:
    print(fatal)
 
print('=' * 80)

At the end of the translation the shutdown script will output a synopsis of all the warnings, errors and fatal errors with line numbers (referencing the written log), e.g.:

XpVDAOh 

Notice that the synopsis is output using the python 'print' function and that the text is therefore NOT included in the log file, it is only sent to the translation log console window. This is by design, but the script can of course be extended to also write to the log file, if necessary.

If there are no warnings etc, it looks like this:

eL8YZW0 

Tested with Python 2.7 and 3.5.

Template file: fmelogsummarywriter.fmwt

Works a charm @david_r. I think I'll add this to my default template from now on.

 

david_r
Rank
Userlevel 4
Works a charm @david_r. I think I'll add this to my default template from now on.

 

Cool, glad to hear it's useful!
T

Hey all!

 

 

Thanks for taking this issue so seriously :) It was me posting the question on stack exchange and last week I also asked for it here. Richard from the support here told me he made a change request for newer versions of FME as well.

@rollo It is indeed the fact that I cant (or hardly can) tell red from black. If I would scroll carefully line by line I could find the red lines (it is like searching the 90% grey value line in all 100% black lines)

@carmijo not in all cases the words 'error' or 'warning' are used when something is going wrong so searching for those words is not a guarantee of finding all red parts. Like in this example:

@bruceharold that would be the best solution I think.

david_r
Rank
Userlevel 4

Hey all!

 

 

Thanks for taking this issue so seriously :) It was me posting the question on stack exchange and last week I also asked for it here. Richard from the support here told me he made a change request for newer versions of FME as well.

@rollo It is indeed the fact that I cant (or hardly can) tell red from black. If I would scroll carefully line by line I could find the red lines (it is like searching the 90% grey value line in all 100% black lines)

@carmijo not in all cases the words 'error' or 'warning' are used when something is going wrong so searching for those words is not a guarantee of finding all red parts. Like in this example:

@bruceharold that would be the best solution I think.

Great to hear from the OP, thanks for filling us in on the details.

 

Regarding the FME log console, perhaps it can be helpful to you if you activate the timestamp information:

 

 

 

That way you also get the log entry type:

 

 

 

I find it easier to search for specific message types that way.
davideagle
Rank
Badge +10
No answer, just a comment to tie these together but there's a related idea over here https://knowledge.safe.com/idea/21940/fme-workbench-accessibility-options-color-blindnes.html. Also, a bit of thanks as this thread and the comments on Twitter helped me to find out about the 'Deuteranopia' theme that appears to have sneakily appeared in FME2017 to help work with Red\\Green colour blindness. More here http://www.color-blindness.com/deuteranopia-red-green-color-blindness/. Thanks!

 

 

katt
Rank
Badge +8

I'm new to Python as well and based my script on  @david_r 's code from this post: https://knowledge.safe.com/questions/24394/how-to-write-a-python-script-to-log-fme-error-to-t.html

I have it using ctypes to bring up a dialog box.

Successful - Displays total features written.

0684Q00000ArKJmQAN.png

Failure - Displays failed message

Startup Script

from fmeobjects import *
global my_fme_messages


my_fme_messages = []
 
def LogSkimmer(severity, text):
    if severity in (FME_FATAL, FME_ERROR):
        my_fme_messages.append(text)
 
FMELogFile().setCallBack(LogSkimmer)


Shutdown Script

import fme
import fmeobjects
import ctypes
import winsound


global my_fme_messages
runtime = fme.elapsedRunTime / 60
status = fme.status
totalfeatureswritten = fme.totalFeaturesWritten


# Translation failed message. Filters only relevant messages
if status is False:
    for msg in my_fme_messages:
        if 'logfile' not in msg:
            winsound.PlaySound('SystemExit', winsound.SND_ASYNC) # plays windows sound
            ctypes.windll.user32.MessageBoxA(0, 'FME translation failed' + msg, 'Error', 0)
            
      
# Translation successful message. Custom message    
if status is True:
    ctypes.windll.user32.MessageBoxA(0, 'FME translation was successful. Total features written: ' + str(totalfeatureswritten), 'Successful', 0)
    winsound.PlaySound('SystemExit', winsound.SND_ASYNC) # plays windows sound


@cwarren Thank you very much. This is very helpful. I am using the ctypes.windll.user32.MessageBoxA as you do now. But in my case only the first letter of my title and of my text are displayed. Why? I even changed the title and text to some hardcoded strings but it doesn't help. I haven't figured out how to fix it.

N
Badge

Here's a suggestion using the fmeobjects log file callback and some global variables.

In the startup python script:

from fmeobjects import *
 
# Global dictionary used to stor important log messages
global fme_log_messages
fme_log_messages = {FME_WARN: [], FME_ERROR: [], FME_FATAL: []}
 
# Global integer to keep track of current log line number
global fme_log_message_count
fme_log_message_count = 0
 
def catch_log_messages(severity, message):
    global fme_log_message_count
    global fme_log_messages
    fme_log_message_count += 1
    if severity in (FME_WARN, FME_ERROR, FME_FATAL) and message:
        fme_log_messages[severity].append('%8d : %s' % (fme_log_message_count, message))
 
FMELogFile().setCallBack(catch_log_messages)

In the shutdown python script:

from __future__ import print_function
 
global fme_log_messages
no_messages = [' ' * 15 + 'None']
 
print('=' * 80)
print(' Line number   Log message')
print('-' * 80)
print('WARNINGS')
for warning in fme_log_messages[FME_WARN] or no_messages:
    print(warning)
    
print('ERRORS')
for error in fme_log_messages[FME_ERROR] or no_messages:
    print(error)
    
print('FATAL ERRORS')
for fatal in fme_log_messages[FME_FATAL] or no_messages:
    print(fatal)
 
print('=' * 80)

At the end of the translation the shutdown script will output a synopsis of all the warnings, errors and fatal errors with line numbers (referencing the written log), e.g.:

XpVDAOh 

Notice that the synopsis is output using the python 'print' function and that the text is therefore NOT included in the log file, it is only sent to the translation log console window. This is by design, but the script can of course be extended to also write to the log file, if necessary.

If there are no warnings etc, it looks like this:

eL8YZW0 

Tested with Python 2.7 and 3.5.

Template file: fmelogsummarywriter.fmwt

Hi  David, Thanks for this code, it works well, however when I run the template, I get a warning.

Warning: not all FMESessions that were created were destroyed before shutdown. This may cause instability.

 

Do other people get this error?

      Is this a concern if running on server?

          Is there a known fix?

 

Many Thanks

 

 

N
Badge

Hi David, Thanks for this code, it works well, however when I run the template, I get a warning.

Warning: not all FMESessions that were created were destroyed before shutdown. This may cause instability.

 

Do other people get this error?

Is this a concern if running on server?

Is there a known fix?

 

Many Thanks

 

 

I found clearing variables at the end of script resolved this warning. All good.

david_r
Rank
Userlevel 4

Hi  David, Thanks for this code, it works well, however when I run the template, I get a warning.

Warning: not all FMESessions that were created were destroyed before shutdown. This may cause instability.

 

Do other people get this error?

      Is this a concern if running on server?

          Is there a known fix?

 

Many Thanks

 

 

I've updated the startup script above to avoid the issue. Basically replace these two lines:

fme_log = FMELogFile()
fme_log.setCallBack(catch_log_messages)

With this single line:

FMELogFile().setCallBack(catch_log_messages)

And the message should disappear.

S

Jxnkz ox ​

S

Mm kj om o on XpVDAOh​

Reply


Community Stats

28,664
Posts
109,336
Replies
36,532
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.