Skip to main content
Solved

PythonCaller error

  • October 28, 2013
  • 5 replies
  • 141 views

Forum|alt.badge.img
We are using a PythonCaller to get some simple info from an input file to be used to validate the file is OK to process:

 

 

import hashlib import fmeobjects     def fileInfo(feature):     log = fmeobjects.FMELogFile()     try:         dir = FME_MacroValues['SourceDirectory']         fileName = FME_MacroValues['SourceTileId']         file = dir + "\\\\" + fileName + ".TIF"                  digest = hashlib.md5(open(file, 'rb').read()).hexdigest()                  fileSize = os.path.getsize(file) >> 20                  feature.setAttribute("actual_fileSize", fileSize)         feature.setAttribute("actual_md5", digest)         pass     except Exception, e:         log.logMessageString(str(e), fmeobjects.FME_ERROR)

 

 

Even though this works and we can test on the 2 attributes created we get the following errors:

 

  Bridge failed to output feature on tag `PYOUTPUT' PythonFactory failed to process feature PythonFactory failed to process feature

 

 

and so the validation messages I want to get out are lost in the log.

 

 

Any ideas what I am doing wrong here?

 

Best answer by osmark

Ok solved this by moving the code to fetch md5 and file size to scripted parameters, then using these parameters in my testers.

 

 

Thanks for your help David

 

 

mark
View original
Did this help you find an answer to your question?
This post is closed to further activity.
It may be a question with a best answer, an implemented idea, or just a post needing no comment.
If you have a follow-up or related question, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

5 replies

Forum|alt.badge.img
  • Author
  • October 28, 2013
Just to clarify. There is a tester and terminator if the 2 extracted values do not match the expected values and the error occurs if the terminator is hit. If the tests pass the workspace completes successfully.

 

 

I have just tried this with a python class as well:

 

 

class FileInfo(object):     def __init__(self):        pass       def input(self,feature):        try:            dir = FME_MacroValues['SourceDirectory']            fileName = FME_MacroValues['SourceTileId']            file = dir + "\\\\" + fileName + ".TIF"                        digest = hashlib.md5(open(file, 'rb').read()).hexdigest()                        fileSize = os.path.getsize(file) >> 20                        feature.setAttribute("actual_fileSize", fileSize)            feature.setAttribute("actual_md5", digest)                        self.feature = feature          except:             print "Exception in user code:"             print '-'*60             traceback.print_exc(file=sys.stdout)             print '-'*60          def close(self):         self.pyoutput(self.feature)

 

  and get:

 

 

Validation_Terminator: Aborting Translation as directed by mapping file. Message is 'Image md5 checksum does not match that supplied;'.

Aborting Translation as directed by mapping file. Message is '%0'.

PythonFactory failed to close properly

PythonFactory failed to close properly

A fatal error has occurred. Check the logfile above for details


david_r
Evangelist
  • October 28, 2013
Hi,

 

 

first of all, if you use a Python class definition, you probably want to output every feature that passes through, not just the last. Modify your code like this:

 

 

class FileInfo(object):     def __init__(self):        pass       def input(self,feature):        try:            dir = FME_MacroValues['SourceDirectory']            fileName = FME_MacroValues['SourceTileId']            file = dir + "\\\\" + fileName + ".TIF"                        digest = hashlib.md5(open(file, 'rb').read()).hexdigest()                        fileSize = os.path.getsize(file) >> 20                        feature.setAttribute("actual_fileSize", fileSize)            feature.setAttribute("actual_md5", digest)                        self.pyoutput(feature)          except:             print "Exception in user code:"             print '-'*60             traceback.print_exc(file=sys.stdout)             print '-'*60          def close(self):         self.pyoutput(self.feature)

 

 

I also suspect that there might be a typo in your filesize calculation:

 

    fileSize = os.path.getsize(file) >> 20

 

 

It should perhaps read:

 

    fileSize = os.path.getsize(file) >> 0x20

 

 

This is to force a hexadecimal interpretation of the "20" literal. Although I'm not quite sure I understand why you shift the result to begin with... :-)

 

 

David

Forum|alt.badge.img
  • Author
  • October 29, 2013
I shift the result to get an answer in MB as the min/max are defined in MB.

 

 

I made the changes you suggested and although I no longer get the errors above the output is not as expected.

 

 

To explain the flow further:

 

1) code above is run to find md5 checksum and file size in MB

 

2) 2 testers compare the result with expected values passed to the workspace and a string is generated on failure (using a string concatenator)

 

3) A tester checks the existence of the above string and passes to a terminator if is found.

 

 

Now before adding this new validation check (others in the flow check image attributes) the output if tests failed included the messages from the terminator and the workspace failed. But for some reason with the python caller introduced I don't get a translation failure I get successful and the message isn't shown, even though the UI shows the process running to the terminator.

Forum|alt.badge.img
  • Author
  • Best Answer
  • October 29, 2013
Ok solved this by moving the code to fetch md5 and file size to scripted parameters, then using these parameters in my testers.

 

 

Thanks for your help David

 

 

mark

david_r
Evangelist
  • October 29, 2013
Hi,

 

 

good to hear that you found a solution.

 

 

Just be aware that by shifting bytes is an integer operation, i.e. the result will 0 when the filesize is less than 1MB, etc.

 

 

If you need fractional filesizes, consider the formula

 

    size_in_mb = size_in_bytes / float(2**20)

 

 

David

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings