• EN
Solved

PythonCaller error

  • 28 October 2013
  • 5 replies
  • 52 views
O
Badge
We are using a PythonCaller to get some simple info from an input file to be used to validate the file is OK to process:

 

 

import hashlib import fmeobjects     def fileInfo(feature):     log = fmeobjects.FMELogFile()     try:         dir = FME_MacroValues['SourceDirectory']         fileName = FME_MacroValues['SourceTileId']         file = dir + "\\\\" + fileName + ".TIF"                  digest = hashlib.md5(open(file, 'rb').read()).hexdigest()                  fileSize = os.path.getsize(file) >> 20                  feature.setAttribute("actual_fileSize", fileSize)         feature.setAttribute("actual_md5", digest)         pass     except Exception, e:         log.logMessageString(str(e), fmeobjects.FME_ERROR)

 

 

Even though this works and we can test on the 2 attributes created we get the following errors:

 

  Bridge failed to output feature on tag `PYOUTPUT' PythonFactory failed to process feature PythonFactory failed to process feature

 

 

and so the validation messages I want to get out are lost in the log.

 

 

Any ideas what I am doing wrong here?

 

icon

Best answer by osmark 29 October 2013, 10:24

View original

5 replies

O
Badge
Just to clarify. There is a tester and terminator if the 2 extracted values do not match the expected values and the error occurs if the terminator is hit. If the tests pass the workspace completes successfully.

 

 

I have just tried this with a python class as well:

 

 

class FileInfo(object):     def __init__(self):        pass       def input(self,feature):        try:            dir = FME_MacroValues['SourceDirectory']            fileName = FME_MacroValues['SourceTileId']            file = dir + "\\\\" + fileName + ".TIF"                        digest = hashlib.md5(open(file, 'rb').read()).hexdigest()                        fileSize = os.path.getsize(file) >> 20                        feature.setAttribute("actual_fileSize", fileSize)            feature.setAttribute("actual_md5", digest)                        self.feature = feature          except:             print "Exception in user code:"             print '-'*60             traceback.print_exc(file=sys.stdout)             print '-'*60          def close(self):         self.pyoutput(self.feature)

 

  and get:

 

 

Validation_Terminator: Aborting Translation as directed by mapping file. Message is 'Image md5 checksum does not match that supplied;'.

Aborting Translation as directed by mapping file. Message is '%0'.

PythonFactory failed to close properly

PythonFactory failed to close properly

A fatal error has occurred. Check the logfile above for details

david_r
Rank
Userlevel 5
Hi,

 

 

first of all, if you use a Python class definition, you probably want to output every feature that passes through, not just the last. Modify your code like this:

 

 

class FileInfo(object):     def __init__(self):        pass       def input(self,feature):        try:            dir = FME_MacroValues['SourceDirectory']            fileName = FME_MacroValues['SourceTileId']            file = dir + "\\\\" + fileName + ".TIF"                        digest = hashlib.md5(open(file, 'rb').read()).hexdigest()                        fileSize = os.path.getsize(file) >> 20                        feature.setAttribute("actual_fileSize", fileSize)            feature.setAttribute("actual_md5", digest)                        self.pyoutput(feature)          except:             print "Exception in user code:"             print '-'*60             traceback.print_exc(file=sys.stdout)             print '-'*60          def close(self):         self.pyoutput(self.feature)

 

 

I also suspect that there might be a typo in your filesize calculation:

 

    fileSize = os.path.getsize(file) >> 20

 

 

It should perhaps read:

 

    fileSize = os.path.getsize(file) >> 0x20

 

 

This is to force a hexadecimal interpretation of the "20" literal. Although I'm not quite sure I understand why you shift the result to begin with... :-)

 

 

David
O
Badge
I shift the result to get an answer in MB as the min/max are defined in MB.

 

 

I made the changes you suggested and although I no longer get the errors above the output is not as expected.

 

 

To explain the flow further:

 

1) code above is run to find md5 checksum and file size in MB

 

2) 2 testers compare the result with expected values passed to the workspace and a string is generated on failure (using a string concatenator)

 

3) A tester checks the existence of the above string and passes to a terminator if is found.

 

 

Now before adding this new validation check (others in the flow check image attributes) the output if tests failed included the messages from the terminator and the workspace failed. But for some reason with the python caller introduced I don't get a translation failure I get successful and the message isn't shown, even though the UI shows the process running to the terminator.
O
Badge
Ok solved this by moving the code to fetch md5 and file size to scripted parameters, then using these parameters in my testers.

 

 

Thanks for your help David

 

 

mark
david_r
Rank
Userlevel 5
Hi,

 

 

good to hear that you found a solution.

 

 

Just be aware that by shifting bytes is an integer operation, i.e. the result will 0 when the filesize is less than 1MB, etc.

 

 

If you need fractional filesizes, consider the formula

 

    size_in_mb = size_in_bytes / float(2**20)

 

 

David

Reply


Community Stats

28,812
Posts
109,833
Replies
36,629
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.