why does python "close" function wait for the termination of the whole workspace ?

jeroenstiers wrote:

Could you provide us the Workspace? I would like to take a look at how you tested this.

Thanks!

You can reproduce using something like this and some print statements in the PythonCaller:

 

 

 

With the Terminator enabled the following is called:

__init__()
input()

 

If you disable the Terminator:

 

__init__()
input()
input()
close()

david_r wrote:

FME doesn't call the close() method before it has terminated the translation, e.g. at the end of the whole workspace. If the workspace crashes before the end, the close() method doesn't get called.

Then how does a python caller in the middle of the translation closes ? I forced mine to properly close by simply adding a logger with no input after the python caller... I observed this waiting only when there was no transformer/logger/etc. following the python caller, when it was a "terminal leaf" of my workflow tree.

 

 

jeroenstiers wrote:

Could you provide us the Workspace? I would like to take a look at how you tested this.

Thanks!

I had mongodb updates in the python callers, the "feature" function was stacking a bunch of write operations in a bulk operation, then the "close" function would execute the bulk. I figured out that even there was nothing sent to mongo even if the python had passed every feature for a while before the error.

 

vchalmel wrote:

Then how does a python caller in the middle of the translation closes ? I forced mine to properly close by simply adding a logger with no input after the python caller... I observed this waiting only when there was no transformer/logger/etc. following the python caller, when it was a "terminal leaf" of my workflow tree.

 

 

I'm not sure I understand what you mean. Can you give a concrete example, or even better, a sample workspace that demonstrates the issue?

 

vchalmel wrote:

I had mongodb updates in the python callers, the "feature" function was stacking a bunch of write operations in a bulk operation, then the "close" function would execute the bulk. I figured out that even there was nothing sent to mongo even if the python had passed every feature for a while before the error.

 

Well, yes, as you can see the criteria for when FME calles the close() method isn't when there are no more features going through the PythonCaller, but when the entire workspace has ended. This is by design.

jeroenstiers wrote:

Could you provide us the Workspace? I would like to take a look at how you tested this.

Thanks!

Thanks for sharing @david_r

 

I wasn't aware of this.

 

david_r wrote:

I'm not sure I understand what you mean. Can you give a concrete example, or even better, a sample workspace that demonstrates the issue?

 

Well, to make it simple, consider the example given by the PythonCaller Documentation

 

import fmeobjects   class FeatureProcessor(object): 	def __init__(self): 		self.featureList = [] 		self.totalArea = 0.0   	def input(self,feature): 		self.featureList.append(feature) 		self.totalArea += feature.getGeometry().getArea()   	def close(self): 	 	for feature in self.featureList: 	 	feature.setAttribute("total_area", self.totalArea) 	 	self.pyoutput(feature) 

How could the following transformer receive its input, considering that self.pyoutput(feature) is in the closing function, if the closing function is waiting for the termination of the translation ? 

 

So I must conclude that the closing function is indeed called right after every featured is processed if and only if there is a transformer following the python caller.

 

david_r wrote:

Well, yes, as you can see the criteria for when FME calles the close() method isn't when there are no more features going through the PythonCaller, but when the entire workspace has ended. This is by design.

Oups, that wasn't an answer to your howto comment, I was explaining to jeroenstiers how i noticed this.

 

vchalmel wrote:

Well, to make it simple, consider the example given by the PythonCaller Documentation

 

import fmeobjects   class FeatureProcessor(object): 	def __init__(self): 		self.featureList = [] 		self.totalArea = 0.0   	def input(self,feature): 		self.featureList.append(feature) 		self.totalArea += feature.getGeometry().getArea()   	def close(self): 	 	for feature in self.featureList: 	 	feature.setAttribute("total_area", self.totalArea) 	 	self.pyoutput(feature) 

How could the following transformer receive its input, considering that self.pyoutput(feature) is in the closing function, if the closing function is waiting for the termination of the translation ? 

 

So I must conclude that the closing function is indeed called right after every featured is processed if and only if there is a transformer following the python caller.

 

Excellent catch, I was wrong when saying that the close() method gets called when the workspace has terminated, it isn't strictly correct, which you've pointed out. The close() method gets called when there are no more features to process*, which is an important distinction.

 

 

From the doc:

 

The PythonCaller will call two methods on the class: input() and close(). The input() method will be called for each FMEFeature that comes into the input port. When no more FMEFeatures remain, the close() method will be called. Features that need to continue through the workspace for further processing must be explicitly written out using the pyoutput() method.

 

 

*) We'd need Safe to give us the definitive answer to exactly what the criteria is, but according to my testing the close() method gets called as soon as FME is certain that no additional feature will ever enter the PythonCaller during that execution. So it might not be at the end of the translation, but it will be guaranteed to be the end of the translation for that particular PythonCaller.

erik_jan wrote:

As @david_r says. And one of the reasons is that the close function needs to be able to access workspace statistics and log file and those are produced last by the workspace.

Sorry, this is true for the Python Shutdown script and not for the PythonCaller.