Skip to main content

I'm struggling with the PythonCaller Transfromer regarding multiprocessing. The script below is a simple example to illustrate my problem. How can I get it running as a PythonCaller script? Currently the translation log says:

File -c could not be opened

Program Terminating

Translation FAILED.

Error running translation.

Translation Aborted

import multiprocessing
import random
import time
 
def worker(t, i, return_dict):
    print("Sleep {} seconds".format(t))
    return_dictii] = t
    time.sleep(t)
 
def multiprocessing_test(n):
    print("in multiprocessing_test")
    manager = multiprocessing.Manager()
    return_dict = manager.dict()
    jobs = u]
    for i in range(n):
        p = multiprocessing.Process(target = worker, args=(random.randint(3,6), i, return_dict,))
        jobs.append(p)
        p.start()
    for p in jobs:
        p.join()
    print(str(return_dict))
 
if __name__ == "__main__":
    def processFeature(feature=None):
        print("in processFeature")
        multiprocessing_test(10)
 
    #processFeature() # Uncomment to test the script with the python interpreter outside of FME. Output is:
# in processFeature
# in multiprocessing_test
# Sleep 4 seconds
# Sleep 6 seconds
# Sleep 3 seconds
# Sleep 6 seconds
# Sleep 6 seconds
# Sleep 4 seconds
# Sleep 6 seconds
# Sleep 5 seconds
# Sleep 6 seconds
# Sleep 6 seconds
# {0: 4, 8: 6, 3: 6, 2: 3, 5: 6, 1: 4, 7: 6, 9: 5, 4: 6, 6: 6}

 

Maybe the answer by @jeroenstiers on the question below may concern your question?

https://community.safe.com/s/question/0D54Q000080hL2TSAU/are-features-ran-through-pythoncaller-sequentially-or-with-multithreading-


What I don't like about this solution is that stdout and stderr are only displayed in the translation log after the script has finished and not during the script. If the script runs for a long time it might seem as the process got stuck. But it is better than doing it without multiprocessing.


Can you flip the problem on its edge? As in, can you let FME handle the multiprocessing and Python handle the processing? I.e. making a custom transformer with just a PythonCaller (and whatever else is needed) inside it. That way, you at least get the parallellism, but still sequential within the group.


Reply