Skip to main content
Solved

PythonCaller Transfromer and Multiprocessing


I'm struggling with the PythonCaller Transfromer regarding multiprocessing. The script below is a simple example to illustrate my problem. How can I get it running as a PythonCaller script? Currently the translation log says:

File -c could not be opened

Program Terminating

Translation FAILED.

Error running translation.

Translation Aborted

import multiprocessing
import random
import time
 
def worker(t, i, return_dict):
    print("Sleep {} seconds".format(t))
    return_dict[i] = t
    time.sleep(t)
 
def multiprocessing_test(n):
    print("in multiprocessing_test")
    manager = multiprocessing.Manager()
    return_dict = manager.dict()
    jobs = []
    for i in range(n):
        p = multiprocessing.Process(target = worker, args=(random.randint(3,6), i, return_dict,))
        jobs.append(p)
        p.start()
    for p in jobs:
        p.join()
    print(str(return_dict))
 
if __name__ == "__main__":
    def processFeature(feature=None):
        print("in processFeature")
        multiprocessing_test(10)
 
    #processFeature() # Uncomment to test the script with the python interpreter outside of FME. Output is:
# in processFeature
# in multiprocessing_test
# Sleep 4 seconds
# Sleep 6 seconds
# Sleep 3 seconds
# Sleep 6 seconds
# Sleep 6 seconds
# Sleep 4 seconds
# Sleep 6 seconds
# Sleep 5 seconds
# Sleep 6 seconds
# Sleep 6 seconds
# {0: 4, 8: 6, 3: 6, 2: 3, 5: 6, 1: 4, 7: 6, 9: 5, 4: 6, 6: 6}

 

Best answer by __alex__

Maybe the answer by @jeroenstiers on the question below may concern your question?

https://community.safe.com/s/question/0D54Q000080hL2TSAU/are-features-ran-through-pythoncaller-sequentially-or-with-multithreading-

View original
Did this help you find an answer to your question?

3 replies

__alex__
Contributor
  • Contributor
  • Best Answer
  • January 12, 2021

  • Author
  • January 12, 2021
__alex__ wrote:

What I don't like about this solution is that stdout and stderr are only displayed in the translation log after the script has finished and not during the script. If the script runs for a long time it might seem as the process got stuck. But it is better than doing it without multiprocessing.


Forum|alt.badge.img+1
  • January 14, 2021

Can you flip the problem on its edge? As in, can you let FME handle the multiprocessing and Python handle the processing? I.e. making a custom transformer with just a PythonCaller (and whatever else is needed) inside it. That way, you at least get the parallellism, but still sequential within the group.


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings