WorkspaceRunner - Identify when all child processes are complete

As I keep running into this problem myself I decided to try finding a workaround using Python. I put the below code into a PythonCaller and fed the Summary port (from the WorkspaceRunner) into it.

 

 

import fme import fmeobjects import subprocess import re import time pattern = re.compile(r"fme[A-Z]*\\.exe {1,}([0-9]{1,})") wait_secs = 30 class FeatureProcessor(object):     def __init__(self):         pass     def check_tasks_running(self, processes):         tasks_running = False         for task in subprocess.check_output("tasklist").splitlines():             match = pattern.search(str(task))             if match:                 if match.group(1) in processes:                     tasks_running = True         if tasks_running:             return False             time.sleep(wait_secs)         else:             return True     def input(self,feature):         processes = feature.getAttribute('_processes')         jobs_finished = False         while not jobs_finished:             jobs_finished = self.check_tasks_running(processes)         self.pyoutput(feature)     def close(self):         pass

 

The Summary feature appears to be sent once the last batch of jobs have been kicked off, this includes a list of process IDs. I cross reference this list to what's running on the system, if there is a match then that means one or more jobs are still running, so I keep checking until that's no longer the case. The wait was added to make it a bit more sensible, e.g. if you know your jobs will take hours then there's no point checking multiple times a second. In my case the jobs took minutes each so I opted to check every 30 seconds.

 

 

I'm very much a beginner in Python so I'm sure this could have been done in a much nicer way...