Skip to main content
Question

How to wait for (log) file to finish ?

lifalin2016
Contributor
Forum|alt.badge.img+29

Hi,

If I create one master workspace, that runs a number of sub-workspaces, these start and create their own log file. Each workspace will run in its own core/thread, which performs faster.

If I (in the WorkspaceRunner) select to wait for each sub-workspace to finish, all sub-workspaces are run sequentially, one after the other, in a single thread. This is not what I want.

If I want the master workspace to wait for all sub-processes to finish (without blocking other processes), e.g.. by waiting for them to release the write access to their log files, how would I go about that ? I want to be able to gather and check the content of all log files for some after-process Q&A.;

I've tried to utilize Python to test for write access ("if not os.access(filename, os.W_OK):"), but it doesn't seem to work.

Has anyone succesfully done something of this sort ?

Cheers

 

Lars I.
-- Cheers, Lars I.

11 replies

verdoodtdries
Supporter
Forum|alt.badge.img+18

Hi @lifalin2016,

Can your problem be resolved by connecting the three output ports of the WorkspaceRunner transformer with a FeatureHolder?

takashi
Influencer
  • takashiInfluencer
  • October 12, 2016

Hi @lifalin2016, it seems to be difficult to directly detect log file completion. However, log file of a workspace will be closed before the shutdown process, so creation of a file while the shutdown process may be used as a signal of log file completion.

If the master workspace generates unique signal file path and passes it to the sub workspace through the WorkspaceRunner (i.e. through user parameter of the sub workspace), a PythonCaller after the WorkpaceRunner can wait for creation of the signal files. e.g.

import os, time
class FeatureProcessor(object):
    def __init__(self):
        self.features = []
        
    def input(self, feature):
        self.features.append(feature)
        
    def close(self):
        for feature in self.features:
            path = feature.getAttribute('_signal_file_path')
            while not os.path.exists(path):
                time.sleep(0.2)
            self.pyoutput(feature)
david_r
Celebrity
  • david_rCelebrity
  • October 12, 2016

In addition to the other answers, you could also simply read the log file and look for the following text on position 33 of each line:

INFORM|END - ProcessID:

This would signal the last line of the log file after the workspace has completed. If you don't find it, wait a couple of seconds and try again.

lifalin2016
Contributor
Forum|alt.badge.img+29
  • lifalin2016Contributor
  • Author
  • Contributor
  • October 13, 2016
verdoodtdries wrote:

Hi @lifalin2016,

Can your problem be resolved by connecting the three output ports of the WorkspaceRunner transformer with a FeatureHolder?

Hi @verdoodtdries

 

 

Using a FeatureHolder after each launch defeats my purpose, as I want to submit all workspaces more-or-less simultaneously, and only wait for their completion after all have launched.

 

 

And if I only add a single FeatureHolder after workspace #4, I don't think that I can be sure that the other workspaces have finished. Or am I missing something here?

 

 

Cheers

 

Lars I.

 

-- Cheers, Lars I.
lifalin2016
Contributor
Forum|alt.badge.img+29
  • lifalin2016Contributor
  • Author
  • Contributor
  • October 13, 2016
david_r wrote:

In addition to the other answers, you could also simply read the log file and look for the following text on position 33 of each line:

INFORM|END - ProcessID:

This would signal the last line of the log file after the workspace has completed. If you don't find it, wait a couple of seconds and try again.

Hi @david_r

 

 

How do I wait a couple of seconds in the workspace without hanging or blocking the process ?

 

 

Cheers

 

Lars I

 

-- Cheers, Lars I.
lifalin2016
Contributor
Forum|alt.badge.img+29
  • lifalin2016Contributor
  • Author
  • Contributor
  • October 13, 2016
david_r wrote:

In addition to the other answers, you could also simply read the log file and look for the following text on position 33 of each line:

INFORM|END - ProcessID:

This would signal the last line of the log file after the workspace has completed. If you don't find it, wait a couple of seconds and try again.

Ah, just saw the Python command "time.sleep()" mentioned in @takashi

 

's post. That might do the trick.

 

-- Cheers, Lars I.
lifalin2016
Contributor
Forum|alt.badge.img+29
  • lifalin2016Contributor
  • Author
  • Contributor
  • October 13, 2016
takashi wrote:

Hi @lifalin2016, it seems to be difficult to directly detect log file completion. However, log file of a workspace will be closed before the shutdown process, so creation of a file while the shutdown process may be used as a signal of log file completion.

If the master workspace generates unique signal file path and passes it to the sub workspace through the WorkspaceRunner (i.e. through user parameter of the sub workspace), a PythonCaller after the WorkpaceRunner can wait for creation of the signal files. e.g.

import os, time
class FeatureProcessor(object):
    def __init__(self):
        self.features = []
        
    def input(self, feature):
        self.features.append(feature)
        
    def close(self):
        for feature in self.features:
            path = feature.getAttribute('_signal_file_path')
            while not os.path.exists(path):
                time.sleep(0.2)
            self.pyoutput(feature)
Hi @takashi

 

 

Thanks for the example.

 

 

Using a file to signal something is a strategy I did think about, but it's very old-school, and not very elegant. I had hoped for some better and more elegant idea, but maybe it's the only way.

 

 

I assume that the features in your example are the "succeeded" features from each workspace, that I get immidiatedly after launching the workspaces (because I don't want to wait for them to complete) ? The attribute "_signal_file_path" should then be unique to each workspace, correct ?

 

 

Anyway, thanks for the tip on "time.sleep()", that tidbit eluded me when I looked into using Python myself.

 

 

Cheers

 

Lars I.

 

-- Cheers, Lars I.
lifalin2016
Contributor
Forum|alt.badge.img+29
  • lifalin2016Contributor
  • Author
  • Contributor
  • October 13, 2016
david_r wrote:

In addition to the other answers, you could also simply read the log file and look for the following text on position 33 of each line:

INFORM|END - ProcessID:

This would signal the last line of the log file after the workspace has completed. If you don't find it, wait a couple of seconds and try again.

Hi again @david_r,

 

 

Do you know whether it's possible to read the log file (multiple times) while the workspace is still running ?

 

 

Cheers

 

Lars I.

 

-- Cheers, Lars I.
verdoodtdries
Supporter
Forum|alt.badge.img+18
verdoodtdries wrote:

Hi @lifalin2016,

Can your problem be resolved by connecting the three output ports of the WorkspaceRunner transformer with a FeatureHolder?

Hi Lars @lifalin2016,,

 

In that case, you can combine all the Summary output-ports of the WorkspaceRunner transformer to one FeatureHolder.

 

I think that can be sufficient to test if all sub-workspaces have finished.

 

 

If not, you can add a PythonCaller with a script that executes following while-loop:

 

- Read all sub-workspace logfiles.

 

- Check whether the sub-workspace has finished by searching for the specific line in each logfile as @david_r suggested.

 

- Continue or wait for a moment with the time.sleep() command as @takashi suggested.

 

Good luck !

 

takashi
Influencer
  • takashiInfluencer
  • October 13, 2016
lifalin2016 wrote:
Hi @takashi

 

 

Thanks for the example.

 

 

Using a file to signal something is a strategy I did think about, but it's very old-school, and not very elegant. I had hoped for some better and more elegant idea, but maybe it's the only way.

 

 

I assume that the features in your example are the "succeeded" features from each workspace, that I get immidiatedly after launching the workspaces (because I don't want to wait for them to complete) ? The attribute "_signal_file_path" should then be unique to each workspace, correct ?

 

 

Anyway, thanks for the tip on "time.sleep()", that tidbit eluded me when I looked into using Python myself.

 

 

Cheers

 

Lars I.

 

Exactly. The TempPathnameCreator (FME 2016+) is useful to create unique file paths for each feature.

 

david_r
Celebrity
  • david_rCelebrity
  • October 13, 2016
lifalin2016 wrote:
Hi again @david_r,

 

 

Do you know whether it's possible to read the log file (multiple times) while the workspace is still running ?

 

 

Cheers

 

Lars I.

 

Sure, just be sure to close and re-open the file handle for each iteration.

 

Reply


Helpful Members This Week

Community Stats

31,330
Posts
118,863
Replies
38,957
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.