Running FME Workspace reader from WorkspaceRunner

Hi, I have a list of fmw files like this one:

That I want to use to go through and extract some data or each fmw using the FME Workspace (FMW) reader.

To do this, I have created a workbench “FileProperties.fmw” that takes the feature type FileProperties of a workbench using FME Workspace (FMW) reader and puts data to a csv.

Then, because I want to apply this process to multiple workbenches and not just one, I have created a separate workbench "RunWorkbench.fmw" which uses a WorkspaceRunner that runs FileProperties.fmw workbench for each of the files of the list. Each line of the list is used as a variable, so FileProperties workbench analyses each fmw file in the list and extracts the data into csv.

In the WorkspaceRunner configuration I run fmw_properties.fmw. WorkspaceRunner knows which workbenches to analyze each time because in fmw_properties.fmw I created create a published parameter that will pass the workbench to be analysed.

This is how I setup the published parameter in fmw_properties.fmw:

The logic of workflow is correct. However, when running it, it starts reading files, and eventually starts sending errors and warnings and it ends up crashing with some sort of "not enough memory" message. There isn't a clear pattern of which workspaces fail and which succeed. Some of them always work some of the just crash from time to time, others always crash. Have checked those that failed and apparently there're no errors in these workspaces, some of them are quite simple apparently.

Additionally, have run the FileProperties.fmw separately for some of the workspaces that fail, and they run perfectly fine.

Don't know what's going on here, it looks like FME Workspace Reader is pretty much inefficient when ran from WorkspaceRunner. I am surprised that reading an FME Workbench can consume a large amount of memory (it is not reflected in the Windows Task Manager).

Wanted to ask if there's an alternative way of doing this, or if somebody else has found similar issues with WorkspaceRunner/FME Workspace Reader? Also, not sure if I'm doing something wrong... Am I setting up the Published Parameter correctly?

Also, I a interested in reading other FeatureTypes in these workbenches, but can't even make this basic one (FileProperties) work, which is a couple of fields only.

Our current build is FME 2019.0 19246

I attach both workbenches.

Page 1 / 1

Hi @miquelrs,

I have tried your solution with the workspacerunner and can confirm getting errors for some workspaces.

Actually you dont really need a parent- child approach, you can do this in one workspace using the Directory and File Pathnames reader and a FeatureReader to read the workspaces.

Here I am reading all the workspace feature types, this you can adjust to read only the ones you are interested in or keep the generic port for future flexibility.

Hope this helps,

Itay

Hi @miquelrs,

I have tried your solution with the workspacerunner and can confirm getting errors for some workspaces.

Actually you dont really need a parent- child approach, you can do this in one workspace using the Directory and File Pathnames reader and a FeatureReader to read the workspaces.

Here I am reading all the workspace feature types, this you can adjust to read only the ones you are interested in or keep the generic port for future flexibility.

Hope this helps,

Itay

Hi @itay,

Thanks for your answer. I tried your approach as well, reading all or only some of the feature types in the workbenches (so using generic output port or specific ones). However, doing this causes FME to crash due to multiple errors as well. From "Error running translation" to "FME EXE has stopped working" prompt, to more complex error messages. These errors arise now for workspaces that were being read fine with the previous approach.

Also noticed that can't use many output ports in this transformer. The reading process becomes really impossible, again apparently due to inefficiency of reading an FME workspace.

Miquel

Hi @itay,

Also noticed that can't use many output ports in this transformer. The reading process becomes really impossible, again apparently due to inefficiency of reading an FME workspace.

Miquel

Hi @miquelrs,

I agree that the FMW can be 'heavy' but this also depends on your system resources, if several workspaces are causing your system to overload, I suggest having a look at your system resources and temp location.

Another issue can be network capability, I have noticed you are reading the workspaces from an external location, possibly copying the workspaces locally and then reading them might help.

Hi @miquelrs,

Another issue can be network capability, I have noticed you are reading the workspaces from an external location, possibly copying the workspaces locally and then reading them might help.

Wow, I have to say I'm impressed. My system fails in the second workspace, and forget about using feature caching mode. I'll try you suggestions, Thanks itay.

Hmmm. This is nothing that I've heard before, and I'm probably the one at Safe who's used this reader most. I've tried your technique and it seems to work fine for me.

You're only running one workspace at a time in the WorkspaceRunner (wait for completion = yes) so I think the out of memory error is a bunch of hooey. How much memory do you have and what Windows version do you have? That might have an effect.

I have a couple of guesses. Firstly that the problem is in how the paths are being passed - I tried it with unc paths but it worked fine, so maybe something in your paths are different (spaces?)

The other guess is encoding. Maybe the text file has an odd encoding, or maybe it has a BOM character that FME reads and tries to interpret as a filename? Maybe it's the end-of-line marker that is an issue? Basically any of these could corrupt the file name, though I would expect a simple "file not found" rather than an "out of memory" issue.

I usually prefer the technique Itay uses (Directory/Path reader) but I'd like to get to the bottom on your textfile problem.

Can you try and break the process down into steps. Step 1: Can the Data Inspector be used to read a failing workspace with the FMW reader? If not, then we know the reader is the issue.

Step 2: If step 1 works, can you run the helper (child) workspace by itself? Does that work?

Step 3: If step 2 works then try the WorkspaceRunner again, but hard code the file name (don't read it from the txt file and don't set it as an attribute in the WorkspaceRunner parameters).

Basically bit by bit narrow it down to where something fails and see if it's consistent.

Otherwise.... well an "out of memory" error is usually our problem, so please feel free to log a support case with us (safe.com/support) and we'll try harder to diagnose and replicate the problem.

It would be helpful to see the full log file (especially the header part telling us system information) and to maybe get a copy of your text file, to see if there is anything odd about it.

Hi @miquelrs,

Another issue can be network capability, I have noticed you are reading the workspaces from an external location, possibly copying the workspaces locally and then reading them might help.

Hi Itay, after struggling with a lot of errors I ended up going through each workbench, saving it in the current FME version, upgrading all transformers, also replacing readers by newer versions. After this, I got much less errors, but also have improved spec by getting a new machine W10/64bit (previous spec was W7/32bit) and now this reader works much better. It is still a little bit slow to go through each workbench, but at least it doesn't crash randomly due to memory resources.