Speeding up a workspace runner

Could you read file B and create a ffs file? ffs is the internal file format for FME and is read faster than any other format.

Create a (temporary) workspace and read file B and write to the ffs Writer. Then use the ffs file in your process instead of file B.

No, you can't, unfortuately.

But you should consider the format of file B, as it can make a big difference as some formats are an order of magnitude faster than others. You could try pre-processing file B into the native FFS format and see if that makes a difference.

If that doesn't improve the performance, a more general analysis of your workflow is in order. For instance, if you're repeatedly using a FeatureMerger on large datasets, you might benefit from first loading your data into a relational database and using SQL join queries using indexed fields (either as a view or from an SQLExecutor/SQLCreator), it can make significant difference.

As an addition you might want to look at the FeatureReader transformer.

That transformer allows you to read the part of file B that you need in comparison with file A (and potentially not the complete file B).

Well... I think it partly depends on what the workspace is doing. You say it "matches variables in A to another static file, file B" - does that mean a join on an attribute key? Maybe with the FeatureMerger? And how many features are there? Are there fewer features in A than B? eg 1,000 features in A are matched against 1,000,000 in B?

If so you could replace the FeatureMerger with a Joiner transformer. You'd have to put B into a database format of some sort, even just temporarily, but then you wouldn't have to read all 1,000,000 features from B just to match 1,000 of them.

Similarly you could use a FeatureReader or SQLExecutor or any other transformer that could retrieve selected features from dataset B. So that's one technique: reading B the same number of times but reducing the amount of data you need to read from it.

That's the first law of performance, btw. Performance is defined as "useful work" carried out, so if you're reading data that isn't used, it's not useful work!

Anyway, to read B only once, then I think you would need to just have one run of the workspace, and load all of the data from all of the A files at one time. Basically one big process. How many features are in each A file? Are they non-spatial or very complex geometry? That will be the big issue.

I'm assuming you have your current setup - to read each A file separately - because combined there is too much data. But if not, and you are only doing it as a quick way to batch process data, there are other solutions. Considering reading all of the A files at once, then using a Dataset Fanout on the output to split them back out again. Group-by parameters in transformers can help keep each A file separate if necessary.

So that's a second technique: reading all of the A files at once, so you only need to read B once.

Finally, and this is where my imagination starts to exceed my knowledge, I wonder if you can set up a continuously running workspace. That workspace reads file B and then receives features from all of the file As one at a time. It keeps running as long as you are passing file A features to it.

We do have transformers like that - TCPIPReceiver, SQSReceiver, etc. I'm partly unsure because I don't know how a workspace like that deals with a reader. I assume it reads the data once and then - if you have a group-based transformer - holds that data for processing against any incoming feature, but I don't know for sure. Also, of course, I don't really know what action you are carrying out or how many features you have, or how often you run this process (cause this would take a lot of setting up, but could save you lots of time if you repeat the process daily).

So, that's the third technique: a continuous process that just listens for new A features to compare against B.

There are some other things like this, but they're fairly specialized; for example if you are doing something with a DEM, the SurfaceModeller can save its internal workings as a file for re-use at a later point. And I think there are some web formats/transformers that will cache data locally, so you don't have to keep rereading it from a remote source again and again. But I don't know if they will help you here.

Anyway, I hope something here helps. If you can give us a few more details about what you are doing exactly, then it might be that there's something we can do to help. Other than that, as others have said, use a good, fast format. I seem to recall trying out different formats at one time, and MicroStation v8 was the fastest, but whether that's still true, and whether I included a comparison with FFS, I don't know.

If I have any more bright (or even not-so-bright) ideas, I'll let you know,

Regards

Mark

PS: Thanks for letting me know about this one Bruce. Not quite a blog, but getting on for blog length!

Could you read file B and create a ffs file? ffs is the internal file format for FME and is read faster than any other format.

Create a (temporary) workspace and read file B and write to the ffs Writer. Then use the ffs file in your process instead of file B.