Running part of a workspace depending on a variable or parameter
There are situations when I want a variable or parameter to determine whether a certain process is run or not. I could add the variable as an attribute to each feature and then test for the value of the variable. But that way the variable controls the process for each feature, when I already know that I don't want put any of the features through the process. And it feels like creating an unneccesary amount of extra data (even though temporarily), which I have to remove later on.
I've come across this before, and now I have a case where I want to run part of my workspace once a week and another part every day. (An alternative is to distribute the processes across separate workspaces, of course.)
What I'm thinking of is functionality like the feature reader has: it only reads features when triggered by an initiator. But I can't think of a way to do this with other transformers.
Does anyone know of a way to achieve this in FME Workbench / Form?
Page 1 / 1
Yes, you can test on user parameter values in a TestFilter just as easily as attribute values.
And for the example you mention, running part of the process on a certain day of the week only, you can make use of the DateTime functions in a TestFilter:
So what that does is it takes a timestamp (DateTimeNow) and formats that as %u, which is the day of week (monday being 1). Alternatively, %a or %A will give you abbreviated or full weekday names, which may be better from a documentation point of view.
That would be my suggestion as well. You can also use Testers or TestFilters on user parameter values, and you can even combine the two options (datetime/user parameter).
You can even use a user parameter or a (formatted) DateTimeNow() in the where-clause of an SQL Creator or SQL Executor - that way you can even prevent FME from loading any data at all.
I’ve been doing this to run weekly reports, using Testers with @DateTimeFormat(@DateTimeNow(),%A) In $(DayOfWeek). This way the DayOfWeek (which could be named better at this point, but does have a more descriptive prompt) can be set at runtime in Form and in Flow, and can accept one or more days of the week, comma separated - the In operator makes it a bit more flexible. For example: Saturday; or: Monday, Wednesday, Friday.
Since I have a number of data flows that need to be constrained, I do have to use a number of Testers, but it has been very useful and not too much overhead to maintain. If there’s a better way to do this at scale, I’d certainly be interested in that!
Some parts of my workspaces need to run weekly, while the rest run daily, hence peppering Testers in as-needed for my use case.
I like to use NoFeatureTesters and the FeatureFlowValve to help control the flow of features. A lot will depend on whether or not these make sense to use in your case. But combining the FeatureFlowValie with what others have suggested about TestFilters and User Parameters can be nice and I think can help with readability later
The feature Flow value is blocking but can be tweaked to be non-blocking (it’s basically just a FeatureMerger).
Thank you all for looking into my question! I'm getting some bonus tips, like using %A to get the name of the day (I like to make my workbenches as self-explanatory as possible). Or using the date constraint for an SQL query to determine whether features are read at all. I will keep that in mind.
And the FeatureFlowValve is exactly what I was looking for! The only objection I have is that it seems to be a transformer from the 32-bit era, so a bit outdated. In the example provided by @redgeographics, my question is how to use the output of the testfilter as a trigger for starting a subprocess within the workspace (or not).
As I mentioned in my opening post, the only way I can think of running either process is using a feature reader, because that transformer has an initiator-input:
In my case, features have already been read previously, and they are needed in all processes. In more detail:
During the week I want to see if there are any changes in features, that need attention;
During the weekend I want to give an overview of all features that still need attention
And there could be processes that run everyday, too. It looks like this, generically:
I've replaced the FeatureReader with a FeatureFlowValve.
Is there a way to do this with current transformers?
Is there a way to do this with current transformers?
If you Embed the FeatureFlowValve (right mouse click → Embed), you can then right mouse click again and edit it, and update the Transformers if you want. It’ll also allow you to see how it works, and it is (like Matt says) basically a FeatureMerger - so you could implement this yourself as well with a FeatureMerger.
It seems to me, @s.jager, that there is no version of the FeatureFlowValve more recent than this:
(copied from the Help for the transformer) I do not get the option to update the transformer. It isn't listed in the Help of FME 2024 either. Am I overlooking something?
Using the FeatureMerger would certainly work. But, as I mentioned in my opening post, I would be adding the same value to each feature as an attribute. I'm looking for a more elegant option, like the FeatureFlowValve. If that transformer was discontinued, my wish is probably not that common.
Am I overlooking something?
Yes. If you Embed the FeatureFlowValve, you can then Edit it, as I explained above. When editing it, you can then upgrade all the Transformers that are inside the Custom Transformer to your current version (a right-click on the Transformer, then at the bottom there’s an Upgrade-option).
If you do not want to do so, you can still Embed and Edit it, and see how it works, then implement that workflow yourself.
Using the FeatureMerger would certainly work. But, as I mentioned in my opening post, I would be adding the same value to each feature as an attribute. I'm looking for a more elegant option, like the FeatureFlowValve.
The FeatureFlowValve IS using a FeatureMerger, as Matt already mentions. It is also using a BulkAttributeRemover, so it looks like no attributes were added to each feature when they come out the Custom Transformer.
Ah! Now I see, thanks @s.jager . So it ends up being this:
The inside of a FeatureFlowValve
The trick used in the FeatureMerger is this:
Nothing specific is used for the merge, and any merged attributes are removed based on the prefix.
It works, but I find it unsatisfactory to join the same value to all features and then drop them again, to prevent all features from being processed when that was the goal to start with.
