How to order features and schemas for a FeatureWriter

Hi ​@chau,

The SchemaScanner has an option to output Schema features before Data features. Set this to Yes, then route the Schema output directly to the FeatureWriter. The Schema features rarely need to be modified after scanning, so this should get them to the FeatureWriter before any of the data features.

But I am in one of the rare situations where I do need to modify the schema features which due to blocking transformers does not guarantee that the schema feature arrives first.

Set the creator to create X features. 

Using the default @Value(_creation_instance) will output values 1 onwards

TestFilter on this instance count value. Connect each port to the FeatureReader/SchemaScanner transformer in the order of operations you want

I don’t think I completely understand this approach?

ok, just a plan to manage the initiating of data streams, if you have many featurereader and featurewriters 

What operations are you doing to the schema features? Perhaps we can find a way to perform those without blocking.

I didn’t find any satisfactory method when i was doing something similar.

My final workflow involved running the workspace once per schema with a workspace runner in a parent workspace on form or using a automations writer on flow where i could make use of a  holder to ensure the schema hit the writer first, without it having to hold everything in memory.

​@daveatsafe - the reason for altering schemas in the use case we had was to resolve all attributes that were linked to domains and use the attribute name with the resolved value. With the added complication that not all resolved fields are exposed in a schema when reading utility network data.

I probably could rewrite the workspace to work around it, but I think its a valid wish to have something to handle this issue since FME naturally requires a specific order of things in the FeatureWriter. 

Could the FeatureWriter get a new option to “Wait for schema feature”?

That would be a good option to have on the FeatureWriter, since it would simplify the workspace by removing the need for the Sorter. However, it would still need to cache the data features while waiting for the schema, so is unlikely to resolve the memory issues you are seeing.

Ideally, the SchemaScanner should be right before the FeatureWriter, so that no changes should be required to the schema features, and the order will be undisturbed.

But what if the SchemaScanner doesn’t produce the result I want? Then I still need to change the schema feature and then we’re back to the beginning again.

After thinking about the issue again I think I would rather like an option in the FeatureWriter like “Wait for Schema feature”. This way the FeatureWriter can hold the features until the schema feature arrives. Any memory used here would be acceptable because it would be my responsibility to optimise the workspace. Having the race between the features and the schema feature requires me to make workarounds. But by removing that race I can optimise without the workarounds.

Thank you for your perseverance but I have moved on now. I will no doubt run into this issue yet again in the future, so I appreciate you creating the IDEA-3345. I’m looking forward to hear about this.