Add a Rejected port to FeatureWriter

I would very much like a rejected port. Stopping with a fatal error means I can't gracefully handle the issue.

I would also like it. When a database rejects a feature, I prefer to handle it to failing the workspace or silently reject it.

Another use case: Be able to handle logged warnings when e.g. constraint violations occur. These are currently difficult to handle in an easy way.

I think this would be an excellent addition, for a collection of reasons:

• Loop re-attempts in a custom transformer if the writer fails
• Custom logs attached to rejected port for more intuitive feedback for users
• Avoid having workbenches fail upon a writer error, effectively giving more options for graceful error handling.
• Enable additional logic to attempt to resolve the cause before failing (check if DB has fallen over, network drive has disconnected, etc....)
• And more.....

When automating things this would also help to control workflow quality.

If you are writing data to some place which is not managed by yourself and someone changes something (eg. table/feature name, column/attribute names, grants etc.), then having info about that through "Rejected" port would help signal the problem via your preferred channel (email, rest) and/or possibly re-create the target if applicable ...

In fact, similar port should be in Writers too.

this would help us to identify an issue we are experiencing with corrupting file geodatabases

Please note there is a more general idea too. https://community.safe.com/s/idea/0874Q000000Tl3kQAC/detail

I’ve tested this in FME 2022.2.4.0, with a NoFeaturesTester added to the Summary port of the FeatureWriter to fire if nothing came out of the Summary port.

Specifying a read only ArcSDE connection file for the FeatureWriter results in a Workspace termination, regardless of whether Rejected Feature Handling is set to Terminate or Contine - presumably because there is no <Rejected> port on the FeatureWriter to reject features through, therefore the Rejected Feature Handling setting is not invoked.  So the translation never gets as far as the NoFeaturesTester.

The same thing happens with a FeatureWriter set to write an XLSX.

One scenario where the NoFeaturesTester does work is if no features get sent to the FeatureWriter.

@safe I’ve arrived at the catch 22 of this situation.

My scenario is that we have a lot of data updates that take place, and we have a general approach to doing this: 1) archive the current version of the data from the live ArcSDE db to an archive ArcSDE db; 2) Overwrite live ArcSDE data with new data (from whatever source it comes from).  I only want to proceed to step 2 if step 1 has completed successfully.  And I want to send a notification at the end confirming success or failure of the update.  Sometimes I also produce a validation report and want to include the path to that in the notification.

So far we have achieved this by having a parent Workspace with 2 child Workspaces (one for archiving, one for updating).  This is good when you’re running it in desktop, because ANY failure of the archiving Workspace can be detect via the Failed port of the WorkspaceRunner in the parent, and the translation can be handled accordingly (don’t run the update Workspace, send a failure email for archiving).

But if we wanted to move the process on to FME Server, we can’t use WorkspaceRunners.  So we’d either have to faff about changing them to JobSubmitters, which would mean developing/updating Workspaces would take more time.  Or we could put everything in one Workspace and use FeatureReaders and FeatureWriters instead of Readers and Writers.  But we can’t adequately handled failure with this option, because there’s no <Rejected> port on the FeatureWriter.  That’s the catch 22.

So if writing the data to the archive db fails because it can’t get an ArcGIS licence, a child Workspace will fail which can be detected in the parent.  But with a FeatureWriter, the translation will just fail (even with Rejected Feature Handling set to “Continue Translation”, and any Emailers etc downstream will not do anything, because the translation never gets there.  As previously mentioned, NoFeaturesTester can cover some failure scenarios, but not all.

I’ve got as far as making a dynamic Workspace which I can use as a generic ArcSDE archiver.  I then moved on to start turning it into a custom transformer to plug in to a single Workspace solution, but the FeatureWriter issue means it’s really hard to adequately catch errors and direct to the <Rejected> port on my custom transformer.

A rejected port would be ideal, though at the minimum it must be made possible to stop a FeatureWriter error causing a FATAL error stop with no way to trap the error. I have spent the last few weeks kicking off overnight processing runs of multiple database datasets of varying quality, only to find each morning that it has ground to a halt part-way through. My only options at the moment are to re-run another night with problematic data manually masked out, or to write ever more complicated FME workspaces to try to identify things that will crash FeatureWriter.

Note to others: the other “ideas” referenced in above posts have all been archived; if anyone knows of any alternative “ideas” that may help solve this problem grateful if they would add links in this chat.

This is of interest to me as well.

I’m in a situation where I want to do some post-processing after a FeatureWriter has written data. I’ve included several FeatureWriters, one for each feature type, in an attempt to catch and do no post-processing for feature types that fail, but continue with post-processing on feature types that write successfully. I’ve come to realize that any FeatureWriter that fails, stops the entire workspace.

I’ve also tried setting “Ignore Failed Feature Types” to Yes on my FeatureWriters (Geodatabase SDE), but what that ends up doing is including all features, failed or successful, in the summary and feature type output port, misrepresenting the number of features actually successfully written (if any failed).

Does anyone have any ideas on this one?

A Rejected port on my FeatureWriters in this case I think would solve my problem, but it doesn’t look like this is the case - I’m on FME Form v2024.1.1.1.