Add a Rejected port to FeatureWriter

I would very much like a rejected port. Stopping with a fatal error means I can't gracefully handle the issue.

I would also like it. When a database rejects a feature, I prefer to handle it to failing the workspace or silently reject it.

Another use case: Be able to handle logged warnings when e.g. constraint violations occur. These are currently difficult to handle in an easy way.

I think this would be an excellent addition, for a collection of reasons:

• Loop re-attempts in a custom transformer if the writer fails
• Custom logs attached to rejected port for more intuitive feedback for users
• Avoid having workbenches fail upon a writer error, effectively giving more options for graceful error handling.
• Enable additional logic to attempt to resolve the cause before failing (check if DB has fallen over, network drive has disconnected, etc....)
• And more.....

When automating things this would also help to control workflow quality.

If you are writing data to some place which is not managed by yourself and someone changes something (eg. table/feature name, column/attribute names, grants etc.), then having info about that through "Rejected" port would help signal the problem via your preferred channel (email, rest) and/or possibly re-create the target if applicable ...

In fact, similar port should be in Writers too.

this would help us to identify an issue we are experiencing with corrupting file geodatabases

Please note there is a more general idea too. https://community.safe.com/s/idea/0874Q000000Tl3kQAC/detail

I’ve tested this in FME 2022.2.4.0, with a NoFeaturesTester added to the Summary port of the FeatureWriter to fire if nothing came out of the Summary port.

Specifying a read only ArcSDE connection file for the FeatureWriter results in a Workspace termination, regardless of whether Rejected Feature Handling is set to Terminate or Contine - presumably because there is no <Rejected> port on the FeatureWriter to reject features through, therefore the Rejected Feature Handling setting is not invoked.  So the translation never gets as far as the NoFeaturesTester.

The same thing happens with a FeatureWriter set to write an XLSX.

One scenario where the NoFeaturesTester does work is if no features get sent to the FeatureWriter.

@safe I’ve arrived at the catch 22 of this situation.

My scenario is that we have a lot of data updates that take place, and we have a general approach to doing this: 1) archive the current version of the data from the live ArcSDE db to an archive ArcSDE db; 2) Overwrite live ArcSDE data with new data (from whatever source it comes from).  I only want to proceed to step 2 if step 1 has completed successfully.  And I want to send a notification at the end confirming success or failure of the update.  Sometimes I also produce a validation report and want to include the path to that in the notification.

So far we have achieved this by having a parent Workspace with 2 child Workspaces (one for archiving, one for updating).  This is good when you’re running it in desktop, because ANY failure of the archiving Workspace can be detect via the Failed port of the WorkspaceRunner in the parent, and the translation can be handled accordingly (don’t run the update Workspace, send a failure email for archiving).

But if we wanted to move the process on to FME Server, we can’t use WorkspaceRunners.  So we’d either have to faff about changing them to JobSubmitters, which would mean developing/updating Workspaces would take more time.  Or we could put everything in one Workspace and use FeatureReaders and FeatureWriters instead of Readers and Writers.  But we can’t adequately handled failure with this option, because there’s no <Rejected> port on the FeatureWriter.  That’s the catch 22.

So if writing the data to the archive db fails because it can’t get an ArcGIS licence, a child Workspace will fail which can be detected in the parent.  But with a FeatureWriter, the translation will just fail (even with Rejected Feature Handling set to “Continue Translation”, and any Emailers etc downstream will not do anything, because the translation never gets there.  As previously mentioned, NoFeaturesTester can cover some failure scenarios, but not all.

I’ve got as far as making a dynamic Workspace which I can use as a generic ArcSDE archiver.  I then moved on to start turning it into a custom transformer to plug in to a single Workspace solution, but the FeatureWriter issue means it’s really hard to adequately catch errors and direct to the <Rejected> port on my custom transformer.

A rejected port would be ideal, though at the minimum it must be made possible to stop a FeatureWriter error causing a FATAL error stop with no way to trap the error. I have spent the last few weeks kicking off overnight processing runs of multiple database datasets of varying quality, only to find each morning that it has ground to a halt part-way through. My only options at the moment are to re-run another night with problematic data manually masked out, or to write ever more complicated FME workspaces to try to identify things that will crash FeatureWriter.

Note to others: the other “ideas” referenced in above posts have all been archived; if anyone knows of any alternative “ideas” that may help solve this problem grateful if they would add links in this chat.

This is of interest to me as well.

I’m in a situation where I want to do some post-processing after a FeatureWriter has written data. I’ve included several FeatureWriters, one for each feature type, in an attempt to catch and do no post-processing for feature types that fail, but continue with post-processing on feature types that write successfully. I’ve come to realize that any FeatureWriter that fails, stops the entire workspace.

I’ve also tried setting “Ignore Failed Feature Types” to Yes on my FeatureWriters (Geodatabase SDE), but what that ends up doing is including all features, failed or successful, in the summary and feature type output port, misrepresenting the number of features actually successfully written (if any failed).

Does anyone have any ideas on this one?

A Rejected port on my FeatureWriters in this case I think would solve my problem, but it doesn’t look like this is the case - I’m on FME Form v2024.1.1.1.

Hi everyone!

We are looking into potentially adding this feature in the 2026.2 release of FME. I’m trying to gather use cases and particularly which formats (files / DBs) you’d like to see supporting this Rejected Port. We unfortunately will have to handle it format by format, and will likely implement it only for requested formats.

Thank you all for your input

Three i’d love to see is for GDB/SDE is:

• When an invalid geometry is passed. Also when the precision of the destination means that a polygon/line is generalised to a line/point (respectively)
• A invalid attribute value is written (trying to write a string to a double for example)
• A reserved attribute name is used

Here is another use case:

https://community.esri.com/t5/arcgis-data-interoperability-blog/writing-html-and-other-amp-lt-and-amp-gt-tagged/bc-p/1681656#M275

I would like to see a rejected port for Esri ArcGIS Portal Feature Service Writer. To handle individual rejected features, it is currently necessary to “recreate” the writer using HTTPCallers.

I would love to see this added, so that I can catch writing errors to the PostGIS database. Currently, the FeatureWriter is rejecting features and “Warning” messages are sent to the translation log, but not in the summary report. Even when all the features are rejected, the FeatureWriter considers that it has successfully completed its task. The FeatureWriter is typically used in the middle of a workflow, so it would be great if we could catch these, either by the addition of a specific rejected port, or by the addition of attribute in the summary port detailing the error or possibly just the % of features imported. That way, I could stop the workflow gracefully.

I don’t necessarily need a comprehensive list of the errors, but I do need be able to catch that there were errors/rejected features.

Hello ​@LizAtSafe, 

I would love to see that feature for theses formats we use a lot and have rejections :

• Definetly PostgreSQL and PostGIS format
• A must have for open source : geopackage
• Could be great for Oracle / Oracle Spatial

Thank you

Hi ​@arthur_bazin,

We have added these in our list of formats to add rejection for in the featureWriter.

They won’t be available from the get go once the rejection in the featureWriter becomes available but more likely sprinkled in the next few releases. For context we have to enable the feature for each format and it’s a non trivial amount of work unfortunately.

Cheers

Thank you ​@PierreAtSafe