Skip to main content
Open

FeatureWriter Schema port

Related products:IntegrationsTransformers
  • March 10, 2024
  • 4 replies
  • 110 views

ctredinnick
Supporter
Forum|alt.badge.img+20

If a FeatureWriter is set to write dynamically, then the port which accepts the schema features should be a dedicated schema port, and not the main port that handles the features to be written.

This is the design the FeatureReader transformer uses, and more recently the SQLCreator, to have a dedicated schema port. Dynamic writing can be done either by having a schema feature enter the writer before the features to be written, or by having each feature carry schema list attributes that describe how it should write. Potentially adding a schema port would make that second dynamic write pattern unviable, but we have tools now like the SchemaScanner to produce a schema feature for any workflows which would need to change from this.

Also there are occasionally bugs where the writer tries to write out the schema feature, or schema attributes. Recently in 2023, a GDB writer will warn you that the schema attributes need to be renamed to follow GDB field name restrictions.

It would be much more consistent design, and easier to teach how dynamic writing works if the FeatureWriter had a schema port to receive the schema. It only needs to appear when the write mode is set to dynamic.

 

4 replies

LizAtSafe
Safer
Forum|alt.badge.img+18
  • Safer
  • June 26, 2024
NewOpen

jkr_wrk
Influencer
Forum|alt.badge.img+36
  • June 27, 2024

Also it would be nice if it would be easier to send the schema first to the writer. It is very hard to debug when features are arriving earlier then the schema and writing fails.

Seeing that there was no schema send to the writer first is a start, but better would be to wait for schema’s before trying to write features.


vlroyrenn
Enthusiast
Forum|alt.badge.img+14
  • Enthusiast
  • June 27, 2024

Agreed. The current approach of only testing the first incoming feature runs the risk of the schema and data arriving out of order, with no existing means of forcing them to arrive in a specific order.

 

Also it would be nice if it would be easier to send the schema first to the writer. It is very hard to debug when features are arriving earlier then the schema and writing fails.

Seeing that there was no schema send to the writer first is a start, but better would be to wait for schema’s before trying to write features.

I had a community idea around this, but it has yet to gain any sort of traction.
 

 


andreaatsafe
Safer
Forum|alt.badge.img+15
The following idea has been merged into this idea:

All the votes have been transferred into this idea.