Question

FeatureWriter cannot configure dynamic schema based on schema feature when fanning out destination dataset?

  • 25 November 2017
  • 10 replies
  • 59 views
takashi
Rank
Userlevel 2
Badge +17

Hello,

FME 2017.1.1.1 build 17652, 2017.1.2 beta build 17717, 2018.0 beta build 18208

The scenario is:
  1. The source dataset contains multiple feature types. e.g. "A", "B", and "C".
  2. Read every feature from the source dataset and simply write them into Shapefile files for each feature type separately, with the same schema as corresponding source feature type.
  3. Here, the destination dataset (folder) names should be the same names as the source feature types. i.e. "A", "B", and "C".
  4. And all the destination files (feature types) should have an identical name. e.g. "output.shp"
  5. In fact, the names and schema of source feature types are unknown when creating the workspace, so you want to configure Dynamic Schema and Dataset Fanout.
It's easy to realize this using FeatureReader + regular Shapefile writer with Dynamic Schema setting and Dataset Fanout configuration. However, I was't able to realize this with a FeatureWriter. According to my observation, it doesn't seem that the FeatureWriter can recognize schema information stored in a schema feature if you specify the Dataset as a string expression concatenating a feature attribute value. e.g. Dataset: result/featurewriter/@Value(fme_feature_type) See also the attached workspace (FME 2017.1.1.1): Is it a restriction regarding Dynamic Schema and Dataset Fanout with the FeatureWriter?

Feature Writer Parameters:

Shapefile Writer Configuration:

10 replies

jneujens
Badge

I have implemented something similar at a customer. Instead of a schema feature, the schema was read from a workspace resource. That worked fine. Off course, the processing is a bit different then.

 

 

I should check if I can reproduce your problem.
takashi
Rank
Userlevel 2
Badge +17

I have implemented something similar at a customer. Instead of a schema feature, the schema was read from a workspace resource. That worked fine. Off course, the processing is a bit different then.

 

 

I should check if I can reproduce your problem.
Hi @jneujens, thanks for your answer. I confirmed that the FeatureWriter works fine to fanout dataset with either "derive schema from workspace resource" or "derive schema from source dataset" method. It seems that only "derive schema from schema feature" has this issue.

 

Hope someone from Safe will take a closer look at this.

 

G
Badge +3

@takashi

I tried it and it seems that the schema definition is not propagated to the features this way.

When you check inspector the attributes are empty on the features themselves, with on row at the top of each feature type with the definition.

If you merge the schema with the generic output using fme_feature_type=fme_feature_type_name it will write out the features to the folders as you intended. (though in my test the feature writer outputs

1 feature less then fed ..which currently eludes me..)

G
Badge +3

@takashi

I tried it and it seems that the schema definition is not propagated to the features this way.

When you check inspector the attributes are empty on the features themselves, with on row at the top of each feature type with the definition.

If you merge the schema with the generic output using fme_feature_type=fme_feature_type_name it will write out the features to the folders as you intended. (though in my test the feature writer outputs

1 feature less then fed ..which currently eludes me..)

a pic.....

 

 

G
Badge +3

@takashi

I tried it and it seems that the schema definition is not propagated to the features this way.

When you check inspector the attributes are empty on the features themselves, with on row at the top of each feature type with the definition.

If you merge the schema with the generic output using fme_feature_type=fme_feature_type_name it will write out the features to the folders as you intended. (though in my test the feature writer outputs

1 feature less then fed ..which currently eludes me..)

@takashi

 

It seems to not write the first record of each feature type.

 

Tried everything, but i seem not able to prevent this.

 

Maybe you can?

 

takashi
Rank
Userlevel 2
Badge +17

@takashi

I tried it and it seems that the schema definition is not propagated to the features this way.

When you check inspector the attributes are empty on the features themselves, with on row at the top of each feature type with the definition.

If you merge the schema with the generic output using fme_feature_type=fme_feature_type_name it will write out the features to the folders as you intended. (though in my test the feature writer outputs

1 feature less then fed ..which currently eludes me..)

@gio, thanks for your feedback.

 

The schema feature output from the <Schema> port of the FeatureReader contains an attribute called "fme_schema_handling", which has a string value "schema_only".

 

According to my observation, a dynamic writer won't write the first feature into the destination dataset if it has the attribute.

 

I think the writer would write the first feature if you removed the "fme_schema_handling" from the features.

 

G
Badge +3

@takashi

the writer in dynamic mode seems to expect that the first record of each feature type is a schema record, even if it fails to propagate the schema to the features.

So, apart form the features, the schema records need to be passed to the writer (and if need sorted to the top of each feature type)

It will then not write these schema records but will write all the features. ( a dummy record for each feature type sorted to the top of each feature should also work)

have not actually tried this tough...

(b.t.w. in version 2016 it seems not to propagate the schema however i tied)

G
Badge +3

@takashi

This way it works.

Add a dummy for each feature_type. Counter set to local etc. start 0.

after feature_merge use the attribute "fme_feature_type_name" instead of fme_feature_type. (for feature_type will be reset)

Attribute "schema only" does not prevent writing. Check output after u run it.

Like this.

b17652-featurewriter-dataset-fanout-does-it.fmwt

takashi
Rank
Userlevel 2
Badge +17

@gio, thanks for your feedback. I confirmed that your solution works as expected.

 

Also, the traditional "Destination Schema is Derived from List" (FME 2016.0 and earlier) method still works as well.

b17291-featurewriter-dataset-fanout-does-it-2.fmwt (FME 2017.0.1.1)

The point is that you have to remove the "fme_schema_handling" attribute (value = "schema_only") from the schema feature. If the features have the attribute, the dynamic writer considers the first feature for each feature type as a schema feature and discards it.

There are some workarounds, but I hope that this simple workflow from my first question could work too.

S
@gio, thanks for your feedback.

 

The schema feature output from the <Schema> port of the FeatureReader contains an attribute called "fme_schema_handling", which has a string value "schema_only".

 

According to my observation, a dynamic writer won't write the first feature into the destination dataset if it has the attribute.

 

I think the writer would write the first feature if you removed the "fme_schema_handling" from the features.

 

Well spotted - I couldn't figure out why a dynamic FeatureWriter was always dropping the first record and removing the fme_schema_handling attribute solved it.

Reply


Community Stats

28,670
Posts
109,368
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.