Dynamic schema partial merge from 2 data sources

Hi, I am attempting to create a workspace that requires a dynamic schema based on most attributes from 1 SHP file plus a couple of attributes from another SHP file. I will be intersecting the 2 SHP files using a SpatialFilter. And I want to output a SHP file with all but 1 of the attributes from SHP file A (always the same one) plus one or more of the attributes from SHP file B (these will be input as a comma-separated list in a published text parameter).

I have gone through all of the examples I could find, including the ones with workspace resource readers and custom formats and Python scripts, but I can't for the life of me get them to work!

Any insight into what my workspace should look like would be greatly appreciated!

Page 1 / 1

I think you'll need to look into 'joining' your features together on a common ID. Can you perhaps take a look at using either the FeatureMerger or the Joiner to see if they give you what you need?

Thanks for the response, 1SpatialDave. I have no problem with the feature merging/joining part of the workspace. The problem I am having is making the output SHP file have the appropriate attributes, based on both datasets, and having one attribute name specified by an input parameter.

Hi,

I think it will need a little complicated tricks. For example (not tested):

1) Create a workspace to

- read the souce feature schema (Schema (Any Format) Reader)

- fetch the parameter defining additional attributes.

- create dynamically a schema table based on above and save it.

2) Modify the main workspace to read the schema table, and use the table as the schema source of the writer.

3) Create the 3rd workspace to

- run the 1st workspace to create and save a schema table.

- run the 2nd (main) workspace to perform translation.

Takashi

Since it's difficult to explain only by words, I created simplified example workspaces. Please download here and evaluate them (32KB zipped file, including test data).

My strategy is: create dynamically a schema table based on the source datasets and a user parameter (comma separated attribute names), and use the table as the destination schema definition when running the main workspace. Schema table should contain these 5 columns and it will be read with "Schema (From Table) Reader" as a workspace resource in the main workspace.

feature_type | attribute_name | attribute_data_type | geometry_type | order

See more information about the Workspace Resource here: Dynamic Schema Examples http://fmepedia.safe.com/articles/Samples_and_Demos/Dynamic-Schema-Examples-FME2011 > Dynamic Schema Components > Workspace Resources This approach should work, but it's a little complicated. Hope simpler solution will be provided.

Takashi

This looks great takashi! I am going to try it out...

Thanks again, Takashi, this works great for me, thanks for putting together the example! I am wondering if somehow saving it as a custom format would allow for this to be simplified to just the 1 main workspace (and the custom format) without requiring the workspace runner. This example does that and uses a Python script, but I couldn't get it to work: http://fmepedia.safe.com/articles/How_To/Extracting-a-schema-subset-for-dynamic-schemas

I think the 1st workspace can be changed to a custom format. You need to modify a few points, not so many. Mainly, 1) Re-build list attributes. attribute{}.name, attribute{}.fme_data_type etc. 2) Change the CSV writer to FFS writer 3) Save the workspace as a Custom Format To re-build list attributes, Python script might be a quick way. Just be aware that the Python script in the linked article is using old module "pyfme" for FME Objects API. It was replaced with "fmeobjects" in FME 2012. New Python FME Objects API in FME 2012 http://fmepedia.safe.com/articles/How_To/New-Python-FME-Objects-API-in-FME-2012

Takashi

Using Python script, the 1st workspace probably will be extremely simple. I'll also try.

Hi, I created an example of custom format for dynamic schema.

Download here (22KB zipped file including test data).

"SCHEMA_EX0.fds": custom format definition.

"MainWorkspace.fmw": main workspace using the custom format.

FME 2013 SP3 Build 13528

Before testing, import the custom format to your FME environment.

Open "Reader and Writer Gallery" window. Workbench Menu: Tools > Browse Readers and Writers

Click /Import...] button in the "Custom Formats" group, and select "SCHEMA_EX0.fds".

Then, "SCHEMA_EX0.fds" will be copied into My Documents/FME/Formats folder, and oSCHEMA_EX0] format will be added the Gallery.

You can also edit or delete a custom transformer on the Gallery.

The functionality of these workspaces is same as the previous example, but they were very simplified! To see detailed definition, open those with the Workbench. Takashi

In fact, Python script is not essential for defining a custom format.

This "SCHEMA_EX1" is an example of non-scripted version, it's compatible with "SCHEMA_EX0". Download here (23KB zipped file). It might help to understand how we can create dynamically feature schema using a custom format. After understanding the mechanism, any practical application will be easier :-)

Hope this helps.

Takashi

Thanks, takashi, this works great! I never would have figured this out with just the documentation and existing examples.

Reply

Community Stats

Reply

Community Stats

Sign up

An FME Account is required to contribute

Login to the community

An FME Account is required to contribute

Scanning file for viruses.

This file cannot be downloaded