Question

Dynamic Writer based on Transformer?

J
I would like to achieve the following:

 

 

Create a workspace that can be used to read a large number of tables, create geometries with the vertexcreator and write them to a database after changing attribute names based on regular expressions.

 

Attributes differ from table to table and should be preserved.

 

 

I can probably achieve this by writing the schema to a feel and rereading that using the workspace runner. However, I feel like I am missing something really obvious. Isn't there a way to use a dynamic writer and derive the schema from a transformer rather than a reader?

 

 

Jakob

 

 

12 replies

G
Badge +3
Alas, no.

 

 

As trhe schema is created dynamically, you cannot refer to it in the a dynamic writer.

 

You would have to enter a schema name, wich does not exist. And even if you would create a empty schema (to be filled during runtime) to refer to in the dynamic writer, you would not be able to refer to any attributes. As they are created dynamically.

 

 

 

You must therefore first create the schema. And (wether or not you use a workspacerunner) refer to it in a second stage.
fmelizard
Rank
Userlevel 4
Badge +13
HI,

 

Have you tried using the schema parameters to be found in the writer feature type properties> schema definition> configure dynamic schema parameters> schema definition> named by attribute ?
J
Hi,

 

do you mean creating a list attribute to be filled on runtime and then reused as a schema provider?

 

 
fmelizard
Rank
Userlevel 4
Badge +13
I have never tired it with a list, and have my doubts if it will work, but I know that an attribute will work.
G
Badge +3
the problem stays the same, you can't apply a schema that does not exists yet.

 

You must somehow write it out, wich is no problem.

 

Like filettributewriter.

 

 

The problem arrises when u have to apply it.

 

Prior to runtime, with no schema... well you can't.

 

 

There are a ways to turn dynamiccaly created attribute names to attributes.

 

Either turn to python or tcl.

 

 

Or Create a output schema by constructing a textfile write it out using a attributefilewriter ( u only need 1 row basiclaly) and then use a workspacecaller to read this file.

 

Workspace to be called must be parametrized to do so.

 

 

I have posted a example a while ago ( i think 2014, attributes on the fly , or called something lke that.)

 

 

Using schema definition by atribute won't help create priorly (to runtime) unknown attributes ( aka "attributes on the fly"). It works for existent schema's, like reading some type of file.

 

 
takashi
Rank
Userlevel 2
Badge +17
Hi,

 

 

I think there are two types of changing attribute names through transformers, and the solution will also be different depending on the type.

 

1. Generate the names dynamically through transformers (resulting names are unknown before running translation).

 

2. Change the names through transformers, but the resulting names will be always same (resulting names can be known when creating workspace).

 

 

If your case was type 1, one possible way would be to create a custom format to generate the schema dynamically based on the source datasets, add it to the workspace as a resource, and set it to schema source of the dynamic writer feature type.

 

In the type 2, you can apply dynamic schema writer with creating a look-up table beforehand.

 

 

Takashi
takashi
Rank
Userlevel 2
Badge +17
The method using list, as Itay mentioned, can be applied to both type 1 and type 2. (http://fmepedia.safe.com/articles/Samples_and_Demos/Dynamic-Workflow-Tutorial-Destination-Schema-is-Derived-from-List-Attributes)

 

 

Maybe the way suggested by Gio is a method using a look-up table. (http://fmepedia.safe.com/articles/Samples_and_Demos/Dynamic-Workflow-Tutorial-Destination-Schema-is-Derived-from-a-Lookup-Table) 
J
Thanks everyone, I'll give it a go later.

 

 
G
Badge +3
attributes created on the fly cannot be accessed during runtime through dynamicwriter.

 

A schema of these unknown attributenames must be output first.

 

This will always involve 2 stage workspace.

 

 

Unless something drastically has changed in fme 2015.

 

This is an oldstanding problem.

 

 

Has been asked and  covered many a time.

 

 

Show me how to create attributes on the fly and acces them within one single workbench (single stage) I'd like to know this process myself of course.

 

 

My method does not use a lookup table. Lookup table has to be created, wich you can not do if attrbiutes are created on the fly and you n=know not their names...

 

 

What I do is create a txt file with 1st line the attributes created on the fly:

 

row1 att_name{1}, att_name{2},...

 

row2 att_value{1},att_value{2},...

 

row3

 

.

 

.

 

 

Write it to txt using a attributefilewriter.

 

 

Workspacecaller, read same file as csv --> results in attrbutes

 

 

2stage rpoces

 

And it does away with having to "expose" any hidden attributes, wich attributes on the fly are.

 

 

Look at you rwon posts, couple of days previous tho this one.
takashi
Rank
Userlevel 2
Badge +17
I put a workspace example and a source dataset here, which demonstrates "how to configure dynamic schema through list attribute".

 

Download (https://drive.google.com/file/d/0B0ufVP2t0eApTWZDay0yVUxrVU0/view?usp=sharing)

 

FME 2014 SP5 build 14440
takashi
Rank
Userlevel 2
Badge +17
Another example for Custom Format approach. (https://drive.google.com/file/d/0B0ufVP2t0eApMnJkZUN5bmdtNjg/view?usp=sharing)

 

This was created with FME 2014 SP1.

 

Includes 2 workspaces named "SchemaProvider.fmw" and "Main.fmw", and source datasets.

 

First, you have to export "SchemaProvider.fmw" as a custom format named "SCHEMA_PROVIDER" in your system. And then open "Main.fmw".

 

See detailed description in the "SchemaProvider.fmw".
G
Badge +3
I think his question ".. derive the schema from a transformer " is still a no.

 

 

Unless one wants to argue that using for instance a SchemaMapper is "deriving a schema form a transformer".

 

Wich i think not, because without writing out the schema first, you cannot map any. (existing schema's can be considered as already available of course)

 

 

AS your own examples shows (like it has been shown lots of time...) it is a 2 stage process)

 

Your first example is the same as creating the thing using a attrbiutefilewriter as i posted earlier in this thread : " ..constructing a textfile, write it out using a attributefilewriter"

 

and...

 

"What I do is create a txt file with 1st line the attributes created on the fly:

 

row1 att_name{1}, att_name{2},...

 

row2 att_value{1},att_value{2},...

 

row3

 

.

 

.

 

"

 

..again posted eralier in this same thread.

 

 

You can write it to a file, but you cannot acces it in the same workspace. Because you cannot expose what is unknow.

 

Attributefilewriter has the advantage that you can use a Workspacecaller at its output.

 

 

 

So,  QED it is a 2 stage problem.

 

(unless the writing out is the final goal of the workspace)

 

 

 

 

 

 

Reply


Community Stats

28,731
Posts
109,528
Replies
36,565
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.