Question

How to expose attributes from FeatureReader if feature type is dynamic?

tb09114
Rank
Userlevel 1
Badge +8

I am trying to construct my first custom transformer and have some issue with the schema handling.

To explain what I am doing in my transformer:

The SQLCreator is pulling information from a PostgreSQL table. The information retrieved are used to extract polygons from a Oracle database in the subsequent FeatureReader (FR_Municipalities). here the requested feature type is static. Cleaning up some attributes and joining a timestamp from the initial PostgreSQL table, the polygons are used to extract data from another table in the Oracle database. The difference here is, that the user has to have the possibility to select the feature type in the last FeatureReader(FR_AnalysisData).

Since the feature type is based on a published parameter, there is no specific output port, but all goes trough the generic port. Here I have the problem that basically all attributes should be pulled, but how to say that in the dialogue where I specify the attributes to expose?

Is there some sort of wildcard to do so?

19 replies

stalknecht
Rank
Userlevel 1
Badge +18

You have to type them in manually.

The AttributeExposer has a import function so maybe it's easier to use.

    Another solution is to dynamically iterate over all the attributes present at run-time using the AttributeExploder transformer.

    nielsgerrits
    Rank
    Userlevel 6
    Badge +32

    Edit: After reading your question again I need to rewrite my answer.

    With generic translations, as long as the workbench isn't run, it can knot now what attributes it will get. I can imagine a way of interacting with user input but this would use more then one workbench. Not sure a dynamic pull down list is going to work.

    For this kind of questions we use geocortex essentials workflows for user interface which communicates with fme server via the api.

    ----

    All columns / attributes are pulled. They only are not exposed. Check the result in the inspector, click a feature and check the Feature Information window.

    The big question is why you want this. Because you can't use an attribute when you don't know the name of it in any of the transformers.

    If you want to write the data dynamically, just use the schema feature from the FeatureReader to define the schema in the FeatureWriter.

    tb09114
    Rank
    Userlevel 1
    Badge +8

    @nielsgerrits, you are right and it cannot know what it will get. What I do not get, how to expose the attributes. When connecting an inspector to the generic port, and running the workbench, then I am getting the inspector to show the geometries, the table though is showing a single, empty column, named no_schema. However, when I select one of the items in the table, the information window of the inspector is showing all the attributes I wish for.

    Interesting enough, when looking at the attributes of the schema returned by the last FeatureReader, I am getting the schema of the initiator, but not of the feature type requested.

    @stalknecht, yes, without using the import function of the AttributeExposer you have to type them manually in. However, using the import function doesn't bring me any further. That's because the import wizard doesn't allow to set the dataset via user parameter. The subsequent dialogue where I have to select the table, I can most likely work with a user parameter, but with only able to use a predefined dataset I can't even get there.

    The AttributeExposer gives me indeed all the attributes, but at the moment I do not get the resulting list turned into a list of unique attribute names, and how to get the list then to be used to expose the members of the list?

    nielsgerrits
    Rank
    Userlevel 6
    Badge +32

    @nielsgerrits, you are right and it cannot know what it will get. What I do not get, how to expose the attributes. When connecting an inspector to the generic port, and running the workbench, then I am getting the inspector to show the geometries, the table though is showing a single, empty column, named no_schema. However, when I select one of the items in the table, the information window of the inspector is showing all the attributes I wish for.

    Interesting enough, when looking at the attributes of the schema returned by the last FeatureReader, I am getting the schema of the initiator, but not of the feature type requested.

    @stalknecht, yes, without using the import function of the AttributeExposer you have to type them manually in. However, using the import function doesn't bring me any further. That's because the import wizard doesn't allow to set the dataset via user parameter. The subsequent dialogue where I have to select the table, I can most likely work with a user parameter, but with only able to use a predefined dataset I can't even get there.

    The AttributeExposer gives me indeed all the attributes, but at the moment I do not get the resulting list turned into a list of unique attribute names, and how to get the list then to be used to expose the members of the list?

    For attributenames I know of 2 ways:

     

    - Use the FeatureReader Schema output port to get the schema, listexploder to get individual attribute names.

     

    - Use a Sampler to get 1 feature, AttributeExploder to get individual attribute names.
    jneujens
    Badge

    If you only need to write your data after the second FeatureReader (so no other transformers are used) you do not need to expose the attributes.

     

     

    When using a writer in dynamic mode and using the schema of the feature type the user wants, FME will automatically get the attributes (as specified in your schema) from each feature, even if the attribute is not exposed yet. Maybe you already knew this, but I think it is important to point out.

     

     

    If you do need to transform the data, and actually use the attribute names in other transformers, it might be a quick workaround to first write the data (dynamically with schema) so you have all the attributes exposed. Then read your temporary output and continue processing?
    stalknecht
    Rank
    Userlevel 1
    Badge +18

    @nielsgerrits, you are right and it cannot know what it will get. What I do not get, how to expose the attributes. When connecting an inspector to the generic port, and running the workbench, then I am getting the inspector to show the geometries, the table though is showing a single, empty column, named no_schema. However, when I select one of the items in the table, the information window of the inspector is showing all the attributes I wish for.

    Interesting enough, when looking at the attributes of the schema returned by the last FeatureReader, I am getting the schema of the initiator, but not of the feature type requested.

    @stalknecht, yes, without using the import function of the AttributeExposer you have to type them manually in. However, using the import function doesn't bring me any further. That's because the import wizard doesn't allow to set the dataset via user parameter. The subsequent dialogue where I have to select the table, I can most likely work with a user parameter, but with only able to use a predefined dataset I can't even get there.

    The AttributeExposer gives me indeed all the attributes, but at the moment I do not get the resulting list turned into a list of unique attribute names, and how to get the list then to be used to expose the members of the list?

    Please explain what you are trying to achieve with the to be exposed attributes?

     

     

     

    f.kemminje
    Badge +3

    just type the attribute name in the attribute to expose dialog box

    tb09114
    Rank
    Userlevel 1
    Badge +8

    just type the attribute name in the attribute to expose dialog box

    What if the attribute names change from feature type to feature type... and they do?

     

     

    tb09114
    Rank
    Userlevel 1
    Badge +8

    If you only need to write your data after the second FeatureReader (so no other transformers are used) you do not need to expose the attributes.

     

     

    When using a writer in dynamic mode and using the schema of the feature type the user wants, FME will automatically get the attributes (as specified in your schema) from each feature, even if the attribute is not exposed yet. Maybe you already knew this, but I think it is important to point out.

     

     

    If you do need to transform the data, and actually use the attribute names in other transformers, it might be a quick workaround to first write the data (dynamically with schema) so you have all the attributes exposed. Then read your temporary output and continue processing?
    I do need to work further with the data. The custom transformer shall be used in several workbenches that analyse different feature types based on the results of some analysis inside the custom transformer.

     

     

    tb09114
    Rank
    Userlevel 1
    Badge +8
    Please explain what you are trying to achieve with the to be exposed attributes?

     

     

     

    Well, one task could be to analyse the features based on the value of a specific attribute. Let's say there should be an attribute 'temporary' and it holds values of 1 or 0, and I only want to work with the 1s...

     

     

    david_r
    Rank
    Userlevel 4

    @nielsgerrits, you are right and it cannot know what it will get. What I do not get, how to expose the attributes. When connecting an inspector to the generic port, and running the workbench, then I am getting the inspector to show the geometries, the table though is showing a single, empty column, named no_schema. However, when I select one of the items in the table, the information window of the inspector is showing all the attributes I wish for.

    Interesting enough, when looking at the attributes of the schema returned by the last FeatureReader, I am getting the schema of the initiator, but not of the feature type requested.

    @stalknecht, yes, without using the import function of the AttributeExposer you have to type them manually in. However, using the import function doesn't bring me any further. That's because the import wizard doesn't allow to set the dataset via user parameter. The subsequent dialogue where I have to select the table, I can most likely work with a user parameter, but with only able to use a predefined dataset I can't even get there.

    The AttributeExposer gives me indeed all the attributes, but at the moment I do not get the resulting list turned into a list of unique attribute names, and how to get the list then to be used to expose the members of the list?

    If you need to make decisions based on run-time attributes, you should consider using the PythonCaller. It's fairly easy to write code that requests a feature's (dynamic) attributes and does something with them.

     

    For example if you need to check for the presence of one or several attributes names that follow a certain pattern and act upon it.
    tb09114
    Rank
    Userlevel 1
    Badge +8
    If you need to make decisions based on run-time attributes, you should consider using the PythonCaller. It's fairly easy to write code that requests a feature's (dynamic) attributes and does something with them.

     

    For example if you need to check for the presence of one or several attributes names that follow a certain pattern and act upon it.
    I still have to get into the use of Python in FME, and sadly the editor doesn't help. ;-)

     

    For now I simply took the last transformer out of the custom transformer. It was a simple 'solution' and certainly worked with respect to my time frame.

     

    I will keep the PythonCaller in mind!

     

     

    jneujens
    Badge
    I do need to work further with the data. The custom transformer shall be used in several workbenches that analyse different feature types based on the results of some analysis inside the custom transformer.

     

     

    In that case, you only have to specify which schema the dynamic FeatureWriter has to use. And that should be it? I might not think of other problems since I dont know your data...

     

     

     

    G
    Badge +3

    If you create a attribute based on an expression like

    tolower "@Value(filter_{2}.part)"])

    or

    ("@Value(filter_{2}.part)" + 2)])

    you can interact with it when exposed using the same expression in some other transformer (like a tester).

    (a not "@" character needs to be inserted before the @Evaluate, else it won't create btw)

    Of course the trick is to expose it....shame exposer does not allow for expressions..or regexp...that would be nice indeed.

    bo
    Badge +2

    Edit: After reading your question again I need to rewrite my answer.

    With generic translations, as long as the workbench isn't run, it can knot now what attributes it will get. I can imagine a way of interacting with user input but this would use more then one workbench. Not sure a dynamic pull down list is going to work.

    For this kind of questions we use geocortex essentials workflows for user interface which communicates with fme server via the api.

    ----

    All columns / attributes are pulled. They only are not exposed. Check the result in the inspector, click a feature and check the Feature Information window.

    The big question is why you want this. Because you can't use an attribute when you don't know the name of it in any of the transformers.

    If you want to write the data dynamically, just use the schema feature from the FeatureReader to define the schema in the FeatureWriter.

    @nielsgerrits, I ran into this post today as I struggled to carry attributes in a "simple-generic" translation from a source format to a target format, without having to make any attribute manipulation. Your example here worked in the case of writing to an FGDB, but it failed when writing to Shapefile or Oracle Spatial. The msg was "Features With No Schema defined." In fact, with FME 2018.0, even with the FGDB target output, the error msg showed up but somehow the featurewriter managed to dump the content to a .ffs file and later "played back" to get it right. I was wondering if you had any comments or experience in this pattern working for other writer formats. Thanks!

     

     

    kimo
    Badge +1

    After many years there is still not a simple solution to expose feature types in a dynamic workbench. Only the schema of known feature types is dynamic. The feature types have to be read before the workbench is configured and run. I need the feature types to be exposed to use other transformers on the data stream. I know you can pair dynamic readers and writers and it will work on any input without the feature types exposed. But this means that for example the ChangeDetector transformer cannot be used.

    But here is a workaround without resorting to python scripts:

    1. Create a workbench that reads in the feature types using the schema reader.
    2. Build a list separated by linefeed characters (LF in the text editor special chars)
    3. Send the list as a parameter to a second dynamic workbench that needs the feature types exposed

    This first workbench has the two arbitrary file geodatabases that need to be compared. It uses the schema reader to obtain the featuretypes in the sources dynamically. Normally it would find these when you add the sources in a static way.

    Here is part of the log showing the list of feature types. These are in an attribute 'concatenated' that can be passed into the second dynamic workbench. Here i have highlighted the first few (less the quote) which can also be copy/paste into the second workbench for debugging.

    The input dialog in the second workbench with default parameters which can be populated from the first runner workbench. Note that the LF characters are invisible, but they must be there. You could copy/paste from a text file if each name is on a separate line, like the log above. The attribute name list must be a parameter, it cannot be a global variable set after the workbench has started to run.

    Here is the actual workbench that I want to run on any two pairs of geodatabases to find the changes.

    Note the "attributes to expose" in the dynamic feature reader is supplied by the input parameter. These are needed for the ChangeDetector further on.

    The whole point of this gymnastic display is to be able to put this up on FME Server so that we can get a change detection report on any two geodatabases without editing the workbench. Surely this is what we all need and it must be able to be done more easily in a built-in transformer?

    harmen_kampinga
    Badge +2

    You have to type them in manually.

    The AttributeExposer has a import function so maybe it's easier to use.

      Another solution is to dynamically iterate over all the attributes present at run-time using the AttributeExploder transformer.

      It's indeed that simple. Especially it will come in handy when you know some standard attributes should be there and should contain a value.

      But it's more difficult when you want to find out whether a specific attribute is present in a schema?

      david_r
      Rank
      Userlevel 4

      It's indeed that simple. Especially it will come in handy when you know some standard attributes should be there and should contain a value.

      But it's more difficult when you want to find out whether a specific attribute is present in a schema?

      You can e.g. use a Tester with the "Attribute is missing" operator to check for the presence of any attribute after it has been exposed. Or am I missing something?

      L
      Badge +1

      After many years there is still not a simple solution to expose feature types in a dynamic workbench. Only the schema of known feature types is dynamic. The feature types have to be read before the workbench is configured and run. I need the feature types to be exposed to use other transformers on the data stream. I know you can pair dynamic readers and writers and it will work on any input without the feature types exposed. But this means that for example the ChangeDetector transformer cannot be used.

      But here is a workaround without resorting to python scripts:

      1. Create a workbench that reads in the feature types using the schema reader.
      2. Build a list separated by linefeed characters (LF in the text editor special chars)
      3. Send the list as a parameter to a second dynamic workbench that needs the feature types exposed

      This first workbench has the two arbitrary file geodatabases that need to be compared. It uses the schema reader to obtain the featuretypes in the sources dynamically. Normally it would find these when you add the sources in a static way.

      Here is part of the log showing the list of feature types. These are in an attribute 'concatenated' that can be passed into the second dynamic workbench. Here i have highlighted the first few (less the quote) which can also be copy/paste into the second workbench for debugging.

      The input dialog in the second workbench with default parameters which can be populated from the first runner workbench. Note that the LF characters are invisible, but they must be there. You could copy/paste from a text file if each name is on a separate line, like the log above. The attribute name list must be a parameter, it cannot be a global variable set after the workbench has started to run.

      Here is the actual workbench that I want to run on any two pairs of geodatabases to find the changes.

      Note the "attributes to expose" in the dynamic feature reader is supplied by the input parameter. These are needed for the ChangeDetector further on.

      The whole point of this gymnastic display is to be able to put this up on FME Server so that we can get a change detection report on any two geodatabases without editing the workbench. Surely this is what we all need and it must be able to be done more easily in a built-in transformer?

      Hello @kimo,

      not a bad way :-) But, what type of Published parameter you used to put there list of attributes to read?

       

      Thank You!

      Lubo

      Reply


      Community Stats

      28,669
      Posts
      109,357
      Replies
      36,531
      Members
      Terms & ConditionsCookie settings
      A product of
      Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.