Question

Attributes in a list transposed across as new columns in excel


Badge

There's a number of other similar posts out there, but I can't seem to find a solution that works. I'm still trying to build a workflow for this on my end, but I figured I'd post here to see if anyone knows of an easy solution.

I have data in a table that looks as follows...

IDSecondary ID1ABC1231DEF3451GHJ6782DEF3452XYZ7893VWX5674ABC123

 

And I'd like to transpose this across a spreadsheet so that it looks as follows...

IDSecondary_ID_1

 

Secondary_ID_2Secondary_ID_31ABC123

 

DEF345

 

GHJ678

 

2DEF345

 

XYZ789

 

 

3VWX567

 

 

 

4ABC123

 

 

 

 

The problem I'm hitting a wall with is that there may be any number of Secondary_IDs for a given ID...potentially a couple hundred for a single ID. So, I'm looking to have a tabular data set in the end that may have a couple hundred columns, with a single row for each ID.

Is this possible? I'm trying to wrap my head around using list attributes, but am struggling to find a solution to output each attribute value in a list as it's own column. I only need one list attribute for the Secondary_ID values as far as I'm aware.

I'll keep playing around and I'll post any solutions that I find.

Thanks for your time.

FYI...I'm running FME 2015.0 (20150114 - Build 15245 -WIN32) via ArcGIS data interoperability extension.


14 replies

Userlevel 1
Badge +10

It's possible but not particularly straightforward in current versions of FME. Not sure whether the same sort of workflow would work in 2015

Badge +16

Ragged edge tables are seldom fun to work with, can you refactor the problem to (say) output a worksheet per ID value and keep Secondary ID in one column?

Badge

Ragged edge tables are seldom fun to work with, can you refactor the problem to (say) output a worksheet per ID value and keep Secondary ID in one column?

Potentially, but I may have to change things further down the line & we already have a script set up to work with such a ragged edge table. Perhaps we need to make some changes in that regards though.

Now I know I could do that with a ListBuilder/ListConcatenator, and it seems it'd be wise for me to use a ListElementCounter to get an idea of how many columns I would need.

Anyhow, I've built a very non-elegant solution. I'm just building a list and setting up an AttributeCreator to output a new attribute for each list element number.

e.g., AttributeName = Secondary_ID_1, Value = @Value(_list{0}.Secondary_ID)

AttributeName = Secondary_ID_2, Value = @Value(_list{1}.Secondary_ID)

AttributeName = Secondary_ID_3, Value = @Value(_list{2}.Secondary_ID)

...and so on until I've created enough new attributes to hold the max count from the ListElementCounter.

Perhaps a more elegant solution will come to mind & if so I'll post here.

Thanks all for the quick feedback.

Userlevel 1
Badge +10

This is how I would do it in 2018

0684Q00000ArLa3QAF.png

and the python to avoid having to manually expose attributes

import fme
import fmeobjects

def create_schema(feature):
    attrlist = feature.getAttribute('_list{}.Sec_ID_Name')
    feature.setAttribute('attribute{'+str(0)+'}.name',"ID")
    feature.setAttribute('attribute{'+str(0)+'}.fme_data_type','fme_char(20)')
    for i, val in enumerate(attrlist):
        feature.setAttribute('attribute{'+str(i+1)+'}.name',val)
        feature.setAttribute('attribute{'+str(i+1)+'}.fme_data_type','fme_char(20)')
Badge

This is how I would do it in 2018

0684Q00000ArLa3QAF.png

and the python to avoid having to manually expose attributes

import fme
import fmeobjects

def create_schema(feature):
    attrlist = feature.getAttribute('_list{}.Sec_ID_Name')
    feature.setAttribute('attribute{'+str(0)+'}.name',"ID")
    feature.setAttribute('attribute{'+str(0)+'}.fme_data_type','fme_char(20)')
    for i, val in enumerate(attrlist):
        feature.setAttribute('attribute{'+str(i+1)+'}.name',val)
        feature.setAttribute('attribute{'+str(i+1)+'}.fme_data_type','fme_char(20)')
Thanks a boat load egomm. I should be able to set up that same workflow in 2015.

 

Do you mind sharing the Python code in your PythonCaller? I will admit... my python skills are lacking.

 

Thanks again.

Badge +3

@bwasserstein

Listbuilder on ID, explode. Increase value of elemtn index by 1.

Concatenate this with Secondary_ID_.

Create attribute @Value(_result) = @Value(Secondary ID)

Expose Secondary_ID_1 to Secondary_ID_n (create list in excel by increment and import in exposer)....

aaah..can't import in fme_2015. Well hope you don't have hundreds of ID nrs then. (upgrade FME?)

 

aggregator on ID, drop attributes, use attriubtus to concatenate on Secondary_ID_1 to Secondary_ID_n with no separator.

 

Now you're done

 

 

 

 

 

Badge +3

@bwasserstein

Listbuilder on ID, explode. Increase value of elemtn index by 1.

Concatenate this with Secondary_ID_.

Create attribute @Value(_result) = @Value(Secondary ID)

Expose Secondary_ID_1 to Secondary_ID_n (create list in excel by increment and import in exposer)....

aaah..can't import in fme_2015. Well hope you don't have hundreds of ID nrs then. (upgrade FME?)

 

aggregator on ID, drop attributes, use attriubtus to concatenate on Secondary_ID_1 to Secondary_ID_n with no separator.

 

Now you're done

 

 

 

 

 

here is result

Userlevel 2
Badge +17

If you populate Secondary IDs grouping by primary ID into a list, you can just rename each element to a desired name with the AttributeRenamer.

Badge +3

If you populate Secondary IDs grouping by primary ID into a list, you can just rename each element to a desired name with the AttributeRenamer.

@takashi

 

Of course that's possible.

Just not very attractive because then you can't import a attribute names . Unless you create a mapping first and import that, which kind of defeats the purpose as it is clearly less efficient.

I simply create a mapping for this type of enumerated attribute name in say excell, making it very as large as anticipated need.

When not enumerated I just copy the list from the feature information window and parse it using like notepad++.

You could replace the exposer with python attributecreation ass the data is enumerated.

I guess doing that would increase the dataset by 2.

 

This will go on till safe figure out how to blindly expose everything hidden.

Userlevel 1
Badge +10

@takashi

 

Of course that's possible.

Just not very attractive because then you can't import a attribute names . Unless you create a mapping first and import that, which kind of defeats the purpose as it is clearly less efficient.

I simply create a mapping for this type of enumerated attribute name in say excell, making it very as large as anticipated need.

When not enumerated I just copy the list from the feature information window and parse it using like notepad++.

You could replace the exposer with python attributecreation ass the data is enumerated.

I guess doing that would increase the dataset by 2.

 

This will go on till safe figure out how to blindly expose everything hidden.

If you are writing it out dynamically to excel, there's no need to expose the attributes in the workspace

Badge +3

If you populate Secondary IDs grouping by primary ID into a list, you can just rename each element to a desired name with the AttributeRenamer.

@egomm

 

So I can dynamically write dynamically created attributes?

How would the schema be created?

Always keen to learn some new technique, I am.

Userlevel 1
Badge +10

@egomm

 

So I can dynamically write dynamically created attributes?

How would the schema be created?

Always keen to learn some new technique, I am.

pivot_example.fmwt An example @gio

Badge +3

If you populate Secondary IDs grouping by primary ID into a list, you can just rename each element to a desired name with the AttributeRenamer.

@egomm

 

To get to the attributes I would have to read the file again and still not know what the attributes are, unless I sacrifice dynamism.

Exposure of enumerated attribute names makes it possible to access them in the workbench. (I simply create a long import list)

 

Not required for the solution of this topic I admit.

But for this topic probably adequate.

Badge

If you populate Secondary IDs grouping by primary ID into a list, you can just rename each element to a desired name with the AttributeRenamer.

Thanks Takashi. This is yet another solution! I currently have an AttributeCreator built into a similar workflow rather than an AttributeRenamer.

I'm still probably going to take a crack at setting up egomm's workflow if I have time. I'll be pushing a number of disparate datasets with identical schema thru this workflow, and I never know how many Secondary_ID values there may be for a given ID... I suspect up to a few hundred for some of these datasets. I can set up the AttributeRenamer/Creator to accommodate this, but egomm's workflow is definitely more elegant.

Thanks all for your help. It's much appreciated.

Reply