Skip to main content
Solved

creating a data dictionary in FME?

  • June 2, 2020
  • 11 replies
  • 224 views
P
Forum|alt.badge.img

Hi,

I have different councils Pipe material list with a specific example of say Copper – which different council has a different way to define it.. say for example one council called it “CU”, other council calls it “COP” and some other councils call it with some other name…however, the material is the same…and in general we want to define a common name for it..... say "Copper"? Can we create a data dictionary in FME?...which can identify and recognize the correct name and replace it with a generic name?

 

 

Best answer by jdh

I would probably do this as a mapping table.

 

Have an excel or csv file with two columns the OriginalName and the CommonName and then use a DatabaseJoiner to attach the common name as an attribute.

 

 

This way the "dictionary" can be maintained without any edits to the FME workspace.

 

Ex.OriginalNameCommonNameCUCopperCOPCopperFEIronIRIron

 

You would also be able to handle any unjoined features, where the council name is not in the dictionary. Whereas the StringPairReplacer option would just pass those features through untouched.

    This post is closed to further activity.
    It may be an old question, an answered question, an implemented idea, or a notification-only post.
    Please check post dates before relying on any information in a question or answer.
    For follow-up or related questions, please post a new question or idea.
    If there is a genuine update to be made, please contact us and request that the post is reopened.

    11 replies

    umapper1
    Contributor
    Forum|alt.badge.img+5
    • umapper1Contributor
    • Contributor
    • June 2, 2020

    I’m not sure about a dictionary but you can use conditional statements in the attributecreator transformer to assign a common name to all features that match your search criteria.

    takashi
    Celebrity
    • takashiCelebrity
    • June 2, 2020

    Hi @ppp19, if all the possible aliases of "Copper" are known, the StringPairReplacer would be available to convert every alias to "Copper".

    Why not inspect features with Visual/Data Preview and Feature/Record Information before writing them into a destination dataset?
    jdh
    Contributor
    Forum|alt.badge.img+40
    • jdhContributor
    • Contributor
    • Best Answer
    • June 2, 2020

    I would probably do this as a mapping table.

     

    Have an excel or csv file with two columns the OriginalName and the CommonName and then use a DatabaseJoiner to attach the common name as an attribute.

     

     

    This way the "dictionary" can be maintained without any edits to the FME workspace.

     

    Ex.OriginalNameCommonNameCUCopperCOPCopperFEIronIRIron

     

    You would also be able to handle any unjoined features, where the council name is not in the dictionary. Whereas the StringPairReplacer option would just pass those features through untouched.

    J
    Forum|alt.badge.img+2

    I would probably do this as a mapping table.

     

    Have an excel or csv file with two columns the OriginalName and the CommonName and then use a DatabaseJoiner to attach the common name as an attribute.

     

     

    This way the "dictionary" can be maintained without any edits to the FME workspace.

     

    Ex.OriginalNameCommonNameCUCopperCOPCopperFEIronIRIron

     

    You would also be able to handle any unjoined features, where the council name is not in the dictionary. Whereas the StringPairReplacer option would just pass those features through untouched.

    I agree with this approach. Preserves the original data and creates the alias field.

    P
    Forum|alt.badge.img
    • ppp19
    • Author
    • June 3, 2020

    I would probably do this as a mapping table.

     

    Have an excel or csv file with two columns the OriginalName and the CommonName and then use a DatabaseJoiner to attach the common name as an attribute.

     

     

    This way the "dictionary" can be maintained without any edits to the FME workspace.

     

    Ex.OriginalNameCommonNameCUCopperCOPCopperFEIronIRIron

     

    You would also be able to handle any unjoined features, where the council name is not in the dictionary. Whereas the StringPairReplacer option would just pass those features through untouched.

    Hi @jdh, this DatabaseJoiner worked well for me...However, I have few raws in my csv file which is blank (having no material name in OriginalName column as you described)...and don't know how to map those rows with common name...because those raws showing unjoined in Databasejoiner.

    Regards,

     

    P
    Forum|alt.badge.img
    • ppp19
    • Author
    • June 3, 2020

    Hi @jdh, this DatabaseJoiner worked well for me...However, I have few raws in my csv file which is blank (having no material name in OriginalName column as you described)...and don't know how to map those rows with common name...because those raws showing unjoined in Databasejoiner.

    Regards,

     

    Hi @jdh, please ignore my previous reply...I have now figured out my blank raw issue.

    Regards,

    P
    Forum|alt.badge.img
    • ppp19
    • Author
    • June 3, 2020

    Hi @ppp19, if all the possible aliases of "Copper" are known, the StringPairReplacer would be available to convert every alias to "Copper".

    Hi @takashi, the above idea of databasejoiner worked for me..thanks for your time to reply me on this issue.

    Regards,

    P
    Forum|alt.badge.img
    • ppp19
    • Author
    • June 3, 2020

    I’m not sure about a dictionary but you can use conditional statements in the attributecreator transformer to assign a common name to all features that match your search criteria.

    Hi @umapper1, the below idea of databasejoiner worked for me.. thanks for your time to reply me on this issue.

    Regards,

    P
    Forum|alt.badge.img
    • ppp19
    • Author
    • June 9, 2020

    Hi @jdh, please ignore my previous reply...I have now figured out my blank raw issue.

    Regards,

    Hi@jdh,

    Hi I have created an excel sheet with adding attribute values for Material and used the DatbaseJoiner function of FME to Map column A with Column B and C as per our previous discussion. Well, it works good for all other material but it is showing unjoined only for MDPE, AK and mPVC. I did everything to rejoin this mapping but it is not taking only these 3 material names in consideration and shows unjoined in FME.... :(

    Any reason you can see here?

    Regards,

     

    jdh
    Contributor
    Forum|alt.badge.img+40
    • jdhContributor
    • Contributor
    • June 9, 2020

    Hi@jdh,

    Hi I have created an excel sheet with adding attribute values for Material and used the DatbaseJoiner function of FME to Map column A with Column B and C as per our previous discussion. Well, it works good for all other material but it is showing unjoined only for MDPE, AK and mPVC. I did everything to rejoin this mapping but it is not taking only these 3 material names in consideration and shows unjoined in FME.... :(

    Any reason you can see here?

    Regards,

     

    The first thing to do is check of there are any white spaces in either the csv or the actual council data. 'MDPE ' is not the same as 'MDPE'. If it's in the data an AttributeTrimmer can strip the extraneous white space.

    P
    Forum|alt.badge.img
    • ppp19
    • Author
    • June 10, 2020

    The first thing to do is check of there are any white spaces in either the csv or the actual council data. 'MDPE ' is not the same as 'MDPE'. If it's in the data an AttributeTrimmer can strip the extraneous white space.

    Hi @jdh...it worked well!!!...Thanks for your help... :)

    Regards

     

    Community Stats

    32,207
    Posts
    122,305
    Replies
    39,891
    Members

    Latest FME

    Terms & ConditionsCookie settingsAccessibility statement
    A product of
    Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.