Skip to main content
Solved

creating a data dictionary in FME?

P
Forum|alt.badge.img

Hi,

I have different councils Pipe material list with a specific example of say Copper – which different council has a different way to define it.. say for example one council called it “CU”, other council calls it “COP” and some other councils call it with some other name…however, the material is the same…and in general we want to define a common name for it..... say "Copper"? Can we create a data dictionary in FME?...which can identify and recognize the correct name and replace it with a generic name?

 

 

Best answer by jdh

I would probably do this as a mapping table.

 

Have an excel or csv file with two columns the OriginalName and the CommonName and then use a DatabaseJoiner to attach the common name as an attribute.

 

 

This way the "dictionary" can be maintained without any edits to the FME workspace.

 

Ex.OriginalNameCommonNameCUCopperCOPCopperFEIronIRIron

 

You would also be able to handle any unjoined features, where the council name is not in the dictionary. Whereas the StringPairReplacer option would just pass those features through untouched.

View original
    Did this help you find an answer to your question?

    11 replies

    umapper1
    Contributor
    Forum|alt.badge.img+5
    • umapper1Contributor
    • Contributor
    • June 2, 2020

    I’m not sure about a dictionary but you can use conditional statements in the attributecreator transformer to assign a common name to all features that match your search criteria.

    takashi
    Influencer
    • takashiInfluencer
    • June 2, 2020

    Hi @ppp19, if all the possible aliases of "Copper" are known, the StringPairReplacer would be available to convert every alias to "Copper".

    jdh
    Contributor
    Forum|alt.badge.img+28
    • jdhContributor
    • Contributor
    • Best Answer
    • June 2, 2020

    I would probably do this as a mapping table.

     

    Have an excel or csv file with two columns the OriginalName and the CommonName and then use a DatabaseJoiner to attach the common name as an attribute.

     

     

    This way the "dictionary" can be maintained without any edits to the FME workspace.

     

    Ex.OriginalNameCommonNameCUCopperCOPCopperFEIronIRIron

     

    You would also be able to handle any unjoined features, where the council name is not in the dictionary. Whereas the StringPairReplacer option would just pass those features through untouched.

    J
    Forum|alt.badge.img+2
    jdh wrote:

    I would probably do this as a mapping table.

     

    Have an excel or csv file with two columns the OriginalName and the CommonName and then use a DatabaseJoiner to attach the common name as an attribute.

     

     

    This way the "dictionary" can be maintained without any edits to the FME workspace.

     

    Ex.OriginalNameCommonNameCUCopperCOPCopperFEIronIRIron

     

    You would also be able to handle any unjoined features, where the council name is not in the dictionary. Whereas the StringPairReplacer option would just pass those features through untouched.

    I agree with this approach. Preserves the original data and creates the alias field.

    P
    Forum|alt.badge.img
    • ppp19
    • Author
    • June 3, 2020
    jdh wrote:

    I would probably do this as a mapping table.

     

    Have an excel or csv file with two columns the OriginalName and the CommonName and then use a DatabaseJoiner to attach the common name as an attribute.

     

     

    This way the "dictionary" can be maintained without any edits to the FME workspace.

     

    Ex.OriginalNameCommonNameCUCopperCOPCopperFEIronIRIron

     

    You would also be able to handle any unjoined features, where the council name is not in the dictionary. Whereas the StringPairReplacer option would just pass those features through untouched.

    Hi @jdh, this DatabaseJoiner worked well for me...However, I have few raws in my csv file which is blank (having no material name in OriginalName column as you described)...and don't know how to map those rows with common name...because those raws showing unjoined in Databasejoiner.

    Regards,

     

    P
    Forum|alt.badge.img
    • ppp19
    • Author
    • June 3, 2020
    ppp19 wrote:

    Hi @jdh, this DatabaseJoiner worked well for me...However, I have few raws in my csv file which is blank (having no material name in OriginalName column as you described)...and don't know how to map those rows with common name...because those raws showing unjoined in Databasejoiner.

    Regards,

     

    Hi @jdh, please ignore my previous reply...I have now figured out my blank raw issue.

    Regards,

    P
    Forum|alt.badge.img
    • ppp19
    • Author
    • June 3, 2020
    takashi wrote:

    Hi @ppp19, if all the possible aliases of "Copper" are known, the StringPairReplacer would be available to convert every alias to "Copper".

    Hi @takashi, the above idea of databasejoiner worked for me..thanks for your time to reply me on this issue.

    Regards,

    P
    Forum|alt.badge.img
    • ppp19
    • Author
    • June 3, 2020
    umapper1 wrote:

    I’m not sure about a dictionary but you can use conditional statements in the attributecreator transformer to assign a common name to all features that match your search criteria.

    Hi @umapper1, the below idea of databasejoiner worked for me.. thanks for your time to reply me on this issue.

    Regards,

    P
    Forum|alt.badge.img
    • ppp19
    • Author
    • June 9, 2020
    ppp19 wrote:

    Hi @jdh, please ignore my previous reply...I have now figured out my blank raw issue.

    Regards,

    Hi@jdh,

    Hi I have created an excel sheet with adding attribute values for Material and used the DatbaseJoiner function of FME to Map column A with Column B and C as per our previous discussion. Well, it works good for all other material but it is showing unjoined only for MDPE, AK and mPVC. I did everything to rejoin this mapping but it is not taking only these 3 material names in consideration and shows unjoined in FME.... :(

    Any reason you can see here?

    Regards,

     

    jdh
    Contributor
    Forum|alt.badge.img+28
    • jdhContributor
    • Contributor
    • June 9, 2020
    ppp19 wrote:

    Hi@jdh,

    Hi I have created an excel sheet with adding attribute values for Material and used the DatbaseJoiner function of FME to Map column A with Column B and C as per our previous discussion. Well, it works good for all other material but it is showing unjoined only for MDPE, AK and mPVC. I did everything to rejoin this mapping but it is not taking only these 3 material names in consideration and shows unjoined in FME.... :(

    Any reason you can see here?

    Regards,

     

    The first thing to do is check of there are any white spaces in either the csv or the actual council data. 'MDPE ' is not the same as 'MDPE'. If it's in the data an AttributeTrimmer can strip the extraneous white space.

    P
    Forum|alt.badge.img
    • ppp19
    • Author
    • June 10, 2020
    jdh wrote:

    The first thing to do is check of there are any white spaces in either the csv or the actual council data. 'MDPE ' is not the same as 'MDPE'. If it's in the data an AttributeTrimmer can strip the extraneous white space.

    Hi @jdh...it worked well!!!...Thanks for your help... :)

    Regards

     

    Reply


    Helpful Members This Week

    Community Stats

    31,378
    Posts
    119,012
    Replies
    39,009
    Members

    Latest FME

    Terms & ConditionsCookie settings

    Cookie policy

    We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

     
    Cookie settings
    A product of
    Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.