Question

Check if set of attributes exist or not

  • 1 September 2017
  • 16 replies
  • 81 views
D
Badge

Hi,

I want to check if set of attributes(not the values in attributes) exist in feature type or not?

I also want to get the name of attributes to generate HTML Report which are Present or not present in Featuretypes.

Case: I have multiple EXCEL files

First file with column name (A,B,C,D,F)

Second file with column name (A,B,D,E)

I want to check in which file which columns are missing out of Set {A,B,C,D,E,F,G}

I want to retrieve the column names as well to generate HTML Report also.

Please help on this topic, Thanks in advance.

16 replies

ebygomm
Rank
Userlevel 1
Badge +10

You can use an attributecreator with conditional statements to create a flag for if the attribute is missing or not

Or you could use an attribute validator, although I've not yet found it possible to input a list of attributes to test, so you have to set it up when you have a file with all attributes linked.

This then gives output such as "Attribute 'F' fails check for Has a Value" *this will work if input is excel, may not work for other inputs

D
Badge

You can use an attributecreator with conditional statements to create a flag for if the attribute is missing or not

Or you could use an attribute validator, although I've not yet found it possible to input a list of attributes to test, so you have to set it up when you have a file with all attributes linked.

This then gives output such as "Attribute 'F' fails check for Has a Value" *this will work if input is excel, may not work for other inputs

Hi,

 

 

Thanks for your answer, But 'Attribute is missing' condition in 'Test Filter' is not checking whether there is attribute present or not in a file, It is checking that Whether the attribute 'A' has values or not?

 

As i mentioned above i want to check whether is there any attribute in file or not?

 

 

Thanks again.
ebygomm
Rank
Userlevel 1
Badge +10
Hi,

 

 

Thanks for your answer, But 'Attribute is missing' condition in 'Test Filter' is not checking whether there is attribute present or not in a file, It is checking that Whether the attribute 'A' has values or not?

 

As i mentioned above i want to check whether is there any attribute in file or not?

 

 

Thanks again.
Attribute is missing checks whether the attribute is present or not, try it and see

 

missing-attribute.fmw

 

 

mark2atsafe
Rank
Userlevel 4
Badge +25

If you just need to know what columns exist, and don't care about the data (for this process at least) then try reading the Excel files with the format "Schema (Any Format)" reader. This will return a list of attributes that you can examine.

However, as others have noted, the Tester "Attribute is Missing" test does test for the existence of an attribute (not whether it has a value) so would be a good alternative if you do need to read the actual data values.

erik_jan
Rank
Userlevel 2
Badge +12

Following up on what @Mark2AtSafe is suggesting:

I would use the Schema (Any format) reader to read the schema of the files to check.

Then have a list of required attributes in another file (Excel, csv or the like) and read that data.

Use the FeatureMerger to merge the read schema (As Requestor) and list of required attributes (as Supplier).

The Merged port will output the attributes that are in the schema, the Unused Supplier port will output the missing attributes.

D
Badge

Following up on what @Mark2AtSafe is suggesting:

I would use the Schema (Any format) reader to read the schema of the files to check.

Then have a list of required attributes in another file (Excel, csv or the like) and read that data.

Use the FeatureMerger to merge the read schema (As Requestor) and list of required attributes (as Supplier).

The Merged port will output the attributes that are in the schema, the Unused Supplier port will output the missing attributes.

Not working for Attributes in CAD file as Schema(Any format) reader is not reading attribute names in CAD file it is readable only using AutoCAD Map 3D Object Data Reader.

 

 

D
Badge

I think TCL caller is checking if attribute exist or not but i am wondering if it allow me to check multiple fields.

FME_AttributeExists <attributeName>

Issue : its just taking one argument/field

If any one can help me on this?

all above solutions are not serving the purpose.

geosander
Badge
Hi,

 

 

Thanks for your answer, But 'Attribute is missing' condition in 'Test Filter' is not checking whether there is attribute present or not in a file, It is checking that Whether the attribute 'A' has values or not?

 

As i mentioned above i want to check whether is there any attribute in file or not?

 

 

Thanks again.
Yes, what @egomm says should work for you. 'Attribute has a value' should check if the attribute has a value, as the name already implies ;)

 

Strange actually that the AttributeValidator does not have a 'Attribute exists' option. Maybe because you don't have an option to type in the name of an attribute (when not exposed)...

 

By the way, you can also test directly if a feature has all required attributes or not by using a Tester like so (i.e. you don't need to flag using an AttributeCreator if you don't want to add more attributes to your features):

 

 

 

nielsgerrits
Rank
Userlevel 6
Badge +32
Not working for Attributes in CAD file as Schema(Any format) reader is not reading attribute names in CAD file it is readable only using AutoCAD Map 3D Object Data Reader.

 

 

If you use a FeatureReader instead of a normal reader you might get better results. Use Schema output port.
mark2atsafe
Rank
Userlevel 4
Badge +25
Not working for Attributes in CAD file as Schema(Any format) reader is not reading attribute names in CAD file it is readable only using AutoCAD Map 3D Object Data Reader.

 

 

Add a second reader that reads the data as AutoCAD Map3D too. Then the parameters you use for that reader will also apply to the Schema reader (if you set its format to be Map3D)

 

 

takashi
Rank
Userlevel 2
Badge +17

If the FME_AttributeExists function can check if "attribute exists" as you say, a TclCaller with this procedure (Source Code) should work for multiple attributes.

proc findMissingAttributes {} {
    set attrs [list "A" "B" "C" "D" "E" "F" "G"]
    set result {}
    foreach attr $attrs {
        if {![FME_AttributeExists $attr]} {
            lappend result $attr
        }
    }
    return [join $result ","]
}
A PythonCaller with this script brings the same result.
def findMissingAttributes(feature):
    attrs = ['A', 'B', 'C', 'D', 'E', 'F', 'G']
    result = []
    for attr in attrs:
        _, isMissing, _ = feature.getAttributeNullMissingAndType(attr)
        if isMissing:
            result.append(attr)
    feature.setAttribute('_result', ','.join(result))
However, the results from both scripts above are equivalent to the result from @egomm's suggestion - use of the Attribute is Missing operator. Perhaps do you intend to check whether some specific user attributes (e.g. A, B, C, D, E, F, G) are defined in the schema of a source feature type?If so, you can read schema features from the source dataset with the Schema (Any Format) reader, and see the elements of "attribute{}.name" list to check if names of required attributes are defined.
takashi
Rank
Userlevel 2
Badge +17

If the FME_AttributeExists function can check if "attribute exists" as you say, a TclCaller with this procedure (Source Code) should work for multiple attributes.

proc findMissingAttributes {} {
    set attrs [list "A" "B" "C" "D" "E" "F" "G"]
    set result {}
    foreach attr $attrs {
        if {![FME_AttributeExists $attr]} {
            lappend result $attr
        }
    }
    return [join $result ","]
}
A PythonCaller with this script brings the same result.
def findMissingAttributes(feature):
    attrs = ['A', 'B', 'C', 'D', 'E', 'F', 'G']
    result = []
    for attr in attrs:
        _, isMissing, _ = feature.getAttributeNullMissingAndType(attr)
        if isMissing:
            result.append(attr)
    feature.setAttribute('_result', ','.join(result))
However, the results from both scripts above are equivalent to the result from @egomm's suggestion - use of the Attribute is Missing operator. Perhaps do you intend to check whether some specific user attributes (e.g. A, B, C, D, E, F, G) are defined in the schema of a source feature type?If so, you can read schema features from the source dataset with the Schema (Any Format) reader, and see the elements of "attribute{}.name" list to check if names of required attributes are defined.
@erik_jan has already suggested the Schema (Any Format). If "check missing attributes" approach and "check schema" approach both couldn't resolve your issue, anyone haven't understand your actual requirement yet.
takashi
Rank
Userlevel 2
Badge +17

If the FME_AttributeExists function can check if "attribute exists" as you say, a TclCaller with this procedure (Source Code) should work for multiple attributes.

proc findMissingAttributes {} {
    set attrs [list "A" "B" "C" "D" "E" "F" "G"]
    set result {}
    foreach attr $attrs {
        if {![FME_AttributeExists $attr]} {
            lappend result $attr
        }
    }
    return [join $result ","]
}
A PythonCaller with this script brings the same result.
def findMissingAttributes(feature):
    attrs = ['A', 'B', 'C', 'D', 'E', 'F', 'G']
    result = []
    for attr in attrs:
        _, isMissing, _ = feature.getAttributeNullMissingAndType(attr)
        if isMissing:
            result.append(attr)
    feature.setAttribute('_result', ','.join(result))
However, the results from both scripts above are equivalent to the result from @egomm's suggestion - use of the Attribute is Missing operator. Perhaps do you intend to check whether some specific user attributes (e.g. A, B, C, D, E, F, G) are defined in the schema of a source feature type?If so, you can read schema features from the source dataset with the Schema (Any Format) reader, and see the elements of "attribute{}.name" list to check if names of required attributes are defined.
I re-read the comments posted under @erik_jan's answer. @nielsgerrits and @Mark2AtSafe have posted good suggestions. Why not try those?
G
Badge +3

@dheeraj

explode and create list of attr_names.

Use a llistsearcher on it with regexp Attr1||Attr2|||Attr..||..

Very conveniently put in a custom. Works nice for me.

D
Badge

@dheeraj

explode and create list of attr_names.

Use a llistsearcher on it with regexp Attr1||Attr2|||Attr..||..

Very conveniently put in a custom. Works nice for me.

Can you please explain the answer a little bit more step by step?

 

I think it might be helpful to me as well..

 

Thanks in advance.
G
Badge +3

@dheeraj

explode and create list of attr_names.

Use a llistsearcher on it with regexp Attr1||Attr2|||Attr..||..

Very conveniently put in a custom. Works nice for me.

@dheeraj

 

 

Here is a method. Wich only requires the test string as input.File reading by either FeatureReader or aReader in advanced mode.

 

Reply


Community Stats

28,670
Posts
109,365
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.