Question

How can i pass this published parameter to "Attributes to remove" field to the writer dialog box

f.kemminje
Badge +3

Hi Friends,

two questions :

1. We can type the attribute names in this text field, but we cannot pass it like variable or text string(list). How can i pass this published parameter to "Attributes to remove" field to the writer dialog box?

2. If i write to shape and it crosses 2gb size fme gives error. how vcan i split the shape file if it crosses 2gb? (like fme splits v7 dgn file size in V7 writer)

Explanation :

My fmw has Dynamic reader and writers. I am reading from oracle spatial (every time input tablename changes, user types in table name through published parameter) and writing to any 4 formats.(shp, tab,dgn and acad) here my fmw is dynamic. there are some attribute names that i dont want to write to destination. user types in these attribute names(space delimited). through published parameters. Every time this list changes and different attribute names comes in. In the Dynamic properties, there is a provision called "Attributes to remove". Now I have two questions :

 

1. We can type the attribute names in this text field, but we cannot pass it like variable or text string(list). How can i pass this published parameter to "Attributes to remove" field to the writer dialog box? 2. If i write to shape and it crosses 2gb size fme gives error. how vcan i split the shape file if it crosses 2gb? (like fme splits v7 dgn file size in V7 writer)

12 replies

redgeographics
Rank
Userlevel 5
Badge +25

Regarding your 2nd question, that's a shapefile spec limitation, there's an idea outstanding at the moment to fix that: Automatically tile shapefile when approaching 2gb, I suggest you vote for that if you haven't done so already.

In the meantime, there's not much you can do. Since you know your data better than we do, do you have any way of predicting how big the output could be based on the user input? If so you could add a Tiler to automatically cut the data up.

pratap
Rank
Badge +2

Hi,

1. As you are writing to 4 formats so you would have placed 4 writers. So place attribute keeper based on the writer and pass only the required attributes. Hope this helps

2. No idea

f.kemminje
Badge +3

Hi,

1. As you are writing to 4 formats so you would have placed 4 writers. So place attribute keeper based on the writer and pass only the required attributes. Hope this helps

2. No idea

pratap,

 

 

But Dynamic writer takes the structure(schema definition) from the reader. I mean writer creates the column names from the reader. if you use attribute keeper , still the writer will cteate these columns and null values will be there.

 

 

L
Badge

For the attributes to remove, I would try to use a FeatureReader to read from Oracle and read the schema to. Then, alter the schema feature to remove the attributes to remove from the attribute list and use this schema feature with the dynamic writer.

pratap
Rank
Badge +2
pratap,

 

 

But Dynamic writer takes the structure(schema definition) from the reader. I mean writer creates the column names from the reader. if you use attribute keeper , still the writer will cteate these columns and null values will be there.

 

 

Hi,

 

When we select "Automatic" in User Attributes --> Attribute Definition then attributes will be created in writer only when you are pass the attribute.

 

 

david_r
Rank
Userlevel 4

As mentioned by @larry, I think the only solution is to remove the attribute definition in the schema feature. Here's a custom transformer using a PythonCaller that can remove one or several attributes from the schema defintion. The list of attribute names to remove can be linked to a published or private parameter.

Example use case:

0684Q00000ArKKlQAN.png

Workspace template: removeschemafeatures.fmwt

Note that the attribute names listed in the custom transformer parameter are case sensitive. The custom transformer will print a warning message to the FME log whenever an attribute definition has been removed from the schema feature.

PythonCaller code:

import fmeobjectsclass RemoveSchemaAttrs(object):    def input(self,feature):        attr_names = feature.getAttribute('attribute{}.name')        attr_fme_datatypes = feature.getAttribute('attribute{}.fme_data_type')        attr_native_datatypes = feature.getAttribute('attribute{}.native_data_type')        attrs_to_remove = feature.getAttribute('_SCHEMA_ATTRS_TO_REMOVE').split(',')        fme_feature_type_name = feature.getAttribute('fme_feature_type_name')        for attr_index, attr_name in enumerate(attr_names):            if attr_name in attrs_to_remove:                fmeobjects.FMELogFile().logMessageString(                    'Removing schema attribute "%s" from schema "%s"' %                     (attr_name, fme_feature_type_name), fmeobjects.FME_WARN)                del attr_names[attr_index]                del attr_fme_datatypes[attr_index]                del attr_native_datatypes[attr_index]        feature.setAttribute('attribute{}.name', attr_names)        feature.setAttribute('attribute{}.fme_data_type', attr_fme_datatypes)        feature.setAttribute('attribute{}.native_data_type', attr_native_datatypes)        self.pyoutput(feature)
f.kemminje
Badge +3

Thank you, Larry And David.

f.kemminje
Badge +3

@david_r

 

 

Hi David, I used your python code to remove attributes. but python shuffles the list as soon as we delete the attr_names. so here index will be different when we delete. i mean if i delete second item then third item becomes second in the index. so it gives wrong attributenames at the end writer.

 

 

for this issue i used opposite 

 

i created the new list which has attributes to keep

 

i modified your python code, but still i am not successful, in the destination dataset has all the attributes name.

 

 

please help. i just pasted my py code.

 

import fmeobjects
class RemoveSchemaAttrs(object):
    def input(self,feature):
        newattr_names=[]
        newattr_fme_datatypes=[]
        newattr_native_datatypes=[]
        attr_names = feature.getAttribute('attribute{}.name')
        attr_fme_datatypes = feature.getAttribute('attribute{}.fme_data_type')
        attr_native_datatypes = feature.getAttribute('attribute{}.native_data_type')
        attrs_to_remove = feature.getAttribute('_SCHEMA_ATTRS_TO_REMOVE').split(',')
        fme_feature_type_name = feature.getAttribute('fme_feature_type_name')
        for attr_index, attr_name in enumerate(attr_names):
            if attr_name not in attrs_to_remove:
                newattr_names.append(attr_names[attr_index])
                newattr_fme_datatypes.append(attr_fme_datatypes[attr_index])
                newattr_native_datatypes.append(attr_native_datatypes[attr_index])
                fmeobjects.FMELogFile().logMessageString(
                    'qqq "%s" from schema "%s"' % 
                    (attr_names[attr_index], fme_feature_type_name), fmeobjects.FME_WARN)
        feature.setAttribute('attribute{}.name', newattr_names)
        feature.setAttribute('attribute{}.fme_data_type', newattr_fme_datatypes)
        feature.setAttribute('attribute{}.native_data_type', newattr_native_datatypes)
        b=0
        for b, attr_name in enumerate(newattr_names):
            fmeobjects.FMELogFile().logMessageString(
                    'end1 "%s" from schema "%s"' % 
                    (newattr_names[b], fme_feature_type_name), fmeobjects.FME_WARN)
        self.pyoutput(feature)

david_r
Rank
Userlevel 4

@david_r

 

 

Hi David, I used your python code to remove attributes. but python shuffles the list as soon as we delete the attr_names. so here index will be different when we delete. i mean if i delete second item then third item becomes second in the index. so it gives wrong attributenames at the end writer.

 

 

for this issue i used opposite 

 

i created the new list which has attributes to keep

 

i modified your python code, but still i am not successful, in the destination dataset has all the attributes name.

 

 

please help. i just pasted my py code.

 

import fmeobjects
class RemoveSchemaAttrs(object):
    def input(self,feature):
        newattr_names=[]
        newattr_fme_datatypes=[]
        newattr_native_datatypes=[]
        attr_names = feature.getAttribute('attribute{}.name')
        attr_fme_datatypes = feature.getAttribute('attribute{}.fme_data_type')
        attr_native_datatypes = feature.getAttribute('attribute{}.native_data_type')
        attrs_to_remove = feature.getAttribute('_SCHEMA_ATTRS_TO_REMOVE').split(',')
        fme_feature_type_name = feature.getAttribute('fme_feature_type_name')
        for attr_index, attr_name in enumerate(attr_names):
            if attr_name not in attrs_to_remove:
                newattr_names.append(attr_names[attr_index])
                newattr_fme_datatypes.append(attr_fme_datatypes[attr_index])
                newattr_native_datatypes.append(attr_native_datatypes[attr_index])
                fmeobjects.FMELogFile().logMessageString(
                    'qqq "%s" from schema "%s"' % 
                    (attr_names[attr_index], fme_feature_type_name), fmeobjects.FME_WARN)
        feature.setAttribute('attribute{}.name', newattr_names)
        feature.setAttribute('attribute{}.fme_data_type', newattr_fme_datatypes)
        feature.setAttribute('attribute{}.native_data_type', newattr_native_datatypes)
        b=0
        for b, attr_name in enumerate(newattr_names):
            fmeobjects.FMELogFile().logMessageString(
                    'end1 "%s" from schema "%s"' % 
                    (newattr_names[b], fme_feature_type_name), fmeobjects.FME_WARN)
        self.pyoutput(feature)

The thing with lists in FME is that they have to be sequential starting from 0, with no gaps. 

 

 

If you have gaps in your list sequence, you will lose everything starting from the first gap. So if you delete item 2, item 3 has to be the new item 2, otherwise it won't work.
f.kemminje
Badge +3
The thing with lists in FME is that they have to be sequential starting from 0, with no gaps.

 

 

If you have gaps in your list sequence, you will lose everything starting from the first gap. So if you delete item 2, item 3 has to be the new item 2, otherwise it won't work.

 

@david_r

 

 

So David what can I do now?

 

i do not have gaps. More over i created new list (the attributes i needed) and sent this new list to the feature. But still fme writes all the columns(including unwanted columns)

 

 

pls help?

 

 

 

david_r
Rank
Userlevel 4

 

@david_r

 

 

So David what can I do now?

 

i do not have gaps. More over i created new list (the attributes i needed) and sent this new list to the feature. But still fme writes all the columns(including unwanted columns)

 

 

pls help?

 

 

 

I'm not sure I understand the problem. The code I posted works for my test case. Can you give a concrete example of what you think is wrong with it, and I'll have a look.
f.kemminje
Badge +3

@david_r

 

 

Hi David, I used your python code to remove attributes. but python shuffles the list as soon as we delete the attr_names. so here index will be different when we delete. i mean if i delete second item then third item becomes second in the index. so it gives wrong attributenames at the end writer.

 

 

for this issue i used opposite 

 

i created the new list which has attributes to keep

 

i modified your python code, but still i am not successful, in the destination dataset has all the attributes name.

 

 

please help. i just pasted my py code.

 

import fmeobjects
class RemoveSchemaAttrs(object):
    def input(self,feature):
        newattr_names=[]
        newattr_fme_datatypes=[]
        newattr_native_datatypes=[]
        attr_names = feature.getAttribute('attribute{}.name')
        attr_fme_datatypes = feature.getAttribute('attribute{}.fme_data_type')
        attr_native_datatypes = feature.getAttribute('attribute{}.native_data_type')
        attrs_to_remove = feature.getAttribute('_SCHEMA_ATTRS_TO_REMOVE').split(',')
        fme_feature_type_name = feature.getAttribute('fme_feature_type_name')
        for attr_index, attr_name in enumerate(attr_names):
            if attr_name not in attrs_to_remove:
                newattr_names.append(attr_names[attr_index])
                newattr_fme_datatypes.append(attr_fme_datatypes[attr_index])
                newattr_native_datatypes.append(attr_native_datatypes[attr_index])
                fmeobjects.FMELogFile().logMessageString(
                    'qqq "%s" from schema "%s"' % 
                    (attr_names[attr_index], fme_feature_type_name), fmeobjects.FME_WARN)
        feature.setAttribute('attribute{}.name', newattr_names)
        feature.setAttribute('attribute{}.fme_data_type', newattr_fme_datatypes)
        feature.setAttribute('attribute{}.native_data_type', newattr_native_datatypes)
        b=0
        for b, attr_name in enumerate(newattr_names):
            fmeobjects.FMELogFile().logMessageString(
                    'end1 "%s" from schema "%s"' % 
                    (newattr_names[b], fme_feature_type_name), fmeobjects.FME_WARN)
        self.pyoutput(feature)

IT WORKED Now. Thanks David

 

we need to remove fme attributes first befoe setting.

 

 

and i used attributes to keep. built new list

 

and sent to fme

 

 

        feature.removeAttribute('attribute{}.name')
        feature.removeAttribute('attribute{}.native_data_type')
        feature.removeAttribute('attribute{}.fme_data_type') 

 

 

 

Reply


Community Stats

28,696
Posts
109,440
Replies
36,546
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.