Question

A field type of '-999' was specified

B
Badge

I'm trying to mimic the FeatureWriter available in FME 2016 in FME2015.1 since upgrading is not an option. I'm using a pythoncaller with FMEObjects to try and do this. As a basis I use the sample python scriptBaseLineWriter.py that came with the installation with FME.

I start out by creating a unique filename for my GDB, then I a getting the schema from the feature, set the schema on the writer and write to the GDB.

import fmeobjects

 

from fmeobjects import FMEUniversalWriter

 

from fmeobjects import FMEUniversalReader

 

import random

 

import os, sys

# Get variables and create output name for temp fGDB

 

folder = FME_MacroValues['FME_SHAREDRESOURCE_TEMP']

 

ext = '.gdb'

 

current_files = os.listdir(folder)

# Get a list of current FGDB's

 

fgdbs = []

 

current_numbers = []

 

for curfile in current_files:

 

if curfile.endswith(ext):

 

fgdbs.append(curfile)

 

# Extract the numbers from the GDB filenames

 

for fgdb in fgdbs:

 

s = int(filter(str.isdigit, fgdb))

 

current_numbers.append(s)

 

print "List of fGDBs: ",fgdbs

 

print "List of fGDBs: ",current_numbers

#Prefix for the FGDB names

 

db = '\\\\\\conversion_temp'

# create a list of 0 - 999 and remove the existing numbers

 

numbers = range(0,1000)

 

for x in current_numbers:

 

numbers.remove(x)

 

nr = random.choice(numbers)

 

# Create the filename

 

db = str(folder + db + str(nr) + ext)

 

print db

# Create a log file object.

 

logger = fmeobjects.FMELogFile()

 

logger.setFileName("baseLineWriter.log", False)

#Create a writer

 

ObjWriter = FMEUniversalWriter('GEODATABASE_FILE',db)

 

ObjWriter.open(db)

def processFeature(feature):

 

# Create schema feature to describe features sent to writer.

 

schemaFeature = feature

 

ft = feature.getAttribute('fme_feature_type')

 

print "FT: ",(ft)

 

schemaFeature.setFeatureType(ft)

 

# Log the schema feature.

 

logger.logMessageString("Schema feature:")

 

logger.logFeature(schemaFeature)

 

# Add the schema feature to the writer.

 

ObjWriter.addSchema(feature)

 

# Log the data feature.

 

logger.logFeature(feature)

 

ObjWriter.write(feature)

 

ObjWriter.close()

 

feature.setAttribute('_loc',db)

My workspace consists of a SHAPE-reader, the python caller and an inspector, so not much that could confuce FME. When I run the workspace, it throws me the error:

A field type of '-999' was specified, which is not valid. Please check the documentation on the Geodatabase Reader/Writer for information on valid field types.

When I look at the writer docs, the tekst "field type" is not on the page, so that doesn't help too much. My data contains columns with the value -999, but when I uncheck those on the reader, it still produces the same error. Does anyone have any thoughts?

11 replies

takashi
Rank
Userlevel 2
Badge +17

Hi @bzwemmer, you will have to read schema features from the source dataset and add them to the writer before writing data features. This is a minimal example.

# PythonCaller Script Example: Dynamic Schema [GEODATABASE_FILE] Writer
# Destination schema is derived from Source Esri Shapefile Dataset.
# Each schema definition name is equal to reader feature type name.
# The writer overwrites existing GDB dataset.
# All writer directives and parameters except 'OVERWRITE_GEODB' are left as default.
# Error handling is omitted.
import fmeobjects
class FeatureProcessor(object):
    def __init__(self):
        gdb = 'C:/FME/tmp/test.gdb' # TODO: Change it to the actual destination dataset.
        directives = [] # All default.
        parameters = ['OVERWRITE_GEODB','YES'] # All default except 'OVERWRITE_GEODB'.
        self.writer = fmeobjects.FMEUniversalWriter('GEODATABASE_FILE', directives)
        self.writer.open(gdb, parameters)
        source = []
        ################################################################
        # TODO: Append all source Shapefile paths to the 'source' list.
        ################################################################
        for src in source:
            reader = fmeobjects.FMEUniversalReader('ESRISHAPE', False)
            reader.open(src)
            self.writer.addSchema(reader.readSchema()) # Add SCHEMA feature.
            reader.close()
        
    def input(self, feature):
        feature.setFeatureType(feature.getAttribute('fme_feature_type'))
        self.writer.write(feature) # Write DATA feature.
        
    def close(self):
        self.writer.close()
B
Badge

@takashi Thanks, but we have a workflow where the user is able to make modifications to the schema prior to this writer. This means that the schema from the source data is possibly not the same as the one we're trying to write. If I would include an inspector prior to this python script, the data inspector is showing the table with the modified column and feature class names. Therefor I assumed it would be possible to derive the schema from the feature itself.

B
Badge

@takashi Thanks, but we have a workflow where the user is able to make modifications to the schema prior to this writer. This means that the schema from the source data is possibly not the same as the one we're trying to write. If I would include an inspector prior to this python script, the data inspector is showing the table with the modified column and feature class names. Therefor I assumed it would be possible to derive the schema from the feature itself.

Sorry, I realised something else: is there a specification for the schema in pyton. Is it a list, or list of dicts or something? If that is the case, I can possibly recreate it in the pyhton.

takashi
Rank
Userlevel 2
Badge +17

@takashi Thanks, but we have a workflow where the user is able to make modifications to the schema prior to this writer. This means that the schema from the source data is possibly not the same as the one we're trying to write. If I would include an inspector prior to this python script, the data inspector is showing the table with the modified column and feature class names. Therefor I assumed it would be possible to derive the schema from the feature itself. 

If all the attribute names and their FME generic data types (fme_*), geometry type, and feature type name are known, you can create a schema feature and then set it to the writer object in a Python script. e.g.

        # Pairs of Attribute Name and FME Generic Data Type Name
        attrs = ['attr1', 'fme_char(20)', 'attr2', 'fme_date', 'attr3', 'fme_int32']
        
        # Create a schema feature.
        schema = fmeobjects.FMEFeature()
        for i in range(0, len(attrs), 2):
            # name: attribute name, value: data type name
            schema.setSequencedAttribute(attrs[i], attrs[i+1])
        schema.setAttribute('fme_geometry{0}', 'fme_line')
        schema.setFeatureType('FeatureTypeName')
        schema.setCoordSys('LL84') # optional

        # Set 'schema' to the writer object with the 'addSchema' method.

Regarding FME generic data types, see this article: Destination Schema is Derived from a Lookup Table

takashi
Rank
Userlevel 2
Badge +17

@takashi Thanks, but we have a workflow where the user is able to make modifications to the schema prior to this writer. This means that the schema from the source data is possibly not the same as the one we're trying to write. If I would include an inspector prior to this python script, the data inspector is showing the table with the modified column and feature class names. Therefor I assumed it would be possible to derive the schema from the feature itself.

Since you are using FME 2015.1, this method also might be apply to your workspace : Destination Schema is Derived from a Schema Feature

The method described in this article was known as "Destination Schema is Derived from a List" before. The mechanism itself has not been changed.

B
Badge

If all the attribute names and their FME generic data types (fme_*), geometry type, and feature type name are known, you can create a schema feature and then set it to the writer object in a Python script. e.g.

        # Pairs of Attribute Name and FME Generic Data Type Name
        attrs = ['attr1', 'fme_char(20)', 'attr2', 'fme_date', 'attr3', 'fme_int32']
        
        # Create a schema feature.
        schema = fmeobjects.FMEFeature()
        for i in range(0, len(attrs), 2):
            # name: attribute name, value: data type name
            schema.setSequencedAttribute(attrs[i], attrs[i+1])
        schema.setAttribute('fme_geometry{0}', 'fme_line')
        schema.setFeatureType('FeatureTypeName')
        schema.setCoordSys('LL84') # optional

        # Set 'schema' to the writer object with the 'addSchema' method.

Regarding FME generic data types, see this article: Destination Schema is Derived from a Lookup Table

Thanks @takashi, I'll give it a go by trying to create the list with python, since I don't know what it will be. With getSequencedAttribute and getFeatureType, I think it might be possible to get everything dynamically.  

B
Badge

I have a feeling we're getting close, so thanks a lot already. However; with keeping it dynamically I have the following code to get the attributes and their types:


attrs  = []<br>  san = feature.getAllAttributeNames()<br>  print "SAN: ",san<br>  for a in san:<br>  	at = feature.getAttributeType(a)<br>  	attrs.append(a)<br>  	attrs.append(at)
ft = feature.getAttribute('fme_feature_type')
cs = feature.getCoordSys()
ge = feature.getAttribute('fme_geometry')

I use these to set the schema:


# Create a schema feature.<br>  schema = fmeobjects.FMEFeature()<br>  for i in range(0, len(attrs), 2):<br>  # name: attribute name, value: data type name<br>  	schema.setAttribute(attrs[i], attrs[i+1])<br>  schema.setAttribute('fme_geometry{0}', ge)<br>  schema.setFeatureType(ft)<br>  schema.setCoordSys(cs)

This gives me a list of all attributes, which I loop over to get their attribute types. The attribute types are represented by numbers:


ATTRS:  ['SHAPE_Length', 9, 'DTB_ID', 6, 'geodb_oid', 6, 'geodb_type'

The list is longer, but this is to give an idea. When addind the schema to the writer:


# Add the schema feature to the writer.<br>  ObjWriter.addSchema(schema)

This gives the error:


No geometry mapping entry found for '11' in metafile GEODATABASE_FILE.fmf'. 
takashi
Rank
Userlevel 2
Badge +17

I have a feeling we're getting close, so thanks a lot already. However; with keeping it dynamically I have the following code to get the attributes and their types:


attrs  = []<br>  san = feature.getAllAttributeNames()<br>  print "SAN: ",san<br>  for a in san:<br>  	at = feature.getAttributeType(a)<br>  	attrs.append(a)<br>  	attrs.append(at)
ft = feature.getAttribute('fme_feature_type')
cs = feature.getCoordSys()
ge = feature.getAttribute('fme_geometry')

I use these to set the schema:


# Create a schema feature.<br>  schema = fmeobjects.FMEFeature()<br>  for i in range(0, len(attrs), 2):<br>  # name: attribute name, value: data type name<br>  	schema.setAttribute(attrs[i], attrs[i+1])<br>  schema.setAttribute('fme_geometry{0}', ge)<br>  schema.setFeatureType(ft)<br>  schema.setCoordSys(cs)

This gives me a list of all attributes, which I loop over to get their attribute types. The attribute types are represented by numbers:


ATTRS:  ['SHAPE_Length', 9, 'DTB_ID', 6, 'geodb_oid', 6, 'geodb_type'

The list is longer, but this is to give an idea. When addind the schema to the writer:


# Add the schema feature to the writer.<br>  ObjWriter.addSchema(schema)

This gives the error:


No geometry mapping entry found for '11' in metafile GEODATABASE_FILE.fmf'. 

Since data features don't have schema information which can be used directly to configure schema, it's not easy to create a schema feature based on them.

  • The list from the 'getAllAttributeNames' method contains format attributes unnecessary (unpreferable) for schema configuration. You should exclude them.
  • The value from the 'getAttributeType' is an integer constant, and it cannot be used as a data type for schema configuration. You should use FME generic data type names. e.g. 'fme_int32', 'fme_char(20)', etc. Please see this article again: Destination Schema is Derived from a Lookup Table
  • The value of 'fme_type' should be assigned to 'fme_geometry{0}', rather than 'fme_geometry'.

If you are going to create schema features based on data features, the implementation of the SchemaSetter custom transformer from the FME Hub might be helpful.

takashi
Rank
Userlevel 2
Badge +17

I have a feeling we're getting close, so thanks a lot already. However; with keeping it dynamically I have the following code to get the attributes and their types:


attrs  = []<br>  san = feature.getAllAttributeNames()<br>  print "SAN: ",san<br>  for a in san:<br>  	at = feature.getAttributeType(a)<br>  	attrs.append(a)<br>  	attrs.append(at)
ft = feature.getAttribute('fme_feature_type')
cs = feature.getCoordSys()
ge = feature.getAttribute('fme_geometry')

I use these to set the schema:


# Create a schema feature.<br>  schema = fmeobjects.FMEFeature()<br>  for i in range(0, len(attrs), 2):<br>  # name: attribute name, value: data type name<br>  	schema.setAttribute(attrs[i], attrs[i+1])<br>  schema.setAttribute('fme_geometry{0}', ge)<br>  schema.setFeatureType(ft)<br>  schema.setCoordSys(cs)

This gives me a list of all attributes, which I loop over to get their attribute types. The attribute types are represented by numbers:


ATTRS:  ['SHAPE_Length', 9, 'DTB_ID', 6, 'geodb_oid', 6, 'geodb_type'

The list is longer, but this is to give an idea. When addind the schema to the writer:


# Add the schema feature to the writer.<br>  ObjWriter.addSchema(schema)

This gives the error:


No geometry mapping entry found for '11' in metafile GEODATABASE_FILE.fmf'. 

Addition.

  • You should use 'setSequencedAttribute' method to add a pair of attribute name and data type to the schema feature, rather than 'setAttribute' method.
B
Badge

Since data features don't have schema information which can be used directly to configure schema, it's not easy to create a schema feature based on them.

  • The list from the 'getAllAttributeNames' method contains format attributes unnecessary (unpreferable) for schema configuration. You should exclude them.
  • The value from the 'getAttributeType' is an integer constant, and it cannot be used as a data type for schema configuration. You should use FME generic data type names. e.g. 'fme_int32', 'fme_char(20)', etc. Please see this article again: Destination Schema is Derived from a Lookup Table
  • The value of 'fme_type' should be assigned to 'fme_geometry{0}', rather than 'fme_geometry'.

If you are going to create schema features based on data features, the implementation of the SchemaSetter custom transformer from the FME Hub might be helpful.

Thanks again. The article you mention shows indeed the data type names that FME uses. I know I have to use those names. What I can't find in the documentation is a way to "translate" the numbers into the names. I guess I can make a mapping list based on what I see in the inspector where I see the names and look for those names to find the corresponding numbers. This doesn't seem very efficient though and I am a little surprised that FMEobjects can't do that. Or am I not looking in the propper places?

takashi
Rank
Userlevel 2
Badge +17

Thanks again. The article you mention shows indeed the data type names that FME uses. I know I have to use those names. What I can't find in the documentation is a way to "translate" the numbers into the names. I guess I can make a mapping list based on what I see in the inspector where I see the names and look for those names to find the corresponding numbers. This doesn't seem very efficient though and I am a little surprised that FMEobjects can't do that. Or am I not looking in the propper places?

No, there is no pre-defined function to map a data type code in Python API to an FME generic data type name.

Every feature attribute doesn't have a fixed data type, since they can be changed in the data flow. For example, a numeric value might be changed to a character string through a transformer, and vice versa. The length of a string may also be changed. Either a 'date', 'datetime' or 'time' is treated as just a string. and so on.

Did you see the SchemaSetter custom transformer (FME Hub)? Its implementation contains a Python script which performs the mapping, although it may not be ideal.

Reply


Community Stats

28,709
Posts
109,481
Replies
36,554
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.