Skip to main content
Solved

Converting CSV to GDB feature classes and every field shows up as text

  • November 29, 2017
  • 3 replies
  • 117 views
B
Forum|alt.badge.img

Hi!

 

 

I have a successful workflow going to convert excel files to geodatabase feature classes and I'm trying to create a similar workflow but with CSV input.

 

 

I have hundreds of files that will need to be converted but am testing with 8 for now. I set up everything in the same way as XLSX -> GDB but with my CSV workflow every single field in my GDB shows up as text, despite the fact that some (most) of them contain purely numerical data.

 

 

Is this normal for CSV formats? Is there a way to prevent all these text fields?

 

 

Thanks!

Best answer by andreaatsafe

Hi @badams,

Have you tried setting the Scan for Schema Type to Yes on the CSV Reader? By default this is set to No. This is found when adding the CSV reader here: Advanced > Schema Generation.

 

With this setting set to Yes, FME will attempt to determine the correct type for each attribute. https://www.screencast.com/t/l5w3qRBkHKQ

Alternatively, you can also set the data types in the Attribute Definition setting when Manual mode is selected.

View original
Did this help you find an answer to your question?

3 replies

andreaatsafe
Safer
Forum|alt.badge.img+10
  • andreaatsafeSafer
  • Safer
  • Best Answer
  • November 29, 2017

Hi @badams,

Have you tried setting the Scan for Schema Type to Yes on the CSV Reader? By default this is set to No. This is found when adding the CSV reader here: Advanced > Schema Generation.

 

With this setting set to Yes, FME will attempt to determine the correct type for each attribute. https://www.screencast.com/t/l5w3qRBkHKQ

Alternatively, you can also set the data types in the Attribute Definition setting when Manual mode is selected.

B
Forum|alt.badge.img
  • badams
  • Author
  • November 29, 2017

Thanks @AndreaAtSafe, the numerical fields came out numerical!

 

Something else is going on now - when I look at the attribute tables in Arc, all the numerical fields in some of the feature classes are blank (<null>). In the excel sheet, each cell has a value.

atttab.jpg

 

 

Any idea what might be going on here?

 

 

andreaatsafe
Safer
Forum|alt.badge.img+10
badams wrote:

Thanks @AndreaAtSafe, the numerical fields came out numerical! 

 

Something else is going on now - when I look at the attribute tables in Arc, all the numerical fields in some of the feature classes are blank (<null>). In the excel sheet, each cell has a value. 

atttab.jpg

 

 

Any idea what might be going on here?

 

 

 
Glad it worked @badams!

 

 

With the <null> values: The data type that FME automatically selected for that field is likely not large enough to hold some of the values. If a value is outside of the data type size limit, it will make the value null. This can occur if the value occurred in a row that was not included when scanning for the schema. You would have seen a warning in the log file stating something similar to:

 

CSV reader: The string '283' read in file 'data file' for field 'num_measures' is not a valid value for type 'UInt8'.  The value will be set to <null>. ...
In order to read the value correctly, recreate the source feature type and set a larger value for the 'Maximum Rows to Scan' option to scan additional rows for fields. 

 

This should help solve it!

Reply


Helpful Members This Week

Community Stats

30,942
Posts
117,252
Replies
38,742
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.