Skip to main content
Archived

Provide a convinient way to dynamically set writer attribute width according to

Related products:FME Form
  • April 25, 2019
  • 2 replies
  • 98 views
fmelizard
Safer
Forum|alt.badge.img+22

***Note from Migration:***

Original Title was: Provide a convinient way to dynamically set writer attribute width according to incoming data


Sometimes, we have the need to be economical with the output string attribute width, to avoid creating large datasets. Especially when the dataset already has a large number of attributes, and features.

It would be really helpful to be able to set the output string attribute width to accommodate the longest string in each field, but no longer.

Currently, the only way I can think of is the method suggested by david_r here, which involves exploding all attributes, calculating string length, and find the max length of each attribute. This process will be quite slow when dealing with a large dataset.

Any way to make this easier will be appreciated.

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

2 replies

kimo
Contributor
Forum|alt.badge.img+10
  • kimoContributor
  • Contributor
  • March 18, 2020

Since the CSV2 reader scans the file for datatypes it wouldn't be a stretch to also scan for field width for string values.

I have an awk tool that scans a csv in milliseconds, perhaps call that? It isn't imbedded into the schema, just gives a text report.

# scan.awk # scan a text file for field widths # given a separator switch # does not handle imbedded separators in strings # but now does skip header line # 5 June 2004 # Ollivier & Co BEGIN { mf=0;} NR == 1 { # skip header     next; } {     for (i=1; i <= NF; i++) {         if ( length($i) > m[i])         m[i] = length($i);     }     if (NF > mf)         mf = NF; } END    { print "field : max width";   for (i=1; i <= mf; i++)       print i,":",m[i]; }
ziur
Contributor
Forum|alt.badge.img+15
  • ziurContributor
  • Contributor
  • June 16, 2022

... and why not add a field type detector as well. It could be that your're better off with a number field than a char. So you can set integer/decimal/precision/double/... according your data.

Community Stats

32,713
Posts
124,128
Replies
41,198
Members
Terms & ConditionsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.