• EN
Question

Issue with FeatureWriter (CSV)

  • 24 April 2020
  • 3 replies
  • 18 views
I

Hi all,

I'm working with FME Desktop 2019.2 and I have encountered a problem with the FeatureWriter.

I need to save features in a CSV file and I want to use FeatureWriter settings to manage escape char and to qualify field values only if needed. Below the settings used.

The problem is that FeatureWriter doesn't qualify a string value like the following:

- Lorem ipsum dolor "sit amet"

While it works correctly with a string like the following:

- "Lorem ipsum" dolor sit amet

The CSV file created by FeatureWriter using the two string above is:

"Test_2","Test_1"

"""Lorem ipsum"" dolor sit amet",Lorem ipsum dolor "sit amet"

 

3 replies

B
Badge +3

That is because Qualify Field Values is set to "If Needed" instead of "Yes".

The implementation of CSV "standards" is loose, but if we take Microsoft Excel as an example, you do not need a qualifier character " for:

  • ...,Lorem ipsum dolor "sit amet"

Because it sees the delimiter character (,) and knows that what follows is unqualified text, and that the middle " characters are not qualifiers and part of the text string.

On the other hand:

  • "Lorem ipsum" dolor sit amet

has " as the leading character in the string which is the same character nominated as the qualifier character, so this must get qualified for most applications to handle this correctly, including MS Excel.

 

Hence, FME doesn't qualify the first string (because qualification isn't needed in most common applications) whereas the second string is. If instead you wish to qualify everything, set "Qualify Field Values" to "Yes".

 

B
Badge +3

That is because Qualify Field Values is set to "If Needed" instead of "Yes".

The implementation of CSV "standards" is loose, but if we take Microsoft Excel as an example, you do not need a qualifier character " for:

  • ...,Lorem ipsum dolor "sit amet"

Because it sees the delimiter character (,) and knows that what follows is unqualified text, and that the middle " characters are not qualifiers and part of the text string.

On the other hand:

  • "Lorem ipsum" dolor sit amet

has " as the leading character in the string which is the same character nominated as the qualifier character, so this must get qualified for most applications to handle this correctly, including MS Excel.

 

Hence, FME doesn't qualify the first string (because qualification isn't needed in most common applications) whereas the second string is. If instead you wish to qualify everything, set "Qualify Field Values" to "Yes".

 

However, I do agree that the FME treatment of these string is not consistent with the CSV specification RFC4180, which specifies that strings with double quotes " or line breaks should be enclosed with double quotes

But again, many applications, including MS Excel, MS Access etc. don't strictly obey this specification. Changing the Parameter "Qualify Field Values" to "Yes" I would expect would then switch the output to RFC4180 specifications, which say that field values may be enclosed with double quotes.

I

Thanks for the explanation!

From the beginning I have discarded the alternative of set "Qualify Field Values" to "Yes" because it involves larger output CSV files. So I think the only remaining alternative is to manage attribute values by string manipulation and set to "No" the option "Qualify Field Values".

Reply


Community Stats

28,777
Posts
109,724
Replies
36,604
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.