Skip to main content
Question

Merging CSV files

  • November 12, 2013
  • 9 replies
  • 224 views
A
Forum|alt.badge.img+1
Hi,

 

I have 6 CSV files that contain just X,Y,Z (no ID fields).

 

 

I want to merge all the files into one large CSV file, but the transformers I have tried: FeatureMerger and Joiner appear to require an ID field to match common IDs.

 

 

Does anyone know of a simple Transformer that just lets me add all the XYZs into one file?

 

 

Thanks
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

9 replies

david_r
Celebrity
  • david_rCelebrity
  • November 12, 2013
Hi,

 

 

a couple of alternatives:

 

  • if the X,Y,Z values are exactly identical in all the CSV files, you can concatenate the values (e.g. StringConcatenator) and use the resulting string as an ID for a FeatureMerger or InlineQuerier.
  • if the X,Y,Z values vary somewhat, send all the features through a 3DPointReplacer to convert them into point geometries and then use a NeighborFinder with a suitably small tolerance to group the features that are within the tolerance value from each other.
David
fmelizard
Safer
Forum|alt.badge.img+21
  • fmelizardSafer
  • Safer
  • November 12, 2013
Hi,

 

 

You can use the UnconditionalFeatureMerger, or just create a dummy ID and use the regular FeatureMerger.

 

 

Also a possibility is to read them as XYZ, use the PointCloudCombiner and write to csv.

 

 

Also the AttributeAccumulator with a dummy ID might work.

 

 

Itay

 

 

A
Forum|alt.badge.img+1
  • aquamarine
  • Author
  • November 12, 2013
Thanks guys.

 

 

How do I create a dummy ID? each csv has 1 million rows, so I don't want to do it manually...

 

 

thanks,
fmelizard
Safer
Forum|alt.badge.img+21
  • fmelizardSafer
  • Safer
  • November 12, 2013
AttributeCreator> attribute name: ID, attribute value :1
A
Forum|alt.badge.img+1
  • aquamarine
  • Author
  • November 12, 2013
Also, when I try merging 2 csvs with just one row each of data using AttributeAccumulator and a Dummy ID field, my output CSV is empty.
david_r
Celebrity
  • david_rCelebrity
  • November 12, 2013
Hi,

 

 

I'm not sure I understand what you really want to achieve, but do you only want to merge all 6 CSV files into one big output file?

 

 

If so, you need no transformer, just route all the readers into the same writer, like so:

 

 

 

 

David
A
Forum|alt.badge.img+1
  • aquamarine
  • Author
  • November 12, 2013
Hi David,

 

 

I tried just routing them all into one output CSV, but the output CSV is empty....
david_r
Celebrity
  • david_rCelebrity
  • November 12, 2013
Have you checked that the attributes on the Writer all have a green triangle next to them? This signifies that the attributes are correctly mapped. The triangles should not be red, like this:

 

 

 

 

David
A
Forum|alt.badge.img+1
  • aquamarine
  • Author
  • November 12, 2013
HI David,

 

Yes thanks for that - I don't think I had exposed the attributes correctly!

 

Thankyou :)

Community Stats

32,207
Posts
122,305
Replies
39,891
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.