Question

Merging CSV files

  • 12 November 2013
  • 9 replies
  • 38 views
A
Badge +1
Hi,

 

I have 6 CSV files that contain just X,Y,Z (no ID fields).

 

 

I want to merge all the files into one large CSV file, but the transformers I have tried: FeatureMerger and Joiner appear to require an ID field to match common IDs.

 

 

Does anyone know of a simple Transformer that just lets me add all the XYZs into one file?

 

 

Thanks

9 replies

david_r
Rank
Userlevel 4
Hi,

 

 

a couple of alternatives:

 

  • if the X,Y,Z values are exactly identical in all the CSV files, you can concatenate the values (e.g. StringConcatenator) and use the resulting string as an ID for a FeatureMerger or InlineQuerier.
  • if the X,Y,Z values vary somewhat, send all the features through a 3DPointReplacer to convert them into point geometries and then use a NeighborFinder with a suitably small tolerance to group the features that are within the tolerance value from each other.
David
fmelizard
Rank
Userlevel 4
Badge +13
Hi,

 

 

You can use the UnconditionalFeatureMerger, or just create a dummy ID and use the regular FeatureMerger.

 

 

Also a possibility is to read them as XYZ, use the PointCloudCombiner and write to csv.

 

 

Also the AttributeAccumulator with a dummy ID might work.

 

 

Itay

 

 

A
Badge +1
Thanks guys.

 

 

How do I create a dummy ID? each csv has 1 million rows, so I don't want to do it manually...

 

 

thanks,
fmelizard
Rank
Userlevel 4
Badge +13
AttributeCreator> attribute name: ID, attribute value :1
A
Badge +1
Also, when I try merging 2 csvs with just one row each of data using AttributeAccumulator and a Dummy ID field, my output CSV is empty.
david_r
Rank
Userlevel 4
Hi,

 

 

I'm not sure I understand what you really want to achieve, but do you only want to merge all 6 CSV files into one big output file?

 

 

If so, you need no transformer, just route all the readers into the same writer, like so:

 

 

 

 

David
A
Badge +1
Hi David,

 

 

I tried just routing them all into one output CSV, but the output CSV is empty....
david_r
Rank
Userlevel 4
Have you checked that the attributes on the Writer all have a green triangle next to them? This signifies that the attributes are correctly mapped. The triangles should not be red, like this:

 

 

 

 

David
A
Badge +1
HI David,

 

Yes thanks for that - I don't think I had exposed the attributes correctly!

 

Thankyou :)

Reply


Community Stats

28,661
Posts
109,324
Replies
36,528
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.