Skip to main content
Question

Merging CSV files

  • November 12, 2013
  • 9 replies
  • 190 views
A
Forum|alt.badge.img+1
Hi,

 

I have 6 CSV files that contain just X,Y,Z (no ID fields).

 

 

I want to merge all the files into one large CSV file, but the transformers I have tried: FeatureMerger and Joiner appear to require an ID field to match common IDs.

 

 

Does anyone know of a simple Transformer that just lets me add all the XYZs into one file?

 

 

Thanks

9 replies

david_r
Celebrity
  • david_rCelebrity
  • November 12, 2013
Hi,

 

 

a couple of alternatives:

 

  • if the X,Y,Z values are exactly identical in all the CSV files, you can concatenate the values (e.g. StringConcatenator) and use the resulting string as an ID for a FeatureMerger or InlineQuerier.
  • if the X,Y,Z values vary somewhat, send all the features through a 3DPointReplacer to convert them into point geometries and then use a NeighborFinder with a suitably small tolerance to group the features that are within the tolerance value from each other.
David
fmelizard
Safer
Forum|alt.badge.img+18
  • fmelizardSafer
  • Safer
  • November 12, 2013
Hi,

 

 

You can use the UnconditionalFeatureMerger, or just create a dummy ID and use the regular FeatureMerger.

 

 

Also a possibility is to read them as XYZ, use the PointCloudCombiner and write to csv.

 

 

Also the AttributeAccumulator with a dummy ID might work.

 

 

Itay

 

 

A
Forum|alt.badge.img+1
  • aquamarine
  • Author
  • November 12, 2013
Thanks guys.

 

 

How do I create a dummy ID? each csv has 1 million rows, so I don't want to do it manually...

 

 

thanks,
fmelizard
Safer
Forum|alt.badge.img+18
  • fmelizardSafer
  • Safer
  • November 12, 2013
AttributeCreator> attribute name: ID, attribute value :1
A
Forum|alt.badge.img+1
  • aquamarine
  • Author
  • November 12, 2013
Also, when I try merging 2 csvs with just one row each of data using AttributeAccumulator and a Dummy ID field, my output CSV is empty.
david_r
Celebrity
  • david_rCelebrity
  • November 12, 2013
Hi,

 

 

I'm not sure I understand what you really want to achieve, but do you only want to merge all 6 CSV files into one big output file?

 

 

If so, you need no transformer, just route all the readers into the same writer, like so:

 

 

 

 

David
A
Forum|alt.badge.img+1
  • aquamarine
  • Author
  • November 12, 2013
Hi David,

 

 

I tried just routing them all into one output CSV, but the output CSV is empty....
david_r
Celebrity
  • david_rCelebrity
  • November 12, 2013
Have you checked that the attributes on the Writer all have a green triangle next to them? This signifies that the attributes are correctly mapped. The triangles should not be red, like this:

 

 

 

 

David
A
Forum|alt.badge.img+1
  • aquamarine
  • Author
  • November 12, 2013
HI David,

 

Yes thanks for that - I don't think I had exposed the attributes correctly!

 

Thankyou :)

Reply


Helpful Members This Week

Community Stats

31,235
Posts
118,439
Replies
38,833
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.