Skip to main content
Solved

Copy a single row into an existing ASCII file (quick way?)

  • December 19, 2017
  • 4 replies
  • 53 views
S
Forum|alt.badge.img

Dear all,

I am currently trying to expand an existing ASCII file with a single row. I have an ASCII file with X, Y, Z, R, G, B and an Intensity (Scalar field). My second ASCII file has the same X, Y, Z and Intensity but instead of R, G and B one single row, named Scalar field #2.

I have both files imported as readers in CSV and connected them with an Aggregator. Now I get an ASCII file which has everything I want: X, Y, Z, R, G, B, Intensity and Intensity 2. My problem is, this task takes +40 hours, since I'm dealing with over 40.000.000 data points each row.

 

 

Is there a chance to just copy X, Y, Z, R, G, B and Scalar field from ASCII 1 to a file and extend it with Scalar field #2 from ASCII 2? Do I really have to Aggregate the whole process?

Thank you in advance!

Best answer by sigtill

  1. FeatureMerger - with Suppliers first
  2. Remove the CSV WRITER and replace it with a FeatureWRITER to be able to write the features at once - so you dont need to keep all the data in memory. You should be able to write the output-csv as soon as all the Suppliers are read!

Consider replacing the CSV reader/writer with Pointcloud XYZ-format - it should be waywaywayway faster as it stores - have a look at this for more info: https://knowledge.safe.com/articles/1337/reading-point-clouds.html

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

4 replies

erik_jan
Contributor
Forum|alt.badge.img+23
  • erik_janContributor
  • Contributor
  • December 19, 2017

I believe FeatureMerger is faster than the Aggregator (especially if you read the Supplier first: first in the list of readers).

But if you have an option to read both CSV files as tables in a database (temporarily), you could use indexes to speed up the joining process and then write the join back to CSV.

sigtill
Supporter
Forum|alt.badge.img+25
  • sigtillSupporter
  • Supporter
  • Best Answer
  • December 19, 2017
  1. FeatureMerger - with Suppliers first
  2. Remove the CSV WRITER and replace it with a FeatureWRITER to be able to write the features at once - so you dont need to keep all the data in memory. You should be able to write the output-csv as soon as all the Suppliers are read!

Consider replacing the CSV reader/writer with Pointcloud XYZ-format - it should be waywaywayway faster as it stores - have a look at this for more info: https://knowledge.safe.com/articles/1337/reading-point-clouds.html

The real SigTill
sigtill
Supporter
Forum|alt.badge.img+25
  • sigtillSupporter
  • Supporter
  • December 19, 2017
  1. FeatureMerger - with Suppliers first
  2. Remove the CSV WRITER and replace it with a FeatureWRITER to be able to write the features at once - so you dont need to keep all the data in memory. You should be able to write the output-csv as soon as all the Suppliers are read!

Consider replacing the CSV reader/writer with Pointcloud XYZ-format - it should be waywaywayway faster as it stores - have a look at this for more info: https://knowledge.safe.com/articles/1337/reading-point-clouds.html

Also - have the SMALLEST dataset as the SUPPLIER - because you need all of them in memory (it seems to be the bottom CSV-file with fewer attributes) and remove all unneded attributes.

 

 

The real SigTill
S
Forum|alt.badge.img
  • sebastian
  • Author
  • December 20, 2017

Good morning,

hey you guys are awesome! It didn't even take 24 hours to get an answer to my questions, thank you very much! I will try your suggestions and write the outome into this post.

Have a nice christmas!

Greetings Sebastian

Community Stats

32,154
Posts
122,048
Replies
39,812
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.