Skip to main content
Solved

Copy a single row into an existing ASCII file (quick way?)

  • December 19, 2017
  • 4 replies
  • 27 views
S
Forum|alt.badge.img

Dear all,

I am currently trying to expand an existing ASCII file with a single row. I have an ASCII file with X, Y, Z, R, G, B and an Intensity (Scalar field). My second ASCII file has the same X, Y, Z and Intensity but instead of R, G and B one single row, named Scalar field #2.

I have both files imported as readers in CSV and connected them with an Aggregator. Now I get an ASCII file which has everything I want: X, Y, Z, R, G, B, Intensity and Intensity 2. My problem is, this task takes +40 hours, since I'm dealing with over 40.000.000 data points each row.

 

 

Is there a chance to just copy X, Y, Z, R, G, B and Scalar field from ASCII 1 to a file and extend it with Scalar field #2 from ASCII 2? Do I really have to Aggregate the whole process?

Thank you in advance!

Best answer by sigtill

  1. FeatureMerger - with Suppliers first
  2. Remove the CSV WRITER and replace it with a FeatureWRITER to be able to write the features at once - so you dont need to keep all the data in memory. You should be able to write the output-csv as soon as all the Suppliers are read!

Consider replacing the CSV reader/writer with Pointcloud XYZ-format - it should be waywaywayway faster as it stores - have a look at this for more info: https://knowledge.safe.com/articles/1337/reading-point-clouds.html

View original
Did this help you find an answer to your question?

4 replies

erik_jan
Contributor
Forum|alt.badge.img+18
  • erik_janContributor
  • Contributor
  • December 19, 2017

I believe FeatureMerger is faster than the Aggregator (especially if you read the Supplier first: first in the list of readers).

But if you have an option to read both CSV files as tables in a database (temporarily), you could use indexes to speed up the joining process and then write the join back to CSV.

sigtill
Supporter
Forum|alt.badge.img+24
  • sigtillSupporter
  • Supporter
  • Best Answer
  • December 19, 2017
  1. FeatureMerger - with Suppliers first
  2. Remove the CSV WRITER and replace it with a FeatureWRITER to be able to write the features at once - so you dont need to keep all the data in memory. You should be able to write the output-csv as soon as all the Suppliers are read!

Consider replacing the CSV reader/writer with Pointcloud XYZ-format - it should be waywaywayway faster as it stores - have a look at this for more info: https://knowledge.safe.com/articles/1337/reading-point-clouds.html

The real SigTill
sigtill
Supporter
Forum|alt.badge.img+24
  • sigtillSupporter
  • Supporter
  • December 19, 2017
sigtill wrote:
  1. FeatureMerger - with Suppliers first
  2. Remove the CSV WRITER and replace it with a FeatureWRITER to be able to write the features at once - so you dont need to keep all the data in memory. You should be able to write the output-csv as soon as all the Suppliers are read!

Consider replacing the CSV reader/writer with Pointcloud XYZ-format - it should be waywaywayway faster as it stores - have a look at this for more info: https://knowledge.safe.com/articles/1337/reading-point-clouds.html

Also - have the SMALLEST dataset as the SUPPLIER - because you need all of them in memory (it seems to be the bottom CSV-file with fewer attributes) and remove all unneded attributes.

 

 

The real SigTill
S
Forum|alt.badge.img
  • sebastian
  • Author
  • December 20, 2017

Good morning,

hey you guys are awesome! It didn't even take 24 hours to get an answer to my questions, thank you very much! I will try your suggestions and write the outome into this post.

Have a nice christmas!

Greetings Sebastian

Reply


Helpful Members This Week

Community Stats

31,341
Posts
118,860
Replies
38,931
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.