Skip to main content
Solved

Splitt Large csv into a more Datasets

  • November 4, 2019
  • 1 reply
  • 117 views
terezia86
Forum|alt.badge.img

Hi,

How can I split a large dataset into multiple CSV datasets?

I have to split them so that each record contains max 5 million datas (rows).

I have about 25 mils of data. So need to have about 5 separated CSVs.

How can I set the writer (or reader) or maybe which transformer ta add to splitt them after writing 5 mils of them?

Thanks a lot!

Best answer by ebygomm

Look at the custom transformer grouper - you can set this to 5 million and use the group count to fan out your csv

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

1 reply

ebygomm
Influencer
Forum|alt.badge.img+44
  • ebygommInfluencer
  • Influencer
  • 3427 replies
  • Best Answer
  • November 4, 2019

Look at the custom transformer grouper - you can set this to 5 million and use the group count to fan out your csv

Community Stats

32,079
Posts
121,810
Replies
39,714
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.