Skip to main content
Question

limit CSV writer size

  • October 13, 2014
  • 2 replies
  • 82 views

bubblebeb
Contributor
Forum|alt.badge.img+6
I have a CSV output file which is currently 54mb. 

 

I need to upload it in a system which has a limit of 5mb. 

 

 

Is there a way of limiting the writer to 5mb limit and then write to another CSV. 

 

 

i.e. I want to get 11 csv's of the same data. Ideally with a counter in the name i.e. upload1, upload2, etc.

 

 

Is there a way of either setting a writter parameter or a transformer to do this? 

 

 

 

2 replies

david_r
Celebrity
  • October 13, 2014
Hi,

 

 

one way to do it:
  • find out approximately how many lines it takes to get 5MB
  • use a ModuloCounter set to this number (or slightly lower to be conservative)
  • use _modulo_count as a basis for a feature fanout

 

David

bubblebeb
Contributor
Forum|alt.badge.img+6
  • Author
  • Contributor
  • October 13, 2014
Many thanks. 

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings