Question

socrata writer - can I write data one year at a time?

  • 7 October 2015
  • 1 reply
  • 1 view
J
I'm using the socrata writer in FME 2014 to take data from my Oracle data warehouse and post to Socrata open data set.  It is a current limitation of my workplace that I cannot use FME 2015 as there is no 2015 Server available (and I don't want to design a solution where I have to run this from my desktop).

 

 

I currently have a functional workspace, but I seem to be hitting an error based on the volume of data I'm trying to write to socrata.  If I upload only 1 year worth of data at a time (about 50k rows), everything works fine.  Once I try over 150k, I start running into problems and I have about 1.6 million rows total.

 

 

I was wondering if there's a way for me to use FME to loop my writer and write only one year at a time.  The first year would need to replace the existing dataset, then I would append data for all the remaining years, and publish the dataset with the last pass.  

 

 

I'm very noobish when it comes to FME, and am learning as I go.  Does anyone have a direction to point me in?  Ideal solution would also be based on a formulaic approach to loading the data (e.g., currentyear, currentyear - 1, currentyear -2, ...currentyear-35).

 

 

My apologies if this is very unclear... I've been wrestling with the problem a while so I'm getting caught up in my own head.

 

 

thank you for any ideas,

 

Julia

1 reply

pratap
Rank
Badge +2
Hi,

 

 

I understood that you want to write the data of the current year first and then execute the script again to write the data of previous data and similarly you want to do for until last year (curent year -35 ) reached.

 

 

1.     a. Add published parameter with value as curernt year (i.e. 2015)

 

        b. Place a tester before writer with condition year (add an attribute if not exists) should be equal to                     parameter defined in step a.

 

 

2. Use another script with workspace runner and provide the input with credentials with number of year with help of csv or xls file. This will execute the script is batch mode for 35 times.

 

 

Hope it helps

 

 

Pratap

Reply


Community Stats

28,669
Posts
109,359
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.