Skip to main content
Question

Hello! My team is using the Socrata Writer to load some large datasets (1-2 million records) daily. I noticed that there isn't a "Features per Transaction" param for this writer. Do you have any suggestions for speeding up writing?


These jobs currently take 1-2 hours to run on FME Server (6 CPU, 48 GB) each day. Due to the nature of the data source, incremental updates are not possible at this time.

2 replies

david_r
Evangelist
  • April 21, 2021

It seems that FME has no transaction support for the Socrata writer, see https://docs.safe.com/fme/2020.2/html/FME_Desktop_Documentation/FME_ReadersWriters/safe.socrata.socrata2/quick_facts_safe.socrata.socrata2.htm

I have no experience using Socrata, but based on the way FME uploads the data, I'm not sure transactions would make much sense: "The Socrata Writer works by saving output features to a CSV file, which is then uploaded and transformed using the Socrata Publishing API." See https://docs.safe.com/fme/2020.2/html/FME_Desktop_Documentation/FME_ReadersWriters/safe.socrata.socrata2/safe.socrata.socrata2.htm


  • Author
  • April 27, 2021
david_r wrote:

It seems that FME has no transaction support for the Socrata writer, see https://docs.safe.com/fme/2020.2/html/FME_Desktop_Documentation/FME_ReadersWriters/safe.socrata.socrata2/quick_facts_safe.socrata.socrata2.htm

I have no experience using Socrata, but based on the way FME uploads the data, I'm not sure transactions would make much sense: "The Socrata Writer works by saving output features to a CSV file, which is then uploaded and transformed using the Socrata Publishing API." See https://docs.safe.com/fme/2020.2/html/FME_Desktop_Documentation/FME_ReadersWriters/safe.socrata.socrata2/safe.socrata.socrata2.htm

Thank you @david_r​! The documentation is a good reference. "Features per Transaction" may not be the right terminology. The issue I'm having is that the Socrata Writer outputs the features to a CSV rather quickly, but then uploads only 1,000 rows at a time using the Socrata Publishing API. This parameter does not appear to be configurable.


Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings