These jobs currently take 1-2 hours to run on FME Server (6 CPU, 48 GB) each day. Due to the nature of the data source, incremental updates are not possible at this time.
Question
Hello! My team is using the Socrata Writer to load some large datasets (1-2 million records) daily. I noticed that there isn't a "Features per Transaction" param for this writer. Do you have any suggestions for speeding up writing?
- 8392 replies
It seems that FME has no transaction support for the Socrata writer, see https://docs.safe.com/fme/2020.2/html/FME_Desktop_Documentation/FME_ReadersWriters/safe.socrata.socrata2/quick_facts_safe.socrata.socrata2.htm
I have no experience using Socrata, but based on the way FME uploads the data, I'm not sure transactions would make much sense: "The Socrata Writer works by saving output features to a CSV file, which is then uploaded and transformed using the Socrata Publishing API." See https://docs.safe.com/fme/2020.2/html/FME_Desktop_Documentation/FME_ReadersWriters/safe.socrata.socrata2/safe.socrata.socrata2.htm
It seems that FME has no transaction support for the Socrata writer, see https://docs.safe.com/fme/2020.2/html/FME_Desktop_Documentation/FME_ReadersWriters/safe.socrata.socrata2/quick_facts_safe.socrata.socrata2.htm
I have no experience using Socrata, but based on the way FME uploads the data, I'm not sure transactions would make much sense: "The Socrata Writer works by saving output features to a CSV file, which is then uploaded and transformed using the Socrata Publishing API." See https://docs.safe.com/fme/2020.2/html/FME_Desktop_Documentation/FME_ReadersWriters/safe.socrata.socrata2/safe.socrata.socrata2.htm
Thank you @david_r! The documentation is a good reference. "Features per Transaction" may not be the right terminology. The issue I'm having is that the Socrata Writer outputs the features to a CSV rather quickly, but then uploads only 1,000 rows at a time using the Socrata Publishing API. This parameter does not appear to be configurable.