Skip to main content
Archived

Google BigQeury Package: Add support for streaming data into BigQuery

Related products:Integrations
  • April 4, 2020
  • 0 replies
  • 18 views

takashi
Celebrity

It seems that the current Google BigQuery writer firstly writes all the features into a temporary CSV file then loads them into the destination BigQuery table. I thinks it's a good mechanism if you load many features into a BigQuery table at once, but it may not be good if you need to write a single (or a few) records into the table frequently (e.g. write 1 feature per a minute), since BigQuery has limits the maximum rate of incoming requests.

Quotas and limits: https://cloud.google.com/bigquery/quotas


However, there is another option - Streaming data into BigQuery in order to perform frequently writing features into BigQuery table avoiding the limits above.

Streaming data into BigQuery: https://cloud.google.com/bigquery/streaming-data-into-bigquery


It would be wonderful if the Google BigQuery writer could also support Streaming data into BigQuery optionally or you could add a new transformer (e.g. GoogleBigQueryStreamer) to the package.


This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.