Skip to main content
Solved

Unable to Connect to BigQuery

rzel24
Contributor
Forum|alt.badge.img+7

Hello,

I was wondering if someone here could give some guidance to a problem we are experiencing.

We our trying to connect FME to BigQuery but regardless if it is the reader or writer we are unable to connect using Web Connection.

For the Web Connection settings we use or own accounts which have access to BigQuery. In fact when we connect directly to BigQuery we are able to see the project, datasets and tables and we are able to run queries on them.

But when trying to establish the connection from FME using Web Connection we get the following error:
 

BigQuery Connection Error

Any help on how to solve the issue or to even find out what exactly is wrong would be appreciated.

Thanks in advance.

RZ.-

Best answer by rzel24

Unfortunately the Web Connection does not seems to work in our case, so we had to set up a Service Account with the access needed.

The following were the permissions given to the account:
bigquery.jobs.create

bigquery.jobs.list

bigquery.jobs.listAll

bigquery.tables.create

bigquery.tables.update

bigquery.tables.updateData

Additionally we had to add the BQ Data Viewer role to the account.

 

We had to set up a new system environment variable (in both Form and Flow):

GOOGLE_APPLICATION_CREDENTIALS with value “Path_to_SA_Key_File”

 

After that we are now able to connect to BigQuery from FME Form and FME Flow.

Regards,

RZ.-

 

 

View original
Did this help you find an answer to your question?

2 replies

emmaatsafe
Safer
Forum|alt.badge.img+8
  • emmaatsafeSafer
  • Safer
  • October 16, 2024

Hi @rzel24,

Thank you for your question! We also have received your support ticket. Here’s a summary of our recommendations:

  • Ensure the Google BigQuery FME package is updated to the latest released version (1.1.5). You can check the package version installed in Workbench using Tools > FME Options > Packages. Delete any existing Big Query connections and recreate it. This ensures the web connection is using the latest web service definition.
  • If the error still persists after these, we would recommend to create and use their own OAuth2.0 app for Big Query, as the default Big Query Web Service definition that is provided by Safe Software is not recommended for production use. The preferred option would be to create your own Google App (client ID/secret) to use within FME.

Emma

rzel24
Contributor
Forum|alt.badge.img+7
  • rzel24Contributor
  • Author
  • Contributor
  • Best Answer
  • October 22, 2024

Unfortunately the Web Connection does not seems to work in our case, so we had to set up a Service Account with the access needed.

The following were the permissions given to the account:
bigquery.jobs.create

bigquery.jobs.list

bigquery.jobs.listAll

bigquery.tables.create

bigquery.tables.update

bigquery.tables.updateData

Additionally we had to add the BQ Data Viewer role to the account.

 

We had to set up a new system environment variable (in both Form and Flow):

GOOGLE_APPLICATION_CREDENTIALS with value “Path_to_SA_Key_File”

 

After that we are now able to connect to BigQuery from FME Form and FME Flow.

Regards,

RZ.-

 

 

emmaatsafe
1 person upvoted this
  • emmaatsafe
    emmaatsafe

Reply


Helpful Members This Week

Community Stats

30,942
Posts
117,256
Replies
38,742
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.