We have not been successful in setting up a web connection for our GCP environment.
Therefore, we use JSON key files.
Has worked just fine via a PythonCaller to read the file and fill in GOOGLE_APPLICATION_CREDENTIALS. If we had a need to write to a different GCP project we would change the project_id in the JSON file.
However, we were contacted by billing recently saying that since we use the project in our JSON file that another area was getting billed instead of our ETL project.
We tried to fully qualify using the project id and then the schema in the writer, but that did not work. It always assumes that you are writing to the project in the JSON file.
Is there a way around this issue?
Google BigQuery Connector and Auth
+6
- Contributor
- 32 replies
Userlevel 3
+13
- Safer
- 607 replies
-
3 May 2024
Hello
When using that method, the JSON Service Account Key File generated from Google contains the project and there's no real actionables or reactions on the Safe side (so this is more of a limitation of using this method).
If you want the ability to choose the project in the GoogleBigQueryConnector, you will need to use an OAuth2.0 web connection. We realize this isn’t ideal as it requires authorizing the connection with a Google account that has access to the resources in Big Query (an admin is recommended). The steps to create the web service and connection are below:
- Upgrade to the latest Google Big Query package
- to get the latest web service template
- Create an app for OAuth in GCP with a client secre
- https://support.google.com/cloud/answer/6158849?hl=en
- In the GCP app, please make sure that the following redirect URIs have been added:
- https://localhost for FME Form (the old urn:ietf:wg:oauth:2.0:oob is no longer supported)
- https://<yourfmeflow>/fmeoauth for FME Flow
- On the web services page in FME Workbench, create a new web service from the 'Google BigQuery (safe.google-bigquery)' web service that comes with the Google Big Query package
- see https://docs.safe.com/fme/html/FME-Form-Documentation/FME-Form/!NamedConnections/options_web_service_connections.htm
- Enter the client ID and secret from your GCP OAuth2.0 app
- Create and authorize a web connection that uses your new web service.
- The Google account you authorize with must have access to the resources you want to use
- Publish the connection to FME Flow (this can be done when publishing a workspace using the connection - both the connection and underlying web service will be published)
- In FME Flow, go to connections > web connections > manage web services
- Click on your GCP OAuth web service and make the sure info is correct. The Auth URL needs to be https://<yourfmeflow>/fmeoauth matching what was entered in the GCP OAuth app exactly
- In FME Flow, go to connections > web connections, click on the web connection and Authorize it
Otherwise, if you’re willing too, you could try creating an FME Idea and linking it here for other users to find or upvote. Sorry I don’t have better news, Kailin.
Reply
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.