I have a workspace that is writing to Google BigQuery and is connecting using the service account key file option. For this to work, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be set to the path of the key file. I have uploaded the file to my FME Cloud instance, but need to know how to set the environment variable.
Solved
How can I set the GOOGLE_APPLICATION_CREDENTIALS to the path to my service account key file in FME Cloud?
Best answer by takashi
Hi @bill.dollins , since the GoogleBigQueryConnector is implemented with Python, you can overwrite the environment variable at run-time with a PythonCaller inserted before the GoogleBigQueryConnector.
For example, a PythonCaller with this script will overwrite the environment variable by the full path of the key file which is saved in the folder where the workspace is saved.
import fme
import os
def setGoogleAppCredentialsEnviron(feature):
dir = fme.macroValues['FME_MF_DIR']
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = os.path.join(dir, 'your-key-file.json')When publishing the workspace to the FME Server, upload the key file into the repository simultaneously. Alternatively you can also upload the key file individually into the repository through the FME Server Web Interface later. You can then run the workspace containing the GoogleBigQueryConnecter that uses the key file.
It works, however, I would recommend you to create and use a Web Connection rather than use a key file, due to security reason.
- Best Answer
Hi @bill.dollins , since the GoogleBigQueryConnector is implemented with Python, you can overwrite the environment variable at run-time with a PythonCaller inserted before the GoogleBigQueryConnector.
For example, a PythonCaller with this script will overwrite the environment variable by the full path of the key file which is saved in the folder where the workspace is saved.
import fme
import os
def setGoogleAppCredentialsEnviron(feature):
dir = fme.macroValues['FME_MF_DIR']
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = os.path.join(dir, 'your-key-file.json')When publishing the workspace to the FME Server, upload the key file into the repository simultaneously. Alternatively you can also upload the key file individually into the repository through the FME Server Web Interface later. You can then run the workspace containing the GoogleBigQueryConnecter that uses the key file.
It works, however, I would recommend you to create and use a Web Connection rather than use a key file, due to security reason.
takashi wrote:
Hi @bill.dollins , since the GoogleBigQueryConnector is implemented with Python, you can overwrite the environment variable at run-time with a PythonCaller inserted before the GoogleBigQueryConnector.
For example, a PythonCaller with this script will overwrite the environment variable by the full path of the key file which is saved in the folder where the workspace is saved.
import fme
import os
def setGoogleAppCredentialsEnviron(feature):
dir = fme.macroValues['FME_MF_DIR']
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = os.path.join(dir, 'your-key-file.json')When publishing the workspace to the FME Server, upload the key file into the repository simultaneously. Alternatively you can also upload the key file individually into the repository through the FME Server Web Interface later. You can then run the workspace containing the GoogleBigQueryConnecter that uses the key file.
It works, however, I would recommend you to create and use a Web Connection rather than use a key file, due to security reason.
Oh, are you using the Google BigQuery Writer?
The current BigQuery writer belonging to the Google BigQuery package is also implemented with Python, so you can apply the solution I suggested above.
Hi @Takashi Iijima,
I am using the BigQuery writer. The script you provided worked perfectly. Thank you!
Bill
bill.dollins wrote:
Hi @Takashi Iijima,
I am using the BigQuery writer. The script you provided worked perfectly. Thank you!
Bill
Good to hear the solution works for you.
Since I assumed that it's a case of GoogleBigQueryConnector use at first, I suggested the solution with inserting a PythonCaller before the GoogleBigQueryConnector, but it would be better to set the environment variable in the Startup Python Script if you write many features with the BigQuery Writer, as in:
import fme, os
dir = fme.macroValues['FME_MF_DIR']
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = os.path.join(dir, 'your-key-file.json')
takashi wrote:
Good to hear the solution works for you.
Since I assumed that it's a case of GoogleBigQueryConnector use at first, I suggested the solution with inserting a PythonCaller before the GoogleBigQueryConnector, but it would be better to set the environment variable in the Startup Python Script if you write many features with the BigQuery Writer, as in:
import fme, os
dir = fme.macroValues['FME_MF_DIR']
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = os.path.join(dir, 'your-key-file.json')
Too funny! I had actually already made that change. Thanks for all of your help. My workspace is now running smoothly in FME Cloud.
Reply
Rich Text Editor, editor1
Editor toolbars
Press ALT 0 for help
Helpful Members This Week
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.