I have a workspace that is writing to Google BigQuery and is connecting using the service account key file option. For this to work, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be set to the path of the key file. I have uploaded the file to my FME Cloud instance, but need to know how to set the environment variable.
Best answer by takashi
View original
+17
Hi @bill.dollins , since the GoogleBigQueryConnector is implemented with Python, you can overwrite the environment variable at run-time with a PythonCaller inserted before the GoogleBigQueryConnector.
For example, a PythonCaller with this script will overwrite the environment variable by the full path of the key file which is saved in the folder where the workspace is saved.
import fme
import os
def setGoogleAppCredentialsEnviron(feature):
dir = fme.macroValues['FME_MF_DIR']
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = os.path.join(dir, 'your-key-file.json')When publishing the workspace to the FME Server, upload the key file into the repository simultaneously. Alternatively you can also upload the key file individually into the repository through the FME Server Web Interface later. You can then run the workspace containing the GoogleBigQueryConnecter that uses the key file.
It works, however, I would recommend you to create and use a Web Connection rather than use a key file, due to security reason.
+17
Hi @bill.dollins , since the GoogleBigQueryConnector is implemented with Python, you can overwrite the environment variable at run-time with a PythonCaller inserted before the GoogleBigQueryConnector.
For example, a PythonCaller with this script will overwrite the environment variable by the full path of the key file which is saved in the folder where the workspace is saved.
import fme
import os
def setGoogleAppCredentialsEnviron(feature):
dir = fme.macroValues['FME_MF_DIR']
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = os.path.join(dir, 'your-key-file.json')When publishing the workspace to the FME Server, upload the key file into the repository simultaneously. Alternatively you can also upload the key file individually into the repository through the FME Server Web Interface later. You can then run the workspace containing the GoogleBigQueryConnecter that uses the key file.
It works, however, I would recommend you to create and use a Web Connection rather than use a key file, due to security reason.
Oh, are you using the Google BigQuery Writer?
The current BigQuery writer belonging to the Google BigQuery package is also implemented with Python, so you can apply the solution I suggested above.
Hi @Takashi Iijima,
I am using the BigQuery writer. The script you provided worked perfectly. Thank you!
Bill
+17
Hi @Takashi Iijima,
I am using the BigQuery writer. The script you provided worked perfectly. Thank you!
Bill
Good to hear the solution works for you.
Since I assumed that it's a case of GoogleBigQueryConnector use at first, I suggested the solution with inserting a PythonCaller before the GoogleBigQueryConnector, but it would be better to set the environment variable in the Startup Python Script if you write many features with the BigQuery Writer, as in:
import fme, os
dir = fme.macroValues['FME_MF_DIR']
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = os.path.join(dir, 'your-key-file.json')
Good to hear the solution works for you.
Since I assumed that it's a case of GoogleBigQueryConnector use at first, I suggested the solution with inserting a PythonCaller before the GoogleBigQueryConnector, but it would be better to set the environment variable in the Startup Python Script if you write many features with the BigQuery Writer, as in:
import fme, os
dir = fme.macroValues['FME_MF_DIR']
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = os.path.join(dir, 'your-key-file.json')
Too funny! I had actually already made that change. Thanks for all of your help. My workspace is now running smoothly in FME Cloud.
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.