Solved

Google Drive token refresh fails through http/https proxy.

  • 23 December 2020
  • 4 replies
  • 7 views

Badge +4

Hi everyone,

 

I developed a simple workspace to read a list of filenames from a Google Drive folder. It works fine when it's run with FME Desktop.

 

When it's run with FME Server it works fine as long as the OAuth2 token is valid, but when the token expires the Google Drive connector seems not to be able to refresh the token.

I point out that FME Server is configured to use a proxy server. I suppose this issue could be related to the proxy, but I don't know how to debug it further.

 

I'm using FME Server 2020.2.1 build 20806 - linux-x64.

 

Any suggestion? Has anyone encountered anything similar?

See the attached excerpt of the logfile.

 

Thanks!

icon

Best answer by hollyatsafe 29 December 2020, 20:41

View original

4 replies

Badge +10

A few recommendations to consider that I found from a similar report:

 

Have you set the proxy for FME Server following the instructions here https://docs.safe.com/fme/html/FME_Server_Documentation/WebUI/Proxy.htm?

 

If you are using NTLM authentication and your security policy allows connections to Google Drive URLs, try adding an exception for Google Drive URLs at either the system level or using Custom Proxy URL option.

 

It sounds like you're running the job an a regular basis, you'll likely have to re-authenticate every 30 or 90 days (depending on the account setting). There are number of articles that describes best practices i.e. create your own custom app for FME Server https://knowledge.safe.com/articles/55445/fme-server-troubleshooting-connections-web-and-dat.html. Instead of re-publishing consider the "Authorize the Web Connection" option in FME Server WebUI 

 

There have been reports where it appears that in some systems/organizations the Redirect URI in the Manage Web Service page in FME Server, may lead to some issues

 

Configure the GoogleDrive Web Service

  • From the left sidebar go to Connections > Web Connections.
  • Click Manage Web Services on the Web Connections page.
  • Select GoogleDrive Editing Web Service "GoogleDrive" page opens.
  • Fill in the Client Information parameters as follows:
    • Client Id: <your client ID>
    • Client Secret: <your_client_secret>
    • Redirect Uri: http://servername/fmeoauth -- In case this is already set, try changing this to IP Address of the FME Server

 

In case the default setting for this is not working, consider creating your own app for GoogleDriveAPI instead of using the default FME version: https://developers.google.com/drive/api/v3/about-auth (recommended for production, FME Server not required), and see if changing redirect URI helps.

 

If you are still experiencing issues, please consider submitting a support case (providing the full logs of both failed and successful runs if possible).

Badge +4

A few recommendations to consider that I found from a similar report:

 

Have you set the proxy for FME Server following the instructions here https://docs.safe.com/fme/html/FME_Server_Documentation/WebUI/Proxy.htm?

 

If you are using NTLM authentication and your security policy allows connections to Google Drive URLs, try adding an exception for Google Drive URLs at either the system level or using Custom Proxy URL option.

 

It sounds like you're running the job an a regular basis, you'll likely have to re-authenticate every 30 or 90 days (depending on the account setting). There are number of articles that describes best practices i.e. create your own custom app for FME Server https://knowledge.safe.com/articles/55445/fme-server-troubleshooting-connections-web-and-dat.html. Instead of re-publishing consider the "Authorize the Web Connection" option in FME Server WebUI 

 

There have been reports where it appears that in some systems/organizations the Redirect URI in the Manage Web Service page in FME Server, may lead to some issues

 

Configure the GoogleDrive Web Service

  • From the left sidebar go to Connections > Web Connections.
  • Click Manage Web Services on the Web Connections page.
  • Select GoogleDrive Editing Web Service "GoogleDrive" page opens.
  • Fill in the Client Information parameters as follows:
    • Client Id: <your client ID>
    • Client Secret: <your_client_secret>
    • Redirect Uri: http://servername/fmeoauth -- In case this is already set, try changing this to IP Address of the FME Server

 

In case the default setting for this is not working, consider creating your own app for GoogleDriveAPI instead of using the default FME version: https://developers.google.com/drive/api/v3/about-auth (recommended for production, FME Server not required), and see if changing redirect URI helps.

 

If you are still experiencing issues, please consider submitting a support case (providing the full logs of both failed and successful runs if possible).

Thanks for your answer @nampreetatsafe​ .

 

To be clear, this issue was already present in FME 2019 and I hoped it would be solved with FME 2020... no luck :-(

 

Here is the situation with respect to your suggestions:

  • the FME Server host use a proxy to exit to Internet via http/https;
  • the proxy is correctly configured in FME Server WebUI;
  • the proxy does not need authentication;
  • I already tried the custom proxy url (as you know, the configuration is quite tricky) with FME Server 2019 and it seeemd not to work (the proxy log did not even show incoming traffic from the FME Server host);
  • I'm already using a custom Google App for GoogleDriveAPI;
  • the redirect uri is urn:ietf:wg:oauth:2.0:oob; it always works in FME Desktop and works in FME Server as long as OAuth2 token is valid;
  • when the workspace ends unsuccessfully the proxy log did not show incoming traffic from the FME Server host.

 

All tests lead to think that the GoogleDriveConnector in FME Server (I'm currently using FME Server 2020.2.1 build 20806 - linux-x64) fails to refresh the token when it has to traverse a proxy (and probably doesn't even contact the proxy at all and, therefore, can't get out on the Internet).

 

I finally point out that the token gets refreshed if I use an HTTPCaller to make the API call, so this confirms that the issue seems to be related to the GoogleDriveConnector.

 

Could it be a bug?

Badge +2

Thanks for your answer @nampreetatsafe​ .

 

To be clear, this issue was already present in FME 2019 and I hoped it would be solved with FME 2020... no luck :-(

 

Here is the situation with respect to your suggestions:

  • the FME Server host use a proxy to exit to Internet via http/https;
  • the proxy is correctly configured in FME Server WebUI;
  • the proxy does not need authentication;
  • I already tried the custom proxy url (as you know, the configuration is quite tricky) with FME Server 2019 and it seeemd not to work (the proxy log did not even show incoming traffic from the FME Server host);
  • I'm already using a custom Google App for GoogleDriveAPI;
  • the redirect uri is urn:ietf:wg:oauth:2.0:oob; it always works in FME Desktop and works in FME Server as long as OAuth2 token is valid;
  • when the workspace ends unsuccessfully the proxy log did not show incoming traffic from the FME Server host.

 

All tests lead to think that the GoogleDriveConnector in FME Server (I'm currently using FME Server 2020.2.1 build 20806 - linux-x64) fails to refresh the token when it has to traverse a proxy (and probably doesn't even contact the proxy at all and, therefore, can't get out on the Internet).

 

I finally point out that the token gets refreshed if I use an HTTPCaller to make the API call, so this confirms that the issue seems to be related to the GoogleDriveConnector.

 

Could it be a bug?

Hi @davtorgh​ ,

 

You mentioned you have set the redirect URL in the Web Service on FME Server to the same one as FME Desktop and I think this is where the issue lies. From the Connections page you should be able to select the Authorize button and it will open a new pop up screen where you can log in to GSuite and allow Safe access to your Google Drive:

Screen Shot 2020-12-29 at 11.19.58 AMOnce complete you'll get this response in the Web UI:

Screen Shot 2020-12-29 at 11.19.58 AMUntil you are able to successfully Authorize the Web Connection from the Connections page on FME Server any jobs run will not be able to refresh the token if it has expired and with the redirect URI you have set currently I do not believe this will work as intended.

 

In FME Server the Redirect URI is different than for FME Desktop, this is how it should be set up:

  1. In Google APIs you'll need to create OAuth Client ID credentials and select Web Application as the Application Type.
  2. Under the Authorized redirect URIs section select add URI and add http(s)://<FMEServerHost>:<port>/fmeoauth
  3. Save the App
  4. In FME Server go to Files & Connections > Web Connections > Manage Web Services > Select Google Drive and update the Client ID and Secret to the ones generated in your app and set the Redirect URI to your FME Server oauth URI set in step 2 e.g. http(s)://<FMEServerHost>:<port>/fmeoauth
  5. Save and return to your Web Connection and try authorizing it now.

After performing these steps you should get the same response as in the images above and FME Server should now have the ability to refresh the token once it expires. If after generating an app for a web application and specifying the correct fmeoauth redirect URI this is still not working please let me know.

Badge +4

Thanks for your answer @nampreetatsafe​ .

 

To be clear, this issue was already present in FME 2019 and I hoped it would be solved with FME 2020... no luck :-(

 

Here is the situation with respect to your suggestions:

  • the FME Server host use a proxy to exit to Internet via http/https;
  • the proxy is correctly configured in FME Server WebUI;
  • the proxy does not need authentication;
  • I already tried the custom proxy url (as you know, the configuration is quite tricky) with FME Server 2019 and it seeemd not to work (the proxy log did not even show incoming traffic from the FME Server host);
  • I'm already using a custom Google App for GoogleDriveAPI;
  • the redirect uri is urn:ietf:wg:oauth:2.0:oob; it always works in FME Desktop and works in FME Server as long as OAuth2 token is valid;
  • when the workspace ends unsuccessfully the proxy log did not show incoming traffic from the FME Server host.

 

All tests lead to think that the GoogleDriveConnector in FME Server (I'm currently using FME Server 2020.2.1 build 20806 - linux-x64) fails to refresh the token when it has to traverse a proxy (and probably doesn't even contact the proxy at all and, therefore, can't get out on the Internet).

 

I finally point out that the token gets refreshed if I use an HTTPCaller to make the API call, so this confirms that the issue seems to be related to the GoogleDriveConnector.

 

Could it be a bug?

Hi @hollyatsafe​ ,

 

the involved FME Server host is not directly reachable from Internet (it is if the connection is initiated by the host itself), so I can't set up the redirect uri as you suggested, at the moment. I'm going to plan a change in network architecture to be able to try out your suggested solution.

 

Thank you

Reply