create Token Based web service (not oauth 2) to connect to ArcGIS Portal from FME

Have you looked through this article?

https://support.safe.com/hc/en-us/articles/25407772142733-How-to-Create-an-ArcGIS-Enterprise-Portal-Web-Connection-OAuth-2-0

@hkingsbury Yes I have but that article is about oauth 2 connection and my question is about token based web service

I use a token-based setup to run processes on this community’s api. Having generated a token, I just use the HTTPCaller with a line in the Headers that is Authorization: Bearer @Value(APIToken).

Obviously it’s not the same as creating and using a web service/web connection, but it’s just as simple to use.

How would you create a token? Try the new EsriPortalTokenGetter transformer that @bruceharold just published.

https://hub.safe.com/publishers/bruceharold/transformers/esriportaltokengetter

Hi,

Can you allude a bit more about the endpoint you are trying to hit (e.g. a FeatureServer Url or a Portal Item) and how? The reason I ask is that the token for Portal vs Server differs and depending on exactly what you are trying to do, you may be able to select you oauth2 webconnection as the authorization without needing to expose the token to the workspace.

There certainly is an answer, but just need to understand your use case.

As Mark alludes to, there are plenty of transformers on the hub to get the token by putting username/password. If there is enough demand, I might combine a couple of process I already have to get the token from the token from portal/server from an existing webconnection (so not to expose passwords in workspace)

Cheers,

Todd

OAuth2 is a process of authentication that then generates a token for FME to use to access secured services.It also provides methods of renewing that token once it expires.

Portal access tokens only last for 120 minutes by default - https://enterprise.arcgis.com/en/portal/latest/administer/windows/specify-the-default-token-expiration-time.htm

If you’re wanting to connect to Portal, Oauth2 is the recommended approach as that will generate fresh tokens when calls are made to the service.

OAuth2 will also be able to utilise SAML and other external identity providers that the Portal instance may be using.

If you must not use OAuth2 you can use the generate token operation, i’m not sure how well this will work as a web connection - https://developers.arcgis.com/rest/services-reference/enterprise/generate-token/

 @todd_davis  Tnx

I am trying to hit a ArcGIS Portal hosted feature service using http caller. in that http caller i need to use web connection object made from token based web service.

Primary reason i wanted to avoid using web connection that authenticate using oauth is because the issue of expiry pf refresh token. not sure who but one of the safer also confirmed that when using oauth the refresh token wont be updated if that oauth based connection is not used for more than 2 weeks. and that issue has cause workspaces to fail here in our end

Esri also recommend to use token based connection for api interaction.

I dont know how well token based web connection will do the job either but wanted to test.

i’d love to see example on how I have to fill this parameters on the screenshot below to create token based web connection for ArcGIS Portal. the example on the knowledge base is for yelp only

 

@todd_davis 
to be clear i have success in using this method (https://developers.arcgis.com/rest/services-reference/enterprise/generate-token/) to generate token in http caller and it works fine.

but i don't want to pass the user name and password with the http request to generate the token. however bake the creds in the token based web service  and create a web connection out of that as web connections are encrypted

Cool...so you know the refresh/access token details and how that works. And yeah, I too don’t like putting the username/password into a transformer either and the Oauth2 webconnection doesn’t hold the credentials, which is better for security. But as you mention the refresh token will expire after two weeks if not renewed (A process using the connection is not run)

When you authenticate via oauth2 in a webconnection, it is creating a refresh token and access token which is stored. But there is no automatic refreshing of these details until a process is run. For some of our clients that have a non-default refresh token like 8 hours, I built a process that performs the exchange of the refresh token in FME with as little calls as possible, but this can also be achieved, by undertaken a simple http call to an authenticated endpoint via FME (there are more calls involved, but it will update your refresh/access token) and avoid the need to reauthenticate.

In reality you have to run a process to ensure that refresh does not die, to get away from providing your username/password to a token service which then authenticates each time.

100 % @todd_davis we actually have workspace that consume resource that does not require authentication from ArcGIS portal and it runs nightly. i see that can be a good candidate to keep the oauth going as you have suggested.

i still see a value in creating a token based web service/connection for the ease of sharing with other users. I look forward to seeing example of how that can be set up
 

You can always create a token based web service in FME and share that. But personally, I always prefer to share an Oauth2 connection that users can’t get to the authentication details.

Not sure if you do this already, but depending on your user-base and how that is setup, if you have a centralised connection location, you can share connections across all users (Public meaning all users with access to that connection database or keep them private to yourself). Also helps ensure that there is more chance that the Oauth2 connections will be used within the default timelines.

Here is a example (reminds me that I need to tidy this up). In this case many of the public items here have been created by our other staff, but I can still use them.

Hello everyone,
 

i have a problem with Oauth 2.0 and the ArcGIS Enterprise Portal Web Connection (OAuth 2.0).
The service and the web connection were created based on the instructions: https://support.safe.com/hc/en-us/articles/25407772142733-How-to-Create-an-ArcGIS-Enterprise-Portal-Web-Connection-OAuth-2-0

Everything is clear to me so far.
After extensive testing, I have noticed that the refresh token is not renewed. 
Based on the information here, which todd_davis writes, I have so far assumed that an ArcGIS Enterprise Portal Web Connection (OAuth 2.0) renews the refresh token when this connection is used in FME.
To do this, I created a workspace that uses the connection and is executed every week. 

When using Oauth 2.0 and the connection, I noticed a few things. I only see the logging of whether the refresh token is used to request a new acess_token in FME form (translation log), but not in FME flow.
I also get the message from our endpoint: http://<yourServerhost>/fmeoauth:
 

The use of that ArcGIS Enterprise Portal Web Connection in FME Form & FME Flow works so far, but updating the refresh_token is not working.
I need to authorize the connection all the time manually in the UI (2 weeks lifetime for refresh_token).

I am not sure why this is not working.

Does anyone had a similar problem in the past?

Kind regards,

Tim Freisen

Hi Tim,

which version of ArcGIS portal , FME flow you are running...

I am on 10.9.1 for portal and 2023.2 for Flow. i have had similar problem until I made sure that one FME flow instance per registered app in ArcGIS Portal. 

if you are using the registered app to support FME oauth connection between multiple users in Form and/or multiple FME flow instance, then you face a problem of refresh token not being updated. also the connection object must be used every couple weeks to make sure refresh token be updated, which you already are aware

In summary, worth to test by only keeping a registered app to establish oauth 2.0 connection object per FME flow instance. (kinda 1:1). I used service account to create the registered app but not sure that matters

Anatoli

Hey Anatoli,

we are using ArcGIS Enterprise 11.3.
FME flow ist at Version FME Flow 2024.0.1.

We are using only one registered App in Portal and we only have one FME instance running.

Do you get the acess_token “refresh” information in the FME flow log? I only get this information in the FME form tranlsation log, which is kinda strange.

Kind regards,

Tim

 

Do you access via a proxy?

​@tfn_wv 
Hi Tim,

I tried to find the refresh token info in FME Flow job log but I see none too…

You said you only have one registered app and if you are using that to establish web connection in your FME flow instance and FME form, then that could cause problem. 

safe explicitly express the above as a common issue for ArcGIS online oauth2.0 but not for portal oauth2.0 web connections. I believe the behavior remains the same regardless of portal or AGOL ( because until I kept the registered app client creds to establish only one connection in flow, i struggled with the issue of refresh tokens not being renewed and jobs failing saying “unauthorized token”) - Safe’s explanation about the agol oauth 2.0 cases is that the refresh token in one instance would invalidate the token in the other if same Oauth app is used to establish web connection object in more that one FME instance ( flow and/or form )

pls see this for more https://support.safe.com/hc/en-us/articles/25407697173645-ArcGIS-Online-Web-Connection-Troubleshooting#h_01HX9M6KESZ4X3GKDFJPRGRMRV

I would create brand new registered app in portal and use the client secret and client ID to establish a brand new web service and web connection object in FME flow and see what happens ( you still need to use it every 2 weeks to keep the it alive) 
 

good luck!

Anatoli

Hello ebygomm,

yes we are using an proxy server.

I thought about the poxy as an indication for the not working refresh too.

Kind regards,

Tim

In the environment I’m working in, to get the arcgis portal connection working properly I had to make sure that the proxy and no_proxy environment variables were set on any machine using the connection. 

On your original question, to set up an Esri ArcGIS Portal Token web connection there’s no requirement to edit anything under manage services. Just add a web connection, choose Esri ArcGIS Portal Token under web service, give it a connection name and enter the token url, username and password

Hello Anatoli,

thanks for your message and the hint to that issue.

I will try that out!

Currently i do the bad hack, to run my workspace every 25 mins to keep the acess_token alive, which is not the best solution in my eyes. :D

Kind regards,

Tim
 

Tnx! ​@ebygomm 

I am on 2023.2 and I dont see a web service that says “Esri ArcGIS Portal Token”  as the one on the screenshot you attached
 

what version of Form you are running?  are you able to share how “Esri ArcGIS Portal Token”  web service is configured from “Manage Services” window?

Anatoli

Sorry, i thought i’d replied to this but must have got distracted.

Running 2024.1, this is how it is set up in manage services