Solved

HTTP caller of other possible method to post data online?

F
Badge

I regularly have to send geodata to others outside of the company I work from.

In order to send any type of data to external parties or clients, we have to upload the data to an online portal.

This portal requires me to use a 2 step verification process to log in (first my company username and password, and then with an sms code send to my phone). After logging in, I'm able to upload the data to specific environments (folders) for each external party.

Is it possible to automatically upload the data through FME, either through an HTTP caller, or some other method? I tried using the HTTP caller, and switched around certain settings and parameters, but so far no success.

The portal used is this one, it doesn't seem to be internationally known, but I could be mistaken. The information on their website seems to be in Dutch only, sorry.

https://try.msafe.nl/

 

(I'm using FME DEsktop 2019

icon

Best answer by fmenco 25 August 2020, 13:52

View original

17 replies

redgeographics
Rank
Userlevel 4
Badge +25

You'll need to check with them to see if they have an API that you can use that doesn't have that 2-step verification process. If that's the case one HTTPCaller should be enough. If they have other authentication options that don't require your phone it might still work.

david_r
Rank
Userlevel 4

Multi-factor authentication (MFA) such as SMS or token generators are meant for physical users. For API or automated access it is not a viable solution, as you're seeing. A very common way of securing such services is to disable MFA and rather use specific API keys, which are basically long-lived tokens that are associated with a specific service account.

My recommendation would be to reach out to the portal administrators and ask for API access to their upload service. I would say that most such data portals would (or should...) provide such an API. If that is the case, they probably have some documentation and maybe even some examples on how to interact with it.

From there on, it should be fairly easy to make FME upload the files using the HTTPCaller.

F
Badge

@Hans van der Maarel​ @david_r​  I have decided to go a different route when it comes to uploading this particular data.

However, I'm still interested in the HTTP caller option in combination with APIs for future datauploads etc. Say there's indeed an API and documentation to go with it, what information should I look for in the documentation in case I want to retrieve data from an external party? And how should I register this in FME? The only mandatory field in the HTTP caller transformer are the request url, and the HTTP method, which in the case of the latter should be GET in case I will be retrieving data and POST in case I will be uploading data, I assume.

(Using API in this form, is new for me, so can I get the "for dummies" version...lol).

redgeographics
Rank
Userlevel 4
Badge +25

@Hans van der Maarel​ @david_r​  I have decided to go a different route when it comes to uploading this particular data.

However, I'm still interested in the HTTP caller option in combination with APIs for future datauploads etc. Say there's indeed an API and documentation to go with it, what information should I look for in the documentation in case I want to retrieve data from an external party? And how should I register this in FME? The only mandatory field in the HTTP caller transformer are the request url, and the HTTP method, which in the case of the latter should be GET in case I will be retrieving data and POST in case I will be uploading data, I assume.

(Using API in this form, is new for me, so can I get the "for dummies" version...lol).

My personal preference is to have examples, rather than documentation, but your mileage may vary of course. If I have an example of how to do it I can usually tinker with it and make it do what I want.

Key parts to look out for (in either documentation or examples) is authentication and how to upload files. It makes sense to try authentication first, if you can connect to the API that's half the battle done. As for uploading, the HTTPCaller has the option to upload files in a PUT or POST request. Again, try it out and see what works.

david_r
Rank
Userlevel 4

@Hans van der Maarel​ @david_r​  I have decided to go a different route when it comes to uploading this particular data.

However, I'm still interested in the HTTP caller option in combination with APIs for future datauploads etc. Say there's indeed an API and documentation to go with it, what information should I look for in the documentation in case I want to retrieve data from an external party? And how should I register this in FME? The only mandatory field in the HTTP caller transformer are the request url, and the HTTP method, which in the case of the latter should be GET in case I will be retrieving data and POST in case I will be uploading data, I assume.

(Using API in this form, is new for me, so can I get the "for dummies" version...lol).

The problem is that there are a myriad of ways to implement these things, so it's not possible to give one answer that fits all cases.

The documentation will be very important, yes, but as @Hans van der Maarel​  mentioned, examples can also be really helpful, if they're available and easily transferable (e.g. an example in C# using lots of internal modules may not be evident if you want to use FME).

F
Badge

My personal preference is to have examples, rather than documentation, but your mileage may vary of course. If I have an example of how to do it I can usually tinker with it and make it do what I want.

Key parts to look out for (in either documentation or examples) is authentication and how to upload files. It makes sense to try authentication first, if you can connect to the API that's half the battle done. As for uploading, the HTTPCaller has the option to upload files in a PUT or POST request. Again, try it out and see what works.

@Hans van der Maarel​ @david_r​ 

This is the public documentation which also contains examples: https://docs.withthegrid.com/javascript/introduction

or

https://docs.withthegrid.com/curl/introduction

 

I've received an assertion from them. Logging in is a two step process:

  1. With the assertion I need to request a json webtoken (jwt), https://docs.withthegrid.com/curl/post-authenticationmachinelogin
  2. With every other request I need to send the jwt in the headers , https://docs.withthegrid.com/curl/models-and-routes

 

(edited to add):

I've also received this information, when I specified what info I wanted:

https://docs.withthegrid.com/curl/get-report 

https://docs.withthegrid.com/curl/get-issue.

 

How do I implement this in FME, and is it even possible to do so, considering this 2 step process?

F
Badge

@david_r​  @Hans van der Maarel​ 

 

I've tried implementing it, but I'm stuck. According to their information it's a two step process (see my earlier messaging).

 

When I try to get the webtoken, I get a 204 HTTP status code (see attached image), with both saving the response body to a file or saving it to an attribute. This means I'm able to connect with the provided assertion, however the response is empty.

What am I doing wrong here?

1 Attachment
david_r
Rank
Userlevel 4

@david_r​  @Hans van der Maarel​ 

 

I've tried implementing it, but I'm stuck. According to their information it's a two step process (see my earlier messaging).

 

When I try to get the webtoken, I get a 204 HTTP status code (see attached image), with both saving the response body to a file or saving it to an attribute. This means I'm able to connect with the provided assertion, however the response is empty.

What am I doing wrong here?

HTTP 204 means that the operation was successful and that that there is no data in the response body: https://httpstatuses.com/204 Which seems to match up with your observations. Did you get any info in the headers?

The exact method for authentication should be described in the API documentation of the service provider, what does it say?

F
Badge

HTTP 204 means that the operation was successful and that that there is no data in the response body: https://httpstatuses.com/204 Which seems to match up with your observations. Did you get any info in the headers?

The exact method for authentication should be described in the API documentation of the service provider, what does it say?

@david_r​ 

 

This is the information that I have received from them:

 

Logging in is a two step process:

I have received an sssertion key, which I need to request a JSON webtoken (JWT), hence why I tried to save the response to a jwt file. However, this file is empty. This is the documentation for requesting the jwt (https://docs.withthegrid.com/curl/post-authenticationmachinelogin )

 

After that, I need to request the reports and other data I need, by implementing one of these

https://docs.withthegrid.com/curl/get-report 

https://docs.withthegrid.com/curl/get-issue.

 

However, I need the jwt to implement the last two, and I don't seem to be able to get the jwt from the 1st step.

 

 

david_r
Rank
Userlevel 4

HTTP 204 means that the operation was successful and that that there is no data in the response body: https://httpstatuses.com/204 Which seems to match up with your observations. Did you get any info in the headers?

The exact method for authentication should be described in the API documentation of the service provider, what does it say?

What does the response header say? Note that you'll have to manually specify a list name in the HTTPCaller to get the header back.

F
Badge

HTTP 204 means that the operation was successful and that that there is no data in the response body: https://httpstatuses.com/204 Which seems to match up with your observations. Did you get any info in the headers?

The exact method for authentication should be described in the API documentation of the service provider, what does it say?

@david_r​ 

Everything seems to be empty...,this is from the inspector (see image)

Unless there is another way to see the response header?

 

 

1 Attachment
david_r
Rank
Userlevel 4

HTTP 204 means that the operation was successful and that that there is no data in the response body: https://httpstatuses.com/204 Which seems to match up with your observations. Did you get any info in the headers?

The exact method for authentication should be described in the API documentation of the service provider, what does it say?

Did you specify the header list attribute? I cannot see it.

F
Badge

HTTP 204 means that the operation was successful and that that there is no data in the response body: https://httpstatuses.com/204 Which seems to match up with your observations. Did you get any info in the headers?

The exact method for authentication should be described in the API documentation of the service provider, what does it say?

No, the only thing I did was insert the assertion I received from them, in the url provided in the 1st step...(see image).

 

@david_r​ 

 

 

 

 

1 Attachment
david_r
Rank
Userlevel 4

HTTP 204 means that the operation was successful and that that there is no data in the response body: https://httpstatuses.com/204 Which seems to match up with your observations. Did you get any info in the headers?

The exact method for authentication should be described in the API documentation of the service provider, what does it say?

Why is the http method = OPTIONS? Is that correct?

F
Badge

HTTP 204 means that the operation was successful and that that there is no data in the response body: https://httpstatuses.com/204 Which seems to match up with your observations. Did you get any info in the headers?

The exact method for authentication should be described in the API documentation of the service provider, what does it say?

Because everything else resulted in an error...Per the 1st step it should be POST, and I implemented that but got a 404 HTTP status error.But then again,I left everything empty, because I didn't know how to implement this in FME. Maybe I needed to fill in something else? Like I said before, all of this is completely new to me, sorry....:-)

 

@david_r​ 

david_r
Rank
Userlevel 4

HTTP 204 means that the operation was successful and that that there is no data in the response body: https://httpstatuses.com/204 Which seems to match up with your observations. Did you get any info in the headers?

The exact method for authentication should be described in the API documentation of the service provider, what does it say?

If it says to use the POST method then you need to leave it as such, if you get an error then it's for a reason and you'll need to fix it, not change the method ;-)

Unfortunately I cannot tell you what to change, since this API is unknown to me. I suggest going back to the API documentation.

F
Badge

@david_r​  @Hans van der Maarel​ 

I managed to connect to the API, get a webtoken and get the reports. Thank you for your help.

I had to fill in the assertion in the upload body. That did the trick (see image). Right now I'm manually inserting the JWT in a 2nd HTTP caller for the 2nd step. I'm still changing the workbench around to not have to manually insert the JWT in the header. But at leastapi_test4 so far so good..:-)

Reply


Community Stats

28,662
Posts
109,325
Replies
36,528
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.