HTTP caller of other possible method to post data online?

You'll need to check with them to see if they have an API that you can use that doesn't have that 2-step verification process. If that's the case one HTTPCaller should be enough. If they have other authentication options that don't require your phone it might still work.

Multi-factor authentication (MFA) such as SMS or token generators are meant for physical users. For API or automated access it is not a viable solution, as you're seeing. A very common way of securing such services is to disable MFA and rather use specific API keys, which are basically long-lived tokens that are associated with a specific service account.

My recommendation would be to reach out to the portal administrators and ask for API access to their upload service. I would say that most such data portals would (or should...) provide such an API. If that is the case, they probably have some documentation and maybe even some examples on how to interact with it.

From there on, it should be fairly easy to make FME upload the files using the HTTPCaller.

@Hans van der Maarel​ @david_r​  I have decided to go a different route when it comes to uploading this particular data.

However, I'm still interested in the HTTP caller option in combination with APIs for future datauploads etc. Say there's indeed an API and documentation to go with it, what information should I look for in the documentation in case I want to retrieve data from an external party? And how should I register this in FME? The only mandatory field in the HTTP caller transformer are the request url, and the HTTP method, which in the case of the latter should be GET in case I will be retrieving data and POST in case I will be uploading data, I assume.

(Using API in this form, is new for me, so can I get the "for dummies" version...lol).

My personal preference is to have examples, rather than documentation, but your mileage may vary of course. If I have an example of how to do it I can usually tinker with it and make it do what I want.

Key parts to look out for (in either documentation or examples) is authentication and how to upload files. It makes sense to try authentication first, if you can connect to the API that's half the battle done. As for uploading, the HTTPCaller has the option to upload files in a PUT or POST request. Again, try it out and see what works.

The problem is that there are a myriad of ways to implement these things, so it's not possible to give one answer that fits all cases.

The documentation will be very important, yes, but as @Hans van der Maarel​  mentioned, examples can also be really helpful, if they're available and easily transferable (e.g. an example in C# using lots of internal modules may not be evident if you want to use FME).

@Hans van der Maarel​ @david_r​ 

This is the public documentation which also contains examples: https://docs.withthegrid.com/javascript/introduction

or

https://docs.withthegrid.com/curl/introduction

I've received an assertion from them. Logging in is a two step process:

1. With the assertion I need to request a json webtoken (jwt), https://docs.withthegrid.com/curl/post-authenticationmachinelogin
2. With every other request I need to send the jwt in the headers , https://docs.withthegrid.com/curl/models-and-routes

(edited to add):

I've also received this information, when I specified what info I wanted:

https://docs.withthegrid.com/curl/get-report 

https://docs.withthegrid.com/curl/get-issue.

How do I implement this in FME, and is it even possible to do so, considering this 2 step process?

@david_r​  @Hans van der Maarel​ 

I've tried implementing it, but I'm stuck. According to their information it's a two step process (see my earlier messaging).

When I try to get the webtoken, I get a 204 HTTP status code (see attached image), with both saving the response body to a file or saving it to an attribute. This means I'm able to connect with the provided assertion, however the response is empty.

What am I doing wrong here?

HTTP 204 means that the operation was successful and that that there is no data in the response body: https://httpstatuses.com/204 Which seems to match up with your observations. Did you get any info in the headers?

The exact method for authentication should be described in the API documentation of the service provider, what does it say?

@david_r​ 

This is the information that I have received from them:

Logging in is a two step process:

I have received an sssertion key, which I need to request a JSON webtoken (JWT), hence why I tried to save the response to a jwt file. However, this file is empty. This is the documentation for requesting the jwt (https://docs.withthegrid.com/curl/post-authenticationmachinelogin )

After that, I need to request the reports and other data I need, by implementing one of these

https://docs.withthegrid.com/curl/get-report 

https://docs.withthegrid.com/curl/get-issue.

However, I need the jwt to implement the last two, and I don't seem to be able to get the jwt from the 1st step.

What does the response header say? Note that you'll have to manually specify a list name in the HTTPCaller to get the header back.

@david_r​ 

Everything seems to be empty...,this is from the inspector (see image)

Unless there is another way to see the response header?

Did you specify the header list attribute? I cannot see it.

No, the only thing I did was insert the assertion I received from them, in the url provided in the 1st step...(see image).

@david_r​ 

Why is the http method = OPTIONS? Is that correct?

Because everything else resulted in an error...Per the 1st step it should be POST, and I implemented that but got a 404 HTTP status error.But then again,I left everything empty, because I didn't know how to implement this in FME. Maybe I needed to fill in something else? Like I said before, all of this is completely new to me, sorry....:-)

@david_r​ 

If it says to use the POST method then you need to leave it as such, if you get an error then it's for a reason and you'll need to fix it, not change the method ;-)

Unfortunately I cannot tell you what to change, since this API is unknown to me. I suggest going back to the API documentation.