Solved

Help with webhooks and FME Cloud

D
Badge +6

I am trying to create a webhook for a sheet in Smartsheet (https://www.smartsheet.com/ ). API documentation on webhooks can be found here: https://smartsheet-platform.github.io/api-docs/?shell#creating-a-webhook

As you can see in the diagram for creating a webhook, the webhook needs to be enabled after creation: this is where I'm having a problem and need some help! The "subscriber" (the callback URL, which for my test case I'm using a topic posted to FME cloud) has to respond with a 200 status and also echo back a "challenge" random number that comes in the response header by including it in the response header "Smartsheet-Hook-Response". Or it can respond with a json that contains this value in "smartsheetHookResponse" attribute.

As you can see above, when I try to enable the webhook, the callback URL is triggering the topic (all good here; two attempts shown above). The highlighted string is what I need to return (step #5 of the image in the API documentation linked above). If Smartsheet doesn't receive this, the webhook is not enabled.

The problem I'm having is twofold:

  1. The topic is responding with a 202 status, so it is failing;
  2. I don't know how to respond back with the correct header or json response

How can I have the topic send the appropriate response back so the webhook can be enabled? Thanks for any help on this! PS: below is the response body I get in FME when I try to enable the webhook via an HTTPcaller).

icon

Best answer by dbaldacchino 6 May 2019, 16:20

View original

12 replies

D
Badge +6

Ok so I've made some progress (I think). I created a subscription to run a simple FME workspace to format the JSON response needed. The reader reads the $(FME_TOPIC_MESSAGE) and writes a new JSON to $(FME_SHAREDRESOURCE_DATA). The Notification Service was checked when posting and the topic was assigned. I cannot figure out why the "Parameter to Get Topic Message" drop down is empty as there are published parameters to pick from...what type do they need to be? Anyway, it doesn't seem to matter as the workspace is processing the data (added a logger to it and see the data in the job log; also new JSON is being written).

The problem I continue having is that the response is a 202 and SmartSheet's API is expecting a 200 in order to enable the webhook. I also do not know whether the new JSON I'm writing is being sent back. How can I inspect the response that FME Cloud is sending back? Is there a way to change the status from 202 to 200?

I do have a new question: I used the topic's "Message as Subscriber Content " URL as the webhook's callback URL, which includes an FME token. However this expires so the webhook becomes invalid unless I keep updating it with a new callback URL that contains a new FME token. I really don't want to be constantly updating the webhook and I don't know enough on this subject to know whether this is the way it has to be or not! I would appreciate if anyone can chime in on this. Can the callback URL (the topic URL) be a constant, or does it have to be refreshed? Mentioning some Safers :) @gerhardatsafe @jlutherthomas. Thanks.

stewartatsafe
Rank
Badge +5

Ok so I've made some progress (I think). I created a subscription to run a simple FME workspace to format the JSON response needed. The reader reads the $(FME_TOPIC_MESSAGE) and writes a new JSON to $(FME_SHAREDRESOURCE_DATA). The Notification Service was checked when posting and the topic was assigned. I cannot figure out why the "Parameter to Get Topic Message" drop down is empty as there are published parameters to pick from...what type do they need to be? Anyway, it doesn't seem to matter as the workspace is processing the data (added a logger to it and see the data in the job log; also new JSON is being written).

The problem I continue having is that the response is a 202 and SmartSheet's API is expecting a 200 in order to enable the webhook. I also do not know whether the new JSON I'm writing is being sent back. How can I inspect the response that FME Cloud is sending back? Is there a way to change the status from 202 to 200?

I do have a new question: I used the topic's "Message as Subscriber Content " URL as the webhook's callback URL, which includes an FME token. However this expires so the webhook becomes invalid unless I keep updating it with a new callback URL that contains a new FME token. I really don't want to be constantly updating the webhook and I don't know enough on this subject to know whether this is the way it has to be or not! I would appreciate if anyone can chime in on this. Can the callback URL (the topic URL) be a constant, or does it have to be refreshed? Mentioning some Safers :) @gerhardatsafe @jlutherthomas. Thanks.

Hi. I spent a bit of time looking into this and I think posting to a topic will always return a 202 currently, there is no getting away from that. As defined in the API. I think unless we make some code changes on our end, it means hitting a topic is a no go.

I started going down the path of using the data streaming service and calling a workspace directly that just reads in the JSON and then writes out the JSON back again using this technique. This should give us a 200. The trouble is, I can't find a way at the moment to get the Smartsheet-Hook-Challenge value out of the header. I don't think FME lets you get at the header info, only the body.

I'll get back to you on the token part as it has all changed in 2019.

D
Badge +6

Hi. I spent a bit of time looking into this and I think posting to a topic will always return a 202 currently, there is no getting away from that. As defined in the API. I think unless we make some code changes on our end, it means hitting a topic is a no go.

I started going down the path of using the data streaming service and calling a workspace directly that just reads in the JSON and then writes out the JSON back again using this technique. This should give us a 200. The trouble is, I can't find a way at the moment to get the Smartsheet-Hook-Challenge value out of the header. I don't think FME lets you get at the header info, only the body.

I'll get back to you on the token part as it has all changed in 2019.

Thanks a lot @stewartharper, very much appreciated even though the news isn't great :( I think getting that 202 response is a show stopper, unless Smartsheet makes a change on their end, which is highly unlikely. I was hoping there would be a way to customize the response, as I'm sure no one out there follows the same standard.

If topics are a dead end for now, I'll take a look at your second option. The webhook sends the "random number" (more like a token) in both the header and as JSON. The latter is what I parsed out in my workspace and re-wrote a response JSON with the attribute "smartsheetHookResponse ". As to the token and the topic, I'm wondering if you can make a topic "public" and thus not need an FME token as part of the URL?

If you have any other ideas on how we can trigger FME to run workspaces when certain events happen that would not require a webhook, I'm all ears :) Thanks again for looking into it.

stewartatsafe
Rank
Badge +5

Thanks a lot @stewartharper, very much appreciated even though the news isn't great :( I think getting that 202 response is a show stopper, unless Smartsheet makes a change on their end, which is highly unlikely. I was hoping there would be a way to customize the response, as I'm sure no one out there follows the same standard.

If topics are a dead end for now, I'll take a look at your second option. The webhook sends the "random number" (more like a token) in both the header and as JSON. The latter is what I parsed out in my workspace and re-wrote a response JSON with the attribute "smartsheetHookResponse ". As to the token and the topic, I'm wondering if you can make a topic "public" and thus not need an FME token as part of the URL?

If you have any other ideas on how we can trigger FME to run workspaces when certain events happen that would not require a webhook, I'm all ears :) Thanks again for looking into it.

I'll give you the workspace that I have so far. It looks good to me so I am not sure what I am doing wrong. I am still getting the error: "Response was missing verification response in both header and body, or body JSON may have been invalid. (ref id: z9jdx0a1q9qz)". I registered this with the data download service and then set the following on upload:

 

 

smartsheets.fmw

D
Badge +6

I'll give you the workspace that I have so far. It looks good to me so I am not sure what I am doing wrong. I am still getting the error: "Response was missing verification response in both header and body, or body JSON may have been invalid. (ref id: z9jdx0a1q9qz)". I registered this with the data download service and then set the following on upload:

 

 

smartsheets.fmw

Thanks @stewartharper. Are you using the streaming service or data service URL as the callback URL in the webhook?

D
Badge +6

I'll give you the workspace that I have so far. It looks good to me so I am not sure what I am doing wrong. I am still getting the error: "Response was missing verification response in both header and body, or body JSON may have been invalid. (ref id: z9jdx0a1q9qz)". I registered this with the data download service and then set the following on upload:

 

 

smartsheets.fmw

I'm attaching my workspace, which I used successfully to take the subscriber content (JSON reader) and write out the new JSON. But I'm at a lost on how I can call this from the webhook without a topic.

Smartsheet_WHenable.fmw

stewartatsafe
Rank
Badge +5

I'll give you the workspace that I have so far. It looks good to me so I am not sure what I am doing wrong. I am still getting the error: "Response was missing verification response in both header and body, or body JSON may have been invalid. (ref id: z9jdx0a1q9qz)". I registered this with the data download service and then set the following on upload:

 

 

smartsheets.fmw

@dbaldacchino You have to use the data streaming service. Register the workspace with the data streaming service and then (assuming you are using 2019) get the shareable URL.

Smartsheet will then call the data streaming service, and you can configure a text reader to receive the contents of the POST into the workspace, you parse the JSON and then rebuild the new JSON (see workspace above) and output it via a Text Line Writer. When the workspace runs, the data streaming service will take the contents of the writer and stream it back to the service that called it. It's basically this article:

https://knowledge.safe.com/articles/1295/send-source-data-to-fme-server-with-http-post.html

I must admit I can't seem to get it going though. If we do get this going for the validation, I can show you how we can repurpose the workspace to handle all events that come in.

G
Badge

Hi @dbaldacchino,

Stewart's example workspace registered as a Job Submitter service should do the trick. You can also have a look at these two examples:

 

https://hub.safe.com/manage/publishers/gerhardatsafe/fme-projects/azurestorageevents

 

https://hub.safe.com/manage/publishers/gerhardatsafe/fme-projects/microsoft-graphsubscription

 

 

Both of these projects include a workspace that handles webhook validation for two different services (Azure Event Grid & Microsoft Graph).

 

 

One thing you could try is to play around with the line termination. Some services might be picky about that so I would definitely try out to set Line Termination to None and Write Last Line Terminator to No. I recall that one of the services I did the webhook validation for rejected the response if there was a last line terminator written.

Hope this helps!

D
Badge +6

Hi @dbaldacchino,

Stewart's example workspace registered as a Job Submitter service should do the trick. You can also have a look at these two examples:

 

https://hub.safe.com/manage/publishers/gerhardatsafe/fme-projects/azurestorageevents

 

https://hub.safe.com/manage/publishers/gerhardatsafe/fme-projects/microsoft-graphsubscription

 

 

Both of these projects include a workspace that handles webhook validation for two different services (Azure Event Grid & Microsoft Graph).

 

 

One thing you could try is to play around with the line termination. Some services might be picky about that so I would definitely try out to set Line Termination to None and Write Last Line Terminator to No. I recall that one of the services I did the webhook validation for rejected the response if there was a last line terminator written.

Hope this helps!

Thanks @gerhardatsafe, will definitely try this. Do you recommend using Text readers/writers vs. JSON for this type of thing? My original workspace was not receiving the data when used with data streaming or download service but it did when used in a subscription. Will post back my findings.

D
Badge +6

Hi all, just wanted to close the loop on this. It appears we won't be using webhooks for now, but thanks to @stewartharper we did get this to work (and thanks @gerhardatsafe as well!). So the issue was that the Smartsheet API requires a 200 response to be returned and thus Topics cannot be used (they return a 202). Using a workspace with Data Streaming to return the expected JSON reponse by the SmartSheet API does the trick. Since the webhook requires re-authentication every 100 requests, the same workspace has to be used at the endpoint and some logic added (ex: Tester) to determine whether to execute some othert logic or whether to re-approve the webhook.

For now we are moving forward with a scheduled task to update data from Trimble ProjectSight (no webhook functionality) to SmartSheet and it seems to be working nicely.

G
Badge

Hi all, just wanted to close the loop on this. It appears we won't be using webhooks for now, but thanks to @stewartharper we did get this to work (and thanks @gerhardatsafe as well!). So the issue was that the Smartsheet API requires a 200 response to be returned and thus Topics cannot be used (they return a 202). Using a workspace with Data Streaming to return the expected JSON reponse by the SmartSheet API does the trick. Since the webhook requires re-authentication every 100 requests, the same workspace has to be used at the endpoint and some logic added (ex: Tester) to determine whether to execute some othert logic or whether to re-approve the webhook.

For now we are moving forward with a scheduled task to update data from Trimble ProjectSight (no webhook functionality) to SmartSheet and it seems to be working nicely.

Thanks for the update!

vhruska
Rank
Badge +3

Hi @dbaldacchino,

Stewart's example workspace registered as a Job Submitter service should do the trick. You can also have a look at these two examples:

 

https://hub.safe.com/manage/publishers/gerhardatsafe/fme-projects/azurestorageevents

 

https://hub.safe.com/manage/publishers/gerhardatsafe/fme-projects/microsoft-graphsubscription

 

 

Both of these projects include a workspace that handles webhook validation for two different services (Azure Event Grid & Microsoft Graph).

 

 

One thing you could try is to play around with the line termination. Some services might be picky about that so I would definitely try out to set Line Termination to None and Write Last Line Terminator to No. I recall that one of the services I did the webhook validation for rejected the response if there was a last line terminator written.

Hope this helps!

Hello,

 

I'm running into the same webhook validation issue with Cheqroom. I tried to follow both links and I am getting 404 errors.

 

Would you mind re-posting one or both of the workspaces?

 

Thank you!

Reply


Community Stats

28,731
Posts
109,530
Replies
36,565
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.