Skip to main content
  • EN

We just upgraded from 2018 to 2021 and like the new option of "Streams". Using this for a RabbitMQ job and working great! But my question is around creating a stream for a job without a stream mode transformer.

 

The documentation says the following: "The workspace must contain a transformer that can run in Mode: Stream"

 

This made me think it would not work, but tried it in place of a job we currently use the "Run Until Cancelled" option with and it seems to be working fine. The job is just constantly polling a REST API with an HTTPCaller for updates to real-time data (wish the vendor had an event based option but do not).

 

I like the idea of having all our near real-time integrations running in this separate grouping of streams. And that you can dedicate an engine to it vs. Run Until Cancelled just grabbing any which one and holding on to it.

 

Any concerns with using streams like this or know why the documentation seems to imply it will not work?

Interesting (and also good) to see that this works with an HTTPCaller - is it just inside a loop? or does the workspace just keep getting resubmitted to the engine after successfully running?

I would also have assumed that this workflow wouldn't work.

Personally, I wouldn't have concerns about it if you see that it's working fine.

Not in a loop, it seems to just resubmit the job in the same way "Run Until Cancelled" did. Still seems to be working fine a day later. So feeling pretty good that it is ok to run it this way. Thanks @virtualcitymatt​ !

I asked someone at Safe to check this out. I wasn't sure if the HTTPCaller counts as a stream mode or not.

Hi @ryan! You're correct, the Streams will automatically resubmit jobs if they "finish" whether it's a success or failure much like Run Until Cancelled. This is part of our automatic job resubmissions in Streams to keep them active at all times. I tested this out and didn't see any issues so you should be fine. Fantastic work, this is a great use case for Streams I haven't seen until now!

 

As you've seen, we don't specifically block non-Stream Mode workspaces from being used in Streams but it's more of a guideline. Other users would probably find the constantly polling HTTPCaller quite useful though. We'll update our documentation with this information.

Reply


Community Stats

30,351
Posts
115,341
Replies
38,080
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.