Question

Differences between different ways of using Stream type of Workspaces.

  • 1 June 2022
  • 1 reply
  • 2 views
N

I have created a workspace having AzureQueueStorageConnector, which should listen to an Azure queue continuously.

I am able to run it as a Job as well and create a stream as well dedicated to this workspace.

 

I have also checked this article about the streams:

https://docs.safe.com/fme/2021.0/html/FME_Server_Documentation/WebUI/Streams.htm

 

I was wondering, what are the differences between these 2 types of running a workspace i.e.

  • Running a stream type of workspace with normal execution flow. (from workspace run button)

 

Running a job from WorkspaceOR 

  • Running it as a dedicated Stream (by creating a Stream dedicated to this workspace).

 

Running as a Stream 

I observed both of the methods listen to the queue continuously and process till the time it is running.

 

Few of the differences I was able to find are:

  1. Streams are capable of restarting themselves, regardless of whether the job was completed successfully, failed, or the server crashed or was shut down. But normal Job executions are not capable of doing this by their own.
  2. Streams need dedicated engines for them to process. Those engines can't be used by any other queues. Which means, 
Total number of engines = Total number of streams (one per workspace) * number of engines we want to keep for high availability or scaling purpose

But in the case of normal job execution, it seems no dedicated engine is required.

 

Are there any other differences between these 2 types of executions?

 

Thanks in advance!

1 reply

markw
Badge

Hi @neha.singh​! These are excellent questions, and your observations are correct. While we don't limit where you can use Stream workspaces, you'll want to use FME Server Streams for the following benefits:

 

  • Starting/stopping a Stream also starts/stops the job. Running a Stream job through Run Workspace would require manually cancelling that job which can be a bit tedious and difficult to track.
  • Assigning Engines in Streams easily gives you control over how many/which Engines are being used for your Stream jobs. When running a job manually, you'd have to use a Job Queue or Queue Control settings to make sure the job runs on a certain Engine.
    • Assigning multiple Engines lets you scale up your stream jobs if the volume is too high to handle with just 1 job.
    • Assigning Engines can also be very useful for FME Server environments that are far away from the stream data source/message broker. Installing and assigning a distributed Engine that's near the stream source will reduce network latency.
  • As you noted, Stream jobs have automated restarts to help them run again especially if it was just a temporary issue.

 

Streams in FME Server is mainly a centralized place to manage your Stream jobs and dedicated Engines. This was a huge step up from the old method of submitting a job with "Run Until Cancelled". Please let me know if you have any other questions!

Reply


Community Stats

28,663
Posts
109,330
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.