Skip to main content
Question

Restart stream to free memory

  • December 2, 2022
  • 4 replies
  • 90 views
ryan
Participant
Forum|alt.badge.img+2

We have a job running as a stream, it is not using any specific stream transformers, but it calls an API endpoint and just runs over and over (~10s) as fast as it can poll the API and write the data.

 

The problem is the engine does not release memory between runs, so it slowly grows and grows. And if left for weeks unchecked the server starts having low memory issues. We can stop and start the stream and it will release the memory back.

 

There is the server config setting MAX_TRANSACTION_RESULT_SUCCESSES which we have set to the default 100, but this does not seem to restart the engine and release the memory. Not sure if a stream just counts as 1 success so it never gets to 100 or what.

 

So, wanted to reach out for any possible solutions for this? Only idea we have is to make and schedule a job that can call the v4 API to stop and start the stream every night. Any better ideas? Maybe some configuration we are missing?

 

Thanks a lot!

 

 

    4 replies

    richardatsafe
    Safer
    Forum|alt.badge.img+10

    Hi @ryan​ ,

     

    Thanks for bringing this up! I think you have the only solution I can see other than having a scheduled service restart on the machines. I've made a request to have a parameter added on the streams page for the engine restart. If anyone else has this issue or a need for the restart please let us know.

      ryan
      Participant
      Forum|alt.badge.img+2
      • ryanParticipant
      • Author
      • Participant
      • December 7, 2022

      That is a great feature request @richardatsafe​ , thanks for capturing that. And thanks for confirming there are not really any better ways than the API call. Thanks again!

      stlj
      Contributor
      Forum|alt.badge.img+5
      • stljContributor
      • Contributor
      • November 21, 2025

      We are facing the same memory issuses in our solution with workspaces running in stream mode. The FMEEngine process that runs the ws uses more and more memory upp to a point where the entire Flow server starts to go really slow and more or less crash. Our tests shows that small and simple workspaces grow the FMEEngine process slower, while workspaces with more datahandling internally, makes it grow faster. Currently, we don´t really know how to solve this. Restarting streams while processing high volume input requires a method to terminate the workspace and stream in a controlled way I suppose. 
      Stefan

      mattmatsafe
      Safer
      Forum|alt.badge.img+14
      • mattmatsafeSafer
      • Safer
      • November 27, 2025

      Hi, below are some steps to help troubleshoot the memory consumption issue:

      1. Confirm you’re using the most recent version of the streaming connector’s package
        Most streaming connectors are FME Hub packages that can be upgraded independently of FME. 

      2. Try running a test workspace as a Stream with only the streaming connector. Does memory consumption still gradually increase over time?

      3. Gradually add additional transformers, readers, and writers in small groups. Test the workspace and see if memory is still stable or if it starts to increase over time. If the issue is not with the streaming connector itself, there is likely another part of the workflow that is causing the memory leak. 

      If you’ve gone through these steps are and seeing memory consumption on the engine grow over time, please submit a support ticket so we can take a closer look! Thanks.

      Side note: the original post from 2 years ago was using a stream to continuously poll an API, instead of connecting to a message queue. If you’re using an HTTPCaller in a streaming workspace, we recommend setting the Maximum Concurrent HTTP  Requests to 1, as discussed here: https://support.safe.com/hc/en-us/articles/25407408057229-FME-Form-Tips-for-Working-with-Continuous-Data-Streams#h_01JWXP6ABHBK89BAQCS766TRGG

      Community Stats

      32,240
      Posts
      122,400
      Replies
      39,941
      Members

      Latest FME

      Terms & ConditionsCookie settingsAccessibility statement
      A product of
      Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.