Skip to main content
  • EN

We have a job running as a stream, it is not using any specific stream transformers, but it calls an API endpoint and just runs over and over (~10s) as fast as it can poll the API and write the data.

 

The problem is the engine does not release memory between runs, so it slowly grows and grows. And if left for weeks unchecked the server starts having low memory issues. We can stop and start the stream and it will release the memory back.

 

There is the server config setting MAX_TRANSACTION_RESULT_SUCCESSES which we have set to the default 100, but this does not seem to restart the engine and release the memory. Not sure if a stream just counts as 1 success so it never gets to 100 or what.

 

So, wanted to reach out for any possible solutions for this? Only idea we have is to make and schedule a job that can call the v4 API to stop and start the stream every night. Any better ideas? Maybe some configuration we are missing?

 

Thanks a lot!

 

 

Hi @ryan​ ,

 

Thanks for bringing this up! I think you have the only solution I can see other than having a scheduled service restart on the machines. I've made a request to have a parameter added on the streams page for the engine restart. If anyone else has this issue or a need for the restart please let us know.

That is a great feature request @richardatsafe​ , thanks for capturing that. And thanks for confirming there are not really any better ways than the API call. Thanks again!

Reply


Community Stats

30,095
Posts
114,253
Replies
37,725
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.