Skip to main content
Question

FME_ENGINE_MEMORY_REDLINE on a desktop license ?

  • November 17, 2017
  • 8 replies
  • 193 views
lifalin2016
Contributor
Forum|alt.badge.img+29

Hi,

This issue have been revisited multiple times, and I've tried to browse these. But I can't seem to find out, whether the "FME_ENGINE_MEMORY_REDLINE" fix is only applicable to FME Server, or whether it (or something similar) also works with FME Desktop ?

If it does work with desktop, where and how is it set ?

Our FME license is running a small number of nightly jobs, but if I manually start an ad-hoc job before leaving for the day, having the job run all night, it hogs all resources, and all the nightly jobs fails. It would be beneficial to limit the ad-hoc job to use e.g. just half of the available memory.

Cheers.

-- Cheers, Lars I.

8 replies

david_r
Celebrity
  • david_rCelebrity
  • November 17, 2017

You have to add it to the workspace header, see under Tools / Edit header... in FME Workbench, e.g.

When you execute the workspace you will see that it's mentioned somewhere at the start of the log:

rylanatsafe
Safer
Forum|alt.badge.img+13
  • rylanatsafeSafer
  • Safer
  • November 17, 2017
david_r wrote:

You have to add it to the workspace header, see under Tools / Edit header... in FME Workbench, e.g.

When you execute the workspace you will see that it's mentioned somewhere at the start of the log:

@david_r has hit the nail on the head!

 

 

@lifalin2016 - The FME_ENGINE_MEMORY_REDLINE configuration, for FME Desktop, is discussed in this Knowledge Center article (see bottom).
steveatsafe
Safer
Forum|alt.badge.img+12
  • steveatsafeSafer
  • Safer
  • November 20, 2017

@lifalin2016, What version of FME is involved here?

I'm curious if it is FME 2017 and if you might be interested in testing FME 2018 beta?

I want to add that this parameter is not a 'fix', rather it helps FME influence its use of memory resources. We might say that we would discourage users from changing the default value and instead determine if there are other ways to avoid the memory issue.

From time to time changes are made in the product that may not have the desired affect on the performance and some workflows could be impacted. If this is one of those examples please don't hesitate to share with support at support@safe.com to help investigate.

For the record changes were made in FME 2017 that reduced this settings effectiveness on influencing FME's use of memory resources. Further, in FME 2017 changes were made that pushed the memory resource management to the OS.

In FME 2018 we have returned the memory resource management to FME do to some technical issues that limited us on further improving issues around memory related topics.

SteveatSafe
steveatsafe
Safer
Forum|alt.badge.img+12
  • steveatsafeSafer
  • Safer
  • November 21, 2017
david_r wrote:

You have to add it to the workspace header, see under Tools / Edit header... in FME Workbench, e.g.

When you execute the workspace you will see that it's mentioned somewhere at the start of the log:

Moving forward we would discourage the use of this parameter as a first step in resolving a memory issue with an FME Workspace. I is absolutely a last resort and should not be adjusted until it is recommended by Safe Support.

 

 

We recommend working in the confines of the workspace by making changes to the workflow, Readers/Writers, or Transformers or increasing the memory of the system in question (FME 64 bit). If this doesn't resolve the memory problem then please report the issue to Safe Support (support@safe.com) and we will work on improving FME.

 

 

SteveatSafe
lifalin2016
Contributor
Forum|alt.badge.img+29
  • lifalin2016Contributor
  • Author
  • Contributor
  • November 21, 2017
rylanatsafe wrote:
@david_r has hit the nail on the head!

 

 

@lifalin2016 - The FME_ENGINE_MEMORY_REDLINE configuration, for FME Desktop, is discussed in this Knowledge Center article (see bottom).
Hi Rylan.

 

I did see that article, but it uses the term "FME Engine", which is normally associated with FME Server, not Desktop. And it has no mention of how to apply this in the desktop application as such.

 

 

-- Cheers, Lars I.
lifalin2016
Contributor
Forum|alt.badge.img+29
  • lifalin2016Contributor
  • Author
  • Contributor
  • November 21, 2017
david_r wrote:

You have to add it to the workspace header, see under Tools / Edit header... in FME Workbench, e.g.

When you execute the workspace you will see that it's mentioned somewhere at the start of the log:

Thanks David.

 

Ok, manually edit what appears to be an auto-generated header portion ??

 

This doesn't sound like a sound implementation, but if it works, great.

 

 

However, Steve says below, that this feature will be discouraged (and discontinued?) in future versions ? Will it be replaced by something better I wonder (and sincerely hope so) ?

 

 

Because, as I see it, just adding more RAM may not be an option. E.g., my desktop machine maxes out with 16 Gb (tell me about it), so that path would be exhausted on it. So some sort of mechanism to limit how many resources each FME session may consume (apart from hard drive space) is still necessary in a production setup. An FME Server would of course be better and very nice, but we don't have the volume for that investment just yet.

 

Cheers

 

 

-- Cheers, Lars I.
david_r
Celebrity
  • david_rCelebrity
  • November 21, 2017
lifalin2016 wrote:
Thanks David.

 

Ok, manually edit what appears to be an auto-generated header portion ??

 

This doesn't sound like a sound implementation, but if it works, great.

 

 

However, Steve says below, that this feature will be discouraged (and discontinued?) in future versions ? Will it be replaced by something better I wonder (and sincerely hope so) ?

 

 

Because, as I see it, just adding more RAM may not be an option. E.g., my desktop machine maxes out with 16 Gb (tell me about it), so that path would be exhausted on it. So some sort of mechanism to limit how many resources each FME session may consume (apart from hard drive space) is still necessary in a production setup. An FME Server would of course be better and very nice, but we don't have the volume for that investment just yet.

 

Cheers

 

 

I suspect modifying the header isn't meant to be something you do, unless you're trying to debug something out of the ordinary.

 

I agree with Steve that you should probably look into how your workspace has been authored before playing around with this setting. Tip: Blocking transformers are horrible for memory consumpting when dealing with large datasets. See here for a starting point on how to un-block them: https://knowledge.safe.com/articles/38700/clearing-blocking-transformers.html
rylanatsafe
Safer
Forum|alt.badge.img+13
  • rylanatsafeSafer
  • Safer
  • November 21, 2017
lifalin2016 wrote:
Hi Rylan.

 

I did see that article, but it uses the term "FME Engine", which is normally associated with FME Server, not Desktop. And it has no mention of how to apply this in the desktop application as such.

 

 

Hi @lifalin2016 - I am sorry about the terminology confusion. Both FME Desktop and FME Server are powered by the 'FME Engine'.

 

The header I am referring to for the article I referenced above is "Controlling How FME Engine Consumes Memory" – it's in the very last paragraph: "To set this value in FME Workbench use Tools - Edit Header and enter: FME_ENGINE_MEMORY_REDLINE 0.5 as the very first line." Hope that helps! I'm sorry about the ambiguity. I referenced the article instead of simply pasting in here because the article contains important information concerning the parameter.

 

As @SteveAtSafe has mentioned this parameter is really the "last resort". It would be great if you opened a support ticket so that we can have our developers investigate if there are any improvements we can make to the product.

 

Reply


Helpful Members This Week

Community Stats

31,347
Posts
118,905
Replies
38,964
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.