Solved

Hidden Attributes memory usage

S
Badge

I'm still learning the basics in FME. I'm asking myself a lot of questions about the mecanism into FME.

Let's say, you have a 10 attributes in a reader. You use a lot of transfermer sush as AttributeKeeper. So you don't have the same amount of attributes along the process, maybe 5 left.

The other 5 are gone?

Are there still in the process but hidden?

How the memory is manage in FME?

If you have a massive project lasting a long time, could you possibly have a memory leak and crash your computer?

By the way, is AttributeExposer show hidden attributes?

Thanks in advance for all your answear.

icon

Best answer by redgeographics 6 July 2016, 10:37

View original

8 replies

redgeographics
Rank
Userlevel 5
Badge +25

If you use the AttributeRemover (and AttributeKeeper) you really remove the attributes from the features and thus reduce memory usage.

AttributeExposer indeed shows hidden attribute, you can also hide and unhide attributes in the input feature type. However, hiding them there just means you won't be able to access them with a transformer later on, it doesn't actually remove them.

If you have a big transformation with a lot of features and a lot of attributes you can certainly run into memory issues. Removing unneccesary attributes is one way of solving that, the Advanced training course has a good section on how to track and improve performance.

S
Badge

If you use the AttributeRemover (and AttributeKeeper) you really remove the attributes from the features and thus reduce memory usage.

AttributeExposer indeed shows hidden attribute, you can also hide and unhide attributes in the input feature type. However, hiding them there just means you won't be able to access them with a transformer later on, it doesn't actually remove them.

If you have a big transformation with a lot of features and a lot of attributes you can certainly run into memory issues. Removing unneccesary attributes is one way of solving that, the Advanced training course has a good section on how to track and improve performance.

If I understand correctly, you have hidden attributes from uncheck attributes in the reader. And you can show them with the AttributeExposer.

AttributeRemover and AttributeKeeper litteraly remove attributes and there is no way you can access them later in the process.

Is it possible to hide attributes during the process? Not only at the beginning?

takashi
Rank
Userlevel 2
Badge +17

If I understand correctly, you have hidden attributes from uncheck attributes in the reader. And you can show them with the AttributeExposer.

AttributeRemover and AttributeKeeper litteraly remove attributes and there is no way you can access them later in the process.

Is it possible to hide attributes during the process? Not only at the beginning?

Hi @slerendu, there isn't a transformer that is designed specially to hide attributes, but the PythonCaller and TclCaller (FME 2016) have 'Attributes to Hide' parameter. You can use one of them if necessary.

S
Badge

Hi @slerendu, there isn't a transformer that is designed specially to hide attributes, but the PythonCaller and TclCaller (FME 2016) have 'Attributes to Hide' parameter. You can use one of them if necessary.

Thanks. It's clearer now.

M

Just to be clear, there won't be a memory "leak" as such, and you should never get a crash, but consuming more data than you need does use up memory and can slow your translation.

Look in the log window and you'll see a line like:

INFORM|FME Configuration: Start freeing memory when process usage exceeds 2.83 GB of memory or 3.41 GB of address space

Then check the bottom line of the log window for:

INFORM|END - ProcessID: 27568, peak process memory usage: 100516 kB, current process memory usage: 100516 kB

When peak memory usage exceeds the amount specified in the configuration comment, that's when you'll start having a slowdown. Removing excess attributes helps keep the peak usage down. So does keeping unwanted features out of the translation (eg using a Where clause on a database reader instead of a Tester inside the workspace).

Switching to 64-bit FME also helps by increasing the amount of memory available to the point where the peak usage is rarely a problem.

Hope this helps. Check out articles on the knowledge centre tagged with "performance" for more information, or chapter 2 of the advanced training materials.

Regards

Mark

S
Badge

Just to be clear, there won't be a memory "leak" as such, and you should never get a crash, but consuming more data than you need does use up memory and can slow your translation.

Look in the log window and you'll see a line like:

INFORM|FME Configuration: Start freeing memory when process usage exceeds 2.83 GB of memory or 3.41 GB of address space

Then check the bottom line of the log window for:

INFORM|END - ProcessID: 27568, peak process memory usage: 100516 kB, current process memory usage: 100516 kB

When peak memory usage exceeds the amount specified in the configuration comment, that's when you'll start having a slowdown. Removing excess attributes helps keep the peak usage down. So does keeping unwanted features out of the translation (eg using a Where clause on a database reader instead of a Tester inside the workspace).

Switching to 64-bit FME also helps by increasing the amount of memory available to the point where the peak usage is rarely a problem.

Hope this helps. Check out articles on the knowledge centre tagged with "performance" for more information, or chapter 2 of the advanced training materials.

Regards

Mark

Ok I see. Is there an user manual for decrypting the FME log? It could be very useful for a better handling of the projects.

Thanks.

M

Ok I see. Is there an user manual for decrypting the FME log? It could be very useful for a better handling of the projects.

Thanks.

Yes, if you check out the training materials, chapter 2 includes a section on how to deconstruct and interpret an FME log file.

G
Badge +3

If you remove attributes to, like, lighten the load on some process, you can always reaquire the attribute by merging them back from a location before the attributeremover or keeper.

I assume that's likish to unloading and reloading them to memory.

Reply


Community Stats

28,669
Posts
109,360
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.