I have a large amout of data in csv files 4 mil in each. Always gets memory problem when reading the data. The rest of the process is fine. I use a 64 bit fme on a powerful machine and the memory is only on 60%. Any suggestions?
Henrik Rinne
Solved
optimizing memory usage
Hi,
Best answer by david_r
Hi,
difficult to tell without knowing your workspace in detail, but some general tips are:
- Remove as many attributes as possible, as early as possible in your workflow (AttributeKeeper is your friend)
- Beware of large lists that are kept "indefinitely" (AttributeRemover is your friend)
- Don't store geometries in an attribute, e.g. GeometryExtractor
- Beware of blocking transformers, e.g. FeatureMergers, they fill up memory quickly
David
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
- Best Answer
Hi,
difficult to tell without knowing your workspace in detail, but some general tips are:
- Remove as many attributes as possible, as early as possible in your workflow (AttributeKeeper is your friend)
- Beware of large lists that are kept "indefinitely" (AttributeRemover is your friend)
- Don't store geometries in an attribute, e.g. GeometryExtractor
- Beware of blocking transformers, e.g. FeatureMergers, they fill up memory quickly
David