I have a large amout of data in csv files 4 mil in each. Always gets memory problem when reading the data. The rest of the process is fine. I use a 64 bit fme on a powerful machine and the memory is only on 60%. Any suggestions?
Henrik Rinne
Solved
optimizing memory usage
Hi,
Best answer by david_r
Hi,
View original
difficult to tell without knowing your workspace in detail, but some general tips are:
- Remove as many attributes as possible, as early as possible in your workflow (AttributeKeeper is your friend)
- Beware of large lists that are kept "indefinitely" (AttributeRemover is your friend)
- Don't store geometries in an attribute, e.g. GeometryExtractor
- Beware of blocking transformers, e.g. FeatureMergers, they fill up memory quickly
David
This post is closed to further activity.
It may be a question with a best answer, an implemented idea, or just a post needing no comment.
If you have a follow-up or related question, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be a question with a best answer, an implemented idea, or just a post needing no comment.
If you have a follow-up or related question, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
- Best Answer
Hi,
difficult to tell without knowing your workspace in detail, but some general tips are:
- Remove as many attributes as possible, as early as possible in your workflow (AttributeKeeper is your friend)
- Beware of large lists that are kept "indefinitely" (AttributeRemover is your friend)
- Don't store geometries in an attribute, e.g. GeometryExtractor
- Beware of blocking transformers, e.g. FeatureMergers, they fill up memory quickly
David
Helpful Members This Week
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.