In short it means that FME is running out of memory. Some things to look out for when working with huge datasets:
- Avoid blocking transformers at all cost! Many transformers can be configured to be non-blocking but you'll have to enable this manually and make sure that your workspace is configured appropriately. A very common culprit is the FeatureMerger which is blocking by default, read up on the option "Suppliers first" in the documentation to understand how to unblock.
- Make sure Windows has enough swap space. Some windows installations have a default swap file of only 4GB which can be on the short side if you have a lot of stuff going on at the same time.
- If you have several writers, make sure that you order the writers so that the writer with the heaviest load comes first. Alternatively replace the traditional writers with FeatureWriters, it can potentially be a lot more efficient.
- Consider using the 64-bit version of FME if you can
abp-csv-nlpg-join-log.txt
This is the log file showing the progress of the data load (trimmed to not show all of the features read)
abp-csv-nlpg-join-log.txt
This is the log file showing the progress of the data load (trimmed to not show all of the features read)
Based on your log file FME is indeed reading the entire dataset into memory before continuing the translation. However, without knowing your workspace it is difficult to say why.
Just to add to Davids response,.. consider replacing any FeatureMerger transformers with InlineQuerier transformer(s). The InlineQuerier stores the input in a temporary database, from which it is then retrieved - it's much more efficient, and resolved an issue I was experiencing with FME storing data in memory recently.
Mary
You seem to be using 64-bit and have a lot of memory available, which is good.
I do notice that you have the line:
FME Configuration: Reader Keyword is `MULTI_READER'
...which seems to mean you have multiple readers in the workspace. Is this for any reason? In fact, in all I find:
INFORM|Creating reader for format:
INFORM|Creating reader for format: CSV (Comma Separated Value) (deprecated)
INFORM|Creating reader for format: Tabular Data
INFORM|Creating reader for format: CSV (Comma Separated Value) (deprecated)
INFORM|Creating reader for format: Tabular Data
INFORM|Creating reader for format: CSV (Comma Separated Value) (deprecated)
INFORM|Creating reader for format: Tabular Data
INFORM|Creating reader for format: CSV (Comma Separated Value) (deprecated)
INFORM|Creating reader for format: Tabular Data
...which seems to suggest that you have nine readers in your workspace! If they are all reading the same CSV, I'm not surprised there is a memory issue! Make sure you have no readers without feature types and delete them from the Navigator window.
Also, you have this in your log:
Creating reader for format: CSV (Comma Separated Value) (deprecated)
Trying to find a DYNAMIC plugin for reader named `CSV'
So you are using FME2017 but the reader is called CSV, which says to me that the workspace was created in an older build, because that reader is now deprecated. If you add a CSV reader in 2017 it should be called CSV2. For example, I have:
Trying to find a DYNAMIC plugin for reader named `CSV2'
The FME2017 CSV reader is way more efficient, so that might be a way to read the data quicker.
Hope this helps,
Mark
