Skip to main content
Solved

Optimizing memory usage - Direct read and write

M

I have searched through almost every question relating to optimizing memory in the Knowledge Centre. With my computer set up, there technically shouldn't be any memory issue.

The workbench is to read 15 non spatial and 2 spatial tables from Oracle and write the same table but into 3 different file formats - ESRI Geodatabase (File Geodb ArcObjects), TAB and shapefile.

The reader is a SQLCreator connecting to the Oracle database with a statement ' SELECT * FROM PRODUCT.$(STATE)_HOUSE.

I have created a Published Parameter $(STATE) so I can run the same workbench 9 times. The parameter is to specify what State Oracle Table I want to read from. There is a total of 9 variables.

My log file reads:

  • ME Configuration: FME_PRODUCT_NAME is 'FME(R) 2015.1.3.1'
  • System Status: 269.57 GB of disk space available in the FME temporary folder (C:\\Users\\MARIAA~1.PSM\\AppData\\Local\\Temp)
  • System Status: 8.00 TB of virtual memory available
  • Operating System: Microsoft Windows 7 64-bit Service Pack 1 (Build 7601)
  • FME Platform: WIN64
  • Locale: en_AU
  • Code Page: 1252 (ANSI - Latin I)
  • FME Configuration: Process limits are 15.94 GB of physical memory and 8.00 TB of address space
  • FME Configuration: Start freeing memory when process usage exceeds 9.57 GB of memory or 8.00 TB of address space
  • FME Configuration: Stop freeing memory when process usage is below 7.17 GB of memory and 6.00 TB of address space

The total number of features read is approximately around 20mil (80% is non spatial).

The total output size is only 6GB.

Questions:

  1. Before I start playing around with the FME_ENGINE_MEMORY_REDLINE, has anyone found a suitable value that prevent the 'ResourceManager: Optimizing Memory Usage. Please wait....' line appearing AND if it did, it did not compromise the workbench?
  2. Is it quicker to have the reader as a Oracle Non Spatial vs. SQLCreator?
  3. My TEMP folder is located in my C Drive, but memory is not be an issue. The folder is empty most of the time until I run a FME workbench.

I am appreciative for any advice that would make my workbench run quicker.

Best answer by fmelizard

Okay, assuming there is no blocking transformer (like an Overlay or Aggregate), then I think the issue here is the 3 outputs at once.

Traditional FME was twitchy about having multiple writers going at the same time. So we "cache" the features/rows headed to all but the first writer until the end.

This has the effect of being a total bear in a case such as this.

Strongly advise NOT using traditional Writers in your workflow but instead use 3 FeatureWriter transformers.

FeatureWriters DO work in parallel. All data will be written as soon as it is received.

It won't matter what type of reader you're using (SQL creator or traditional reader). Just route whatever is coming out to each of the 3 FeatureWriters that you've configured.

Give that a spin and do advise on the outcome. You should see no resource manager popping up in the log at all. Hope this helps @matkins

View original
Did this help you find an answer to your question?

12 replies

M
Forum|alt.badge.img+2
  • mark_f
  • June 6, 2017

@matkins

Did you see article:

https://knowledge.safe.com/articles/44349/writers-and-performance.html

Could be that you're writing to three formats at once?

itay
Supporter
Forum|alt.badge.img+16
  • itaySupporter
  • Supporter
  • June 6, 2017

    Is it quicker to have the reader as a Oracle Non Spatial vs. SQLCreator? > doesnt matter the Oracle reader sends a SQL statement to the database.

    If the reading is the issue, try saving the data into a FFS and read from there.

    Upgrade to a newer version of FME.

    I guess it also depends on what you do with the data, so some more information on that part might help understand your translation.

    david_r
    Evangelist
    • david_rEvangelist
    • June 6, 2017

    In addition to the other tips here, one of the most imporant things to look out for when working on large datasets is blocking transformers. Often, the number one culprit is the FeatureMerger, but there are quite a few of other transformers that are blocking by default (and therefore consume a lot of memory), but can be made non-blocking with some careful tweaking of your workspace.

    Some tips on how to "un-block" transformers here: https://knowledge.safe.com/articles/38694/blocking-transformers-and-the-flow-of-features.html

    M
    • matkins
    • Author
    • June 6, 2017
    itay wrote:

      Is it quicker to have the reader as a Oracle Non Spatial vs. SQLCreator? > doesnt matter the Oracle reader sends a SQL statement to the database.

      If the reading is the issue, try saving the data into a FFS and read from there.

      Upgrade to a newer version of FME.

      I guess it also depends on what you do with the data, so some more information on that part might help understand your translation.

      The data will be used so its accessible for end users to consume in their end products. It is basically a straight conversion from Oracle to the three file formats.

       

       

      M
      • matkins
      • Author
      • June 6, 2017
      david_r wrote:

      In addition to the other tips here, one of the most imporant things to look out for when working on large datasets is blocking transformers. Often, the number one culprit is the FeatureMerger, but there are quite a few of other transformers that are blocking by default (and therefore consume a lot of memory), but can be made non-blocking with some careful tweaking of your workspace.

      Some tips on how to "un-block" transformers here: https://knowledge.safe.com/articles/38694/blocking-transformers-and-the-flow-of-features.html

      Thank you @david_r, but the workflow is pretty much straight forward.

       

      Reader = SQLCreator and connected to a writer (either .gdb, TAB or .shp).

       

      fmelizard
      Contributor
      Forum|alt.badge.img+17
      • fmelizardContributor
      • Contributor
      • Best Answer
      • June 6, 2017

      Okay, assuming there is no blocking transformer (like an Overlay or Aggregate), then I think the issue here is the 3 outputs at once.

      Traditional FME was twitchy about having multiple writers going at the same time. So we "cache" the features/rows headed to all but the first writer until the end.

      This has the effect of being a total bear in a case such as this.

      Strongly advise NOT using traditional Writers in your workflow but instead use 3 FeatureWriter transformers.

      FeatureWriters DO work in parallel. All data will be written as soon as it is received.

      It won't matter what type of reader you're using (SQL creator or traditional reader). Just route whatever is coming out to each of the 3 FeatureWriters that you've configured.

      Give that a spin and do advise on the outcome. You should see no resource manager popping up in the log at all. Hope this helps @matkins

      fmelizard
      Contributor
      Forum|alt.badge.img+17
      • fmelizardContributor
      • Contributor
      • June 6, 2017
      fmelizard wrote:

      Okay, assuming there is no blocking transformer (like an Overlay or Aggregate), then I think the issue here is the 3 outputs at once.

      Traditional FME was twitchy about having multiple writers going at the same time. So we "cache" the features/rows headed to all but the first writer until the end.

      This has the effect of being a total bear in a case such as this.

      Strongly advise NOT using traditional Writers in your workflow but instead use 3 FeatureWriter transformers.

      FeatureWriters DO work in parallel. All data will be written as soon as it is received.

      It won't matter what type of reader you're using (SQL creator or traditional reader). Just route whatever is coming out to each of the 3 FeatureWriters that you've configured.

      Give that a spin and do advise on the outcome. You should see no resource manager popping up in the log at all. Hope this helps @matkins

      Note that we have plans to remove this restriction on the traditional writers at some future point.

       

       

      M
      • matkins
      • Author
      • June 13, 2017
      fmelizard wrote:

      Okay, assuming there is no blocking transformer (like an Overlay or Aggregate), then I think the issue here is the 3 outputs at once.

      Traditional FME was twitchy about having multiple writers going at the same time. So we "cache" the features/rows headed to all but the first writer until the end.

      This has the effect of being a total bear in a case such as this.

      Strongly advise NOT using traditional Writers in your workflow but instead use 3 FeatureWriter transformers.

      FeatureWriters DO work in parallel. All data will be written as soon as it is received.

      It won't matter what type of reader you're using (SQL creator or traditional reader). Just route whatever is coming out to each of the 3 FeatureWriters that you've configured.

      Give that a spin and do advise on the outcome. You should see no resource manager popping up in the log at all. Hope this helps @matkins

      Hi @daleatsafe

       

      Thank you for your advice! I replaced all my writers to three FeatureWriter. It reduced from a 13hr translation to 3hr translation for processing 26 million features in total.

       

      There was no 'Optimizing memory usage' message during the translation.

       

      The two issues I experienced was near the end of the translation where my computer started to lag and also the FeatureWriter did not give me an option to align attributes between the reader and the FeatureWriter, I had to use AttributeRenamer to rename the attributes.

       

      Thank you again.
      M
      • matkins
      • Author
      • June 13, 2017
      itay wrote:

        Is it quicker to have the reader as a Oracle Non Spatial vs. SQLCreator? > doesnt matter the Oracle reader sends a SQL statement to the database.

        If the reading is the issue, try saving the data into a FFS and read from there.

        Upgrade to a newer version of FME.

        I guess it also depends on what you do with the data, so some more information on that part might help understand your translation.

        Hi @itay, thankyou for your advice, yes the new version dramatically reduced the translation time (13hrs to 3hrs) with the use of transformer FeatureWriter instead of the traditional writer.

         

        david_r
        Evangelist
        • david_rEvangelist
        • June 14, 2017
        matkins wrote:
        Hi @daleatsafe

         

        Thank you for your advice! I replaced all my writers to three FeatureWriter. It reduced from a 13hr translation to 3hr translation for processing 26 million features in total.

         

        There was no 'Optimizing memory usage' message during the translation.

         

        The two issues I experienced was near the end of the translation where my computer started to lag and also the FeatureWriter did not give me an option to align attributes between the reader and the FeatureWriter, I had to use AttributeRenamer to rename the attributes.

         

        Thank you again.
        Wow, that's an impressive performance gain, thanks for sharing!
        fmelizard
        Contributor
        Forum|alt.badge.img+17
        • fmelizardContributor
        • Contributor
        • June 14, 2017
        matkins wrote:
        Hi @daleatsafe

         

        Thank you for your advice! I replaced all my writers to three FeatureWriter. It reduced from a 13hr translation to 3hr translation for processing 26 million features in total.

         

        There was no 'Optimizing memory usage' message during the translation.

         

        The two issues I experienced was near the end of the translation where my computer started to lag and also the FeatureWriter did not give me an option to align attributes between the reader and the FeatureWriter, I had to use AttributeRenamer to rename the attributes.

         

        Thank you again.
        Could you send the workspace in to support@safe.com -- it sounds like we have a slight memory leak if the computer was laggy by the end. Would also be interested to see the log file.

         

         

        tim_wood
        Contributor
        Forum|alt.badge.img+8
        • tim_woodContributor
        • Contributor
        • October 1, 2018
        fmelizard wrote:

        Okay, assuming there is no blocking transformer (like an Overlay or Aggregate), then I think the issue here is the 3 outputs at once.

        Traditional FME was twitchy about having multiple writers going at the same time. So we "cache" the features/rows headed to all but the first writer until the end.

        This has the effect of being a total bear in a case such as this.

        Strongly advise NOT using traditional Writers in your workflow but instead use 3 FeatureWriter transformers.

        FeatureWriters DO work in parallel. All data will be written as soon as it is received.

        It won't matter what type of reader you're using (SQL creator or traditional reader). Just route whatever is coming out to each of the 3 FeatureWriters that you've configured.

        Give that a spin and do advise on the outcome. You should see no resource manager popping up in the log at all. Hope this helps @matkins

        Would this explain why I got "WARN |Failed to free sufficient memory to reach the process usage limit" in my log file running one translation processing 69,000 features from 2 Readers into one Writer at the same time as another translation was running processing 563,000 features from a CSV file into 2 SQL Server tables and another CSV file?

         

        64-bit FME 2018.1.0.1 on Windows Server 2012 R2 DataCenter with 4 CPUs and 12 Gb RAM.

         

        Reply


        Helpful Members This Week

        Community Stats

        30,922
        Posts
        117,201
        Replies
        38,727
        Members

        Latest FME

        Terms & ConditionsCookie settings

        Cookie policy

        We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

         
        Cookie settings
        A product of
        Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.