Skip to main content
Solved

FeatureReader Cache

arnold_bijlsma
Enthusiast
Forum|alt.badge.img+14

I just found the "Enable Cache" option on the FeatureReader, but it's not entirely clear from the documentation when this cache is renewed.

Assume I have set the "Cache Timeout" parameter to a very long time, say a week or longer.

  1. Will the cache only remain during a session, or will it still exist when I close the workbench, and reopen it later that day?
  2. Will the cache still exist when I close the workbench, close down the machine, and reopen the workbench the next day?

The reason for asking is that some remote datasets are pretty heavy and only change occasionally, so I wish I could minimise the number of times reading it.

Note: I am not talking about workbench feature caching here!

Best answer by bwn

I agree the documentation is not clear, but it is fairly easy to test. Surprisingly, it appears the cache IS permanent between FME sessions, and so this option is smarter than I thought.

In my testing, the first run of the FeatureReader will build the cache in the FME Temp folder.

Every subsequent run WorkBench reads from the cache, even if:

  • You run the Workflow again from scratch a second time
  • You close the project and re-open it and run from scratch
  • You close FME Workbench, re-open the application, re-open the Project, and run form scratch

The documentation hints at this by suggesting that the expiry time on the cache is actually stored as a physical data attribute within the cache files themselves rather than being only stored during run-time.

That's actually quite handy from a development perspective for design/testing purposes, as that means your test Workspace working with a static offline data extract doesn't need to re-read the data on every session, saving test time. I almost wish the plain Readers would have something like this, because there are things the Readers can do (like limiting which attributes to extract from the Source with SQL), that FeatureReader can't.

View original
Did this help you find an answer to your question?

2 replies

bwn
Evangelist
Forum|alt.badge.img+26
  • bwnEvangelist
  • Evangelist
  • Best Answer
  • April 11, 2020

I agree the documentation is not clear, but it is fairly easy to test. Surprisingly, it appears the cache IS permanent between FME sessions, and so this option is smarter than I thought.

In my testing, the first run of the FeatureReader will build the cache in the FME Temp folder.

Every subsequent run WorkBench reads from the cache, even if:

  • You run the Workflow again from scratch a second time
  • You close the project and re-open it and run from scratch
  • You close FME Workbench, re-open the application, re-open the Project, and run form scratch

The documentation hints at this by suggesting that the expiry time on the cache is actually stored as a physical data attribute within the cache files themselves rather than being only stored during run-time.

That's actually quite handy from a development perspective for design/testing purposes, as that means your test Workspace working with a static offline data extract doesn't need to re-read the data on every session, saving test time. I almost wish the plain Readers would have something like this, because there are things the Readers can do (like limiting which attributes to extract from the Source with SQL), that FeatureReader can't.

arnold_bijlsma
Enthusiast
Forum|alt.badge.img+14
bwn wrote:

I agree the documentation is not clear, but it is fairly easy to test. Surprisingly, it appears the cache IS permanent between FME sessions, and so this option is smarter than I thought.

In my testing, the first run of the FeatureReader will build the cache in the FME Temp folder.

Every subsequent run WorkBench reads from the cache, even if:

  • You run the Workflow again from scratch a second time
  • You close the project and re-open it and run from scratch
  • You close FME Workbench, re-open the application, re-open the Project, and run form scratch

The documentation hints at this by suggesting that the expiry time on the cache is actually stored as a physical data attribute within the cache files themselves rather than being only stored during run-time.

That's actually quite handy from a development perspective for design/testing purposes, as that means your test Workspace working with a static offline data extract doesn't need to re-read the data on every session, saving test time. I almost wish the plain Readers would have something like this, because there are things the Readers can do (like limiting which attributes to extract from the Source with SQL), that FeatureReader can't.

Thanks for checking this out. I probably could (should?) have tested it myself as well. At the same time, it's probably worth putting it out here in the open, as I think it's indeed a very underrated feature, which more people should know about!

Reply


Helpful Members This Week

Community Stats

31,018
Posts
117,555
Replies
38,794
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.