Skip to main content
Solved

InlineQuerier: Location of SQLite DB?


bwn
Evangelist
Forum|alt.badge.img+26
  • Evangelist

This may or may not be only answerable by the good people at Safe, but does anyone know where the SQLite database generated by InlineQuerier is created? Is is a temporary physical file, or instead uses SQLite's temporary in-memory (virtual) instance option?

Just thinking ahead to some workspace performance tuning options. My workspace InlineQueriers work fairly fast but then the underlying storage is my local PC's SSD so that performance may not translate when executing where the read/write to the SQLite DB involves a different physical file storage environment.

Best answer by david_r

As far as I can tell, it's a physical file that's created in the temporary directory used by FME, e.g. %TEMP% or %FME_TEMP%. The filename seems rather random:

0684Q00000ArKz8QAF.png

The location is usually mentioned in the FME log, e.g.

FME Configuration: Temporary folder is `c:\temp', set from environment variable `FME_TEMP'

See also: https://knowledge.safe.com/articles/176/fme-temp-environment-variable.html

View original
Did this help you find an answer to your question?

4 replies

david_r
Celebrity
  • Best Answer
  • December 9, 2019

As far as I can tell, it's a physical file that's created in the temporary directory used by FME, e.g. %TEMP% or %FME_TEMP%. The filename seems rather random:

0684Q00000ArKz8QAF.png

The location is usually mentioned in the FME log, e.g.

FME Configuration: Temporary folder is `c:\temp', set from environment variable `FME_TEMP'

See also: https://knowledge.safe.com/articles/176/fme-temp-environment-variable.html


bwn
Evangelist
Forum|alt.badge.img+26
  • Author
  • Evangelist
  • December 9, 2019
david_r wrote:

As far as I can tell, it's a physical file that's created in the temporary directory used by FME, e.g. %TEMP% or %FME_TEMP%. The filename seems rather random:

0684Q00000ArKz8QAF.png

The location is usually mentioned in the FME log, e.g.

FME Configuration: Temporary folder is `c:\temp', set from environment variable `FME_TEMP'

See also: https://knowledge.safe.com/articles/176/fme-temp-environment-variable.html

Thanks @david_r, that is indeed what I suspected.  The Journal file (temporary file that stores uncommitted DB transactions) confirms that is the SQLite DB creation method used by InlineQuerier.

Hmmm, well then next is probably for me to submit an idea for InlineQuerier to expose the in-memory option, which is generally programatically simple to do, all you do when calling something like the sqlite3_open()  function is to pass it a file path string argument =  ":memory:" instead of a more conventional folder/file name path like eg. "%FME_TEMP%tempdb.db".

I've tested that the equivalent SQLite FeatureWriter accepts and writes to a file path of ":memory:" so it seems that the underlying technology to do the equivalent in InlineQuerier is already there in FME?  From experience, these RAM instances are lightning fast for query execution and not relying on having a physical SSD or similar to have high performance.


david_r
Celebrity
  • December 9, 2019
bwn wrote:

Thanks @david_r, that is indeed what I suspected. The Journal file (temporary file that stores uncommitted DB transactions) confirms that is the SQLite DB creation method used by InlineQuerier.

Hmmm, well then next is probably for me to submit an idea for InlineQuerier to expose the in-memory option, which is generally programatically simple to do, all you do when calling something like the sqlite3_open() function is to pass it a file path string argument = ":memory:" instead of a more conventional folder/file name path like eg. "%FME_TEMP%tempdb.db".

I've tested that the equivalent SQLite FeatureWriter accepts and writes to a file path of ":memory:" so it seems that the underlying technology to do the equivalent in InlineQuerier is already there in FME? From experience, these RAM instances are lightning fast for query execution and not relying on having a physical SSD or similar to have high performance.

I think that's a great idea, but I agree that it needs to be an option. Unfortunately there are too many servers out there already struggling with limited memory.

Perhaps also post a link to the idea in this thread, so that it's easy to find for those stopping by this question later.


bwn
Evangelist
Forum|alt.badge.img+26
  • Author
  • Evangelist
  • December 15, 2019
david_r wrote:

I think that's a great idea, but I agree that it needs to be an option. Unfortunately there are too many servers out there already struggling with limited memory.

Perhaps also post a link to the idea in this thread, so that it's easy to find for those stopping by this question later.

Thanks @david_r, to link this to the Idea: posted here:

https://knowledge.safe.com/idea/104456/inlinequerier-expose-virtual-ram-option.html?

I've also discussed that there is a 3rd option if concerned about RAM availability that SQLite has, that you can also use the same base database creation command to let SQLite take care of creating and managing its own temporary database file. In these instances it uses this file to "page" any in-memory data overflow to disk but still executes the bulk of the SQL against an in-memory representation. This would address the environment you've noted where might want to be more protective of RAM.

https://www.sqlite.org/inmemorydb.html

It kind of what the background to the original question was: SQLite already provides a programmatic way to call temporary databases and you can let it manage the whole temporary instance instead of FME having to manually create and delete its own on-disk file within the code for InlineQuerier. The authors of SQLite already provide this functionality, which in most cases probably would perform better in Transformer speed.


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings