Idea is to add new InlineQuerier Parameter “Use In-Memory DB”
Previously established that InlineQuerier essentially creates a temporary, on-disk SQLite database file, writes input features into it, performs the queries and deletes the SQLite file afterwards.
The current FME approach has a number of drawbacks however including:
- Overhead to write the input feature tables to disk
- Query performance throttled by disk I/O
- Can consume a not in-significant amount of disk space for the temporary query
Having tested these back-to-back in Python environments (when I was scripting in Python before discovering FME 😉 ) there is often a very large performance difference for large datasets between on-disk and in-memory databases.
Programmatically, I suspect this is likely a fairly easy enhancement, because this would utilise exactly the same API sqlite3_open() command that creates+opens the SQLite database as InlineQuerier would use now.
Instead of the current InlineQuerier SQLite API call that I suspect looks like this
sqlite3_open(FME_Temporary_SQLite_file_name, ...)
...the suggested optional parameter above for in-memory DB would simply change the InlineQuerier API call to this
sqlite3_open(":memory:", ...)Also running out of memory is not usually an issue as the standard PRAGMA settings will cache to disk any of the in-memory DB where necessary to to keep within RAM limits.
