I have been playing around a bit with Duckdb lately and its performance is impressive, even with its spatial extension. I understand that 0.8.1 is still a young version and lacks some backward capability but it would be interesting how Duckdb could play with FME.
Duckdb reader/writer
Integrations
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
Thanks for suggesting a DuckDB Reader/Writer in FME.
We'd love to hear how you and others might use DuckDB with FME. Sharing your specific scenarios or challenges could help us understand the benefits and guide our development.
Please reply with your use cases or ideas or send me a message to connect.
Looking forward to your insights!
Looks like Safe is thinking about duckdb integration
“Cloud-native data formats
With growing data sizes, data gravity is becoming critical. That is, you have to put the processing next to the data. This is why we introduced Remote Engines in FME—to let people run data integration workflows as close to the data as possible.
Expect a rise in formats engineered to store huge amounts of data on cloud providers and let you work with the data in the cloud. Taking data down from the cloud to work with it is no longer practical for performance and security reasons.
Over the last year, a handful of cloud-native formats have matured, and at Safe, we’re jumping on this trend by adopting technology around these formats, such as DuckDB support.”
Looks like DuckDb will go out of Beta and into production with version 1.0 later this year
I thought one good idea for using duckdb in FME would be to update the InLineQuerier allowing you to choose sqllite or duckdb as the query engine. The advantage being that duckdb is specifically designed for on the fly processing large amounts of data quickly. If the spatial extension was included you could also carry out spatial queries within the InLineQuerier.
We’re pleased to let you know a DuckDB Reader is now available in 2024.1. This is available as a stand alone reader, or within the SQLCreator and SQLExecutor.
If you are still interested in a DuckDB Writer, please submit a new idea.