Skip to main content
Open

Connecting to parquet files in SQLExecutor/SQLCreator (using DuckDB Reader?)

Related products:Integrations
  • April 22, 2024
  • 0 replies
  • 73 views

nissedahlsten
Contributor
Forum|alt.badge.img+9
When I got a humongous pile of parquet-files I had to find a way to read and filter the data in them. I found this and it works great, https://duckdb.org/docs/guides/import/parquet_import.html.Fast SQL-queries on all files without having to move the data into something else first!Using DBeaver and a DuckDB connection ":memory:" I can do queries like this:select count(distinct Id) FROM "D:\myfiles\**\*.parquet"where startingdate >= '2020-07-27 00:00:00' Perhaps this will be a bonus with the upcoming DuckDB-reader/writer...??? :) Migration Note: Idea originally posted on 2024-02-01