From Query Engine to Drag-and-Drop

PrivStack has always stored data encrypted in the entity system — great for privacy, less great for running SELECT * FROM expenses WHERE amount > 500 GROUP BY category. These last two pushes change that. There's now a full DuckDB-backed dataset engine in the Rust core, a cross-plugin capability system in the SDK, and a drag-and-drop data sources panel in the Notes plugin that lets you drop a live dataset table onto any page.

A New Rust Crate: privstack-datasets

The centerpiece is privstack-datasets, an entirely new crate in the Rust workspace. It provides a separate DuckDB database (data.datasets.duckdb) that stores raw columnar data without encryption, enabling full SQL — WHERE, GROUP BY, JOIN, aggregations, window functions — with DuckDB's native read_csv_auto() for imports.

Each imported CSV becomes a native DuckDB table named ds_. The crate is split into the modules you'd expect:

• Schema initialization with WAL recovery (mirrors the pattern from privstack-storage)
• CRUD for dataset lifecycle — import, duplicate, rename, delete
• Query engine with pagination, column introspection, and a DESCRIBE-based column type resolver (working around a DuckDB 1.4.4 panic where column_count() crashes on un-executed statements)
• Mutations — INSERT, UPDATE, DELETE, CREATE TABLE, ALTER TABLE with dry-run support
• Views — saved filter/sort/group configurations per dataset
• Saved queries — persisted SQL with an is_view flag for queries that act as virtual tables
• Relations — cross-dataset foreign key links
• Row page links — a row in a dataset can deep-link into a Notes page
• SQL preprocessor — resolves source:"Dataset Name" tokens into actual ds_ table names so cross-dataset queries feel natural

The preprocessor is worth calling out. Instead of forcing users to type SELECT * FROM ds_a1b2c3d4, they write SELECT e.name, SUM(t.hours) FROM source:"Employees" e JOIN source:"Time Entries" t ON e.employee_id = t.employee_id. The Rust preprocessor does a regex pass, resolves each source: reference against the dataset catalog, and replaces it with the real table name before DuckDB sees the query.

FFI: Bridging Rust and .NET

Every store operation gets a corresponding FFI export — CRUD, queries, mutations, relations, views, saved queries — each in its own file under privstack-ffi/src/datasets/. The usual FFI ceremony applies: #[repr(C)] structs, CString conversions, // SAFETY: comments on every unsafe block. On the C# side, DatasetNativeLibrary provides the P/Invoke bindings, and DatasetService (split across three partials: main, mutations, SQL v2) wraps them into the IDatasetService interface that plugins consume through the capability broker.

The native binary also got leaner — the .dylib dropped from ~46MB to ~34MB from dependency tree cleanup.

SDK: Three New Capability Interfaces

The SDK gained three cross-plugin interfaces so any plugin can discover and consume datasets without depending on the Data plugin directly:

• IDataObjectProvider — query datasets, run aggregations, list saved query views
• IPluginDataSourceProvider — expose named groups of data source entries (datasets, views, projects) that other plugins can browse and query
• DataSourceModels — the DataSourceGroup / DataSourceEntry hierarchy that providers return

This is the capability broker pattern PrivStack already uses for other cross-plugin contracts. The Data plugin registers itself as the IDataObjectProvider and IPluginDataSourceProvider on initialization; the Notes plugin (or any future consumer) requests those capabilities without knowing which plugin provides them.

A new EntityMetadataService also landed in the shell, giving a centralized way to look up entity metadata across all plugins — a missing piece several features were working around.

The Data Plugin: Full Dataset UI

The entire PrivStack.Plugin.Data plugin is new in this push. It's a complete dataset management UI:

• Data grid browser with pagination, inline cell editing, column sort, and saved views
• Query editor with syntax highlighting (AvaloniaEdit + TextMate), source: autocomplete that suggests dataset names after typing source:, alias-qualified column completion (type e. and get columns from the aliased dataset), live query execution, dry-run previews for mutations, and a saved queries panel
• Schema builder for creating empty datasets with defined column types
• CSV import — both file-based and clipboard paste via ImportFromContentAsync
• Notion import — a dedicated import path for Notion database exports
• Deep linking — implements IDeepLinkTarget so other plugins can navigate to a specific dataset row

Notes Integration: Data Sources Panel and Drag-to-Insert

This is where it all comes together for the end user. The Notes plugin now has a Data Sources panel in the sidebar that shows all datasets and saved query views, fetched through IDataObjectProvider. The panel is resizable via a GridSplitter between the page tree and data sources — no more fixed height fighting for space.

The key interaction: drag a data source item onto a page to insert a live dataset table. The drag implementation went through a couple iterations — Button swallows pointer events internally, so the template was switched to Border with dynamic PointerMoved/PointerReleased wiring in PointerPressed (matching the working page tree drag pattern). Drop targets on both BlockWrapper and the empty editor area create a TableBlock backed by either a dataset ID or inline SQL (for views).

View-backed tables needed their own loading path. The existing TableBlockEditor only handled DatasetId-based queries. Views set InlineSql instead, so a new RebuildViewTableAsync method routes those through ExecuteSqlV2Async — the v2 path that resolves source: aliases before executing. This was also where a macOS pasteboard UTI warning surfaced: the drag data key was changed from "DataSourceItem" to "com.privstack.datasourceitem" (reverse-DNS format) to suppress it.

Tasks and Contacts also got IPluginDataSourceProvider implementations via partial classes (TasksPlugin.DataSources.cs, ContactsPlugin.DataSources.cs), exposing their data through the same panel. The architecture is open — any plugin that implements the interface shows up automatically.

The Chart Block

Tucked inside the Notes diff is a ChartBlockEditor — over 1,100 lines of a new chart visualization block that renders dataset-backed charts directly in a Notes page. It includes a chart export renderer with a SkiaSharp drawing backend for PDF/DOCX export, so charts aren't just screenshots — they're vector-rendered in exports.

What's Next

The dataset engine has the SQL foundation but no demo content yet — new users land on an empty grid. Seed data with sample datasets, pre-configured views, and saved queries that showcase cross-dataset joins and CTEs is the immediate next step. There's also a two-line fix needed in the Rust preprocessor: WITH (CTE) queries currently get classified as StatementType::Other and routed to the mutation path instead of the read path. The C# side already handles this correctly — the Rust side just needs to match.

Need I note it's also pretty rough around the edges as it was just setup. I will be working on polishing it up over the coming weeks to make sure it is fast, speedy, and most importantly, useful.