Changelog

Phase 6 of the DuckDB -> SQLite migration. This is the most complex

migration because datasets used DuckDB for OLAP features.

Key changes:

• Replace duckdb dependency with privstack-db + csv crate
• CSV import: replace DuckDB's read_csv_auto() with Rust-side csv crate

parsing with automatic type inference (Integer -> Float -> Text widening)

• Column introspection: replace information_schema.columns with

PRAGMA table_info()

• ALTER COLUMN SET DATA TYPE: replaced with table rebuild pattern

(create new table -> copy with CAST -> drop old -> rename) since

SQLite doesn't support ALTER COLUMN

• DESCRIBE SELECT: replaced with stmt.columns() using rusqlite's

column_decltype feature for type introspection

• ILIKE -> LIKE (SQLite's LIKE is case-insensitive for ASCII by default)
• BOOLEAN columns stored as INTEGER (0/1) with conversion in read/write
• Dry-run mutations use SAVEPOINT/ROLLBACK TO instead of DuckDB's

explicit BEGIN/ROLLBACK transaction pattern

• DDL types normalized: VARCHAR->TEXT, BIGINT->INTEGER, BOOLEAN->INTEGER
• open_datasets_db() removed; replaced by privstack_db::open_db_unencrypted()
• Added open_with_conn() constructor for external connection management

All 73 tests pass, zero warnings. The privstack-ffi crate (which depends

on this crate) compiles cleanly.

Replace DuckDB with SQLite via privstack-db across the entire

privstack-storage crate. This is the largest piece of the DuckDB-to-

SQLite migration.

Key changes:

Cargo.toml: Replace duckdb + uuid deps with privstack-db.

error.rs: Replace duckdb::Error with rusqlite::Error + DbError variants.

Remove the Encryption error variant (encryption now at SQLCipher level).

lib.rs: Remove open_duckdb_with_wal_recovery() and apply_resource_limits()

(SQLite handles WAL recovery automatically). Update re-exports from

scan_duckdb_* to scan_db_*.

event_store.rs: Use privstack_db::open_db_unencrypted/open_in_memory.

Add open_with_conn() for shared connection support. Use

privstack_db::checkpoint(). Translate schema: VARCHAR->TEXT,

BIGINT->INTEGER, TIMESTAMP->INTEGER.

entity_store.rs (~1860 lines rewritten):

• Remove ALL application-level encryption: encryptor field,

encrypt_data_json(), decrypt_data_json(), migrate_unencrypted(),

re_encrypt_all(), base64 helpers. Encryption is now at the SQLCipher

database level.

• Remove open_with_encryptor() constructor, add open_with_conn() for

shared connection pattern.

• SQL dialect translation: VARCHAR->TEXT, BIGINT->INTEGER,

BOOLEAN DEFAULT FALSE->INTEGER DEFAULT 0, VARCHAR[]->TEXT (JSON arrays),

DOUBLE[]->TEXT (JSON arrays), is_trashed=FALSE->is_trashed=0,

information_schema->sqlite_master/PRAGMA, ILIKE->LIKE,

CHECKPOINT->PRAGMA wal_checkpoint(TRUNCATE),

list_cosine_similarity->cosine_similarity (custom fn from privstack-db),

::DOUBLE[] array literals->JSON text strings.

• Tags stored as JSON arrays via serde_json::to_string() instead of

DuckDB array literal format.

• Embeddings stored as JSON text instead of DOUBLE[] arrays.
• Compact: Replace DuckDB ATTACH/COPY FROM DATABASE/DETACH/swap with

SQLite VACUUM INTO via privstack_db::compact().

• Diagnostics: Replace duckdb_tables()/duckdb_views()/duckdb_indexes()

with sqlite_master queries and PRAGMA index_list().

• Schema init: Replace information_schema introspection with

privstack_db::table_exists/column_exists/add_column_if_not_exists.

• Register cosine_similarity custom function on connection open.

Tests: Remove encryption-specific tests (FailingEncryptor,

UnavailableEncryptor, migrate_unencrypted, re_encrypt_all). Replace

uuid::Uuid::new_v4() with atomic counter. All 97 tests pass.

FFI (privstack-ffi): Update callers — replace open_with_encryptor with

open(), remove migrate_unencrypted/re_encrypt_all calls, rename

scan_duckdb_file/compact_duckdb_file to scan_db_file/compact_db_file.

Replace duckdb with privstack-db (rusqlite/SQLCipher) in the vault crate.

Vault blob encryption (ChaCha20-Poly1305) is preserved — vault blobs

contain secrets (cloud keys, recovery material) that need crypto

protection even within SQLCipher.

Changes:

• Replace duckdb dependency with privstack-db
• Translate SQL types: VARCHAR→TEXT, BIGINT→INTEGER
• Translate param syntax: positional ?→?N
• VaultManager: add open_with_conn() for shared connection support
• VaultManager::open() uses privstack_db::open_db_unencrypted()
• VaultManager::checkpoint() uses privstack_db::checkpoint()
• Update all 3 test files (145 tests): replace duckdb::Connection

with privstack_db::open_in_memory(), update file extensions

All 145 tests pass (32 recovery + 91 integration + 22 unit).

Note: Build may fail at this commit due to privstack-ffi still

referencing removed DuckDB vault APIs — will be resolved when FFI

is migrated.

Phase 1 (Foundation):

• Add rusqlite 0.31 with bundled-sqlcipher to workspace dependencies
• Create privstack-db crate: unified SQLCipher connection management with

open_db(), rekey(), checkpoint(), compact(), and helper utilities

• Implement cosine_similarity() as a registered Rust scalar function

(replaces DuckDB's list_cosine_similarity for RAG vector search)

• Add derive_sqlcipher_key() to privstack-crypto for raw hex key formatting
• Align privstack-sync to use workspace rusqlite (resolves libsqlite3-sys

link conflict between bundled and bundled-sqlcipher features)

• 24 tests for privstack-db (encryption, rekey, FTS5, cosine similarity)

Phase 2 (BlobStore Migration):

• Replace duckdb dependency with privstack-db in privstack-blobstore
• Remove all per-entity encryption from BlobStore: encryptor field,

open_with_encryptor(), encrypt/decrypt in store/read, re_encrypt_all(),

migrate_unencrypted(), blob_entity_id() -- at-rest encryption now handled

by SQLCipher at the database file level

• Remove BlobStoreError::Encryption variant (no longer needed)
• Remove privstack-crypto dependency from blobstore
• Translate schema: VARCHAR→TEXT, BIGINT→INTEGER
• Translate SQL: positional params now use ?N syntax
• BlobStore accepts shared Arc<Mutex<Connection>> via open_with_conn()
• 36 tests pass (removed 15 encryption-specific tests, all CRUD/error tests

preserved and passing)

Note: Build may fail at this commit due to privstack-ffi still referencing

removed BlobStore APIs (open_with_encryptor, re_encrypt_all,

migrate_unencrypted) -- will be resolved when FFI is migrated.