September Release Notes

Category: Product
Security & Performance: Yes
Featured?: Yes
Quarter: 72fa4497711a77b89bf2a5c4d4460464

TerrariumDB v8.0 introduces a new Storage Core Format (SCF4) with pluggable storage engines and execution-path isolation, alongside major SQL additions — DELETE, WITH, multi-JOINs, and TRUNCATE ALL.

‍

Security is strengthened with TLS-secured gRPC by default, and observability is upgraded across metrics and logging. Operationally, v8.0 reduces ingest service CPU utylization, and speedup backups. Under the hood, you’ll see faster, more predictable query execution via refined thread-pooling and scheduler work, plus numerous fixes that improve SQL correctness, stabilize aggregator ↔ worker flows, and ensure cleaner shutdown behavior.

‍

What’s new

‍

Pluggable storage engines — a stable Storage Engine API that decouples query/execution from persistence, letting the database load a single, process-wide engine implementation (e.g., row, columnar, LSM, in-memory, cloud-backed) at startup and swap/upgrade it without changing application code.
Parallel interpreter unification — merged GROUP BY and ORDER BY interpreters to improve parallel processing.
SQL: DELETE statement — native row removal in the SQL layer.
Segment index lookup endpoint — returns the matched segment index (or null) for a client through the aggregator/worker path.
SQL: WITH (CTE) support — richer query composition.
SQL: Multi-JOINs — support for multiple JOIN operations in a single query.
Rebalance tooling — shard rebalance script to assist controlled data movement across the cluster and aggregator.
gRPC TLS by default — single-file certificate chains, TLS 1.2+ (1.3 preferred), SAN validation.
Client-Segment Index Lookup - a new endpoint in analytics layer returns the matched segment index per client to speed real-time routing decisions.

‍

Improvements

‍

SCF4 engine support — new on-disk format with zero SCF2 conversion during reads; foundation for future type-specific optimizations.
SCF4 compression strategies — evaluated per-segment and whole-column approaches to cut storage and IO.
SQL thread-pool isolation — dedicated pool on workers for SQL layer (JOIN, GROUP BY, ORDER BY, etc.).
Dedicated ingestion pool (analytics layer) — separates ingestion from other SQL work to protect latency under write pressure.
TRUNCATE ALL TABLES — management command for bulk table truncation within a database.
Observability: SQL logging — richer fields (start/end, duration, parameters, rows, user/host, status) and worker execution times; reduced footprint for non-tracked query types.
Operational metrics — added user/host to MYSQL_ACTIVE_CONNECTIONS and new metrics for the aggregator/controller.
Security & auth UX — password-cache invalidation across gateways on ALTER USER; test suites for user/role and privilege surfaces; SHOW ROLE EDGES.
Concurrency — RAND() rewritten to be multi-thread friendly.
Client compatibility — added default_storage_engine system variable required by sql client connectivity.

‍

Fixes

‍

Aggregator stability — safer error-data handling; prevent disconnections when coordination service session not initialized; cancel streaming RPCs before shutdown.
Time-zone performance — resolved slowdown caused by a custom date/time runtime layer.
SQL correctness
- proper deserialization for GROUP BY;
- fix * schema expansion; CAST … AS CHAR now copies by byte size;
- proper escaping for patterns like \[abc] in LIKE;
- whitespace stripped from SQL logs;
Build & robustness — Debian Trixie build fixed; guard against excessively deep JSON in serialized queries with a clear exception instead of generic “Unknown statement.”.

‍

Compatibility & rollout notes

‍

Storage format (SCF4). SCF4 becomes the primary format and removes on-read SCF2 conversions. Expect fewer physical files per dataset, faster backups, and a better path for future per-type optimizations. Plan migrations and backup policies accordingly.
Thread-pool separation. New pools for SQL and ingestion isolate workloads; validate CPU reservations/limits and update SLOs/alerts to reflect the new pools.
Secure transport. Enforce TLS for all gRPC listeners/clients using single-file PEM bundles (server and client trust). Verify SANs and cipher policies before enabling cluster-wide.

‍