01 · Start
Quickstart #
Use Arkilian anywhere you use SQLite. Open a database, run DDL through it, and every write is captured, batched, and streamed to S3 — with no code changes beyond the constructor.
import Arkilian from 'arkilian';
const db = new Arkilian('app.sqlite');
db.exec(`CREATE TABLE IF NOT EXISTS users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL
)`);
db.run('INSERT INTO users (name) VALUES (?)', ['Ada']);
console.log(db.all('SELECT * FROM users'));
db.close();from arkilian import Arkilian
db = Arkilian("app.sqlite")
db.exec("""CREATE TABLE IF NOT EXISTS users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL
)""")
db.run("INSERT INTO users (name) VALUES (?)", ["Ada"])
print(db.all("SELECT * FROM users"))
db.close()package main
import (
"fmt"
"github.com/arkiliandb/arkilian/bindings/go/arkilian"
)
func main() {
db, _ := arkilian.OpenDB("app.sqlite")
defer db.Close()
db.Exec(`CREATE TABLE IF NOT EXISTS users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL
)`)
stmt, _ := db.Prepare("INSERT INTO users (name) VALUES (?)")
defer stmt.Finalize()
stmt.BindText(1, "Ada")
stmt.Step()
fmt.Println("changes:", db.Changes())
}Run DDL through Arkilian — not a raw handle — so capture triggers are wired automatically.
resyncTriggers() afterwards, or enable auto-resync.Tables without a PRIMARY KEY are skipped; count them with backupSkippedTableCount.
02 · Start
Configuration #
Set ARKILIAN_ENABLE_BACKUP=0 to run embedded-only with no S3 traffic. Otherwise Arkilian reads every variable below from the process environment or a ./.env file — environment wins.
npm install arkilian # Node.js / Bun — prebuilt N-API binary, no compiler needed
pip install arkilian # Python (CFFI)
go get github.com/arkiliandb/arkilian/bindings/go/arkilian| Variable | Default — behavior |
|---|---|
| ARKILIAN_DB_PATH | app.sqlite — path to the primary database file |
| ARKILIAN_BACKUP_PATH | backup.sqlite — staging path for snapshot copies |
| ARKILIAN_BACKUP_INTERVAL | 3600 — snapshot interval in seconds (minimum 1) |
| ARKILIAN_CHUNK_INTERVAL_SEC | 1 — seconds between packaging outbox rows into S3 chunks |
| ARKILIAN_MANIFEST_INTERVAL_SEC | 30 — minimum seconds between manifest publishes |
| ARKILIAN_S3_ENDPOINT | (empty) — S3 URL. Unset means shipping stays dormant |
| ARKILIAN_S3_BUCKET | (empty) — target bucket for snapshots, chunks, manifests |
| ARKILIAN_S3_REGION | us-east-1 — SigV4 signing region |
| ARKILIAN_S3_PREFIX | db_default — isolates one database inside a shared bucket |
| ARKILIAN_MANIFEST_HMAC_KEY | (empty) — REQUIRED for publishing and hydration. Fail closed |
| ARKILIAN_MAX_QUEUE_DEPTH | 100000 — outbox rows before capture pauses (writes never block) |
| ARKILIAN_MAX_ATTEMPTS | 100 — upload retries per chunk before rows move to the DLQ |
| ARKILIAN_ENABLE_BACKUP | 1 — set 0 to disable outbound shipping at runtime |
| ARKILIAN_OUTBOX_DURABLE | 1 — FULL synchronous outbox flush. Set 0 (NORMAL) for max throughput |
03 · Core
Capture & snapshots #
DDL executed through Arkilian creates row-level triggers for every table with a primary key, backed by a transaction-aware sidecar journal. Each INSERT, UPDATE, and DELETE lands in the local _pending_backup outbox in strict LSN order.
Chunks — every ARKILIAN_CHUNK_INTERVAL_SEC a flush thread batches outbox rows into SHA-256-verified SQL chunks, uploaded with SigV4 presigned PUTs. Delivery is at-least-once: replay uses idempotent REPLACE INTO / DELETE, so overlapping ranges are safe.
Snapshots — every ARKILIAN_BACKUP_INTERVAL a snapshot thread takes a non-blocking online backup (sqlite3_backup_step) to backup.sqlite. If the outbox ever hits capacity, capture pauses while application writes proceed — check capturePaused and take a fresh snapshot to close the gap.
04 · Core
Hydration #
Cold-start any replica from object storage before opening it. Hydration downloads the latest snapshot, verifies every digest and the HMAC manifest signature, then replays incremental chunks. Missing or mismatched signatures refuse to restore.
import Arkilian from 'arkilian';
Arkilian.hydrateS3('app.sqlite', {
endpoint: 'https://s3.amazonaws.com',
bucket: 'my-app-backups',
region: 'us-east-1',
accessKey: process.env.ARKILIAN_S3_ACCESS_KEY,
secretKey: process.env.ARKILIAN_S3_SECRET_KEY,
prefix: 'tenant-production',
});
const db = new Arkilian('app.sqlite');Requires ARKILIAN_MANIFEST_HMAC_KEY in the environment — the same key that signed the manifest.
05 · SDKs
Node.js / Bun #
Install arkilian from npm. The postinstall pulls a prebuilt N-API binary for your platform (Linux x64/arm64 glibc + musl, macOS x64/arm64, Windows x64) and falls back to building from bundled C sources offline.
npm install arkilianWrites, reads, transactions
import Arkilian from 'arkilian';
const db = new Arkilian('app.sqlite');
db.exec('CREATE TABLE IF NOT EXISTS orders (id INTEGER PRIMARY KEY, item TEXT, qty INT)');
db.run('INSERT INTO orders (item, qty) VALUES (?, ?)', ['Widget', 5]);
console.log(db.lastInsertRowid); // id of the row just written
console.log(db.changes); // rows modified by the last statement
const orders = db.all('SELECT id, item, qty FROM orders');
db.transaction((tx) => {
tx.run('INSERT INTO orders (item, qty) VALUES (?, ?)', ['Gadget', 10]);
tx.run('INSERT INTO orders (item, qty) VALUES (?, ?)', ['Sprocket', 2]);
}); // rolls back automatically on throwBackup health
console.log(db.backupHealthy); // 1 when all core checks pass
console.log(db.backupHealthFlags.toString(2)); // ARK_HF_* bitmask — exact fault
console.log(db.backupQueueDepth); // rows waiting in _pending_backup
console.log(db.backupOldestPendingAgeSec); // real-time replication lag
console.log(db.backupChunkCount); // chunks shipped so far
db.close();06 · SDKs
Python #
Install from bindings/python. The binding is an idiomatic class over CFFI — rows come back as lists of dicts.
cd bindings/python
pip install -e .from arkilian import Arkilian
db = Arkilian("app.sqlite")
db.exec("CREATE TABLE IF NOT EXISTS orders (id INTEGER PRIMARY KEY, item TEXT, qty INT)")
db.run("INSERT INTO orders (item, qty) VALUES (?, ?)", ["Widget", 5])
print(db.last_insert_rowid)
for row in db.all("SELECT id, item, qty FROM orders"):
print(row["item"], row["qty"])
db.begin()
try:
db.run("INSERT INTO orders (item, qty) VALUES (?, ?)", ["Gadget", 10])
db.commit()
except Exception:
db.rollback()
raise
print(db.backup_is_healthy, db.backup_queue_depth)
db.close()Hydrate with Arkilian.hydrate_s3(db_path, endpoint, bucket, region, access_key, secret_key, prefix) before constructing the instance.
07 · SDKs
Go #
Import github.com/arkiliandb/arkilian/bindings/go/arkilian. The package compiles SQLite and Arkilian directly via cgo — no system libraries needed.
go get github.com/arkiliandb/arkilian/bindings/go/arkiliandb, err := arkilian.OpenDB("app.sqlite")
if err != nil {
log.Fatal(err)
}
defer db.Close()
db.Exec("CREATE TABLE IF NOT EXISTS players (id INTEGER PRIMARY KEY, name TEXT, score INT)")
stmt, _ := db.Prepare("INSERT INTO players (name, score) VALUES (?, ?)")
defer stmt.Finalize()
stmt.BindText(1, "PlayerOne")
stmt.BindInt(2, 4200)
stmt.Step()
fmt.Println(db.Changes(), db.BackupQueueDepth(), db.BackupIsHealthy())Use arkilian.Open(token, dbPath) when you manage auth tokens explicitly, and db.ResyncTriggers() after DDL issued through a raw handle.
08 · SDKs
Rust #
Add the arkilian crate. It exposes safe abstractions over arkilian-sys; prepare + use_stmt select the active statement for stepping and column reads.
use arkilian::Database;
let db = Database::new("app.sqlite")?;
db.exec("CREATE TABLE IF NOT EXISTS scores (id INTEGER PRIMARY KEY, player TEXT, points INT)")?;
db.prepare("INSERT INTO scores (player, points) VALUES (?, ?)")?;
db.use_stmt(0)?;
db.bind_text(1, "Hero")?;
db.bind_int(2, 100)?;
db.step()?;
db.finalize()?;
println!("healthy: {}", db.backup_is_healthy());Hydrate with arkilian::hydrate_s3(path, prefix, &s3config) before opening. Transactions use begin / commit / rollback.
09 · SDKs
PHP #
Requires PHP 7.4+ with FFI. Include bindings/php/Arkilian.php — reads return plain arrays, and every method returns $this for chaining where it makes sense.
<?php
require_once 'Arkilian.php';
$db = new Arkilian('app.sqlite');
$db->exec("CREATE TABLE IF NOT EXISTS messages (id INTEGER PRIMARY KEY, text TEXT)");
$db->run("INSERT INTO messages (text) VALUES (?)", ["Hello from PHP"]);
print_r($db->all("SELECT id, text FROM messages"));
echo $db->backupIsHealthy() ? "healthy" : "degraded", "\n";
$db->close();Prepared statements follow prepare → useStmt → bindText / bindInt → step → finalize. Hydrate statically with Arkilian::hydrateS3($path, $prefix, $s3Config).
10 · SDKs
C / C++ #
Link the static or shared library from CMake (ARKILIAN_BUILD_SHARED / ARKILIAN_BUILD_STATIC). Three independent connections eliminate cross-thread lock contention — never share one handle across threads.
#include "class.h"
arkilian *db = NULL;
db_init(&db, "app.sqlite");
db_exec(db, "CREATE TABLE items (id INTEGER PRIMARY KEY, title TEXT);");
db_prepare(db, "INSERT INTO items (title) VALUES (?);");
db_bind_text(db, 1, "Toolbox");
db_step(db);
db_finalize(db);
printf("queue=%d healthy=%d\n",
db_backup_queue_depth(db), db_backup_is_healthy(db));
db_close(db);gcc -I/usr/local/include/arkilian myapp.c -L/usr/local/lib -larkilian -lcurl -lpthread -lm -o myapp11 · Operate
Telemetry #
Every binding exposes the same signals — Node.js uses getters (db.backupQueueDepth), Python properties (db.backup_queue_depth), Go methods (db.BackupQueueDepth()), and C functions (db_backup_queue_depth()).
| Signal | Meaning |
|---|---|
| backupQueueDepth | Rows in _pending_backup waiting for S3 |
| backupOldestPendingAgeSec | Age of the oldest pending row — replication lag |
| backupDeadLetterCount | Poison rows parked in _dead_backup |
| backupThreadHeartbeatAgeMs | Ms since flush heartbeat (-1 when idle) |
| backupSnapshotHeartbeatAgeMs | Ms since snapshot heartbeat (-1 when idle) |
| backupChunkCount | Chunks shipped successfully, lifetime |
| backupLastChunkFlushAgeMs | Ms since the last successful upload |
| backupTriggerCoverage | 0 when every PK table has triggers; >0 otherwise |
| backupSkippedTableCount | Tables without a PRIMARY KEY, ignored by capture |
| backupHealthy / backupHealthFlags | 1 when core checks pass; bitmask below on failure |
Health bitmask (ARK_HF_*)
| Bit | Flag — set when… |
|---|---|
| 0 · 0x001 | BACKUP_ENABLED — kill-switch off |
| 1 · 0x002 | DEST_CONFIGURED — bucket + credentials present |
| 2 · 0x004 | FLUSH_ALIVE — flush heartbeat fresh |
| 3 · 0x008 | SNAPSHOT_ALIVE — snapshot heartbeat fresh |
| 4 · 0x010 | QUEUE_BELOW_CAP — outbox under capacity |
| 5 · 0x020 | SCHEMA_IN_SYNC — triggers match tables |
| 6 · 0x040 | NO_DEAD_LETTER — DLQ empty |
| 7 · 0x080 | MANIFEST_RESOLVED — manifest registry active |
| 8 · 0x100 | NO_CAPTURE_GAP — no undrained drop gap |
12 · Operate
Dead-letter queue #
Rows that fail upload ARKILIAN_MAX_ATTEMPTS times are parked in _dead_backup so one poison row never blocks the pipeline. Inspect and replay them with arkilian-dlq — it compiles against the bundled amalgamation with zero extra dependencies.
cc tools/arkilian-dlq.c src/deps/sqlite/sqlite3.c -Isrc/deps/sqlite -o arkilian-dlq
./arkilian-dlq app.sqlite --count # how many poison rows
./arkilian-dlq app.sqlite --list # payloads + error reasons
./arkilian-dlq app.sqlite --replay --dry-run
./arkilian-dlq app.sqlite --replay # re-queue into _pending_backup13 · Operate
Benchmarks #
Arkilian vs raw SQLite 3.46.1, same machine, schema, and connection settings (tests/bench_1m.c, Intel i7-9750H, journal_mode=WAL). Latency percentiles are identical — P50/P99 INSERT 256/512 µs, point SELECT 8/16 µs.
−2.2%
INSERT overhead vs raw SQLite
−0.2%
UPDATE overhead by primary key
29.4k
Batched ops/s — matches raw SQLite
8 µs
Point-select P50 latency, identical
| Single-row op (100k) | Raw SQLite → Arkilian |
|---|---|
| INSERT autocommit | 5,262 → 5,144 ops/s (−2.2%) |
| UPDATE by PK | 5,030 → 5,018 ops/s (−0.2%) |
| SELECT point by PK | 139,147 → 135,875 ops/s (−2.4%) |
| SELECT range 100 rows | 6,182 → 6,017 ops/s (−2.7%) |
| Batched INSERT (100k) | Raw SQLite → Arkilian |
|---|---|
| Batch 10 | 10,354 → 10,516 ops/s (+1.6%) |
| Batch 100 | 22,975 → 23,231 ops/s (+1.1%) |
| Batch 1,000 | 28,604 → 28,383 ops/s (−0.8%) |
| Batch 100,000 | 29,304 → 29,432 ops/s (+0.4%) |
Full curves, RSS telemetry, and reproduction steps live in BENCHMARK.md.
14 · Operate
Reference #
One surface, six spellings. Core verbs are identical everywhere: open, exec, run / prepare, read, transact, observe, close.
| Binding | Open · write · read · close |
|---|---|
| Node.js | new Arkilian(path) · exec/run · all<T> · transaction(fn) · close() |
| Python | Arkilian(path) · exec/run · all · begin/commit/rollback · close() |
| Go | OpenDB(path) · Exec/Prepare+Step · Begin/Commit/Rollback · Close() |
| Rust | Database::new(path) · exec/prepare+use_stmt+step · begin/commit/rollback |
| PHP | new Arkilian(path) · exec/run · all · begin/commit/rollback · close() |
| C | db_init · db_exec/db_prepare+db_step · explicit SQL txn · db_close |
| Binding | Hydrate before open |
|---|---|
| Node.js | Arkilian.hydrateS3(path, {endpoint, bucket, region, accessKey, secretKey, prefix}) |
| Python | Arkilian.hydrate_s3(path, endpoint, bucket, region, access_key, secret_key, prefix) |
| PHP | Arkilian::hydrateS3($path, $prefix, $s3Config) |
| Rust | arkilian::hydrate_s3(path, prefix, &s3config) |
| Go / C | Configure env (.env) — hydration runs inside open/init |

