Arkilian mark

Documentation

Use Arkilian
in your language.

One embedded engine, six SDKs. Open a database, run DDL through it, and every write streams to S3 — verified, snapshotted, replayable.

Start with Quickstart
On this page

01 · Start

Quickstart #

Use Arkilian anywhere you use SQLite. Open a database, run DDL through it, and every write is captured, batched, and streamed to S3 — with no code changes beyond the constructor.

app.js
import Arkilian from 'arkilian';

const db = new Arkilian('app.sqlite');

db.exec(`CREATE TABLE IF NOT EXISTS users (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  name TEXT NOT NULL
)`);

db.run('INSERT INTO users (name) VALUES (?)', ['Ada']);
console.log(db.all('SELECT * FROM users'));

db.close();

Run DDL through Arkilian — not a raw handle — so capture triggers are wired automatically.

Run DDL through Arkilian. Raw handles bypass trigger wiring — use resyncTriggers() afterwards, or enable auto-resync.

Tables without a PRIMARY KEY are skipped; count them with backupSkippedTableCount.

02 · Start

Configuration #

Set ARKILIAN_ENABLE_BACKUP=0 to run embedded-only with no S3 traffic. Otherwise Arkilian reads every variable below from the process environment or a ./.env file — environment wins.

terminal
npm install arkilian        # Node.js / Bun — prebuilt N-API binary, no compiler needed
pip install arkilian        # Python (CFFI)
go get github.com/arkiliandb/arkilian/bindings/go/arkilian
VariableDefault — behavior
ARKILIAN_DB_PATHapp.sqlite — path to the primary database file
ARKILIAN_BACKUP_PATHbackup.sqlite — staging path for snapshot copies
ARKILIAN_BACKUP_INTERVAL3600 — snapshot interval in seconds (minimum 1)
ARKILIAN_CHUNK_INTERVAL_SEC1 — seconds between packaging outbox rows into S3 chunks
ARKILIAN_MANIFEST_INTERVAL_SEC30 — minimum seconds between manifest publishes
ARKILIAN_S3_ENDPOINT(empty) — S3 URL. Unset means shipping stays dormant
ARKILIAN_S3_BUCKET(empty) — target bucket for snapshots, chunks, manifests
ARKILIAN_S3_REGIONus-east-1 — SigV4 signing region
ARKILIAN_S3_PREFIXdb_default — isolates one database inside a shared bucket
ARKILIAN_MANIFEST_HMAC_KEY(empty) — REQUIRED for publishing and hydration. Fail closed
ARKILIAN_MAX_QUEUE_DEPTH100000 — outbox rows before capture pauses (writes never block)
ARKILIAN_MAX_ATTEMPTS100 — upload retries per chunk before rows move to the DLQ
ARKILIAN_ENABLE_BACKUP1 — set 0 to disable outbound shipping at runtime
ARKILIAN_OUTBOX_DURABLE1 — FULL synchronous outbox flush. Set 0 (NORMAL) for max throughput

03 · Core

Capture & snapshots #

DDL executed through Arkilian creates row-level triggers for every table with a primary key, backed by a transaction-aware sidecar journal. Each INSERT, UPDATE, and DELETE lands in the local _pending_backup outbox in strict LSN order.

Chunks — every ARKILIAN_CHUNK_INTERVAL_SEC a flush thread batches outbox rows into SHA-256-verified SQL chunks, uploaded with SigV4 presigned PUTs. Delivery is at-least-once: replay uses idempotent REPLACE INTO / DELETE, so overlapping ranges are safe.

Snapshots — every ARKILIAN_BACKUP_INTERVAL a snapshot thread takes a non-blocking online backup (sqlite3_backup_step) to backup.sqlite. If the outbox ever hits capacity, capture pauses while application writes proceed — check capturePaused and take a fresh snapshot to close the gap.

04 · Core

Hydration #

Cold-start any replica from object storage before opening it. Hydration downloads the latest snapshot, verifies every digest and the HMAC manifest signature, then replays incremental chunks. Missing or mismatched signatures refuse to restore.

app.js
import Arkilian from 'arkilian';

Arkilian.hydrateS3('app.sqlite', {
  endpoint: 'https://s3.amazonaws.com',
  bucket: 'my-app-backups',
  region: 'us-east-1',
  accessKey: process.env.ARKILIAN_S3_ACCESS_KEY,
  secretKey: process.env.ARKILIAN_S3_SECRET_KEY,
  prefix: 'tenant-production',
});

const db = new Arkilian('app.sqlite');
Fail closed by design. Hydration refuses unsigned or mismatched manifests — there is no insecure fallback.

Requires ARKILIAN_MANIFEST_HMAC_KEY in the environment — the same key that signed the manifest.

05 · SDKs

Node.js / Bun #

Install arkilian from npm. The postinstall pulls a prebuilt N-API binary for your platform (Linux x64/arm64 glibc + musl, macOS x64/arm64, Windows x64) and falls back to building from bundled C sources offline.

terminal
npm install arkilian

Writes, reads, transactions

app.js
import Arkilian from 'arkilian';
const db = new Arkilian('app.sqlite');

db.exec('CREATE TABLE IF NOT EXISTS orders (id INTEGER PRIMARY KEY, item TEXT, qty INT)');
db.run('INSERT INTO orders (item, qty) VALUES (?, ?)', ['Widget', 5]);

console.log(db.lastInsertRowid); // id of the row just written
console.log(db.changes);         // rows modified by the last statement

const orders = db.all('SELECT id, item, qty FROM orders');

db.transaction((tx) => {
  tx.run('INSERT INTO orders (item, qty) VALUES (?, ?)', ['Gadget', 10]);
  tx.run('INSERT INTO orders (item, qty) VALUES (?, ?)', ['Sprocket', 2]);
}); // rolls back automatically on throw

Backup health

health.js
console.log(db.backupHealthy);              // 1 when all core checks pass
console.log(db.backupHealthFlags.toString(2)); // ARK_HF_* bitmask — exact fault
console.log(db.backupQueueDepth);              // rows waiting in _pending_backup
console.log(db.backupOldestPendingAgeSec);     // real-time replication lag
console.log(db.backupChunkCount);              // chunks shipped so far

db.close();

06 · SDKs

Python #

Install from bindings/python. The binding is an idiomatic class over CFFI — rows come back as lists of dicts.

terminal
cd bindings/python
pip install -e .
app.py
from arkilian import Arkilian

db = Arkilian("app.sqlite")
db.exec("CREATE TABLE IF NOT EXISTS orders (id INTEGER PRIMARY KEY, item TEXT, qty INT)")

db.run("INSERT INTO orders (item, qty) VALUES (?, ?)", ["Widget", 5])
print(db.last_insert_rowid)

for row in db.all("SELECT id, item, qty FROM orders"):
    print(row["item"], row["qty"])

db.begin()
try:
    db.run("INSERT INTO orders (item, qty) VALUES (?, ?)", ["Gadget", 10])
    db.commit()
except Exception:
    db.rollback()
    raise

print(db.backup_is_healthy, db.backup_queue_depth)
db.close()

Hydrate with Arkilian.hydrate_s3(db_path, endpoint, bucket, region, access_key, secret_key, prefix) before constructing the instance.

07 · SDKs

Go #

Import github.com/arkiliandb/arkilian/bindings/go/arkilian. The package compiles SQLite and Arkilian directly via cgo — no system libraries needed.

terminal
go get github.com/arkiliandb/arkilian/bindings/go/arkilian
main.go
db, err := arkilian.OpenDB("app.sqlite")
if err != nil {
  log.Fatal(err)
}
defer db.Close()

db.Exec("CREATE TABLE IF NOT EXISTS players (id INTEGER PRIMARY KEY, name TEXT, score INT)")

stmt, _ := db.Prepare("INSERT INTO players (name, score) VALUES (?, ?)")
defer stmt.Finalize()
stmt.BindText(1, "PlayerOne")
stmt.BindInt(2, 4200)
stmt.Step()

fmt.Println(db.Changes(), db.BackupQueueDepth(), db.BackupIsHealthy())

Use arkilian.Open(token, dbPath) when you manage auth tokens explicitly, and db.ResyncTriggers() after DDL issued through a raw handle.

08 · SDKs

Rust #

Add the arkilian crate. It exposes safe abstractions over arkilian-sys; prepare + use_stmt select the active statement for stepping and column reads.

main.rs
use arkilian::Database;

let db = Database::new("app.sqlite")?;
db.exec("CREATE TABLE IF NOT EXISTS scores (id INTEGER PRIMARY KEY, player TEXT, points INT)")?;

db.prepare("INSERT INTO scores (player, points) VALUES (?, ?)")?;
db.use_stmt(0)?;
db.bind_text(1, "Hero")?;
db.bind_int(2, 100)?;
db.step()?;
db.finalize()?;

println!("healthy: {}", db.backup_is_healthy());

Hydrate with arkilian::hydrate_s3(path, prefix, &s3config) before opening. Transactions use begin / commit / rollback.

09 · SDKs

PHP #

Requires PHP 7.4+ with FFI. Include bindings/php/Arkilian.php — reads return plain arrays, and every method returns $this for chaining where it makes sense.

app.php
<?php
require_once 'Arkilian.php';

$db = new Arkilian('app.sqlite');
$db->exec("CREATE TABLE IF NOT EXISTS messages (id INTEGER PRIMARY KEY, text TEXT)");
$db->run("INSERT INTO messages (text) VALUES (?)", ["Hello from PHP"]);

print_r($db->all("SELECT id, text FROM messages"));

echo $db->backupIsHealthy() ? "healthy" : "degraded", "\n";
$db->close();

Prepared statements follow prepare useStmtbindText / bindIntstep finalize. Hydrate statically with Arkilian::hydrateS3($path, $prefix, $s3Config).

10 · SDKs

C / C++ #

Link the static or shared library from CMake (ARKILIAN_BUILD_SHARED / ARKILIAN_BUILD_STATIC). Three independent connections eliminate cross-thread lock contention — never share one handle across threads.

main.c
#include "class.h"

arkilian *db = NULL;
db_init(&db, "app.sqlite");

db_exec(db, "CREATE TABLE items (id INTEGER PRIMARY KEY, title TEXT);");

db_prepare(db, "INSERT INTO items (title) VALUES (?);");
db_bind_text(db, 1, "Toolbox");
db_step(db);
db_finalize(db);

printf("queue=%d healthy=%d\n",
  db_backup_queue_depth(db), db_backup_is_healthy(db));

db_close(db);
terminal
gcc -I/usr/local/include/arkilian myapp.c -L/usr/local/lib -larkilian -lcurl -lpthread -lm -o myapp

11 · Operate

Telemetry #

Every binding exposes the same signals — Node.js uses getters (db.backupQueueDepth), Python properties (db.backup_queue_depth), Go methods (db.BackupQueueDepth()), and C functions (db_backup_queue_depth()).

SignalMeaning
backupQueueDepthRows in _pending_backup waiting for S3
backupOldestPendingAgeSecAge of the oldest pending row — replication lag
backupDeadLetterCountPoison rows parked in _dead_backup
backupThreadHeartbeatAgeMsMs since flush heartbeat (-1 when idle)
backupSnapshotHeartbeatAgeMsMs since snapshot heartbeat (-1 when idle)
backupChunkCountChunks shipped successfully, lifetime
backupLastChunkFlushAgeMsMs since the last successful upload
backupTriggerCoverage0 when every PK table has triggers; >0 otherwise
backupSkippedTableCountTables without a PRIMARY KEY, ignored by capture
backupHealthy / backupHealthFlags1 when core checks pass; bitmask below on failure

Health bitmask (ARK_HF_*)

BitFlag — set when…
0 · 0x001BACKUP_ENABLED — kill-switch off
1 · 0x002DEST_CONFIGURED — bucket + credentials present
2 · 0x004FLUSH_ALIVE — flush heartbeat fresh
3 · 0x008SNAPSHOT_ALIVE — snapshot heartbeat fresh
4 · 0x010QUEUE_BELOW_CAP — outbox under capacity
5 · 0x020SCHEMA_IN_SYNC — triggers match tables
6 · 0x040NO_DEAD_LETTER — DLQ empty
7 · 0x080MANIFEST_RESOLVED — manifest registry active
8 · 0x100NO_CAPTURE_GAP — no undrained drop gap

12 · Operate

Dead-letter queue #

Rows that fail upload ARKILIAN_MAX_ATTEMPTS times are parked in _dead_backup so one poison row never blocks the pipeline. Inspect and replay them with arkilian-dlq — it compiles against the bundled amalgamation with zero extra dependencies.

terminal
cc tools/arkilian-dlq.c src/deps/sqlite/sqlite3.c -Isrc/deps/sqlite -o arkilian-dlq

./arkilian-dlq app.sqlite --count     # how many poison rows
./arkilian-dlq app.sqlite --list      # payloads + error reasons
./arkilian-dlq app.sqlite --replay --dry-run
./arkilian-dlq app.sqlite --replay    # re-queue into _pending_backup

13 · Operate

Benchmarks #

Arkilian vs raw SQLite 3.46.1, same machine, schema, and connection settings (tests/bench_1m.c, Intel i7-9750H, journal_mode=WAL). Latency percentiles are identical — P50/P99 INSERT 256/512 µs, point SELECT 8/16 µs.

−2.2%

INSERT overhead vs raw SQLite

−0.2%

UPDATE overhead by primary key

29.4k

Batched ops/s — matches raw SQLite

8 µs

Point-select P50 latency, identical

Single-row op (100k)Raw SQLite → Arkilian
INSERT autocommit5,262 → 5,144 ops/s (−2.2%)
UPDATE by PK5,030 → 5,018 ops/s (−0.2%)
SELECT point by PK139,147 → 135,875 ops/s (−2.4%)
SELECT range 100 rows6,182 → 6,017 ops/s (−2.7%)
Batched INSERT (100k)Raw SQLite → Arkilian
Batch 1010,354 → 10,516 ops/s (+1.6%)
Batch 10022,975 → 23,231 ops/s (+1.1%)
Batch 1,00028,604 → 28,383 ops/s (−0.8%)
Batch 100,00029,304 → 29,432 ops/s (+0.4%)

Full curves, RSS telemetry, and reproduction steps live in BENCHMARK.md.

14 · Operate

Reference #

One surface, six spellings. Core verbs are identical everywhere: open, exec, run / prepare, read, transact, observe, close.

BindingOpen · write · read · close
Node.jsnew Arkilian(path) · exec/run · all<T> · transaction(fn) · close()
PythonArkilian(path) · exec/run · all · begin/commit/rollback · close()
GoOpenDB(path) · Exec/Prepare+Step · Begin/Commit/Rollback · Close()
RustDatabase::new(path) · exec/prepare+use_stmt+step · begin/commit/rollback
PHPnew Arkilian(path) · exec/run · all · begin/commit/rollback · close()
Cdb_init · db_exec/db_prepare+db_step · explicit SQL txn · db_close
BindingHydrate before open
Node.jsArkilian.hydrateS3(path, {endpoint, bucket, region, accessKey, secretKey, prefix})
PythonArkilian.hydrate_s3(path, endpoint, bucket, region, access_key, secret_key, prefix)
PHPArkilian::hydrateS3($path, $prefix, $s3Config)
Rustarkilian::hydrate_s3(path, prefix, &s3config)
Go / CConfigure env (.env) — hydration runs inside open/init

Ship your first snapshot today

Install the SDK, point it at SQLite, and let S3 hold the truth.