duckdb-harbor

Many clients, one DuckDB, over plain HTTP. POST a statement, read NDJSON back.

Only one process can access a DuckDB file at a time. Harbor fixes that: it puts a small server in front of the file so all your apps can share it — and it feels exactly like the duckdb shell, except a database can also serve.

DuckDB Harbor is harbor, one small Rust binary with one grammar:

$ harbor                       # what's running
$ harbor mydata.duckdb         # open it — REPL, or -c "SQL", or stdin
$ harbor mydata.duckdb start   # bring it up in the background, until you stop it

harbor mydata.duckdb is the duckdb-shell muscle memory, kept: a REPL with highlighting and completion, -c for one-shots, stdin for scripts. The difference is what happens behind it — if nothing serves the file yet, a server is spawned for it, and every other client of the same file joins that server instead of hitting “database is locked”. The two lifetimes, in one breath: bare, the server is everyone’s — it lives while anyone is connected; start, the server is yours — it lives until you stop it.

The Elevator Pitch

Two files. That’s the entire install.

The library is DuckDB — all of it, one dynamic library, vanilla, compiled and shipped by the DuckDB team. We never patch it, fork it, or wrap it in bindings. Version hop = swap the file.

The binary is harbor: one small executable that is both sides of the conversation. As a server it loads libduckdb and serves your database over HTTP, on a Unix socket or TCP. As a client it connects to any harbor and gives you a modern shell — syntax highlighting, completion, history — in place of the DuckDB CLI.

You never choose which one you’re running. harbor mydb.duckdb connects if the database is already being served, and spawns a server and connects to it if it isn’t. Your connection is the server’s lifeline: the database stays served while anyone is connected — a second client makes it two, your exit makes it one — and when the last client leaves, the server checkpoints and departs. Nothing to daemonize, nothing to clean up. (Want it to outlive its clients? harbor mydb.duckdb start — then it’s yours until you stop it.)

While it’s up, anything that speaks HTTP can query it: curl, your app, another harbor.

harbor by itself shows what’s being served.

Zero-config by default. No drivers, no ORM, no fleet manager. Built directly on DuckDB’s v2 C API, the API of the 2.0 line.

If it can speak HTTP and parse JSON, it can query your database.

$ curl -s 127.0.0.1:9495/sql -H 'Content-Type: application/json' \
       -d '{"sql":"SELECT id, total FROM orders LIMIT 2"}'
{"type":"schema","columns":[{"name":"id","duckdbType":"BIGINT","lossless":true},
                            {"name":"total","duckdbType":"DECIMAL(10,2)","lossless":true,
                             "decimal":{"width":10,"scale":2}}]}
{"type":"row","values":[1,"19.99"]}
{"type":"row","values":[2,"4.50"]}
{"type":"end","rowCount":2,"timeMs":3}

One schema message, one row per row, one end. Rows go out as DuckDB produces them, so a client can start on row one while the server is still producing the last one.

Ten routes. That is the whole surface — one to run a statement and one to ask whether a server can, four so a transaction can outlive one request, one to stop a statement that is running, one to read the schema without asking five questions, one that says who a server is, and one graceful shutdown route:

GET  /ready                can this server answer a query?
POST /shutdown             drain, checkpoint, and stop

GET  /info                 identity — database path, versions, pid, uptime,
                           and the live client count
GET  /catalog              everything about the database in one stable JSON
                           document — schema, sizes, exact row counts, DDL
                           (?style=lite for the count-free inventory)

POST /sql                  run one statement, stream the result as NDJSON
                           (Accept: application/json for one document instead)
POST /sql/sessions         take a connection and hold it, for a transaction
GET  /sql/sessions         list ALL open sessions — who holds each, how long
DELETE /sql/sessions/<id>  give that one back
POST /sql/sessions/<id>/renew  keep a session alive, running nothing
DELETE /sql/queries/<id>   stop a statement the caller named when it sent it

Three alias spellings are served beside the canonical routes: POST /sql/sessions/new, GET /sessions, and DELETE /shutdown.

POST /sql streams by default. Send Accept: application/json and the same result comes back as one document instead:

{"ok":true,
 "columns":[{"name":"id","duckdbType":"INTEGER","lossless":true}],
 "data":[[1],[2]],
 "rowCount":2,
 "timeMs":3}

Same columns, same values, same encoder — only the framing differs. It is worth asking for when the result is small and a single JSON.parse is simpler than reading lines; it is the wrong choice for anything large, because a JSON document is not valid until its last byte, so nothing can be flushed as it is built. Harbor holds at most 32 MiB for one and refuses past that with a 406 naming NDJSON as the remedy. Streaming has no such limit.

The one thing one-shot does better: since nothing has been sent when the last row lands, a failure is still a real status code. The same query that streams a 200 with an {"type":"error"} line at the end answers 400 in this shape.

The stream compresses on request — Accept-Encoding: zstd, which browsers, Node and Bun send on their own and curl --compressed sends when curl has zstd. The wrapped bytes are the identical NDJSON: a 5M-row integer result measures 169 MB plain and 1.1 MB as zstd, and when the query does any real work the compression rides the writer thread for free. Anything else — gzip included, its encoder would throttle the stream — gets identity, and curl -H 'Accept-Encoding: zstd' ... | zstd -d recovers the stream byte-for-byte.

/ready normally runs SELECT 1 through an ordinary executor and answers 200 {"status":"ready"} or 503. Under sustained worker saturation, the dedicated probe lane asks the control connection instead, so a load balancer can still distinguish busy from dead. It is not a process-liveness check: a process can be running while its database path is broken. Verdicts are cached for one second, so polling costs at most one probe query per second however often it is asked.

Stopping a statement

A statement that has entered DuckDB does not come back until it is done, and harbor runs a small, bounded number at once. So a query nobody wants any more is not a slow request — it is a connection out of service, and enough of them are the whole server.

Name a statement when you send it, and you can stop it:

$ curl -s 127.0.0.1:9495/sql -H 'Content-Type: application/json' \
       -d '{"sql":"SELECT count(*) FROM huge","queryId":"report-7"}' &
$ curl -s -X DELETE 127.0.0.1:9495/sql/queries/report-7
{"cancelled":true}

When cancellation lands before streaming begins, the statement answers 499 with {"code":"cancelled"} — nginx’s code, because there is no standard one for “the caller withdrew” and neither 400 nor 500 is true. If a streaming response already began with 200, cancellation arrives as its final NDJSON error event instead; an HTTP status cannot be changed after its headers were sent. Cancelling something that already finished is {"cancelled":false}, not an error: by the time a Stop button is pressed, the query it refers to may well be over.

The id is chosen by the caller rather than issued by harbor, and it has to be: the response does not begin until the statement is streaming or done, so an id in the reply would arrive too late to be any use. It is a non-empty string of at most 128 characters, and it is refused with a 409 while a statement of that name is already running, so two live queries can never share one name and make a cancel a coin flip.

A deadline is the backstop. {"timeoutMs": N} on a request, or HARBOR_STATEMENT_TIMEOUT_MS for a whole deployment, stops a statement without anyone having to ask. There is no default, deliberately: harbor streams 300,000-row results and is used for queries that take minutes on purpose, so a default deadline would break correct programs to catch incorrect ones. With no deployment cap, zero on a request means no limit. When a deployment cap is set, it is a hard ceiling: a request may ask for less time, but neither a larger value nor zero can opt out of the operator’s limit.

Explicit cancellation remains reachable when every executor is inside a long statement: after sustained saturation, a connection-free probe lane accepts query cancellation, session release, readiness, and inspection requests. The reaper is the independent backstop. It runs on its own thread and never touches HTTP, so deadlines are still enforced if no cancellation request arrives or a client disappears. If a deployment’s worry is runaway queries rather than impatient users, set HARBOR_STATEMENT_TIMEOUT_MS or --statement-timeout <duration>.

A client that goes away has its statement stopped without asking. The server watches the connection while a statement computes and while its rows are written, and a client that closes it, resets it or shuts down its sending side has its statement cancelled, in the minutes a plan computes before its first row as well as mid-stream. In a session a statement cut short this way leaves its transaction aborted, as a cancel does: a transaction with a statement missing does not commit. A result small enough to have been sent whole before the client left was not cut short, and its transaction stands.

Shutting down the sending side is departure even with a request still in flight. A client that pipelines two requests and then half-closes — sends both, then shutdown(SHUT_WR) — has the first cancelled and the second never run, the same as a client that sends one request and half-closes to say it wants nothing more. A client that means to send a second request keeps its sending side open until it has the first answer.

Two smaller things follow from the same machinery. Releasing a session whose statement is still running stops it — {"released":false,"cancelling":true} — and the connection comes back on a reaper tick after execution stops. And a lease that blows its TTL while busy is reclaimed: the lease that most needs taking back is the one wedged inside a runaway statement.

Cancelling a statement inside a transaction aborts that transaction, exactly as it does in Postgres. Harbor does not paper over it — the next statement gets Current transaction is aborted (please ROLLBACK) until you do. Rolling back silently would let the statement after a cancellation commit in autocommit under a client that still believed it was in a transaction. The rule has no exceptions: a 499 inside a transaction means that transaction is over. A cancel that lands before the statement has begun to execute — while an object or array parameter is still being bound, or before an executor has picked the statement up — answers the same 499 and harbor leaves the transaction aborted itself, so a client that carries on cannot commit a transaction with a statement missing. A COMMIT sent to an aborted transaction, whatever aborted it, is a 400 that says it was rolled back: the engine alone would roll back and answer success, and a client reading that answer would believe its work was kept. In autocommit there is no transaction to abort, and the next statement runs.

A COMMIT is the one statement a cancel does not reach once it has begun. An interrupt that arrives as one finishes is reported by the engine after the transaction is durable, which would be a 499 for work that was kept. So a cancel, or a deadline, either lands before the COMMIT starts, which is a 499 with nothing kept, or does not land, and the answer is the engine’s own.

Transactions

A transaction lives on a connection and HTTP requests do not, so one request per statement means no transaction can span two. A session bridges that: a connection pinned to you until you release it or its lease expires. COMMIT and ROLLBACK end the transaction; DELETE releases the session.

$ sid=$(curl -s -X POST 127.0.0.1:9495/sql/sessions | jq -r .sessionId)
$ post() { curl -s 127.0.0.1:9495/sql -H 'Content-Type: application/json' -d "{\"sql\":\"$1\",\"sessionId\":\"$sid\"}"; }
$ post "BEGIN"
$ post "INSERT INTO orders (total) VALUES (19.99) RETURNING id"
$ post "INSERT INTO order_items (order_id, price) VALUES (1, 19.99)"
$ post "COMMIT"
$ curl -s -X DELETE 127.0.0.1:9495/sql/sessions/$sid

This is PgBouncer’s transaction pooling, or ActiveRecord checking a connection out of its pool — with an HTTP request where they have a socket and a thread. The session rules are:

A transaction begins in a session, or not at all. BEGIN or START TRANSACTION without a sessionId is a 400, and so is one behind an analyzed EXPLAIN, which runs what it explains. The pooled connection it would run on goes back when the request ends, so it would answer success and hold nothing, and every statement meant to be inside it would commit on its own.

The command line opens the session for you. In the REPL, a -c string, a piped script or .read, a BEGIN takes a session, every statement after it runs there, and the COMMIT or ROLLBACK that ends the transaction releases it. One that never reached the engine, a typo the parser refuses or a request that failed on the way, ends nothing, and the session is kept; on a transaction an error has already aborted, a COMMIT is rolled back by the server whatever follows the word. A script that fails, or ends with its transaction still open, has it rolled back and says so. The session is an ordinary one. At the REPL it is kept alive while the prompt waits or a result is being read, so the thirty-second idle limit does not take a transaction from someone thinking; the five-minute ceiling still ends it, and the REPL says so when one opens. The keeping is a renewal every ten seconds, which runs nothing; a server that renews only backup sessions gets a SELECT 1 on the session instead. A script’s statements follow one another and need no keeping.

Sessions draw from their own connections. HARBOR_POOL_SIZE (default 16) is opened at load and split: the workers take theirs, sessions get the rest. A pool serving both would run out of workers the moment enough clients held transactions open, and then answer nothing at all. With none free, opening a session is a 503 with Retry-After — queries keep working throughout.

Ordinary sessions have a fixed deadline. Ask for a lifetime with {"ttlMs": N}; Harbor caps it at five minutes and returns the granted lifetime alongside a thirty-second idle timeout. Running SQL does not count as idle, but the fixed deadline applies even while a statement is running. A client that holds one open between statements renews it with POST /sql/sessions/<id>/renew, which restarts the idle timeout and leaves the deadline where it was. A renewal runs nothing: it is not counted as a statement, it waits for no worker and is answered while a statement runs on the session, and it returns {"renewed":true}. A session that has idled out is gone even before it is reclaimed, and a renewal of it returns 404.

Backup sessions renew their deadline. Open one with {"purpose":"backup"}; the response carries "purpose":"backup" when the session granted is one, and a client that depends on it checks. Its ttlMs is a renewal window, default and maximum 60 seconds; idleTtlMs is zero. Send POST /sql/sessions/<id>/renew well before each deadline (the CLI renews every 20 seconds). A renewal returns {"renewed":true} and starts a fresh window, even while SQL is running, and the renewals take the place of both the five-minute ceiling and the thirty-second idle timeout. Expired or released leases return 404 and cannot be revived. /sessions reports renewable and expiresInMs. Renewals use the control path, so a busy SQL worker cannot block them indefinitely. A shorter window must allow for network and scheduling delays, including up to five seconds before the control lane takes forwarded lease work; the CLI uses the full 60-second window.

Expired sessions are reclaimed. If a client disappears, its ordinary session expires at its fixed deadline or idle timeout; a backup session expires when its renewal window runs out. For either kind, Harbor cancels outstanding SQL, then rolls back any open transaction and reclaims the connection once execution stops. Explicitly releasing a session also rolls back an open transaction. Before reuse, Harbor replaces the engine connection, clearing temporary tables, variables, prepared statements, and connection-local settings. These remain available between requests within the same live session. One-shot requests also discard local state before the next caller; use a session whenever later statements depend on it.

One statement at a time. A second statement sent while the first is running gets a 409: a transaction is a sequence, and two of them interleaving inside one is something no client could reason about.

GET /sessions shows what is held — age, idle time, statements, whether a transaction is open — and the connection accounting behind it. Free plus live plus in-flight always equals total; balanced is that checked at the moment you asked. A pool leaks connections silently and the symptom shows up weeks later as “everything hangs”, so the arithmetic is worth being able to read.

Note that DuckDB resolves write conflicts optimistically: two transactions touching the same row do not queue, the second is refused the moment it writes. The answer is to run the transaction again, which is what rip/db does for you.

Brace expansion

A statement can carry the shell’s {a,b}, and Harbor expands it before the engine sees it. It was made for reaching into a VARIANT several fields at a time without repeating the path:

select
  id,
  raw_request.{
    requisitionNumber,
    visitDate,
    patient.{
      lastName,
      firstName,
    },
  },
from
  orders
where
  raw_request.patient.lastName ilike 'morel'
;

is what the engine runs as

select id, raw_request.requisitionNumber, raw_request.visitDate,
           raw_request.patient.lastName, raw_request.patient.firstName,
  from orders where raw_request.patient.lastName ilike 'morel';

Items are separated by commas. Whitespace or a comment alone separates them too, so the comma is never required, only clearer. A comment between items goes with the item before it, and one before the first item with the first: it comes out after each alternative that item makes, so r.{a /* the a */, b} is r.a /* the a */, r.b, a line comment keeps the line break that ends it, and an item commented out stays a comment in the statement the engine runs. A group nests. It can sit anywhere in a term — orders_{2025,2026}, {first,last}Name, r.{a,b}::VARCHAR — where a term ends at whitespace, a comment, a comma, a parenthesis or a semicolon, whitespace being whatever the engine takes for it, the vertical tab and the Unicode spaces it strips among it. A cast’s type parameters written right after its type name stay in the term, so r.{a,b}::DECIMAL(10,2) casts both, and a subscript or slice written right after a term stays in it whatever it holds, so r.{a,b}[ 1 ] and r.{a,b}[i : j] take it to both. A type the engine reads on past a space or comment, parameters or array bounds after one (::DECIMAL (10,2), ::INT []) or a type of several words (::TIMESTAMP WITH TIME ZONE, ::DOUBLE PRECISION), would reach only the last alternative, so the statement is refused with a 400 that gives the one-word spelling where there is one, or says to cast each alternative on its own. CAST(r.{a,b} AS …) puts both inside the one CAST, which the engine refuses. Two groups in one term multiply, as in a shell: r.{a,b}.{x,y} is four paths. The alternatives are joined with , , which is what a select list, a FROM list and an argument list all take.

A struct literal is DuckDB’s own use of braces, and it always carries a lone : at its top level — never the :: of a cast — so {'a': 1} is left as it came, and so is an empty {}. Strings, quoted identifiers, dollar quotes and comments are never touched: the statement is read once, as the engine reads it, and only a group in code expands. An expansion the engine would read another way than the statement it came from, as when a quote its parser misreads is repeated or an E comes to begin the word before a string and make it take backslash escapes, is refused with a 400 that says what to write instead: most often a space between the group and the quote, or a plain space for a Unicode one. So is an expansion that would surprise: a group inside a list’s brackets, [r.{a,b}], which would make a list of each alternative rather than one list of them all; an empty group before another in its term, r.{}.{a,b}, which would leave the other unexpanded; and an item cut inside the parentheses of a call or a lambda at a space or comma, r.{list_transform(l, lambda x: x), b}. A call with nothing to split, r.{f(a), b}, and a group in a call inside a list, [f(r.{a,b})], expand. A brace that never closes leaves the statement as it came; the engine then reports the syntax error at it.

The expansion happens in the server, once, for every client: the REPL, curl, a Rip app. EXPLAIN and the engine’s error messages show the expanded statement. DuckDB itself is untouched; it only ever receives plain SQL.

Groups multiply, so a short statement can stand for an enormous one: thirty two-item groups are a billion alternatives. An expansion that would write or re-read more than 16 MiB of text is refused with a 400 before anything runs.

The whole database, one call

GET /catalog answers what a client would otherwise ask in a dozen queries: every table with its columns, primary key, unique constraints, foreign keys, indexes and sequences — including whether a column is generated and its generation expression — plus its exact row count and the engine’s own CREATE TABLE rendering per table, and the database and WAL file sizes in exact bytes at the top:

{"harborVersion":"0.32.1","duckdbVersion":"v2.0.0-dev83323",
 "databaseSizeBytes":12582912,"walSizeBytes":0,
 "tables":[{"name":"orders","schema":"main","rowCount":300000,
            "columns":[{"name":"id","type":"BIGINT","notNull":true,
                        "default":null,"generated":false,
                        "generationExpression":null,"primary":true},…],
            "primaryKey":["id"],
            "ddl":"CREATE TABLE orders(id BIGINT PRIMARY KEY, …);"}, …],
 …}

Views and attached databases follow the tables:

 "views":[{"name":"sales","schema":"main",
           "columns":[{"name":"id","type":"BIGINT",…},…],
           "error":null,
           "ddl":"CREATE VIEW sales AS SELECT * FROM read_csv('/data/sales.csv');"},
          {"name":"gone","schema":"main","columns":[],
           "error":"IO Error: No files found that match the pattern \"/data/gone.csv\"",
           "ddl":"…"}],
 "attached":[{"name":"app","path":"/data/app.sqlite","type":"sqlite",
              "readOnly":false,"error":null,"tables":[…],"views":[…]}]

A view has no rowCount: counting a view over a file reads the file. Its columns are the ones it reads as when the catalog is asked, since each view is bound first (a CSV’s sample, a Parquet footer; a GeoJSON file whole, for GDAL parses it to open it). A view that does not bind — its file gone, or its extension not loaded — has no columns and the engine’s reason in error, and the rest of the document is unaffected. Views are bound on a connection of their own, never a worker’s, each within two seconds: one that takes longer (a mount that hangs, a slow URL) is reported unbound with why, and one over a pipe, a socket or a device is not bound at all, since a read of it waits on another program. A view over one file, as harbor <file> makes one, is bound again only once the file changes (its size, its modification time or its inode), so a GeoJSON file is parsed once, not at every catalog read. attached lists every database attached beside the served one, system and temp never, with its path (null in memory), its storage type, whether it is read-only, and its tables and views in the same shapes. A DuckDB catalog’s tables carry exact counts; another engine’s carry rowCount: null, since a count there reads every row. On the workspace, a file it records as attached and could not attach at its start, or whose catalog did not read once attached, is listed too, holding nothing, with why in error ("/data/app.sqlite: no such file"); error is null for a database that is attached. The engine reads every attached database’s catalog together and lists none while one does not, so an attached SQLite file whose catalog does not read fails every listing, on any server. One that does not read for good — not SQLite after all, replaced by a file that is not, or a read-only one deleted — is detached by the read that finds it and listed the same way, holding nothing, with the engine’s reason in error, while the rest of the catalog lists. The first read after it reads attaches it again under its name when the workspace attached it from its record; one a client attached is left for the client to attach again, and a client’s DETACH of its name forgets it. The workspace records either as attached. One that is only busy — held locked by a long write, harbor’s own or another program’s — stays attached, since detaching it would take it from every connection using it: the read answers with the catalog the server last read, the busy file’s entry saying why in error, and a stale field at the document’s end saying that every database in it may be stale (a document read anew has no stale), or, when the server has read none yet, a 400 that names the file. The read that meets a lock waits out the file’s own wait on it, and a read while it stays busy asks the file alone, waiting a tenth of a second. A SQLite file harbor attaches waits a second on a lock (BUSY_TIMEOUT 1000; the engine’s own default is five). A table or view another client drops, or a database it detaches, while the catalog is being read is left out; it never fails the read. An ATTACH or DETACH runs alone against every statement that reads the catalog — /catalog itself, the duckdb_… and pragma_… functions, information_schema, SHOW, DESCRIBE, the engine’s completer, and query and query_table, which run a statement or read a table named in a string — waiting for those running and holding back those that come after, since the engine brings itself down when a SQLite database is detached while a catalog read walks it. A view, macro or prepared statement whose body reads the catalog, itself or through another, reads it too: the server learns each one’s body from the engine’s catalog when it starts and after each ATTACH, IMPORT DATABASE and COPY FROM DATABASE, from each CREATE VIEW, CREATE MACRO, PREPARE and ALTER VIEW … RENAME before it runs, and from the schema.sql an IMPORT DATABASE runs, before it runs.

The document is stable — same database, same bytes — so clients can diff it. A client that only wants an inventory asks GET /catalog?style=lite and gets the versions, the sizes, {name, schema} per table and view, and each attached database with its tables and views named the same way: what exists, without counting it or describing how it is built, at a fraction of the work and bytes. An unknown style value is a loud 400; unknown parameters pass.

Get it running

One binary, ready without configuration. The client half never touches DuckDB — the engine (libduckdb) loads on demand, only when this process is the one serving a file, so the same harbor is a pure protocol client on machines that never host a database. make fetch-duckdb puts an engine into ~/.duckdb/cli/2.0.0/, one of the places harbor looks at runtime; then:

$ make fetch-duckdb            # libduckdb + duckdb CLI -> ~/.duckdb/cli/2.0.0/
$ make harbor                  # -> target/release/harbor (no engine needed to build)
$ harbor mydata.duckdb
mydata>

make bootstrap does the whole thing in one shot — fetch the engine into ~/.duckdb, then build and install harbor into ~/.local/bin. No step needs root.

make fetch-duckdb installs the engine and the duckdb CLI of one build: the DuckDB release harbor binds, DuckDB v2.0.0, taken from DuckDB’s own GitHub release (duckdb-shared-libs-<platform> and duckdb-cli-<platform>). Each archive is checked against the SHA-256 the script records for it, and each binary must say it is that release. The script refuses a library that lacks the v2 C API harbor binds, before it installs anything.

CI and the release builds fetch through the same script, and each release archive bundles the engine it was built and tested with, so a local fetch tests against the same engine.

No toolchain? One command installs the latest release — it picks the right archive for the platform, verifies its sha256 against the published checksums, and installs harbor into ~/.local/bin with libduckdb in ~/.local/lib (override with BIN=... LIB=...):

# macOS and Linux
curl -fsSL https://raw.githubusercontent.com/shreeve/duckdb-harbor/main/install.sh | bash

# Windows
irm https://raw.githubusercontent.com/shreeve/duckdb-harbor/main/install.ps1 | iex

Uninstall with ... | bash -s -- --uninstall — the binary and libduckdb go; your databases, state, and config stay. It refuses while a login item still runs that copy, since launchd or systemd would retry a missing binary until logout, and names the autostart off stop that takes each down.

Homebrew has it too, as duckdb-harbor, which installs the harbor command:

brew install shreeve/tap/duckdb-harbor

Use one or the other on a machine, not both. Each keeps its own copy and upgrades only its own, harbor update for the script’s and brew upgrade duckdb-harbor for Homebrew’s, so two copies drift apart and the version you get depends on which one PATH reaches first. Config and state are shared, in your home directory, so nothing is lost by changing over. harbor says so when it finds a second copy, names it, and gives the command that removes it.

Installed, it updates itself. harbor update runs that same installer over the binary that ran it, so a copy outside ~/.local updates in place (with sudo in front when it is system-wide), and harbor update 0.42.0 picks a release, older or newer. A running server keeps the code it started with until it is restarted, so the update ends by naming every server still on the old version and the restart that brings each forward; --restart runs them, and --check only says whether a newer release exists. Each restart runs out of reach of a hangup, so an ssh session that closes mid-update does not leave a server down, and its word is kept in ~/.local/state/harbor/runtime/log/update.log. A server that cannot come back as it was is named with its own restart instead and left running: one whose config will not load, or one started by a harbor that kept no record of its options (below).

Nothing there asks for root. ~/.local/bin is where the XDG base directory spec puts user executables; Debian and Fedora already have it on PATH, macOS does not, and the installer says so rather than putting binaries somewhere you cannot see. A system-wide install is BIN=/usr/local/bin LIB=/usr/local/lib with sudo in front of the whole command — the installer never escalates on its own. On Windows the binary lands in %LOCALAPPDATA%\Programs\harbor\bin, which the installer adds to your user PATH.

Pin a version with ... | bash -s v0.32.1, or on Windows with & ([scriptblock]::Create((irm .../install.ps1))) -Tag v0.32.1, since a parameter cannot pass through | iex. Each release ships one self-contained archive per platform (osx-arm64, linux-amd64, linux-arm64, windows-amd64, windows-arm64): harbor and the exact DuckDB shared library it was tested against. Unix archives carry bin/, lib/ and install.sh; Windows archives put duckdb.dll beside the executable and run in place.

On Windows, banners and fleet displays omit the \\?\ prefix for readability. File access, configuration, and /info keep native canonical paths, including that prefix where needed for long paths.

Linux archives are built on Ubuntu 24.04 and need glibc 2.39 or newer; on an older distribution, build from source with cargo build --release.

cmd.exe does not treat single quotes as quoting, so a curl example written for a Unix shell sends its body starting with a literal ' there. Put the JSON in a file and pass -d @query.json, which reads the same in every shell, or double-quote the body and escape the quotes inside it. In PowerShell, curl is an alias for Invoke-WebRequest; call curl.exe.

The two lifetimes

harbor <db.duckdb> — the server is everyone’s. On a terminal it is the REPL — highlighting, completion (Down on the live line or Ctrl-Space lists, Tab accepts; Up and Down are history everywhere else), the duckdb-shell dot commands. With -c or stdin it runs statements and exits; a dot command at the start of a line runs there as at the prompt, and one that fails — an unknown command, a bad argument, a .read of a file that is missing or whose statement fails — ends the script with exit 1, as a failed statement does. Either way, if nothing serves the file yet, a server is spawned behind the scenes: detached, refcounted, alive while anyone is connected. Every client holds one quiet connection for its lifetime, so a human thinking at a prompt counts as presence; when the last client leaves, the server drains, CHECKPOINTs, sweeps its socket, and exits a few seconds later. A second harbor on the same file — any spelling of the same path — joins the same server instead of reporting “database is locked”.

harbor <data file> — the workspace. A CSV, a folder of Parquet, a spreadsheet or a SQLite file opens the same way, in the workspace: a DuckDB database at ~/.local/share/harbor/workspace.duckdb (in $XDG_DATA_HOME or $HARBOR_HOME when one is set). Before the prompt, harbor makes a view over the file, named for it, or attaches the SQLite file, and says what it made:

$ harbor ~/x/sales.csv
sales: a view over ~/x/sales.csv
workspace> SELECT region, sum(amount) FROM sales GROUP BY ALL;

The file	Opens as
`.csv` `.tsv` `.txt` `.csv.gz`	a view over `read_csv`
`.parquet`, or a folder of them	a view over `read_parquet` (a folder’s `*/.parquet`, `key=value` directories as columns)
`.json` `.ndjson` `.jsonl`	a view over `read_json`
`.xlsx`	a view over `read_xlsx`, its first sheet
`.geojson` `.shp` `.gpkg`	a view over `ST_Read`, with `spatial` installed and loaded first
SQLite (`.sqlite`, `.db`, …)	attached: `ATTACH … (TYPE sqlite, BUSY_TIMEOUT 1000)`

A file is known by its first bytes where its kind has a signature (DuckDB, SQLite, Parquet, Excel’s zip, gzip, a shapefile) and by its name where it has none. A DuckDB file, a path ending .duckdb, and a path that does not exist yet open as databases, as above; a file harbor cannot tell is refused, naming what it looked at. The view reads the file where it lies, by its absolute path, so the file is never copied or changed and a 10 GB Parquet file opens at once. A name another relation or a reserved word has gets _2, _3; opening the same file again finds the view the first opening made. The excel, sqlite and spatial extensions install on first use.

Every server that opens the workspace makes it whole before it serves anyone, whichever client summoned it: it loads spatial when a view calls an ST_ function, since spatial does not load on demand (excel, json, parquet and sqlite do), and installs it in the background when it is not installed, never between the database and the socket, since an install is a download (the views that need it say why they do not read until it loads). The SQLite files it records attach through sqlite, which the engine installs on demand, so when sqlite is not installed it is installed in the background too, and those files are attached there once it loads, each listed in /catalog with why until then; and it attaches again each database file the workspace records. The record is the table _harbor.attached (name, path, type, read_only) in the workspace itself: harbor <SQLite file> writes its row, and the server rewrites it from what is attached when it stops, so a plain ATTACH or DETACH of a SQLite or DuckDB file, from any client, lasts past the server; one that ends without stopping (a crash, a kill -9) leaves the record as last written. A database --init attaches is the operator’s, attached by each start that passes it, and is never recorded; a recorded file of the same name is not attached beside it, and its row stays as it is. /catalog does not list the record. A recorded file that is gone at the start is not attached, since attaching a missing path would make an empty database there, nor is a row whose path is not absolute, which would name a file in whichever directory a server starts in; one whose catalog does not read once attached (not SQLite after all, or locked by another program) is detached again, since it would fail every catalog read of the server; the server’s log and /catalog say so, the server starts all the same, and the row stays, so the file is attached again at a later start once it reads. Deleting the row forgets it. The verbs (start, backup, …) act on databases and refuse a data file.

harbor <db.duckdb> start — the server is yours. No refcount: it lives until you stop it. At a terminal the server comes up in the background and start returns — under the database’s login item when it has one, so the session manager owns it from the first second, otherwise as a detached process that harbor <db> stop ends. Headless, meaning stdin or stdout is not a terminal, it serves in place until SIGTERM, which is the shape launchd, systemd and a spawn want; --foreground asks for that shape at a terminal, to watch a server work. SIGINT and SIGHUP stop it the same way, so a terminal or an ssh session that closes still folds the WAL, and on Windows Ctrl-C does. Every stop is clean — drain, CHECKPOINT so the next open never replays a WAL, socket swept.

A server in the background, started by hand or on use, writes its output to runtime/log/<basename>-<hash>.log in the state directory, which starts over at a start once it has passed 1 MiB; a login item’s server writes to runtime/log/<name>.log.

harbor <db.duckdb> autostart — the server is the session manager’s. harbor never becomes a supervisor; it hands exactly this start to one that already exists. On macOS it writes a LaunchAgent and loads it, on Linux a systemd user unit, enabled and started. The server comes up now, at every login, and again after a crash — never after a clean exit, so stop stays stopped until the next login and restart bounces it with a fresh read of its config. autostart off drops the login item and leaves a running server alone; autostart off stop takes both down. A login item runs a bare start, so its options are the database’s entry in config.toml, never flags. The verbs move two independent facts, the way brew services and systemctl do:

You type	Registered at login	Running now
`autostart`	yes	yes
`autostart stop`	yes	no
`stop`	unchanged	no
`restart`	unchanged	yes
`autostart off`	no	unchanged
`autostart off stop`	no	no
`detach`	no	unchanged
`detach stop`	no	no, and forgotten

Removing the login item never kills a server: autostart off and detach take the registration away and leave whatever is running to stop — though until logout the manager still restarts that server after a crash, since it holds the job it loaded.

restart brings a server back as it was. Under its login item, re-reading config.toml; otherwise in the background, with the options it was started with, from the directory it was started in (so a relative path in an --init names the same file), and with the same lifetime — unless options are typed. A server keeps those options beside its socket, in runtime/<base>-<hash>.args, readable by its user alone, and never in /info, which answers anyone who reaches its TCP door: an --init can hold a secret. Everything the start will read is read before anything stops, so a restart that cannot start refuses and leaves the server running: a config that will not load, a typed option that does not parse, or a server whose harbor kept no record of its options. That one is restarted with its options typed — the refusal names its --port — or stopped and started.

There is no registry. The socket is the runtime registration: its name is derived from the database’s canonical path (~/.local/state/harbor/runtime/<basename>-<hash>.sock), so discovery is readdir plus a GET /info to each socket — which is precisely what bare harbor prints:

$ harbor
╭───────────────╮
│ harbor 0.32.1 │
├───────────────┴────┬───────────────────────┬─────────┬───────┬─────────┬────────╮
│ DATABASE           │ URL                   │ VERSION │ PID   │ CLIENTS │ UPTIME │
├────────────────────┼───────────────────────┼─────────┼───────┼─────────┼────────┤
│ ~/Data/labs.duckdb │ http://127.0.0.1:9495 │ 0.32.1  │ 72840 │       2 │     3d │ ¹
╰────────────────────┴───────────────────────┴─────────┴───────┴─────────┴────────╯
¹ ~/.local/state/harbor/runtime/labs.duckdb-1a2b3c4d.sock

The caption is the installed CLI version. Each row’s version comes from that running server, so an installation update is visible immediately and servers that still need a restart stand out. The socket path hangs below the grid under the row’s footnote, and the URL column exists only while some server has a TCP door — an all-socket fleet keeps the five-column shape.

A running database answers to four spellings, and the last two come straight off this list:

$ harbor ~/.local/state/harbor/runtime/labs.duckdb-1a2b3c4d.sock   # its socket
$ harbor http://127.0.0.1:9495                                     # its URL
$ harbor labs                                                      # its name
$ harbor 1                                                         # its footnote

A name or a footnote reaches what is listed — a running server by the name it declares, an attached database by its config key — and a bare word never becomes a file: a folder of Parquet in the current directory is harbor ./lake, and harbor lake, naming nothing listed, says so. A name two running databases share is refused as ambiguous rather than guessed. The verbs take the same spellings: harbor labs stop, harbor 3 start; opening a stopped one summons it the way its path would.

A socket nothing answers on is a leftover from a kill -9, and the list unlinks it. Set HARBOR_HOME (absolute path) to collapse configuration, runtime state — sockets, logs, and history — and the workspace into one directory; the test suites use it to keep their servers out of the real fleet view. A relative or empty HARBOR_HOME is an error, never a quiet fall back to the real config and fleet.

config.toml

~/.config/harbor/config.toml (in $XDG_CONFIG_HOME/harbor when that is set, and in $HARBOR_HOME when that is) is optional: without it every default stands. It holds one [connection.<name>] entry per database, and DuckTable reads the same file. An entry with path is a database on this machine, which harbor can start and which harbor <name> reaches; an entry with url is a server elsewhere, for DuckTable. attach and detach add and remove entries, and an entry may be written by hand:

[connection.medlabs]
path = "~/db/medlabs.duckdb"
memory-limit = "8GB"
statement-timeout = "5m"
init = ["INSTALL httpfs", "LOAD httpfs"]

[connection.medlabs.settings]
enable_progress_bar = true
default_null_order = "NULLS LAST"

[connection.warehouse]
url = "http://warehouse.example.com:9495"

The entry whose path names the file being started, in any spelling, supplies its settings to every start of it: a start typed by hand, a summon, and the login item. Each key is the start option of the same name, and an option typed on the command line overrides the key:

Key	Value	Default
`path`	the database file; `~` expands	—
`memory-limit`	DuckDB’s `memory_limit`, as `"8GB"`	`"2GB"`
`threads`	DuckDB’s threads	DuckDB’s own
`workers`	statements run at once	`6`
`statement-timeout`	the ceiling on every statement: `"90s"`, `"10m"`, `"2h"`	none
`max-temp-size`	the cap on disk spill, as `"10GB"`	DuckDB’s own
`block-size`	the block size of a file this start creates: `"16k"` to `"256k"`	DuckDB’s own
`sealed`	`true`: no host file access, no community extensions	off
`unsigned`	`true`: unsigned extensions load	off
`log`	`true`: one line per request in the server’s log	off
`init`	SQL run at boot, before serving, in order	none
`port`	a loopback TCP port beside the socket, for a start that is not a summon	none

[connection.<name>.settings] passes any DuckDB setting through: each key = value runs as SET key = value after init, so it can tune an extension init just loaded, in key order, a string quoted and a number or boolean bare. A typed --init runs after both. default_block_size is chosen when a file is created, where no SET reaches, so it is skipped with a word that names block-size.

A config.toml that will not load stops every start and summon, since it may hold sealed, a statement ceiling or the boot SQL, and a server that came up without them would look configured while being open. An unknown key, a [defaults] section, a value of the wrong type, two entry names that are one name once normalized, and a file or a directory holding it that others can write or another user owns are all refused, in one line that names the file and, for what is written in it, the line and column. A server already running is not touched.

Output modes

--mode <m> picks how results are rendered, and .mode <m> changes it at the prompt. The display modes — duckbox (the default at a terminal), duckboxy (the same without the type row), markdown, line and list — are for eyes. The data modes — csv, json and jsonlines (--json is shorthand) — are for programs, and boxed output on a pipe gets a hint to pick one. trash discards results and reports only errors. csv writes a NULL as an empty field, an empty string as "" and the word NULL as itself: three values, as DuckDB’s COPY writes them. .nullvalue sets what a NULL shows as in the display modes. Output into a pipe whose reader has gone, as in harbor db.duckdb < script.sql | head, ends the script there: no statement after the one whose output failed runs.

The modes differ in how they treat a VARIANT or JSON cell, which the wire carries as JSON text. A display mode shows a VARIANT string bare — a value read by path, raw_request.requisitionNumber, shows as L2605106156, the way a VARCHAR always has — while a number, a boolean, a null, an object or an array shows as its JSON; a JSON column shows its text with the quotes, as DuckDB’s own table does. Cast a path to JSON to see the quotes in a table. csv carries the JSON text CSV-escaped, so 42 and "42" stay apart for the program on the other end. json and jsonlines splice the cell in as the JSON it is — 43, not "43"; a document, not a string holding one — for both column types; a SQL NULL and a JSON null are both null there. The text is checked before it is spliced, so a record is always well-formed: NaN and Infinity, which the engine writes bare and JSON cannot say, stay the strings "NaN" and "Infinity". The check does not recurse and has no depth limit, so a document is spliced whole however deep it nests, and a consumer whose parser stops at some depth meets that in its own parser. A pretty-printed JSON column keeps its newlines in the engine; jsonlines is one record per line, so between tokens they become spaces. JSON nested inside a struct, list or map column is a string, as the wire holds it. The wire itself is untouched by any mode.

Backup and restore

A .duckdb file is only as portable as the engine that wrote it, so copying one is a snapshot, not a backup. backup writes the durable thing instead — schema.sql, load.sql, one tab-separated file per table, and after.sql when a table has a VARIANT column (below):

$ harbor medlabs backup
harbor: backed up 14 tables to ~/db/medlabs.backups/20260909044118 (612K)
harbor: restore it with — harbor <new.duckdb> restore ~/db/medlabs.backups/20260909044118

$ harbor fresh.duckdb restore ~/db/medlabs.backups/20260909044118 --block-size 64k
harbor: restored 14 tables into fresh.duckdb (740K)

Tab-separated rather than parquet, deliberately: both round-trip exactly and both come to about the same size, so the tie goes to what you can do with the artifact six months from now — grep it, diff two of them, read one in an editor, keep one in a repo. Sequences come back at their current value and indexes come back with them.

Three values and one escape cover the whole dialect: a bare NULL is a real null, a quoted "NULL" is the string, and an empty field is an empty string. Quotes are always allowed and rarely required — a value containing a tab, a newline or a quote is wrapped and a " inside doubles to "" (RFC 4180, which every CSV reader already knows), and everything else is written plain. Nothing is backslash-escaped, so a backslash in the data is only ever a backslash. The reader takes a bare field and a written "" the same way, so a backup stays editable by hand.

One file per backup may look different, and it is a row of data rather than a matter of taste. A one-column table holding an empty string would write an empty LINE, and every CSV reader skips those — that row would not come back and nothing would say so. FORCE_QUOTE is the only lever DuckDB offers and it takes a column list rather than a predicate, so it is spent per file: a table whose export contains a blank record is written again with every value quoted, said out loud, and no other file pays for it. One thing worth knowing before you reach for wc -l: a value holding a newline spans physical lines, so a row is a record, not always a line.

A VARIANT column travels as JSON. Its display rendering cannot come back — the number 42 and the string "42" both print as 42, and the reader hands every cell back as a string — but JSON tells them apart, and a value that entered as JSON returns exactly as it entered, every inner type and every nesting. So the column is written as JSON text (42 for the number, "42" for the string, still greppable) and decoded on restore. For a stock duckdb the decode is an UPDATE, and DuckDB’s IMPORT DATABASE takes nothing but COPY, so it lives in after.sql beside load.sql: importing the directory by hand gives the JSON text, and running the second file turns it back into documents. harbor restore needs no second pass. It runs schema.sql and each COPY of load.sql itself, which is all IMPORT DATABASE does, and loads a table with VARIANT columns as documents: the table’s own COPY, options and all, fills a staging table that holds those columns as text, one INSERT moves the rows across with the text cast through JSON, and the staging table is dropped. The table never holds the JSON text, so a CHECK on a VARIANT column is shown documents and nothing else. What JSON has no word for — a DATE, a DECIMAL, a BLOB, a TIMESTAMP put inside a variant from SQL — comes back as JSON’s nearest type, and the backup says so, once per column:

$ harbor mydata.duckdb backup
harbor: events."payload" holds DATE — written as JSON, which has no such type; --format parquet keeps it
harbor: backed up 14 tables to ~/db/mydata.backups/20260909051315 (612K)

An integer stored from SQL as a narrow type comes back as JSON’s wide one — the same value, a different width label — which is what “written as JSON” means and is not reported. --strict refuses instead of writing the note.

A GENERATED column is computed, not carried. schema.sql holds its expression and no data file holds its values, in either format, since no COPY takes one back. The restored table computes it from the rows it is given, a column that reads a VARIANT document (doc['a']::VARCHAR) from the restored document. A text file that does hold the generated columns says so in its header, and restores the same way with those fields left behind.

Neither format holds every shape, and that is why --format exists:

	tsv	parquet
`UNION`	loses its tag — the restore refuses	✅
`VARIANT` nested in a `STRUCT`, `LIST` or `MAP`	would come back retyped — refused	no writer for it — refused
`VARIANT` holding a `DATE`, `DECIMAL`, `BLOB`, …	JSON’s nearest type, said out loud	✅
`VARIANT` holding an `INTERVAL`, `BIGNUM`, `BIT`, …	JSON’s nearest type, said out loud	refused outright
negative `INTERVAL`	✅	refused outright
`TIMETZ` with an offset	✅	normalised to UTC, silently

A table the chosen format cannot carry is written in the other format when that format can preserve all its types, and the change is reported:

$ harbor mydata.duckdb backup
harbor: settings is parquet, not text — a UNION loses its tag
harbor: backed up 14 tables to ~/db/mydata.backups/20260909051315 (612K)

load.sql names the format per table, so the directory stays self-describing and the choice is visible in ls. --format parquet asks for one format throughout — with the same swap running the other way for a TIMETZ column, whose table is then text in every respect, a VARIANT beside it included — and --strict refuses rather than swapping, for a backup that has to be one format or nothing. What no mode will do is write something that will not come back without saying so: a negative interval under --format parquet is an error. A table combining UNION with TIMETZ is refused because neither whole-table format preserves it. So is a table holding a VARIANT[], a STRUCT(v VARIANT) or a MAP(VARCHAR, VARIANT), in either format: text reaches only a plain VARIANT column and parquet has no writer for one below the root. The whole backup stops there and no directory is written —

$ harbor mydata.duckdb backup
harbor: settings cannot round-trip in either backup format: parquet has no writer for a VARIANT nested inside another type

— and under --format parquet the refusal is the engine’s own Not implemented Error. Keep a VARIANT a plain column, or hold the nested shape inside one VARIANT document, and the table backs up.

The whole of this is a test suite rather than a claim: test/scripts/roundtrip.py backs up and restores every type in the shared corpus, a schema of constraints and indexes and views and sequences, the strings that attack the format, and a seeded fuzz of random tables — then attaches both databases and asks DuckDB whether anything differs.

Backup holds one transaction for the initial export and every rewrite pass, so concurrent committed table changes cannot mix snapshots. It needs a free session connection. The CLI renews a 60-second lease every 20 seconds during both SQL and file inspection, so there is no fixed total backup lifetime. If the client disappears, the lease expires 60 seconds after its last successful renewal (or creation if never renewed). Harbor cancels active work, then rolls back and reclaims the connection once execution stops. A renewal failure or failed pass aborts the backup and removes its incomplete directory; it never resumes on a newer snapshot. The operator’s --statement-timeout still limits each SQL statement; configure it to accommodate the longest export pass. A server that grants no backup session is refused with a message that says to upgrade it. Exported data is scanned with a bounded buffer. Sequence counters are not transactional in DuckDB; quiesce sequence users when their exact position must correspond to the exported rows.

The directory is self-contained. Each COPY in load.sql names its file and nothing more, so the backup can be moved, renamed, copied to another machine or committed to a repo and still restore — an absolute path would have nailed it to the machine that wrote it.

Each appears whole or not at all. A backup is written under <dir>.partial-<pid>, a directory only its owner can enter, and renamed into place as its last step, and a restore builds <db>.restoring-<pid> and moves it into place once its server has folded the WAL in; a Ctrl-C or a failure takes the partial one back out, and a kill leaves only that name, which restore refuses. restore also refuses a directory EXPORT DATABASE wrote itself: its load.sql reads a quoted "NULL" back as a null.

restore always builds a new file and refuses one that exists. A restore that can overwrite is a restore that can be run at the wrong moment and take the very thing it was meant to protect; moving the restored file into place is a human’s job, and a deliberate one. It is also the only moment --block-size can be applied, since DuckDB fixes that when a file is created and offers no ALTER — which makes backup then restore the way to change it.

Sockets and TCP

The Unix socket is always there, protected by the 0700 runtime directory. --port adds loopback TCP beside the socket, never in place of it, so the server stays visible to the fleet (the list, DuckTable, join-before-summon) like any other. Both doors are machine-local:

$ harbor mydata.duckdb start --port 9495
$ harbor http://127.0.0.1:9495 -c "SELECT count(*) FROM orders"

An explicit start can also take its port from the database’s port key in config.toml; a summon stays on the Unix socket, so opening a database never silently adds a TCP listener. TCP binds IPv4 loopback only: 127.0.0.1. --port 0 takes a free port, and /info and the list show the one bound. harbor http://127.0.0.1:9495 stop stops a server by its URL — drained and checkpointed, as over its socket — which is how a server on Windows is stopped from anywhere but its own terminal.

Remote access is Caddy’s job at the edge (TLS and access policy); harbor itself speaks plain HTTP over a unix socket or a loopback TCP port. A human reaches a remote host over ssh and uses the socket.

Ordinary DuckDB SQL can read host files or load extensions. For a server whose callers should not receive those capabilities, --sealed disables host-file access and community extensions. Once initialization has run, Harbor locks its memory, thread and spill settings inside DuckDB, and the setting that unlocks them: the engine refuses every statement that would change one, with its own message naming the setting, and inside a transaction that refusal aborts the transaction as any error does. Other settings registered at startup remain changeable unless --init imposed a stricter lock. Load extensions that need configurable settings during initialization; settings registered later are outside the allowed list.

--max-temp-size bounds disk spill, and --statement-timeout places the hard statement ceiling described above. These are independent of Caddy’s transport and HTTP policy.

Request logging

--log writes one line per HTTP request to stderr:

harbor: 2026-08-12T04:31:07Z 127.0.0.1 POST /sql 200 12ms

Timestamp, peer, method, path, status, duration — measured to the last body byte rather than the first, so a slow query and a slow client both show — and, for a status of 400 or more, the error code. Off by default. The SQL itself is never logged: it arrives in the request body, it can be megabytes, and on this endpoint it is as likely to hold customer data as the tables it reads.

stderr, not stdout, so it stays clear of anything a client reads. Send it wherever the log belongs — 2>>/var/log/harbor.log, a pipe, or a supervisor’s collector. There is no --log FILE: rotation and permissions are the shell’s job, and it does them better than harbor would.

Any language

There is nothing to install on the client side. Shell:

$ curl -sN 127.0.0.1:9495/sql -H 'Content-Type: application/json' \
       -d '{"sql":"SELECT count(*) FROM orders"}'

Python, standard library only — NDJSON means one message per line, so the response reads as it arrives:

import http.client, json

conn = http.client.HTTPConnection("127.0.0.1", 9495)
conn.request("POST", "/sql", json.dumps({"sql": "SELECT id, total FROM orders"}),
             {"Content-Type": "application/json"})

for line in conn.getresponse():
    msg = json.loads(line)
    if msg["type"] == "row":
        print(msg["values"])

JavaScript, with fetch — and params, which is how values are passed:

const res = await fetch("http://127.0.0.1:9495/sql", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ sql: "SELECT id, total FROM orders WHERE id > ?",
                         params: [100] }),
});

const decoder = new TextDecoder();
let pending = "";
for await (const chunk of res.body) {
  pending += decoder.decode(chunk, { stream: true });
  const lines = pending.split("\n");
  pending = lines.pop();
  for (const line of lines) {
    if (!line.trim()) continue;
    const msg = JSON.parse(line);
    if (msg.type === "row") console.log(msg.values);
  }
}
pending += decoder.decode();
if (pending.trim()) {
  const msg = JSON.parse(pending);
  if (msg.type === "row") console.log(msg.values);
}

Performance

DuckDB answers the query; DuckDB Harbor’s job is to stay out of the way. test/scripts/bench.py measures how well it does. It serves a fresh database, sends each query shape over one keep-alive connection, round after round, and reports the server’s own timeMs beside the wall time the client saw. The shapes are chosen because each fails differently:

Shape	Query	What it catches
point	`SELECT 42`	the round trip, with nothing to encode
ints	5,000,000 integers	the integer encoder
strings	1,000,000 strings	the JSON escape scan
mixed	2,000,000 rows of three columns	per-row framing
temporal	1,000,000 timestamps	date and time formatting
heavy	`count(DISTINCT i)` over 100,000,000	the engine’s own work, one row out

make bench runs it against target/release/harbor, and make bench BENCH_ARGS="./harbor-a ./harbor-b" compares two builds. Their servers run side by side and each round rotates which goes first, so whatever else the machine is doing lands on both columns rather than one. A spread past 2× between a shape’s fastest and slowest round is reported as noise, and such numbers are re-earned on quiet hardware rather than quoted. Measure the build and the engine you deploy: an engine moves these numbers as much as harbor does.

Streaming matters more than the rate for large results. A 300,000-row result starts arriving before the query has finished running, because nothing is buffered: a client can start work on row one while the server is still producing row 300,000. (Whether the query materialises is DuckDB’s business: ORDER BY, hash aggregates and joins all build state first.)

A repeated statement skips DuckDB’s parse, the largest engine cost of a small statement: each connection keeps the parsed form of the SQL texts it has run — at most 64 texts and 1 MiB of SQL, no single statement over 64 KiB — and clears it when the connection is replaced. A cached statement is the parser’s output, not a plan, so each execution binds against the current catalog. The limits bound the retained SQL text, not the engine’s memory for the parsed statements.

Many connections, few queries: DuckDB Harbor accepts many concurrent connections and executes a small, bounded number of statements — six by default, settable with --workers. DuckDB parallelises a single query across every core, so running hundreds at once produces thrashing, not throughput. A request normally waits for a worker. If every worker has been inside a statement for at least 250 ms, the dedicated probe lane keeps control routes responsive, relays a session’s statement to the session’s own connection, and sheds other /sql and /catalog work with a retryable 503 instead of hiding an unbounded queue behind saturated analytics.

Why it looks like this

Plain HTTP, on purpose. It binds loopback and speaks HTTP, not HTTPS. TLS belongs at the edge, where certificates, renewal, and HTTP/2 and /3 are already solved by software that does nothing else. Put Caddy or nginx in front and terminate there.

One statement per request. A request has one answer — one schema, one stream of rows, one end — so it carries one statement. The engine’s own parser counts them in the text it would run, before anything runs, and a second is a 400, exactly one SQL statement is allowed per request. A comment is not a statement, so SELECT 1; -- note runs.

Values travel as params. Each ? or $1 takes one element of the params array. A string binds as text, a boolean as a boolean, a whole number as a 64-bit integer, and a number with a fraction or an exponent as the double its text names. A whole number past what 64 bits hold is a 400 that says to send it as a string and cast it (?::HUGEINT), since as a double it would be a different number. An object or an array is a document (below). A null is bound untyped, as DuckDB’s own EXECUTE p(NULL) binds it: the statement types it, so coalesce(?, 'x') works, and where nothing does, as in CREATE TABLE t AS SELECT ?, the column takes the type a NULL literal gets, which holds nothing else. Cast it there (?::INTEGER).

Types survive the trip. Every column carries its duckdbType, plus width and scale for DECIMAL and nested child/fields for LIST and STRUCT, so a typed client can reconstruct exactly what DuckDB had rather than a lossy JSON approximation. Values JSON cannot hold exactly are quoted rather than emitted as bare numbers, so an integer past 2^53 does not silently reprecision in a JavaScript client. Dates and timestamps are ISO 8601. A year before year 0 takes the expanded form, a minus and six digits (-000043-03-15 is 44 BC), which a JavaScript Date reads and DuckDB reads back; a Date reads -0043-03-15 as the year 2043. A year past 9999 is its bare digits (10000-01-01), as DuckDB writes it, since DuckDB refuses the +010000 form. Where something genuinely cannot survive, the schema says so with "lossless": false instead of returning a plausible wrong answer. A list, array, struct, map or union is lossless only when every member is, so the column’s own entry answers for the whole of it: a GEOMETRY[], a STRUCT(v VARIANT) or a MAP(VARCHAR, VARIANT) reads "lossless": false, and so does the child, the field or the valueType that loses, which says why in its encoding.

Where it fits

DuckDB’s ecosystem already covers DuckDB talking to DuckDB. DuckDB Harbor covers everyone else.

Serves	Client needs
`quack` — other DuckDB instances	DuckDB
`harbor` — everything else	`curl`

quack is a DuckDB extension; harbor is a standalone server. It can still load an extension into its own database with harbor db.duckdb start --unsigned --init 'LOAD <ext>', so one process can answer HTTP clients and other DuckDB instances over one file at once. Harbor ships no extension of its own — whatever LOAD resolves by name in ~/.duckdb is what it gets, matching that to the loaded engine is the operator’s call, and Harbor does not patch extension source while loading it. For a desktop face on Harbor servers, DuckTable is the native client, developed in this repository beside harbor.

Known limitations

VARIANT arrives as JSON text, GEOMETRY as display text. Neither has a committed vector layout in the v2 C API. A VARIANT column is cast to JSON by the engine on the way out, a chunk at a time, so 42 and '42' stay apart and a document that entered as JSON leaves as the same document, each object’s keys in the sorted order the engine stores them in; the column says "lossless": false, "encoding": "json" because JSON has no DATE or TIMESTAMP of its own (those arrive as strings). A GEOMETRY goes out as the engine’s own text rendering under "encoding": "varchar-cast", a cell at a time. A cell that cannot be rendered fails the stream with an error rather than arriving as null.

A VARIANT is JSON at every edge, and not quite JSON inside. A column that always enters as JSON can be stored as VARIANT for the engine’s typed field access, and still be handled as JSON by everything outside: harbor delivers it as JSON text, and harbor backup writes it as JSON text. The trip JSON → VARIANT → JSON keeps what real documents carry — any string, any Unicode, 1 versus 1.0, "42" versus 42, integers up to 64 bits, nested nulls — and gives each object’s keys back in sorted order, with these known edges, all measured on the engine:

Text written into a VARIANT needs ::JSON. A plain string written into a VARIANT column, whether a literal, an UPDATE, or a string parameter, is stored as a string, not parsed: '{"a":1}' lands as the text {"a":1}, a client gets "{\"a\":1}" back, and every path into it is NULL. Write '{"a":1}'::JSON, or $1::JSON for a parameter, and it lands as an object. '…'::VARIANT does not parse either. The same applies to a column conversion: use ALTER TABLE t ALTER COLUMN c SET DATA TYPE VARIANT USING c::JSON::VARIANT. harbor’s object and array parameters, Rip’s ORM and DuckTable’s editor all write a document as a document, so the mistake is one hand-written SQL makes.
A CHECK on a VARIANT column restores. CHECK (v IS NULL OR variant_typeof(v) LIKE 'OBJECT%') does refuse the unparsed string at write time, and with it every array, every scalar and every JSON string, which are documents too, so it is not a general guard. harbor backup writes such a table as text like any other and harbor restore brings it back, SQL NULL included, because it loads documents as documents and the constraint is never shown the JSON text they travelled as. A stock duckdb cannot restore that table from the same directory: IMPORT DATABASE lands each cell as a VARIANT string for after.sql to decode, and the constraint refuses the strings first, with Constraint Error: CHECK constraint failed. --format parquet carries the column as itself, and stock duckdb imports that directory.
An object or array parameter is a document. "params": [{"a":1}] aimed at a VARIANT — a column in SET or VALUES, a comparison against one, a coalesce with one — is bound as the document, so SET doc = ? stores an object and WHERE doc = ? finds one. harbor asks the engine what each parameter expects, once, and only for a request that carries an object or an array. Aimed anywhere else it is its JSON text, a VARCHAR, for the statement to cast: an untyped slot (SELECT ?, INSERT … SELECT ?), a VARCHAR column, a $1 used against two types, a VARIANT[] or a STRUCT holding one. A string parameter is a string wherever it goes, whatever it spells: a parameter that looks like JSON is data, as one that looks like SQL is. A client holding JSON text — a grid cell, a file — says so with ?::JSON. An object or array parameter nests at most 100 levels, its own levels counted: {} is one and [[1]] is two. One level more is a 400 bad_request that says a document param nests at most 100 levels, and nothing is bound. Rip’s ORM and DuckTable’s editor keep the same number, so a document is refused at the same depth whichever layer meets it first. A string parameter is never inspected: what it spells is data, and one cast through ?::JSON nests as deep as the engine reads.
Integers beyond 64 bits become doubles and lose digits past the 17th; numbers past the range of a double come back as Infinity, which is not JSON. Integers within INT64/UINT64 are exact.
A top-level JSON null is SQL NULL once stored; nested nulls survive.
Duplicate keys keep the last value, and once a table is large enough for the engine to shred the column into typed paths, keys may come back sorted. The document is the same; its text is not. Anything that hashes or signs the body text must keep the original text.
Normalization, the same as DuckDB’s json(): whitespace and \u escapes are dropped, -0 is 0, 1.50 is 1.5, 1e10 is 10000000000.0. An object or array parameter is read by harbor’s request parser first, and two numbers differ there: -0 is stored as the DOUBLE -0.0, where the ?::JSON text path stores the integer 0, and 1e400 fails the request with a 400, number out of range, where the text path stores Infinity.
Fields are typed. v.status = 500 finds rows; v.status = '500' finds none, without an error. v.tests.price through an array is NULL, not an error.
v::VARCHAR is display text, not JSON, so v->'a', json_*(v), LIKE on the column and COPY … TO 'x.csv' all see {'a': 1}; use v.a, v['a'], or v::JSON->'a', and cast to ::JSON before a text export.
Delivery casts one cell at a time. The engine’s C API has no batch cast, so harbor delivers a VARIANT column at roughly 20,000 rows a second against 400,000 for the same documents as VARCHAR, measured on 1.5 KB documents. SELECT v::JSON in the query is vectorized and delivers at 130,000. Fetching one document, or a hundred, does not notice.

Need more fidelity than that? Back the table up with --format parquet, which keeps the VARIANT as itself.

Bodies are capped at 8 MiB, declared or delivered; over that is a 413. There is no rate limiting and no CORS. A web page cannot reach the TCP listener: a request carrying an Origin header, or a Host that is a hostname rather than localhost or an IP address, is refused with 403 forbidden before anything runs. That shuts out a page on the same machine posting SQL to loopback, and DNS rebinding reading the answer. Harbor’s own clients, Rip and curl send no Origin, and their Host is whatever address they were given: http://127.0.0.1:9495 or http://localhost:9495 passes, including through an SSH tunnel, but a name that only /etc/hosts maps to loopback does not. A browser client belongs behind an edge proxy that enforces its own policy, drops Origin, and sends the upstream’s address as Host (Caddy: header_up -Origin and header_up Host {upstream_hostport}). The unix socket is out of a browser’s reach and skips the check. Request logging is available with --log, off by default.

Windows serves over loopback TCP only. Unix sockets — and with them spawn-on-use, the list and joining a server by its file or name — are a unix feature. On Windows, serving is explicit (harbor <db> start --port <p>), a bare harbor says so instead of listing the server, and a client reaches it by URL: harbor http://127.0.0.1:<p>. Everything past the connection is the same everywhere.

The engine is the loaded libduckdb, not the binary. Nothing is linked: harbor loads the engine on demand, from the first of these that loads and serves the v2 C API:

HARBOR_LIBDUCKDB, when it is set — that file and no other, or the start fails naming it;
../lib beside the binary, then the binary’s own directory — for a symlinked binary on macOS, beside the file it links to first, then beside the link;
~/.local/lib;
~/.duckdb/cli/latest, then the newest ~/.duckdb/cli/<version>, compared number by number — DuckDB’s own world, disposable and refetchable;
on Linux, the dynamic loader’s own search.

A library without the v2 C API is passed over for the next, and the working directory is never searched. Harbor binds DuckDB’s v2 C API, so DuckDB 2.0 is the engine floor. CI runs the suite, and the release archives are built, against the DuckDB release scripts/fetch-duckdb.sh pins. Treat that as tested compatibility, not a promise that an arbitrary future DuckDB ABI will work. Your database files need no such care: a file created by a 1.5-era DuckDB opens as-is, because 2.0’s storage layer reads it. A machine with no engine at all still runs the client half; only serving needs the library, and the error says exactly where it looked.

Working on it

Building is only needed to change it. The workspace has five first-party crates:

harbor — the server engine, the client (src/repl/, which never touches DuckDB), and the CLI, all in one binary;
harbor-common — paths, names, permissions, durations: the vocabulary shared with DuckTable so the two cannot drift;
wire — protocol request and response types consumed by the client half;
harbor-http — the client’s transport, sessions, keep-alive, summoning and server discovery, used by the REPL and DuckTable; and
justhttp — Harbor’s small synchronous HTTP/1.1 server over TCP and Unix sockets.

The server implements its protocol shapes directly rather than depending on wire, so a wire change needs tests on both sides; drift is not a Rust compile error. Nothing links libduckdb — the engine loads on demand — so no DuckDB source tree, library, or header is required to build: make harbor works on a bare machine, and make fetch-duckdb fetches the duckdb CLI plus a library (under Get it running). The crate ships pregenerated bindings, so there is no bindgen.

make unit runs the Rust tests and make test runs them with the integration suites (SUITES="spec fuzz" picks some), building the sample.duckdb fixture first when it is missing. The suites use independent oracles where answers need comparison — values read from the database file before the server takes the lock, and Python’s own datetime and base64 for fuzzed values. An oracle that shares an implementation with the thing it checks confirms only that the code is self-consistent.

Status

Pre-1.0, with protocol v1 held compatible: a release adds to the wire and never removes or renames (see PRODUCT.md). Deploy remote TCP behind Caddy, which owns TLS and edge request policy; Harbor independently owns SQL statement deadlines.

License

MIT.

This site is open source. Improve this page.