Connections

A Connection copies tables from somewhere else into a DuckHouse database and keeps them current on a schedule. There is nothing to deploy: the sync runs on our side, and the tables simply appear.

Setup steps, caveats and the full list of tables for each of these are in the source setup guides.

Add a Connection

1. Open a running database in the dashboard and find Connections.
2. Choose a source and fill in its details. The form tests the connection before saving, and tells you what is wrong if it fails.
3. Choose which tables to sync, and how often.
4. The first sync starts straight away. Sync now runs one on demand at any time.

A database can have one Connection per source: one Stripe, one Shopify, one GitHub, and so on. A second one would only copy the same account again. The exceptions are PostgreSQL, MySQL, DuckDB, SFTP and Google Sheets, where another database, folder or spreadsheet is the normal thing to want; add as many of those as you like. To sync a second account of a single-Connection source, give it its own database.

A large first load is done in stages: a sync works for a while, saves its place, and carries on in the next run. Tables fill in progressively rather than appearing all at once.

Where the data lands

Each Connection writes into its own schema, so sources never collide: stripe, shopify, github and so on, the name of the source database for Postgres and MySQL, and duckdb for DuckDB. You can change it when you create the Connection.

FROM wh.query('
  SELECT status, count(*) AS invoices, sum(amount_due) / 100 AS amount
  FROM stripe.invoices
  GROUP BY status
');

The REST API, MCP and the dashboard's query console run on the server already, so schema names work there as written. More in Querying and loading data.

PostgreSQL and MySQL

Prepare the source

• Create a user for DuckHouse that can only read: SELECT on the tables you want, nothing more.
• Allow connections from the internet over TLS. TLS is required by default.
• If you have a read replica, point the Connection at it, so that syncs never compete with production traffic.

CREATE ROLE duckhouse_reader LOGIN PASSWORD 'choose-a-long-one';
GRANT USAGE ON SCHEMA public TO duckhouse_reader;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO duckhouse_reader;

How tables are kept current

DuckHouse picks a suitable column automatically, usually updated_at, and you can choose a different one per table. If a large table has no such column, adding one to the source is the single most effective thing you can do for sync speed.

Rows deleted in the source are removed from the copy as well. Detecting them costs one scan of each table's primary keys per sync.

Choosing tables and columns

• Pick tables when you create the Connection, and change them later with Edit tables on the Connection. New tables that appear in the source are synced unless you turn them off.
• Turning a table off stops syncing it. The copy already in your database stays until you drop it.
• Exclude columns that should never leave the source, such as password hashes, API tokens or personal details. Excluded columns are left out of the query sent to the source, so their values never reach DuckHouse. If a column was already synced, excluding it drops it from the copy on the next sync. Backups taken before that still hold it.
• Primary key columns, and the column used for incremental syncs, cannot be excluded.

DuckDB

Copies tables from another DuckDB over Quack, DuckDB's own client-server protocol. The source can be another DuckHouse database, or your own DuckDB running quack_serve. Your database connects to the source and reads it directly.

• For a DuckHouse database, paste its address from its page (quack:…:443) into Host, and create a token just for this Connection so you can revoke it on its own.
• Quack tokens can write as well as read. DuckHouse only sends SELECT statements to the source, but the token itself is not limited to them.
• Tables in every schema are copied, not just main; views are not. A table in sales lands as sales_orders. DuckHouse's own _duckhouse schema is skipped.
• Tables are kept current the same way as PostgreSQL and MySQL, above. DuckDB tables often have no primary key, and those are copied in full on every sync; declare a PRIMARY KEY on large tables so they can sync incrementally.
• Column types arrive as they are at the source, including lists and timestamps with time zone.
• TLS is on by default, which is what a DuckHouse database expects. Turn it off only for a quack_serve with nothing providing TLS in front of it; the token then travels unencrypted.

Stripe

Create a restricted key

1. In Stripe, open Developers → API keys → Create restricted key.
2. Give it Read access to Customers, Products, Prices, Subscriptions, Invoices, Charges, PaymentIntents, Disputes, Payouts, Balance transaction sources and Events.
3. Paste the key, which starts with rk_live_, into the form. DuckHouse never needs write access.

A full secret key (sk_live_) also works, but it can move money, and the sync only reads. Prefer a restricted key.

Organization keys

If your key starts with sk_org_, it belongs to a Stripe organization and can reach several accounts, so Stripe needs to be told which one. Fill in the Account ID field with the account to sync (acct_…). A database holds one Stripe Connection, so to sync several accounts, give each its own database. If you only have one account, a restricted key created inside it is simpler.

What the tables look like

There is one table per Stripe object type, with the commonly used fields as columns. Three extra columns appear on every table:

-- A field that has no column of its own
FROM wh.query('
  SELECT id, _raw->>''$.metadata.plan'' AS plan
  FROM stripe.customers
  WHERE NOT _deleted
');

After the first load, changes arrive through Stripe's event stream, so one request covers every object type. That keeps syncs well inside Stripe's rate limits, alongside your live traffic.

Pausing, editing and removing

• A paused Connection keeps its tables and its place, and resumes where it stopped.
• Deleting a Connection stops the syncing. The tables it created stay in your database until you drop them.
• Each Connection keeps a history of its runs, with rows per table and any error, on the database's page.

How credentials are handled

Source passwords and API keys are encrypted before they are stored, each Connection under its own key, and are never shown again, in the dashboard or through the API. Only the sync process decrypts them, at the moment it connects.