Architecture

ServiceLabs is a gateway, job system and console in front of the utility functions. The utilities themselves keep running where they always did; everything around them — auth, jobs, storage, metrics, the UI — lives here.

Components

                         Cloudflare (TLS, edge)
                                  │
                              host nginx
                                  │  127.0.0.1:8097
                ┌─────────────────▼──────────────────┐
 caller ──────▶ │ api  (Hono)                         │
 Bearer du_…    │  /v1           machine API          │
                │  /console/api  console API (cookie) │
 browser ─────▶ │  /             console SPA          │
                │  /docs/        these docs           │
                └──┬───────────────┬──────────────────┘
       sync: runs  │               │ async: enqueue
       in-process  │               ▼
                   │     Redis (BullMQ) ──▶ worker ×2 ──┐
                   │                                    │
                   ▼                                    ▼
          ┌─────────────────── one execution path ───────────────────┐
          │ validate → invoke utility (AWS Lambda Invoke, IAM)       │
          │         → parse its answer → copy outputs S3 → R2        │
          │         → finish job → signed callback (async)           │
          └──────────────────────────────────────────────────────────┘
                   │                              │
               Postgres                       Cloudflare R2
     jobs · job_events · keys · users     <utility>/<date>/<job>/<file>
     usage_5m rollups (histograms)        public or presigned links

Piece	What it does
api	Serves /v1 (API keys), /console/api (sessions), the console SPA and these docs. Runs sync jobs in-process. Applies migrations on boot.
worker (×2)	Runs async jobs from BullMQ, delivers callbacks, and every 30 s rolls finished jobs into 5-minute usage buckets. Also runs the janitor (every 5 min) and retention (daily).
Postgres	Jobs and their event timeline, API keys, users, sessions, invites, settings, usage rollups.
Redis	BullMQ queues (jobs, callbacks, maintenance), rate-limit windows, login throttles.
R2	Where outputs live and where callers download them from.
Utilities	The existing functions, invoked directly by the gateway.

Decisions

These are the choices that shape the codebase, and why.

1. A gateway over the existing functions

The utilities stay where they are; this platform is the front door. Porting them (Chromium for HTML → PDF, a PDF library for splitting) was deliberately not part of v1. The runner interface is the seam where a native, Worker-based or container runner can later replace any single utility without callers noticing.

2. Invoke functions directly, not through their old HTTP endpoints

The old per-function HTTP endpoints cap a request at 29 s; the functions themselves run up to 15 minutes. Calling them with a direct invoke removes that ceiling and moves the trust boundary to cloud IAM. The functions still perform their own token check, so the gateway mints a short-lived token per call with each function's current signing key. An HTTP transport remains as a fallback.

3. Outputs are re-hosted on R2 by the gateway

Callers get R2 links. Because the functions are unchanged and still write to S3, the gateway streams each output S3 → R2 after the function answers. The cost is one extra transfer per file; if a function is later changed to write to R2 directly, its links are recognised and passed through without a copy.

4. Every call is a job

Sync and async share one row type, one execution path, one timeline and one metrics stream. A sync call is simply a job the request thread runs and waits on; past 90 s (under Cloudflare's 100 s edge limit) it answers 202 with the job id and keeps going — "sync" never fails just because a utility is slow. If the api restarts mid-run, unfinished sync jobs are handed to the worker queue so they still complete.

Only failures to reach a utility are retried (throttling, 502/503, network errors) — up to 3 attempts. A 4xx/5xx answer from the utility is final: those handlers have side effects, and a bad input fails the same way twice.

5. A clean break for callers

No compatibility routes, no legacy tokens. Callers move to /v1/utilities/{slug} with an API key, a normalised { input } envelope and a normalised result.files[] (migration guide). Input is validated at the gateway, which also closes a zip-slip hole in the zip utility's entry names.

6. Latency as mergeable histograms

Usage rollups store fixed-boundary latency histograms next to the counts. Histograms add exactly across time, keys and utilities, so any window's p50/p95/p99 comes from summing rows — averaging stored percentiles would be wrong. Raw jobs are pruned after the retention period; rollups are kept, so usage history outlives the raw rows. Details in Usage & latency.

7. Console sessions are cookies, not tokens in JS

The console and the API share one origin, so an httpOnly, SameSite=Lax session cookie (stored hashed) keeps credentials out of JavaScript. Mutations must carry an X-DU-Console: 1 header that cross-site forms can't set. Passwords use argon2id; TOTP secrets are encrypted at rest; Google sign-in is restricted to allowed domains unless the person was invited.

8. Callbacks follow Standard Webhooks

Callbacks are signed with webhook-id / webhook-timestamp / webhook-signature: v1,… over id.timestamp.body, so any Standard Webhooks library can verify them. Destinations are DNS-resolved before every attempt and refused if private (the worker shares a host with the datastores). Eight attempts with exponential back-off, about 21 minutes in total. See Callbacks.

9. External partners are guests; every key has its own signing secret

Staff (admin, member) see the whole workspace. A guest — an external partner — sees only the keys they created, the jobs those keys (or they, in the playground) ran, and that usage. Every API key has its own callback signing secret, so no key holder can forge callbacks for another key's jobs. See API keys & access.

The life of a job

Step	Event on the timeline
Request validated, job row written (running for sync, queued for async)	accepted (+ queued)
A worker (or the request thread) claims it	started
The utility is invoked	upstream_request
It answers (status, duration, its request id)	upstream_response
Transient failure with attempts left	retry
Outputs copied to R2	storage
Terminal state written with timings	succeeded / failed / cancelled
Callback attempts (async with callbackUrl)	callback

The console's job page shows exactly this timeline, with millisecond offsets.