Architecture
Hosts, services, data flow, and environments.
High-level diagram
┌─────────────────────────────────────────────────────────────────────┐
│ CLIENTS │
│ Browser │ Mobile (responsive) │ API integrations │
└──────────────────────┬───────────────────────────┬───────────────────┘
│ HTTPS │ HTTPS + API key
▼ ▼
┌─────────────────────────────────────────────────────────────────────┐
│ VERCEL (Next.js) │
│ • Routes: /hub, /admin, /dashboard, /partner, /client, /workbench │
│ • API routes: /api/* │
│ • SSR / RSC; server actions │
└─────────┬─────────────────────────┬──────────────────────┬──────────┘
│ │ │
│ OIDC │ SQL │ HTTPS
▼ ▼ ▼
┌──────────────────┐ ┌─────────────────────┐ ┌──────────────────┐
│ AUTHENTIK │ │ NEON POSTGRES │ │ POSTMARK / S3 │
│ (Render-hosted) │ │ (managed) │ │ (transactional │
│ • OIDC/OAuth │ │ • Per-env DBs │ │ email / blob) │
│ • SSO / MFA │ │ • Migrations │ │ │
│ • Recovery │ │ • Audit log │ │ │
└──────────────────┘ └─────────┬───────────┘ └──────────────────┘
│
│
┌─────────────────────┴──────────────────────┐
│ │
▼ ▼
┌────────────────────────────┐ ┌────────────────────────┐
│ RENDER WORKER │ │ TEMPORAL CLOUD │
│ • Temporal worker process │ workflows │ • Namespace per env │
│ • Provider adapters │ ◄──────────► │ • Task queues │
│ • Enrichment / enforcement│ │ • Durable state │
│ • Verification scheduler │ │ │
└──────────────┬─────────────┘ └────────────────────────┘
│
│ HTTPS (provider APIs)
▼
┌─────────────────────────────────────────────────────────────────────┐
│ PROVIDER INTEGRATIONS │
│ URLScan │ WhoisXML │ NothingPhishy │ CleanDNS │ WhoisFreaks │
│ Cloudflare │ Google Safe Browsing │ Microsoft SmartScreen │
│ Meta │ X │ Registrars │ Hosting providers │
└─────────────────────────────────────────────────────────────────────┘
Hosts and what runs where
| Host | Purpose | Why there |
|---|---|---|
| Vercel | Next.js app, preview deployments, route handlers, SSR/RSC. | First-class Next.js hosting, branch previews, env scoping per-environment. |
| Render | Long-running Temporal worker. Authentik. Other always-on services. | Vercel functions are not appropriate for long-running workers; Render gives us durable processes. Authentik must run outside Vercel for its own DB / Redis / TLS / SMTP / backups. |
| Temporal Cloud | Workflow state and orchestration. | Durable, replay-able, retryable workflows for ingestion, enrichment, enforcement, verification, scheduled reports. |
| Neon | Primary Postgres. | Managed Postgres with branching, pooling, backups. Per-environment databases or branches. |
| S3 / Vercel Blob | Evidence files (screenshots, attachments), generated reports, provider response artifacts. | Object storage is the right tool for binary evidence; databases store relative paths. |
| Postmark | Transactional email: invitations, notifications, reports. | Established v1 dependency with sender reputation; we keep it. |
Environments
Four environments. They share routes but differ by host, data, and provider mode.
| Environment | Host(s) | Database | Provider modes | Real effects |
|---|---|---|---|---|
| Hub | unphish.engram.org | Internal platform DB | n/a | Vercel env writes, Authentik API calls, secret metadata writes, audit. |
| Staging | unphish-staging.engram.org | Staging DB / branch | fixture or sandbox | Postmark email bin; sandbox provider calls; no real takedowns. |
| Demo | unphish-demo.engram.org | Demo fixtures | fixture | No real mutations; scripted/no-op. |
| Workbench | /workbench (or unphish-workbench.engram.org) | Staging or fixtures | fixture or sandbox | Sandbox provider calls; no real takedowns. |
| Production (future) | unphish-prod.engram.org | Production DB | live (where ready), otherwise fixture / unconfigured | Real takedowns, real provider submissions, real client emails. |
Data flow: from threat to enforcement
A threat moves through the system as follows:
- Intake. Provider feeds (URLScan, WhoisXML, etc.), scheduled scans, watchlist updates, user submissions, and API calls all converge on the same
/api/threat-feed/ingestshape. Intake validates ownership, whitelist, and required fields. - Enrichment. A workflow runs in Temporal that captures screenshots (desktop + mobile), DNS, WHOIS/RDAP, SSL/CT logs, redirect traces, HTML analysis, language detection, and email evidence as applicable. Files land in S3; rows land in Postgres.
- Classification. The classification engine scores the case across visual, NLP, domain, and evidence dimensions; produces a confidence, label, route, and explanation. Output is stored in
classification_runs. - Routing. Based on classification + client policy, the workflow either auto-enforces (high confidence + permitted), pauses for analyst review, or pauses for client approval.
- Enforcement. The chosen channel adapter (CleanDNS, registrar, hosting provider, social platform, browser blocklist) renders the channel template, submits the request, and stores the provider reference. Submissions are durable; the workflow retries on transient failures.
- Verification. Scheduled verification checks (DNS, HTTP, visual, blocklist) run every 4 hours by default. Status updates flow into the case timeline.
- Closure / resurrection. When the threat is down, the case closes and a 30-day resurrection monitor starts. If the threat reappears, the case reopens.
Every step writes audit entries. Workflow steps store payloads, outputs, durations, retries, and logs — replayable from the workbench.
Service responsibilities
Vercel (Next.js app)
- Route handlers (
app/**/page.tsx,app/api/**/route.ts). - Server actions for mutations.
- OIDC callback handler at
/api/auth/callback/authentik. - Read-side queries against Neon.
- Provider request submissions for fast, synchronous operations only (e.g., one-shot lookups, search calls).
- Long-running operations are enqueued to Temporal, not executed in the request lifecycle.
Render worker
- Temporal worker process executing workflows and activities.
- Provider adapters (URLScan, WhoisXML, NothingPhishy, CleanDNS, WhoisFreaks, Meta, X, Cloudflare, Google Safe Browsing, Microsoft SmartScreen).
- Enrichment activities (screenshot capture, DNS lookup, SSL inspection, HTML analysis).
- Enforcement activities (template rendering, provider submission, response polling).
- Verification activities (DNS / HTTP / visual / blocklist checks).
- Scheduled reporting and migration batch processing.
Temporal Cloud
- Workflow state and history.
- Task queues per environment (
unphish-enforcement-staging,unphish-verification-staging, etc.). - Schedules (4-hour verification cadence, weekly/monthly reports).
Authentik (Render)
- OAuth2/OIDC provider with separate application/provider per environment (local, staging, production).
- Branded login, recovery, MFA, and email templates.
- SMTP for password resets and MFA challenges.
- Optional federation to client SSO providers (Auth0, Okta, Microsoft Entra ID, Google).
Neon (Postgres)
- Identity / tenancy:
users,organizations,clients,memberships,team_invitations,user_sessions. - Domain:
cases,evidence_packages,attachments,notes,tags,enforcements,enforcement_submissions,verification_checks,watchlist_items,whitelist_items,scan_queries. - Operational:
workflow_runs,workflow_steps,provider_secret_metadata,audit_log(append-only). - v1 traceability:
legacy_v1_*columns and thev1_migrationschema for row maps.
Object storage
- Screenshots (desktop, mobile, full-page).
- Attachments uploaded by analysts and clients.
- Migrated v1 evidence files (under a
v1/prefix or viaV1_MEDIA_BASE_URLresolution). - Generated report exports (PDF, CSV, STIX 2.1).
- Provider response artifacts.
Files are addressed by relative path in the database; the application resolves to a presigned URL or CDN URL at request time.
Postmark
- Invitation emails.
- Notification emails (case updates, enforcement responses, scheduled reports).
- Per-environment streams (test stream for staging; production stream for production).
- Sender domain configuration with SPF / DKIM / DMARC.
Provider adapter contract
Every provider integration goes through an adapter module, not raw calls scattered across routes. The contract:
- Inputs include
organization_id,client_id(where applicable), environment, idempotency key, and request metadata. - Outputs include status, response payload reference, retry count, and timestamps.
- Mode is one of
live,fixture,sandbox,unconfigured. Set via env (URLSCAN_PROVIDER_MODE=live, etc.) per environment. - Production UI surfaces show the mode for each provider in
/hub/secretsand/admin/integrations. - Workbench executions use the same adapter contract; only the transport is switched to fixture/sandbox.
What does not run on Vercel
- Long-running workflows (Temporal worker on Render).
- Authentik (its own host).
- Provider polling, verification checks, scheduled reports (all on the worker).
- Migration batch imports (especially the 1.45M-row WhoisFreaks NRD table; this must stream).
This separation is deliberate. Vercel functions are short-lived; durable workflows must run elsewhere. Putting them on Vercel would mean dropping cases on the floor every time a function timed out.
Environment configuration
Required environment variables (per environment):
| Variable | Purpose |
|---|---|
DATABASE_URL | Postgres connection. |
AUTH_AUTHENTIK_ID | OIDC client ID. |
AUTH_AUTHENTIK_SECRET | OIDC client secret. |
AUTH_AUTHENTIK_ISSUER | OIDC issuer URL (no trailing slash, includes app slug). |
AUTH_AUTHENTIK_RECOVERY_URL | Recovery flow URL. |
AUTHENTIK_ADMIN_TOKEN | Non-expiring intent=api token for invite/profile/password operations. |
AUTH_SECRET | App session signing secret. |
NEXT_PUBLIC_APP_URL | Public-facing URL for OIDC redirect URI. |
POSTMARK_API_KEY | Email API key. |
POSTMARK_FROM_EMAIL | Default sender. |
EVIDENCE_STORAGE_BACKEND / S3 vars | Storage configuration. |
*_PROVIDER_MODE | Per-provider mode. |
Vercel env vars are scoped per environment (preview / staging / production). Render env groups handle the worker side.