Unphish v2 Docs

Architecture

Hosts, services, data flow, and environments.

High-level diagram

┌─────────────────────────────────────────────────────────────────────┐
│                            CLIENTS                                   │
│       Browser  │  Mobile (responsive)  │  API integrations          │
└──────────────────────┬───────────────────────────┬───────────────────┘
                       │ HTTPS                     │ HTTPS + API key
                       ▼                           ▼
┌─────────────────────────────────────────────────────────────────────┐
│                          VERCEL (Next.js)                            │
│   • Routes: /hub, /admin, /dashboard, /partner, /client, /workbench │
│   • API routes: /api/*                                              │
│   • SSR / RSC; server actions                                       │
└─────────┬─────────────────────────┬──────────────────────┬──────────┘
          │                         │                      │
          │ OIDC                    │ SQL                  │ HTTPS
          ▼                         ▼                      ▼
┌──────────────────┐    ┌─────────────────────┐    ┌──────────────────┐
│   AUTHENTIK      │    │   NEON POSTGRES     │    │  POSTMARK / S3   │
│  (Render-hosted) │    │  (managed)          │    │  (transactional  │
│   • OIDC/OAuth   │    │   • Per-env DBs     │    │   email / blob)  │
│   • SSO / MFA    │    │   • Migrations      │    │                  │
│   • Recovery     │    │   • Audit log       │    │                  │
└──────────────────┘    └─────────┬───────────┘    └──────────────────┘


            ┌─────────────────────┴──────────────────────┐
            │                                            │
            ▼                                            ▼
┌────────────────────────────┐              ┌────────────────────────┐
│  RENDER WORKER             │              │  TEMPORAL CLOUD        │
│  • Temporal worker process │   workflows  │  • Namespace per env   │
│  • Provider adapters       │ ◄──────────► │  • Task queues         │
│  • Enrichment / enforcement│              │  • Durable state       │
│  • Verification scheduler  │              │                        │
└──────────────┬─────────────┘              └────────────────────────┘

               │ HTTPS (provider APIs)

┌─────────────────────────────────────────────────────────────────────┐
│   PROVIDER INTEGRATIONS                                              │
│   URLScan │ WhoisXML │ NothingPhishy │ CleanDNS │ WhoisFreaks       │
│   Cloudflare │ Google Safe Browsing │ Microsoft SmartScreen         │
│   Meta │ X │ Registrars │ Hosting providers                         │
└─────────────────────────────────────────────────────────────────────┘

Hosts and what runs where

HostPurposeWhy there
VercelNext.js app, preview deployments, route handlers, SSR/RSC.First-class Next.js hosting, branch previews, env scoping per-environment.
RenderLong-running Temporal worker. Authentik. Other always-on services.Vercel functions are not appropriate for long-running workers; Render gives us durable processes. Authentik must run outside Vercel for its own DB / Redis / TLS / SMTP / backups.
Temporal CloudWorkflow state and orchestration.Durable, replay-able, retryable workflows for ingestion, enrichment, enforcement, verification, scheduled reports.
NeonPrimary Postgres.Managed Postgres with branching, pooling, backups. Per-environment databases or branches.
S3 / Vercel BlobEvidence files (screenshots, attachments), generated reports, provider response artifacts.Object storage is the right tool for binary evidence; databases store relative paths.
PostmarkTransactional email: invitations, notifications, reports.Established v1 dependency with sender reputation; we keep it.

Environments

Four environments. They share routes but differ by host, data, and provider mode.

EnvironmentHost(s)DatabaseProvider modesReal effects
Hubunphish.engram.orgInternal platform DBn/aVercel env writes, Authentik API calls, secret metadata writes, audit.
Stagingunphish-staging.engram.orgStaging DB / branchfixture or sandboxPostmark email bin; sandbox provider calls; no real takedowns.
Demounphish-demo.engram.orgDemo fixturesfixtureNo real mutations; scripted/no-op.
Workbench/workbench (or unphish-workbench.engram.org)Staging or fixturesfixture or sandboxSandbox provider calls; no real takedowns.
Production (future)unphish-prod.engram.orgProduction DBlive (where ready), otherwise fixture / unconfiguredReal takedowns, real provider submissions, real client emails.

Data flow: from threat to enforcement

A threat moves through the system as follows:

  1. Intake. Provider feeds (URLScan, WhoisXML, etc.), scheduled scans, watchlist updates, user submissions, and API calls all converge on the same /api/threat-feed/ingest shape. Intake validates ownership, whitelist, and required fields.
  2. Enrichment. A workflow runs in Temporal that captures screenshots (desktop + mobile), DNS, WHOIS/RDAP, SSL/CT logs, redirect traces, HTML analysis, language detection, and email evidence as applicable. Files land in S3; rows land in Postgres.
  3. Classification. The classification engine scores the case across visual, NLP, domain, and evidence dimensions; produces a confidence, label, route, and explanation. Output is stored in classification_runs.
  4. Routing. Based on classification + client policy, the workflow either auto-enforces (high confidence + permitted), pauses for analyst review, or pauses for client approval.
  5. Enforcement. The chosen channel adapter (CleanDNS, registrar, hosting provider, social platform, browser blocklist) renders the channel template, submits the request, and stores the provider reference. Submissions are durable; the workflow retries on transient failures.
  6. Verification. Scheduled verification checks (DNS, HTTP, visual, blocklist) run every 4 hours by default. Status updates flow into the case timeline.
  7. Closure / resurrection. When the threat is down, the case closes and a 30-day resurrection monitor starts. If the threat reappears, the case reopens.

Every step writes audit entries. Workflow steps store payloads, outputs, durations, retries, and logs — replayable from the workbench.

Service responsibilities

Vercel (Next.js app)

  • Route handlers (app/**/page.tsx, app/api/**/route.ts).
  • Server actions for mutations.
  • OIDC callback handler at /api/auth/callback/authentik.
  • Read-side queries against Neon.
  • Provider request submissions for fast, synchronous operations only (e.g., one-shot lookups, search calls).
  • Long-running operations are enqueued to Temporal, not executed in the request lifecycle.

Render worker

  • Temporal worker process executing workflows and activities.
  • Provider adapters (URLScan, WhoisXML, NothingPhishy, CleanDNS, WhoisFreaks, Meta, X, Cloudflare, Google Safe Browsing, Microsoft SmartScreen).
  • Enrichment activities (screenshot capture, DNS lookup, SSL inspection, HTML analysis).
  • Enforcement activities (template rendering, provider submission, response polling).
  • Verification activities (DNS / HTTP / visual / blocklist checks).
  • Scheduled reporting and migration batch processing.

Temporal Cloud

  • Workflow state and history.
  • Task queues per environment (unphish-enforcement-staging, unphish-verification-staging, etc.).
  • Schedules (4-hour verification cadence, weekly/monthly reports).

Authentik (Render)

  • OAuth2/OIDC provider with separate application/provider per environment (local, staging, production).
  • Branded login, recovery, MFA, and email templates.
  • SMTP for password resets and MFA challenges.
  • Optional federation to client SSO providers (Auth0, Okta, Microsoft Entra ID, Google).

Neon (Postgres)

  • Identity / tenancy: users, organizations, clients, memberships, team_invitations, user_sessions.
  • Domain: cases, evidence_packages, attachments, notes, tags, enforcements, enforcement_submissions, verification_checks, watchlist_items, whitelist_items, scan_queries.
  • Operational: workflow_runs, workflow_steps, provider_secret_metadata, audit_log (append-only).
  • v1 traceability: legacy_v1_* columns and the v1_migration schema for row maps.

Object storage

  • Screenshots (desktop, mobile, full-page).
  • Attachments uploaded by analysts and clients.
  • Migrated v1 evidence files (under a v1/ prefix or via V1_MEDIA_BASE_URL resolution).
  • Generated report exports (PDF, CSV, STIX 2.1).
  • Provider response artifacts.

Files are addressed by relative path in the database; the application resolves to a presigned URL or CDN URL at request time.

Postmark

  • Invitation emails.
  • Notification emails (case updates, enforcement responses, scheduled reports).
  • Per-environment streams (test stream for staging; production stream for production).
  • Sender domain configuration with SPF / DKIM / DMARC.

Provider adapter contract

Every provider integration goes through an adapter module, not raw calls scattered across routes. The contract:

  • Inputs include organization_id, client_id (where applicable), environment, idempotency key, and request metadata.
  • Outputs include status, response payload reference, retry count, and timestamps.
  • Mode is one of live, fixture, sandbox, unconfigured. Set via env (URLSCAN_PROVIDER_MODE=live, etc.) per environment.
  • Production UI surfaces show the mode for each provider in /hub/secrets and /admin/integrations.
  • Workbench executions use the same adapter contract; only the transport is switched to fixture/sandbox.

What does not run on Vercel

  • Long-running workflows (Temporal worker on Render).
  • Authentik (its own host).
  • Provider polling, verification checks, scheduled reports (all on the worker).
  • Migration batch imports (especially the 1.45M-row WhoisFreaks NRD table; this must stream).

This separation is deliberate. Vercel functions are short-lived; durable workflows must run elsewhere. Putting them on Vercel would mean dropping cases on the floor every time a function timed out.

Environment configuration

Required environment variables (per environment):

VariablePurpose
DATABASE_URLPostgres connection.
AUTH_AUTHENTIK_IDOIDC client ID.
AUTH_AUTHENTIK_SECRETOIDC client secret.
AUTH_AUTHENTIK_ISSUEROIDC issuer URL (no trailing slash, includes app slug).
AUTH_AUTHENTIK_RECOVERY_URLRecovery flow URL.
AUTHENTIK_ADMIN_TOKENNon-expiring intent=api token for invite/profile/password operations.
AUTH_SECRETApp session signing secret.
NEXT_PUBLIC_APP_URLPublic-facing URL for OIDC redirect URI.
POSTMARK_API_KEYEmail API key.
POSTMARK_FROM_EMAILDefault sender.
EVIDENCE_STORAGE_BACKEND / S3 varsStorage configuration.
*_PROVIDER_MODEPer-provider mode.

Vercel env vars are scoped per environment (preview / staging / production). Render env groups handle the worker side.