Operations

This page covers what operating Unphish v2 looks like day-to-day and during incidents.

Health checks and readiness

Every environment exposes readiness signals at /hub/readiness. The signals cover:

• Database. Connectivity, pool health, migration version vs. expected, recent error rate.
• Authentik. OIDC discovery reachable, admin token valid, recent auth failures within tolerance.
• Providers. Per-provider mode (live / fixture / sandbox / unconfigured) and last-call status.
• Worker. Render worker heartbeat, Temporal task-queue lag, recent retries and DLQ counts.
• Storage. S3 / Vercel Blob reachable; recent upload success.
• Queues and schedules. Verification schedule on time; report schedule on time.

A green readiness page is a precondition for promoting deployments. An amber/red signal blocks promotion until resolved or an explicit override (capability-gated, audited) is recorded.

Observability

Application logs

• Structured JSON logs with subject IDs (user, organization, client, case) but no payload bodies.
• Errors include stack traces and request metadata.
• Access logs are aggregated at the Vercel layer; analyst-facing surfaces produce specific events for important transitions.

Worker logs

• Per-workflow run / step IDs for traceability.
• Activity start/finish events with duration and retry counts.
• Provider request/response references (the actual payloads are stored in S3 + Postgres; logs reference paths).

Metrics we watch

• Sign-in success rate per environment.
• Auth failure rate per environment (sustained spike = potential phishing campaign or Authentik issue).
• Provider call success rate per provider per environment.
• Workflow retry rate, failure rate, DLQ count.
• Verification schedule on-time percentage.
• Migration phase status (during cutover).
• API response latency by route family.

Alerting

Alerts fire to operations on-call for:

• Readiness red.
• Authentik admin token failure.
• Provider error rate over threshold for any live provider.
• Worker heartbeat loss.
• DLQ growth.
• Migration phase failure.
• Backup completion failure.

Backups and restore

Component	Backup mechanism	Restore RPO/RTO target
Postgres (Neon)	Managed continuous backup with point-in-time recovery.	RPO ≤ 5 min; RTO ≤ 1 hour for a known good point in time.
Object storage (S3)	Versioning + lifecycle. No deletes by default.	RPO ≈ 0 for current versions; older versions per lifecycle.
Authentik	Self-managed Postgres backups for the Authentik host. Configuration as code where possible.	Tested per restore drill cadence.
Temporal Cloud	Managed by Temporal Cloud; retention per the chosen tier.	Per Temporal Cloud SLA.

Restore procedures are documented in the production-readiness runbook (docs/production-readiness-runbook.md). They are exercised periodically to validate they actually work; an untested restore is not a backup.

Migration runbooks

For v1 → v2 migration:

• Audit before import. Run the read-only v1 audit (pnpm audit:v1-dump) and produce a migration report with row counts, distinct values, orphan checks, and file inventory.
• Stream high-volume tables. unphish_whoisfreaks_nrd_domain (1.45M rows) and unphish_scan_query (250K+ rows) require streaming with checkpoints and resumability.
• Phase the import. Phase 1 (clients/brands/users) before Phase 2 (cases/notes) before Phase 3 (enforcements) before Phase 4 (watchlist) before Phase 5 (whitelist/scans/quotas/API/SSO) before Phase 6 (validation).
• Preserve legacy IDs. Every imported record carries legacy_v1_* for traceability.
• Validate after each phase. Row counts match; required foreign keys resolve; enum values are normalized; orphans are quantified.
• Label imported data. Imported records carry source: imported and the UI labels them.
• Approve the migration report before promoting the imported data to production.

The migration is resumable. Failed batches roll back and re-run; checkpoints persist so we never re-import already-imported records.

Deployment

Pull request

Vercel creates a preview deployment for every PR. The preview is the closest reproduction of production we have for short-lived testing. Run route smoke and affected canonical Playwright/API checks against the preview before merging.

Staging

Merging to main deploys to staging. Staging shares the same routes as production but uses staging environment variables, fixture/sandbox provider modes, and a staging database / branch.

The post-merge checklist is in Release gates → Promotion to staging.

Production

Production promotion is manual. The release manager:

1. Confirms the staging build is healthy.
2. Runs through the production gate checklist.
3. Promotes via Vercel.
4. Watches readiness, recent activity, and worker health for 30 minutes post-deploy.
5. Verifies first real user sign-in succeeded.
6. Documents the deployment in the activity log.

Rollback is documented per release; for a fast revert, redeploy the prior commit. For a migration that must be undone, follow the documented migration rollback (per phase).

Incident response

Severity

• Sev 1. Customer-impacting outage or data integrity issue. Examples: sign-in down, dashboard returns wrong tenant's data, enforcement submission fails platform-wide, audit log write failures. Page on-call immediately.
• Sev 2. Significant degradation. Examples: one provider down, one tenant cannot load a specific surface, scheduled report failed to deliver. On-call notified during business hours.
• Sev 3. Minor degradation or known-issue tracking. Examples: cosmetic UI bug, single non-critical workflow retry, intermittent provider warning. Filed and triaged.

First response

For Sev 1:

1. Acknowledge in the incident channel within 5 minutes.
2. Page any required additional responders.
3. Capture initial state. Take screenshots; record current versions; note recent deploys.
4. Mitigate. If a recent deploy is implicated, roll back. If a provider is implicated, switch to fixture/sandbox mode for that provider while investigating.
5. Communicate. Update the status surface; notify affected customers if customer-impacting.
6. Resolve. Address the root cause once stabilized.
7. Postmortem. Within 5 business days for any Sev 1.

Common runbooks

Symptom	First action
All sign-ins failing	Check Authentik readiness; check AUTH_AUTHENTIK_* env vars; check OIDC discovery reachable.
Dashboard loading "unavailable" everywhere	Check /hub/readiness → database tile; check pool exhaustion; check recent migration.
One provider failing	Check /hub/secrets → provider last-check; rotate credential if compromised; switch to fixture mode while triaging.
Worker stopped processing	Check Render worker heartbeat; check Temporal Cloud namespace; restart the worker process.
Migration phase failed	Check /admin/imports → the run; review error; address root cause; resume from checkpoint.
Customer reports wrong-tenant data	Treat as Sev 1; check audit log; isolate the affected route; verify tenant scoping at the data-access layer.
Provider returned partial / rejected at scale	Check classification confidence and evidence quality for affected cases; check provider-side policy changes; consider escalation.

Post-incident

• Postmortem document with timeline, root cause, contributing factors, action items, and a parity matrix entry if behavior is now expected to be different.
• Action items tracked in the Delivery board with owners and dates.
• Customer communication for affected tenants, scoped to what they need to know.

Routine maintenance

Task	Cadence
Restore drill (Postgres, evidence, Authentik)	Quarterly
Provider credential rotation	Per provider policy; minimum annually
Authentik admin token rotation	On Authentik secret_rotate events; minimum semi-annually
Capability/role audit	Quarterly
Audit log review (sample)	Monthly
Backup verification	Monthly
Dependency updates (security)	As security advisories warrant; minimum monthly

Capacity planning

Inputs we track:

• Cases / day per tenant.
• Enforcement submissions / day per provider.
• Verification check load (4-hour cadence × active cases).
• Watchlist enrichment load (per-watchlist cadence × subscribed items).
• Scheduled report generation load.
• Migration backfill load (during cutover phases).

These feed Render worker count, Temporal task-queue concurrency, and database pool sizing. Increases trigger an environment configuration change rather than ad-hoc scaling.

Data subject requests

When a customer or end-user requests access, export, or deletion:

1. Verify the requester through the customer's admin channel.
2. Scope the request to the affected records.
3. Execute via admin surfaces — export, soft-delete, or hard-delete with the documented two-person approval where required.
4. Audit the action with reason and approver.
5. Confirm completion to the requester.

Hard deletion is exceptional; default is soft deletion preserving audit history.