Runbook (VPS operations)

Host: HestiaCP server, domain krunc.com, document root /home/nerdy/web/krunc.com/public_html, PHP-FPM 8.3 runs as user nerdy, MariaDB database nerdy_krunc.

Deploy

cd /home/nerdy/web/krunc.com/public_html
git pull
composer dump-autoload -o
php bin/migrate            # applies pending SQL migrations exactly once
php bin/lint && php bin/test
chown -R nerdy:nerdy . && chmod 600 .env
curl -fsS https://krunc.com/healthz && curl -fsS https://krunc.com/readyz

readyz returns 503 when the database is unreachable or storage/ is not writable.

If the deploy changed CSS, JS or an image, bump ASSET_VERSION in .env. Everything under /assets is

served with Cache-Control: max-age=315360000 (ten years), and every asset URL carries ?v=<ASSET_VERSION>.

Without a bump, browsers that already have the old file keep it — a stylesheet change can look like it simply

did not deploy. Any new value works; a date (2026-10-08) is easiest to read.

Environment

All configuration lives in .env (chmod 600, never committed). .env.example lists every variable. Generate secrets with php bin/keygen. Rotating ENCRYPTION_KEY requires re-encrypting stored tokens — do not rotate casually (it also encrypts platform_settings, below).

Social OAuth apps do not need a server login. /admin/integrations (platform admin) writes each provider's client id/secret to the platform_settings table, encrypted with ENCRYPTION_KEY, and those values take precedence over .env. The same page shows the redirect URI to paste into each developer console and a read-only panel for the services that do need a deploy (Stripe, Anthropic, image, email, Krunc ingest). Clearing a provider there falls back to .env. See social → setting up each OAuth app.

Reverse proxies / Cloudflare

krunc.com is proxied by Cloudflare (orange cloud) in front of nginx → Apache → PHP-FPM. Client IPs are recovered in layers: nginx trusts CF-Connecting-IP from Cloudflare's published ranges (/etc/nginx/conf.d/cloudflare.inc), Apache's mod_remoteip trusts X-Real-IP from nginx, and the app (Request::clientIp()) honours CF-Connecting-IP / X-Real-IP / X-Forwarded-For only when the direct peer is loopback or listed in TRUSTED_PROXIES. A client that reaches the origin directly therefore cannot spoof its IP for rate limits or audit logs. Keep Cloudflare SSL mode on Full (strict) — the origin has a real Let's Encrypt certificate.

Worker & scheduler

One process drains the job queue (jobs) and ticks the scheduler (live polling, scheduled posts, token refresh,

offline detection, roll-ups, earnings accrual). Run exactly one of the two options below: Scheduler::tick()

guards each task with an interval row rather than a lock, so a cron pass racing a daemon can fire a task twice.

Jobs people wait on: stream-key tests (ingest.test_destinations), transactional email, scheduled publishing,

Discord delivery, AI generations, sponsor video renditions.

sudo bash deploy/worker/install.sh        # unit file, disables the cron line, enables + starts, verifies
systemctl status krunc-worker
journalctl -u krunc-worker -f            # app log is still storage/logs/app.log
systemctl restart krunc-worker           # after any deploy that changes job handlers or job code
sudo bash deploy/worker/install.sh --uninstall   # back to the cron, cron line restored

A queued job starts within ~3 s (the idle poll interval) instead of up to a minute. The installer backs the

crontab up to storage/crontab.<user>.<stamp>.bak before commenting the worker line out; --keep-cron leaves

the crontab alone, at the cost of two runners (see the warning above).

The worker is Nice=5 so ffmpeg and AI work stays behind the web server, and Restart=always: on a broken

database connection it exits 75 and systemd restarts it (MariaDB closes idle connections after 300 s, though

the 3 s poll keeps this one warm).

Stopping it is cooperative, because pcntl_signal is disabled in this host's PHP: ExecStop touches

storage/worker.stop, the loop notices within ~3 s, finishes the job in hand and exits 0. If you ever kill it

instead, the job it was running is reclaimed by the queue's 600 s stale-reservation sweep and retried.

Option B — minute cron (the fallback, and what a fresh install gets)

* * * * * cd /home/nerdy/web/krunc.com/public_html && php bin/worker --once >> storage/logs/worker.log 2>&1

Everything works, but anything queued waits up to a minute. QUEUE_DRIVER=sync (the default) also means a job

pushed from a web request runs inline in that request — which is why anything needing ffmpeg is pushed with a

run_at a few seconds out: proc_open and popen are disabled in the FPM pool, so only PHP-CLI can spawn it.

Logs

  • App: storage/logs/app.log (JSON lines, secrets redacted)
  • Mail (log driver): storage/logs/mail.log
  • Worker: storage/logs/worker.log
  • Web: /var/log/apache2/domains/krunc.com.*.log

Backups

bin/backup runs nightly from the app user's cron (30 3 *, log in ~/backups/backup.log) and does:

  1. mariadb-dump --single-transaction of the app database → ~/backups/db/<db>-<UTC stamp>.sql.gz (mode 600), verified (gzip -t + "Dump completed" marker) — a failed verification deletes the file and exits non-zero.
  2. rsync mirror of storage/uploads → ~/backups/uploads/, plus a dated tarball every Sunday.
  3. Retention: dumps 14 days (BACKUP_KEEP_DAYS), weekly upload tarballs 6 weeks.
  4. Off-box copy when BACKUP_RCLONE_REMOTE is set (e.g. b2:krunc-backups): rclone sync ~/backups <remote>. rclone is installed; configure a remote as the app user (rclone config) and set the variable in .env. Until then the copies live only on this VPS — do this before real customer data arrives.

Hestia's own daily user backup (/backup, admin account) is a second local copy of web + DB.

sudo -H -u nerdy /home/nerdy/web/krunc.com/public_html/bin/backup     # run by hand
tail -20 /home/nerdy/backups/backup.log                               # last runs
crontab -u nerdy -l                                                   # worker + backup entries

Restore

# database (restore into a scratch DB first to check, then the real one)
mysql -e 'CREATE DATABASE krunc_restoretest'
zcat ~/backups/db/nerdy_krunc-YYYY-MM-DD_HHMM.sql.gz | mysql krunc_restoretest
mysql -N -e "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='krunc_restoretest'"   # expect ~70
zcat ~/backups/db/nerdy_krunc-YYYY-MM-DD_HHMM.sql.gz | mysql nerdy_krunc
# uploads
rsync -a ~/backups/uploads/ /home/nerdy/web/krunc.com/public_html/storage/uploads/
php bin/migrate --status   # confirm schema matches the code

Restore tested 2026-09-24 into a scratch database (68 tables, tenants intact).

Stripe Connect (creator payouts)

Status 2026-09-24: sandbox keys are in .env (STRIPE_SECRET_KEY, STRIPE_PUBLISHABLE_KEY), both v1 webhook endpoints (STRIPE_WEBHOOK_SECRET, STRIPE_CONNECT_WEBHOOK_SECRET) and the v2 event destination for connected-account status (STRIPE_EVENTS_SECRET) were created through the API, and STRIPE_CONNECT_ENABLED=true. Verified against the sandbox: Express (v2) account create / status / onboarding link, invoice create + send + void, top-up Checkout session, and a signed thin-event ping to the live endpoint. Swap in live keys and re-create the endpoints in live mode when going live (stripe listen or POST /v1/webhook_endpoints as done here).

  1. Stripe Dashboard → Connect → enable Express accounts; set the platform name/branding shown during onboarding. Connected accounts use Accounts v2 (see docs/sponsors.md).
  2. .env: STRIPE_CONNECT_ENABLED=true.
  3. Developers → Webhooks → add a second endpoint for https://krunc.com/webhooks/stripe with "Listen to events on Connected accounts" and event account.updated; put its signing secret in STRIPE_CONNECT_WEBHOOK_SECRET. Add transfer.reversed, invoice.voided, invoice.marked_uncollectible to the platform endpoint. Then create a v2 event destination (Workbench → Event destinations, or POST /v2/core/event_destinations with events_from: ["@accounts"], thin payload, the v2.core.account[…] events listed in docs/sponsors.md) pointing at the same URL; its signing secret goes in STRIPE_EVENTS_SECRET. Going live means repeating all three in live mode.
  4. Transfers draw on the platform's available Stripe balance (advertiser top-ups). Keep a buffer; a short balance makes Pay via Stripe fail with Stripe's message and records nothing.
  5. Test: a creator opens My earnings → Set up payouts (use Stripe test mode + the test onboarding data), then pay a small balance from Creator payouts.

Stripe webhooks

Endpoint: https://krunc.com/webhooks/stripe. Set STRIPE_WEBHOOK_SECRET from the Stripe dashboard. Locally: stripe listen --forward-to https://krunc.com/webhooks/stripe. Failed events show up in /admin/webhooks with a reprocess button.

Email (SMTP) & reports

EMAIL_PROVIDER=smtp, SMTP_HOST=fusion.mxrouting.net, SMTP_PORT=465 (SSL/TLS), EMAIL_FROM="… <[email protected]>", [email protected] with its mailbox password in SMTP_PASS. Live since 2026-10-07 — before that every email (password resets, email verification, invites, receipts, reports) went to storage/logs/mail.log, which is still the fallback if either value is blank (Mailer picks LogEmail then).

Authenticating as info@ while sending as no-reply@ is accepted by MXroute. DNS as it stands: SPF v=spf1 include:mxroute.com -all ✓, DKIM at x._domainkey.krunc.com ✓, no DMARC record — adding _dmarc.krunc.com TXT "v=DMARC1; p=none; rua=mailto:[email protected]" in Cloudflare is worth doing for deliverability and reports.

Checks:

php -r '$a=require "bin/bootstrap.php"; echo $a->make("mailer")->provider()->name(), "\n";'   # expect: smtp (not log)
grep email.sent storage/logs/app.log | tail -5                                                # or email.failed
tail -20 storage/logs/mail.log                                                                # only written when unconfigured

A test with real delivery: /dashboard/analytics → Email me last week's report, or any password reset. Mail is queued (email.send), so with the cron fallback it leaves within a minute — see worker & scheduler. Reports: docs.

Site videos

assets/video/<docs-page>.mp4 (+ .jpg poster, .vtt captions) is picked up automatically by /docs/<page> and, for streamer-setup, by the dashboard guide. The MP4s are not in git (gitignored, mirrored nightly by bin/backup into video/); the poster and captions are. Re-render the setup explainer with bash deploy/video/build-streamer-setup.sh (needs OPENAI_API_KEY for narration; music is optional — drop a music.mp3 next to the script). Renders take ~10 minutes on this box.

Blog writer & sitemap

blog.autopublish runs from the worker every BLOG_AUTOPUBLISH_HOURS (48). It needs ANTHROPIC_API_KEY; without it every run is recorded as skipped on /admin/blog. /sitemap.xml is generated on request (cached 30 min) — submit it once in Google Search Console; see blog docs.

Twitch EventSub webhooks

Endpoint: https://krunc.com/webhooks/twitch (HMAC-signed by Twitch with TWITCH_EVENTSUB_SECRET; no Twitch dashboard setup — the app registers subscriptions itself). Off until the social.twitch_eventsub flag is enabled; see social → Twitch EventSub. Subscription state is on /dashboard/live; stored events and reprocessing are in /admin/webhooks (provider twitch). If the secret is rotated, the existing subscriptions keep the old secret: turn the flag off, run php bin/worker --once (or press Sync subscriptions) so they are deleted, then turn it back on.

Krunc ingest

MediaMTX runs as mediamtx.service (user nerdy, config /opt/mediamtx/mediamtx.yml, chmod 600 — it holds the hook secret). Install or upgrade with sudo bash deploy/ingest/install.sh (idempotent: .env settings, /etc/hosts pin for krunc.com, binary, config, service, Hestia firewall rule for TCP 1935).

systemctl status mediamtx && journalctl -u mediamtx -n 50 --no-pager   # server log (auth denials show here and in app.log as ingest.auth.denied)
tail -f storage/logs/ingest.log                                        # ffmpeg relay output via `ingest-relay watch` (per-destination errors, keys masked)
curl -s 127.0.0.1:9997/v3/paths/list | jq                              # what is publishing right now
php bin/ingest-relay reconcile                                         # close stale sessions by hand
  • OBS says "failed to connect": port 1935 open? (v-list-firewall), service running?, key enabled on /dashboard/social/ingest?, slots free? (ingest.auth.denied … ingest slots full in storage/logs/app.log).
  • Connected but a platform is not receiving: the workspace is notified and the session row says which (last_error, and not receiving on /dashboard/social/ingest); storage/logs/ingest.log has the masked URL and ffmpeg's reason. Input/output error means the platform rejected the key — re-fetch or re-paste it on the Channels card. The other destinations keep going.
  • A key test stays on "testing…": the probe is a queued job (ingest.test_destinations) because proc_open/popen are disabled in the FPM pool — only PHP-CLI may spawn ffmpeg. Check the worker is running (systemctl status krunc-worker, or the minute cron and storage/logs/worker.log) and that ffmpeg is on PATH for the app user. See worker & scheduler.
  • Session shows live after OBS stopped: the scheduler tick (php bin/worker --once) reconciles within a minute; php bin/ingest-relay reconcile does it now.
  • Rotating the hook secret: change INGEST_HOOK_SECRET in .env, re-run deploy/ingest/install.sh (rewrites mediamtx.yml), systemctl restart mediamtx.
  • Moving ingest off this box: keep the app URL in authHTTPAddress, point INGEST_API_URL at the new host, and run bin/ingest-relay there with a copy of .env (it needs DB + ENCRYPTION_KEY).

Studio (2026-09-25): re-run sudo bash deploy/ingest/install.sh after pulling — it rewrites mediamtx.yml (RTSP on loopback 8554, WebRTC/WHIP on loopback 8889 with the UDP mux on 8189, webrtcAdditionalHosts = the public IP), opens UDP 8189 in the Hestia firewall, creates storage/ingest/overlays, and adds INGEST_LOCAL_RTSP, INGEST_WHIP_URL, INGEST_MAX_COMPOSITED to .env. Checks: ss -ltn | grep -E ':8554|:8889', ss -lun | grep :8189. Browser go-live needs the UDP port reachable from the internet (Cloudflare is not involved: the browser talks to the public IP directly). Composited sessions show libx264 in ps; keep INGEST_MAX_COMPOSITED at 1 on this box. See social → browser go-live / server-side overlay.

Troubleshooting

  • 500 on every page: check storage/logs/app.log and Apache error log; usually .env unreadable by nerdy or a bad ENCRYPTION_KEY.
  • HTTP (port 80) shows a different site: a catch-all Apache vhost (/etc/apache2/conf.d/custom-domains-catchall.conf, ServerAlias *) captures plain-HTTP requests on this host. Force-SSL is enabled for krunc.com, so HTTP redirects to HTTPS which is served correctly.
  • Jobs stuck: /admin/jobs lists failed jobs; retry there or php bin/worker --once.
  • Migrations: never edit an applied file; add a new numbered .sql under database/migrations.