GameForge AI

An autonomous gaming content system: AI agents play free games in a remote browser on a schedule, FFmpeg records

(and streams) the session, and clips and videos are cut and published through Krunc. Built in eight phases from

the owner's brief of 2026-11-09; this file records what each finished phase established.

Status: Phase 2 of 8 done (assessment, then the browser worker and its capture path). No game, agent,

scheduler, tables or dashboard exist yet.

Where things run

Part	Where	Why
Control plane: schedule, `gf_` tables, dashboard, API, publish queue	The Krunc app (PHP, this repo), on the Krunc web server	It already has a persistent scheduler, a job queue, `krunc-worker.service`, audit log, the ingest relay and the Bunny storage provider
Gameplay worker: browser, game adapters, agents, recording	A Docker container on the worker server (Contabo, 6 vCPU, 12 GB, no GPU)	The Krunc web server has 2 CPU threads shared with ~29 sites and the live relay; a 720p30 encode alone takes half of it

The worker pulls its jobs from the Krunc API and pushes results back (recordings to Bunny, live video to Krunc

ingest as RTMP), so it needs no inbound ports and can be replaced by a GPU machine without touching the control

plane. This departs from the brief's FastAPI + Celery + Redis on the control plane; the owner agreed on 2026-11-09.

The worker server is shared with the owner's other applications (an LLM runtime, an agent gateway, a voice

service, PostgreSQL). GameForge installs nothing on that host: one image, one folder (/srv/gameforge), and every

docker run carries --cpus and --memory limits.

Phase 1: assessment (2026-11-09)

Krunc web server: 1 core / 2 threads, 3.9 GB RAM with swap in use, 13 GB disk free, no GPU, no Docker. Not a

place for gameplay or recording.

Reusable in the app: Scheduler + jobs, Krunc ingest (the worker can stream to it like OBS and every enabled

stream destination gets the feed), BunnyStorage, the YouTube and TikTok providers, audit_logs, media_assets.

Gaps to close in later phases: no BUNNY_* credentials configured; BunnyStorage::put and the YouTube upload

hold the whole file in memory (fine for ads, not for multi-GB recordings); YouTube publishes as public with

no unlisted option; no YouTube account is connected; nothing exists for Groq, Ollama or ElevenLabs; Twitch and

Kick have no upload API, so they are live destinations only.

Games: SuperTuxKart, 0 A.D. and Mindustry are native desktop games, not browser games, and the two 3D ones need

a GPU. Licence terms for monetised footage must be confirmed per game before one is enabled (Phase 3).

Phase 2: browser worker (2026-11-09)

Code: integrations/gameforge/worker/ (see its README for build and run commands).

What it is: Ubuntu 24.04 image with Xvfb, PulseAudio (null sink gf_sink), Chromium 156 via Playwright 1.64.1 and

FFmpeg, running as an unprivileged user. Chromium opens the page as an app window (no tabs or address bar), FFmpeg

grabs display :99 and gf_sink.monitor into H.264 + AAC in MKV.

Measured on the worker, with its other workloads running:

Check	Result
Recording	1280x720, 30 fps, H.264 + AAC, MKV, about 6 Mbit/s; a 5-minute run finished cleanly with 8,913 of 9,000 frames
Audio	Captured from the virtual sound card (peak about -9 dB on the test tone)
2D canvas scene (600 sprites)	40-55 fps
WebGL scene (600 sprites)	about 20 fps
Full-screen WebGL shader	about 12 fps
Encoder cost	`ultrafast` about 0.5 core, `veryfast` about 1.4 cores
Picture vs sound	Picture about 100 ms behind, steady across the run
Dropped frames	1% to 7.5%, worst while the host's other workloads held every core
Isolation	From the container the host's PostgreSQL, Redis, Ollama and loopback apps are unreachable; only the ports the host already exposes publicly answer

Decisions that follow from it:

GF_GL=software (Chromium with no GPU process) is the default: compositing the whole page through SwiftShader

ran the 2D scene at 7 fps.

ultrafast at 6 Mbit/s is the capture default; the master is re-encoded when clips are cut.
The first game has to be a 2D canvas or light WebGL title. Anything 3D waits for a GPU worker.
Open for Phase 3: correct the ~100 ms audio lead with the real game, and decide how sessions share the CPU with

the host's other workloads (schedule around them, or give the container a larger CPU share).

Phase 3: the first game (2026-11-09)

The owner asked for the game most likely to get views. Checked candidates (licence, browser build, agent control,

fit for a GPU-less worker): Mindustry, Dungeon Crawl Stone Soup (WebTiles), Freeciv-web, BrowserQuest, 2048;

Shattered Pixel Dungeon has no official web build. Mindustry first, Crawl second: Mindustry is the only one

with a large existing audience, and tower defence makes clips.

Why the desktop build and not the browser one

The official Mindustry Classic HTML5 build (build 40, GPLv3) runs in the worker's Chromium, but at 6-8 fps even

after forcing its 4:3 canvas to 16:9, and the game then runs in slow motion. Keyboard input only reaches it as

KeyboardEvents dispatched on document; Playwright's keyboard never arrives. It has no state API: an agent

would play from screenshots.

The current desktop Mindustry (v160.7 jar from the GitHub releases, GPLv3, 29k stars) runs in the same

container with OpenJDK 17 and Mesa llvmpipe (software OpenGL), has a JavaScript console (F8) and loads

JavaScript mods at start. That gives the agent real game state and real actions instead of pixels.

What was built

integrations/gameforge/worker/:

Piece	Role
`games/mindustry/data-template/mods/gameforge/`	Bridge mod: auto-starts the configured map as a survival game, writes `gf/state.json` once a second (wave, countdown, enemies, core items and health, player unit, buildings, enemy units), executes `gf/cmd.js` in its own scope and answers in `gf/cmd.out`. `gf.*` helpers: `build`, `buildLine`, `goto`, `mine`, `ores`, `spawns`, `scan`, `skipWave`, `clearPlans`, `restart`. Walking and mining run in a per-frame hook on the player's own unit.
`games/mindustry/write_settings.py`	Generates the game's binary `settings.bin` (console on, bloom and lighting off, fps counter, fullscreen, no autosave, no community-server fetch).
`games/mindustry/adapter.py`	Launches the jar on the virtual display, waits for a playing state, reads state, runs JavaScript, restarts maps, keeps the game log with the session.
`agent.py`	Every `GF_AGENT_INTERVAL` seconds: state + map scan + recent results to the language model, which answers with a line of commentary, a plan and up to six actions; a deterministic scripted opening (hand-mine copper, drill + conveyor into the core, two duos towards the spawn) is step one of every session and the whole policy when `GF_AGENT=scripted`.
`llm.py`	One `chat_json` call (OpenAI chat completions, JSON mode, default model `gpt-5.4-mini`); other providers plug in here.
`recorder.py`	The Phase 2 FFmpeg capture (display + virtual sound card to H.264/AAC MKV) as a class.
`session.py`	Orchestrates one session: prepare, start, record, agent loop, screenshots every minute, state samples every 15 s, restart after game over, `report.json`.
`explore.py`	Scripted click/key/screenshot runs against a browser game while writing an adapter.

Run (the jar lives on the worker host, mounted read-only):

docker run --rm --cpus 4 --memory 3g --shm-size 1g --cap-add SYS_NICE \
  -v /srv/gameforge/worker:/app:ro -v /srv/gameforge/games/mindustry:/game:ro -v /srv/gameforge/recordings:/recordings \
  --env-file /srv/gameforge/worker.env \
  -e GF_MINUTES=60 -e GF_MAP=Fortress -e GF_NICE="nice -n -15" krunc-gameforge-worker python session.py
# worker.env holds OPENAI_API_KEY (and optionally GF_LLM_MODEL); add -e GF_AGENT=scripted to run without a model

Sessions land in /recordings/sessions/<timestamp>-mindustry/ with master.mkv, events.jsonl (agent steps with

commentary), state-samples.jsonl, shots/, the game log and report.json.

Measured (6-minute scripted session, Fortress, worker shared with the owner's other apps)

Check	Result
Recording	1280x720 at 30 fps, H.264 + AAC, 4.5 Mbit/s, 314 of 10,474 frames dropped (3%)
Game frame rate	4-11 fps, average 8; up to 24 fps with `nice -n -15`; the same scene is 12-14 fps without the encoder running
Game speed	below 30 fps the game slows down with the frame rate (about half speed at 14 fps)
Bridge	all commands answered within a second; mining raised core copper from 100 to 411 in 3 minutes; drill, conveyors and turrets were built by the player unit; game over and restart handled
Audio	captured from the game's SoLoud output through the null sink

The frame rate is the blocker, and it is not the container. The container uses 1.5-2.8 of its 4 cores and is

never throttled. The worker host runs the owner's Ollama (llama-server, 415-480% CPU, qwen3:4b kept loaded)

and OpenClaw (115-160%) around the clock, so every thread of the game waits for a core. Raising the container's

scheduling priority doubles the frame rate and is the only lever available without touching those apps.

Options for the owner: pause or throttle Ollama and OpenClaw during sessions (2 h every other night), pin them to

fewer cores, accept 15-20 fps video with priority, or move GameForge to its own machine.

Schedule and the CPU decision (owner, 2026-11-09)

Ollama and OpenClaw are paused for sessions, 2:00 to 4:00 AM Los Angeles time. deploy/gameforge/ holds the

pieces for the worker host: gameforge-session.sh stops ollama.service and openclaw-js.service, runs a

110-minute session in the container (the language model if /srv/gameforge/worker.env exists, the scripted agent

otherwise), and restarts both services on exit, error or signal; gameforge-session.timer fires it daily at 02:00

America/Los_Angeles and the script skips when the previous session was under 36 hours ago (one session per 48 h,

as the brief asks); gameforge-resume.timer at 04:05 starts both services if anything is still down. Install or

update on the worker as root with bash /srv/gameforge/deploy/install.sh (the files are synced there; the

classifier stops Claude from enabling systemd units itself). Logs: /srv/gameforge/logs/,

journalctl -u gameforge-session. --uninstall reverts.

Open before the 60-minute run

OPENAI_API_KEY must exist on the worker in /srv/gameforge/worker.env (mode 600); the auto-mode classifier

stops Claude from copying it there.

The timers above must be enabled by the owner (bash /srv/gameforge/deploy/install.sh on the worker).
The agent's first real runs will show whether gpt-5.4-mini plays well enough; the scripted opening alone loses

at wave 2 because the duos never get ammo.

Still needed from the owner

An SSH key login for the worker server (password login was used for Phase 2).
Bunny storage zone name, access key and pull-zone host.
A Groq API key (Phase 4); ElevenLabs if commentary is wanted (Phase 5).