Reference-to-Scene Pipeline

Drop any visual reference — a Hollow Knight gameplay clip, a Forza screenshot, a Midjourney render, a YouTube link — and Buril builds a playable Unity scene matching the reference's layout, palette, and gameplay pattern. The headline demo: paste a 30-second gameplay video, get a whitebox + greybox scene roughly 90 seconds later, then iterate via Buril's vision feedback loop.

This is the pillar feature of 1.1. Everything else in the AI features section feeds into it.

How it works

One call to build_scene_from_reference chains four stages:

1. Extract     vision LLM → ReferenceProfile (palette + mood + composition
                            + gameplay_pattern + camera_mode + notes)
2. Plan        WhiteboxPlanner → scene_template_* + tuned args (deterministic)
3. Whitebox    invoke the selected scene_template_* → gameplay primitives
   + Skybox    invoke scene_template_skybox_atmosphere (mapped mood)
4. Greybox     GreyboxStylist → palette → materials, skybox tint, sun, fog

Stages 1 and 4 hit the vision LLM. Stage 2 is pure C#, deterministic and unit-testable. Stage 3 invokes existing scene_template_* ITools (the same ones you can call directly). After step 4, Buril's existing vision feedback loop auto-attaches a screenshot of the result for the next agent turn — that's where the "final art" refinement loop closes.

Why is the planner separate from the LLM? Two reasons:

• Determinism. LLMs map gameplay_pattern → template correctly maybe 90% of the time. Code maps it 100% of the time.
• Replayability. The plan is JSON — you can inspect what Buril picked, edit the args, and replay the build.

The 6 modes

Mode	When to use	Input	Requires
Image	You have a screenshot or concept art	PNG / JPG / GIF / WebP (file or URL)	Vision-capable provider
Video / YouTube	You have a gameplay clip	MP4 / WebM / MOV / AVI / MKV file, or YouTube / Vimeo URL	ffmpeg + ffprobe + yt-dlp
Asset library	You bought packs and want Buril to compose from what you own	(auto-scans AssetDatabase)	Open Unity project
AI-generated URL	You ran image-gen elsewhere	URL of GPT Image 2 / Nano Banana 2 / FLUX / Imagen / Recraft / Ideogram output	Allowlisted host
Style guide only	You want palette + mood, NOT scene composition	Text prompt	Vision-capable provider
Whitebox-first	You want pure gameplay layout, no art	Text prompt	None (text-only)

The mode picks ingestion strategy; the four pipeline stages are the same for all modes.

Quick start

Chat panel

> creá una pista de carreras matching /Users/me/Pictures/forza-screenshot.png
[Buril] Step 1/4: Extracting reference profile (vision LLM call)...
[Buril] Step 2/4: Planning — selected scene_template_racetrack (32 segments, dusk mood)
[Buril] Step 3/4: Building whitebox...
[Buril] Step 4/4: Applying greybox style (palette: #3a8c2e + 3 more, skybox: dusk)
[Buril] Done. Vision feedback auto-attaches a screenshot for your next turn.

Programmatic (agent invocation)

{
  "tool": "build_scene_from_reference",
  "input": {
    "mode": "image",
    "local_path": "/path/to/forza-screenshot.png"
  }
}

Returns a structured envelope:

{
  "success": true,
  "pipeline": "build_scene_from_reference",
  "mode": "image",
  "skipped": { "whitebox": false, "greybox": false },
  "steps": {
    "extract":  { "profile": { "palette": ["#3a8c2e", "..."], "mood": "dusk", "gameplay_pattern": "racing", "camera_mode": "third-person", "notes": ["..."] } },
    "plan":     { "strategy": "driving", "selected_template": "scene_template_racetrack", "template_args": { "segments": 32, "track_width": 8, "palette_hint": ["..."] }, "rationale_bullets": ["..."] },
    "whitebox": { "template_invoked": "scene_template_racetrack", "result_raw": { "...": "..." } },
    "skybox":   { "template_invoked": "scene_template_skybox_atmosphere", "mood": "dusk", "result_raw": { "...": "..." } },
    "greybox":  { "materials_created": 4, "renderers_assigned": 18, "skybox_applied": true }
  },
  "next_steps": ["Inspect visually...", "Tweak with material_set_property...", "Re-run with skip_whitebox=true..."]
}

On failure the envelope has success: false, failed_at_step: <stage>, the error, and partial_results carrying anything that completed before the failure — useful for retrying with skip flags.

Mode deep-dive

Image reference

The most direct mode. Pass local_path (absolute path) OR url — not both.

• Local file: PNG, JPG, GIF, WebP.
• URL: must be on the allowlist (see below). The allowlist deliberately covers AI image-gen output hosts so URLs you copy from GPT, Gemini, FLUX, Replicate, Ideogram, and Recraft work as-is.

What kinds of references work best:

• Game screenshots. Best signal-to-noise — composition + palette + gameplay all readable in one frame.
• Concept art. Good for palette + mood; gameplay pattern often ambiguous → planner falls back to scene_template_rpg_environment.
• Real-world photos. Buril extracts palette + mood; gameplay pattern defaults to "unknown" → RPG environment template.

The URL allowlist (UrlSafetyValidator):

youtube.com         oaiusercontent.com (GPT Image 2)         imgur.com
youtu.be            googleusercontent.com (Nano Banana 2)    githubusercontent.com
vimeo.com           fal.media (FLUX / Seedream)              githubassets.com
                    replicate.delivery                       i.redd.it
                    ideogram.ai                              pinimg.com
                    recraft.ai                               wikimedia.org

URLs are also validated against SSRF — any URL that resolves to a private IP range (RFC1918, link-local 169.254.x including the AWS metadata endpoint, IPv6 fc00::/7, loopback) is rejected before the fetch. If you need to ingest from a host that isn't on the allowlist, upload the file directly.

Video reference

Two ingestion paths, both unified into the same downstream extractor:

• Local video file: MP4 / WebM / MOV / AVI / MKV.
• YouTube / Vimeo URL: yt-dlp downloads the clip into a temp directory.

ffmpeg samples 8 keyframes evenly across the clip and stitches them into a contact sheet (or sends them as separate frames, depending on provider). The vision LLM uses temporal reasoning across the 8 frames to infer gameplay patterns — differences between frames are the strongest signal for "this is a platformer" (vertical movement between frames) vs. "this is a driving game" (continuous horizontal flow) vs. "this is an FPS" (camera yaw + weapon viewmodel).

Required external tools:

# macOS
brew install ffmpeg yt-dlp

# Linux (Debian/Ubuntu)
apt-get install ffmpeg
pip install yt-dlp

ffprobe ships inside the ffmpeg package on every platform Buril supports. Buril checks ffmpeg, ffprobe, and yt-dlp on PATH before starting the extraction — if any is missing the pipeline fails fast at step 1 with a clear install hint.

End-to-end timing for video mode:

Stage	Typical time
yt-dlp download (YouTube)	10-30s
ffmpeg keyframe sampling	5-10s
Vision LLM extraction	5-10s
Plan + whitebox + greybox	10-30s
Total	~30-80s

Asset library

Zero image-gen cost. Best for studios with a curated paid asset library.

Buril scans the AssetDatabase for prefabs, materials, and textures, builds a 4x4 thumbnail grid (synthetic reference image), and feeds that to the vision LLM as if you'd uploaded a real screenshot. The extractor picks dominant palette + mood from the thumbnails; the planner picks a template from the inferred gameplay pattern.

Filters in place:

• Packages/ paths are excluded (Unity-shipped + third-party packages don't represent your taste).
• Editor-only assets are excluded.
• Empty Assets/ folders fall back to "no reference" mode → defaults to scene_template_rpg_environment.

This mode is industry-novel — no other tool composes scenes from your AssetDatabase. Use it when you've spent $500+ on packs and want Buril to remix what you already own instead of generating new content.

AI-generated URL

The path for users who prefer to iterate on the reference image in the image-gen model's own UI. Workflow:

1. Generate an image in GPT Image 2 / Nano Banana 2 / FLUX / Imagen / Recraft / Ideogram.
2. Copy the result URL directly out of the model's UI.
3. Paste into Buril's url field.

Buril detects the source model from the URL host and tags the reference profile with provenance — the rationale bullets will read "Reference sourced from GPT Image 2 generation" so you can trace the design lineage. Internally this is the same code path as the Image mode with a host-tagging pre-step; the allowlist (see Image section above) is what allows AI image-gen URLs through.

Style guide only

You provide a text prompt; Buril returns palette + mood only. No scene composition, no template selection, no whitebox built. The greybox stage receives the palette + mood and applies them to whatever scene is currently active (or to a default empty scene if none is open).

Use this when you want aesthetic anchors — "what palette would fit a moonlit cyberpunk alley?" — without scene-level bias from a specific reference image. The output is fast (one vision LLM call, ~3-5s) and cheap (no whitebox build).

Whitebox-first

Pure gameplay layout from a text prompt. No reference, no vision call. The agent picks a scene_template_* based on gameplay keywords in the prompt ("FPS", "racetrack", "platformer", "RPG", "dialog", "puzzle") and builds primitives — cubes, planes, lights, spawn markers.

This is the industry-standard "blockout" workflow that level designers have used for 20+ years. Buril ships it as one of the 6 modes because sometimes you really do just want geometry first, art second — and pretending otherwise is dishonest.

Whitebox-first does not require a vision-capable provider — any active provider works because no vision call happens. It's the only mode that works fully offline against a local LLM.

Mode → template mapping

The WhiteboxPlanner is a pure function from ReferenceProfile.gameplay_pattern to template + args:

Inferred gameplay	Template	Default args
platformer	scene_template_platformer_level	8 platforms, difficulty from pace
fps, combat	scene_template_fps_arena	arena_size from notes, wall_height 6, 12 cover blocks
third-person, top-down, exploration, rpg	scene_template_rpg_environment	biome from mood, size 150, 30 trees, 15 rocks
driving, racing	scene_template_racetrack	32 segments, track_width 8
vn, dialog	scene_template_dialog_panel	speaker "Character", sample text from first note
puzzle	scene_template_inventory_grid	6 rows x 8 cols
unknown, anything else	scene_template_rpg_environment (safe default)	meadow biome

Mood from the vision LLM is mapped through an alias table onto the 6-preset skybox enum (dawn / noon / dusk / night / overcast / stormy). Adjacent vocabulary collapses correctly:

• "sunrise", "morning" → dawn
• "sunset", "twilight", "evening", "golden hour" → dusk
• "moonlit", "midnight", "neon" → night
• "rainy", "thunder" → stormy
• "cloudy", "foggy", "misty" → overcast
• "midday", "bright", "sunny" → noon

Unknown moods default to noon (neutral lighting that works for the broadest range of styles).

Workflow knobs

Two boolean flags compose to four pipeline shapes:

skip_whitebox	skip_greybox	Use case
false	false	Default. Full pipeline: extract → plan → whitebox → greybox.
true	false	Re-skin. Keep the existing scene layout, re-apply palette + mood from a fresh reference. Cheaper than rebuilding from scratch when A/B-testing styles.
false	true	Layout only. Build the whitebox but leave materials as defaults. Useful when you want to hand-tune materials and don't want the auto-palette overriding your choices.
true	true	Extract-only. Only useful for inspecting what Buril sees in a reference — no scene changes.

Limitations

• Vision LLM cost. Each extraction is one round-trip with N images attached (1 for image mode, 8 for video mode). Budget ~$0.05-$0.20 per extraction depending on provider + model. Cloud-bundled credits in 1.2 will absorb this cost (see roadmap).
• Image quality bias. AI-generated references tend toward photorealism; FLUX / SDXL outputs require extra prompting if you want a specific stylized look. Game screenshots and concept art produce more art-directable extractions than diffusion outputs.
• Video processing time. ~30-80s end-to-end (download + sample + extract + plan + build + style). Budget accordingly when scripting batch flows.
• Whitebox + greybox, not final art. The scene Buril builds is playable and visually styled but not ship-ready. The vision feedback loop closes the gap over subsequent agent turns — expect 3-6 turns of "compare to reference, adjust materials, re-screenshot" before final.
• The planner is deterministic, not psychic. When gameplay_pattern extraction returns "unknown" — common for abstract concept art — the planner falls back to scene_template_rpg_environment. If you know the gameplay you want, write it into the prompt explicitly or use Whitebox-first mode instead.

Troubleshooting

Symptom	Fix
Active provider 'X' doesn't SupportsVision	Switch to one of: anthropic, openai, gemini, claude-code, codex-cli, antigravity-cli, azure-openai, bedrock, vertex-ai.
ffmpeg not found on PATH	brew install ffmpeg (macOS) or apt-get install ffmpeg (Linux).
ffprobe not found on PATH	Ships in the ffmpeg package — same install fixes both.
yt-dlp not found on PATH	brew install yt-dlp (macOS) or pip install yt-dlp (Linux).
Host '...' is not in the allowlist	The URL host isn't on the SSRF allowlist. Download the file and pass local_path instead.
URL '...' resolves to a private IP range	SSRF protection — the URL points at an internal endpoint (10.x, 192.168.x, 127.x, 169.254.x metadata). Use a public URL or upload the file.
DNS lookup failed for '...'	Offline or unreachable host. Check connectivity.
WhiteboxPlanner returned no selected_template	Profile extraction returned empty gameplay_pattern. Should never happen — file a bug if you hit this.
Scenario template '...' not found in ToolRegistry	A scene_template_* ITool the planner expects is missing. Re-import the bridge package; this means the registry is incomplete.
Vision LLM returns empty / nonsense profile	Provider quota exhausted or model context truncated the image. Try a different model (Claude Opus / GPT-4o / Gemini 1.5 Pro all work).

Roadmap

• 1.2 — Buril Cloud Credits integration: the vision LLM call uses bundled credits instead of BYOK, so users on the Buril Cloud paid tier don't need to wire their own provider key just to use this feature.
• 1.3 — Templates inferred from imported UnityPackages: when you import a Stylized Racing Kit, Buril auto-generates a scene_template_* that knows how to assemble its assets, and the planner can route to it.
• 1.5 — Reference template marketplace: users publish + share their own reference profiles (palette, mood, gameplay) and template arg presets, so the community can ship "Hollow Knight starter scene" / "Forza starter scene" packs.

Read next

• Multi-Agent Studio — where the scene-builder specialist lives and routes to this tool.
• AI 3D generation — to populate the whitebox with hero assets after Buril picks the layout.
• AI animation — to bring the scene's characters to life once art is final.