mark rosenberg
d4a8a587f0
Migrates the reaper-king-no-more proof of concept into decks/reaper-king-no-more/
as the first deck under a shared tools/ engine, so the repo can hold many
Commander decks instead of one implicit one.
- tools/deckctx.py: new shared DeckContext, resolves --deck / $MTG_DECK / cwd,
loads a deck's config.yaml, and derives color identity from the commander's
own card data (never a hardcoded color combination).
- cards.py: drop the JUND=frozenset("BRG") constant; jund_legal() renamed to
identity_legal(card, identity); DB() takes an explicit blacklist_path since
the blacklist is per-deck, not shared.
- combos.py, roles.py, similar.py, similar_text.py, bracket_check.py: take
--deck, use the deck's real identity instead of Jund/BRG.
- power.py, card_stats.py, forge_report.py: OURS deck-name constant now comes
from the deck's config (name: / commander), matched via DeckContext instead
of hardcoded "Reaper King No More".
- to_dck.py, batch_run.py, ab_test.py: .dck filename and [metadata] Name= are
derived from the deck (slug / config name), not a hardcoded "reaper.dck".
- generate.py: RECIPES and make_variants.py's VARIANTS (deck-specific
hypotheses) moved out of Python into decks/reaper-king-no-more/variants.yaml;
protected cards now come from config.yaml's protected: list instead of a
hardcoded set.
- New tools/mtg dispatcher: `tools/mtg decks/<name> <tool> [args...]` runs any
tool against any deck without cd-ing into its directory.
- data/ (Scryfall/MTGJSON/Spellbook bulk, ~1.2GB, gitignored) and embeddings.npz
move to the repo root as shared, commander-format-wide data.
- docs/howto.md: moved from the deck dir (it documents the tool, not the deck)
and updated its example commands to the real generalized CLI.
- README.md: rewritten to describe the multi-deck repo and point at
decks/reaper-king-no-more as the reference example.
Verified after the change: bracket_check, combos, and roles against
reaper-king-no-more reproduce the exact PASS/11-combos/role-profile numbers
from before the move; generate.py --list now loads recipes from YAML;
make_variants.py reproduces all 7 hand-curated variants including the
mikaeus FAIL control case; to_dck.py derives the right Name= and filename;
variant.py and compare.py reproduce decks/README.md's original numbers
exactly. The original POC at ~/mtg/reaper-king-no-more is untouched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QPKgLYpMR8szWNFXyK1XNu
335 lines
13 KiB
Markdown
335 lines
13 KiB
Markdown
# How to use this system
|
||
|
||
A pipeline that generates thousands of legal Commander decks, plays them against each
|
||
other in a real rules engine, and ranks **cards and card pairs** by how much they
|
||
actually contribute to winning.
|
||
|
||
It is not a deckbuilder that hands you "the best deck". It is a measuring instrument.
|
||
What it produces is *evidence*; you still make the decisions.
|
||
|
||
**This is the generic tool.** It works on any deck in `decks/<name>/` that supplies
|
||
the four inputs below. Every command takes `--deck decks/<name>` (or a positional
|
||
deck path via `tools/mtg decks/<name> <tool> ...`), or falls back to `$MTG_DECK`, or
|
||
to the current directory if you `cd` into a deck first. Nothing here is hardcoded to
|
||
one commander or one color identity — see `decks/reaper-king-no-more/` for a worked
|
||
example.
|
||
|
||
---
|
||
|
||
## What you supply
|
||
|
||
Four things, in a deck's own `config.yaml`. Everything else is derived.
|
||
|
||
### 1. Your commander
|
||
|
||
```yaml
|
||
commander: "The Reaper, King No More"
|
||
```
|
||
|
||
From this the system derives your **color identity** (which cards are legal at all),
|
||
the legality filter, and which combos in the 108k-combo database are reachable. Color
|
||
identity is *always* looked up from the commander's own card data — it is never a
|
||
hardcoded color combination anywhere in `tools/`.
|
||
|
||
### 2. Your blacklist — cards you won't play
|
||
|
||
`<deck>/blacklist.txt` (path configurable via `blacklist_file:`), one per line, with a reason:
|
||
|
||
```
|
||
Sol Ring | boycott, not a power-level call
|
||
Talisman of Indulgence | prefer land ramp
|
||
Mind Stone | prefer land ramp
|
||
```
|
||
|
||
Reasons are for you, not the tool. Blacklisted cards are excluded from the pool,
|
||
from generation, and from "you're one card away" combo suggestions — so they never
|
||
come back as a recommendation.
|
||
|
||
### 3. Your card pool
|
||
|
||
The set of cards decks may be built from. Two ways to define it:
|
||
|
||
```yaml
|
||
pool:
|
||
max_edhrec_rank: 1000 # by popularity, or...
|
||
file: mypool.txt # ...an explicit list you curate
|
||
```
|
||
|
||
**Pool size is the most consequential choice you make.** See *Choosing a pool size*
|
||
below — it directly determines whether your results mean anything.
|
||
|
||
### 4. Your constraints
|
||
|
||
```yaml
|
||
bracket: 3 # enforces game changers / MLD / extra turns
|
||
house_rules:
|
||
no_vacuum_combos: true # no 2-card infinites needing no board state
|
||
lands: 36
|
||
```
|
||
|
||
---
|
||
|
||
## What you do NOT supply
|
||
|
||
Derived from the card databases, no input needed:
|
||
|
||
- **Color identity** — from the commander
|
||
- **Which cards are legal** — Scryfall
|
||
- **Game Changers list** — Scryfall's `is:gamechanger` (53 cards, auto-updates)
|
||
- **What each card does** — 4,369 Scryfall Tagger tags, so the system knows what a
|
||
sac outlet or a drain payoff *is* without you labelling anything
|
||
- **Which card combos exist** — 108,535 combos from Commander Spellbook
|
||
- **How popular a card is** — MTGJSON `edhrecRank`
|
||
- **Opponents** — 173 Commander precons ship with Forge; 4 pods rotate automatically
|
||
|
||
Optionally you can tune the **role budget** (how many sac outlets, removal spells,
|
||
ramp pieces a deck should have) but sensible defaults are provided.
|
||
|
||
---
|
||
|
||
## Choosing a pool size
|
||
|
||
Each deck has ~63 nonland slots. Coverage is what makes the statistics work:
|
||
|
||
| pool | 100 decks | 500 decks | 1000 decks |
|
||
| --- | --- | --- | --- |
|
||
| 300 | 21 decks/card · 4.4x per pair | 105 · 21.8x | 210 · 43.5x |
|
||
| **451** | 14 · 1.9x | **70 · 9.6x** | **140 · 19.2x** |
|
||
| 978 | 6.4 · 0.4x | 32 · 2.0x | 64 · 4.1x |
|
||
|
||
**Recommendation: 300–500 cards.** A 978-card pool at 1000 decks gives only 4x
|
||
coverage per card *pair*, which is not enough to measure synergy. A smaller, curated
|
||
pool measured well beats a large pool measured badly.
|
||
|
||
Note the combinatorics: C(451, 63) is about **10^78** possible decks — comparable to
|
||
the number of atoms in the observable universe. You are never searching this space.
|
||
You are *sampling* it well enough to score individual cards.
|
||
|
||
---
|
||
|
||
## Choosing a game count
|
||
|
||
Win rate in a 4-player pod has a 25% random baseline. Precision:
|
||
|
||
| games per deck | 95% confidence interval |
|
||
| --- | --- |
|
||
| 30 | ±15.5pp — meaningless |
|
||
| 50 | ±12.0pp — still noise |
|
||
| 100 | ±8.5pp |
|
||
| **150** | **±6.9pp** — usable |
|
||
| 300 | ±4.9pp |
|
||
|
||
**Use 150.** Below that you cannot distinguish a good deck from a lucky one.
|
||
|
||
**But deck-level precision is not the point.** Card-level data pools across every
|
||
deck a card appeared in:
|
||
|
||
| card appears in | total games behind it | CI |
|
||
| --- | --- | --- |
|
||
| 14 decks × 150 | 2,100 | ±1.9pp |
|
||
| 70 decks × 150 | 10,500 | ±0.8pp |
|
||
| 140 decks × 150 | 21,000 | ±0.6pp |
|
||
|
||
This is the real payoff. Individual decks stay noisy forever; **cards get sharp fast.**
|
||
You are not running 1000 decks to find the best one — you are running 1000 decks as a
|
||
sampling apparatus to score 451 cards.
|
||
|
||
---
|
||
|
||
## How long it takes
|
||
|
||
Measured: ~48s/game on a modern desktop i7; expect ~120s/game on older Xeons.
|
||
Forge is **CPU-only** — single-threaded per instance, no GPU involvement.
|
||
|
||
Sizing: **1 worker ≈ 1 core ≈ 1.5 GB RAM.**
|
||
|
||
| hardware | workers | limit |
|
||
| --- | --- | --- |
|
||
| 16 cores / 32 GB | 16 | cores (needs 27GB) |
|
||
| 16 cores / 64 GB | 16 | still cores — extra RAM does not help |
|
||
| 32 cores / 64 GB | 32 | cores (needs 51GB) |
|
||
|
||
On a 16-core box at ~120s/game (~480 games/hour):
|
||
|
||
| run | games | time |
|
||
| --- | --- | --- |
|
||
| 100 decks × 150 | 15,000 | **1.3 days** |
|
||
| 500 decks × 150 | 75,000 | 6.5 days |
|
||
| 1000 decks × 150 | 150,000 | 13 days |
|
||
|
||
This is a job that *cooks*. The queue is persistent and resumable — kill it, reboot,
|
||
resume, and you lose at most one deck's work.
|
||
|
||
---
|
||
|
||
## What you get out
|
||
|
||
### Card rankings
|
||
|
||
For each card: how often it was cast, what turn it landed, and its win-rate
|
||
contribution — computed by **ridge regression over deck composition**, not a raw
|
||
average. Regression matters because good cards cluster together; it estimates each
|
||
card's contribution *holding the rest of the deck constant*.
|
||
|
||
### Synergy pairs
|
||
|
||
For cards A and B, compare the observed win rate of decks containing both against
|
||
what you'd predict from their individual effects. A positive residual means the pair
|
||
is worth more than the sum of its parts. This is what finds "these two belong
|
||
together" rather than "both of these are good".
|
||
|
||
### Deck rankings
|
||
|
||
Ranked list with confidence intervals. Use it to identify the top and bottom
|
||
*deciles* — trust the tails, not the exact ordering.
|
||
|
||
### Never-cast list
|
||
|
||
Cards that never got played across thousands of games. The most immediately
|
||
actionable output: each one is dead weight, uncastable by the AI, or never drawn.
|
||
All three are worth knowing.
|
||
|
||
---
|
||
|
||
## Tuning the AI (and why the defaults lie to you)
|
||
|
||
Forge's AI is heuristic and its own docs say it is *"pretty bad for most combo decks."*
|
||
Three fixable problems distort results before any deck question is asked:
|
||
|
||
**1. The default clock silently discards your best games.** Forge's `-c` is a *per-game*
|
||
wall clock defaulting to **120s**; on expiry the game is abandoned as a **Draw**. Nobody
|
||
wins a draw. In the first 39 games logged here, **7 (18%) hit that limit** — and they are
|
||
overwhelmingly the long grindy games an aristocrats deck is built to win. The default
|
||
biases the measured win rate *downward*. `batch_run.py` now defaults to `--clock 600`.
|
||
|
||
**2. Forge refuses to play 2,506 cards.** Cards carrying `AI:RemoveDeck:All` in their
|
||
script are ones the engine will *never* cast. **7 cards in this deck** carry it, and
|
||
**32 of the 451-card pool (7%)**. They appear as "never cast" in results, which reads as
|
||
deck weakness but is a simulator artifact. `card_stats.py` now reports two lists:
|
||
|
||
```
|
||
NEVER CAST — AI-BLOCKED: 7 (simulator artifact, IGNORE)
|
||
NEVER CAST — PLAYABLE: 7 (real signal, worth investigating)
|
||
```
|
||
|
||
They are blocked for real reasons, and the reasons differ:
|
||
|
||
| card | why the AI can't cope | treatment |
|
||
| --- | --- | --- |
|
||
| Viscera Seer | bare sac outlet, **no AI logic written** | override (Woe Strider proves the pattern works) |
|
||
| Goblin Bombardment | same, but the whole sac-for-damage class is blocked | override, tested separately |
|
||
| Beast Within | gives an opponent a 3/3; can't weigh the downside | **substitute** |
|
||
| Toxic Deluge | X-cost paid with *life*; no logic to size X | **substitute** |
|
||
| Greater Good | draw X then discard 3; symmetrical drawback | **substitute** |
|
||
|
||
*Overriding* only helps where nobody wrote logic. Where the AI genuinely can't evaluate a
|
||
card, forcing it on makes it play the card badly — so `sim_substitutions.yaml` swaps in an
|
||
AI-playable analogue **for simulation only**, keeping the deck's shape intact:
|
||
|
||
```yaml
|
||
Beast Within: Feed the Swarm
|
||
Toxic Deluge: Blasphemous Edict
|
||
Greater Good: Umbral Collar Zealot
|
||
```
|
||
|
||
`decklist.txt` stays the truth; every swap is recorded in run metadata and printed. Note
|
||
the tradeoff: per-card stats for a substituted card describe **the stand-in, not the real
|
||
card**. This buys whole-deck realism at the cost of per-card precision.
|
||
|
||
**3. `Default.ai` disables the sacrifice fallback entirely.** `SACRIFICE_DEFAULT_PREF_ENABLE`
|
||
is **false** in stock Forge, so any sac outlet lacking a card-specific hint is simply never
|
||
fed — fatal for this archetype. `forge-overrides/ai/Combo.ai` turns it on and widens the
|
||
caps (`MAX_CREATURE_EVAL` 135 → 400, so the AI will feed real creatures rather than only
|
||
tokens).
|
||
|
||
```bash
|
||
python3 tools/batch_run.py --deck decks/<name> --ai-profile Combo --overlay --clock 600
|
||
```
|
||
|
||
Our seat only — **opponents always stay `Default`**, or you just move the baseline.
|
||
|
||
## Measuring: power level, not win rate
|
||
|
||
Win rate throws away everything that happened in games you lost, and needs 150 games to
|
||
say anything. `tools/power.py` scores what the deck *did*:
|
||
|
||
```
|
||
DID THE DECK DO ITS THING?
|
||
commander deployed 6/6 games (100%) median round 5
|
||
engine assembled 5/6 games ( 83%) median round 14
|
||
life taken from table mean 105.8
|
||
rounds survived mean 17.7
|
||
```
|
||
|
||
A 17% win rate reads as failure; "commander down every game, engine online in 83%, 106
|
||
life drained off the table, half the games ran out of clock" is a deck that works. Use
|
||
these to A/B a change — they move on far fewer games because they score *every* game.
|
||
|
||
`tools/ab_test.py` runs two configs on identical decks and seeds and reports per-metric
|
||
deltas with confidence intervals, labelling anything inside noise as `noise` rather than
|
||
pretending it is a result. **Always run `--aa` first** — two identical arms must show no
|
||
difference; if the null test reports `MOVED`, the harness is broken.
|
||
|
||
## The honest caveats
|
||
|
||
**Forge's AI is heuristic and, by its own documentation, weak at combo decks.** It
|
||
under-plays exactly the cards an aristocrats deck cares about. The tuning above
|
||
mitigates this; it does not remove it. **A tuned AI is not stock Forge** — results are
|
||
not comparable with untuned runs, which is why every run records its config. Everything here
|
||
measures *what Forge's AI can pilot*, which is a proxy for your pod, not a
|
||
substitute. Cards that need a human to see the line will score badly.
|
||
|
||
**Opponents are stock precons** (roughly bracket 2). Your results describe
|
||
performance against that field.
|
||
|
||
**The AI cannot do politics.** In a real pod, threat assessment and table talk decide
|
||
games. Forge plays a free-for-all somewhat like three parallel duels.
|
||
|
||
**Correlation, not causation.** A card cast in long games looks good because long
|
||
games are ones you survived. Regression helps; it does not eliminate this.
|
||
|
||
Use the output to generate hypotheses, then test them with a variant. Do not let a
|
||
win-rate column pick your deck.
|
||
|
||
---
|
||
|
||
## Running it
|
||
|
||
```bash
|
||
# one-time setup on a new machine (data/ is shared across every deck in the repo)
|
||
python3 tools/fetch_data.py # ~1.2GB of card data
|
||
python3 tools/embed.py # -> data/embeddings.npz (needs GPU or ~20min CPU)
|
||
# install Forge 2.0.14 to ~/forge (needs Java 17+)
|
||
|
||
# configure a deck
|
||
$EDITOR decks/<name>/config.yaml # commander, pool, constraints, roles
|
||
$EDITOR decks/<name>/blacklist.txt # cards you won't play
|
||
$EDITOR decks/<name>/decklist.txt # the 100 cards
|
||
|
||
# check it, offline and instant
|
||
python3 tools/bracket_check.py --deck decks/<name>
|
||
python3 tools/combos.py --deck decks/<name>
|
||
python3 tools/roles.py --deck decks/<name>
|
||
|
||
# generate role-budget variants (hypotheses live in decks/<name>/variants.yaml)
|
||
python3 tools/generate.py --deck decks/<name> --list
|
||
python3 tools/generate.py --deck decks/<name>
|
||
# or hand-curated cut/add variants (variants.yaml `handwritten:` section)
|
||
python3 tools/make_variants.py --deck decks/<name>
|
||
|
||
# run the full analysis suite for one variant, optionally with Forge games
|
||
python3 tools/variant.py --deck decks/<name> no-mikaeus # analysis only, ~10s
|
||
python3 tools/variant.py --deck decks/<name> no-mikaeus --sim 24 # + 24 Forge games, ~25 min
|
||
|
||
# batch-simulate every variant in the deck in parallel
|
||
python3 tools/batch_run.py --deck decks/<name> --games 150 --workers 16 --heap 2g
|
||
python3 tools/compare.py --deck decks/<name> # rebuild decks/<name>/decks/README.md
|
||
|
||
# same thing through the single dispatcher, without cd-ing into the deck
|
||
tools/mtg decks/<name> bracket_check
|
||
tools/mtg decks/<name> variant no-mikaeus --sim 24
|
||
```
|
||
|
||
Everything lands in `decks/<name>/decks/<variant>/` — decklist, legality check, combo
|
||
analysis, per-card stats, and every simulation run as `.log` / `.txt` / `.json`. See
|
||
the top-level `README.md` for the full repo layout.
|