mtg/docs/howto.md

How to use this system

A pipeline that generates thousands of legal Commander decks, plays them against each other in a real rules engine, and ranks cards and card pairs by how much they actually contribute to winning.

It is not a deckbuilder that hands you "the best deck". It is a measuring instrument. What it produces is evidence; you still make the decisions.

This is the generic tool. It works on any deck in decks/<name>/ that supplies the four inputs below. Every command takes --deck decks/<name> (or a positional deck path via tools/mtg decks/<name> <tool> ...), or falls back to $MTG_DECK, or to the current directory if you cd into a deck first. Nothing here is hardcoded to one commander or one color identity — see decks/reaper-king-no-more/ for a worked example.

---

What you supply

Four things, in a deck's own config.yaml. Everything else is derived.

1. Your commander

commander: "The Reaper, King No More"

From this the system derives your color identity (which cards are legal at all), the legality filter, and which combos in the 108k-combo database are reachable. Color identity is always looked up from the commander's own card data — it is never a hardcoded color combination anywhere in tools/.

2. Your blacklist — cards you won't play

<deck>/blacklist.txt (path configurable via blacklist_file:), one per line, with a reason:

Sol Ring | boycott, not a power-level call
Talisman of Indulgence | prefer land ramp
Mind Stone | prefer land ramp

Reasons are for you, not the tool. Blacklisted cards are excluded from the pool, from generation, and from "you're one card away" combo suggestions — so they never come back as a recommendation.

3. Your card pool

The set of cards decks may be built from. Two ways to define it:

pool:
  max_edhrec_rank: 1000     # by popularity, or...
  file: mypool.txt          # ...an explicit list you curate

Pool size is the most consequential choice you make. See Choosing a pool size below — it directly determines whether your results mean anything.

4. Your constraints

bracket: 3                  # enforces game changers / MLD / extra turns
house_rules:
  no_vacuum_combos: true    # no 2-card infinites needing no board state
lands: 36

---

What you do NOT supply

Derived from the card databases, no input needed:

• Color identity — from the commander
• Which cards are legal — Scryfall
• Game Changers list — Scryfall's is:gamechanger (53 cards, auto-updates)
• What each card does — 4,369 Scryfall Tagger tags, so the system knows what a sac outlet or a drain payoff is without you labelling anything
• Which card combos exist — 108,535 combos from Commander Spellbook
• How popular a card is — MTGJSON edhrecRank
• Opponents — 173 Commander precons ship with Forge; 4 pods rotate automatically

Optionally you can tune the role budget (how many sac outlets, removal spells, ramp pieces a deck should have) but sensible defaults are provided.

---

Choosing a pool size

Each deck has ~63 nonland slots. Coverage is what makes the statistics work:

pool	100 decks	500 decks	1000 decks
300	21 decks/card · 4.4x per pair	105 · 21.8x	210 · 43.5x
451	14 · 1.9x	70 · 9.6x	140 · 19.2x
978	6.4 · 0.4x	32 · 2.0x	64 · 4.1x

Recommendation: 300–500 cards. A 978-card pool at 1000 decks gives only 4x coverage per card pair, which is not enough to measure synergy. A smaller, curated pool measured well beats a large pool measured badly.

Note the combinatorics: C(451, 63) is about 10^78 possible decks — comparable to the number of atoms in the observable universe. You are never searching this space. You are sampling it well enough to score individual cards.

---

Choosing a game count

Win rate in a 4-player pod has a 25% random baseline. Precision:

games per deck	95% confidence interval
30	±15.5pp — meaningless
50	±12.0pp — still noise
100	±8.5pp
150	±6.9pp — usable
300	±4.9pp

Use 150. Below that you cannot distinguish a good deck from a lucky one.

But deck-level precision is not the point. Card-level data pools across every deck a card appeared in:

card appears in	total games behind it	CI
14 decks × 150	2,100	±1.9pp
70 decks × 150	10,500	±0.8pp
140 decks × 150	21,000	±0.6pp

This is the real payoff. Individual decks stay noisy forever; cards get sharp fast. You are not running 1000 decks to find the best one — you are running 1000 decks as a sampling apparatus to score 451 cards.

---

How long it takes

Measured: ~48s/game on a modern desktop i7; expect ~120s/game on older Xeons. Forge is CPU-only — single-threaded per instance, no GPU involvement.

Sizing: 1 worker ≈ 1 core ≈ 1.5 GB RAM.

hardware	workers	limit
16 cores / 32 GB	16	cores (needs 27GB)
16 cores / 64 GB	16	still cores — extra RAM does not help
32 cores / 64 GB	32	cores (needs 51GB)

On a 16-core box at ~120s/game (~480 games/hour):

run	games	time
100 decks × 150	15,000	1.3 days
500 decks × 150	75,000	6.5 days
1000 decks × 150	150,000	13 days

This is a job that cooks. The queue is persistent and resumable — kill it, reboot, resume, and you lose at most one deck's work.

---

What you get out

Card rankings

For each card: how often it was cast, what turn it landed, and its win-rate contribution — computed by ridge regression over deck composition, not a raw average. Regression matters because good cards cluster together; it estimates each card's contribution holding the rest of the deck constant.

Synergy pairs

For cards A and B, compare the observed win rate of decks containing both against what you'd predict from their individual effects. A positive residual means the pair is worth more than the sum of its parts. This is what finds "these two belong together" rather than "both of these are good".

Deck rankings

Ranked list with confidence intervals. Use it to identify the top and bottom deciles — trust the tails, not the exact ordering.

Never-cast list

Cards that never got played across thousands of games. The most immediately actionable output: each one is dead weight, uncastable by the AI, or never drawn. All three are worth knowing.

---

Tuning the AI (and why the defaults lie to you)

Forge's AI is heuristic and its own docs say it is "pretty bad for most combo decks." Three fixable problems distort results before any deck question is asked:

1. The default clock silently discards your best games. Forge's -c is a per-game wall clock defaulting to 120s; on expiry the game is abandoned as a Draw. Nobody wins a draw. In the first 39 games logged here, 7 (18%) hit that limit — and they are overwhelmingly the long grindy games an aristocrats deck is built to win. The default biases the measured win rate downward. batch_run.py now defaults to --clock 600.

2. Forge refuses to play 2,506 cards. Cards carrying AI:RemoveDeck:All in their script are ones the engine will never cast. 7 cards in this deck carry it, and 32 of the 451-card pool (7%). They appear as "never cast" in results, which reads as deck weakness but is a simulator artifact. card_stats.py now reports two lists:

NEVER CAST — AI-BLOCKED: 7   (simulator artifact, IGNORE)
NEVER CAST — PLAYABLE:   7   (real signal, worth investigating)

They are blocked for real reasons, and the reasons differ:

card	why the AI can't cope	treatment
Viscera Seer	bare sac outlet, no AI logic written	override (Woe Strider proves the pattern works)
Goblin Bombardment	same, but the whole sac-for-damage class is blocked	override, tested separately
Beast Within	gives an opponent a 3/3; can't weigh the downside	substitute
Toxic Deluge	X-cost paid with life; no logic to size X	substitute
Greater Good	draw X then discard 3; symmetrical drawback	substitute

Overriding only helps where nobody wrote logic. Where the AI genuinely can't evaluate a card, forcing it on makes it play the card badly — so sim_substitutions.yaml swaps in an AI-playable analogue for simulation only, keeping the deck's shape intact:

Beast Within:  Feed the Swarm
Toxic Deluge:  Blasphemous Edict
Greater Good:  Umbral Collar Zealot

decklist.txt stays the truth; every swap is recorded in run metadata and printed. Note the tradeoff: per-card stats for a substituted card describe the stand-in, not the real card. This buys whole-deck realism at the cost of per-card precision.

3. Default.ai disables the sacrifice fallback entirely. SACRIFICE_DEFAULT_PREF_ENABLE is false in stock Forge, so any sac outlet lacking a card-specific hint is simply never fed — fatal for this archetype. forge-overrides/ai/Combo.ai turns it on and widens the caps (MAX_CREATURE_EVAL 135 → 400, so the AI will feed real creatures rather than only tokens).

python3 tools/batch_run.py --deck decks/<name> --ai-profile Combo --overlay --clock 600

Our seat only — opponents always stay Default, or you just move the baseline.

Measuring: power level, not win rate

Win rate throws away everything that happened in games you lost, and needs 150 games to say anything. tools/power.py scores what the deck did:

DID THE DECK DO ITS THING?
  commander deployed    6/6 games (100%)   median round 5
  engine assembled      5/6 games ( 83%)   median round 14
  life taken from table   mean  105.8
  rounds survived         mean   17.7

A 17% win rate reads as failure; "commander down every game, engine online in 83%, 106 life drained off the table, half the games ran out of clock" is a deck that works. Use these to A/B a change — they move on far fewer games because they score every game.

tools/ab_test.py runs two configs on identical decks and seeds and reports per-metric deltas with confidence intervals, labelling anything inside noise as noise rather than pretending it is a result. Always run --aa first — two identical arms must show no difference; if the null test reports MOVED, the harness is broken.

The honest caveats

Forge's AI is heuristic and, by its own documentation, weak at combo decks. It under-plays exactly the cards an aristocrats deck cares about. The tuning above mitigates this; it does not remove it. A tuned AI is not stock Forge — results are not comparable with untuned runs, which is why every run records its config. Everything here measures what Forge's AI can pilot, which is a proxy for your pod, not a substitute. Cards that need a human to see the line will score badly.

Opponents are stock precons (roughly bracket 2). Your results describe performance against that field.

The AI cannot do politics. In a real pod, threat assessment and table talk decide games. Forge plays a free-for-all somewhat like three parallel duels.

Correlation, not causation. A card cast in long games looks good because long games are ones you survived. Regression helps; it does not eliminate this.

Use the output to generate hypotheses, then test them with a variant. Do not let a win-rate column pick your deck.

---

Running it

# one-time setup on a new machine (data/ is shared across every deck in the repo)
python3 tools/fetch_data.py                                    # ~1.2GB of card data
python3 tools/embed.py                                         # -> data/embeddings.npz (needs GPU or ~20min CPU)
# install Forge 2.0.14 to ~/forge (needs Java 17+)

# configure a deck
$EDITOR decks/<name>/config.yaml                                # commander, pool, constraints, roles
$EDITOR decks/<name>/blacklist.txt                               # cards you won't play
$EDITOR decks/<name>/decklist.txt                                # the 100 cards

# check it, offline and instant
python3 tools/bracket_check.py --deck decks/<name>
python3 tools/combos.py --deck decks/<name>
python3 tools/roles.py --deck decks/<name>

# generate role-budget variants (hypotheses live in decks/<name>/variants.yaml)
python3 tools/generate.py --deck decks/<name> --list
python3 tools/generate.py --deck decks/<name>
# or hand-curated cut/add variants (variants.yaml `handwritten:` section)
python3 tools/make_variants.py --deck decks/<name>

# run the full analysis suite for one variant, optionally with Forge games
python3 tools/variant.py --deck decks/<name> no-mikaeus              # analysis only, ~10s
python3 tools/variant.py --deck decks/<name> no-mikaeus --sim 24     # + 24 Forge games, ~25 min

# batch-simulate every variant in the deck in parallel
python3 tools/batch_run.py --deck decks/<name> --games 150 --workers 16 --heap 2g
python3 tools/compare.py --deck decks/<name>                    # rebuild decks/<name>/decks/README.md

# same thing through the single dispatcher, without cd-ing into the deck
tools/mtg decks/<name> bracket_check
tools/mtg decks/<name> variant no-mikaeus --sim 24

Everything lands in decks/<name>/decks/<variant>/ — decklist, legality check, combo analysis, per-card stats, and every simulation run as .log / .txt / .json. See the top-level README.md for the full repo layout.