Part 5 of 5 · Asset Forge dev blog

Making merges faster

Every merge ran every UI check, so a card that changed one screen still waited about an hour. Picking only the checks a change can affect fixed that for small cards. Most cards still touch shared files, and moving tests to my second PC freed a lock more than it saved time.

8 min read1,657 words

Asset Forge's theme menu open over the AI Image Gen page. It lists eleven themes, from Wasteland Field Kit (ticked) through Pixel Workshop, Clean Studio Light, Blueprint, Retro Desktop, 1996 Screen-Print Zine, Amber Terminal, Craft Notebook, Brutalist Mono and Arcade Sunset to Classic
Eleven themes. The layout check opens every page in all of them at five widths, and until 6 October every merge ran it in full.

At the end of post 4 I said a full merge took about an hour. This post is about why, what I changed, and how much it actually helped. The answer is "a lot for small cards, not much for big ones", and the numbers below show both.

Why a merge took an hour

Every lane reaches main through one script, af_merge.sh. Before it lets anything in, it builds the frontend, runs the full pytest suite and then runs the UI checks: scripts that click through the real app in a browser and fail on anything wrong.

The biggest of those is the layout check. It opens every page in all 11 themes at 5 screen widths and looks for clipped text, overflow and squeezed columns. The themes check is the next biggest. Together they took about 30 minutes of every merge.

Asset Forge's Pixel Snap page in the Pixel Workshop theme: wood-brown header, cream parchment panels with chunky outlined borders, a list of image models with prices on the left, an empty Pixel Snap gallery in the middle and the prompt and estimate panel on the right
The Pixel Snap page in the Pixel Workshop theme.
The same Pixel Snap page in the Blueprint theme: dark navy panels on a faint grid, white monospace text and yellow prices, with the same model list, empty gallery and prompt panel in the same places
The same page in Blueprint. The layout check looks at every page like this in 11 themes and 5 widths, which is why it was the slowest part of a merge.

Before the change, a full run of all 33 UI checks took 3,121 to 3,834 seconds. The whole merge took 58 to 74 minutes. And the merge held the heavy-CPU lock for all of it, because only one heavy job runs on my PC at a time. Nothing else could build or test until it finished.

That would be fine if every card changed every page. Most don't. The card that fixed this put it in one line: "Most cards touch one or two pages."

MERGE-FAST: only the checks a change can affect

MERGE-FAST went in as v1.17.1 on 6 October. It reads the list of files a lane changed and works out which UI checks those files can affect.

  • A tool page, or its CSS, maps to every check that opens that page, plus the checks that open every page.
  • A backend module maps through the modules that import it, to the API routes they serve, to the pages whose code calls those routes.
  • Docs, change notes and test files map to no UI check at all.
  • Shared files always mean the full sweep: the app shell, themes, the main stylesheet, the API client, the merge scripts themselves, and the backend core with everything it imports.

The layout check got its own rule. It runs the changed pages at all 11 themes and 5 widths, plus every other page once, in one theme at one width, so a broken import anywhere still shows up. pytest still runs in full every time.

The rule I cared most about was in the card body: "When unsure, run everything. Never skip a check a changed file can affect." A file no rule knows, a module no route reaches, or an error inside the map itself all fall back to the full run. The merge log prints which checks ran and why, and which were skipped and why. There's a test for the map too, so a change to a tool page has to pick that tool's checks, and a change to the shell has to pick everything.

Measured

For a change that touched only the HUD Kit page, the selected UI checks took 667 seconds instead of 3,121 to 3,834. Adding the normal build and pytest times puts that merge at about 18 minutes instead of 58 to 74. A docs-only lane comes to about 8. The 667 seconds was measured. The 18-minute total adds the usual build and pytest times to it.

MERGE-FAST's own merge couldn't benefit. It changed the lane tools and the UI-check scripts, which are shared files, so it ran the full sweep by design: 4,329 seconds.

The catch

Most cards still run everything.

Since MERGE-FAST, 24 merges have recorded how long they took to get through their tests. 14 of the 24 ran the full sweep. Here they are, one bar per 5 minutes:

text
v1.18.0   72 min  ██████████████            all
v1.19.0   73 min  ███████████████           all
v1.20.0   33 min  ███████                   some
v1.21.0   80 min  ████████████████          all
v1.22.0   85 min  █████████████████         all
v1.23.0   81 min  ████████████████          all
v1.24.0   15 min  ███                       some
v1.25.0  180 min  ████████████████████████████████████  all (waited for locks)
v1.26.0   95 min  ███████████████████       all
v1.27.0   87 min  █████████████████         all   *
v1.28.0   28 min  ██████                    some
v1.29.0   97 min  ███████████████████       all
v1.30.0   14 min  ███                       none (passed a full sweep on the try before)
v1.31.0   98 min  ████████████████████      all
v1.32.0  101 min  ████████████████████      all   *
v1.32.1  176 min  ███████████████████████████████████  all   * (waited for locks)
v1.33.0   93 min  ███████████████████       all   *
v1.33.1  114 min  ███████████████████████   all   *
v1.33.2  126 min  █████████████████████████ some (one check; waited ~100 min for the merge lock)
v1.34.0   12 min  ██                        some
v1.35.0   20 min  ████                      some
v1.36.0   22 min  ████                      some
v1.36.1   25 min  █████                     some  *
v1.36.2   14 min  ███                       some  *

all = every UI check, some = only the affected ones, * = tests ran on my second PC

The selection works. The median of the merges that ran only some checks, or none, is about 21 minutes. The median of the full sweeps is about 94.

The trouble is how often a card touches something shared. The 3D polish card changed the main stylesheet and backend core. The second-PC card changed the backend core, the model map and the lane scripts. The scenery card found a stranger one: it committed sample export files, and because the sheet exporter mentions those file names, the map treated them as shared and ran everything. Its handoff warned the next cards to expect "an approximately 80 min merge".

So MERGE-FAST cut the wait for small cards. It didn't change much for the big feature cards, and those were most of the board for a few days.

A second PC

My other PC came back on 9 October with a new CPU. Before I trusted it with anything, it ran a one-hour stress test with zero failures.

Then a card measured what it could take over. Full pytest took 287 seconds there against 444 to 485 here. Single UI checks ran faster too: 64 seconds for one check that takes 68 to 114 here, 54 for another that takes 667 to 3,048, and 445 for the themes check against 511.

The same card added a remote-heavy slot to the locks. When the other PC is online, pytest and the UI checks go there. When it isn't, they fall back to the heavy-CPU lock on this PC. The card's rule was that an offline second PC must never fail a card; the work just runs here instead. The frontend build, git, swapping in the new build and restarting the live app all stay here.

As a side effect I got a free image route on that PC's graphics card: a real 1024 by 1024 image in 32.8 seconds, at $0.

The first real remote merge

The second-PC work merged as v1.27.0, and that merge was the first to use it. pytest and the UI checks ran on the other machine, and nothing fell back.

pytest took 294 seconds there, sync included, against 444 to 485 here. The full UI sweep took 4,919 seconds against 4,242 to 5,248 here. The whole merge reached "tests done" in 5,236 seconds, against 4,878 to 5,728 for the full-sweep merges just before it.

So it wasn't faster. The layout and themes checks dominate the full sweep, and they took about as long there as here. A full-sweep merge still takes about 87 minutes on either machine.

What changed is where those 87 minutes go. The notes I keep for the agents put it this way: "The real gain is that this PC's heavy-cpu lock stays free for those 87 minutes, so other lanes can build and test here meanwhile." It also spares this PC the long all-core load, which is why the heavy-CPU lock exists.

Of the 15 merges from v1.27.0 on, 7 ran pytest on the other PC and one more ran only its UI checks there. The rest ran here.

What went wrong

The remote box broke merges of its own. The text-to-animation card's first two merge attempts died on filesystem input/output errors on the other PC, halfway through the UI sweep. Main didn't change, which is the point of the gate, but that's two full sweeps lost. Its third attempt ran here and stopped on a timing flake. The fourth merged.

Different machine, different flakes. One guide card's check failed on the remote box and passed here. The remote copy of the code also has no .git folder, so a test that checked git-ignore rules failed there until it learned to skip.

Flaky checks cost whole merges. The Image Expander check sometimes got "stuck on Expanding...". It stopped a merge for the canvas card and another for the help card. Neither card had caused it, and the canvas card's stopped attempt had already run the full sweep.

Merges wait for each other. The 180-minute merge in the chart includes time spent waiting for locks. So does the 176. The 126-minute one ran a single UI check that took 163 seconds, and spent about 100 minutes waiting for the merge lock first. The lock isn't first-come, first-served.

One lane can hold the slot for everyone. On 11 October one card's UI sweep held the remote-heavy slot for over 40 minutes. Another card queued behind it and stopped its run rather than bypass the lock, because the rules say never to. That's the rule working as written, and still a lost run.

Where it is now

The last five merges, v1.34.0 to v1.36.2, took 12 to 25 minutes from start to tests done. They were small cards, so they ran only some checks, and that's exactly the case MERGE-FAST was built for.

I still run two cards at a time. The rule is to raise that only after measuring, and the tighter limit is the Claude and ChatGPT plans, which drain faster with every extra card, not the CPU.

If I did this again I'd put the shared-file problem first. The selection is only as good as the code is split up, and a lot of Asset Forge still lives in a few files that everything touches.