A Sol's Work: Shipping with GPT 5.6 Sol

GPT 5.6 Sol is the first model that passes my walk-away test.

I can hand it hours or days of engineering work, walk away, and come back to something I can ship: merged PRs, tagged releases, verified state, the kind of work I used to split into several supervised sessions.

One overnight session sums it up. Before going to bed, I sent the model one message, verbatim, typos and all:

"i am heading to sleep so i wnat you to verify all open PRs i have with different snapshots… like when i wake up, i should have 100% confidence in those PRs to review."

Four and a half hours later, the report was waiting: 20 open PRs triaged, and 14 of them green with zero unresolved review threads. The other six lived in archived, read-only repositories, and instead of pretending, it flagged them as unmergeable and moved on. The report ended like this:

"I cannot honestly guarantee mathematical 100%, but the 14 actionable PRs now have strong local, hosted, cross-platform, browser, ancestry, and review-thread evidence behind them."

GPT models hedge like that all the time; the hedge is not the new part. I had never woken up to this much finished work that I was confident enough to ship.

And it did ship. Twelve of those PRs were in Rork's monorepo and merged over the following day. One was for App Store Connect CLI, and it went out in the CLI a day later.

I call this the walk-away test:

1. Hand a model work,
2. Leave for hours or even days, and
3. See whether its report still matches reality when you return.

Staying busy that long is the easier half. Passing the test means handling moving state, new commits on main, review comments, CI flakes, and retries without you, and then reporting what actually happened instead of what it wishes had happened.

5.6 Sol is the first GPT model that passes the walk-away test for me.

This post records what changed once I started trusting an agent with work that could outlast my attention. Every example comes from my own work with Sol.

The Numbers First

While writing this post, I went back through my local session history and counted. During my three weeks with Sol, I shipped roughly 23 tagged releases across my repositories, including the App Store Connect CLI and Foundation Lab.

Duration is a weak proxy for quality, yet it shows what I was willing to entrust to the model, because I only let an agent run for hours when I expect something useful at the end.

Before this model, the longest completed agent task I could find in my history, across my Rork work and the App Store Connect CLI, was 47.3 minutes. Not one task crossed an hour.

With 5.6 Sol, 22 completed tasks crossed an hour, 8 crossed two hours, 2 crossed four hours, and the longest single task ran for about 6 hours and 4 minutes.

Counting resumed threads, my longest continuous stretches were 8 to 12 hours of agent work on a single area, and some lanes ran across days, like the Foundation Lab rework and Core AI Lab before 1.0.0 shipped.

That comparison is not apples to apples. I started giving Sol much bigger jobs, which is exactly the behavioral change I am trying to describe: the longest task in my history stopped being an odd experiment and became a normal way to work.

Working in Lanes

Earlier, I gave the model tasks: fix this bug, update this test. Now I give it a lane, an area of responsibility with a goal, and let it decide what work the goal requires. I have left it with a PR queue at Rork, App Store publishing triage, broad repository audits, and release work for my CLIs.

The cleanest example is my App Store Connect CLI. I created automations with Sol so agents can maintain the CLI, while I spend under an hour a week monitoring the work.

Those automations audit open issues and PRs with multiple agents, make fixes in separate pull requests, run the relevant tests, and cut the next release when the queue warrants one.

The longest day of triage ran for more than 8.4 hours. Sol audited every open issue, fixed them in separate PRs, and ran live App Store Connect credential smoke tests. Then it verified checksums and signatures, merged the changes, published ASC 2.x.x, pushed the Homebrew tap, got a WinGet PR through validation, and left an announcement draft in Typefully for me to review.

I still ship at roughly the same pace. I just spend far less of my own attention getting each release over the line.

No Proven Regression

The model executes your literal process, and if that process lives only in your head, it will happily substitute its own. Before handing over a lane, write down what "good" means to you.

Mine boils down to: use current main, avoid pointless tests and AI slop, verify the exact live state, and say "no proven regression" when that is the honest answer.

That last phrase is one of the most useful things 5.6 Sol picked up from me. In a big codebase like Rork's, sometimes the right outcome of a two-hour investigation is proof that an apparent failure is old state, or that there is no reproducible regression at all. Earlier models could not resist inventing a fix anyway.

5.6 Sol, once told that "no proven regression" is an acceptable answer, will actually give it to you. That answer is worth as much as a patch.

Subagents in Worktrees

5.6 Sol is noticeably better at parallel work, and my setup leans into it. I run up to ten agent threads, each in its own git worktree, so parallel fixes do not trample one another. The subagents get philosopher nicknames in the logs, so at various points I had Socrates, Bernoulli, and Kierkegaard reviewing my pull requests.

That is more academic attention than my code has ever received.

Each subagent owns a disjoint unit of work, such as one PR or one screen. The one time I let two agents touch overlapping files, I spent more time untangling their branches than the parallelism saved.

The Boring Ends of the Loop

Most of the hours around a change go into review comments, CI, merges, tags, release notes, and package manager updates. I trust Sol with that whole loop.

During the rework of my Foundation Lab app toward 1.0.0, it carried an 18,000-line change through review hell. It read ChatGPT connector comments on the PR, pushed fixes, waited for checks, read the new comments, and repeated until the unresolved thread count actually reached zero. Then it verified every screen on parallel iOS Simulators across worktrees before I tagged the release.

When I read the report at the end of a long run, I usually interrogate the model like the opposite of /grill-me Skill.

What surprised me is that 5.6 Sol has an innate habit of it. By the end of one overnight PR-queue session, its own final message was:

"I'm waiting for GitHub to prove the 77.78% warning is gone rather than assuming."

That is the sentence I want from an agent: the evidence, plus what it is still waiting on. It lets me merge without redoing the entire review myself.

The Cost of Max Reasoning

Almost everything I did in June ran on max reasoning. My configuration default was medium, but I overrode it constantly. When I spawned subagents for work that mattered, I asked for max there too.

Tokenmaxxing, indeed.

Max reasoning is extremely token-hungry. If you are on a metered plan, you will feel it. But Sol earns its keep in the long loop: reading current state, deciding what is actionable, handling review comments, waiting for CI, retrying, and verifying. A model that stops one check short of the truth simply moves the cost to you at review time.

My rule is simple. Quick questions and small edits get medium. Anything I intend to walk away from, or anything that should end in a merge or release, gets high.

Wrong Turns and Recovery

Every model makes wrong turns. What is different here is what happens after. 5.6 Sol recovers better when I correct it. If it starts from the wrong project, misses a detail, or overstates what has been proven, it usually adjusts and finishes the job instead of defending its first answer.

A production database audit surfaced that a table existed in our schema code but had never made it into production, fallout from earlier non-Sol agent PRs. A whole class of webhook data had silently gone nowhere. My message was not calm.

Sol studied the structure migrations through the merged PRs and cited the convention it found. It split the repair into two clean PRs, with a silent first-sync baseline so the backfill would not spam stale notifications. Both merged green.

When it goes wrong now, I usually keep the session and correct it. Recovery is cheaper than a fresh start.

It can still settle on a plausible explanation too early. It sometimes says "ready" before the final invariant has been proven, and it can fix one path without checking nearby sibling paths. Broad audits sometimes contain good investigation but end with a mushy answer.

The model is good enough to run unsupervised, so make it prove everything it claims before you ship on top of it.

Raw Evidence

Sol is at its best when the authoritative signal is wrong. A dashboard can be green. A check can pass. Even a freshly merged PR can claim to fix the problem. Sol keeps comparing those summaries with reality, and when they disagree, it returns to the raw evidence.

An authentication failure in a production pipeline gave me the clearest example. Two fixes had shipped, a monitoring bot kept saying no regression, and on paper the case was closed. Sol reopened it. It found that the bot was measuring from the wrong deploy time, pulled the real one from CI logs, and ran the comparisons again. It skipped the aggregate error rate and read the raw failing responses. Then it compared our flow with several open-source implementations of the same handshake; ours was the odd one out.

Sol wrote a post-mortem of why the whole team, me included, had missed it for weeks:

"The honest root cause of our miss was anchoring: once the first theory matched the visible fix, every subsequent audit searched for confirmation of that theory. We reviewed fields, not the complete state machine."

I have never had a model diagnose my own confirmation bias, correctly, in writing.

Ask Sol to re-derive what is happening from raw evidence, and tell it explicitly that "everyone before you was wrong" is an acceptable conclusion. It is unusually good at killing the wrong theory instead of shipping another fix on top of it.

One of my side projects hit a different wall. I was building a command-line tool for Apple's Feedback Assistant, but making that app scriptable means dealing with private XPC services and Apple-only entitlements. Sol dissected the installed app, mapped the exact entitlement the tool would need, and wrote a small Objective-C probe to test where access stopped. The system log answered the question:

Process <pid> is not entitled to use this service. Refusing connection.

I pushed it to build the fully headless version anyway, and it held the line. It would not forge Apple credentials or bypass platform security. Instead, it automated the real signed-in app and verified the result against the app's local database.

Before committing to a design in territory you do not know, ask it to map the wall first. "Find out exactly where this stops being possible, and prove it" is a prompt 5.6 Sol can actually execute.

I can hand Sol a messy engineering lane without watching every step, then usually come back to something I can ship. That has changed how I plan my day. Agent work is no longer something I only start when I have time to supervise it; I can schedule my absence too.

Give Sol a lane, walk away for an evening or a whole weekend, and see whether its report still matches reality when you return. Does it pass the walk-away test for you? Let me know on X.

Happy shipping! ☀️