danluu.com
-
Test and verification technique effectiveness with GPT-6 Astra
When we looked at how well agents used test/verification techniques in August1, we saw that GPT-5.6 Sol generally did what you might call a caricature of various techniques when asked to use a technique or library such as TDD, QuickCheck, Lean 4, etc.; not specifying any particular test method (which we'll call Default) resulted in many worthless unit tests, which was one of the best performing conditions because asking agents to test in specific ways mostly made them worse. In that eval, we...
-
“We should be more ambitious with the software we build”
Since early this year, Michael Malis (of pgrust) has been showing me interesting things he's done and saying [W]e should be more ambitious1 with the software we build There is, understandably, a lot of skepticism about projects like pgrust because there are so many low-quality AI-generated projects out there. If I didn't know Malis and didn't look into it, I would just assume that pgrust was another vibe slopped project that's never going to work. But I know Malis well enough to trust his...
- There's no point at which turning your brain off will work
- How well do agents use test and verification techniques?
- Ed Zitron's AI prediction track record
- Bug blindness
-
There's no reason for software to be slow anymore
The other day, I saw a viral tweet saying that people talking about how LLMs are causing slow, bloated, code are going to eat crow once they re-write everything in super-optimized assembly. We're not quite at the point where we want to write everything in assembly, but some variant of what Nolan Lawson said about testing, you can choose how many bugs you want now, which I less eloquently noted here, is becoming more true for performance. In response to a comment in my last post that the cost...
- There's no reason for software to be slow anymore
- The benchmarkpocalypse
- How does programming language affect token efficiency and correctness?
- Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires
- Agentic test processes, LLM benchmarks, and other notes on agentic coding from Galapagos Island
- Steve Ballmer was an underrated CEO
- How good can you be at Codenames without knowing any words?
- A discussion of discussions on AI bias
- What the FTC got wrong in the Google antitrust investigation
- How web bloat impacts users with slow devices
- Diseconomies of scale in fraud, spam, support, and moderation
- Why it's impossible to agree on what's allowed
- Notes on Cruise's pedestrian accident