Submit

Back to the RSS Reader

danluu.com

131 items · latest · The site

  1. Test and verification technique effectiveness with GPT-6 Astra danluu.com ·

    When we looked at how well agents used test/verification techniques in August1, we saw that GPT-5.6 Sol generally did what you might call a caricature of various techniques when asked to use a technique or library such as TDD, QuickCheck, Lean 4, etc.; not specifying any particular test method (which we'll call Default) resulted in many worthless unit tests, which was one of the best performing conditions because asking agents to test in specific ways mostly made them worse. In that eval, we...

  2. “We should be more ambitious with the software we build” danluu.com ·

    Since early this year, Michael Malis (of pgrust) has been showing me interesting things he's done and saying [W]e should be more ambitious1 with the software we build There is, understandably, a lot of skepticism about projects like pgrust because there are so many low-quality AI-generated projects out there. If I didn't know Malis and didn't look into it, I would just assume that pgrust was another vibe slopped project that's never going to work. But I know Malis well enough to trust his...

  3. There's no point at which turning your brain off will work danluu.com ·
  4. How well do agents use test and verification techniques? danluu.com ·
  5. Ed Zitron's AI prediction track record danluu.com ·
  6. Bug blindness danluu.com ·
  7. There's no reason for software to be slow anymore danluu.com ·

    The other day, I saw a viral tweet saying that people talking about how LLMs are causing slow, bloated, code are going to eat crow once they re-write everything in super-optimized assembly. We're not quite at the point where we want to write everything in assembly, but some variant of what Nolan Lawson said about testing, you can choose how many bugs you want now, which I less eloquently noted here, is becoming more true for performance. In response to a comment in my last post that the cost...

  8. There's no reason for software to be slow anymore danluu.com ·
  9. The benchmarkpocalypse danluu.com ·
  10. How does programming language affect token efficiency and correctness? danluu.com ·
  11. Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires danluu.com ·
  12. Agentic test processes, LLM benchmarks, and other notes on agentic coding from Galapagos Island danluu.com ·
  13. Steve Ballmer was an underrated CEO danluu.com ·
  14. How good can you be at Codenames without knowing any words? danluu.com ·
  15. A discussion of discussions on AI bias danluu.com ·
  16. What the FTC got wrong in the Google antitrust investigation danluu.com ·
  17. How web bloat impacts users with slow devices danluu.com ·
  18. Diseconomies of scale in fraud, spam, support, and moderation danluu.com ·
  19. Why it's impossible to agree on what's allowed danluu.com ·
  20. Notes on Cruise's pedestrian accident danluu.com ·

Search posts