Vibe Coding Created a Testing Crisis — WebTestBench Measures Exactly How Badly Agents Fail at Verifying Their Own Web Apps
Vibe coding unlocked the ability for anyone to build a working web app from natural language. It quietly created a harder second-order problem: if an agent builds the app autonomously, how do you automatically verify it works correctly? Static visual checks fail because vibe-coded apps vary too widely in structure. Predefined checklists fail because they can't anticipate what the agent decided to build. WebTestBench formalizes this gap into a measurable benchmark across two cascaded sub-tasks — generating a test specification from the app itself, then autonomously executing that checklist against the live app to catch failures — including the "latent logical constraints" that look correct visually but fail silently at runtime.
The results expose two distinct failure modes in every tested LLM: insufficient test completeness (checklists that miss entire categories of defects, especially around inter-component logic) and detection bottlenecks (agents that generate a complete checklist but then fail to identify violations when running it). The reinforcing problem the authors identify is worth noting: agents that built the app in the first place often lack the documentation and intent signals they'd need to properly test it. For teams with rapidly AI-generated frontends, the checklist generation sub-task is directly extractable as a standalone tool — generate a test spec from your app before you ship it, regardless of what testing infrastructure you run downstream.