“WANDR is designed to test whether research agents can discover many qualifying entities and back each one with cited, re-verifiable evidence.”
It’s about time we stopped pretending that “searching” is just a fancy wrapper for a single Google query. For too long, the industry has accepted agents that find the first three relevant links and then stop, acting as if they’ve exhausted the internet. Perplexity is finally putting a number on this laziness.
The new benchmark, WANDR, consists of 500 evidence-heavy tasks. The goal isn’t just to find an answer, but to find all the qualifying entities that fit a specific, complex set of criteria. It’s the difference between asking an agent to find “a good laptop for coding” and asking it to find “every laptop released in 2024 with a 120Hz OLED screen and a 100Wh battery.” One is a suggestion; the other is a research project.
Unsurprisingly, Perplexity’s own “Search as Code” is currently leading the pack. They’ve reported a soft F1 score of 0.363 and a hard F1 score of 0.133. (Which is basically a fancy way of saying it’s barely working).
Who actually believes a benchmark created by the current leader is unbiased? Probably nobody. It’s a classic move: define the metric in a way that favors your current internal architecture, then release it as an “open standard” to force the rest of the industry to play catch-up on your turf.
The real story isn’t that Perplexity is winning; it’s how badly everyone is losing.
Look at that hard F1 score: 0.133. For those who don’t spend their weekends reading evaluation papers, the “hard” metric requires the agent to not only find the entity but to provide a citation that explicitly verifies the claim. The “soft” metric is more lenient. The massive gulf between the two is a loud admission that current agents are mostly guessing or hallucinating the connective tissue between a search result and a final answer.
It’s like a paralegal who finds three relevant cases but misses the ten that actually win the trial. They’ve done the “search” part, but they failed the “research” part.
The gap between soft and hard F1 is embarrassing.
This tells us that the “agentic” workflow we’ve been hyped about for the last year is still incredibly shallow. We’ve built systems that can loop through a few API calls, but we haven’t built systems that can actually verify evidence with any degree of rigor. The friction here isn’t just about the LLM’s reasoning capabilities; it’s about the latency and cost of performing the “wide and deep” search required to hit a high hard F1 score. If you actually want to verify 20 different entities across 50 sources, your token spend and your wait time spike. Most developers have been optimizing for speed, not truth.
By prioritizing “Search as Code,” Perplexity is betting that the path to reliability is through programmatic execution rather than just prompting a model to “be more thorough.” They want the agent to write a script that scrapes, filters, and validates, rather than hoping the model’s internal attention mechanism doesn’t drift after the third webpage.
I suspect this is a preemptive strike. By establishing WANDR as the gold standard for “deep research,” they are trying to move the goalposts away from simple chat benchmarks. Expect a competing “Deep Research” benchmark from OpenAI or Google by Q4 to reclaim the narrative. Until then, we’re stuck with agents that are great at summarizing the first page of Google but terrible at actually doing the work of a researcher.
