Writing
After RecBench, I Care More About Whether Models Can Actually Recommend
Comparative evaluation made me more suspicious of fluent recommendation language and more careful about what the benchmark is truly measuring.
Many large-model papers quietly treat “sounding like a recommender” as a substitute for “being good at recommendation.” A serious benchmark forces those apart. A model may explain beautifully and still fail at ranking, preference tracking, or consistency.
The most important thing RecBench changed for me is not that it produced another benchmark. It sharpened research judgment: what are we truly measuring, and why should we trust that measurement at all?
That judgment matters for collaborators and hiring committees too. It says something about whether a researcher gets seduced by surface intelligence.