HumanSignal

You can't unit test whether a website feels right. You have to ask people. For this study we wrote a rubric, ran 450 head-to-head comparisons with ten votes each, and ranked the models with a Bradley-Terry fit. The full method and raw data are public if you want to check our work.

One model topped our boutique hotel brief and came in tenth out of ten on a science publication. Same model, same evaluators, same rubric. Average it into one leaderboard number and you'd never see that. That's the case for running evals with people.

We gave ten AI models the same ten website briefs, from a boutique hotel to a telemedicine app, then asked people to pick the better design, two at a time. 4,500 votes later, here's where every model landed. All 100 designs are live on the page, along with the rankings and the raw data.

I've been going through the color explanations from our AI taste study (all 563 of them) and one pattern keeps showing up. A lot of them don't really say what the color is for. They say what it isn't. "Without the coldness of hospital blue." "Rather than corporate-cold." About 36% of them are written that way. Take the telemedicine brief, for example. The models were trying hard not to look like a hospital, and 8 of 10 landed on the same deep teal. Two of them were one hex digit apart. It's a good reminder that "avoid the cliché" is also something a model can learn, and once they all learn it, it stops being an escape. This blog dives deeper: https://lnkd.in/g7UuJdJf

We gave ten AI models the same website brief and asked people to pick the better design. Then we changed the brief. The model that won the boutique hotel brief finished dead last on a science publication. One leaderboard score would have hidden that, which is why we run evals with people. All 100 designs and the full rankings are on the page.

The model people liked most for a science publication finished last when we asked it for an Italian soda pitch deck. Same model, same evaluators, same rubric. Only the brief changed, and a single benchmark score would never show it. See all 100 designs and how every model ranked.