r/AIQuality • u/penguinothepenguin • 23h ago
Experiments Lessons from scoring writing quality on 70 LLMs without an LLM judge: one overall rank hides a lot (Gemini 3.5 Flash is 12th sloppiest on email and 9th cleanest on essays)
Rank direction first: #1 = sloppiest, #70 = cleanest. Every rank below uses the mechanical score only (padding, templating, rhythm, AI tells), nothing from votes.
I've been running a writing benchmark for a while: 70 models, 77,943 outputs, 112 scenarios across emails, Slack messages, social posts and short essays. I didn't want an LLM grading LLM prose, so the scorer is plain code. A few things I learned that might be useful to anyone building evals:
One overall number hides the format. Gemini 3.5 Flash is 32nd overall, but 12th on email and 62nd on essays. Seed 2.1 Turbo is #1 on both email and social posts and 33rd on essays. DeepSeek V4 is 69th on email and 10th on social posts.
Price tells you little. Fable 5 and GPT-6 Astra ($10 in / $50 out per million tokens) rank 7th and 10th. Nemotron 3.5 Lightning ($0.08 / $0.20) ranks 69th.
Code-only scoring is reproducible (you can rebuild the mechanical ranks from the repo and dataset), but it has blind spots. The biggest one I know of: the mechanical axes miss repeated sentence constructions, which human readers catch quickly. So there is also a blind arena (2,655 votes so far), and the site's combined score blends those in. I'm not quoting combined ranks here because they move daily and many models have few votes.
Other limits: short-form only, no fiction yet.
Links:
Leaderboard: https://theslopindex.com/leaderboard
Arena (play a few rounds to see how your own judgment compares): https://theslopindex.com/arena
Methodology: https://theslopindex.com/methodology
Open dataset: https://huggingface.co/datasets/hgaddipati/slop-index
Mechanical ranking as one image: https://theslopindex.com/mechanical-card.png
I'd like to hear how people here validate a code-based metric against human judgment when you only have a few dozen votes per model.
For context, I'm one of the people behind https://slashy.com, an email client that writes drafts that sound like you. This benchmark started as how we pick models.