r/AIQuality • • 23h ago

Experiments Lessons from scoring writing quality on 70 LLMs without an LLM judge: one overall rank hides a lot (Gemini 3.5 Flash is 12th sloppiest on email and 9th cleanest on essays)

1 Upvotes

Rank direction first: #1 = sloppiest, #70 = cleanest. Every rank below uses the mechanical score only (padding, templating, rhythm, AI tells), nothing from votes.

I've been running a writing benchmark for a while: 70 models, 77,943 outputs, 112 scenarios across emails, Slack messages, social posts and short essays. I didn't want an LLM grading LLM prose, so the scorer is plain code. A few things I learned that might be useful to anyone building evals:

  • One overall number hides the format. Gemini 3.5 Flash is 32nd overall, but 12th on email and 62nd on essays. Seed 2.1 Turbo is #1 on both email and social posts and 33rd on essays. DeepSeek V4 is 69th on email and 10th on social posts.

  • Price tells you little. Fable 5 and GPT-6 Astra ($10 in / $50 out per million tokens) rank 7th and 10th. Nemotron 3.5 Lightning ($0.08 / $0.20) ranks 69th.

  • Code-only scoring is reproducible (you can rebuild the mechanical ranks from the repo and dataset), but it has blind spots. The biggest one I know of: the mechanical axes miss repeated sentence constructions, which human readers catch quickly. So there is also a blind arena (2,655 votes so far), and the site's combined score blends those in. I'm not quoting combined ranks here because they move daily and many models have few votes.

Other limits: short-form only, no fiction yet.

Links:

I'd like to hear how people here validate a code-based metric against human judgment when you only have a few dozen votes per model.

For context, I'm one of the people behind https://slashy.com, an email client that writes drafts that sound like you. This benchmark started as how we pick models.


r/AIQuality • • 1d ago

Can adding more forensic detectors actually make an AI image detection system less reliable?

1 Upvotes

TL;DR: Digital image forensics means examining an image for clues about how it was created or modified, using metadata, compression artifacts, noise patterns, and other measurable properties. I've been investigating a problem in a multi-detector system: several detectors can react to the same underlying artifact, making the evidence appear stronger than it really is. How do you prevent correlated observations from being counted as independent evidence?

I've been developing a multi-signal, image forensics application and recently re-audited its reasoning engine workings.

 One of the problems I encountered was that collecting more forensic observations could produce conclusions that appeared better supported than they actually were.

 Consider how a real smartphone photograph after being resized and recompressed may cause several detectors to signal frequency-domain irregularities, noise inconsistencies, compression artifacts and unusual texture statistics.

 Straightforward evidence aggregation could interpret these signals as four separate indications of synthetic generation. But they may also be the related consequences of the same processing operations mentioned. So, in this case, four measurements do not necessarily represent four independent pieces of evidence.

 This problem became more concrete during a thorough evaluation of the implementation of the scoring mechanism. Some forensic findings were represented at multiple levels: individual detector artifacts, overlapping fragments and summary observations derived from those same measurements.

 Although these representations were useful in investigating images, the reasoning engine could assign evidential credit to related observations more than once. This could affect base evidence weights, domain-level corroboration bonuses, and mechanism-level score contributions. We therefore revised the evidence aggregation to account for duplicated contributions and preserve the lineage between observations and their underlying measurements.

 Interesting consequences became apparent during regression testing.

 For a test image in our ‘Insta-fake’ dataset, the suspiciousness score remained unchanged at 20.4771 after numerical deduplication. The assessment still indicated localized editing with high-confidence. Its relevant contour and localization measurements retained their contributions. For a Midjourney test image, however, the score changed from 6.0375 → 5.4000, because redundant contributions had been removed. The assessment remained at medium-confidence, with an unverified generation declaration in its metadata.

 Note, these were not calibrated probabilities and the test cases do not demonstrate improved classification accuracy. They were regression checks on the (scoring) behaviour of the reasoning engine. But as such they do highlight an important finding: deduplication should remove redundant evidential credit, not suppress genuinely distinct observations.

 I also encountered a related issue with missing EXIF metadata being always interpreted as suspicious, other early findings too heavily influencing decision routes, and confidence assignments made across different components.

 We enforced clearer evidence lineage, more explicit treatment of marginal corroboration and a centralized confidence policy. The revised scoring logic was checked against the existing regression suite with all tests passing.

 Thus, I addressed identifiable engineering weaknesses but that does not solve the full statistical problem (structural dependence vs statistical dependence). Explicitly tracking shared evidence ancestry is not quite the same as learning the conditional dependence between forensic measurements.

 At a certain point, the challenge becomes less about adding detectors and more about determining how much new information each observation actually contributes.

 I'm interested in how others working on ensemble classification and forensic evidence fusion approach this problem. Some questions.

 How do you handle conditionally dependent signals? Do you explicitly group correlated observations, learn their joint distribution, use probabilistic graphical models, or rely on calibration of the final ensemble?

And how do you distinguish genuine corroboration from multiple measurements of the same underlying phenomenon?


r/AIQuality • • 2d ago

Discussion Has anyone found a good automated testing approach with GenAI?

Thumbnail reddit.com
1 Upvotes

r/AIQuality • • 2d ago

Benchmarked 9 models on extracting data out of documents, scoring wrong values as worse than missing ones. Results and open bench+dataset

Thumbnail
reddit.com
1 Upvotes

r/AIQuality • • 2d ago

What makes a test fail without a bug?

Thumbnail reddit.com
0 Upvotes

r/AIQuality • • 3d ago

When your LangGraph run fails, how do you figure out WHICH node broke it . I built a tool that answers with evidence

Thumbnail reddit.com
1 Upvotes

Genuine question first, because it's why I built this: when a multi-agent run (LangGraph, or a graph calling different models per node) produces a wrong final answer, how do you currently find the cause? Every time I've watched someone debug this, it's hours of reading trace logs by hand — and half the time the actual break was the handoff, a value that changed between one node's output and the next node's input, not the node everyone suspected.
So I built BlameGraph (part of AgentsChecked). You upload a run trace and it reconstructs the actual workflow graph, traces important fields from source to final output, and when a handoff fails it names the responsible step with the supporting evidence — the field, the value before, the value after. The checks are deterministic, no LLM judge: values mutated in transit, fields dropped, fields an agent invented, two agents disagreeing with no resolution, retry loops, outputs nobody consumed, cost outliers, and final-answer values that appear in no step's output. You get a per-agent blame %, a score per handoff, and cost per task. Policy weights are customizable, results are exportable through the API, and a run is reproducible with the same trace, engine and policy versions.
One design choice I'd defend: results are counted separately as passed / failed / insufficient_evidence / not_auditable. If the evidence in a trace isn't enough to judge a handoff, the tool says so instead of guessing either way.
On testing, since a blame tool has to earn trust: the engine (v2.1.0) went through an external review batch that closed at effectively 108/108 checks — including one finding the reviewer later conceded was an error in their own test trace, which the tool had flagged correctly.
It's free during early access (the paid tiers are on the site, struck through at $0 for now):
Full disclosure, also stated on the page: the engine runs server-side, and traces you upload and their results are collected and may be used anonymised — please don't upload anything sensitive.
I'm not a coder by background — I'm the founder; this was built with an AI dev team. What I actually need from this sub: run a real trace through it and tell me where the attribution lands wrong. And what do you export traces as today? That's the next thing I'd build.


r/AIQuality • • 3d ago

Lessons from running a RAG support bot for real Shopify stores: the hard part wasn't retrieval, it was knowing when to say "I don't know"

1 Upvotes

I've been building a customer-support chatbot for Shopify stores (order status, product questions, store policies) using RAG. It's now in a pilot with real merchants. A few lessons that weren't obvious to me at the start:
1. A confident wrong answer is worse than no answer.
In e-commerce, a bot that invents a return policy or a delivery date creates real problems for the merchant. So the core design question became "when should the bot refuse?", not "how do I get better answers?"
2. Retrieval score floors did more than prompt engineering.
I spent a lot of time telling the model "only answer from the context." What actually reduced hallucinations was a minimum similarity threshold: if nothing retrieved clears the floor, the bot doesn't get to answer from vibes. It says it doesn't know and offers a human handoff.
3. Build a refusal test set, not just an accuracy test set.
I keep a set of questions the bot should not be able to answer from the store's data (other stores' policies, made-up products, general-knowledge bait). Every change to chunking, prompts or models gets run against it. This caught regressions that normal Q&A tests missed.
4. Model choice mattered more than I expected for grounding.
On a smaller, cheaper model I kept seeing fabricated details even with good context. Moving to a newer model eliminated most of those failures, and the cost difference was worth it for support use cases.
5. Per-shop isolation has to be enforced in retrieval, not just in the prompt.
With multiple merchants on one system, the filter must sit at the retrieval layer. You can't rely on the model to ignore another store's chunks.
6. Show the merchant why the bot answered.
The admin can see each conversation and which document an answer came from. When the bot gets something wrong, the merchant can see whether it's a missing document or a retrieval problem, and fix it. That built more trust than any accuracy number.
7. Human takeover is a feature, not a fallback.
Merchants wanted to jump into a live conversation at any point. Designing for that from the start made "I don't know" answers acceptable instead of embarrassing.
Curious how others handle this:
How do you pick your score threshold? Fixed value, per-tenant, or calibrated on a test set?
Anyone using rerankers specifically to improve refusal behaviour?
(Disclosure: this is my own product, but not linking it here. Happy to go deeper on any of these.)


r/AIQuality • • 6d ago

Building an AI platform with a small team has been quite a journey.

Thumbnail
reddit.com
1 Upvotes

r/AIQuality • • 7d ago

A user tested my AI platform and caught an interesting problem.

1 Upvotes

I’m building SireIQ, and someone recently tested it with a Bayesian probability problem.
The first answer was correct. But when they changed the numbers and added more conditions, SireIQ reused information from the earlier question instead of recalculating.
What caught my attention was that the response still looked perfectly structured.
It’s a reminder that a convincing AI answer isn’t necessarily a correct one.
I’m looking into this because reliability matters just as much as what an AI can do.
For anyone building with LLMs, how are you testing for context errors across conversations?
SireIQ is free to use for now: https://sireiq.com


r/AIQuality • • 7d ago

How do you catch duplicate emails or bookings when every agent trace looks fine?

Thumbnail reddit.com
1 Upvotes

r/AIQuality • • 8d ago

Discussion PSA: pricing hidden behind tabs can trip up AI agents

1 Upvotes

Small thing but it cost us. Our pricing sat inside a tab component that only renders after a click. Looks fine to a person, but some agent paths were only seeing the plan names with no prices.

A teammate spotted it when we were trying out a couple of agent readiness tools, Lighthouse’s agentic browsing audit and Ora. In the Ora replay the agent landed on pricing and gave up.

Moved the prices into the plain HTML and it was fixed. Curious what other “looks fine to humans” stuff people have found.


r/AIQuality • • 9d ago

Discussion Terminal-Bench 4.1 hardens grading across tasks, re-grading the 4.0 leaderboard trials flipped about 1% of rewards

Thumbnail reddit.com
2 Upvotes

Sharing this here as I thought there may be interest.... Thanks!

The release notes and the write-up covering the changes across 4.0 and 4.1 are here:

Release notes: https://github.com/harbor-framework/terminal-bench/releases/tag/v4.1.0

Write-up: https://snorkel.ai/blog/terminal-bench-4-1-hardens-the-verifiers-behind-the-scores/


r/AIQuality • • 9d ago

How do you explain eval results to non-technical leadership?

1 Upvotes

Your evals say a change is better. How do you turn that into something a VP or CFO acts on? Do you translate into cost, time saved, error rates, or does it never leave the engineering team? Curious what has worked and what got ignored. For context, I’m a founder exploring this problem.


r/AIQuality • • 10d ago

Which interactive benchmarks are actually useful for comparing models?

1 Upvotes

Every model release comes with another graph showing it beating everything else by a few percentage points. Useful, sure, but I’d rather watch what the model can actually do.

Where are the decent, visually exciting benchmarks? Models navigating unfamiliar environments, controlling bodies, building things, or competing at tasks where you can see their mistakes and decisions unfold.

Something where you can watch a model get confused, recover, or completely fail, rather than just seeing its final score.

I’m also interested in how these tests could avoid becoming another target to benchmaxx. Randomized environments? New tasks each run? Hidden objectives?

Obviously, making a benchmark entertaining doesn’t automatically make it rigorous. But why can’t we have both?


r/AIQuality • • 11d ago

AI keeps misreading tables in your documents and how to fix it

2 Upvotes

In one review, 64% of submissions had table or data layout faults that caused AI reading tools to misread the numbers. This wasn't due to incorrect underlying data but when the table was converted to plain text the structure didn't survive, so whatever read it pulled the wrong figure from the wrong row.

That matters more than it sounds like it should, because a table can look perfectly fine to a person and still fail completely the moment something automated parses it. A few things that reliably break it: merged cells, multi-row headers, and numbers aligned with spacing instead of actual table markup.

Easiest check before anything table-heavy goes out: export the document to plain text (or copy-paste it into a basic text editor) and see if the rows and columns still line up logically once the formatting is stripped.

If you’re on the receiving end of submissions like this rather than sending them, worth running the same check before trusting an AI summary or score of someone else’s data. A well-formatted table can make bad numbers look credible, and a badly-formatted one can make good numbers look wrong.


r/AIQuality • • 11d ago

Discussion Finetuning models - share your experience

1 Upvotes

Hi folks, those who are working on finetuning, can you share your experience like why did you choose finetuning for your scenario instead of RAG, domain, challenges you faced etc


r/AIQuality • • 12d ago

Dialectic, Requirements, and Verifiable tests will drastically increase your model's code output quality

Thumbnail
reddit.com
1 Upvotes

r/AIQuality • • 12d ago

Reviewing the rubrics behind rubric-graded benchmarks: missing and misaligned criteria were far more common than vague wording

Post image
1 Upvotes

Sharing in this community as well! Thanks y'all!


r/AIQuality • • 12d ago

Experiments What evals actually caught real problems in your RAG chatbot or AI agent, and how do you handle security and data?

Thumbnail reddit.com
1 Upvotes

r/AIQuality • • 12d ago

Discussion "Evals" is such a blanket term

Thumbnail reddit.com
1 Upvotes

r/AIQuality • • 12d ago

Built a lightweight, open-source RAG evaluation tool — looking for developer feedback

Thumbnail
reddit.com
1 Upvotes

r/AIQuality • • 14d ago

what QA owns once engineers test their own changes with agents

Thumbnail reddit.com
1 Upvotes

r/AIQuality • • 14d ago

Same model, same harmful topics, different refusals depending on whether I asked in English, MSA or Egyptian Arabic

0 Upvotes

I built a small English/Arabic quote service using Gemini and created 63 evaluation cases to test its guardrails.

A few findings from testing with gemini-3.5-flash-lite:

Three unexpected bugs: A model refusal was displayed as a quote, a Chinese word slipped into an Arabic response, and stalled API calls failed requests despite having half their time budget remaining.

Prompt injection: The input checks blocked 15 out of 20 attacks. Gemini ignored the remaining five, as well as all 17 canary attacks tested without guardrails. This suggests the checks provide an additional layer of protection rather than being the only defense.

Multilingual safety: Roughly 40% of harmful requests were refused in each language, but the decisions weren't always consistent. For example, a pipe-bomb request was refused in both Arabic varieties but reframed as an anti-violence quote in English. Only 78% of cases received the same decision across all three languages.

An unresolved issue: A request about secretly tracking an ex-partner's location generated romanticized quotes in every language. Neither the model's safety filters nor my checks detected the harmful intent behind the seemingly harmless wording.

The dataset is small and written by one person, so these results are more of a smoke test than a benchmark.

I'm curious how others approach multilingual safety evaluation. How do you create language-specific test cases instead of simply translating English ones? And how do you evaluate responses that avoid harmful instructions by reframing the request, without manually reviewing every output?

Source code (MIT): https://github.com/1AyaNabil1/swan-llm-service


r/AIQuality • • 14d ago

Your new prompt scored higher. That doesn't mean it's better. I built a free tool that tells you if it actually is

Thumbnail
reddit.com
1 Upvotes

r/AIQuality • • 15d ago

Experiments I compared four decision models and five small LLMs on the same six tasks

Thumbnail
reddit.com
2 Upvotes