"Large language models (LLMs) are increasingly proposed as tools to verify online health information. A common assumptio...

"Large language models (LLMs) are increasingly proposed as tools to verify online health information. A common assumption from the self-consistency and ensemble literature is that agreement across models signals reliability. But it remains untested whether inter-model consensus tracks medical truth. We evaluated 10 LLMs spanning frontier, Russian-specific and medical-specialized families on RuHealth, a corpus of 108 short health claims (40 true, 40 false, 28 disputed) validated through multi-stage physician review. Each claim was classified as true, false, or disputed, in both Russian and a validated English translation, across decoding temperatures and 3 stochastic runs (90,000 judgements). Frontier models were most accurate (macro-F1 64–70) but no model was reliable: “disputed” was assigned to 59.8% of judgements versus a 26% true base rate, while false claims were systematically missed. Majority voting did not help. Three frontier models were unanimous on 90 of 108 claims, yet 31% o...