Medium impact

We need more science in the judgment of AI

What happened

The article argues that AI, especially large language models (LLMs), is often overhyped because people judge them based on impressive but unscientific results rather than rigorous, hypothesis-driven evaluation. It warns against practices like HARKing, p‑hacking, and fishing expeditions that can mislead about a model’s true capabilities. The author notes that many impressive demonstrations rely on training data already present in the domain, and that LLMs struggle with niche or corner‑case tasks such as translating specialized code. The piece calls for more scientific testing before making broad claims about AI’s abilities.

Why it matters

If we rely on untested AI claims, we may overestimate its usefulness and miss real limitations. A more rigorous approach could help users set realistic expectations and avoid costly mistakes.

Reality check

An automated, sceptical read of the source

Mostly marketingIncrementalNotableGenuine breakthrough

Incremental. The article focuses on improving evaluation practices rather than introducing a new AI technology.

Who it affectsResearchersDevelopersPolicymakers

What to watchFuture discussions may explore standardized testing protocols for LLMs.

Do you agree with this rating?Agree Disagree

More AI news

Every Way to Get 192GB for Local AI (Without a $9,500 Mac) [video]Industry news · 14 Nov 2026DeepSeek V4 Pro 0813DeepSeek · 14 Nov 2026DeepSeek V4 Pro 0813DeepSeek · 14 Nov 2026Show HN: Portal2Bench – Can AI agents solve Portal 2?Industry news · 14 Nov 2026Anthropic Upgrades Claude Code Projects to Run Parallel AI Workers Hands-FreeAnthropic · 14 Nov 2026

This page was collected from livingsystems.substack.com and summarised automatically by AIPulsee's pipeline. See how it runs.