‹ Hacker News

Astra 6.2 和 Sol 6.2 在 Simple Bench 上失败

原文:Astra 6.2 and Sol 6.2 flop at Simple Bench

Hacker News 🔥 2 家报道 本站已缓存

英文正文:点「译成中文」可整篇机器翻译(免费通道,不花钱),翻好永久缓存、下次秒开。

文章网址:https://simple-bench.com/ 评论网址:https://news.ycombinator.com/item?id=50588916 积分:74 # 评论:60

Introduction

We introduce SimpleBench , a multiple-choice text benchmark for LLMs where individuals with unspecialized (high school) knowledge outperform SOTA models. SimpleBench includes over 200 questions covering spatio-temporal reasoning, social intelligence, and what we call linguistic adversarial robustness (or trick questions). For the vast majority of text-based benchmarks LLMs outperform a non-specialized human, and increasingly, exceed expert human performance. However, on SimpleBench, a non-specialized human baseline is 83.7%, based on our small sample of nine participants, outperforming every tested LLM, including today's top model, Claude Fable, which scored 81.9% . While we expect model performance to improve over time, the results of SimpleBench confirm that the memorized knowledge, and approximate reasoning retrieval, utilized by frontier LLMs is not always enough to answer basic questions just yet.

For a deeper dive into our results and our methods, check out the full technical report here .

Use all of these models on the Simple Bench app - LMcouncil.ai

打开原文 ↗

正文由本站服务器抓取保存(已自动过滤广告、推荐位与追踪图片),排版与原站可能不同;版权归原站所有。

相关报道同一事件 / 同分类