Claude 3.5 Sonnet New Benchmark Details

Claude 3.5 Sonnet New currently shows benchmark results led by HumanEval (3 / 101, score 93.70), BBH (2 / 19, score 92.60), MMLU (19 / 124, score 88.30).

Benchmark Results

Claude 3.5 Sonnet New

Benchmark Results

Knowledge Exams

2 evaluations

Benchmark / mode

Score

Rank/total

MMLU

Standard Mode

88.30

19 / 124

MMLU-Pro

Standard Mode

74 / 175

Code Generation & Editing

1 evaluations

Benchmark / mode

Score

Rank/total

HumanEval

Standard Mode

93.70

3 / 101

Mathematics

5 evaluations

Benchmark / mode

Score

Rank/total

MATH

Standard Mode

78.30

12 / 42

MATH-500

Standard Mode

44 / 46

AIME 2024

Standard Mode

59 / 61

FrontierMath

Standard Mode

2.10

47 / 60

FrontierMath - Tier 4

Standard Mode

72 / 80

General Reasoning

1 evaluations

Benchmark / mode

Score

Rank/total

BBH

Standard Mode

92.60

2 / 19

Scientific Reasoning

1 evaluations

Benchmark / mode

Score

Rank/total

GPQA Diamond

Standard Mode

200 / 253

Honesty & Factuality

1 evaluations

Benchmark / mode

Score

Rank/total

SimpleQA

Standard Mode

28.40

26 / 46

Repository Engineering

1 evaluations

Benchmark / mode

Score

Rank/total

SWE-bench Verified

Standard Mode

99 / 116

Algorithmic Coding

1 evaluations

Benchmark / mode

Score

Rank/total

LiveCodeBench

Standard Mode

38.70

107 / 125

Writing

1 evaluations

Benchmark / mode

Score

Rank/total

Creative Writing

Standard Mode

1450.80

60 / 110

Commonsense

1 evaluations

Benchmark / mode

Score

Rank/total

SimpleBench

Standard Mode

41.40

66 / 96

Agentic Development

1 evaluations

Benchmark / mode

Score

Rank/total

Aider-Polyglot

Standard Mode

51.60

32 / 59

Maintenance & Optimization

1 evaluations

Benchmark / mode

Score

Rank/total

GSO

Standard ModeTools

4.60

17 / 21

ML Engineering

1 evaluations

Benchmark / mode

Score

Rank/total

WeirdML v2

Standard ModeTools

39.97

47 / 52

Visual Understanding

2 evaluations

Benchmark / mode

Score

Rank/total

GeoBench ACW

Standard Mode

18 / 20

VPCT

Standard Mode

24 / 24

Logical Planning

1 evaluations

Benchmark / mode

Score

Rank/total

BALROG

Standard ModeTools

32.60

10 / 12

Capability Frontier Metrics

1 evaluations

Benchmark / mode

Score

Rank/total

METR Time Horizons v1.1

Standard ModeTools

29.58

20 / 22

Compare with other models