HackerRank ASTRA | An AI Benchmark for the SDLC
Meet HackerRank ASTRA
Evaluating LLM performance across the SDLC
HackerRank ASTRA* challenges AI models with real projects, testing how well they solve complex software tasks.
*ASTRA (Assessment of Software Tasks in Real-world Applications)
Methodology
The leaderboard shows results from private datasets with project questions created by the same experts who design HackerRank’s developer assessments.
Real-world datasets
The evaluation focuses on the model’s ability to perform complex tasks across the software development life cycle.
Correctness and consistency
To assess the production reliability of the model, we prioritize median standard deviation with k=32, rather than relying on the industry-standard pass@k.
Skill Leaderboards
To assist in model selection for various tasks, performance is reported at both the skill level and on a global leaderboard.
ASTRA Leaderboard
Model Performance
| Model | Avg. Score | Avg. Pass@1 | Consistency |
|---|---|---|---|
| GPT 4.1 | 81.96% | 71.72 | 0.14 |
| DeepSeek-R1 | 81.49% | 69.09% | 0.11 |
| o3-mini | 80.75% | 71.28% | 0.12 |
| DeepSeek-V3 | 77.89% | 64.11% | 0.16 |
| Claude-3.7-sonnet | 77.82% | 69.54% | 0.10 |
| GPT-4.5-preview | 77.46% | 64.91% | 0.13 |
| o1 | 75.80% | 63.92% | 0.11 |
| o1-preview | 75.55% | 60.89% | 0.17 |
| Llama-4-Maverick | 75.44% | 63% | 0.12 |
| Claude-3.5-sonnet | 75.07% | 62.74% | 0.05 |
| Gemini-1.5-pro | 71.17% | 58.15% | 0.13 |
| GPT-4o | 69.52% | 50.91% | 0.20 |
| Gemini-2.5-pro-exp-03-25 | 67.43% | 58.02% | 0.23 |
| Llama-3.3-70B | 61.65% | 46.54% | 0.09 |
Evaluation Metrics
ASTRA assesses AI models with metrics that matter in actual development.
Average Score
Measures how much of a task a model correctly solves on its first try. This shows how well a model handles complex, layered problems.
Average Pass@1
Reflects how often a model delivers a fully correct solution on the first attempt.
Consistency
Measures the mean standard deviation of scores to track how reliably a model performs. Lower numbers mean more consistent results; higher numbers signal inconsistent performance.
Additional Insights
- Claude 3.5 Sonnet leads in consistency, indicating higher reliability over multiple attempts.
- o1 ranks highest in average score and average pass@1, reflecting stronger correctness across tasks.
- Model performance varies by subskill specialization.
For Developers, Data Scientists, and the Hopelessly Curious
Want to roll up your sleeves and get into the weeds? Check out our full report!