## Meet HackerRank ASTRA

### Evaluating LLM performance across the SDLC

HackerRank ASTRA* challenges AI models with real projects, testing how well they solve complex software tasks.

[View leaderboard](/content/ai/leaderboard#ASTRAleaderboard/index.html)

*ASTRA (Assessment of Software Tasks in Real-world Applications)

## Methodology

The leaderboard shows results from private datasets with project questions created by the same experts who design HackerRank’s developer assessments.

### Real-world datasets

The evaluation focuses on the model’s ability to perform complex tasks across the software development life cycle.

### Correctness and consistency

To assess the production reliability of the model, we prioritize median standard deviation with k=32, rather than relying on the industry-standard pass@k.

### Skill Leaderboards

To assist in model selection for various tasks, performance is reported at both the skill level and on a global leaderboard.

## ASTRA Leaderboard

### Model Performance
| Model                           | Avg. Score | Avg. Pass@1 | Consistency |
|---------------------------------|------------|--------------|-------------|
| GPT 4.1                         | 81.96%     | 71.72       | 0.14        |
| DeepSeek-R1                     | 81.49%     | 69.09%      | 0.11        |
| o3-mini                         | 80.75%     | 71.28%      | 0.12        |
| DeepSeek-V3                     | 77.89%     | 64.11%      | 0.16        |
| Claude-3.7-sonnet               | 77.82%     | 69.54%      | 0.10        |
| GPT-4.5-preview                 | 77.46%     | 64.91%      | 0.13        |
| o1                              | 75.80%     | 63.92%      | 0.11        |
| o1-preview                      | 75.55%     | 60.89%      | 0.17        |
| Llama-4-Maverick                | 75.44%     | 63%         | 0.12        |
| Claude-3.5-sonnet               | 75.07%     | 62.74%      | 0.05        |
| Gemini-1.5-pro                  | 71.17%     | 58.15%      | 0.13        |
| GPT-4o                          | 69.52%     | 50.91%      | 0.20        |
| Gemini-2.5-pro-exp-03-25        | 67.43%     | 58.02%      | 0.23        |
| Llama-3.3-70B                   | 61.65%     | 46.54%      | 0.09        |

## Evaluation Metrics

ASTRA assesses AI models with metrics that matter in actual development.

### Average Score

Measures how much of a task a model correctly solves on its first try. This shows how well a model handles complex, layered problems.

### Average Pass@1

Reflects how often a model delivers a fully correct solution on the first attempt.

### Consistency

Measures the mean standard deviation of scores to track how reliably a model performs. Lower numbers mean more consistent results; higher numbers signal inconsistent performance.

### Additional Insights

- Claude 3.5 Sonnet leads in consistency, indicating higher reliability over multiple attempts.
- o1 ranks highest in average score and average pass@1, reflecting stronger correctness across tasks.
- Model performance varies by subskill specialization.

### For Developers, Data Scientists, and the Hopelessly Curious

Want to roll up your sleeves and get into the weeds? Check out our [full report](/content/ai/astra-reports/index.html)!