HackerRank ASTRA Report
Introduction
HackerRank’s ASTRA benchmark is composed of multi-file, project-based problems designed to closely mimic real-world coding tasks. The objective is to evaluate the capabilities of advanced AI models across the entire SDLC. The initial release (v1) is primarily composed of frontend development problems and includes frameworks such as Node.js, React.js, Angular.js, Django, Java Spring Boot, Ruby on Rails, and .NET. In v1, the evaluation focuses exclusively on the model's ability to perform new feature development, assessed purely through code generation tasks. Both the input and output in the evaluation framework are text-based. The primary emphasis is on the correctness and consistency of the models, as these are fundamental to real-world applications. Evaluation metrics include average score and average pass@1, with the consistency (median standard deviation) considered as an additional reference.
Features of the HackerRank ASTRA Benchmark:
- Diverse skill domains: the v1 ASTRA Benchmark dataset comprises 65 project-based coding questions primarily focused on front-end, categorized into 10 primary coding skill domains and 34 subcategories.
- Long context multi-file project questions: To mimic real-world development, HackerRank’s ASTRA Benchmark Dataset includes, on average, 12 source code and configuration files per question as model inputs. The problem statements alone typically have an average context length of 718 characters, with source code files, input strings averaging 22,863 characters in length, and output strings averaging 2,744 characters. On average, the benchmark dataset requires 84 lines of solution code per question.
- Model correctness and consistency evaluation: To assess the production reliability of the model, we prioritize metrics such as average scores, average pass@1, and median standard deviation with k=32, rather than relying on the traditional industry-standard pass@k. These metrics provide a more precise evaluation of the model's correctness and consistency in real-world scenarios.
- Wide test cases coverage: HackerRank’s ASTRA Benchmark Dataset contains an average of 6.7 test cases per question, designed to rigorously evaluate the correctness of implementations.
- Abundant content availability: HackerRank's expertise in developer skills is built on its extensive content library with a growing library of over 7,500 questions, advanced skills taxonomy, and data-driven insights. Informed by data from tens of thousands of job descriptions, HackerRank’s Roles Directory—spanning 9 job families, 77 roles, and 260 skills—leverages machine learning to identify key skills for various tech roles, ensuring alignment with real-world industry demands.
Evaluation Leaderboard
| Model | Avg. Score | Avg. Pass@1 | Consistency |
|---|---|---|---|
| GPT 4.1 | 81.96% | 71.72 | 0.14 |
| DeepSeek-R1 | 81.49% | 69.09% | 0.11 |
| o3-mini | 80.75% | 71.28% | 0.12 |
| DeepSeek-V3 | 77.89% | 64.11% | 0.16 |
| Claude-3.7-sonnet | 77.82% | 69.54% | 0.10 |
| GPT-4.5-preview | 77.46% | 64.91% | 0.13 |
| o1 | 75.80% | 63.92% | 0.11 |
| o1-preview | 75.55% | 60.89% | 0.17 |
| Llama-4-Maverick | 75.44% | 63% | 0.12 |
| Claude-3.5-sonnet | 75.07% | 62.74% | 0.05 |
| Gemini-1.5-pro | 71.17% | 58.15% | 0.13 |
| GPT-4o | 69.52% | 50.91% | 0.20 |
| Gemini-2.5-pro-exp-03-25 | 67.43% | 58.02% | 0.23 |
| Llama-3.3-70B | 61.65% | 46.54% | 0.09 |
Based on the analysis of average scores, the models o1, o1-preview, and Claude-3.5-Sonnet-1022 demonstrate superior performance for multi-file, real-world front-end coding tasks. However, due to the high variance within the average scores across 65 questions, a paired t-test reveals that, with the exception of GPT-4o-0513, the differences between model performances are not statistically significant. Despite this, the average score with k=32 indicates a meaningful practical impact in real-world production settings. Similar trends were observed when evaluating the models using the average pass@1 metric.
In our benchmark evaluation, we assessed the Consistency of LLMs using the standard deviation (SD) of their scores across 32 independent runs per question and then evaluated the median SD across 65 questions. The models demonstrated varying levels of performance stability, with Claude-3.5-Sonnet-1022 exhibiting the lowest variability (SD = 0.0497), indicating the highest consistency across problems. The difference between Claude-3.5-Sonnet-1022 and the rest of the models are statistically significant based on the paired t-test.
ASTRA Benchmark Dataset Description
The v1 ASTRA Benchmark Dataset comprises 65 project-based coding questions, systematically categorized into 10 primary coding skill domains and 34 subcategories.
Key Statistics
| Statistic | Value |
|---|---|
| Total project questions | 65 |
| Average number of test cases | 10 |
| Count of sub skill category | 34 |
| Average input files | 12 |
| Average input character length | 22,863 |
| Average problem statement length | 718 |
| Average output character length | 2,744 |
| Average expected lines of code | 84 |
| Average modified code files | 2.3 |
Sample Data and Solution
Here is an example of a RESTful API project from the ASTRA Benchmark Dataset. The task involves developing a RESTful API for managing product records using Node.js and Express, reflecting a common real-world e-commerce development scenario. The project structure is depicted in the following screenshot:
API Endpoints:
- POST /products: Add a new product.
- GET /products: Retrieve existing products.
- PATCH /products: Update product details.
Business Logic Requirements:
- MRP ≥ Price
- Stock > 0
Conclusion
The ASTRA benchmark is instrumental in assessing the capabilities and performance of AI models in coding tasks that reflect real-world scenarios, offering valuable insights for developers and researchers alike.