ASTRA: HackerRank's coding benchmark for LLMs
5 by rvivek | 0 comments on Hacker News.
We help companies hire & upskill developers. A customer recently asked: What % of HackerRank problems can LLMs solve? That got us thinking—how should hiring evolve when AI can translate natural language to code? Our belief: AI will handle much of code generation, so developers will be assessed more on SDLC skills with AI assistants. To explore this, we’re benchmarking LLMs on real-world software dev scenarios—starting with 65 unseen problems across 10 domains. Beyond correctness, we evaluated consistency—an often overlooked aspect of AI reliability. We’re open-sourcing the dataset on Huggingface and expanding it to cover more domains, ambiguous specs, and harder challenges. Would love the HN community’s take on this!
Tuesday, February 11, 2025
Subscribe to:
Post Comments (Atom)
Hello, my name is MD Sihab uddin (SR). I'm a 17 year old owner of srtrick.com
No comments:
Post a Comment