ASTRA: HackerRank's coding benchmark for LLMs
5 by rvivek | 0 comments on Hacker News.
We help companies hire & upskill developers. A customer recently asked: What % of HackerRank problems can LLMs solve? That got us thinking—how should hiring evolve when AI can translate natural language to code? Our belief: AI will handle much of code generation, so developers will be assessed more on SDLC skills with AI assistants. To explore this, we’re benchmarking LLMs on real-world software dev scenarios—starting with 65 unseen problems across 10 domains. Beyond correctness, we evaluated consistency—an often overlooked aspect of AI reliability. We’re open-sourcing the dataset on Huggingface and expanding it to cover more domains, ambiguous specs, and harder challenges. Would love the HN community’s take on this!
Tuesday, February 11, 2025
Tags
# Hacker News
![]()
About Sr officials
Templatesyard is a blogger resources site is a provider of high quality blogger template with premium looking layout and robust design. The main mission of templatesyard is to provide the best quality blogger templates which are professionally designed and perfectlly seo optimized to deliver best result for your blog.
Newer Article
New top story on Hacker News: Hackers leak cop manuals for departments after breaching major provider
Older Article
New top story on Hacker News: Show HN: Open-source Hacker News apps
Subscribe to:
Post Comments (Atom)
Hello, my name is MD Sihab uddin (SR). I'm a 17 year old owner of srtrick.com
No comments:
Post a Comment