Research Engineer, Benchmarks

Experience: Technical aptitude and learning potential matter more than years Pay: $100k-$200k, full time Location: San Francisco, CA, US or Singapore (on-site) Visa: Relocation and visa support available for strong candidates The role Build high-quality benchmarks for evaluating frontier agents on domain-specific tasks.

You will create benchmarks that are technically rigorous, practically useful, and credible to frontier labs. What you'll do

• Own the design, implementation, and quality of HUD's internal agent benchmarks
• Work with subject-matter experts to define realistic domain-specific tasks
• Build infrastructure to run models and agents reliably against benchmark tasks

Looking to advance your Devops career with relocation support? Explore Devops Jobs with Relocation Packages that include comprehensive packages to help you move and settle in your new role.

• Develop metrics and analyses for benchmark difficulty, reliability, and failure modes
• Validate whether benchmark performance correlates with real-world evaluations, customer needs, and lab expectations
• Write clear documentation and benchmark reports for technical audiences

What we're looking for

• Proficiency in Python, Docker, and Linux environments
• Published papers or technical blogs about benchmarks, model failure modes, or related topics
• A strong understanding of what makes a benchmark realistic, reliable, and useful
• Experience with environments and evaluations
• Curiosity and the ability to understand how workflows in unfamiliar domains actually work
• First-principles reasoning about task design, scoring, and failure modes

Interested in relocating to United State? Check out our comprehensive Relocation Jobs in United State page with detailed relocation packages and benefits.

Strong signals

• Detail orientation and an eye for subtle inconsistencies and edge cases
• Independent work in unstructured problem spaces
• Early-stage startup experience
• Strong communication across time zones