Test and evaluate AI systems, ensuring reliability and accuracy. Collaborate with engineers to resolve issues. Develop and maintain quality metrics for AI systems.
Key Highlights
Key Responsibilities
Technical Skills Required
Benefits & Perks
Nice to Have
Job Description
About LastHire
We don’t just build software — we build intelligence.
At LastHire, we are building the Autonomous Office: AI systems that actively perform business tasks rather than just answering questions. Our team develops production-grade AI agents, RAG architectures, and workflow automation systems that integrate directly into real business environments.
- As we scale, we are looking for a QA Engineer for AI/ML systems who will ensure our models, agents, and pipelines perform reliably in real-world scenarios.
The Role
As an AI/ML QA Engineer, you will be responsible for testing, validating, and improving AI-driven systems before they are deployed to production environments.
You will work closely with engineers building LLM-powered applications and automation systems.
Responsibilities
AI System Testing
- Test LLM responses, prompt chains, and agent workflows
- Validate system outputs for accuracy, reliability, and safety
RAG Validation
- Test retrieval pipelines and vector database results
- Identify hallucinations, incorrect retrieval, and edge cases
Automation Workflow Testing
- Evaluate multi-agent systems interacting with APIs and external services
- Simulate real-world user scenarios
Interested in remote work opportunities in Machine Learning & AI? Discover Machine Learning & AI Remote Jobs featuring exclusive positions from top companies that offer flexible work arrangements.
Quality Metrics
- Define evaluation metrics for AI systems
- Track system performance and failure patterns
Bug Reporting
- Document issues clearly and collaborate with engineers to resolve them
What We're Looking For
AI/ML Knowledge
- Understanding of LLMs and prompt engineering
- Familiarity with RAG systems and vector databases
Technical Skills
- Python basics
- Experience testing APIs or software systems
- Familiarity with tools like LangChain, CrewAI, or similar frameworks is a plus
Analytical Thinking
- Ability to identify edge cases and unusual model behavior
Browse our curated collection of remote jobs across all categories and industries, featuring positions from top companies worldwide.
Communication
- Clear documentation of testing results and system behavior
Bonus Experience
- Experience testing LLM applications
- Knowledge of prompt evaluation frameworks
- Familiarity with vector databases such as Pinecone, Chroma, or Weaviate
Why Join LastHire?
Work on Cutting-Edge AI
You’ll test and evaluate real production AI systems.
Early Team Impact
As an early contributor, your work will shape the quality standards of the entire platform.
Flexible Remote Work
Work from anywhere with a team focused on speed, innovation, and automation.
How to Apply
Send your application or portfolio through LinkedIn or contact us at:
lasthire.ai@outlook.com
Similar Jobs
Explore other opportunities that match your interests
bizmoni - the next gen ai supe...
Senior Technical Client Leadership - Machine Learning
Caylent
