AI/LLM QA Engineer - Agentic and Multi-Agent System Testing

Key responsibilities

• Design and execute test strategies for Agentic and Multi-Agent systems, ensuring reliability and scalability.
• Perform LLM evaluation using metrics such as Exact match / Soft match, BLEU, ROUGE, BERTScore, and semantic similarity using embeddings.
• Develop and maintain prompt testing frameworks and validate output consistency across use cases.
• Implement and evaluate guardrails for responsible AI behavior (hallucinations, toxicity, bias).
• Build automated test pipelines for LLM-driven workflows and orchestration frameworks.
• Collaborate with AI/ML engineers and product teams to define test cases, benchmarks, and acceptance criteria.
• Identify defects, performance issues, and edge cases in AI systems.
• Integrate testing processes into CI/CD pipelines.

Required skills and experience

• Hands-on experience in Agentic & Multi-Agent Testing.
• Expertise in LLM evaluation frameworks and methodologies.
• Strong understanding of prompt engineering validation, guardrails and safety mechanisms, and output evaluation using embeddings and NLP metrics.
• 3+ years in Software QA/Testing.
• Minimum 2+ years of experience in AI/ML or LLM-based systems.
• Proficiency in Python or TypeScript/JavaScript.

Good to have

• Experience with orchestration frameworks such as LangChain, LangGraph, LlamaIndex, DSPy, or OpenAI Assistants/Actions.
• Familiarity with CI/CD tools like GitLab and Jenkins.
• Knowledge of monitoring tools like Grafana and test management/tracking tools like Jira and X-Ray.
• Exposure to AI Quality Engineering practices.