Senior AI QA Engineer

Krisp

- Develop and execute AI model evaluation strategies, ensuring accuracy, consistency, and fairness - Implement automated and manual testing for LLM-based applications - Collaborate with the AI Engineer to integrate testing into early-stage development - Build and manage test datasets, ensuring high-quality, diverse, and balanced samples - Develop synthetic data pipelines to enhance model evaluation - Design and maintain hallucination, bias, and robustness detection frameworks - Define and track AI performance metrics (e.g., factual accuracy, coherence, latency, response quality) - Work closely with AI engineers to debug failures, identify root causes, and optimize model performance - Provide feedback on prompt effectiveness, suggest improvements, and collaborate with the Prompt Engineer to refine prompts - Implement continuous monitoring tools to track AI model drift, performance degradation, and unexpected failures - Develop and maintain comprehensive test reports, summarizing findings and recommendations

- Experience with AI/ML testing frameworks and LLM evaluation methodologies - Strong understanding of LLM behaviors, biases, failure modes, and edge cases - Proficiency in Python and familiarity with ML testing frameworks (e.g., PyTest, Unittest, custom ML evaluation tools) - Experience with test dataset management and annotation tools - Familiarity with synthetic data generation and adversarial testing techniques - Strong problem-solving and debugging skills to analyze AI failures and inconsistencies - Strong English language proficiency with the ability to evaluate AI-generated text and improve prompts

All interested candidates are encouraged to apply by sending their CV and additional details to talent@krisp.ai. We highly appreciate all applications, however, only shortlisted candidates will be contacted for the next stages. Please mention in your application that you have learned about this position from MyJob.am