Senior Quality Assurance Engineer

About the role

We are seeking a Senior Quality Assurance Engineer to own the quality strategy for our LLM‑powered AI Support Assistant embedded across an EdTech learning platform. The role focuses on ensuring accurate, reliable, and context‑aware conversational experiences for millions of students and instructors.

Key responsibilities

• Design and execute test strategies for conversational flows, measuring response accuracy, relevance, and hallucination prevention.
• Build and maintain evaluation datasets to benchmark assistant performance against target metrics.
• Validate context injection so the assistant correctly uses runtime data such as user role, product, LMS type, and error codes.
• Perform integration and end‑to‑end testing across major LMS platforms (Canvas, Blackboard, Moodle, D2L) and related product lines.
• Develop automated regression suites covering response quality, context propagation, and UI behavior.
• Define performance and load testing approaches for millions of annual interactions.
• Establish monitoring and alerting for production accuracy, escalation rates, and context‑pass‑through failures.
• Collaborate with ML/AI engineers, product managers, and backend teams to translate requirements into testable criteria.

Required profile

• Several years of hands‑on quality engineering experience, preferably in AI or conversational systems.
• Proven ability to design metrics, datasets, and automated test frameworks for large‑scale applications.
• Strong understanding of integration testing across complex LMS ecosystems.
• Experience establishing production monitoring and alerting for AI‑driven services.

Required skills

• AI/LLM conversational testing
• Integration and API testing
• Automated test framework development
• Performance and load testing
• Monitoring and alerting setup
• Evaluation dataset creation
• Knowledge of EdTech/LMS platforms