AI Quality Assurance Specialist

This is an exciting opportunity to work on a nationally significant AI project, contributing to the development of cutting-edge technologies and innovations in the field.

• We are seeking a highly skilled Quality Assurance (QA) Engineer with expertise in Artificial Intelligence (AI) systems. Your primary responsibility will be to ensure the quality and reliability of our Large Language Model (LLM) by designing and implementing robust data quality frameworks, evaluation methodologies, and quality gates throughout the LLM development lifecycle.

The ideal candidate will have a strong understanding of Natural Language Processing (NLP) challenges and Semitic language intricacies. You will be responsible for establishing and enforcing comprehensive data quality frameworks, evaluating LLM performance, and ensuring cultural relevance before deployment.

1. Design and implement data quality frameworks for LLM training and evaluation.
2. Establish and enforce quality gates across all phases: data preparation, model training, evaluation, and RAG implementation.
3. Define acceptance criteria and secure stakeholder sign-offs at each quality gate.
4. Develop annotation quality metrics targeting >90% inter-annotator agreement and >95% contextual/cultural accuracy.
5. Implement QA processes for text normalization, diacritics standardization, and dialect mapping.
6. Optimize tokenization with >98% vocabulary coverage and >95% morphological accuracy.
7. Develop robust RAG (Retrieval-Augmented Generation) quality measurement frameworks.
8. Build real-time dashboards for automated quality monitoring and alerts.
9. Define and execute testing protocols for evaluation across diverse NLP tasks.
10. Establish regression testing frameworks to safeguard model quality during updates.
11. Design processes for bias detection and mitigation in training data and outputs.
12. Lead human evaluation campaigns to assess qualitative aspects of model responses.
13. Maintain thorough documentation on QA processes, metrics, guidelines, and acceptance criteria.
14. Participate in weekly quality governance meetings and RAG evaluation reviews.

• Bachelor's or Master's degree in Computer Science, AI, Machine Learning, or related field.
• 4+ years of experience in AI/ML quality assurance, with an emphasis on NLP and LLMs.
• Proven experience with LLM evaluation frameworks and benchmarking techniques.
• Expertise in setting quality gates and defining acceptance criteria for AI systems.
• Strong understanding of NLP challenges and Semitic language intricacies.
• Experience with multi-level annotation reviews and metrics-driven QA.
• Skilled in RAG system evaluation (retrieval and generation components).
• Proficient in Python and testing/QA libraries for machine learning.
• Background in statistical analysis, human evaluation, and continuous quality tracking.
• Understanding of fairness, bias detection, and responsible AI practices.

• Experience in government or enterprise-scale LLM evaluations.
• Knowledge of NLP benchmarks and datasets.
• Experience using platforms like Scale AI, Human loop, or similar tools.
• Familiarity with hallucination detection, prompt quality assessment, and factual consistency validation.
• Knowledge of ROUGE, BLEU, BERT Score, and custom evaluation metrics.
• MLOps experience including CI/CD quality gates and A/B testing frameworks.
• Background in security, privacy, and UX testing for AI systems.
• Participation in quality governance boards or national standards bodies.

• Opportunity to contribute to a nationally significant and socially impactful AI project.
• Competitive compensation package.
• Collaboration with leading researchers and engineers in LLM development.
• A fast-paced, innovation-driven environment that values ownership and growth.
• Professional development in cutting-edge AI quality assurance.

We are committed to unlocking community knowledge in a new way, leveraging cutting-edge technologies to develop scalable, high-performance solutions for our clients across industries.