Senior Data Quality Engineer

AI Quality Assurance Specialist

We are seeking a highly skilled professional to join our National Large Language Model (LLM) Project. This key role will focus on establishing and implementing robust data quality frameworks, evaluation methodologies, and quality gates throughout the LLM development lifecycle.

The ideal candidate will ensure our Arabic LLM meets the highest standards of performance, reliability, and cultural appropriateness before being deployed to 20,000 government employees.

• Design and implement comprehensive data quality frameworks specific to Arabic language datasets for LLM training and evaluation
• Establish and enforce quality gates at each project phase (data preparation, model training, evaluation, and RAG implementation)
• Develop detailed acceptance criteria for each phase gate requiring formal sign-off from key stakeholders
• Create and implement quality metrics for data annotation, achieving >90% inter-annotator agreement and >95% cultural/contextual accuracy
• Design and maintain data pipeline quality assurance processes for Arabic text normalization, diacritics standardization, and dialect variation mapping
• Implement Arabic-specific tokenization optimization with >98% vocabulary coverage and >95% morphological accuracy
• Develop comprehensive RAG quality measurement frameworks covering both retrieval metrics
• Establish automated monitoring systems for continuous quality assessment with real-time dashboards
• Create and enforce testing protocols for model evaluation across various Arabic language tasks
• Implement robust regression testing frameworks to ensure model updates maintain or improve quality metrics
• Develop protocols for bias detection and mitigation in both training data and model outputs
• Support the implementation of benchmarking against global standards
• Design human evaluation frameworks to assess model outputs qualitatively
• Collaborate with data annotation teams to ensure high-quality ground truth data
• Participate in weekly quality committee meetings and bi-weekly RAG performance reviews
• Create and maintain quality documentation including processes, guidelines, and acceptance criteria

• Bachelor's or Master's degree in Computer Science, AI, Machine Learning, or related field
• 4+ years of experience in AI/ML quality assurance, with specific focus on natural language processing
• Strong understanding of LLM evaluation methodologies and benchmarking techniques
• Experience establishing quality gates and acceptance criteria for AI systems
• Hands-on experience with data quality frameworks and validation techniques
• Experience implementing multi-level annotation review processes with clear metrics
• Proficiency in designing data pipeline quality assurance systems for Arabic language processing
• Experience with RAG quality assessment covering both retrieval and generation components
• Ability to establish and track performance metrics against benchmarks.
• Experience implementing automated testing frameworks and continuous integration for ML systems
• Strong knowledge of bias detection and fairness assessment in AI systems
• Familiarity with Arabic language and NLP challenges specific to Semitic languages
• Experience with human evaluation protocols and annotation quality assessment
• Proficiency in Python and relevant testing/quality assurance libraries
• Understanding of statistical analysis techniques for model evaluation
• Experience with data annotation platforms and quality control mechanisms
• Knowledge of responsible AI practices and ethical considerations

• Opportunity to contribute to a nationally significant AI project
• Competitive compensation package
• Collaboration with world-class AI teams and researchers
• Professional development opportunities in cutting-edge AI quality assurance
• Chance to establish quality standards for Arabic language AI
• Work with advanced language models and state-of-the-art evaluation techniques