LLM Evaluator – Model Response Analyst (Remote)

About the role

We are looking for a detail-oriented LLM Evaluator (Model Response Analyst) to assess and improve the performance of large language models. Working remotely with a global team, you will analyze AI-generated content for accuracy, coherence, factual reliability, bias, safety and alignment with guidelines.

Key responsibilities

• Evaluate and rank model-generated text using complex rubrics covering factuality, coherence, safety, instruction-following and creativity.
• Compare multiple model responses to the same prompt, select the preferred output and justify your choice.
• Provide concise feedback to modeling and training teams on recurring failure patterns.
• Design adversarial prompts to expose biased, harmful or insecure outputs and help patch safety vulnerabilities.
• Collaborate with QA to refine evaluation guidelines and calibrate scoring standards across the global evaluator pool.
• Investigate under-performing model outputs, hypothesize root causes and flag novel behaviors for the research team.

Required profile

• Minimum 2 years of professional experience in computational linguistics, data analysis, technical writing, NLP quality assurance or cognitive science.
• Bachelor’s degree in Computer Science or a related field.
• Proven experience with Reinforcement Learning from Human Feedback (RLHF) data collection and evaluation.
• Ability to analyze IAA scores, conduct calibration sessions and design A/B experiments for model comparison.

Required skills

• Prompt engineering
• RLHF data collection
• Dataset annotation and cleaning
• Data analysis
• A/B testing for AI models