Job Description
Responsibilities
About the Team
The AI Data Service and Operations (ADSO) team provides safety and non-safety data annotation services, search operation services, and customer services for ByteDance's international products, helping them to build their own data ecological security. In order to optimize user experience while upholding negative content governance on our platforms, the Eco Governance team within ADSO focuses on data labeling work to support online content strategies and AI/LLM model development.
As the AI Model Evaluation Project Lead in the Eco Governance team, you will lead end-to-end delivery of AI data annotation projects and play a critical role in evaluating AI/LLM model performance. You will directly manage a team of AI Project Managers while driving rigorous model evaluation, analyzing results, identifying gaps in model behavior, and delivering clear, actionable recommendations to improve model quality. You will bridge annotation operations with model performance insights to ensure high-quality training/evaluation data translates into measurable improvements in AI capabilities.
Responsibilities
- Lead data annotation and model evaluation projects: Manage end-to-end execution of multiple projects, ensuring both annotation quality targets/SLAs and model performance benchmarks are met.
- Design and execute AI model evaluations: Develop or refine evaluation frameworks, create test cases/datasets (including adversarial/safety-focused ones), run evaluations on LLM outputs, and assess metrics such as accuracy, safety, relevance, bias, and robustness in content governance scenarios.
- Analyze model performance and provide recommendations: Deep-dive into evaluation results, perform root cause analysis on model failures or quality issues, identify patterns in errors, and translate findings into concrete recommendations for annotation guideline improvements, data collection strategies, model fine-tuning, or process changes.
- Serve as the primary stakeholder interface: Translate product, safety, and business needs into clear annotation + evaluation requirements; align on targets and success metrics; and present evaluation insights and recommendations to algorithm, product, and leadership teams.
- Drive delivery governance and cross-functional collaboration: Establish operating rhythms, conduct evaluation reviews, and build escalation frameworks across QA, vendors, annotation teams, and business stakeholders.
- Leverage data for performance management: Monitor dashboards for both annotation and model metrics, detect anomalies, conduct in-depth data and root cause analysis, and drive continuous improvements in quality and efficiency.
- Lead continuous improvement and optimization: Identify gaps between annotation quality and model performance; design workflow enhancements, hybrid (machine + human) labeling strategies, and automation opportunities; partner with tooling and algorithm teams to scale evaluation capabilities.
- Risk and change management: Proactively identify risks related to data quality, model safety, or delivery timelines; propose mitigation plans; and lead operational transitions.
- Deliver strategic reporting: Synthesize annotation and model evaluation data into clear insights, performance summaries, and forward-looking recommendations for leadership and cross-functional partners.
Qualifications
Minimum Qualification(s)
- Bachelor's degree or above.
- At least 5 years of project/program management experience in AI data annotation, large-scale data operations, content moderation, or AI model evaluation environments.
- At least 3 years of hands-on experience in AI/LLM model evaluation, including designing evaluation methodologies, analyzing model outputs, conducting root cause analysis on performance issues, and providing actionable recommendations to improve models.
- At least 3 years of regional team management experience, with proven ability to lead a team of project managers or similar roles.
- Strong proficiency in both spoken and written English, with excellent stakeholder management and communication skills — especially in presenting complex evaluation findings and recommendations to regional/global teams in a fast-paced environment.
- Strong data skills (Excel, SQL, dashboard development) and proven experience using data-driven approaches for performance analysis, root cause investigation, and driving operational/process improvements.
Preferred Qualification(s)
- Majors in Data Science, Statistics, Mathematics, Computer Science, or related fields are preferred.
- Direct experience with LLM training, fine-tuning, or evaluation (e.g., safety alignment, red-teaming, preference/reward modeling, or RLHF workflows).
- Hands-on experience implementing or optimizing machine labeling, hybrid (human + machine) annotation, or automated evaluation pipelines.
- Familiarity with evaluation metrics and benchmarks relevant to content safety and generative AI.
- Project Management certifications (PMP, Agile, Lean Six Sigma) or equivalent.
About TikTok
TikTok is the leading destination for short-form mobile video. At TikTok, our mission is to inspire creativity and bring joy. TikTok's global headquarters are in Los Angeles and Singapore, and we also have offices in New York City, London, Dublin, Paris, Berlin, Dubai, Jakarta, Seoul, and Tokyo.
Why Join Us
Inspiring creativity is at the core of TikTok's mission. Our innovative product is built to help people authentically express themselves, discover and connect – and our global, diverse teams make that possible. Together, we create value for our communities, inspire creativity and bring joy - a mission we work towards every day.
We strive to do great things with great people. We lead with curiosity, humility, and a desire to make impact in a rapidly growing tech company. Every challenge is an opportunity to learn and innovate as one team. We're resilient and embrace challenges as they come. By constantly iterating and fostering an Always Day 1 mindset, we achieve meaningful breakthroughs for ourselves, our company, and our users. When we create and grow together, the possibilities are limitless. Join us.
Diversity & Inclusion
TikTok is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workplace. At TikTok, our mission is to inspire creativity and bring joy. To achieve that goal, we are committed to celebrating our diverse voices and to creating an environment that reflects the many communities we reach. We are passionate about this and hope you are too.
