Introduction As Large Language Models (LLMs) move from research prototypes to production systems, the developers of these systems need rigorous performance evaluation. In particular, we need confidence intervals around estimates of system accuracy. However, LLMs introduce a challenge that is unusual for ML systems: they are (operationally) non-deterministic. Even with the temperature set to zero, […]
All posts categorized in: Data Science
%s
Distilling Long-Tail User Behavior into Scalable Embeddings for Job Search
Authors : Marsan Ma, Nikhil Lopes, Raj Amrit, Hong Lu, Dipankar Biswas, Trent KyonoLeadership: Iris Wang, Madhu Kurup Recommendation and ranking systems power many of the most important experiences on large internet platforms. Yet the models that run in production are rarely the largest models we can train. They are usually compact, latency-sensitive supervised models […]
Normalized Entropy or Apply Rate? Evaluation Metrics for Online Modeling Experiments
Introduction At Indeed, our mission is to help people get jobs. We connect job seekers with their next career opportunities and assist employers in finding the ideal candidates. This makes matching a fundamental problem in the products we develop. The Ranking Models team is responsible for building Machine Learning models that drive matching between job […]