Introduction As Large Language Models (LLMs) move from research prototypes to production systems, the developers of these systems need rigorous performance evaluation. In particular, we need confidence intervals around estimates of system accuracy. However, LLMs introduce a challenge that is unusual for ML systems: they are (operationally) non-deterministic. Even with the temperature set to zero, […]
All posts categorized in: Analytics
SHAP Plots: The Crystal Ball for UI Test Ideas
Have you ever wanted a crystal ball that would predict the best A/B test to boost your product’s growth, or identify which part of your UI drives a target metric? With a statistical model and a SHAP decision plot, you can identify impactful A/B test ideas in bulk. The Indeed Interview team used this […]
What’s Up, ASF? Using Imhotep to Understand Project Activity
As I described in an earlier post, we built Imhotep as a data analytics platform for the rapid exploration and analysis of large time-series datasets. In the previous post, I showed how an Imhotep dataset based on Atlassian Jira can drive improvements to the development process. We’re continually searching for new ways to collect metrics. […]