Sr. Applied Scientist, Alexa AI

As part of the Alexa AI team, our mission is to provide scalable and reliable evaluation of state-of-the-art Conversational AI. We are looking for a passionate, talented, and resourceful Applied Scientist in the field of Large Language Models (LLMs), Artificial Intelligence (AI), and Natural Language Processing (NLP) to invent and build the end-to-end evaluation of how customers perceive state-of-the-art, context-aware conversational AI assistants.

A successful candidate will have a strong machine learning background, a deep understanding of the conversational AI stack, and a desire to push the envelope in conversational-AI evaluation. As a senior member of the team, you will own ambiguous, high-impact evaluation problems end-to-end - from defining the scientific direction to shipping the models and metrics that millions of customers and the developers who build for them depend on. The ideal candidate has hands-on experience building Generative AI solutions with LLMs, including LLM-as-a-Judge (LLMaaJ), model distillation, Supervised Fine-Tuning (SFT), In-Context Learning (ICL), and Learning from Human Feedback (LHF).

As an Applied Scientist, you will leverage your technical expertise to set the research agenda for how we measure conversational quality, mentor other scientists and engineers, and partner across science, engineering, and business teams to research and develop novel evaluation methods. You will analyze and understand customer experiences using Amazon"s heterogeneous data sources, and design, train, and maintain the evaluation models that serve as the source of truth for assistant quality.

Key job responsibilities

• Own the design, development, and long-term maintenance of flagship quality-evaluation metrics for a state-of-the-art conversational assistant - spanning ground-truth definition, data preparation, model training, and production maintenance.
• Research and build LLM-based evaluators, including LLM-as-a-Judge systems, and distill large judge models into efficient, cost-effective models suitable for scaled online use.
• Set the technical direction for evaluation science and raise the bar for scientific rigor across the team; mentor scientists and engineers and review their work.
• Ensure data quality throughout all stages of acquisition and processing, including data sourcing/collection, ground-truth generation, normalization, and transformation.
• Present proposals and results to partner teams and leadership in a clear manner, backed by data and coupled with actionable conclusions.
• Partner with engineers to develop efficient data-querying and inference infrastructure for both offline and online use cases.

About the team

Central Analytics and Research Science (CARS) is an analytics, software, and science team within Amazon"s Alexa AI organization. Our mission is to provide an end-to-end understanding of how customers perceive the assistants they interact with - from the metrics themselves to software applications to deep dive on those metrics - allowing assistant developers to improve their services. Learn more about Amazon's approach to customer-obsessed science on the Amazon Science website, which features the latest news and research from scientists across the company. For the latest updates, subscribe to the monthly newsletter, and follow the @AmazonScience handle and #AmazonScience hashtag on LinkedIn, Twitter, Facebook, Instagram, and YouTube.