Responsibilities:• Direct annotation by managing annotation budgets, sourcing data, and providing statistical analysis of annotations• Maintain and expand metrics used to evaluate agent performance and fairness• Create evaluation packages for Data Scientist and Machine Learning Engineers to use when running experiments• Ensure accurate, regular reporting on evaluation metrics to stakeholders• Manage periodic re-evaluation of production agents using user data and annotationQualifications:Required:• Ability to write reliable, extensible python packages• Demonstrated experience translating high level requirements into useful tools• Excellent knowledge of frequentist and Bayesian statistical techniques• Good written communication skillsPreferred:• Experience with basic principles of deep learning and/or natural language processing• Experience working specifically on automated model evaluationCompany:MCG Health, part of the Hearst Health network, provides unbiased clinical guidance that gives healthcare organizations confidence in their patient-centered care decisions. Manager, Data Science, Mid-Level (Remote) Join to apply for the Manager, Data Science, Mid-Level (Remote) role at Jobright.aiManager, Data Science, Mid-Level (Remote) 2 days ago Be among the first 25 applicantsJoin to apply for the Manager, Data Science, Mid-Level (Remote) role at Jobright.aiJobright is an AI-powered career platform that helps job seekers discover the top opportunities in the US.