Applications Manager'',''Applications Manager'',''United States-Massachusetts-Boston-1 Ashburton Place'',''United States-Massachusetts-Boston-1 Ashburton Place'',''Information Systems and Technology'',''Information Systems and Technology'',''Office of the State Auditor'',''Office of the State Auditor'',''Full-time'',''Full-time'',''Day'',''Day'',''Oct 29, 2026, 1:37:04 PM'',''Oct 29, 2026, 1:37:04 PM'',''1'',''1'',''134,991.01'',''89,993.11'',''134,991.01'',''Yearly'',''Marie Tedesco'',''Marie Tedesco'',''6177277200'','''','''','''','''',''Yes'',''Yes'',''false'',''1094553'',''1094553'',''true'',''1094553'',''false'',''Submission for the position: Senior AI Engineer - (Job Number: 260007JK)'',''false'',''1094553'',''false'',''true''. Define and instrument quality and performance measures - groundedness, relevance, citation accuracy (e.g., RAGAS), latency, TTFT, tokens/sec, concurrency, and cost-per-query - with dashboards in Azure Monitor, Log Analytics, and Application Insights, and supply usage and adoption data for the audit workflow KPIs and dashboard maintained by the Manager, System and Process Improvements.