NewSenior AI Engineer JobotSenior AI EngineerNew York, NY$225,000–$280,000 / yearInformation collected and processed as part of your Jobot candidate profile, and any job applications, resumes, or other information you choose to submit is subject to Jobot's Privacy Policy, as well as the Jobot California Worker Privacy Notice and Jobot Notice Regarding Automated Employment Decision Tools which are available at jobot.com/legal. The product sits at the intersection of AI, automation, and real world impact, using LLMs and agentic systems to replace hours or weeks of manual work with intelligent, scalable software.
Generative AI Training Specialist IconmaGenerative AI Training SpecialistNew York, NY$84.66–$89.66 / hourBuild and deliver training on AI coding assistants such as Claude Code and OpenAI Codex, helping engineering and technical teams learn to use these tools effectively in their day-to-day development work. Responsibilities: Design and facilitate live and on-demand training on generative AI tools, including (but not limited to) client GPT, ChatGPT Enterprise, Microsoft Copilot, and Claude.
Accounting Manager (AI, Data Analytics) Vaco LLCAccounting Manager (AI, Data Analytics)New York, NY$150,000Determining compensation for this role (and others) at Vaco by Highspring depends upon a wide array of factors including but not limited to: the individual’s skill sets, experience and training; licensure and certification requirements; office location and other geographic considerations; other business and organizational needs. Determining compensation for this role (and others) at Vaco/Highspring depends upon a wide array of factors including but not limited to the individual’s skill sets, experience and training, licensure and certifications, office location and other geographic considerations, as well as other business and organizational needs.
Sr. SW Engineer | AI Applications | MS Foundry | Hybrid in Midtown Living TalentSr. SW Engineer | AI Applications | MS Foundry | Hybrid in MidtownNew York, NY$200,000–$250,000 / yearBuild AI-driven applications used daily within a global professional services environment Hands-on with Copilot Studio, Azure OpenAI, and custom development. The Role Design and deliver custom AI applications integrated into daily business workflows.
NewSenior AI Enablement Specialist Balyasny Asset Management, LPSenior AI Enablement SpecialistPECK SLIP, NY$190,000–$225,000 / yearWork with data science; scoping, developing, and deploying LLM-based and data science solutions; building LLM-based applications; Python for data manipulation, statistical modeling, and machine learning; and, SQL for database querying. Senior AI Enablement Specialist (New York, NY): Embed with trading desks to maximize the value derived from generative AI and data science tools.
NewGenerative AI Training & Support Specialist JobotGenerative AI Training & Support SpecialistNew York, NY$80–$90 / hourInformation collected and processed as part of your Jobot candidate profile, and any job applications, resumes, or other information you choose to submit is subject to Jobot's Privacy Policy, as well as the Jobot California Worker Privacy Notice and Jobot Notice Regarding Automated Employment Decision Tools which are available at jobot.com/legal. Reporting to the Senior Manager of User Engagement, Education & Tech Academy, this individual will be responsible for designing, delivering, and supporting generative AI training and enablement programs across the enterprise.
IT Project / Program Manager (PMP / AI & Cloud platform) TRIGYN TECHNOLOGIES, INC.IT Project / Program Manager (PMP / AI & Cloud platform)White Plains, NY$75–$78 / hourContractorThe successful candidate will be responsible for ensuring the successful delivery of projects and use cases in this program by coordinating across teams, managing timelines, budgets, and resources, and maintaining alignment with organizational goals. The ideal candidate will possess deep expertise in IT project management, AI technologies, cloud platforms, and governance structures.
NewIT AI Portfolio Manager Blue Ribbon Global TechnologiesIT AI Portfolio ManagerMorris Plains, NJCollaborate with business and IT stakeholders to ensure successful delivery of AI initiatives. Required Skills: Basic Qualification: Additional Skills: Background Check: No.
AI Governance Lead Yantran LLCAI Governance LeadWarren, NJday to day driver of an enterprise AI governance implementation programme, co leading delivery with the client s Privacy and Security leadership. *** represents the connected world, offering innovative and customer-centric information technology experiences, enabling Enterprises, Associates, and Society to Rise.
Python AI Engineer VeeRteq Solutions Inc.Python AI EngineerMorris Plains, NJBuild extensible API Integrations, low code UI/UX solutions, with extremely short cycle times, to extract information from sources, integrate with GPT4, receive insights and make them available in intuitive, high performing dashboards Build solutions that align with responsible AI practices. Work with Product Development as a Generative Artificial Intelligence (AI) subject matter expert and architect and develop scalable, resilient, ethical AI solutions Strong engineering skills to design the output from the AI with nodes and nested nodes in JSON or array, HTML formats as required.
Machine Learning / AI Contractor: 10+ years (Expert) ApolisMachine Learning / AI Contractor: 10+ years (Expert)New York, NYRemote$72–$75 / hourTool & System Integration: Architect the "connective tissue" between our AI agents and live enterprise infrastructure (e.g., Oracle database Google Workspace, Jira, Salesforce, BigQuery) using Model Context Protocol (MCP) servers and custom APIs and Oracle Database MCP server. • Complex Workflow Automation: Develop long-running agents capable of managing multi-day, multi-step workflows, utilizing complex reasoning patterns (e.g., ReAct, hierarchical delegation, self-reflection).
Cloud Security Architect (AI) ApolisCloud Security Architect (AI)New York, NY$60 / hourThe AI Security Team is responsible for establishing secure-by-design principles for AI and Agentic AI systems, ensuring appropriate controls, governance, monitoring, and regulatory alignment across platforms such as ChatGPT Enterprise, AWS Bedrock, Databricks, and other enterprise AI services. The role will serve as a hands-on technical contributor responsible for defining, implementing, and validating security controls and solutions that protect AI and data development, deployment, and usage across the enterprise.
Staff AI Agent Engineer - Moveworks | Implementations & Partner Success ServiceNow IncStaff AI Agent Engineer - Moveworks | Implementations & Partner SuccessNew York, NY$137,700–$241,000 / yearAs a AI Agent Engineer (AIAE), specializing in strategic implementations and partnership initiatives at Moveworks, you'll be responsible for ensuring a successful technical delivery of the Moveworks platform for new customers and initiatives, working alongside Implementation Managers and the broader Partner ecosystem to understand customer requirements, perform discovery, and drive all technical aspects around our post-sales engagements. Experience designing, building, and launching full-stack workflows and automations leveraging REST APIs, iPaaS tools (Workato, Azure Functions, AWS Lambdas, ServiceNow Flow Designer), or scripting languages (Python, JavaScript, Golang).
Solution Engineer - Cloud & AI Infrastructure Microsoft CorpSolution Engineer - Cloud & AI InfrastructureNY$106,400–$203,600 / yearRequired Qualifications: 5+ years technical pre-sales or technical consulting experience OR equivalent experience OR Bachelors Degree in Computer Science, Information Technology, or related field AND 4+ years technical pre-sales or technical consulting experience OR Masters Degree in Computer Science, Information Technology, or related field AND 3+ years technical pre-sales or technical consulting experience. Build trusted advisor relationship with customers' technical decision makers and use proactive effort to find and understand customers' pain points and work together with customers to co-develop secure & resilient solution architecture for production scale delivery using Cloud Adoption Framework (CAF) best practices including Unified Support and Cloud Migration Factory (CMF) offer for every opportunity.
Software Engineer, AI Engineering And Enablement Bristol Myers SquibbSoftware Engineer, AI Engineering And EnablementPrinceton LVL, NJ$96,560–$117,008 / yearBased on eligibility*, additional time off for employees may include unlimited paid sick time, up to 2 paid volunteer days per year, summer hours flexibility, leaves of absence for medical, personal, parental, caregiver, bereavement, and military needs and an annual Global Shutdown between Christmas and New Years Day. Each of us is empowered to apply our individual talents and unique perspectives in a supportive culture, promoting global participation in clinical trials, while our shared values of passion, innovation, urgency, accountability, inclusion and integrity bring out the highest potential of each of our colleagues.
NewDirector, Applied AI PfizerDirector, Applied AINew York City, New York$176,600–$294,300 / yearCandidate demonstrates a breadth of diverse leadership experiences and capabilities including: the ability to influence and collaborate with peers, develop and coach others, oversee and guide the work of other colleagues to achieve meaningful outcomes and create business impact. 3) Integration Ownership — Ensure seamless integration across platforms and services, data systems, AI models, and business workflows — connecting models to real use cases, guardrails to runtime systems, and observability to operational feedback loops within enterprise constraints.
Generative AI Engineer IPolarityGenerative AI EngineerWhippany, New JerseyIPolarity LLC integrates expert industry knowledge, process and technology frameworks, strong partnerships, and a reliable work force to provide strategic solutions that generate sustainable results. We provide our clients in key industry verticals with focused expertise in IT Solution, Application Development, Application Maintenance, Testing services and Systems Integration.
AI Partnerships, Director Salesforce.com, Inc.AI Partnerships, DirectorNew York, NY$197,300–$313,700 / yearBridge Product, GTM, and Operations: Collaborate across Product, Legal, Marketing, and Operations to define scalable partnership models and integration frameworks. Build and Lead Strategic, Complex Partnerships: Source, structure, and close high-value partnerships that advance Salesforce's AI and ecosystem vision.
Manager/Senior Manager, AI Solutions Architect Salesforce.com, Inc.Manager/Senior Manager, AI Solutions ArchitectNew York, NY$148,500–$260,100 / yearOwn agent identity and session design - author and maintain agent context files (identity files, session startup protocols, memory structures, and skill definitions) that govern how agents behave consistently across sessions, users, and evolving workflows. LLM context file design as a craft - knows how to write durable agent context (identity, memory, startup protocols, skill definitions) that holds up across sessions - not one-off prompts, but role-level design.
NewSpecialist Account Executive, AI & Data Driven Agentic Transformation Salesforce.com, Inc.Specialist Account Executive, AI & Data Driven Agentic TransformationNew York, NY$99,500–$214,400 / yearYou will operate as a trusted advisor, guiding customers through complex, multi-cloud solutions and demonstrating how Salesforce delivers measurable business value through AI-powered outcomes. We are seeking a driven, technically fluent sales professional with experience selling complex, multi-product SaaS solutions and a strong understanding of AI and data-driven technologies.
Applied AI Designer Thread AIApplied AI DesignerNew York, New York$135,000–$200,000 / yearHeadquartered in New York, our growing team is a group of AI, product, and engineering experts, who have a track record of creating and executing against complex workflows and infrastructure. You will be responsible for creating innovative, intuitive, and user-centered experiences that make complex and powerful AI workflows simple and accessible to profiles across the technical spectrum.
Applied AI Strategist Thread AIApplied AI StrategistNew York, New YorkThis role requires a curious and analytical approach, a sharp intuition for value-oriented agentic workflows, and a strong degree of user empathy to ultimately empower our customers to build better. Headquartered in New York, our growing team is a group of AI, product, and engineering experts, who have a track record of creating and executing against complex workflows and infrastructure.
AI Forward Deployed Engineer ArloAI Forward Deployed EngineerNew York City, New YorkPrototypes or demos do not measure success in this role; it’s measured by operational leverage: faster workflows, fewer manual processes, lower costs, and teams that can scale far more efficiently because of the systems you deployed. At most insurance companies, teams such as underwriting, claims, premium finance, member support, care navigation, and sales operations each rely on extensive ecosystems of external vendors and legacy SaaS tools.
Software Engineer - AI BrelliumSoftware Engineer - AINew York City, New York$140,000–$200,000 / yearWe’ve built AI-powered technology that helps healthcare providers deliver safer, higher-quality care - starting with the first real-time medical review platform built to fix clinical and compliance risks before they impact patients. Brellium is building the AI-powered platform that helps providers deliver safer, more consistent care by mitigating risk early and aligning patient visits with clinical best practices.
Applied AI Engineer RampApplied AI EngineerNew York City, New YorkYou will work on exciting projects such as AI Agents, Retrieval-Augmented Generation, Structured Extraction (we made https://github.com/1rgs/jsonformer ), internal tooling for customer-facing teams, fine-tuning models, and build infrastructure for LLM inference. We automate how over $200B in annualized spend flows in and out of 70,000+ companies: authorizing payments, flagging risk, categorizing spend, and closing books.
Lead AI Engineer, Data Solutions Salesforce.com, Inc.Lead AI Engineer, Data SolutionsNew York, NY$172,500–$260,100 / yearThis role focuses on developing intelligent decisioning systems and building an agent flywheel-a system of feedback loops that continuously evaluate, optimize, and improve agent performance over time. The world of work as we know it is changing and we're looking for Trailblazers who are passionate about bettering business and the world through AI, driving innovation, and keeping Salesforce's core values at the heart of it all.
AI Forward Deployed Engineer Railway Health IncAI Forward Deployed EngineerNew York City, NY$180,000–$225,000 / yearPrototypes or demos do not measure success in this role; it's measured by operational leverage: faster workflows, fewer manual processes, lower costs, and teams that can scale far more efficiently because of the systems you deployed. At most insurance companies, teams such as underwriting, claims, premium finance, member support, care navigation, and sales operations each rely on extensive ecosystems of external vendors and legacy SaaS tools.
Client Success Specialist- AI The Circana GroupClient Success Specialist- AINY$90,000–$100,000 / yearThis role bridges product support, user experience, and development by investigating issues, coordinating resolutions, coaching commercial teams, and managing communication between users, client service teams, and product development. An employee's position within the salary range will be based on several factors including, but not limited to, relevant education, qualifications, certifications, experience, skills, seniority, geographic location, performance, shift, travel requirements, sales or revenue-based metrics, any collective bargaining agreements, and business or organizational needs.
NewGenAI / Agentic AI Developer Capgemini SEGenAI / Agentic AI DeveloperNew York, NY$80,000–$101,050 / yearImportant Notice: Compensation (including bonuses, commissions, or other forms of incentive pay) is not considered earned, vested, or payable until it becomes due under the terms of applicable plans or agreements and is subject to Capgemini's discretion, consistent with applicable laws. It delivers end-to-end services and solutions leveraging strengths from strategy and design to engineering, all fueled by its market leading capabilities in AI, generative AI, cloud and data, combined with its deep industry expertise and partner ecosystem.
AI Engineering, Vice President (New York, New Jersey, Toronto) The Goldman Sachs Group IncAI Engineering, Vice President (New York, New Jersey, Toronto)New York, NY$115,000–$180,000 / yearJoin our engineering teams that build massively scalable software and systems, architect low latency infrastructure solutions, proactively guard against cyber threats, and leverage machine learning alongside financial engineering to continuously turn data into action. We''re committed to fostering and advancing diversity and inclusion in our own workplace and beyond by ensuring every individual within our firm has a number of opportunities to grow professionally and personally, from our training and development opportunities and firmwide networks to benefits, wellness and personal finance offerings and mindfulness programs.
NewAI Context Operations Lead Mercury Technologies IncAI Context Operations LeadNew York, NY$163,000–$203,800 / yearBehind every product Mercury ships is a fast-growing company: dozens of teams across engineering, product, design, data, and business functions, and hundreds of projects that require alignment across compliance, legal, partnerships, customer support, finance, and leadership. Working closely with Engineering, who own the underlying infrastructure, you''ll design the knowledge layer above it: the taxonomies, schemas, validation workflows, and automations that make company knowledge reliable for both people and AI systems.
Principal Solution Architect, AI Agents Wolters Kluwer N.V.Principal Solution Architect, AI AgentsNY$146,900–$262,400 / yearIn addition to creating proof-of-concepts to demonstrate best practices and accelerate adoption, you'll play a key role in driving the evolution of the GenAI platform by identifying gaps, proposing enhancements, and collaborating with platform engineering teams to deliver new capabilities. Help product engineering teams design and implement scalable AI solutions that leverage WKs GenAI platform to drive impact in healthcare, legal, tax, and compliance industries.
Principal Java Microservices Architect - AI & Agentic Platforms Publicis Groupe SAPrincipal Java Microservices Architect - AI & Agentic PlatformsNew York, NY$165,000–$205,000 / yearResponsibilities Hands-on and possesses deep expertise in: Enterprise Java microservices architecture, Agentic AI systems & tool-calling backend frameworks, AI orchestration frameworks, Cloud-native distributed architectures, Enterprise automation platforms, and High-throughput backend system engineering. This role requires strong experience with: Java 17+/21+ and Spring Boot Google GCP and AI ecosystem, Agent Development Kit (ADK) & Model Context Protocol (MCP), Modern AI application microservices architectures.
Quantitative AI Strategist DRW Holdings LLCQuantitative AI StrategistNY$175,000–$250,000 / yearPrototype and validate quantitative workflows end-to-end - from data retrieval and signal construction through to strategy evaluation, PnL simulation, testing, and risk/scenario analysis - while defining how the AI should interact with data sources, analytics libraries, desk-specific tools, etc., and work with engineers to deliver them as production platform capabilities. Your job is to make it indispensable - by working directly with trading desks to understand their workflows, building the quantitative and AI capabilities they need to generate better ideas and make better decisions, and partnering with software engineers to deliver them at production quality.
Principal Privacy Product Manager, Health AI Microsoft CorpPrincipal Privacy Product Manager, Health AINew York, NY$142,800–$274,800 / yearThere is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $188,000 - $304,200 per year. Be the privacy partner to health engineering, product, and clinical stakeholders, guiding privacy decisions across the end-to-end product experience, including feature design, user flows, defaults, and sensitive data use.
Senior Manager, AI Solutions Architect Salesforce IncSenior Manager, AI Solutions ArchitectNY$172,500–$260,100 / yearOwn agent identity and session design - author and maintain agent context files (identity files, session startup protocols, memory structures, and skill definitions) that govern how agents behave consistently across sessions, users, and evolving workflows. LLM context file design as a craft - knows how to write durable agent context (identity, memory, startup protocols, skill definitions) that holds up across sessions - not one-off prompts, but role-level design.
Digital Experience & AI Enablement Partner World Economic Forum SwitzerlandDigital Experience & AI Enablement PartnerNY$120,000–$140,000 / yearThis position will lead the adoption and effective use of the Forum's core digital and AI-enabled products across internal teams and external users, while continuously identifying opportunities to streamline workflows through practical automation. The Digital Experience & AI Enablement Partner will report to the Head of Digital Success & Support in Geneva and will work closely with the Intelligent Workplace Product Team and internal stakeholders across the New York and San Francisco offices.
Principal AI Engineer Salesforce IncPrincipal AI EngineerNY$172,500–$313,700 / yearYou'll build and maintain the core infrastructure, CI/CD pipelines, and platform services that underpin our machine learning initiatives and go further in designing the harnesses, sandboxes, and evaluation frameworks that let AI agents be developed, tested, and trusted in production. Implement evaluation frameworks for agent behavior using a combination of vendor, open source or in house built tools - covering task success, tool selection accuracy, trajectory evaluation, hallucination rates, latency, and cost.
AI Architect, Finance ZipHQ IncAI Architect, FinanceNew York City, NY$130,000–$150,000 / yearAs an AI Architect, Finance, you will work directly with enterprise customers to design, deploy, and optimize AI agents across the full AP cycle: invoice inbox, PO and invoice coding, three-way matching, approval workflows, invoice processing, disbursements, payments, and wallet management. Design and Build AP AI Agents: Collaborate with customers to understand their AP and P2P workflows and configure agents that handle invoice intake, GL coding, three-way matching, exception routing, and payment approvals.
Lead AI Engineer, Data Solutions Salesforce IncLead AI Engineer, Data SolutionsNY$172,500–$260,100 / yearThis role focuses on developing intelligent decisioning systems and building an agent flywheel-a system of feedback loops that continuously evaluate, optimize, and improve agent performance over time. The world of work as we know it is changing and we're looking for Trailblazers who are passionate about bettering business and the world through AI, driving innovation, and keeping Salesforce's core values at the heart of it all.
Principal or Senior Principal Anthropic AI Solutions East Region Slalom IncPrincipal or Senior Principal Anthropic AI Solutions East RegionNYRemote$157,000–$216,000 / yearLead the design and delivery of enterprise AI solutions using Anthropic technologies, including large language model integrations, agentic workflows, and retrieval-augmented generation pipelines, to drive measurable client outcomes. You spend roughly 40% of your time delivering: architecting Claude-based solutions, leading technical engagements, and acting as the SME on model usage, MCP integrations, agentic workflows, and enterprise deployment patterns.
Lead Software Engineer - Core AI Zeta Global CorpLead Software Engineer - Core AINY$200,000–$215,000 / yearAs a Lead Software Engineer on our AI Platform Team, you will drive the design and development of the next generation of agentic, reliable, and scalable GenAI platforms supporting state-of-the-art large language models, evaluation systems, and secure developer tooling. Zeta Global (NYSE: ZETA) is the AI-Powered Marketing Cloud that leverages advanced artificial intelligence (AI) and trillions of consumer signals to make it easier for marketers to acquire, grow, and retain customers more efficiently.
Solution Engineer - AI Business Process Microsoft CorpSolution Engineer - AI Business ProcessNY$85,100–$169,800 / yearYoull collaborate closely with engineering leaders and platform teams to accelerate Dynamics 365, Power Platform, and AI through hands-on engagements like Proof of Concepts (PoC), hackathons, and architecture workshops. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $112,000 - $185,300 per year.
Vice President, Product Management - Agentic AI Publicis Groupe SAVice President, Product Management - Agentic AINew York, NY$164,500–$351,000 / yearThis is a critical role at the intersection of AI and the future of marketing - an opportunity to shape how the world's largest brands harness AI to evolve planning, execution, measurement, and optimization of marketing campaigns spanning owned and paid advertising channels in an increasingly autonomous era. As VP of Product Management - Agentic AI, you will own the vision, strategy, roadmap, and commercialization of AI agents and agentic worfklows that deliver measureable outcomes for marketers, agencies, and publishers, ultimately translating to commercial value for Epsilon.
Specialist Account Executive, AI & Data driven Agentic Transformation Salesforce IncSpecialist Account Executive, AI & Data driven Agentic TransformationNY$99,500–$247,800 / yearYou will operate as a trusted advisor, guiding customers through complex, multi-cloud solutions and demonstrating how Salesforce delivers measurable business value through AI-powered outcomes. We are seeking a driven, technically fluent sales professional with experience selling complex, multi-product SaaS solutions and a strong understanding of AI and data-driven technologies.
Senior Software Engineer - AI Enablement Mercury Technologies IncSenior Software Engineer - AI EnablementNew York, NY$166,600–$218,700 / yearShape and maintain structured context artifacts-clean, reliable, agent-consumable-so LLMs working in Mercury''s systems can reason accurately about our domain. Has hands-on experience building LLM-powered systems-RAG pipelines, agents, eval frameworks-and has shipped at least one of these to production.
AI Governance Capgemini SEAI GovernanceBrooklyn, NY$103,330–$128,656 / yearThe disclosed range takes into account the wide range of factors that are considered in making compensation decisions including, but not limited to, geographic location, relevant education, qualifications, certifications, experience, skills, seniority, performance, sales or revenue-based metrics, and business or organizational needs. Important Notice: Compensation (including bonuses, commissions, or other forms of incentive pay) is not considered earned, vested, or payable until it becomes due under the terms of applicable plans or agreements and is subject to Capgemini's discretion, consistent with applicable laws.
VP Applications Data & AI Services For The Underserved, Inc.VP Applications Data & AINew York, New YorkFull timeThis role is the organizational authority on the Microsoft Fabric, AI agents, Microsoft Power Platform, process automation, and the full analytics and reporting ecosystem that enables S:US leadership and program staff to make data-informed decisions at scale. Reporting to the CIO, the VP will lead a team of engineers, data professionals, and automation specialists, serving as both a hands-on architect and a strategic partner to Operations, Finance, HR, Clinical Services, and Housing program leadership.
Security Engineer - AI Agent Builder CapgeminiSecurity Engineer - AI Agent BuilderNew York, NY$80,420–$106,050 / yearThe disclosed range takes into account the wide range of factors that are considered in making compensation decisions including, but not limited to, geographic location, relevant education, qualifications, certifications, experience, skills, seniority, performance, sales or revenue-based metrics, and business or organizational needs. It delivers end-to-end services and solutions leveraging strengths from strategy and design to engineering, all fueled by its market leading capabilities in AI, generative AI, cloud and data, combined with its deep industry expertise and partner ecosystem.
Principal AI Engineer Salesforce.com, Inc.Principal AI EngineerNew York, NY$197,300–$313,700 / yearYou'll build and maintain the core infrastructure, CI/CD pipelines, and platform services that underpin our machine learning initiatives and go further in designing the harnesses, sandboxes, and evaluation frameworks that let AI agents be developed, tested, and trusted in production. Implement evaluation frameworks for agent behavior using a combination of vendor, open source or in house built tools - covering task success, tool selection accuracy, trajectory evaluation, hallucination rates, latency, and cost.