Sr. AI Software Engineer (C#/.Net, infrastructure & startup exp. req'd) - 100% REMOTE JobotSr. AI Software Engineer (C#/.Net, infrastructure & startup exp. req'd) - 100% REMOTENew York City, NYRemote$170,000–$270,000 / yearThe ideal candidate will be mid-level backend .Net Software Engineer with recent experience within a small, fast-paced, API-focused AI infrastructure startup company with lean, highly collaborative engineering teams, where they have operated with significant ownership, autonomy, and a strong “go-getter” mentality. Information collected and processed as part of your Jobot candidate profile, and any job applications, resumes, or other information you choose to submit is subject to Jobot's Privacy Policy, as well as the Jobot California Worker Privacy Notice and Jobot Notice Regarding Automated Employment Decision Tools which are available at jobot.com/legal.
Senior AI Engineer JobotSenior AI EngineerNew York, NY$225,000–$280,000 / yearInformation collected and processed as part of your Jobot candidate profile, and any job applications, resumes, or other information you choose to submit is subject to Jobot's Privacy Policy, as well as the Jobot California Worker Privacy Notice and Jobot Notice Regarding Automated Employment Decision Tools which are available at jobot.com/legal. The product sits at the intersection of AI, automation, and real world impact, using LLMs and agentic systems to replace hours or weeks of manual work with intelligent, scalable software.
NewGenerative AI Training & Support Specialist JobotGenerative AI Training & Support SpecialistNew York, NY$80–$90 / hourInformation collected and processed as part of your Jobot candidate profile, and any job applications, resumes, or other information you choose to submit is subject to Jobot's Privacy Policy, as well as the Jobot California Worker Privacy Notice and Jobot Notice Regarding Automated Employment Decision Tools which are available at jobot.com/legal. Reporting to the Senior Manager of User Engagement, Education & Tech Academy, this individual will be responsible for designing, delivering, and supporting generative AI training and enablement programs across the enterprise.
NewGenerative AI Training Specialist IconmaGenerative AI Training SpecialistNew York, NY$84.66–$89.66 / hourBuild and deliver training on AI coding assistants such as Claude Code and OpenAI Codex, helping engineering and technical teams learn to use these tools effectively in their day-to-day development work. Responsibilities: Design and facilitate live and on-demand training on generative AI tools, including (but not limited to) client GPT, ChatGPT Enterprise, Microsoft Copilot, and Claude.
NewRevenue Manager (AI/Tech) Vaco LLCRevenue Manager (AI/Tech)New York, NY$150,000–$180,000Determining compensation for this role (and others) at Vaco by Highspring depends upon a wide array of factors including but not limited to: the individual’s skill sets, experience and training; licensure and certification requirements; office location and other geographic considerations; other business and organizational needs. Industry Tenure: Minimum of 3 years of direct experience in revenue management, pricing strategy, or asset performance roles within multifamily, real estate, or related sectors.
NewCFD & CAD Expert — AI Evaluation & Quality Review BraintrustCFD & CAD Expert — AI Evaluation & Quality ReviewChelsea and Clinton, NYAdditional expertise in one or more of the following: Reacting flows Solid mechanics / structural deformation Fluid-structure interaction (FSI) Heat transfer Chemically reacting flows Multiphase flows Company Braintrust is a global talent network that connects top independent professionals with leading companies for high-quality, flexible work. Job description Final rate depends on location, qualifications and experience We are seeking a Mechanical Engineering expert with advanced CAD and Computational Fluid Dynamics (CFD) experience to support a leading AI lab developing next-generation engineering AI models.
NewTechnical Project Manager III (AI/Initiatives) BC ForwardTechnical Project Manager III (AI/Initiatives)New York, NYFull timeThe ideal candidate will have strong experience in planning and coordinating information systems, systems analysis, and software delivery and a proven ability to lead large cross-functional initiatives to successful outcomes . With delivery centers and offices across North America and India, we take pride in building long-term relationships and delivering excellence through innovation, collaboration, and integrity.
Freelance Annotator (English) - AI Trainer Toloka AnnotatorsFreelance Annotator (English) - AI TrainerNew York, New YorkAs an annotator, you may be invited to take part in online projects such as rating AI-generated content, evaluating factual accuracy, or comparing responses - when projects are available. At Toloka , we connect smart, curious people from around the world with freelance online tasks that train and improve artificial intelligence.
AI Trainer - Freelance Data Annotator Toloka AnnotatorsAI Trainer - Freelance Data AnnotatorNew York, New YorkAs an annotator, you may be invited to take part in online projects such as rating AI-generated content, evaluating factual accuracy, or comparing responses - when projects are available. At Toloka , we connect smart, curious people from around the world with freelance online tasks that train and improve artificial intelligence.
Lead Instructor: MLOps / AI Platform Engineering Adecco Group A.G.Lead Instructor: MLOps / AI Platform EngineeringNYRemote$11,500–$15,500 / yearGeneral Assembly is delivering a specialized reskilling program designed to transition Customer Success and Account Management professionals into MLOps and AI Platform Engineering roles. You will lead experienced customer-facing professionals through the complexities of taking AI systems from pilot to production, ensuring they leave the 2-week intensive with a functional understanding of MLOps governance and deployment.
Lead Instructor: MLOps / AI Platform Engineering General Assembly Space IncLead Instructor: MLOps / AI Platform EngineeringNYRemote$11,500–$15,500 / yearGeneral Assembly is delivering a specialized reskilling program designed to transition Customer Success and Account Management professionals into MLOps and AI Platform Engineering roles. You will lead experienced customer-facing professionals through the complexities of taking AI systems from pilot to production, ensuring they leave the 2-week intensive with a functional understanding of MLOps governance and deployment.
Lead Instructor: Agentic AI Engineering General Assembly Space IncLead Instructor: Agentic AI EngineeringNYRemote$11,500–$15,500 / yearGeneral Assembly is delivering an intensive reskilling program designed to transition experienced Customer Success and Account Management (CSAM) professionals into technical roles focused on Agentic AI Engineering. Lead Live Technical Instruction: Deliver synchronous remote lectures and "build-along" sessions focused on designing agentic systems within Azure AI Foundry, Copilot Studio, and Semantic Kernel.
NewCoding Specialist (Fluent in Dutch) - Freelance AI Trainer Project Meridial Marketplace, by InvisibleCoding Specialist (Fluent in Dutch) - Freelance AI Trainer ProjectNew York, NYOn a typical day, you will converse with the model on software engineering tasks and technical scenarios in Dutch, verify logical accuracy and coding fluency, assess naturalness and correctness of Dutch language usage, capture reproducible error traces, and suggest improvements to our prompt engineering and evaluation metrics. You'll challenge advanced language models on topics like asynchronous programming, RESTful API integration, memory management, object‑oriented design, secure coding practices, and debugging distributed systems—documenting every failure mode so we can harden model reasoning.
NewLitigation Specialist - Freelance AI Trainer Project Meridial Marketplace, by InvisibleLitigation Specialist - Freelance AI Trainer ProjectNew York, NY$50–$75 / hourThat training data begins with you—your expertise will help power the next generation of AI.We're looking for attorneys with substantial litigation experience who can challenge advanced AI models with realistic legal scenarios, evaluate their reasoning, and identify opportunities for improvement. On a typical day, you will engage in substantive legal dialogue with the model, assess its responses for factual accuracy and analytical rigor, identify reproducible failure points, and collaborate with our team to refine prompts and improve evaluation metrics.
NewRemote Litigation Specialist & AI Trainer Meridial Marketplace, by InvisibleRemote Litigation Specialist & AI TrainerNew York, NYRemote$50–$75 / hourThis role requires attorneys with significant litigation experience to engage with a large-scale AI model, assessing its legal reasoning and accuracy. A legal technology firm is seeking a Litigation Specialist for a freelance AI Trainer project.
NewCoding Specialist (Fluent in Bengali) - Freelance AI Trainer Project Meridial Marketplace, by InvisibleCoding Specialist (Fluent in Bengali) - Freelance AI Trainer ProjectNew York, NYThat training data begins with you—we need your expertise to help power the next generation of AI.ResponsibilitiesChallenge advanced language models on topics such as asynchronous programming, RESTful API integration, memory management, object‑oriented design, secure coding practices, and debugging distributed systems—documenting every failure mode to harden model reasoning. Employment DetailsEmployment type: ContractWorkplace type: RemoteSeniority level: Mid‑Senior LevelJob function: Health Care ProviderJob function: Software DevelopmentIndustries: Software DevelopmentReferrals increase your chances of interviewing at Meridial Marketplace, by Invisible by 2x.#J-18808-Ljbffr.
NewDesign Strategist - AI Trainer DataAnnotationDesign Strategist - AI TrainerNew York, NY$20–$40 / hourResponsibilities Review and critique AI-generated UI/UX designs, mockups, and visualsEvaluate the quality produced by AI models for correctness and performanceHelp train models to better understand aesthetic quality and user-centered designQualifications Fluency in English (native or bilingual level)Background in UI/UX design, web design, product design, or graphic designStrong understanding of visual hierarchy, interaction design, and user flowExcellent command of grammar, style, and brand voiceA current, in progress, or completed Bachelor's is preferred but not requiredPayment is made via PayPal. This is a full‑time or part‑time REMOTE positionYou'll be able to choose which projects you want to work onYou can work on your own scheduleGeneral AI training projects start at $20+ USD/hr, while design-focused projects start at $40+ USD/hr, with bonuses for exceptional work.
AI & Product Transformation Manager Sovos Compliance LLCAI & Product Transformation ManagerNY$100,000–$120,000 / yearOperating as a program manager, you''ll lead AI skill-building across the support org, serve as the support-side liaison for major product launches, and make sure every agent has the documentation, training, and tooling they need to succeed---in both English and Spanish. The Work You''ll Do: At Sovos, we''re transforming how companies around the world manage tax compliance---helping half of the Fortune 500 and thousands more stay ahead in a complex, ever-changing global landscape.
AI Governance Lead Yantran LLCAI Governance LeadWarren, NJday to day driver of an enterprise AI governance implementation programme, co leading delivery with the client s Privacy and Security leadership. *** represents the connected world, offering innovative and customer-centric information technology experiences, enabling Enterprises, Associates, and Society to Rise.
NewQuantitative AI Trainer & Evaluation Scientist DataAnnotationQuantitative AI Trainer & Evaluation ScientistNew York, NYRemoteThe job includes tasks like assessing statistical analysis and predictive modeling, all while contributing to cutting-edge AI systems. Candidates should have at least 2 years of experience in a quantitative role, coding skills, and strong analytical capabilities.
NewRemote Labor & Employment Attorney AI Model Trainer DataAnnotationRemote Labor & Employment Attorney AI Model TrainerBrooklyn, NYRemote$50–$60 / hourDataAnnotation is seeking a Labor and Employment Attorney to assist in training AI models by evaluating their logic and solving legal problems. Flexibility is key, allowing you to choose projects, work on your own schedule, and from home.
NewAI Quant Analyst & Trainer Remote & Flexible DataAnnotationAI Quant Analyst & Trainer Remote & FlexibleNew York, NYRemoteIdeal candidates should have 2+ years of experience in areas like data science or statistics, along with coding skills and a bachelor's degree in a quantitative field. A leading AI development firm is looking for quantitative professionals to evaluate AI-generated quantitative outputs and design problems for training AI systems.
NewSecurities Lawyer & AI Model Trainer - Remote, Flexible Hours DataAnnotationSecurities Lawyer & AI Model Trainer - Remote, Flexible HoursNew York, NYRemote$50–$60 / hourDataAnnotation, located in the United States, is seeking an experienced Securities Lawyer to help train AI models. This role requires you to measure the progress of chatbots, evaluate their logic, and enhance the model's quality.
Full Stack AI Software Engineer BC ForwardFull Stack AI Software EngineerJersey City, NJ$68–$68.25 / hourFull timeThe ideal candidate will have strong experience in full stack development across modern UI frameworks and Python backends and a proven ability to deliver scalable, low-latency, AI-enabled enterprise applications. With delivery centers and offices across North America and India, we take pride in building long-term relationships and delivering excellence through innovation, collaboration, and integrity.
Senior AI Engineer BC ForwardSenior AI EngineerJersey City, NJ$68–$68.25 / hourFull timeThe ideal candidate will have strong experience in end-to-end GenAI solution delivery, data management, and Python engineering, and a proven ability to translate business strategy into actionable, governed AI solutions. We are seeking an experienced Application Programmer III to join our team and lead development for a high-visibility Generative AI initiative focused on document summarization and chatbot capabilities across multiple lines of business.
Senior GenAI Engineer / AI Engineer BC ForwardSenior GenAI Engineer / AI EngineerJersey City, NJ$68–$68.25 / hourFull timeWith delivery centers and offices across North America and India, we take pride in building long-term relationships and delivering excellence through innovation, collaboration, and integrity. BCforward is a leading global IT consulting and workforce solutions firm providing services and support to Fortune 500 and government clients.
Senior Python Developer - Generative AI (GenAI) BC ForwardSenior Python Developer - Generative AI (GenAI)Jersey City, NJ$68–$68.25 / hourFull timeThe ideal candidate will have strong experience in Python, GenAI lifecycle management, RAG pipelines, vector databases, and API-first application development and a proven ability to deliver secure, scalable AI-driven applications for document processing, risk analytics, and compliance reporting . With delivery centers and offices across North America and India, we take pride in building long-term relationships and delivering excellence through innovation, collaboration, and integrity.
Technical AI Instructor and Forward-Deployed Engineer II Deloitte Touche Tohmatsu LtdTechnical AI Instructor and Forward-Deployed Engineer IINJ$82,600–$162,800 / yearOur purpose comes through in our work with clients that enables impact and value in their organizations, as well as through our own investments, commitments, and actions across areas that help drive positive outcomes for our communities. The wage range for this role takes into account the wide range of factors that are considered in making compensation decisions including but not limited to skill sets; experience and training; licensure and certifications; and other business and organizational needs.
Technical AI Instructor and Forward-Deployed Engineer (Consultant) Deloitte Touche Tohmatsu LtdTechnical AI Instructor and Forward-Deployed Engineer (Consultant)Morristown, NJ$80,400–$148,000 / yearKey Responsibilities Include: Teaching Technical AI Skills: Develop and deliver comprehensive training programs on generative AI technologies, including tools such as GitHub Copilot, Cursor, Replit, Airtable, and platforms like OpenAI, Anthropic, and Gemini. Our purpose comes through in our work with clients that enables impact and value in their organizations, as well as through our own investments, commitments, and actions across areas that help drive positive outcomes for our communities.
Python AI Engineer VeeRteq Solutions Inc.Python AI EngineerMorris Plains, NJBuild extensible API Integrations, low code UI/UX solutions, with extremely short cycle times, to extract information from sources, integrate with GPT4, receive insights and make them available in intuitive, high performing dashboards Build solutions that align with responsible AI practices. Work with Product Development as a Generative Artificial Intelligence (AI) subject matter expert and architect and develop scalable, resilient, ethical AI solutions Strong engineering skills to design the output from the AI with nodes and nested nodes in JSON or array, HTML formats as required.
Cloud Security Architect (AI) ApolisCloud Security Architect (AI)New York, NY$60 / hourThe AI Security Team is responsible for establishing secure-by-design principles for AI and Agentic AI systems, ensuring appropriate controls, governance, monitoring, and regulatory alignment across platforms such as ChatGPT Enterprise, AWS Bedrock, Databricks, and other enterprise AI services. The role will serve as a hands-on technical contributor responsible for defining, implementing, and validating security controls and solutions that protect AI and data development, deployment, and usage across the enterprise.
Machine Learning / AI Contractor: 10+ years (Expert) ApolisMachine Learning / AI Contractor: 10+ years (Expert)New York, NYRemote$72–$75 / hourTool & System Integration: Architect the "connective tissue" between our AI agents and live enterprise infrastructure (e.g., Oracle database Google Workspace, Jira, Salesforce, BigQuery) using Model Context Protocol (MCP) servers and custom APIs and Oracle Database MCP server. • Complex Workflow Automation: Develop long-running agents capable of managing multi-day, multi-step workflows, utilizing complex reasoning patterns (e.g., ReAct, hierarchical delegation, self-reflection).
Software Engineer, AI Agents NxT LevelSoftware Engineer, AI AgentsNew York, NYRemote$180,000–$320,000 / yearYou’ll work on AI features that directly power customer workflows—especially around analytics and data questions—and you’ll partner closely with product to translate ambiguity into elegant systems. Their AI agents are reimagining marketing workflows—helping teams create content, plan campaigns, and execute strategies with significantly higher velocity and performance.
AI Engineering Enablement Lead Optiver Holding BVAI Engineering Enablement LeadNYAs our AI Engineering Education Lead, you'll partner with the Head of AI Engineering to lead and implement AI education across our teams globally, reducing resistance to adoption and turning AI fluency into measurable business impact across Trading, Technology and Business Operations. Build and support a network of local trainers across our offices to deliver training in-region, including train-the-trainer programs tailored to varied audiences.
NewCoding Specialist - Freelance AI Trainer (Fluent in Italian) Project MeridialCoding Specialist - Freelance AI Trainer (Fluent in Italian) ProjectNew York, NYOn a typical day, you will converse with the model on software engineering tasks and technical scenarios in Italian, verify logical accuracy and coding fluency, assess naturalness and correctness of Italian language usage, capture reproducible error traces, and suggest improvements to our prompt engineering and evaluation metrics. You'll challenge advanced language models on topics like asynchronous programming, RESTful API integration, memory management, object‑oriented design, secure coding practices, and debugging distributed systems—documenting every failure mode so we can harden model reasoning.
NewCoding Specialist (Fluent in Chinese) - Freelance AI Trainer Project MeridialCoding Specialist (Fluent in Chinese) - Freelance AI Trainer ProjectNew York, NYOn a typical day, you will converse with the model on software engineering tasks and technical scenarios in Chinese, verify logical accuracy and coding fluency, assess naturalness and correctness of Chinese language usage, capture reproducible error traces, and suggest improvements to our prompt engineering and evaluation metrics. That training data begins with you—we need your expertise to help power the next generation of AI.We're looking for coding specialists who live and breathe algorithms, data structures, software architecture, frontend and backend development, cloud infrastructure, and systems programming—and who can do so fluently in Chinese.
NewCoding Specialist (Fluent in Swedish) - Freelance AI Trainer Project MeridialCoding Specialist (Fluent in Swedish) - Freelance AI Trainer ProjectNew York, NYOn a typical day, you will converse with the model on software engineering tasks and technical scenarios in Swedish, verify logical accuracy and coding fluency, assess naturalness and correctness of Swedish language usage, capture reproducible error traces, and suggest improvements to our prompt engineering and evaluation metrics. You'll challenge advanced language models on topics like asynchronous programming, RESTful API integration, memory management, object‑oriented design, secure coding practices, and debugging distributed systems—documenting every failure mode so we can harden model reasoning.
NewCoding Specialist (Fluent in German) - Freelance AI Trainer Project MeridialCoding Specialist (Fluent in German) - Freelance AI Trainer ProjectNew York, NYThat training data begins with you—we need your expertise to help power the next generation of AI.ResponsibilitiesWe're looking for coding specialists who live and breathe algorithms, data structures, software architecture, frontend and backend development, cloud infrastructure, and systems programming—and who can do so fluently in German. On a typical day, you will converse with the model on software engineering tasks and technical scenarios in German, verify logical accuracy and coding fluency, assess naturalness and correctness of German language usage, capture reproducible error traces, and suggest improvements to our prompt engineering and evaluation metrics.
Principal or Senior Principal Anthropic AI Solutions East Region Slalom IncPrincipal or Senior Principal Anthropic AI Solutions East RegionNYRemote$157,000–$216,000 / yearLead the design and delivery of enterprise AI solutions using Anthropic technologies, including large language model integrations, agentic workflows, and retrieval-augmented generation pipelines, to drive measurable client outcomes. You spend roughly 40% of your time delivering: architecting Claude-based solutions, leading technical engagements, and acting as the SME on model usage, MCP integrations, agentic workflows, and enterprise deployment patterns.
Solution Engineer - Cloud & AI Infrastructure Microsoft CorpSolution Engineer - Cloud & AI InfrastructureNY$106,400–$203,600 / yearRequired Qualifications: 5+ years technical pre-sales or technical consulting experience OR equivalent experience OR Bachelors Degree in Computer Science, Information Technology, or related field AND 4+ years technical pre-sales or technical consulting experience OR Masters Degree in Computer Science, Information Technology, or related field AND 3+ years technical pre-sales or technical consulting experience. Build trusted advisor relationship with customers' technical decision makers and use proactive effort to find and understand customers' pain points and work together with customers to co-develop secure & resilient solution architecture for production scale delivery using Cloud Adoption Framework (CAF) best practices including Unified Support and Cloud Migration Factory (CMF) offer for every opportunity.
AI Enablement Specialist Kyndryl IncAI Enablement SpecialistNew York, NY$100,080–$190,080 / yearKyndryl currently does not require employees to be fully vaccinated against COVID-19, however, if you are hired to work at a client, customer, or partner location, you may be required to show proof of vaccination to align with their respective COVID-19 vaccination policies. With state-of-the-art resources and Fortune 100 clients, every day is an opportunity to innovate, build new capabilities, new relationships, new processes, and new value.
Principal Solution Architect, AI Agents Wolters Kluwer N.V.Principal Solution Architect, AI AgentsNY$146,900–$262,400 / yearIn addition to creating proof-of-concepts to demonstrate best practices and accelerate adoption, you'll play a key role in driving the evolution of the GenAI platform by identifying gaps, proposing enhancements, and collaborating with platform engineering teams to deliver new capabilities. Help product engineering teams design and implement scalable AI solutions that leverage WKs GenAI platform to drive impact in healthcare, legal, tax, and compliance industries.
AI Governance Capgemini SEAI GovernanceBrooklyn, NY$103,330–$128,656 / yearThe disclosed range takes into account the wide range of factors that are considered in making compensation decisions including, but not limited to, geographic location, relevant education, qualifications, certifications, experience, skills, seniority, performance, sales or revenue-based metrics, and business or organizational needs. Important Notice: Compensation (including bonuses, commissions, or other forms of incentive pay) is not considered earned, vested, or payable until it becomes due under the terms of applicable plans or agreements and is subject to Capgemini's discretion, consistent with applicable laws.
AI Forward Deployed Engineer ZipHQ IncAI Forward Deployed EngineerNew York City, NY$130,000–$150,000 / yearAs an AI Forward Deployed Engineer, you will embed directly inside our most strategic accounts, uncover the operational problems that are hardest to see from the outside, and build working solutions using Zips agent infrastructure and AppStudio. You will work where Zips platform meets real enterprise complexity, across procurement, finance, legal, and risk workflows, and you will be expected to see what needs to exist before the customer can articulate it.
AI Engineering, Vice President (New York, New Jersey, Toronto) The Goldman Sachs Group IncAI Engineering, Vice President (New York, New Jersey, Toronto)Newark, NJ$115,000–$180,000 / yearJoin our engineering teams that build massively scalable software and systems, architect low latency infrastructure solutions, proactively guard against cyber threats, and leverage machine learning alongside financial engineering to continuously turn data into action. We''re committed to fostering and advancing diversity and inclusion in our own workplace and beyond by ensuring every individual within our firm has a number of opportunities to grow professionally and personally, from our training and development opportunities and firmwide networks to benefits, wellness and personal finance offerings and mindfulness programs.
Data Architect with Snowflake & AI Platforms Capgemini SEData Architect with Snowflake & AI PlatformsNY$90,786–$110,762 / yearArchitect and implement AI‑powered solutions using Snowflake Cortex including Cortex LLM functions, Cortex Search for semantic/vector/RAG capabilities, Cortex Analyst for conversational analytics, Document AI, and integration with external LLMs (e.g., for fine‑tuning, agents, and multimodal data processing). Lead end‑to‑end data architecture and solution design for Snowflake‑based data platforms including logical/physical data models, ingestion patterns (batch, streaming, Snowpipe), storage layers (raw, curated, consumption/semantic), and consumption patterns for analytics, BI, and AI/ML.
Client Success Specialist- AI The Circana GroupClient Success Specialist- AINY$90,000–$100,000 / yearThis role bridges product support, user experience, and development by investigating issues, coordinating resolutions, coaching commercial teams, and managing communication between users, client service teams, and product development. An employee's position within the salary range will be based on several factors including, but not limited to, relevant education, qualifications, certifications, experience, skills, seniority, geographic location, performance, shift, travel requirements, sales or revenue-based metrics, any collective bargaining agreements, and business or organizational needs.
Principal AI Engineer Salesforce IncPrincipal AI EngineerNY$172,500–$313,700 / yearYou'll build and maintain the core infrastructure, CI/CD pipelines, and platform services that underpin our machine learning initiatives and go further in designing the harnesses, sandboxes, and evaluation frameworks that let AI agents be developed, tested, and trusted in production. Implement evaluation frameworks for agent behavior using a combination of vendor, open source or in house built tools - covering task success, tool selection accuracy, trajectory evaluation, hallucination rates, latency, and cost.