AI Evaluation Engineer

Request ID: 110224-1
Title: AI Evaluation Engineer
Locations: Tampa, FL
Duration: 6 Months
Pay Range: $38 - $42/Hour on W2/C2C (All inclusive)

Introduction

We are seeking a skilled professional to join our team as an AI Evaluation Engineer, responsible for defining evaluation strategies and success metrics for AI agents. The role involves building test scenarios, developing automated evaluation pipelines, and collaborating with cross-functional teams to improve agent performance.

Required Skills & Qualifications (Must-have qualifications that candidates must meet to be considered)

• Strong understanding of Generative AI, LLMs, RAG, prompts, tool calling, and AI agents.
• Proficiency in Python and experience working with APIs, JSON, datasets, and evaluation frameworks.
• Ability to translate business expectations into measurable evaluation criteria.
• Experience with AI evaluation/observability tools such as LangSmith, Langfuse, Ragas, DeepEval, MLflow, or similar.
• Prior work experience at client or in client's Industry.

Applicants must be able to work directly for Artech on W2.

Preferred Skills & Qualifications (Nice-to-have skills but are not required)

• Experience with automated evaluation pipelines.
• Knowledge of regression testing and establishing quality thresholds.
• Familiarity with edge and adversarial case testing.

Day-to-Day Responsibilities (key tasks and expectations for the role)

• Define evaluation strategies and success metrics for AI agents.
• Build comprehensive test scenarios, including edge and adversarial cases.
• Develop automated evaluation pipelines and conduct regression testing.
• Analyze agent traces and outputs to identify performance bottlenecks.
• Collaborate with AI engineering and product teams for continuous improvement.

Company Benefits & Culture

• Competitive salary and benefits package.
• Opportunities for professional growth and development.
• Supportive and inclusive work environment.

For immediate consideration please click APPLY to begin the screening process with Alex.