Senior Software Engineer, CoreAI Workload Engines

Microsoft Corporation

Redmond, WA

Apply

JOB DETAILS

SALARY

$119,800–$234,700 Per Year

SKILLS

Analysis Skills, Application Programming Interface (API), Architectural Services, Artificial Intelligence (AI), Benchmarking, C Programming Language, C++ Programming Language, CUDA (Compute Unified Device Architecture), Cloud Computing, CoS (Class of Service), Communication Skills, Computer Science, Continuous Improvement, Debugging Skills, Engineering, Establish Priorities, Fortune 500 Customers, GPU (Graphics Processing Unit), High Reliability, Identify Issues, Inference Engine, Infrastructure Software, Integrated Circuits (ICs), Java, JavaScript, Kernel Programming, Load Balancing, Mentoring, Metrics, Microsoft C# (C Sharp), Microsoft Product Family, Microsoft Windows Azure, Modeling Languages, Open Source, Operating Systems, Operational Support Systems (OSS), Performance Analysis, Performance Management, Performance Metrics, Performance Tuning/Optimization, Problem Solving Skills, Process Improvement, Production Systems, Python Programming/Scripting Language, QoS (Quality of Service), Return on Capital Employed (ROCE), Software Engineering, Startup, Team Player, Technical Leadership, Theater Production, Time Management, Vehicle Fleets, Web Client Plug-ins

LOCATION

Redmond, WA

POSTED

30+ days ago

**Overview** The?CoreAI?Workloads team builds the?foundational inference engines and APIs that power largescale AI inference across Azure - from?cutting-edge?startups to Fortune 500 enterprises and Microsoft Copilots and agents. Our mission is to deliver?secure, reliable, and highly efficient GPU inference that enable multitenant AI systems at?global?scale while maximizing?utilization, performance, and developer productivity.?We own inference serving and performance of OpenAI and other state of the art large language model (LLM) models and work directly with OpenAI serving some of the largest workloads on the planet with trillions of inferences per day. Our?converged AI fabric and engines?deliver inference capabilities for all LLMs in?Microsoft catalog (https://azure.microsoft.com/en-us/products/ai-model-catalog#Models) , including OpenAI,?Anthropic,?Mistral, Cohere, Llama, and more. This role sits at the intersection of LLM inference fleets, serving efficiency, rapid experimentation, cloud infrastructure, and systems software-working closely with CoreAI data plane, compute, and partner teams to deliver end-to-end efficiencies and platform capabilities. In this role, you will have the opportunity to work on multiple levels of the AI software stack, including the fundamental abstractions, programming models, OpenAI and OSS engines runtimes, libraries and application programming interfaces (APIs) to enable large scale inferencing of models. You will drive production-grade inference serving improvements for OpenAI and open-source models across Azure, including benchmarking, performance measurement, and disciplined experimentation to improve latency, throughput, availability, and cost at scale. You will both (1) make hands-on engine changes and (2) contribute to the experimentation capabilities that make those changes measurable, safe to ship, and repeatable across teams. Microsoft's mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.? In alignment with our Microsoft values, we are committed to cultivating an inclusive work environment for all employees to positively impact our culture every day.? **Responsibilities** As?the Senior?engineer on the?team, your responsibilities include: ? + Optimize inference engines for OpenAI and open-source models by implementing and shipping performance/efficiency improvements across runtime, scheduling, and serving paths (latency, throughput, utilization, availability, and cost). + Run experiments end-to-end: formulate hypotheses, implement engine changes (including Python/PyTorch integration points where relevant), analyze results, and ship improvements behind guardrails. + Build and use experimentation capabilities for large-scale AI inference (experiment lifecycle, tracking, metric modeling, comparability standards, automated analysis) so the team can iterate quickly and safely. + Own serving availability and efficiency for Azure OpenAI Service workloads through tiered experimentation, lean segmentation, and multi-modal utilization across heterogeneous fleets-turning findings into shipped engine improvements. + Design and evolve inference serving architectures to improve utilization and latency using techniques such as disaggregated serving, multi-token prediction, KV offload/retrieval, and quantization-validated via staged rollouts and production guardrails. + Extend AI infrastructure abstractions to support elastic, heterogeneous inference engines reliably at scale (e.g., dynamic scaling across model families, modalities, and workload classes while maintaining isolation and SLOs). + Tune and scale inference engines across NVIDIA GPU generations (A100, H100, H200) for state-of-the-art OpenAI models, focusing on serving efficiency, utilization, and reliability (not hardware bring-up). + Partner with networking and storage teams to leverage high-performance interconnects (e.g., RDMA/InfiniBand-class fabrics such as RoCE over IB) for distributed inference, without owning low-level kernel/driver enablement. + Drive end-to-end features from design through production: observability, diagnostics, performance regression detection, and operational excellence for inference serving. + Influence platform architecture and technical direction across teams through design reviews, clear metrics, and technical leadership focused on experimentation velocity and production reliability. **Additional Responsibilities** + Work across multiple layers of the AI software stack (abstractions, programming models, engine runtimes, libraries, and APIs) to enable large-scale model inference. + Benchmark OpenAI and other LLMs for performance across Azure OpenAI Service workload tiers and segments, and translate results into production improvements. + Debug, profile, and optimize production inference performance across the stack (abstractions, runtime, scheduling, and serving pipelines) to improve latency, throughput, and utilization. + Monitor performance regressions and drive continuous improvements to reduce time-to-deploy and hardware footprint. + Collaborate across engineering teams to deliver scalable, production-ready serving efficiency and availability improvements, using experimentation results to guide prioritization and rollout. + Build durable engine interfaces that enable fast experimentation and safe shipping of new strategies for class of service (QoS), replica load balancing, KV management (including offload/retrieval), quantization, and sampling (e.g., multi-token prediction and constrained sampling). **Out of Scope** (This role does not focus on) + Novel hardware bring-up or first-party silicon enablement (e.g., Microsoft chips) or expanded support for non-NVIDIA platforms (e.g., AMD). + Low-level kernel, driver, or CUDA optimization as a primary responsibility. + Model pre-training, fine-tuning, or model architecture customization. **Qualifications** + Bachelor's Degree in Computer Science or related technical field and 4+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, Python, or equivalent experience. **Other Requirements:** + Proven ability to design and operate large-scale, production inference services with high reliability and performance requirements, and to ship performance improvements safely via disciplined experimentation. + Strong skills in performance analysis: benchmarking, profiling, diagnosing regressions, and turning results into concrete engine/runtime changes. + Strong problem-solving skills and the ability to debug complex,?cross?layer?systems issues.? + Demonstrated technical leadership, including mentoring engineers, driving cross-team architectural alignment, and leveraging AI tools and AI-assisted workflows to accelerate engineering velocity and quality. + Hands-on experience with Kubernetes (building and operating services on k8s), including debugging production issues and designing platform abstractions (e.g., custom resources/controllers) and scheduling-aware deployments (e.g., node affinity, taints/tolerations, resource requests/limits). + Strong collaboration and communication skills, with the ability to work across organizational boundaries.? **Preferred Qualifications:** + Experience optimizing LLM inference in practice (e.g., PyTorch inference, serving runtimes, model execution, or inference orchestration) in production environments. + Familiarity?with?high?performance?networking?and?low?latency?communication stacks.? + Familiarity with GPU-accelerated inference stacks (e.g., CUDA at the application/runtime level, device plugins, or runtime integration). + Experience building or using experimentation systems (A/B, canarying, tiered rollout), including metric definition and comparability for performance and reliability. + Familiarity with distributed inference stacks (e.g., NCCL-style collectives, model/tensor parallelism) and performance tradeoffs in large-scale serving. **Impact & Growth:** + Work on?mission?critical infrastructure?that directly powers?largescale?AI systems.? + Influence the future of?cloud GPU platforms?used by internal and external customers.? + Collaborate with experts across?OS, hardware, networking, and AI platform teams.? + Opportunity to grow as a?technical leader, shaping long?term platform strategy.? Software Engineering IC4 - The typical base pay range for this role across the U.S. is USD $119,800 - $234,700 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $160,200 - $261,000 per year. Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here: https://careers.microsoft.com/us/en/us-corporate-pay Software Engineering IC5 - The typical base pay range for this role across the U.S. is USD $142,800 - $274,800 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $188,000 - $304,200 per year. Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here: https://careers.microsoft.com/us/en/us-corporate-pay Software Engineering IC5 - The typical base pay range for this role across the U.S. is USD $142,800 - $274,800 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $188,000 - $304,200 per year. Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here: https://careers.microsoft.com/us/en/us-corporate-pay This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled. Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations. (https://careers.microsoft.com/v2/global/en/accessibility.html)

About the Company

Microsoft Corporation

DO WHAT YOU LOVE
Make your mark on the world’s most used technologies. Develop the next hit mobile application. Pioneer a startup that could be the next big thing. At Microsoft, you choose your path.

Headquartered in Redmond, Washington, Microsoft is a top innovator in both the consumer and enterprise technology industry. Just a few of the many things our products do are unleash creativity, connect businesses, and make learning more fun. But our continued success is based on one thing: our employees. We hire amazing, talented people and give them the opportunities—and the tools—to succeed.

WHY MICROSOFT?
As a Microsoft employee, you’re surrounded by a diverse group of the smartest people in your field. This fosters new ideas, better business results, and creates a dynamic work environment. In the office, you’re constantly challenged and supported by your colleagues. Every day holds something new and exciting.

We also offer unparalleled depth and breadth of career opportunities. As an industry leader in multiple fields, working for Microsoft means being able to do whatever you feel passionate about—and being able to make an impact in that field. From day one, we give our employees significant responsibility. This means that you’ll know that you directly contributed to something that has a positive impact on people worldwide. Whether you choose to work in management, dive deep into the newest technology, or explore multiple professions, you’ll find everything you need at Microsoft to drive your career—and to make a difference.

WE GET IT – YOU’RE MORE THAN YOUR JOB
Everyone works differently and is motivated by different things. We also understand that there’s more to you than your job. That’s why we offer competitive pay and a wide assortment of benefits-- to help you make the most of life at work and away from it.

GET THE BALL ROLLING

COMPANY SIZE

10,000 employees or more

INDUSTRY

Other/Not Classified

FOUNDED

1975

WEBSITE

http://www.microsoft.com

Resume Resources

Free Resume Templates Free Resume Builder