$119,800–$234,700 Per Year
Amazon Web Services (AWS), Artificial Intelligence (AI), Background Investigation, Best Practices, Business Analysis, Cloud Computing, Computer Engineering, Computer Science, Data Management, Data Modeling, Data Processing, Data Quality, Data Science, Data Sets, Database Design, GCP (Good Clinical Practices), Information/Data Security (InfoSec), JSON, Math Software, Microsoft Product Family, Microsoft Windows Azure, Performance Tuning/Optimization, Production Systems, Python Programming/Scripting Language, Scalable System Development, Software Development, Software Engineering, Structured Data, Use Cases, Workflow Analysis
We are building a large-scale data platform that transforms raw system logs into high-quality, structured datasets used for experimentation and analytics. The platform processes terabytes to petabytes of data daily and serves as a foundational asset for multiple teams.
This Senior Data Engineer - AI Infrastrucute role focuses on designing and implementing data pipelines, ensuring correctness, and building scalable data models. You will work closely with data scientists and platform engineers to ensure that data is accurate, reliable, and usable for downstream decision-making.
We are looking for engineers who care deeply about data correctness, understand how systems behave at scale, and can translate complex data into well-structured, reliable datasets.
Responsibilities
- Design and implement large-scale data pipelines using PySpark and distributed processing frameworks
- Build and maintain data models that accurately represent underlying system behavior and business logic
- Ensure high standards of data correctness, completeness, and consistency across datasets
- Develop validation, monitoring, and alerting mechanisms to detect data quality issues
- Partner with data scientists to support experimentation and analytics use cases
- Collaborate with platform engineers to ensure efficient data ingestion, processing, and storage
- Optimize pipelines for performance, scalability, and cost efficiency
- Define and enforce best practices for schema design, data transformations, and pipeline reliability
Qualifications
Required/Minimum Qualifications:
Masters Degree in Computer Science, Math, Software Engineering, Computer Engineering, or related field AND 3+ years experience in business analytics, data science, software development, data modeling, or data engineering OR Bachelors Degree in Computer Science, Math, Software Engineering, Computer Engineering, or related field AND 4+ years experience in business analytics, data science, software development, data modeling, or data engineering OR equivalent experience.
Other Requirements:
Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include, but are not limited to the following specialized security screenings: Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud Background Check upon hire/transfer and every two years thereafter.
Preferred Qualifications:
- Experience with Azure technologies such as:
- ADLS Gen2 (Blob Storage)
- Synapse Spark
- Azure Data Explorer (ADX)
- Experience working with structured and semi-structured data (e.g., JSON logs)
- Familiarity with experimentation and analytics workflows
- Experience with orchestration tools (e.g., Airflow)
- Exposure to privacy, compliance, and secure data handling practices
- 5+ years of experience in data engineering or software engineering with a strong focus on data systems
- Strong experience with PySpark or similar distributed data processing frameworks
- Experience building and operating large-scale data pipelines
- Strong understanding of data modeling and schema design
- Experience ensuring data quality and correctness in production systems
- Proficiency in Python
- Experience working with cloud-based data platforms (Azure, AWS, or GCP)
- Ability to reason about data at scale, including performance and failure modes
Data Engineering IC4 - The typical base pay range for this role across the U.S. is USD $119,800 - $234,700 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $158,400 - $258,000 per year.
M
Microsoft
DO WHAT YOU LOVE
Make your mark on the world’s most used technologies. Develop the next hit mobile application. Pioneer a startup that could be the next big thing. At Microsoft, you choose your path.
Headquartered in Redmond, Washington, Microsoft is a top innovator in both the consumer and enterprise technology industry. Just a few of the many things our products do are unleash creativity, connect businesses, and make learning more fun. But our continued success is based on one thing: our employees. We hire amazing, talented people and give them the opportunities—and the tools—to succeed.
WHY MICROSOFT?
As a Microsoft employee, you’re surrounded by a diverse group of the smartest people in your field. This fosters new ideas, better business results, and creates a dynamic work environment. In the office, you’re constantly challenged and supported by your colleagues. Every day holds something new and exciting.
We also offer unparalleled depth and breadth of career opportunities. As an industry leader in multiple fields, working for Microsoft means being able to do whatever you feel passionate about—and being able to make an impact in that field. From day one, we give our employees significant responsibility. This means that you’ll know that you directly contributed to something that has a positive impact on people worldwide. Whether you choose to work in management, dive deep into the newest technology, or explore multiple professions, you’ll find everything you need at Microsoft to drive your career—and to make a difference.
WE GET IT – YOU’RE MORE THAN YOUR JOB
Everyone works differently and is motivated by different things. We also understand that there’s more to you than your job. That’s why we offer competitive pay and a wide assortment of benefits-- to help you make the most of life at work and away from it.
GET THE BALL ROLLING