Sr. Software Engineer

About Blackhawk Network

Today, through BHN’s single global platform, businesses of all kinds can tap into the world’s largest network of branded payment solutions. BHN helps businesses grow revenue, increase loyalty, motivate and reward their teams, disburse funds and engage consumers. Branded payment solutions include the issuance and distribution of gift cards, egifts, corporate payouts and rewards, along with the technology to deliver these products in seamless, integrated ways. BHN’s network spans the globe with more than 400,000 consumer touchpoints. Learn more at BHN.com.

Hybrid flexibility: At Blackhawk Network, you’ll enjoy the best of both worlds—focused remote work plus in-person collaboration at our Coppell, TX office. This rhythm gives you the tools, connection, and autonomy you need to make a real impact.

Overview

We are looking for a Senior or Staff Software Engineer with deep Site Reliability Engineering expertise and strong Python development skills to join our SRE team. You will own the reliability, scalability, and observability of a critical Data pipeline platform.

This is a builder and operator role. you will write production-grade Python services, and implement observability-as-code infrastructure, and drive reliability improvements across a complex AWS-native platform. You will work at the intersection of software engineering and infrastructure, eliminating toil through automation, building internal developer platforms, and ensuring our systems meet the reliability bar our partners and customers expect.

Responsibilities

Site Reliability Engineering

• Own and improve service reliability, availability, and performance across a distributed AWS platform (EKS, EC2, RDS, S3)
• Define and track SLOs, SLIs, and error budgets for critical services and partner-facing integrations
• Drive initiatives identify manual, repetitive operational work and eliminate it through automation
• Partner with engineering teams to embed reliability practices into the SDLC

Platform & Observability Engineering

• Build and maintain observability-as-code infrastructure using Terraform and New Relic NerdGraph 
• Design multi-signal observability pipelines metrics, logs, traces  across Splunk, New Relic, and OpenSearch
• Architect and maintain log routing and sawmills data pipelines using FluentD and Sawmills
• Build New Relic NerdGraph integrations and custom NRQL dashboards for platform health visibility
• Maintain and extend Terraform modules for AWS infrastructure provisioning  Security Groups, EKS node groups, Aurora, IAM roles
• Contribute to the MCP server architecture (Python-based internal developer tooling) integrating Jira, Confluence, Bitbucket, and observability platforms

Python Development

• Design, build, and maintain production-grade Python services and automation REST APIs, background workers, CLI tooling, and data pipeline adapters
• Mentor junior and mid-level engineers  code reviews, pair programming, architecture guidance
• Work closely with program managers and product stakeholders 

Qualifications

• Bachelor's degree in Information Technology, Computer Science, or related field; or equivalent experience.
• 7+ years of software engineering experience 
• 4+ years of SRE or platform engineering experience in a production environment
• Strong Python proficiency production services, automation, REST APIs 
• Experience with AWS EKS, EC2, RDS/Aurora, S3, IAM, VPC, CloudWatch, Security Groups
• Hands-on Terraform experience  writing modules, managing state, CI/CD integration
• Experience with observability platforms  Splunk New Relic (NRQL, NerdGraph)
• Strong understanding of SRE principles SLOs/SLIs/error budgets, toil reduction, incident management, capacity planning
• Experience with Kubernetes / EKS  pod operations
• Strong Linux/ Unix fundamentals log management, performance debugging

Preferred Experience

• Experience with log pipeline tooling  FluentD, Open Telemetry
• Familiarity with Splunk, New Relic, OpenSearch, Elasticsearch for log storage and search
• Experience building internal developer platforms or tooling  CLI tools, internal APIs, automation frameworks
• Familiarity with MCP (Model Context Protocol) server development or AI Gateway architecture

Benefits

Pay is based on several factors including but not limited to education, work experience, certifications, etc. In addition to your salary, Blackhawk Network offers benefits including 401k with employer match, medical, dental, vision, 12 paid holidays throughout the year 2025, sick pay accrual according to state law, parental leave, life insurance, disability insurance, accident and illness insurance, health and dependent care flexible spending accounts, wellness benefits, and flexible time off for all full-time employees. 

EEO Statement

Blackhawk Network provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.  Blackhawk Network believes that diversity leads to strength. This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation, and training.

Blackhawk Network encourages applicants with previous criminal records to apply to all positions and, pursuant to the San Francisco and Los Angeles Fair Chance Acts (and other “Fair Chance” laws), Blackhawk Network will consider for employment qualified applicants with arrest and conviction records.  For Philadelphia applicants or jobs, please see a copy of Philadelphia’s ordinance on this topic by clicking this link: https://codelibrary.amlegal.com/codes/philadelphia/latest/philadelphia_pa/0-0-0-280104.