Senior Staff Engineer - InfraOps Resilience

A perfect recovery design fails if nobody can execute it at three in the morning under pressure. This role owns the engineering of recovery execution: runbooks as code, command centre tooling, incident engineering, and the observability of the recovery itself.

You will build the machinery that makes infrastructure operations genuinely recoverable, and you will personally execute the technical recovery tests that underpin enterprise readiness reporting. You will also define the operational resilience practice that an incoming managed service provider delivers against, which means shaping what that practice is before a provider defines it for us.

Breadth matters here more than in any other role on the team. You cannot be deep in only one domain and succeed in this one.

WHAT YOU WILL DO

• Engineer recovery automation. Runbooks as code: tested, versioned, and executable by an engineer who did not write them. Automated where possible, engineered where not.
• Turn incidents into engineering change. Convert deep postmortems into systemic engineering change across the infrastructure teams and drive it to closure. Not action items on a list.
• Build command centre capability. The technical capability behind major incident and recovery command: observability of the recovery itself, decision instrumentation, escalation and communication mechanics.
• Set the supplier practice. Define the recovery execution standard, runbook quality bar and incident engineering discipline that a large managed service provider is measured against, and validate delivered performance directly.
• Own the dependency map. Maintain the infrastructure operations tier one dependency chain that the other resilience domains build against.
• Run the tests. Design and run the technical execution of recovery tests, and produce the resulting evidence.
• Direct without managing. Provide technical direction and mentoring to infrastructure operations engineers across the global estate.

REQUIRED QUALIFICATIONS

We are looking for the right capability rather than a specific number of years. The experience indication below is typical, not a filter; an engineer who has genuinely done this work in less time will be considered.

• Typically ten or more years in production operations, site reliability engineering or infrastructure engineering, including genuine on call ownership of tier one systems.
• You have built and run a technical recovery or disaster recovery test programme at enterprise scale, with evidenced outcomes.
• Strong automation and observability engineering. Runbooks as code, not documents.
• Major incident command experience on incidents with material business impact.
• You have shaped or governed a managed service provider operational practice, ideally through a transition or onboarding.
• Credible across at least three of compute, network, storage, cloud and data.

PREFERRED QUALIFICATIONS

• Manufacturing, or another twenty four by seven continuous operations environment.
• Game day or chaos engineering programme ownership.
• Experience inheriting or handing over an operational practice at a supplier transition.

LEVEL EXPECTATIONS

• Works across teams on ambiguous problems that have no established playbook.
• Defines objectives and establishes standards that others adopt.
• Influence extends well beyond the immediate team.