Senior Software Engineer, Observability

Role Summary

We have an opening for a Senior Software Engineer on our Infrastructure Team, with specific focus on Observability - both internal and customer-facing. This is an opportunity to join a mission-critical engineering team that is driving the productivity and reliability of Temporal's developers and core platforms, respectively. We are a passionate team of talented developers who truly care about our mission and enjoy working deeply across the entire Temporal ecosystem to be a force multiplier across the organization.

Team Highlights

• Core observability tech stack: Clickhouse, Prometheus, Grafana, Loki, Thanos, Kubernetes, Temporal.

• Our recent observability project had the biggest impact to Temporal margins of any engineering effort in company history while also unlocking unprecedented operator capabilities.

• "Observability is the beating heart of our ability to deliver a mission critical service" - Preeti Somal, Senior VP of Engineering.

What You'll Do

Build

• Lead the end-to-end Software Development Lifecycle: goals & requirements solicitation, design & review, implementation, operationalization & deployment, support & maintenance.

• Formulate feature designs, review with stakeholders, iterate to incorporate feedback and drive consensus.

• Clearly document design choices and operational knowledge to successfully deploy and manage the software you develop.

• Provide appropriate test and production readiness coverage for unit, integration, and performance of your feature ownership area.

Own

• Set a high bar for technical excellence and take pride in the software you develop.

• Design and build multi-component, distributed systems that operate at scale.

• Investigate issues with a methodical approach to identify a root cause.

• Understand performance and reliability implications of design options at scale. Make related tradeoffs.

• Able to participate in the team's on-call rotation.

Learn

• Expert-level knowledge of architecture and services of assigned domain. Strong command over all aspects of the Temporal ecosystem. This role provides a strong opportunity for diverse technical exposure!

• Investigate and understand ways to best leverage Temporal's own software to power our mission.

• Deeply understand the needs of Temporal internal developers and external customers, and leverage that knowledge for product development and feature design.

Collaborate

• Participate in design reviews and contribute to design of other features.

• Share design principles for building reliable systems at scale.

What You'll Bring

• User-first mindset. You're excited by the opportunity to empower others through tooling, and enjoy deeply internalizing user goals and use cases to build effective solutions.

• Motivated by impact. You are driven by a desire to make positive things happen.

• Strong opinions about tools and technology that are equally balanced by a pragmatic drive for impact.

• Ability to work in a self-directed manner in a fast-paced environment.

• Excellent collaboration and communication skills.

Skills & Technologies

• Demonstrated ability to develop horizontally scalable, resilient, and high performance distributed systems in a production environment.

• Experience designing, implementing, deploying, and supporting large scale, geographically distributed observability and/or high throughput data streaming/processing pipelines, or similar.

• Expert in one or more high-level programming languages, preferably Go.

• Expert-level Kubernetes skills.

• Expert-level query development skills, preferably SQL.

• Hands-on experience with one or more cloud providers, preferably AWS, or GCP.

• Thorough understanding of computer architecture, operating systems, and networking.

• Familiarity with best practices regarding monitoring, instrumenting, and configuring infrastructure.