Splunk Administrator/Developer
Overview
We are seeking a Splunk Administrator/Developer to take full ownership of a large-scale Splunk environment supporting mission-critical operational and engineering systems. This role will be responsible for managing the entire Splunk ecosystem from the infrastructure layer (bare metal) through application-level development, in direct support of software engineering teams responsible for complex physical systems and real-time operations.
This is a hands-on role requiring deep technical expertise across infrastructure, data ingestion, and observability, with a strong emphasis on reliability, scalability, and performance.
________________________________________
Key Responsibilities
Own and manage the full Splunk stack
o End-to-end responsibility from bare metal infrastructure OS Splunk architecture, dashboards, and analytics
Design, deploy, and maintain Splunk infrastructure
o Install, configure, and optimize Splunk components (indexers, search heads, forwarders, clustering)
o Manage both on-premise/physical server environments and hybrid integrations
Support real-time operational systems
o Enable monitoring and observability for high-availability, real-time engineering systems
o Build dashboards and alerts used by engineering and operations teams
Develop data pipelines and ingestion strategies
o Integrate data from diverse sources (logs, sensors, applications, control systems)
o Optimize parsing, indexing, and search performance
Partner with software engineering leadership
o Work closely with Software Engineering Managers and system teams
o Translate operational requirements into scalable Splunk solutions
Performance tuning and system reliability
o Identify bottlenecks across infrastructure and data pipelines
o Ensure high availability, redundancy, and system resilience
Security, governance, and access control
o Implement role-based access and data governance best practices
o Maintain compliance with enterprise security standards
Continuous improvement
o Recommend architecture improvements and modernization strategies
o Evaluate tools within the Splunk ecosystem (e.g., ITSI, ES, Cribl, etc.)
________________________________________
Required Qualifications
510+ years of experience with Splunk administration and development
Proven experience owning a Splunk environment end-to-end
Strong hands-on experience with:
o Splunk Enterprise architecture (indexers, search heads, clustering)
o Data ingestion, parsing, and optimization
o Dashboard and alert development
Experience managing on-prem / bare-metal infrastructure
o Linux administration (preferred)
o Server provisioning, performance tuning, and system monitoring
Experience supporting real-time, high-availability systems
Strong scripting skills (Python, Bash, or similar)
________________________________________
Preferred Qualifications
Experience supporting engineering, industrial, or operational technology (OT) environments
Familiarity with observability and logging tools beyond Splunk (e.g., Cribl, ELK, Prometheus)
Experience with event-driven systems, telemetry data, or IoT/sensor integrations
Exposure to high-throughput environments with large-scale data ingestion
Splunk certifications (Admin, Architect, or Developer)
________________________________________
What Success Looks Like
Full ownership of a complex Splunk environment with minimal oversight
Reliable, scalable monitoring of mission-critical systems
Strong partnership with engineering leadership and operational stakeholders
Continuous improvements in performance, visibility, and system uptime
________________________________________
Why This Role Is Unique
Direct impact on real-world systems with physical and operational outcomes
Highly visible role supporting critical engineering and operational teams
Opportunity to own both infrastructure and application layers within a single ecosystem