Principal Splunk Engineer

At Early Warning, we've powered and protected the U.S. financial system for over thirty years with cutting-edge solutions like Zelle®, Paze, and so much more. As a trusted name in payments, we partner with thousands of institutions to increase access to financial services and protect transactions for hundreds of millions of consumers and small businesses.

Positions located in Scottsdale, San Francisco, Chicago, or New York follow a hybrid work model to allow for a more collaborative working environment. Candidates responding to this posting must independently possess the eligibility to work in the United States, for any employer, at the date of hire. This position is ineligible for employment Visa sponsorship.

Purpose

The Principal Splunk Engineer operates as a logging architecture subject matter expert across all domains of logging and event monitoring. This includes working directly in collaboration with Technology and Security teams to design, implement and maintain Production Assurance and Reliability Operations Services and ensure that the company operates a world-class enterprise log management solution. The Principal Splunk Engineer maintains the logging infrastructure for Splunk Enterprise and Cloud and other dependent infrastructure, such as Cribl to ensure that reports are available, fast and efficient. The Principal Splunk Engineer leads and/or consults on efforts pertaining to design, architecture, scalability, business continuity, and disaster recovery as needed.

Essential Functions

• Leads the Splunk Engineering team in technical and risk-based decision making, focusing on data quality and business needs
• Architects Splunk Enterprise infrastructure and leads tuning Splunk for optimal onboarding of data, performance, and capacity management; identifies gaps and areas of duplication; provides recommendations for optimization
• Leads operating and maintaining efforts for a complex multi-site hybrid environment and ensures the infrastructure remains available and scalable
• Significantly influences the overall technology direction of Early Warning from the perspective of log management
• Collaborates with business customers to intake and define new logging architectures and solutions for various business units at Early Warning
• Leads and approves changes for logging infrastructure from a security perspective
• Ensures the logging environment provides for effective threat detection and response in direct partnership with information security teams
• Participates in incident management & incident response during an outage or security investigation when needed
• Creates documentation for any addition or change to the environment. Reviews and updates on a regular basis to ensure accuracy.
• Ensures the tools are supporting all compliance efforts in collaboration with auditors
• Provides metrics for platform performance, capacity, and user management
• Leads root cause analysis efforts pertaining to log engineering issues
• On-call rotation with other Splunk engineers to cover 24x7 response
• Sets and supports best practices for end users and company standards
• Stays current on the latest industry technologies, trends, and strategies
• Advises as the subject matter expert and contributes to the development of Early Warning security policy and procedures
• Mentors new team members

Minimum Qualifications

• Education and experience typically obtained through completion of a Bachelors degree in Computer Science, Engineering, Math or Physical Science
• Typically has 15 years of progressive Splunk administration, Splunk architect and/or logging experience in a multisite environment is necessary
• Certification: Splunk Certified Administrator required
• Subject matter expert: Splunk Enterprise and Cloud, Splunk SPL query language, Common ingestion strategies such as UFs, HFs, HEC, TAs/Add-ons and syslog
• Demonstrated proficiency with the full Splunk lifecycle, including all major components for enterprise deployments
• Must have solid foundation in Linux and possess a competence to troubleshoot various aspects of the integration including operating system, application, and networking components as they relate to both Splunk and syslog implementations
• Experience with supporting technologies such as Cribl
• Ability to handle large projects as well as take care of day-to-day operations
• Strong sense of responsibility, ownership, and pride in delivering quality results
• Demonstrated ability to communicate across all levels of the organization is necessary; must be able to clearly articulate technical ideas to a non-technical audience both verbally and in writing
• Background and drug screen preferred
• Shell/Interpreter scripting a plus (e.g., Bash, PERL, Python)
• Certifications: Splunk Enterprise Certified Architect, Splunk Cloud Certified Admin (working towards or achieved strongly desired)
• Unix/Linux administration background a strong plus
• Experience with logging technologies (implementation/administration) - syslog-ng/rsyslog, ELK, Filebeat, Kafka, Fluentd, Graylog
• X.509/PKI fundamentals with experience on practical implementation
• Practical understanding and troubleshooting experience with the following: UDP, TCP, SSL/TLS, HTTP/S, Rest APIs, load balancers, forward proxies
• Physical Requirements
• Working conditions consist of a normal office environment.
• Work is primarily sedentary and requires extensive use of a computer and involves sitting for periods of approximately four hours.
• Work may require occasional standing, walking, kneeling, and reaching.
• Must be able to lift 10 pounds occasionally and/or negligible amount of force frequently.
• Requires visual acuity and dexterity to view, prepare, and manipulate documents and office equipment including personal computers.
• Requires the ability to communicate with internal and/or external customers.
• Employee must be able to perform essential functions and physical requirements of position with or without reasonable accommodation.

Base Pay Scale

The base pay scale for this position in: Phoenix, AZ/ Chicago, IL in USD per year is: $184,000 - $230,000. San Francisco, CA in USD per year is: $221,000 - $276,000.

Additional Benefits

Candidates are eligible for a discretionary incentive plan and benefits. This pay scale is subject to change and is not necessarily reflective of actual compensation that may be earned, nor a promise of any specific pay for any specific candidate, which is always dependent on legitimate factors considered at the time of job offer. Early Warning Services takes into consideration a variety of factors when determining a competitive salary offer, including, but not limited to, the job scope, market rates and geographic location of a position, candidate's education, experience, training, and specialized skills or certification(s) in relation to the job requirements and compared with internal equity (peers).

Early Warning Services

Early Warning Services is an affirmative action and equal opportunity employer. The company also prohibits discrimination on other bases such as medical condition, marital status or any other factor that is irrelevant to the performance of our employees. Early Warning Services LLC is a proud participant in E-Verify, a federal program to help ensure a legal and authorized workforce. As part of our hiring process, we electronically verify the employment eligibility of all new hires through E-Verify. For more information on your rights and responsibilities under E-Verify please visit Home | E-Verify.