Senior Engineer, Network and Systems

Position Purpose

The Senior Systems and Network Engineer provisions, operates, and secures the Company's core infrastructure, with primary focus on Linux servers, VMware virtualization, and storage. Because the Company has no separate network team, this position also owns the data center and corporate network, including the routers, switches, and firewalls that provide routing, VPNs, and perimeter security. The position is based in Dallas and supports the Company's two Dallas-area offices and its Dallas and Denver data centers, remotely managing Denver and assisting the onsite Denver counterpart with major projects as needed. In addition to day-to-day operations, the position monitors and evaluates systems and procedures to protect the Company from unauthorized access, recommends and implements improvements to security, reliability, and utilization, and helps ensure compliance with software licensing, network, and systems legal requirements.

Essential Duties/Responsibilities

Systems, Virtualization & Storage

• Deploy, maintain and support core IT infrastructure, including Linux servers, VMware vSphere, physical server and storage hardware, and file, email (Postfix), application and DNS servers
• Administer NetApp storage and storage protocols (iSCSI, Fibre Channel, NFS, SMB/CIFS), including provisioning, capacity planning and performance monitoring
• Manage physical server hardware primarily through out-of-band tools (Dell iDRAC/HP iLO), including remote provisioning, firmware lifecycle management and hardware health monitoring
• Perform hands-on data center work in the Dallas data center as needed for installations and projects
• Maintain, patch and upgrade server and network operating systems and applications
• Monitor infrastructure health and performance and respond to alerts.

Network & Perimeter

• Design, implement and maintain the Company's data center and corporate LAN/WAN infrastructure, including switching, routing, firewalls, IPSec VPN tunnels, dynamic routing protocols (including BGP on servers such as load balancers) and perimeter security
• Deploy, maintain and support routers, switches and firewalls
• Administer load balancing services (HAProxy)
• Maintain responsibility for VPN gateways and intrusion detection system

Identity, Authentication & Email Security

• Manage network authentication, authorization and directory services, including RADIUS, TACACS+ and LDAP systems
• Support the Active Directory-based security model, which acts as the root authority for most authenticated applications, by integrating Linux systems and services with it (LDAP, Kerberos, SSSD)
• Manage certificate and PKI lifecycle
• Configure and support the email authentication framework (SPF, DKIM, DMARC)

Security & Risk Management

• Develop and deploy IT infrastructure hardening standards, and help shape the Company's IT security and infrastructure standards
• Troubleshoot and identify possible security threats; anticipate hardware and software degradation issues
• Secure business applications to reduce risk; abide by third party established regulations where applicable
• Help ensure compliance with software licensing, network and systems legal requirements

Backup & Disaster Recovery

• Implement and maintain backup and disaster recovery systems across the Dallas and Denver data centers

Automation

• Automate provisioning, configuration and routine operational tasks through scripting and configuration management

Multi-Site Support (Denver)

• Develop and deploy IT infrastructure hardening standards, and help shape the Company's IT security and infrastructure standards
• Troubleshoot and identify possible security threats; anticipate hardware and software degradation issues
• Secure business applications to reduce risk; abide by third party established regulations where applicable
• Help ensure compliance with software licensing, network and systems legal requirements

Documentation, Standards & Administration

• Maintain Company server and network documentation and information retention policies
• Collaborate with divisional resources to ensure that the Company's IT standards are achieved
• Meet fiscal requirements while ordering new equipment including servers, routers, switches, network storage, software, etc.

Availability & Schedule

• On-call: participates in a rotating after-hours on-call schedule for infrastructure related incidents.
• Work requires periodic evening and weekend work for planned maintenance windows and major projects

Job Requirements

Required

• Bachelor's degree in Information Technology, Computer Science, or a related field, or equivalent experience
• 7+ years of progressive experience in systems and network administration, including hands-on production administration of Enterprise Linux and responsibility for routing and firewall infrastructure (or an equivalent combination of education and experience)
• Expert proficiency with Enterprise Linux (CentOS, Rocky, Red Hat), VMware virtualization (vSphere, vCenter, etc.) and server hardware management (Dell iDRAC/HP iLO, firmware lifecycle)
• Solid understanding of PAM, LDAP, RADIUS, TACACS+, DNS (BIND and domain registration), certificate and PKI lifecycle, host firewalls, HAProxy, Apache, NGINX, Postfix, NTP and other common Linux server applications
• Experience automating infrastructure tasks with scripting (e.g., Bash, Python) and configuration management tools (e.g., Ansible)
• Advanced proficiency with NetApp network storage and storage protocols such as iSCSI, Fibre Channel, NFS and SMB/CIFS
• Advanced proficiency with Palo Alto firewalls, IP routing and IPSec VPNs
• Advanced skills in dynamic routing protocols such as OSPF and BGP, including BGP on Linux hosts; working knowledge of Cisco routers and switches and Spanning Tree
• Ability to work with confidential information in a secure and professional manner

Preferred

• Experience with Meraki equipment and management portal
• Working knowledge of Zabbix network management, Sandvine traffic shaping and Infoblox IP management
• Familiarity with CSG Systems, Sugar Customer Management System and Catcher Middleware, and how they relate to the network and system infrastructure
• Relevant certifications (e.g., RHCSA/RHCE, VCP, PCNSE, CCNP)
• Experience with highly available database clustering, specifically MariaDB (Galera clusters) and PostgreSQL (Patroni clusters)
• Familiarity with Kubernetes container orchestration, including the administration, maintenance, and troubleshooting of self-hosted, in-house clusters

Working Conditions

• Employee remains in the sitting position for prolonged hours. Employee is occasionally required to stand, walk, use hands to handle or feel objects, tools or controls; reach with hands and arms; talk and hear. Employee must occasionally lift and/or move up to 50 pounds. Specific vision abilities required by the job include close vision, distance vision, color vision, peripheral vision, depth perception and the ability to adjust focus.
• Working conditions may include being in an open (shared) cubicle/workspace area.

Disclaimer

This job description is not meant to be an all-inclusive statement of every duty and responsibility which will ever be required of an employee in this position, however, the employee will be held responsible for all duties assigned.