Senior Systems Administrator

ASRC Federal is searching for a Staff HPC Engineer to support ASRC Federal Technology Solution out of Manhattan, KS. ASRC Federal Technology Solution proudly supports the USDA Animal and Plant Health Inspection Service (APHIS) Marketing and Regulatory Programs - Information Technology Division (MRP-ITD). The USDA APHIS mission requires rapid, secure, and reliable access to large-scale scientific data for genomics, diagnostics, modeling, and analytics to support decision-making, research, and emergency outbreak responses.

The Scientific Computing (SciComp) HPC system and the Marketplace for Data Analytics (MDA) form the computational, data analytics, and data-sharing backbone for APHIS's laboratory network as well as other MRP Program areas. These systems facilitate high-speed data transfer, AI modeling, data visualization, and reproducible scientific outcomes through the integration of on-premises HPC clusters, cloud-based analytics platforms, and secure data transfer via Internet2 (I2). Our employees embrace innovation and are committed to a culture of continuous, standards-driven process improvement and assimilation of the industry's best practices.

Summary: The successful candidate will be an active supporting member of the ASRC Federal team. The HPC Systems Engineer/Administrator is responsible for the administration, maintenance, optimization, and lifecycle management of a high-performance computing (HPC) environment supporting scientific research. This position ensures the availability, security, and performance of compute, storage, networking, and authentication infrastructure while providing technical support to USDA APHIS researchers and collaborating with on-site facilities, networking, and cybersecurity teams.

Primary Roles, Duties and Responsibilities

HPC Infrastructure Administration

• Administer and maintain a Linux-based HPC environment consisting of compute, login, management, and GPU nodes.
• Manage Slurm workload manager configuration, partitions, queues, scheduling policies, and resource allocation.
• Perform operating system installation, provisioning, upgrades, and lifecycle management.
• Develop and maintain automation using Ansible, shell scripting, and other administrative tools (NVIDIA Base Command Manager).
• Monitor overall cluster health, capacity, utilization, and performance.

Storage Administration

• Administer and optimize a multi-petabyte Lustre parallel file system (HPE E1000).
• Configure and maintain storage quotas, project directories, tiering policies, and data placement strategies.
• Troubleshoot Lustre performance, metadata operations, changelog processing, and client connectivity.
• Monitor storage utilization, capacity growth, and system performance using enterprise monitoring platforms (Starfish Storage).

High-Speed Networking

• Administer and troubleshoot NVIDIA InfiniBand fabrics, including Subnet Manager, routing, and multi-rail configurations.
• Maintain NVIDIA Base Command Manager and Cumulus Linux infrastructure supporting the HPC environment.
• Optimize MPI, UCX, NCCL, and RDMA communications for distributed scientific workloads.
• Support Internet2 connectivity and high-performance data transfers (Globus Data Transfer Nodes).

GPU Computing Support

• Support NVIDIA GPU compute nodes and associated software stacks.
• Install, configure, and maintain CUDA, NCCL, GPU drivers, and related scientific software.
• Benchmark and validate GPU performance following infrastructure changes.

Software Support

• Install, configure, and maintain scientific software packages and development environments.
• Build and install software from source using Make, CMake, Ninja, and other build systems.
• Configure and troubleshoot software compilation using GCC, G++, and other compiler toolchains.
• Resolve library dependencies, compiler compatibility issues, and software linkage problems.
• Develop and maintain environment modules supporting multiple software versions.
• Support Conda, R, Apptainer, compilers, MPI implementations, CUDA toolkits, and GPU-enabled applications (Apptainer and Singularity containers).
• Assist researchers with software deployment, optimization, and troubleshooting.

Identity and Security

• Administer authentication and authorization services including Active Directory integration, LDAP, SSSD, Kerberos, and single sign-on technologies.
• Coordinate operating system patching and security updates.
• Support vulnerability remediation and infrastructure hardening activities.
• Maintain least-privilege access controls and security best practices.

Performance Engineering

• Evaluate system performance using industry-standard benchmarks.
• Analyze CPU, GPU, memory, storage, and network performance bottlenecks.
• Optimize NUMA configuration, processor affinity, MPI communication, and storage performance for scientific workloads (bioinformatics).
• Validate infrastructure performance following hardware or software changes.

Documentation and Operational Support

• Develop and maintain standard operating procedures, technical documentation, and knowledge base articles.
• Participate in change management processes and maintain accurate operational documentation.
• Coordinate maintenance activities and production system changes.
• Support disaster recovery planning, backup strategies, and infrastructure lifecycle planning.

Collaboration and User Support

• Provide advanced technical support to researchers and scientific staff.
• Collaborate with networking, storage, cybersecurity, and vendor support teams to resolve complex technical issues.
• Mentor junior technical staff through knowledge transfer, collaborative troubleshooting, and hands-on training.
• Participate in procurement planning, hardware evaluations, and technology modernization initiatives.

Typical Daily Activities

• Monitor cluster health and respond to system alerts.
• Troubleshoot compute, storage, network, and authentication issues.
• Deploy operating system, firmware, kernel, and software updates.
• Optimize HPC performance for scientific workloads.
• Install and validate scientific software requested by researchers.
• Develop automation to improve operational efficiency.
• Maintain system documentation and operational procedures.
• Coordinate infrastructure maintenance and change activities.
• Assist users with job scheduling, software execution, and performance optimization.
• Work with vendors to diagnose and resolve hardware and software issues.

Requirements:

• Bachelor's degree in computer science or related field
• Strong computer science background with in-depth systems-level knowledge in operating systems and networking
• 5+ years demonstrated competency in provisioning and configuration management of medium-sized Linux environments using tools such as RHEL Satellite, Ansible, Base Commander
• Ability to work with system configuration management tools (e.g., Puppet, Ansible) and revision control software such as CVS, SVN, Git
• Superior scripting skills and excellent attention to detail; proficiency in at least Python, Perl, or Bash
• Demonstrated troubleshooting skills in a complex environment beyond systems, from networks to other dependent systems
• Excellent customer service skills and attention to detail supporting hundreds of scientific users
• Excellent communication and people skills; excellent time management and organizational skills

Preferred Qualifications:

• Experience with Base Commander or other orchestration platforms
• Previous experience in support of HPC environments and complex scientific computing centers
• Familiarity/proficiency with OpenMP and Message Passing Interface (MPI) programming
• Experience with Lustre and InfiniBand
• Experience with Slurm

#USDA,#HPC,#LINUX,#RHEL,#ANSIBLE,#BaseCommander,#GPU,#Slurm