Job Description
| Job Title: | AI Performance Engineer |
| Location: | Bay Area, CA [ONSITE] |
Job Description:
The engineer will lead performance benchmarking, analysis, and tuning of provisioned AI clusters
across distributed training, inference, and GPU collective communication. This senior specialist role
establishes reproducible baselines, identifies gaps against comparable reference configurations, and
works with platform and network teams to resolve bottlenecks. The engineer will validate improvements
while preserving model quality, numerical correctness, and production stability.
Key responsibilities-
- Define benchmark plans and acceptance criteria covering single-node and multi-node workloads.
- Compare results with references using equivalent hardware, models, precision, software versions,
- and workload settings.
- Build automated benchmark suites using MLPerf-style methodologies, NCCL Tests, NeMo/Megatron, TensorRT-LLM/trtllm-bench, NVIDIA AIPerf, and vLLM/SGLang load testing.
- Measure training throughput, step time, model FLOPs utilization (MFU), scaling efficiency, and collective bandwidth and latency. Track inference throughput, time to first token, inter-token and tail latency, and GPU memory use.
- Profile bottlenecks with PyTorch Profiler, Linux perf, NVIDIA Nsight Systems/Compute, and DCGM.
- Correlate workload behavior with compute, memory, storage, network, and host-level evidence.
- Tune batch sizes, sequence lengths, data loading, tensor/data/pipeline parallelism, and communication overlap. Optimize serving concurrency, batching, and KV-cache settings for representative traffic.
- Evaluate supported precision and quantization options against model quality and numerical correctness requirements. Document throughput, latency, memory, and quality tradeoffs before recommending changes.
- Investigate NCCL, GPU interconnect, RDMA, topology, and host configuration issues with platform and network teams. Separate application and framework bottlenecks from infrastructure limitations.
- Control warm-up, repeat runs, report variability, and check for load-generator bottlenecks. Deliver profiling evidence, prioritized remediation, validated configurations, and automated performance regression tests.
- 7+ years in performance engineering, HPC, ML systems, or GPU infrastructure, including hands-on benchmarking of distributed training and inference workloads.
- Strong Python and Linux skills for benchmark automation, evidence collection, system diagnostics, and repeatable test execution; practical C++/CUDA understanding.
- Experience with PyTorch, GPU profiling, collective communication, and distributed execution, including the impact of parallelism, synchronization, memory use, and data movement.
- Hands-on use of training and inference benchmarking tools, with the ability to select representative workloads and interpret throughput, latency distributions, and scaling behavior.
- Knowledge of benchmark statistics and experimental controls, including warm-up, repeated measurements, run-to-run variation, and fair comparison against reference results.
- Ability to isolate cross-layer bottlenecks, validate performance changes without compromising correctness, and explain findings and remediation priorities to engineering stakeholders.
- Experience qualifying NVIDIA AI clusters at scale, including GPU topology, InfiniBand/RoCE, RDMA, NCCL behavior, and hardware, driver, and runtime compatibility.
- Experience with NeMo/Megatron training, MFU analysis, TensorRT-LLM, vLLM/SGLang, NVIDIA Dynamo, and AIPerf or GenAI-Perf.
- Experience integrating benchmark suites with deployment and release validation, using Nsight and
- DCGM evidence to close performance gaps and detect regressions.
- Preferred: NCP-AIO. Specialization: NCP-GENL. Foundational: NCA-AIIO.
- Certifications are preferred credentials. Candidates should demonstrate the corresponding skills
- through practical benchmarking, profiling, and performance tuning experience.
- Deliver automated benchmark suites, reproducible baseline reports, profiling evidence, prioritized remediation plans, validated tuning configurations, and regression tests. Success is demonstrated through documented closure of gaps against comparable references, improved throughput and scaling or reduced latency, and repeatable results that preserve agreed model quality and numerical correctness.
Numbers & Facts
| Location | San Jose, CA |
| Industry | Computer Software |
| Company Size | 20 to 49 employees |
| Year Founded | 2014 |
| Website | http://www.tekfortune.com/ |
About Company
Tekfortune is a software solutions company providing project development and staff augmentation services to companies in various industries including Healthcare, Banking, Finance, Telecommunication and Aviation.
Tekfortune Inc delivers solutions to a wide variety of organizations of various sizes and industries. Our clients include mid-market to Fortune 1000 companies spanning across industries such as: Insurance, Financial, Pharmaceutical, Manufacturing, Retail and many others.
In Tekfortune Inc, we are offering selection of services and solutions at very reasonable rates. With years of experience in the placement and recruitment Industry and over time we have acquired the knowledge essential to comprehend the exact need and dependence on customers. Therefore we provide the customers the best and also the brightest candidates for vacancies. Moreover, by understanding the working interest of customers who is able to place them within the companies where they are able to get maximum opportunity for professional growth.
Tekfortune Inc has a strong emphasis on quality. Its goal would be to achieve client satisfaction by looking into making deliveries satisfaction of customer's business needs. Our ability to quickly identify and effectively employ proven fundamental principles, techniques and resources yields highly desirable results for our clients.
We understand that each industry and organization share a level of uniqueness that requires a tailored approach in order to deliver specific solutions. Such factors as client culture, industry regulations, organizational maturity, financial health and many more items may impact our approach to delivering client solutions.
Having served a wide variety of clients has provided Tekfortune the ability to build subject matter expertise across numerous industries. Our industry expertise combined with our specialized knowledge of Business Process Improvement, Requirements Management and Program & Project Management provides a powerful combination and a tremendous value to our clients.
We are run by a group of industry experts who have decades of experience in delivering custom solutions to our clients.
Tekfortune Inc delivers solutions to a wide variety of organizations of various sizes and industries. Our clients include mid-market to Fortune 1000 companies spanning across industries such as: Insurance, Financial, Pharmaceutical, Manufacturing, Retail and many others.
In Tekfortune Inc, we are offering selection of services and solutions at very reasonable rates. With years of experience in the placement and recruitment Industry and over time we have acquired the knowledge essential to comprehend the exact need and dependence on customers. Therefore we provide the customers the best and also the brightest candidates for vacancies. Moreover, by understanding the working interest of customers who is able to place them within the companies where they are able to get maximum opportunity for professional growth.
Tekfortune Inc has a strong emphasis on quality. Its goal would be to achieve client satisfaction by looking into making deliveries satisfaction of customer's business needs. Our ability to quickly identify and effectively employ proven fundamental principles, techniques and resources yields highly desirable results for our clients.
We understand that each industry and organization share a level of uniqueness that requires a tailored approach in order to deliver specific solutions. Such factors as client culture, industry regulations, organizational maturity, financial health and many more items may impact our approach to delivering client solutions.
Having served a wide variety of clients has provided Tekfortune the ability to build subject matter expertise across numerous industries. Our industry expertise combined with our specialized knowledge of Business Process Improvement, Requirements Management and Program & Project Management provides a powerful combination and a tremendous value to our clients.
We are run by a group of industry experts who have decades of experience in delivering custom solutions to our clients.
Skills
Level up your application
Professional resume templates
Browse dozens of recruiter approved resume templates, layouts and formats. Choose your favorite and make it your own in minutes.
Free resume templatesResume Builder
Improve your existing resume or start from scratch and create a standout, ATS-friendly resume. Add job-specific content, download and apply.
Resume Builder