14:40 - 15:00
Custodian: Enabling Automatic Performance Engineering Agents on the NERSC HPC System
Abdul Hannan Mohammed, Oscar Antepara, Dhruva Kulkarni, Muaaz Gul Awan, Samuel Williams, Jack Deslippe
Lawrence Berkeley National Laboratory, CA, United States
The advancement of Large Language Models and the rapid evolution of AI assistants and agents create an opportunity for the development of highly specialized HPC agents to enable automatic performance analysis. In this paper, we present Custodian, an HPC performance analysis agent that analyzes previously collected job telemetry and application-output data to characterize application behavior, identify potential performance bottlenecks, and transform performance metrics into natural-language descriptions and figures that suggest optimizations or highlight computing inefficiencies. We validate Custodian’s telemetry-derived application states and roofline analyses against NVIDIA Nsight Systems and Nsight Compute across four representative applications executed on NERSC Perlmutter. Across these applications, the telemetry-derived state distributions differed from Nsight Systems by a mean absolute difference of 1.61 percentage points. In a separate reproducibility experiment involving 27 executions across three language models and three applications, every execution identified the same primary bottleneck and optimization direction. Finally, we discuss the tradeoffs, cost, productivity and systems applicability of HPC agents on the NERSC HPC system.
