1 Introduction↩︎

The revolution of llm for usage in ai and ml workloads has taken the global research community by storm. The hpc community is a representative subset of this user base that can benefit from using ai/ml models to advance science. The interfaces to interact with llms are typically agents – applications that can receive instructions and respond with context-aware, meaningful text in coordination with ai services [1]. Successful integration of agents into scientific workflows depends on understanding strengths and limitations, and strategy to holistically combine human goals with llm productivity. A llm is only as powerful as its ability to focus on a scoped task, and an agentic team of llms and humans will best achieve a desired outcome with proper protocol for guidance and validation. As natural language becomes a part of a new type of software to create revolutionary new systems, the feedback loop between agents and humans must become tighter [2].

A variety of developer frameworks [3], [4] and hosted services are available for inference, including Google’s Gemini, Anthropic’s Claude, OpenAI’s ChatGPT, and Microsoft Copilot. These ai systems are being used in medicine, education, and science [5]–[7]. They are designed to derive objectives from prompts, and synthesize multiple data sources while using internal tools to plan, reason, and execute complex, multi-step tasks. While the structure is not typically transparent to the user, we can speculate that these systems have checks and balances, can adapt to failure, and use a mixture of experts (MOE) [8] to target sub-tasks to models with tuned expertise. Although a response to the user may appear as a single, cohesive output, it likely results from synthesis of multiple sources of information and multiple steps, each with support from tools. Tools might include symbolic solvers for formal logic, code interpreters to produce accurate mathematical results, and search engines or databases for general knowledge retrieval. A planner that initially processes the prompt is likely responsible for orchestrating agents and tools into an execution pipeline. Industry vendors are ahead of the game to structure these interactions, developing the mcp to standardize ai system interactions [9].

While corporations can afford and utilize these services, scientific groups are more cost constrained and can be limited by institutional regulations. While downloading models and using them on-premises might be allowed, the additional tools and agentic ai systems are not included. The inability to fully utilize the latest ai systems hinders the ability of the scientific community to use them for advances and learning. The requirement to deploy models locally is redundant, and adds management complexity along with capital and operational costs. However, the challenges to using ai for computational science does not deem the task impossible. Scientists can develop infrastructure and software approved for scientific institutions. These needs are demanded by national initiatives such as FG-HPCC and Genesis [10], [11]. A shift from laborious manual task execution to proactive, goal-oriented autonomy would be a considerable benefit to scientific discovery.

1.1 Agentic ai for Computational Science↩︎

Research specific to agents oriented to build, deploy, and optimize hpc applications is in its infancy. We might imagine a future when ai systems are used as first class citizens for scientific workflow development and execution. In that many agentic ai systems decompose high-level tasks into directed graphs of general computational resources, systems that support agentic orchestration [12] or graph-based schedulers [13] could match resources to agent needs. Work to integrate ai into scheduling is already years old, however it is often focused on prediction of a specific attribute such as runtime remaining for the job [14]. An agentic framework to orchestrate a multi-step workflow that might require coordination with humans adds additional complexity to the task. Agents may not yet be able to act independently as an hpc scheduler, but they can optimize resource requests for a specific application.

Scientific workflows often require manual tasks to prepare for using workflow tools. For example, running a Snakemake [15] pipeline might require building a container first. Directed acyclic graph-oriented workflows are typically deterministic without llm routing [16]. An agent-based approach could help to make this process flexible and automated, assisting with preparatory steps, runtime parameters, resource requests, topology, and binding.

Agentic ai offers the opportunity to re-imagine this sequence of steps. While many scientific-based ai tools focus on hypothesis generation, data analysis, experiment design, and scientific writing [17], [18], a less studied and more challenging area of work is in the space of workload orchestration. A successful agentic framework for executing workload steps would require collaboration between agents and human experts to reach a desired outcome. Akin to hosted services [19] and workflow tools, a top level manager would handle decision making, supplemented by human intervention to further validate experiment progress. In our work, we prototype and study such a tool – an agentic team of step-level experts orchestrated by a top level manager and supporting sub-agents to achieve a full build, deploy, and optimization of hpc applications. We make the following contributions:

We start with an introduction to agentic roles and definitions (Section 2) and describe our methods to orchestrate agentic execution and respond to failure. We describe experiments and results (Section 3). We finish with a discussion of lessons learned and best practices for collaboration between agents and humans.

2 Methods↩︎

2.1 Overview↩︎

We aimed to orchestrate the entirety of a build, deployment, and optimization cycle for 5 hpc applications of interest in the leading cloud orchestration framework Kubernetes [20]. We chose Kubernetes because it provides declarative management and structured, programmatic interactions that can be easily used by agents. Our applications include a set that vary in difficult to build, including LAMMPS, AMG, Kripke, the OSU Benchmarks, and Laghos, all of which we have deployed and described in previous work [21]. For our setup, to give the optimization step a choice of resources, we aimed to use an autoscaling setup on aws that includes 21 instance types (Table ¿tbl:table:instances?) that include each of arm64 and amd64 platforms. Instance sizes were chosen to be approximately $3.00 or less. Providing a dimension across platforms, micro-architectures, CPU, and memory would give the agents four dimensions to consider when selecting an instance type.

2.2 Agents↩︎

We designed an agentic framework with a preference for simplicity. While several libraries exist to use mcp [22], for our own learning we chose to minimize external dependencies and maximize transparency by designing each agent with a common base class to query the Gemini api and take on a specific role. While most mcp agent apis are expecting request and responses exclusively in text, we wanted our design to be oriented around a directed graph, and based on execution of controlled commands that resulted in a clear result (e.g., a return code) that would not require returning to an llm to complete an interaction. In this controlled environment, multiple agents that work together form an agentic system. Agentic systems contain different types of agents that operate at different path lengths from the root. Together, they form an agentic graph.

During the execution of a workflow, multiple tasks are connected by inputs and outputs and end in a final state. Each task accepts inputs, and executes a function on the inputs to transform them into outputs. Inputs, outputs, and processing together define a step. The inputs and outputs between two adjacent steps form a shared context. Multiple steps (nodes) that have dependency relationships (edges) and a shared context form a graph. In this graph, a step A is adjacent and directed from A to B if the output of step A serves as the input to step B. As an example, the resulting container uri is an output of a build step, and could be used as input to a deployment step.

A step agent is responsible for a specific task. A single step agent can act as an independent unit or entity, and can be represented as a node in a graph. Two step agents can be adjacent in the graph and joined by an edge if their inputs or outputs are compatible. For example, a build agent builds an image before it is provided to a deploy agent to test. The structure of any step agent is simple: a controlled set of tasks specific to the agent that are combined with structured prompts. The prompts are scoped to derive input (parameters, configuration files, or supporting regular expressions) to direct a task. Input comes by way of the context (Section [sec:context]), which includes pre-defined variables specific to the task at hand (e.g, a container uri for a build agent) and a general details section where a user can provide freeform text instruction to the agent. Each step agent is implemented with actions and validation checks for the task. For example, a build agent takes a response from the llm and writes a Dockerfile to a temporary staging directory, and collects output and error to return as feedback to the llm. A return code of “0" indicates success and proceeds to the next step. A deploy agent is then provided with the Dockerfile and tasked to generate a YAML manifest to deploy it. Step agents use a conversational client, so new attempts are made with memory (model context) of previous failures. Each agent instantiates its own llm model, and this is done so that responsibility does not bleed between agents. For our agentic system, we define step agents for a Dockerfile build, a Kubernetes deployment for a Job and Flux Framework MiniCluster, an optimization task, and a scaling decision.

2.2.0.1 Build Agent

The build agent is an expert at building Dockerfiles scoped for an hpc application, and is allowed to design the Dockerfile to be given to a docker build or docker buildx command in a sandbox on the user system. The execution of the build is done by the agent’s class, and provides the logic to check the return code, and either return to the manager to proceed to the next step, or in the case of a build error, send the output to a debug helper agent to diagnose the issue. The debug helper agent then provides scoped feedback to the build agent to retry. Multiple build attempts form a cycle in the graph, and a successful build determines breaking the cycle and moving on to the next step. Context variables for the build agent include a container uri, an application to build, an environment and platforms to build for, and whether to push or load the image into Docker. A push requires the build environment to have permissions for the task, and the build agent is only allowed retries up to a maximum number of attempts for each step.

2.2.0.2 Deploy Agent

The deploy agent is responsible for the single task of taking an instruction for an application of interest and a Kubernetes abstraction (e.g., Job or Flux Framework MiniCluster [23]) and successfully running the application in Kubernetes. The agent is instructed to deploy a container for a specific environment and application, and with additional context provided in the details section. The deploy agent class validates the manifest generated by attempting to load it into YAML, and then checking that the container uri matches what was requested. The class then writes the manifest to a temporary deploy directory, and applies it to the cluster with kubectl. The details in the manifest including the container, command, size, and resources, are decided by the llm and can be influenced by details in the prompt. The application Pods are monitored, and the class looks for changes of state from Running to Completed or Failed. A maximum waiting time for a monitored Pod can be set by the user, which allows for handling cases such as timeout due to error or insufficient resources. The class can handle edge cases such as timeout, out of memory errors OOMKIlled, or in the case of Flux, the request being unsatisfiable. The class also handles unexpected deletion. In the cases of an error code, the error along with cluster diagnostics are sent to a debug agent to determine and summarize the issue to inform the deploy agent how to retry. The deploy is attempted up to a maximum number of attempts or a successful case (Succeeded status).

2.2.0.3 Optimization Agent

The optimization agent can be viewed as a child of the deploy agent in that it is run after a successful deploy to improve upon a metric of interest. The metric of interest must be defined in the context details – an optimize variable – that triggers its creation. The optimization agent works as follows. A completed result log and manifest are passed forward in the context from a deploy step. The prompt instructs the agent to optimize for the metric of interest, and return a JSON structure with variables to update the previous manifest. For the first execution, there is no means for the optimization agent to know what the metric of interest value is from the provided log, and so a child helper agent class is instructed to generate a regular expression to parse the metric of interest from the log. The result parsing helper agent receives a regular expression back from the llm, and tests it against the log. Validation consists of ensuring there is a match, and then asking for human validation. The user deploying the pipeline is given the task to say it is correct (yes), incorrect (no), or to provide custom feedback (feedback). The result parser agent runs in cycles until a successful result – one that passes automated checks and is approved by the human assistant. The result is cached and reused.

The optimization agent llm is provided with the current and all previously parsed metrics of interest, the latest log and manifest, and instruction for how to optimize from the user. The agent is instructed to return a newly optimized manifest in JSON. Each subsequent retry assesses performance of deploys before it. A kubectl explain is done for the resource of interest (e.g., Job or MiniCluster) to provide the exact fields and descriptions of the manifest that can be changed, and the Kubernetes Python SDK is used to get a summary of nodes (counts and resources) that are available for using. A resources field in the plan is also allowed for the user to provide explicit instructions about what resources to use. Such a field is useful for autoscaling clusters, where nodes available might not be present in the cluster. The optimization agent is required to return an updated manifest, a decision to RETRY or STOP, and a reason. In the case of error, execution returns to the debug agent and then deploy to resolve it before optimization is continued. While the current implementation uses the optimization for Kubernetes, it is not tied to Kubernetes. The same optimization agent could be used by a bare-metal execution deployment agent and tasked equivalently.

Due to common interactions with Kubernetes to get logs or status, the two deployment agents share a common set of functions and underlying Python class. These controlled interactions are how agents access the execution environment. The orchestration software carefully monitors each execution and determines when the state is erroneous. For example, deploying an abstraction to Kubernetes could fail immediately if the YAML file is invalid, or later if execution starts and fails. Collection and filter of error information from the right sources is a strategic task that ensures that a failed attempt can be debugged properly. Direction of the next agent to make a request to with a specific update to the prompt is essential. For example, not capturing a timeout or out of memory issue can lead to updates to the manifest that are nonsensical like increasing the problem size. In the case of failure, a cleanup is typically required to prepare for the next attempt.

Each agent has scoped and structured prompts to perform and retry a task. A prompt defines a clear persona, context, task, and set of instructions. The persona clarifies the agent’s role. The task and context clearly define the goal for the interaction. The instructions are concise, single lines that describe what an agent MUST and MUST NOT do. Using capital letters to provide emphasis makes a difference. Not adding this emphasis leads to agents more often not following instructions. Using regular expressions to filter text noise from prompts improves goal attainment. For example, when debugging output from the build agent, many thousands of lines are generated by apt-get during an Ubuntu build. These extra lines distract from identifying the core issue and can be filtered before sending to Gemini. This reduced our token count (Section [ref:token-counts]) from the order of 300,000 tokens down to a few thousand, and reduced time to parse the response and cost.

A helper agent is an agent used by a step that is primed to perform a specific task, and typically one that does not require conversational memory. When agents are created, api clients are instantiated with the class that can either send one-off model messages, or a message to continue a conversation. We consider the conversation a form of superficial memory, as subsequent prompts are informed by previous ones. In the case of helper agents, we would not want any bias based on a previous prompt. Our two use cases for helper agents were debugging and synthesizing errors (i.e., identifying the issue and summarizing it succinctly), and parsing a result from a log. We call these helpers the debugging and result agents, respectively. The pattern of an agent performing a task and asking for human validation is one that can be extended to other use cases.

An agentic team requires a manager to orchestrate the sequence of steps. The manager takes as input a plan derived by a human defining a sequence of steps. The manager is responsible for orchestration of step agents. Each step has a defined context. For example, the build agent minimally requires an application name and environment to tune the build for. The manager is the root of the graph, executing individual steps, and having flow return to it in the case of error or completion. Each step has a maximum number of attempts, or up to the point when a debugging agent directs flow back to the manager. The manager saves an output file with metadata when the workflow completes. The manager combined with step agents and helper agents form an agentic team (Figure 1).

A shared context object is responsible for sharing state between agentic steps, and provides an interface for setting and retrieving input variables. Each step requires a specific set of inputs to be provided in the context, and inputs can be populated in several ways. The developer user adds a function to the agent class that returns a subparser. Any argument defined there is added to the agent context, and can be customized via the command line, the plan YAML file, or by an agent setting it directly on the context. As an example, a build agent by default will generate and set a container uri. When the context is passed to the deploy agent, the input container is not only instructed for use, but validated by the agent class to appear in the output. The validation ensures that cycles are not wasted testing a uri that does not exist. A user might also create a plan for a previously built container, and provide the uri as the container field under the context. This design allows for each agent to define a different set of context variables as inputs, and flexibility for them to be set by humans or another agent. The manager loads the initial context from the user-defined plan, and each step is provided with the context for execution. Common variables between steps that aid with communication include a final result for the current step, and an error_message. The extent to which we can track and explain the decisions of the llms is the degree of explainability [24]. We found it useful to display inputs, outputs, and prompts in the terminal, and always prompt the llm agents to provide rationale for choices. While this is not standardized collection of provenance [24] it allows us to track what is going on at a level appropriate for being able to improve upon it. Visual monitoring is useful to get direct feedback about what prompt or input data led to a decision, what information might be missing, and what factors led to surprising or improper behavior. In addition to terminal output, the manager saves all intermediate steps and output generated. Each step agent saves custom metadata, function timings, and relevant metrics of interest.

We use the Google Gemini api as our llm backend. Each agent has defined environment interfaces that use the Python subprocess library to execute commands for building or deploying configuration files. We typically do not allow agents to write commands that are executed to the system, but instead to populate the content of configuration files used. We currently do not provision connectors to other external databases, search engines, or apis to supplement the agent. The closest that the llm will get to code execution is writing regular expressions for log parsing to be used by the result agent. We looked into a potential security risk with allowing execution of regular expressions, and saw the risk of a ReDOS [25]. From a hosted service like Gemini provided by Google we saw this as unlikely, and if it happened, would be a bug in a product to report to Google. As an extra layer of precaution, all experiments are run on cloud virtual machines. These controlled interfaces are essential for agents to interact with and adapt to the environment.

2.3 Experiments↩︎

We aim to build, deploy, and optimize 4 hpc applications in Kubernetes on a single node. We provide instructions in the plan to use the Flux Operator MiniCluster crd [23] to deploy an hpc cluster to run each application. We give the optimization agent step 21 types of instances (Table ¿tbl:table:instances?) to select from, and request deployment on one node to maximize fom. We start with one node to assess performance of the agentic team when scale is not a factor. For each of LAMMPS, AMG, Laghos, and Kripke, we give the optimization agent instructions for how to make a decision to retry or stop, along with an instruction for an optimization strategy. For all applications we provide instructions to use a smaller problem size for testing, and increase to a larger size for the optimization. In all cases, the optimization agent step is required to return a decision value of RETRY, or STOP with a reason and resources if a retry is needed. We test the following optimization strategies, each of which instructs the agent to maximize the fom with additional instruction.

• llm decision: Optimize fom and decide when to retry/stop

• User function: Require the agent to execute a function.

• User guided function: Require calling a function with scaling metrics to get back optimization hints.

For all optimization strategies, we provide the agent with a description of resources available, including instance types (CPU and memory) in the autoscaling cluster. For the first optimization strategy, llm decision, we leave the entire decision up to the agentic model. This strategy can be considered free-form in that the prompt does not give the llm an instruction or algorithm for making a decision. For the second strategy, user function, we provide a function in the prompt that calculates a configuration to use for the next step, and ask the agent to run it. This strategy is conceptually similar to using an mcp server in that it mimics the llm executing a function and using the exact output. The third strategy is an intermediate between these two extremes, taking in the current resources and returning a decision to RETRY or STOP with a strategy hint, and providing instruction to also return resources and instance type. The main difference between a user-function and a user guided function is that the latter returns metrics that the llm can use to make its own decision.

We perform a modified version of the single node experiments that instruct the agent to do a build and deploy across multiple (N=4) nodes. We decided to use the m7g.16xlarge due to reasonable cost and best availability across the top 3 contenders as determined by testing. Importantly, the selected instance needed to support the aws efa suggested for hpc workloads [26]. efa allows network packets to bypass the operating system and go directly to the device for lower latency. The agents are instructed to build OpenMPI with libfabric intended for aws efa, and that they should maximize fom. We will use the best performing strategy identified in Section 3.2. Since we are running across nodes, we will add in the OSU Benchmarks in place of Laghos to test the ability of the llm to figure out point to point and collective mpi calls.

For a final experiment, we challenge the agents to complete the entirety of a scaling study. For this work, we test LAMMPS, Kripke, and AMG, and pin the container build to a known working variant from our previous experiments. A scaling agent is added to the agentic team, and is prompted to return a response that decides when to continue or stop scaling toward a user-defined goal (Figure 8). The optimization agent is provided a modified prompt with the context of the scaling study that instructs to change the problem size only at the first size, and then hold it constant (strong scaling). An example optimize and scale plan would define a YAML file to run scaling and optimize agents run under the deploy (MiniCluster) agent. The execution of the deploy agent is allowed a maximum number of 10 attempts, and an execution timeout of 300 seconds. The container provided was previously built by a build agent, and provided to save time and not rebuild each time. The environment definition is minimal, as the details for the cluster size and resources are provided programmatically. To ensure that the execution loop has fewer edges, no returns are allowed to a human or to the manager. The scale agent is instructed to maximize fom and pin a problem size from the smallest size (N=1) and to stop when the application stops strong scaling. The optimize instruction is specific, placing emphasis on points with MUST.

The study is a collaboration between the scaling, deploy, and optimization agents, where the optimization agent derives the configuration, the deploy agent is tasked with running work at each subsequent size, and the scaling agent response determines when to stop. A human is asked to intervene to validate the parsing of a fom from the log. We plan to instruct the agent to build and deploy a multi-node cluster starting at size two up to a maximize size of 32 nodes, and to choose the instance type for each that performed optimally for the multi-node experiments. We instruct the agents to stop when strong or weak scaling has ended, and instruct that each decision be explained with evidence from relevant literature.

Across experiments, we are interested in the agents’ choices for the Docker build, Kubernetes specifications, and the decision sequence to retry or stop. Each application will be run for 10 iterations, with a total number of retries of 10 for each step agent, and 15 retries for the manager. Each application will be allowed a maximum running time of 5 minutes, and a timeout will cancel the run and alert the agent that the application run has timed out. Each result will be assessed by our team for quality, and agent logs will be inspected to learn how we can improve and tighten the orchestration. Our team has expertise in building and deploying hpc applications in cloud environments [21]. We evaluate:

• Overall performance of final optimized variant

• Total time to complete each step

• Number of attempts to successfully complete a step

• Choices step agents take between and within steps

• Dockerfile build logic correctness and completeness

• MiniCluster design logic correctness and completeness

• Decision of when to STOP versus RETRY

• Reasons for failure

For failures, we will assess whether an additional prompt or requirement could better guide the agentic team to a successful outcome. We will run experiments in sessions, and make adjustments to prompts and software design to improve upon initial attempts.

2.3.1 Token Counts and Completion Time↩︎

A token is a chunk of text that an llm processes that can be a single word, part of a word, punctuation, or a space. For Gemini, a token is about four characters [27]. We are interested in the relationship between token counts for requests and responses, and time (seconds) for completion. It is not clear if providing more tokens in a request leads to longer processing time. It could be feasible to have a short response that requires more processing time by the llm.

3 Results↩︎

We performed single node build, deploy, and optimization experiments for 4 proxy applications (AMG, LAMMPS, Kripke, Laghos) across 21 instance types on an autoscaling cluster followed by a multi-node (N=4) optimization study to add the networking benchmark OSU. We finished with a scaling study that extended the experiments to deploy and optimize across sizes. Best fom results are shown in Table ¿tbl:table:performance-results? and each experiment discussed below.

3.1 Tokens↩︎

We analyzed the relationship between token counts and seconds to complete the response (Figure 3). We did not observe any strong relationship between prompt token count and candidate token count, however a pattern reflecting the difficulty of building the application was seen in our plot. Applications that are more difficult to build, either for a human or the llm, produced a higher token count for prompts and candidates. The higher token count is caused by the library framework sending more error output, and receiving more text content. As an example, Laghos (green in Figure 3) was hardest to build, and has the highest prompt and candidate token counts. We also see “stuck sequences” in the data, or a sequence of prompts that are similar in count that likely received similar responses from the debugging agent. This pattern – the inability of a request to return a working answer and trying something similar to return a similar result – might be predictive of an oncoming failure state. It could also be indicative of fine-tuning a response that is closer to a successful solution. We hypothesize that the content of the prompts is similar due to what appears to be equivalent lengths.

3.2 Single Node Experiments↩︎

We asked build, deploy, optimization, debugging, and parsing agents to work together with a manager to deploy AMG, LAMMPS, Kripke, and Laghos on one node. Overall success and failure rates are shown in Figure 4 and best performance metrics in Table ¿tbl:table:performance-results?. A failure indicates that the pipeline was not completed through optimization because build or deploy did not complete successfully. Akin to the patterns described in Figure 3, we found that the difficulty an agent had with a build or deploy step matched our perceived human difficulty, with LAMMPS being the easiest to build and execute, and AMG, Kripke, and Laghos more difficult. Subjectively, more difficult builds require more dependencies, and the dependencies need customization with respect to build flags or settings. LAMMPS is easy to build with cmake, while Laghos is much more challenging because it requires specific versions and flags for each of hypre, mfem, and metis.

3.3 Instance Selection↩︎

Instance selection is the task for the llm to choose a cloud instance type to assign a workload to. For Laghos and Kripke, there was a preference for choosing the hpc7g.16xlarge, with the instance type selected more than 50% of the time (Figure 5). Looking at the agent reported reasons, we see a pattern of the first optimization attempt “selecting a 64-core, HPC-optimized instance.” We hypothesize the choice is based on LLM training data that advertises the hp7g as ideal for hpc applications. When exploring multiple instance types, when the agent determines fom has reached a limit for one instance type it often decides to test other types.

Agent responses that have common, general errors warrant an update to the prompt to avoid the erroneous states. An example is related to discovery of data and executables. If the build agent response did not place the final executable on the PATH or add a comment to the Dockerfile about data file names and locations, it was unlikely for the deploy agent to be successful. Other consistent and surprising errors were not installing certificates and deleting binaries that were just built. Although these errors can be fixed by the debugging agent, it makes the loop tighter to prompt the agent to avoid them.

Results for AMG are included in Figure 6. Allowing the llm to make a decision was the most successful strategy. The foms were \(1.604893\times 10^9\), \(1.417084\times 10^9\), and \(8.048641\times 10^7\) for the llm decision, user guided, and user provided functions, respectively. Despite requesting AMG2023 the agent always built the AMG2013 repository. This preference could derive from the llms training set. The difference is significant because the problem the agent was instructed to run (2) cannot be compared between variants.

We observed from agent output that the instruction from the user function was in direct conflict with what the llm determined was best. We think this conflict results from the lack of fidelity of the hard-coded functions in modeling an environment. The user provided function did not work beyond the testing case because the optimization function produced a resource estimate that was too large, leading to execution that would run out of memory or time out. We learned that without providing guidance for the problem size for the first deploy size, the agent always chose a problem size that was too large and would lead to the pod being OOMKilled. Although the agent could adjust the problem size to be smaller each time, in practice it was not enough to get to a working result in under 10 attempts. We chose to advise the agent to choose a small problem size (no greater than 10) for testing of the initial deploy step. The subsequent information provided about how to optimize AMG for memory would then prevent the same error. If asking for a problem size that is too large is a common pattern, the agent might be given instructions for how to adjust problem sizes. Instead of a slow linear decrease, a binary search and then reaching a point when the fom stops improving might be more efficient.

Through a collaborative process of watching experiments run, we noticed that the key to AMG running well was to force OpenMP and any math libraries to use only one thread per MPI rank. While this insight could have come from an expert user, it was the llm that reminded us of threading and we could reliably give the advice to the agent to make it the application run performant.

LAMMPS results are shown in the third panel of Figure 6. While the user provided function mean value outperformed that of the user guided function, the variance across runs was twice as large, suggesting that the result is not consistent. The llm decision outperformed both, leading to a fom that was over 2x improved. The ease of building and deploying LAMMPS was reflected in our results. Builds did not require many debugging prompts (green in Figure 3) and the number of attempts was consistently fewer than five. The only application to consistently reach the maximum attempts was Laghos, which needed an increase to N=20. The ease of building mirrors the human experience.

Laghos was an exemplar of a build that required collaboration. In our testing runs, the agent was unable to build a container in the maximum attempts we allowed (N=10). Laghos is hard to build because several components (hypre, mfem, and metis) each require specific build flags and compile options. While the llm might figure out a correct configuration eventually, the maximum attempts setting exists to disallow an unreasonably large number of builds. We fixed this by providing the agent with human expertise – a description of the versions to build. With this expertise, the agent was successful every time. However, the components still took many attempts to get working together, reflected in a large number of prompts and responses for the build (Figure 3). We did not attempt a user provided function for Laghos due to the length of the build and previously poor performance with AMG.

The Kripke llm decision outperformed the user guided function (Figure 6), as a smaller grind time indicates higher performance. The fom, Grind time per units of work (where smaller values are better) was order \(10^{-8}\).

3.4 Multi-Node Experiments↩︎

A multi-node experiment requires extending the base container to include an OpenMPI build with libfabric enabled for efa. We chose OpenMPI since it comes packaged with the efa installer, and find that it works well across clouds and environments [21]. We learned that asking for this additional dependency, especially for a multi-platform build, increases the time and difficulty of the build. Strategies for caching and reuse of images become increasingly important, and we adopted our strategy to ask the build agent to first build a base image with OpenMPI and libfabric, and then for the application builds we instructed it to use the shared base image.

The optimized fom for AMG on 4 nodes was \(6.132476\times 10^9\) nnz/s on the m7g.16xlarge. We observed that after the deploy agent deemed the application working in Kubernetes, the optimize agent increased the resources of the run to occupy the full node. While the agent was not instructed to, it chose to test two nodes before increasing to full size (N=4). We observed negligible improvement in performance, and attributed it to a small problem size. The next attempt scaled up to the full four nodes at maximum resources, and achieved a fom of \(6.13\times 10^9\). From this point, it increased problem size until performance degradation, which it attributed to “exceeded the available memory or hit other scaling limits.” LAMMPS optimized on 4 nodes resulted in a maximum fom of 2.50 Matom steps/s. During optimization, the agent jumped immediately from one to the maximum of 4 nodes and 256 tasks, and increased problem size to -v x 32 -v y 32 -v z 16. This initial attempt was too aggressive and timed out, leading the agent to adjust to a smaller problem size (-v x 24 -v y 12 -v z 12) to establish a baseline (2.267 Matom/s). The final attempt increased the problem size to the deemed optimized fom (-v x 30 -v y 15 -v z 15).

OSU Benchmarks executed successfully without prompt tuning. OSU Latency had performance comparable to previous performance studies using efa [21] and OSU AllReduce showed high variability across message sizes (Figure 7). Agents successfully enabled collective and point to point communication.

Kripke Grind time optimized to run across 4 nodes was comparable to one node, with a slightly higher time to perform a unit of work (\(8\times 10^{-10}\) and \(7\times 10^{-10}\) for each of 4 and 1 nodes, respectively). The decrease in performance likely results from the added need for network communication. Since it could be challenging in our experience [21] to get Kripke parameters correct, we requested the llm to test running kripke without parameters.

3.5 Scaling Study↩︎

We performed an agentic scaling study for LAMMPS, Kripke, and AMG. We had wanted to use a large cluster with 32 nodes, but were only able to provision 5 hpc7g.16xlarge nodes over the course of a week. The foms are shown in Figure 8.

We closely monitored each study for correctness. We learned that agents need specific instructions about rules for each step. For example, the optimization agent must be given instructions to keep trying until a condition like a timeout, and whether to minimize or maximize fom. The specific size and history of fom progression at each size is helpful. In early testing, we observed that when we got application-specific prompts correct, the study ran smoothly, meaning observing subsequent attempts were carefully changing the application in a way we thought reasonable without making mistakes (e.g., stopping too early, changing a problem size incorrectly).

We carefully monitored each decision and step in the scaling experiments, and learned how to handle unexpected edge cases and how to improve usability. For example, we had a case where the job controller did not recreate a Pod that did not bootstrap correctly with Flux. The ideal outcome would be to delete the Job, disregard the run, and try again. We added support for this case, a “Lost” status that can be triggered by unexpected deletion of the MiniCluster. We also learned that it was strategic to prompt the deploy agent to give unique names to Job with an incrementing number to avoid naming conflicts and to provide a live record via completed Pods for a human participant to inspect. Finally, we identified a tradeoff between choosing fewer edges in the graph and adding edges for error debugging. During the single-node optimization step, it would require fewer edges to give the optimization agent information about a failed run to debug. However, this practice violated our principle to have each agent perform one function, and in practice we saw better debugging by exiting the scaling and optimization loops and sending the diagnostic logs to the debugging agent to synthesize the problem and redeploy back into a working state.

The LAMMPS optimization agent running on five nodes reached a maximum fom of 3.159 Matom steps/s. We observed the scaling agent carefully increasing the problem size until the fom stopped improving, even reporting on the percentage improvement between attempts at times. We observed that agent responses included correctly calculated percent changes in fom output, suggesting that Gemini uses supporting tools on its backend. The agent step stopped and returned the result when the execution timed out. Attempts for 2,3,4, and 5 nodes followed instructions to use the fixed problem size 18 x 18 x 18. Comparing the single node optimization to the single node study, the fom was slightly improved, hitting 743.061 katom-step/s, an improvement from 724.238 katom/steps using a problem size of -v x 16 -v y 16 -v z 16.

The scaling study produced a best fom of \(6.424653\times 10^9\) nnz/s for AMG on 5 nodes. For optimization on one node, the agent immediately increased the problem size to -n 270 270 270, which resulted in the Pods being OOMKilled. Subsequent attempts slowly decreased the size (270, 215, 150, 100) until a successful run was found (-n 80 80 80). The agent then reversed the pattern, increasing the problem size slowly until the “optimal” (-n 90 90 90), where the agent stopped due to OOM. While other applications typically had one run for sizes 2-5, the agent acted differently here, trying several variants of the processor topology for several sizes.

Kripke’s best fom for the scaling study at 5 nodes was a Grind time of \(6.012693\times 10^{-10}\) (s/iter)/unknowns. The optimization agent started at a small number of cubic zones (with edge length 32, 48, 64, 80, 96, 112, 120, and 128), and methodically increased until the execution ran out of memory. Size 128 failed, and size 120 was determined optimal. At sizes 2, 3, 4, and 5 nodes, the same problem size was returned in the response to run, and responses tested three different configurations of processor topology to maximize the fom. We observed a linear speedup that would be possible if the problem size per core became small enough to fit into the CPU’s fast cache (96 MB L3 cache), reducing time waiting for data from the slower main memory. As an example, our problem running on 5 nodes of the hpc7g.16xlarge had a fixed total problem size of 103,555 MB. As we scaled up to 5 nodes, the data footprint per core decreased from 1,618 MB to 324 MB. While the full domain per core still exceeds the cache capacity, the Kripke wavefront of actively computed data is much smaller. By scaling to five nodes, this active wavefront is small enough to be cache-resident. We think that the transition to using the cache shifted the performance bottleneck from slow main memory to the faster L3 cache, resulting in an efficiency greater than 100%. We view this optimization and scaling effort as highly successful, as tuning happened for problem size and processor topology.

3.6 Result Assessment↩︎

For our result assessment, we went through each study reported in Table ¿tbl:table:performance-results? and qualified good and bad design choices. We were interested in Dockerfile and MiniCluster configuration logic correctness and design, and parameter or other environment choices. We include a summary of common good and bad choices for applications in Table ¿tbl:tab:features-summary?.

4pt

Despite not being advised about networking, the LAMMPS build agent set environment variables to assume ethernet, which we considered a non-ideal choice in the case that the user deployed a different device. Most builds, with the exception of Kripke, had too many distinct layers that should have been bundled into one layer. The Laghos build would not have been possible without very strict prompt input.

4 Discussion↩︎

In this work, we demonstrate the ability of an agentic team to semi-autonomously execute the entire workflow to build, deploy, and optimize hpc applications. There are several topics for discussion.

Collaboration between llm agents and humans is important. We learned about a new CPU request unit, a millicore [28]. From observing experiments, we learned that llm agents require succinct prompting, and user validation along the way ensures the agent is parsing correctly. Asking the llm to provide rationale for choices gives insight to see flaws in logic. The choice of number of retries to allow an agent, and the strictness of guardrails provided must strike a balance that allows the agent to explore, but up to reasonable constraints. In any case, the entire system must be able to self-heal, or come back from erroneous state or respond to context drift [29]. While there are many strategies for self-healing systems [30], we chose a simple strategy to ensure several means to restart: a debugging agent can decide at any time to return to the manager, or the same outcome can result if the number of maximum attempts is exceeded. We also give the agent the option to return to a human.

Care must be taken when advising the agents to use specific versions of software or dependencies. We discovered that asking for a specific version of a library tended to pull versions of other software and base images toward specific points in time (e.g., asking for OpenMPI 4.1.2 produced builds with Ubuntu 20.04 and libfabric 1.13 that is no longer compatible for efa).

Asking the llm the algorithm it was using to determine when the application was optimized gave us insight into observed behavior. The llm often reported a greedy hill-climbing approach, where it would evaluate the last choice, determine if the change improved or was detrimental to performance, and then would either step back to a previous configuration or continue in the same direction. Many of the stopping conditions we saw as premature, and we think more human guidance about an approach would be needed.

Our most successful experiments were specific about optimization. Minimizing walltime without specifying the problem size is insufficient because the llm can simply run a small problem more quickly. Not providing a small test case can allow the llm trying to run problems that are too large too quickly. Not providing a strategy or function for incrementing resources can lead to jumps that are too large or too small for the number of attempts allowed. Best practices for individual steps can often make the step more challenging for the agent. For example, while it is the case that a multi-stage build is a best practice for Docker images to reduce image size, in practice it made it more challenging for the build agent, because library dependencies or required data were less likely to be carried forward between stages.

We learned early on that without emphases added to our prompts, (e.g., You MUST) the llm would often not follow the instruction. Adding capital letters made a large difference when we absolutely did not want a behavior. When the llm made a decision that went against one of our requests, it added code commentary that would justify the choice.

The most dangerous failure cases were those that the agent could not learn from (e.g., a timeout where the agent does not get feedback) leading to guessing behavior.

4.1 Future Work and Limitations↩︎

We have completed follow up work that extends this framework to include a hierarchical agentic server, and orchestration with mcp.

We recognize that other LLM api endpoints could have been used given equivalent opportunity. We recognize that more research is needed to better understand when and how a human should intervene.

5 Conclusion↩︎

Agentic frameworks are the future for not just the hpc community, but the entire software ecosystem. Autonomous, converged hpc infrastructure that can collaborate with humans to orchestrate the entire lifecycle of scientific workloads is becoming feasible. From autonomous task execution to failure recovery and ai-supported scheduling techniques, these powerful transitions will fundamentally change our daily work lives and how scientists, engineers, and developers pursue scientific discovery. We look forward to a new era of faster, more accurate, and more accessible computing for the next generation of HPC.

References↩︎