SageMaker AI: Accelerating Agentic Tool Calling with Serverless Model Customization

Agentic AI has revolutionized how we think about automated tasks, enabling systems to make decisions and interact with the world through specialized tools. However, the true utility of AI agents in production hinges on their ability to reliably perform agentic tool calling. This is how agents query databases, trigger complex workflows, retrieve real-time data, and act decisively on a user's behalf. Unfortunately, a common roadblock to broad adoption has been the tendency of base large language models (LLMs) to hallucinate tools, pass incorrect parameters, or attempt actions when clarification is needed. Such failures erode trust and significantly hinder production deployment.

Amazon SageMaker AI is stepping up to solve these critical challenges. By offering serverless model customization, developers can fine-tune LLMs for robust agentic tool calling without the typical operational overhead. Central to this innovation is Reinforcement Learning with Verifiable Rewards (RLVR), a technique that empowers models to generate and validate their own responses, learning to favor successful tool interactions. This post delves into how SageMaker AI, utilizing RLVR, dramatically improves agent reliability, showcasing a 57% enhancement in tool call reward on unseen scenarios with a fine-tuned Qwen 2.5 7B Instruct model.

The Promise and Perils of Agentic Tool Calling

The concept of AI agents interacting with external systems via tools is a cornerstone of advanced AI applications. Imagine an agent that can book flights, summarize documents from a database, or even execute code based on a natural language prompt. This functionality is precisely what agentic tool calling enables. Yet, the path to reliable tool use is fraught with challenges.

Base LLMs, while powerful in language generation, often lack the nuanced understanding required for precise tool invocation. They might infer a tool that doesn't exist, misinterpret user intent leading to incorrect parameter values, or fail to recognize when critical information is missing. These missteps lead to frustrating user experiences and make enterprise-level deployment risky. For organizations looking to operationalize AI agents effectively, ensuring predictable and trustworthy tool execution is paramount. The stakes are high, as reliable agents can unlock unprecedented levels of automation and efficiency, while unreliable ones can lead to costly errors and user dissatisfaction. This is why robust model optimization for agentic workflows is essential, a task made simpler with platforms like SageMaker AI.

Serverless Model Customization: SageMaker AI's Advantage

The traditional approach to improving LLM performance often involves significant infrastructure management – from GPU procurement and memory orchestration to complex reward infrastructure and checkpointing for reinforcement learning. These tasks introduce considerable operational overhead, diverting valuable developer resources from focusing on the core problem: refining model behavior.

Amazon SageMaker AI's serverless model customization removes this burden. Developers can select a foundation model (e.g., Qwen, Llama, GPT-OSS), configure a fine-tuning technique like RLVR, point to their data, and define a reward function. SageMaker AI then manages the entire backend process, from scaling compute resources to managing training phases and hyperparameter tuning. This abstraction allows teams to concentrate on dataset quality and reward function design, which are the true drivers of model improvement. For enterprises, this serverless approach translates to faster iteration cycles, reduced costs, and a lower barrier to entry for advanced LLM customization. It’s a game-changer for those looking to scale AI for everyone by simplifying complex fine-tuning LLMs processes.

Why RLVR Excels for Agentic Tool Calling

When it comes to teaching an AI agent to reliably use tools, not all fine-tuning techniques are created equal. Supervised Fine-Tuning (SFT) requires meticulously labeled examples for every possible behavior a model should exhibit – calling a tool, asking for clarification, or refusing a request. The challenge with SFT is its struggle to generalize the decision-making process between these distinct behaviors, often performing well on patterns seen during training but faltering on novel scenarios.

Reinforcement Learning with Verifiable Rewards (RLVR) offers a more dynamic and effective solution. Unlike SFT, RLVR operates on a feedback loop:

1. Candidate Generation: For each prompt, the model generates multiple (e.g., eight) potential responses.
2. Reward Function Evaluation: A predefined reward function objectively scores each candidate, indicating its quality, correctness, and adherence to desired behavior (e.g., did it call the right tool with the correct parameters?).
3. Policy Update: Using Group Relative Policy Optimization (GRPO), the model's policy is updated to reinforce responses that scored above the average of the generated group. This process iteratively guides the model toward more optimal behavior.

This iterative learning enables the model to understand not just how to perform a specific action, but when to perform it. It learns the nuances of distinguishing between situations where a tool call is appropriate, clarification is needed, or refusal is the best course of action. Because tool calling has a naturally verifiable objective—whether the model called the right function with the right parameters—it maps exceptionally well to the RLVR paradigm, making it ideal for AI agents requiring high reliability. This method effectively addresses the challenge of designing agents to resist prompt injection by reinforcing precise action patterns.

Preparing High-Quality Training Data for RLVR

The success of any fine-tuning effort, especially with RLVR, hinges on the quality and comprehensiveness of the training data. For agentic tool calling, the dataset must teach the model more than just correct API invocations; it needs to encompass the full spectrum of required agent behaviors.

Our approach involved generating 1,500 synthetic training examples using Kiro, Amazon’s AI-powered IDE. These examples covered five distinct tool schemas: get_weather_forecast, search_flights, translate_text, currency_convert, and get_statistics. Crucially, the data was distributed across three primary agent behaviors to ensure balanced learning:

Each training example followed a JSONL format, including a prompt (system instruction and user request) and a ground_truth in the reward_model field that the reward function scores against. Varying phrasing between formal, casual, and terse further enhanced the dataset's robustness. While synthetic data provides a practical starting point, organizations with existing agentic workflows can leverage real user prompts and tool calls from production logs to achieve even higher quality training. This data preparation is a critical step in prompt engineering for complex agent behaviors.

{
  "prompt": [
    {"role": "system", "content": "You are a helpful assistant. When using tools, respond with: [...]"},
    {"role": "user", "content": "Get weather for San Francisco"}
  ],
  "reward_model": {
    "ground_truth": "[{"name": "get_weather_forecast", "arguments": {"city": "San Francisco"}}]"
  }
}

{
  "prompt": [
    {"role": "system", "content": "You are a helpful assistant. When using tools, respond with: [...]"},
    {"role": "user", "content": "Get the weather"}
  ],
  "reward_model": {
    "ground_truth": "To provide you with the weather information, could you please specify the location?"
  }
}

Fine-Tuning Qwen 2.5 7B Instruct with SageMaker AI

The process of fine-tuning a model like Qwen 2.5 7B Instruct within Amazon SageMaker AI Studio is streamlined and intuitive. After ensuring the necessary prerequisites (AWS account, IAM role, SageMaker AI domain, S3 bucket) are met, users can navigate to the Models section in the SageMaker AI Studio.

From there, selecting Qwen 2.5 7B Instruct and choosing Customize with UI opens a dedicated configuration page. This interface allows for:

• Technique Selection: Explicitly choosing Reinforcement Learning with Verifiable Rewards (RLVR) from the dropdown.
• Data Input: Pointing to the prepared training data stored in an Amazon S3 bucket.
• Reward Function: Configuring the tiered scoring mechanism that defines how candidate responses are evaluated against the ground_truth.
• Hyperparameter Configuration: Adjusting parameters like batch size, though SageMaker AI often handles optimal settings automatically.

SageMaker AI supports a diverse range of model families, including Amazon Nova, GPT-OSS, Llama, Qwen, and DeepSeek, alongside various techniques like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), RLVR, and Reinforcement Learning from AI Feedback (RLAIF). Integrated MLflow tracking provides visibility into training and validation metrics, simplifying performance monitoring and iteration. This ease of use dramatically accelerates the development lifecycle for developers building sophisticated github-agentic-workflows.

Evaluation and Deployment Success

The efficacy of our fine-tuned Qwen 2.5 7B Instruct model was rigorously evaluated on held-out data, including scenarios with entirely unseen tools—a crucial test for generalization. The results were compelling: the fine-tuned model achieved a remarkable 57% improvement in tool call reward compared to the base model. This significant leap in performance on scenarios it had not encountered during training underscores the power of RLVR in teaching models robust decision-making abilities for tool interaction.

This enhanced reliability directly translates into higher trust and confidence in deploying AI agents into production environments. By minimizing instances of tool hallucinations, incorrect parameters, and inappropriate actions, businesses can leverage AI agents for more critical and sensitive tasks. With SageMaker AI handling the complexities of model deployment and infrastructure management, developers can seamlessly move from fine-tuning to production, realizing the full potential of their agentic AI solutions. This capability aligns with the broader vision of operationalizing agentic AI for real-world impact.

In summary, the combination of Amazon SageMaker AI's serverless model customization and the robust learning capabilities of RLVR provides a powerful pathway to building highly reliable agentic tool calling systems. This innovative approach accelerates development, reduces operational burden, and ultimately delivers AI agents that perform with unprecedented accuracy and trustworthiness.