1 Introduction↩︎

Artificial intelligence (AI) agents [1], [2] have significantly transformed various industries by enabling autonomous decision-making and improving operational efficiencies [3]–[7]. These agents rely on a critical planning process that involves determining the optimal course of action, executing the planned actions, and summarizing the outcomes. Large Language Models (LLMs) such as Gemini-Pro [8] and GPT-4 [9] have shown potential in this domain. While these models face challenges in executing complex planning tasks at a level comparable to human performance [10], [11], they remain effective in addressing simpler tasks, thereby facilitating practical applications. One such application is the emergence of AI assistant tools from companies like MultiOn [12], Simular AI [13], and Adept AI [14], which leverage the capabilities of LLMs to provide intelligent assistance across various domains. Additionally, consumer-oriented AI hardware products, such as Rabbit R1 [15], Humane AI Pin [16], and Limitless Pendant [17], integrate LLMs into user-friendly devices, making intelligent assistance more accessible and gaining significant traction.

The success of AI agents depends on the performance of the underlying LLMs. Agents using pre-trained models without fine-tuning on task demonstrations have relatively low success rates, ranging from 12% on desktop applications [18] to 46% on mobile applications [19], while those leveraging fine-tuned models can achieve success rates of up to 80% on tasks similar to their training data [20], [21]. However, using LLMs for AI agents is costly due to high computational demands and infrastructure expenses, which limits widespread adoption. The lack of on-device AI agents restricts applications requiring real-time processing, offline functionality, or enhanced privacy.

On-device AI agents offer advantages including reduced latency, offline operation, lower costs, and improved data security [22]–[25]. While action models like Octopus V2 achieve over 95% accuracy in function calling [26], an on-device planning model is still missing. General agent frameworks rely on single-model in-context learning, which requires lengthy function descriptions and planning instructions in each prompt. This approach is impractical for on-device models with limited context lengths, as it causes high latency and battery consumption on edge devices.

In this paper, we introduce Octo-planner, an on-device planning agent that addresses the key challenges of efficiency, adaptability, and resource constraints. Our Planner-Action framework separates planning and action execution into two components: a planner agent, or Octo-planner, optimized for edge devices, and an action agent using the Octopus model for function execution. By prioritizing fine-tuning over few-shot prompting, we reduce computational costs and minimize key-value (KV) cache requirements. Our approach uses GPT-4 to generate and validate planning data, which is then used to fine-tune Phi-3 Mini for on-device deployment. In-domain tests demonstrate that this fine-tuning improves planning success rates to 97%. To address multi-domain planning challenges, we develop a multi-LoRA training method that merges weights from LoRAs trained on distinct function subsets. This method enables flexible handling of complex, multi-domain queries while maintaining computational efficiency on resource-constrained devices. By focusing on pre-defined functions for simpler tasks and leveraging fine-tuning, we aim to make AI agents more practical, accessible, and cost-effective for real-world applications.

This work aims to contribute to the ongoing efforts to make AI more accessible and practical for everyday use. By bridging the gap between AI agent potential and edge computing constraints, we seek to facilitate the adoption of intelligent, on-device assistants across various domains. Through open-sourcing our approach, we hope to inspire further innovations in on-device AI, expanding the reach of advanced planning capabilities to a broader range of applications.

2 Related Work↩︎

Planner agentLanguage models have become essential in planning agent systems. Proprietary models like OpenAI’s assistant API [27] excel in generating strategies based on user queries and available functions. Recent advancements have further expanded the capabilities of language models in planning. The ReAct framework [28] integrates planning and acting for limited action spaces, while research from Alibaba Group [29] highlights the effectiveness of separate planning and action models for complex tasks. In robotics, language models are also increasingly applied to task-level planning [30], [31]. Notable examples include SayCan [32], which uses LLMs to break high-level tasks into concrete sub-tasks, and Video Language Planning (VLP) [33], which enhances long-horizon planning through a text-to-video dynamics model. The broad application of language models in planning systems, from general strategies to specific robotics tasks, underscores their growing importance and adaptability in decision-making processes across diverse domains.

Fine-tuning to replace long contextFine-tuning language models to internalize specific prompts or context information reduces input length and improves efficiency [34], [35]. This approach involves training models on carefully curated, task-specific datasets. For models with limited context windows, this technique is particularly valuable as it enables more efficient query processing without sacrificing response quality. The success of fine-tuning largely depends on the use of diverse, high-quality datasets, which ensure the model can generalize across various prompt phrasings [36]–[39]. When implemented effectively, fine-tuning streamlines application-specific interactions, addressing both context length limitations and computational challenges in practical deployments.

LoRA and Multi-LoRALow-Rank Adaptation (LoRA) efficiently adapts pre-trained language models to specific tasks [40]. Unlike full fine-tuning, which updates all parameters, LoRA freezes the pre-trained weights and adds trainable low-rank matrices to each layer, significantly reducing the number of trainable parameters and the computational demands. Multi-LoRA extends this concept by enabling multiple task-specific adapters to be trained, combined, or switched during inference, allowing a single base model to handle various tasks efficiently [41]. Building on these approaches, researchers have developed several related variants to address different aspects of model adaptation: LoRA+ optimizes learning rates [42], VeRA uses random projections [43], AdaLoRA implements adaptive rank [44], DoRA decomposes weights [45], and Delta-LoRA updates the pre-trained weights [46]. These variants aim to further refine efficiency or performance in specific scenarios.

3 Method↩︎

This section presents our framework for on-device Planner-Action agents. We first describe the integration of planning and action agents for efficient problem-solving. We then detail our approach to dataset design and the training process for the planning agent, including support for extensive function sets and a plug-and-play capability for additional ones. Finally, we outline the benchmark used to evaluate agent performance.

3.1 Planner and action agents framework↩︎

Our Planner-Action approach distinguishes itself from general agent frameworks by separating the planning and action execution processes into two components. This separation improves modularity and allows for specialized optimization of each component. The framework operates as follows:

Planner Phase: Given a user query \(q\), our planning model \(\pi_{plan}\) decomposes the task into a sequence of sub-steps. Formally: \[\{\tau_1, \tau_2, ..., \tau_n\} = \pi_{plan}(q; F),\] where \(F\) is the set of available function descriptions, and \(\tau_i\) is the \(i^{th}\) execution step. \(\pi_{plan}\) internalizes \(F\) during instruction fine-tuning.

Action Phase: For each step in the execution sequence, we employ an action model \(\pi_{action}\). At step \(i\), given the observation of the current state \(O_i\), the action model performs: \[O_{i+1} = \pi_{action}(\tau_i, O_i),\] where \(O_{i+1}\) and \(\tau_{i+1}\) are passed to the next step for continued execution. This iterative process ensures coherent progression through the task’s sub-steps.

For the action model, we utilize the Octopus model, which is specifically designed for on-device function calling. Figure 2 illustrates the difference between our Planner-Action framework and the single-model approach for LLM agents.

The modular design of our framework offers several advantages:

• Specialization: Separating planning and action execution allows optimization of each model for its specific role, enhancing performance in complex tasks.

• Scalability: Independent scaling of planning and action capabilities efficiently accommodates varying task complexities.

• Interpretability: Explicit separation of phases improves transparency in the decision-making process.

• Adaptability: Domain-specific knowledge or constraints can be integrated into either phase without system-wide changes.

3.2 Planning dataset↩︎

Our framework uses the Octopus model as the action model, requiring training only for the planner agent. We fine-tune the planner agent with the following dataset format:

Special tokens such as <|user|> and <|assistant|> are used for chat model pre-training but are optional otherwise. We set \(n\) to range from 1 to 5, based on our finding that most tasks on mobile apps consist of fewer than 5 steps. The dataset generation and curation process includes:

1. Dataset collection: Given the available functions \(F\), we use a large language model (GPT-4) to generate diverse queries answerable by these functions. We increase the model’s temperature setting to ensure query variety. Responses are then generated in the specified dataset format. Importantly, while function descriptions are used during generation, they are not included in the final dataset. Instead, the planner model internalizes this function information during training.

2. Data validation: We employ the same language model as a validation tool to assess the correctness of query-response pairs. Despite some errors in the initial generation process, we found that the model effectively classifies generated content as valid or invalid, allowing us to filter out incorrect outputs and maintain dataset quality.

Example data points with different numbers of sub-steps are shown below:

A visualization of the dataset collection process is shown in Figure 3. Example function descriptions are provided in Appendix 7.1.

3.3 Benchmark design↩︎

Our evaluation relies on a carefully constructed test dataset. This dataset is designed to represent the complexities of real-world planning, employing a multi-stage approach that integrates automated generation, expert validation, and empirical testing.

The process begins with the automated generation of an initial dataset comprising 1,000 data points using GPT-4. These data points then undergo a rigorous quality assurance process to ensure their integrity and relevance. The quality assessment criteria are as follows:

• Each step must correspond to an existing function;

• The sequential order of steps must be correct.

To ensure the reliability of our evaluation, we incorporate an additional phase of manual verification. This phase involves selecting a subset of examples for end-to-end model execution, thereby validating the accuracy of our results and providing a comprehensive assessment of our model’s performance.

For the evaluation of our proposed planning model, we employ GPT-4 as an oracle to determine the correctness of the generated plans. This choice is based on empirical observations indicating GPT-4’s high proficiency in our specific use case.

4 Experimental Design↩︎

Our experimental design assesses Octo-planner’s performance in on-device AI agent planning. We aim to determine the optimal configuration for deploying efficient, accurate planning models on resource-constrained devices while maintaining adaptability to new domains and functions. Our experiments focus on four key areas:

1. Performance and efficiency trade-offs between full fine-tuning and LoRA.

2. Multi-LoRA accuracy in handling different function sets simultaneously.

3. Performance comparison across various base models and sizes.

4. The impact of dataset size on accuracy, ranging from 100 to 1000 training examples.

We conduct supervised fine-tuning on our curated dataset, using Phi-3 Mini and a few other alternatives as base models. Training includes both full fine-tuning and LoRA techniques. For all experiments, we set the dataset size to be 800 times the number of available functions and perform fine-tuning on an NVIDIA A100 GPU. We use optimized hyperparameters across both techniques: a learning rate of \(5\times 10^{-6}\), a batch size of 4, and a warm-up ratio of 0.2, training for 2 epochs. For LoRA, we set target_modules to all-linear.

5 Results↩︎

5.1 Full fine-tuning vs LoRA↩︎

Table 1 presents a detailed comparison of the full fine-tuning and LoRA approaches for our planning model. Our experiments reveal significant differences in performance across these methods. Full fine-tuning achieves the highest accuracy at 98.1%, demonstrating superior performance. In contrast, LoRA’s performance depends on the rank size. With rank 64 and alpha 256, LoRA achieves 85.1% accuracy, while reducing the rank to 16 and alpha to 32 drops accuracy to 72.9%. These results highlight the trade-off between model performance and computational efficiency when using LoRA. While full fine-tuning provides better accuracy, LoRA offers a more resource-efficient alternative, with performance varying according to the rank configuration.

5.2 Multi-LoRA training and merging↩︎

While LoRA-based training is effective for specific function sets, real-world applications often require handling new or expanded sets of functions. To address this challenge, we propose merging multiple LoRA weights, each trained on a distinct function subset, into the same base model. This approach creates a composite model that combines knowledge from various function sets, offering a scalable solution for complex, multi-domain queries in resource-constrained environments.

To evaluate this method, we constructed a benchmark dataset by randomly selecting functions from each LoRA’s domain with equal probability, combining them into workflows. Queries and plans were generated using GPT-4. For example, when testing two merged LoRAs, queries may involve Android functions, E-commerce functions, or both, with equal likelihood.

The code block below shows sample queries from our benchmark dataset and the corresponding inference results from the multi-LoRA merged model:

Table 2 presents the performance results of our multi-LoRA merging technique. Each individual LoRA was trained with consistent hyperparameters: rank 64, lora_alpha 256, and target_modules set to "all-linear". The single-domain Android function set LoRA achieves 85.1% accuracy. When merging LoRAs from two domains (Android and E-commerce), accuracy slightly decreases to 82.2%. Further merging yields lower accuracies: 78.9% for three domains (adding Video Streaming) and 69.7% for four domains (adding Travel). These results reveal a pattern of gradual accuracy decline as more function sets are integrated, with a steeper drop occurring after the third domain is added.

5.3 Full fine-tuning with different base models↩︎

Table 3 presents the benchmark accuracy of different base models after full fine-tuning. Google Gemma 2B achieves 85.6% accuracy, while the larger Gemma 7B excels with 99.7%. Microsoft Phi-3 Mini also performs strongly at 98.1%. These results indicate that our framework adapts well to various on-device LLMs, with larger models generally achieving higher accuracy.

5.4 Full fine-tuning with different dataset sizes↩︎

Our default training dataset contains 1000 data points, evenly distributed across 1–5 step sequences (200 each) to represent varying task complexities. We investigate the impact of dataset size on model performance to optimize the efficiency of function set integration and address the cost of synthetic data generation. Table 4 shows the benchmark accuracy for various training dataset sizes:

The results show a clear correlation between dataset size and accuracy. The full 1000-point dataset achieves 98.1% accuracy, while reducing the dataset to 500 data points drops accuracy to 92.5%. Further reductions to 250 and 100 data points result in accuracies of 85.3% and 78.1%, respectively. These findings suggest that, for optimal performance, a training dataset of more than 1000 data points is recommended.

6 Conclusion↩︎

This paper introduces Octo-planner, an on-device planning agent designed to work alongside action agents such as Octopus V2. By separating planning and action execution, we improve specialization and adaptability. Our approach fine-tunes Phi-3 Mini (a 3.8-billion-parameter LLM) to serve as a planning agent capable of running locally on edge devices, achieving 97% success in in-domain tests. We reduce computational demands, improving latency and battery life, and implement a multi-LoRA technique for expanding model capabilities without full retraining.

Octo-planner contributes to addressing AI deployment concerns, including data privacy, latency, and offline functionality. It represents an advancement towards practical, sophisticated AI agents for personal devices. By open-sourcing our model weights, we aim to drive innovation in on-device AI, promoting the development of efficient, privacy-respecting applications that enhance daily life without compromising performance or security.

Limitations and future work↩︎

Our current model, while effective for specific mobile phone use cases, has limitations in its broader applicability. Unlike frameworks such as ReAct, which alternates between planning steps and executing actions based on real-time feedback, our model conducts all of its planning in advance. This upfront planning approach, while efficient for straightforward tasks, may be less adaptable to complex or unpredictable scenarios in which conditions change during execution.

Future work will focus on exploring an iterative planning methodology that refines plans based on real-time observations, improving adaptability to dynamic environments. We also plan to investigate the integration of our planning model with diverse action models, extending its capabilities beyond mobile applications to areas such as IoT, robotics, and smart home systems. These advancements will address current limitations and expand the versatility of our on-device planning model, bridging the gap between efficient, localized AI processing and the complex demands of real-world applications.

7 Appendix↩︎

7.1 Function/API description examples↩︎

def get_trending_news(query, language):
    """
    Retrieves a collection of trending news articles relevant to a specified query and language.

    Parameters:
    - query (str): Topic for news articles.
    - language (str): ISO 639-1 language code. The default language is English ('en'), but it can be set to any valid ISO 639-1 code to accommodate different language preferences (e.g., 'es' for Spanish, 'fr' for French).

    Returns:
    - list[str]: A list of strings, where each string represents a single news article. Each article representation includes the article's title and its URL, allowing users to easily access the full article for detailed information.
    """


def get_weather_forecast(location):
    """
    Provides a weather forecast for a specified location over a given number of days. Each day's forecast includes a brief description of the expected weather conditions.

    Parameters:
    - location (str): The location for which the weather forecast is desired. Can be a city name, ZIP code, or other location identifiers.

    Returns:
    - list[str]: A list of strings, each representing the weather forecast for one day. Each string includes the date and a brief description of the weather conditions. Formatted in 'YYYY-MM-DD: Description' format.
    """


def send_email(recipient, title, content):
    """
    Sends an email to a specified recipient with a given title and content.

    Parameters:
    - recipient (str): The email address of the recipient.
    - title (str): The subject line of the email. This is a brief summary or title of the email's purpose or content.
    - content (str): The main body text of the email. It contains the primary message, information, or content that is intended to be communicated to the recipient.

    Returns:
    """


def search_youtube_videos(query):
    """
    Searches YouTube for videos matching a query.

    Parameters:
    - query (str): Search query.

    Returns:
    - list[str]: A list of strings, each string includes video names and URLs.
    """


def find_route_google_maps(origin, destination, mode):
    """
    Computes a route using Google Maps from an origin to a destination.

    Parameters:
    - origin (str): Starting location.
    - destination (str): Target location.
    - mode (enum): Mode of transportation, options include 'driving', 'walking', 'bicycling', and 'transit'. The default mode is 'driving'.

    Returns:
    - List[str]:  The string provides the route details.
    """


def send_text_message(contact_name, message):
    """
    Sends a text message to the specified contact.

    Parameters:
    - contact_name (str): The name of the recipient contact.
    - message (str): The content of the message to be sent. This is what the recipient will receive.

    Returns:
    """


def create_contact(name, phone_number):
    """
    Creates a new contact entry in the device's address book.

    Parameters:
    - name (str): Full name of the contact. This should include first and last name.
    - phone_number (str): phone number of the contact. The phone number should be provided in a standard format, preferably in E.164 format (e.g., +12345678900 for an international format).

    Returns:
    """


def set_timer_alarm(time, label):
    """
    Sets a timer or alarm for a specified time.

    Parameters:
    - time (str): Alarm time in "HH:MM" 24-hour format. For example, "07:12" for 7:12 AM.
    - label (str): Custom label for the alarm, default is "alarm".

    Returns:
    """


def create_calendar_event(title, start_time, end_time):
    """
    Schedules a new event in the calendar.

    Parameters:
    - title (str): Event title.
    - start_time (str): Event start time as a string in ISO 8601 format "YYYY-MM-DD-HH-MM". For example, "2022-12-31-23-59" for 11:59 PM on December 31, 2022.
    - end_time (str): Event end time as a string in ISO 8601 format "YYYY-MM-DD-HH-MM". Must be after start_time. For example, "2023-01-01-00-00" for 12:00 AM on January 1, 2023.

    Returns:
    """


def set_volume(level, volume_type):
    """
    Sets the volume level for a specified type : "ring" , "media" , "alarm".

    Parameters:
    - level (int): Target volume level, from 0 (mute) to 10 (maximum).
    - volume_type (enum): The category of volume to adjust, select from "ring" , "media" , "alarm".

    Returns:
    """

References↩︎