On-Device AI Agents with LLM Tool-Calling

An agent that can actually operate a device — schedule, message, file, automate — is dramatically more useful than a chatbot. It's also dramatically easier to get wrong. I've built an on-device agent that drives phone and desktop workflows through LLM tool-calling, and the hard part was never getting it to do things. It was getting it to do the right things, reliably, without surprising anyone.

Here's how I think about building these systems.

Tools are the API — design them like one

The model can only act through the tools you expose. Every tool is a contract: a name, a description, a strict schema, and a clear failure mode. Sloppy tool design is the number-one cause of flaky agents.

• One tool, one job. send_message, not do_comms.
• Validate every argument before acting — reject malformed input instead of guessing.
• Return structured results the model can reason about, including errors.
• Keep the set small. Too many tools and the model picks wrong; five to fifteen is a healthy range.

{
  "name": "create_calendar_event",
  "description": "Create an event. Confirm with the user first.",
  "parameters": {
    "title": "string",
    "start": "ISO-8601 datetime",
    "duration_minutes": "integer"
  }
}

Loop with a budget, not forever

Agents run in a loop: think → call tool → observe → repeat. Ungoverned, that loop burns tokens and can thrash. I always cap it: a maximum number of steps, a wall-clock timeout, and a maximum cost per task. When the budget runs out, the agent stops and reports — it does not keep improvising.

Human-in-the-loop for anything consequential

Autonomy should be proportional to reversibility. Reading a calendar or summarising an inbox can run without asking. Sending a message, making a payment, or deleting files must be confirmed. I classify every tool by risk level and gate the high-risk ones behind an explicit confirmation step.

• Read-only tools — run freely.
• Write tools — show the proposed action, require confirmation.
• Destructive / external tools — confirmation plus an undo path where possible.

Constrained outputs beat clever prompts

Asking a model to "call the right function" in prose is unreliable. Using strict tool schemas — or structured JSON output with validation — turns action selection into a parsing problem you can verify. I validate the model's tool call against its schema and retry once on a formatting error, then fail cleanly.

On-device, I keep a small local model for intent classification and routing, and fall back to a larger cloud model only for genuinely complex steps. That keeps most runs fast, private, and cheap.

State and memory need a plan

Agents that forget context repeat themselves; agents that remember everything become slow and expensive. I keep a short working memory of the current task and a bounded summary of prior steps, and I never let raw tool output flood the context. Summarise and compress as you go.

Determinism where it counts

Local automation — via ADB, accessibility services, or shell — is inherently stateful and can drift. So where I can, I replace "the model figures it out" with deterministic glue: for a known task, the agent selects a workflow and the workflow executes predictable steps. The LLM decides what; tested code decides how.

Observability is mandatory

Log every step: the model's reasoning summary, the tool called, the arguments, the result, and the timing. Without this you cannot debug a non-deterministic system. With it, failures become reproducible and fixable. I also track success rate per tool and per task type — that's where tuning effort actually pays off.

Security, always

An agent with device access is a powerful capability and a serious attack surface. Prompt-injection via content the agent reads is a real threat. My mitigations: never let untrusted content trigger tool calls directly, validate all arguments against schemas, run tools with least privilege, and require confirmation on anything that leaves the device.

What makes an agent good

A good on-device agent is boring in the best way: small, well-described tools; a bounded loop; confirmation for anything irreversible; strict output validation; summarised memory; and full logging. The intelligence is necessary but not sufficient — the engineering discipline around it is what turns a neat demo into something people trust with their actual workflows.