Why Run Local LLMs with Claude Code?
Imagine having a super-smart coding assistant that can not only understand your code but also execute tasks on your computer. Now, imagine you can swap out its "brain" for one you’ve trained yourself or picked from a community of open-source innovators. That's the core idea behind running local LLMs with Claude Code. This integration bypasses cloud-based APIs, delivering a more private, customizable, and often faster development experience right on your local machine.
This setup becomes particularly powerful given Claude Code's recent updates, which enable it to take direct control of your desktop, opening files, using browsers, and running developer tools autonomously, per Ars Technica. While Claude Code prioritizes using direct connectors for services like Slack, it can fall back to directly controlling your mouse, keyboard, and screen when needed. Running local LLMs means the intelligence driving these agentic actions can be entirely tailored to specific needs or constraints, giving developers an unprecedented level of control over their AI co-pilot.
Setting Up Your Local AI Agent
The process hinges on llama.cpp, an open-source framework designed for efficient LLM inference on various devices. Developers first compile `llama.cpp` according to their system (Linux, Mac, Windows, with or without GPU acceleration), then download their preferred open-source model. Unsloth provides dynamically quantized GGUF models, such as Qwen3.5-35B-A3B or GLM-4.7-Flash, optimized for performance and accuracy, even on consumer-grade GPUs with 24GB VRAM.
Once downloaded, the `llama-server` component of `llama.cpp` deploys the model locally, typically on port 8001, providing an OpenAI-compatible endpoint for Claude Code. Specific sampling parameters, like a temperature of 0.6 and top-p of 0.95 for Qwen3.5, are crucial for optimal agentic coding performance. A key optimization involves using `--cache-type-k q8_0 --cache-type-v q8_0` for KV cache quantization, reducing VRAM usage without significant accuracy degradation.
However, a critical challenge arises when integrating with Claude Code: the platform’s attribution header can invalidate the KV cache, leading to 90% slower inference with local models. The fix, known as "The Claude Code Loophole," involves setting `CLAUDE_CODE_ATTRIBUTION_HEADER` to `0` within the `env` section of the `~/.claude/settings.json` file. This prevents Claude Code from prepending the problematic header, restoring full performance to the local LLM.
After configuring `llama-server` and addressing the KV cache issue, developers set the `ANTHROPIC_BASE_URL` environment variable to ` and `ANTHROPIC_API_KEY` to a dummy key like `sk-no-key-required`. This redirects Claude Code's requests to the locally hosted model. With these configurations, Claude Code can then be launched within a project directory, utilizing the specified local LLM to execute complex coding tasks, including autonomous fine-tuning runs with Unsloth.
This localized approach to AI-powered development, particularly when paired with Claude Code's expanding autonomous capabilities on macOS, transforms the developer workflow. It opens the door for highly specialized AI agents that operate with greater efficiency, data privacy, and direct control over the computing environment, heralding a new era of personalized AI assistance in coding.
