Running CPU-Optimized Large Language Model Inference in a Container

2026-11-17

Introduction

Running large language models locally has become increasingly practical with CPU-optimized inference engines.

The llama-modchip project enables running open-weight Mixture-of-Experts models like DeepSeek-V4-Flash (284B parameters) on systems with 64GB of RAM without requiring a discrete GPU.

Combined with OpenCode, an open-source AI coding agent, developers can create a fully local, privacy-preserving development environment.

Containerizing this setup with Podman provides isolation, reproducibility, and the ability to run without root privileges.

Container Setup

I wrote a simple Dockerfile that packages both llama-modchip and OpenCode together in a container.

The container runs as a non-root user, matching your host UID and GID to avoid permission issues with mounted volumes. Bind-mount your workspace directory to persist code and model weights outside the container.

Running the Stack

Inside the container, install a model using the interactive setup script. The script recommends suitable models based on available system RAM.

Access a chat UI in a browser on your Podman host. The llama-modchip gateway and OpenCode each offer a chat server.

Conclusion

This containerized approach provides a reproducible, rootless environment for local LLM inference and AI-assisted coding. The combination of llama-modchip's CPU optimizations and OpenCode's agent capabilities enables fully offline development workflows.

For more details on llama-modchip's quantization techniques and benchmarks, see the project repository and the Hacker News discussion.

Blog Posts

  1. Running Llama-Modchip and OpenCode in a Rootless Podman Container (2026-11-17)
  2. USB Gadget Mode on Linux: From Helpful Tool to Stealthy Attack Vector (2026-11-01)
  3. Finding Weak ECDSA Keys: A Lesson from the eMudhra Incident (2026-10-25)
  4. CVE-2026-113582: Apache Abdera XML External Entity (XXE) Vulnerabilities (2026-10-24)
  5. Bringing TEE-Mediated Private AI to Linux with TPM-Backed Passkeys (2026-02-21)
  6. Sandboxing Applications on Linux: Securing Password Managers with Firejail (2026-02-07)
  7. Running Tailscale in a Container (2026-01-25)
  8. Running dnscrypt-proxy in a Container (2026-01-17)
  9. Proxying Applications Without Native SOCKS Support (2026-01-11)
  10. Running a SOCKS Proxy for the Tor Onion Network in a Container (2026-01-05)
  11. Running OpenSSH Client as a SOCKS Proxy in a Container (2026-01-04)
  12. Adding Poe AI Support to Roo Code (2025-11-23)
  13. Introducing Edi: A CLI Tool for Interacting with the Poe API (2025-09-14)
  14. Deriving the Sum of Powers Formula (2025-08-03)
  15. Scheduling Color Temperature Changes for Your Monitors (2025-06-07)
  16. Stream and Play: Emulating Retro Games on Chromecast with Google TV (2025-05-10)
  17. Simulating the Monty Hall Problem (2025-05-04)
  18. Pi-hole on Raspberry Pi OS with dnscrypt-proxy (2024-01-28)
  19. Prime or Not Prime: That is the Question (2023-07-22)
  20. Switching Monitor Inputs on Computer Wakeup (2021-04-25)
  21. Changing the Playback Speed on Spotify's Web Player (2021-04-24)
  22. Solving Microcorruption Hollywood (2021-03-28)
  23. Font Rendering on macOS (2020-07-02)
  24. Video Disk Recorder (VDR) Tooling (2020-05-30)
  25. Spotify Web Player Failure due to Crash of Connected Device (2020-05-08)
  26. Malicious Forward Proxies (2020-04-27)
  27. Switching to a Dark Theme (2020-04-25)
  28. USB 3.0 to Gigabit Ethernet Adapter with Realtek 8153 Chipset on macOS Catalina (2020-04-23)
  29. Backing up Emails using OfflineIMAP (2020-04-19)
  30. Using a Raspberry Pi as a Wireless Access Point (2020-04-18)
  31. Accessing External Hard Drives in a Virtual Machine (2017-12-11)
  32. Managing Clipboards (2016-11-23)
  33. Multi-Screen Configuration Using Xrandr (2016-07-02)
  34. Email Backup and Migration using Thunderbird (2016-06-29)
  35. Automatic SSH Proxy Selection (2016-02-08)
  36. Set Color Temperature (2015-11-25)
  37. Completion Methods in Vim (2015-10-10)
  38. Fuzzing the mbed TLS Library (2015-09-26)
  39. SSH Weak Diffie-Hellman Group Identification Tool (2015-08-12)
  40. Building a Standing Desk for 50 Bucks (2015-01-25)
  41. High-DPI Displays under Linux (2014-12-23)
  42. Slow Wi-Fi on the Raspberry Pi (2014-06-15)
  43. XBMC on the Raspberry Pi (2014-06-14)
  44. Turning the Raspberry Pi into a Game Console (2014-06-12)
  45. ZFS Backups to External Hard Drives (2014-03-23)
  46. Display the I/O Usage of a Process (2014-02-21)
  47. Determining Wireless Network Channel Usage (2014-02-09)
  48. Managing Web Files through a Repository (2013-08-17)
  49. Simple HTTP File Servers (2013-08-16)
  50. Protect your SSH Private Keys (2013-06-16)
  51. Forward Secrecy for HTTPS and Ephemeral Diffie-Hellman (2013-06-14)
  52. Markdown Preview Chrome Extension (2013-06-13)