The local AI stack

One rule decides everything on this machine: nothing leaves it, and nothing that belongs to my employer arrives on it. The first half is a technical constraint and the second is a policy I set for myself. Local processing does not make someone else’s data mine to carry home.

The hardware is a 2024 desktop with an Intel Core i9, an NVIDIA RTX 4090 with 24 GB of video memory, 64 GB of system memory in two sticks, and an MSI Z790 Edge board. The 4090 is the reason the rest exists. A 32 billion parameter model in 4-bit quantization fits in its memory with room for a usable context window, and that size is where open models stopped feeling like toys for engineering questions.

What runs today is LM Studio as the chat interface and model server, with AnythingLLM beside it for retrieval over my literature folder (the retrieval note covers that). The models are Qwen3 32B for reasoning, Qwen2.5-Coder 32B for code, and bge-m3 for embeddings. The embedding choice mattered more than I expected: retrieval quality moved more when I changed the embedding model than when I changed the chat model.

What is planned is the move from Windows-native tools to a Linux layer with Ollama serving the same models, Open WebUI in Docker as the interface, and Aider or Continue in VS Code for coding with a local model (the Linux note explains why it is a layer and not a replacement). Tailscale gives remote access from the laptop without opening a port.

What is excluded is any tool that needs a cloud API, which rules out the agentic coding assistant I use for this site. That is a real cost. The local coding models are a year or two behind, and I accept the gap on this machine because the whole point of it is the first rule.

What I have not done is fine-tune anything. The use cases so far are chat and writing over my own notes, code, and small machine learning jobs on tabular data, and none of them has needed a model trained on my material rather than a model reading it.