Running a 397B parameter model on my M5 MacBook Pro

A 397 billion parameter mixture-of-experts model does not fit in RAM. It turns out it does not need to. Flash-MoE streams experts off the SSD four at a time, and on an M5 Max it runs at twice the speed the authors reported.

The machine

MacBook Pro, Apple M5 Max, 128GB of unified memory, 2TB internal SSD. The SSD is the interesting part. This whole experiment lives or dies on sequential read speed.

For reference, flash-moe was built and benchmarked on an M3 Max with 48GB. My machine is comfortably over spec, which made it a good candidate for a different question: how much of the reported 4.4 tokens per second was the engine, and how much was the hardware ceiling?

The goal

Flash-MoE is a pure C and Metal inference engine for Qwen3.5-397B-A17B. No Python at inference time, no frameworks. The 209GB of expert weights stay on disk and get read on demand with parallel pread, four experts per layer per token. The OS page cache is the only cache. I wanted it running locally, end to end.

Checking the prereqs

Before touching anything: is this machine actually able to run it? Four checks, thirty seconds.

sysctl -n machdep.cpu.brand_string   # Apple M5 Max
xcode-select -p                      # Xcode installed
python3 --version                    # 3.13
df -h /                              # 1.5TB free

Three out of four passed. The Python packages were the only gap, so they went into a project venv instead of the system Python:

python3 -m venv .venv
.venv/bin/pip install safetensors numpy huggingface_hub

One small thing worth knowing: the hf CLI ships inside huggingface_hub now. The [cli] extra you see in older instructions is obsolete in 1.x.

The 800GB that wasn't

The setup notes I started from claimed you need the original BF16 safetensors, an 800GB download, and a repack from scratch. That would have been the real cost of the whole exercise.

It is not true. Before starting any download I read the repo's own expert_index.json, and its model_path points at mlx-community/Qwen3.5-397B-A17B-4bit, which is already quantized. 46 shards, 209GB. The component table in repack_experts.py confirms it: the expected tensors are packed 4-bit weights with scales and biases, not BF16. The repack is not a quantization pass, it is a re-layout: it shuffles already-4-bit experts into one contiguous binary file per layer so a single pread can grab an expert in one shot.

Reading the code before starting the download saved 600GB of traffic and a day of waiting.

Download in the background, prep in the foreground

The download is the long pole, so it went first, in the background:

git clone https://github.com/danveloper/flash-moe
.venv/bin/hf download mlx-community/Qwen3.5-397B-A17B-4bit

It pulled at roughly 50MB per second, so about an hour and a half for 209GB. Everything else happened while it ran.

First, the build. make in metal_infer/ produced the infer binary clean on the first try, make chat the terminal client. Metal shaders compile at startup, so there is nothing else to prepare on the GPU side.

Second, the paths. The repo assumes the author's machine: expert_index.json, extract_weights.py and export_tokenizer.py all hardcode paths under /Users/danielwoods. The index got patched to point at my HuggingFace cache, the scripts took paths as flags. Conveniently, the snapshot hash in the fresh download matched the hardcoded one exactly, so everything lined up.

Third, the missing tool. The engine needs two generated files for token handling: tokenizer.bin for encoding, which has an exporter in the repo, and vocab.bin for decoding, which has no generator anywhere. I reverse engineered the format from load_vocab() in infer.m: u32 entry count, u32 max id, then a length-prefixed UTF-8 string per token id. A 60-line Python script maps the GPT-2 byte-level BPE strings in tokenizer.json back to raw bytes and writes the file. 248,077 tokens, 2.3MB.

202GB repacked in 45 seconds

The download finished, verification of all 46 shards included. Then the repack:

.venv/bin/python repack_experts.py --index expert_index.json
DONE: 217,432,719,360 bytes (202.5 GB) written
Time: 45.5s
Throughput: 4.5 GB/s

This was the number that surprised me most. 4.5GB per second, sustained, on the internal SSD, and it verified four sample experts per layer against the source shards as it went. All 60 layers passed.

The non-expert weights (attention, norms, routing, the shared expert) extract into a single 5.5GB file with extract_weights.py. That file gets mmap'd at startup and is all the engine ever holds in memory. Total footprint is about 6GB. The other 202GB stay on disk.

First tokens

The 209GB model is never loaded. It is read, four experts at a time, token by token.

The binary defaults to the author's model path, so -m is mandatory:

./infer -m ~/.cache/huggingface/hub/models--mlx-community--Qwen3.5-397B-A17B-4bit/snapshots/<hash> \
    --prompt "Say hello in Swedish, then briefly introduce yourself." --tokens 80
Total time:     8.8 s
TTFT:           2875 ms
Tokens:         53 generated
Generation:     6.0 s (8.73 tok/s)

8.73 tokens per second average, peaking above 10, time to first token under three seconds, cold. The authors reported 4.36 on the M3 Max, so the M5 Max roughly doubles it. The bottleneck moved with the SSD, exactly as their paper predicts.

Quality is real too. It answered in fluent Swedish and closed with a polite question, then emitted a clean end-of-text token on its own. This is a frontier-class open model, not a demo toy.

Serving it

The engine has an OpenAI-compatible server built in:

./infer -m <model-path> --serve 8080

Startup takes about 260ms: mmap the weights, wrap them as a Metal buffer, open the 60 packed layer files, pre-fill the system prompt. Then it listens. Testing it is two curls:

curl http://localhost:8080/health
# {"status":"ok","model":"qwen3.5-397b-a17b"}

curl -N -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Say hi in five words."}]}'

Two quirks to know. The server always streams SSE chunks, even if you pass stream: false, and the model emits <think> blocks in its output, so a client needs to handle both. The bundled terminal client does, but it defaults to port 8000, so it needs ./chat --port 8080. Beyond that, any OpenAI-compatible UI pointed at localhost works.

Total disk cost: 411GB. Total RAM cost: about 6GB. The model never got loaded, and that is the whole point.