I trained a classifier on a Mac with MPS and let a gate decide if it ships

local-ai

My home voice runs on Home Assistant. When the rules do not recognize a phrase, I want a fallback that picks one of 77 categories (labels) and gives the probability for each choice. Without that probability, I cannot decide whether the system should act. The obvious option was a small model, trained for the home's intents and run locally.

I trained it on an M3 Max using PyTorch MPS, Apple's system for running AI calculations on the GPU. It worked, but the time and memory use did not match my estimate.

Where I started

The base is laya-multilingual, an open model from convaiinnovations. It uses a multilingual mmBERT-base encoder, a part that reads the options and helps pick one. The model has 322 million parameters, internal values adjusted during training.

With no training, on October 10, the model had 32% tool accuracy on the home's intents and a 54% false-positive rate. It was not useful. In a test on October 9, after the first fine-tuning, accuracy on the held-out set rose from 0.317 to 0.917. ECE was 0.08 and p50 was 23 ms. Fine-tuning is extra training for an existing model. Held-out is a separate set that is not used for training and helps evaluate the model. ECE measures calibration error: the gap between the confidence a model reports and its real rate of correct answers. P50 is the time by which half of the responses have arrived.

2.5 hours instead of 1

A full run uses about 9.1k items, 77 labels, and 4 epochs, each with about 4,593 micro-steps. At 133 micro-steps per minute, it takes around 2.5 hours. I had estimated one hour.

The setup uses soft cross-entropy as the loss function, shuffled options, device mps, a micro-batch of 2, and gradient accumulation of 32. The loss function measures training errors. A micro-batch is the number of examples processed at once. The small batch was necessary: MPS memory has no limit. At a micro-batch of 8, it reached 39 GB.

I tried to limit memory with the MPS watermark ratios and got it wrong twice before succeeding. Setting only the high ratio fails with invalid low watermark ratio 1.4, because the default low ratio was higher. With the high ratio at 0.5, training exceeded the limit: MPS allocated: 17.97 GiB, max allowed: 18.72 GiB. High at 0.8 and low at 0.65 worked on October 16.

The worst bug gave no warning. An earlier dataset version had 76 labels. The head budget (512/768), sized for fewer labels, silently skipped all 8,800 rows. There was no error or warning. I raised the values to 1024/1280.

The gate

A run only becomes the production model if it passes these automatic rules:

• held-out accuracy of at least 0.90;
• real-utterance accuracy of 1.0;
• ECE no higher than 0.10;
• false-positive action rate no higher than 0.01;
• wrong-action rate no higher than 0.02.

A passing run swaps the active checkpoint on its own. The checkpoint is the file with the model's trained state. That is why every test trains in a staging folder, a temporary area separate from production.

The run promoted on October 16 had held-out 0.987, real-utterance accuracy 1.0, ECE 0.013, false positives 0.0, and wrong actions 0.014. The gate rejected the October 21 run, even though it had the best held-out score: 0.991. It got only 0.889 on real utterances, with 3 errors out of 27, and its false-positive rate was 0.074.

The run promoted that same day had held-out 0.971, real-utterance accuracy 1.0, ECE 0.016, false positives 0.008, and wrong actions 0.0199. That last number was close to the 0.02 limit.

The held-out set comes from the same data distribution used in training. It measures how much the model learned from that set. The real utterances show whether it understands what I actually say. The run with the highest score was the one the gate rejected.

What still bites

On October 17, the first call got slow. It looked like the model was idle. The problem was memory pressure: the process showed RSS of 0.07 GB and physical memory use of 2.73 GB. Swap was at 19.0 of 20.5 GB and the load average was 13.9. RSS, which measures part of the memory used by a process, fooled me again.

A deploy, or release of a new service version, rebuilt the Python environment and killed a longer run after about 4 hours. With runs this long, I would protect the training environment so no deploy could change it.