Smaug-Decisions: Calibrated On-Device Decisions over Audio, Images and Text

1Introduction

Many uses of a language model are decisions rather than conversations: has the speaker finished their turn, which instrument is playing, should this request be escalated, what does this warning sign say. Each is a question with a fixed set of options, and what an application needs back is not a sentence but a probability per option that it can compare against a threshold. We call these typed decisions.

Two properties matter for such a model. It should be calibrated, so that a probability of 0.8 is right about 80% of the time and a threshold means what it says. And it should be cheap enough to run on every event, ideally on the device where the audio or image is captured, without a network round trip.

This report covers three released models and the apps that run them:

• Smaug-Decisions, a full fine-tune of Gemma 4 E4B-it that answers decisions over audio, images and text (Section 2).
• Smaug-Audio-Decisions-0.7B, a 744M audio model that answers in a single pass and knows when it is unsure (Section 2).
• A confidence-gated cascade of the two that matches Smaug-Decisions' audio accuracy at 62% of its latency (Sections 3 and 4).
• Smaug-Decisions-LiteRT, a 3.45 GB build of Smaug-Decisions for the phone GPU, and iOS and Android apps that run the full cascade offline (Section 5).

All models are released under Apache 2.0, the license of their base models.

2Models

2.1Reading a decision instead of generating it

A decision is rendered as a prompt containing the input, the question and the options labelled A, B, C and so on (up to 20). Smaug-Decisions reads the language-model logits of the option letters at the answer position, applies a tanh soft cap of 30, and takes a softmax for one-answer questions or a per-option sigmoid with one learned threshold for select-all questions. Smaug-Audio-Decisions-0.7B places a marker token before each option and scores the hidden state at each marker with a linear head. Either way a decision costs one prefill and no decoding.

2.2The three models

Smaug-Decisions is a full fine-tune of Gemma 4 E4B-it on a broad mix of typed decision tasks over audio, images and text, with calibration as an explicit objective and options shuffled so that the model learns the options, not their positions. It accepts up to 30 s of 16 kHz audio, one image at 560 image tokens, or text. Smaug-Audio-Decisions-0.7B is fine-tuned from EmbeddingGemma 2 with a linear marker head for the same audio inputs, and ships as int8 Core ML for iPhone. Smaug-Decisions-LiteRT is Smaug-Decisions for Google's on-device runtime LiteRT-LM: an int4 decoder, int2 embedding tables, an int8 audio encoder and an output head that holds only the 20 option letters, with a 2,048-token context. Table 1 summarises them.

3The cascade

Smaug-Audio-Decisions-0.7B is fast but less accurate than Smaug-Decisions; Smaug-Decisions is accurate but about twenty times slower per decision on a server GPU. Because both are calibrated, the audio model's own probability tells us when its answer can be trusted. Every audio question goes to the audio model first. Let c be its confidence: the top probability for a one-answer question, and the probability of the least certain option for a select-all question. If c ≥ τ, the audio model's answer stands; otherwise the question is escalated to Smaug-Decisions. Text and image questions go straight to Smaug-Decisions (Figure 1). The default threshold is τ = 0.9.

Routing text past the audio model costs nothing in practice: at τ = 0.9 the audio model escalates 98% of text questions on its own, so a cascade answers text at Smaug-Decisions' level either way.

4Results

4.1Audio decisions

Earshot has 6,671 items and 17.2 hours of audio in four tracks: conversation dynamics (turn completion, next speaker, speaker count and speech activity), external turn completion, sound and music (genre, instrument, tempo, scene and sound events), and a small pilot track. Every model was scored on the same items with the same harness; the APIs received the audio and returned probabilities as JSON. Smaug-Decisions has the highest audio skill and the lowest calibration error of the models we tested (Figure 2, Table 2). Smaug-Audio-Decisions-0.7B, at about a fifth of the size, is ahead of Gemini 3.8 Flash on skill and much better calibrated.

4.2Cascade operating points

Figure 3 sweeps the threshold. At τ = 0.9 the cascade escalates 60% of audio questions, matches Smaug-Decisions alone on skill (60.1) and calibration (0.071 against 0.069), and needs 130 ms per decision on average instead of 210 ms, 38% less. Raising the threshold to 0.95 buys 0.1 points for another 23 ms; lowering it to 0.8 saves 29 ms and costs 0.9 points.

4.3Text and images

On the public Jev Decision Index, Smaug-Decisions scores 56.6 on text and 57.9 on images, the best results for a Gemma 4 E4B model on both. The phone build, at about a fifth of the size, loses 0.7 points of Earshot skill, 0.005 of calibration error, 2.7 points of Decision Index text and 2.1 points on images (Table 1).

5On-device

5.1Runtime

The iOS app runs Smaug-Audio-Decisions-0.7B on Core ML and Smaug-Decisions-LiteRT on LiteRT-LM with the GPU backend (Metal). The Android app is one Kotlin app with the audio model on ONNX Runtime and the same Smaug-Decisions-LiteRT file on LiteRT-LM, sharing the core logic of the iOS app. Smaug-Decisions is downloaded inside the app; its first load writes a cache of about 2.4 GB that makes later loads fast, and the vision encoder is attached only when an image arrives.

5.2Measurements

Table 3 lists measurements on an iPhone 14 Pro Max with 6 GB of memory, offline, with both models resident. Figure 4 shows the app.

(a)Ask screen with both models ready.

(b)Text, routed to Smaug-Decisions: Yes, 0.939, 0.67 s.

(c)Image, routed to Smaug-Decisions: wet floor, 0.999, 7.13 s.

(d)Models sheet: what to install and τ.

5.3Device classes

The app chooses a mode from the phone's memory (Table 4). A phone that cannot run Smaug-Decisions never downloads it and still answers audio questions with the audio model.

6Limitations

• Decisions, not text: each call answers one typed question with a probability per option.
• Per request: one audio clip of up to 30 s (longer audio keeps its last 30 s), one image, up to 20 options; the phone build has a 2,048-token context.
• Set decision thresholds on your own data for high-stakes uses.

7Resources

• huggingface.co/abacusai/Smaug-Decisions: the main model for Transformers (audio, images, text).
• huggingface.co/abacusai/Smaug-Audio-Decisions-0.7B: the fast audio tier for Transformers and Core ML.
• huggingface.co/abacusai/Smaug-Decisions-LiteRT: Smaug-Decisions for phones (LiteRT-LM).
• github.com/abacusai/Smaug-Decisions-Mobile: the iOS and Android Cascade apps.

AEvaluation details

Skill. For each task, skill = (accuracy − chance) / (1 − chance) × 100, so 0 is guessing and 100 is perfect. Earshot's headline audio skill is the mean task skill within each track, averaged over the four tracks. No Earshot recording or item is in the training data.

Calibration. ECE is the expected calibration error of the top probability.

Decision Index. Smaug-Decisions' text and vision scores are our runs of the official Jev Decision Index kit: text on a 3,000-row sample of the public suite, compared on the same 35 benchmarks as the public board; vision at 560 image tokens. They are self-reported, not board submissions.

Latency. Server latency is the mean time per decision on one H100 in bfloat16, including escalations. Phone latency is wall-clock time in the app after the models are loaded.

Citation

@misc{abacusai2026smaugdecisions,
  title  = {Smaug-Decisions: calibrated typed decisions over audio, images and text},
  author = {Abacus.AI},
  year   = {2026},
  url    = {https://huggingface.co/abacusai/Smaug-Decisions}
}