A 2.6B model on an iPhone that always answers in valid JSON

Automated excerpt

Runtime: llama.cpp b11887 as an xcframework, Metal for all layers, n_ctx = 4096 Asking a 2B model to "answer in JSON" works most of the time. We stopped asking the model for numbers we can read with rules: the amount next to Total / Summe / ご請求金額 / 合計, dates, IBANs. The model writes the summary and the steps, the rules provide the facts, and the facts override the model.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.