150 questions in French, in six families: facts, tool calls, calculations, reading a short text, requests (a poem, a list, a message), small talk. A rule grades every answer, so the score needs no judge model and anyone can rerun it. The questions and the scorer are in the Vertigo repository.
The other models were run on this Mac with Ollama, same questions, same length limit, greedy decoding. They answer facts from memory; Vertigo reads Wikipedia when it asks to. The tool family is Vertigo's own tool calls, so it is left out of the comparison column.
Model
Size
Score
faits
calcul
lecture
demandes
conversation
Qwen2.5 1.5BAlibaba
1.0 GB
87.8
75
88.9
100
85
90
Llama 3.2 1BMeta
1.3 GB
81.1
77.5
77.8
70
90
90
Gemma 3 1BGoogle
0.8 GB
79
92.5
33.3
100
82.5
86.7
Qwen2.5 0.5BAlibaba
0.4 GB
74.1
57.5
88.9
80
57.5
86.7
Vertigo 3ParadoxalAI
98 MB
47.7
55
33.3
60
30
60
Score = mean of the five families, out of 100. Vertigo 3 also scores 100 on tool calls (56.4 over six families).
Where we stand
Last, today. Vertigo 3 is ten times smaller than the others and it shows on poems, messages and facts it has to recall. What it does that the others do not: it knows when to call a tool, and it reads before answering. The next version is four times bigger and is training now. It will be added to this table the day it is done, whatever the number.
Every answer
Pick a question to read what each model wrote. Green means the rule counted it right.