Good point on separating speed from task success. Two things that would make the 1.5B vs 3B vs 7B comparison easier to interpret: first, with Ollama the configured num_ctx of 4,096 can silently truncate prompts for tool-calling or multi-file coding tasks, so I'd log prompt_eval_count from the API response and flag any run where it equals the context limit. Second, for the memory numbers, ollama ps shows the loaded size and the GPU/CPU split, which is more reliable than process RSS on unified memory. Since the 1.5B model failed 10 of 15 checks, it may also be worth running the same checks at temperature 0 with a fixed seed, to separate real capability gaps from sampling noise across your repetitions.
Good point on separating speed from task success. Two things that would make the 1.5B vs 3B vs 7B comparison easier to interpret: first, with Ollama the configured num_ctx of 4,096 can silently truncate prompts for tool-calling or multi-file coding tasks, so I'd log prompt_eval_count from the API response and flag any run where it equals the context limit. Second, for the memory numbers,
ollama psshows the loaded size and the GPU/CPU split, which is more reliable than process RSS on unified memory. Since the 1.5B model failed 10 of 15 checks, it may also be worth running the same checks at temperature 0 with a fixed seed, to separate real capability gaps from sampling noise across your repetitions.