Neko code bench

Small local models write Python for pure-function tasks. The bench runs the code against real test cases and judges pass/fail, plus a hallucination flag when a model claims its code passes and it does not. Round 479, task rotation over 3 recent cycles.

modelpassfailerrorhallucinated
qwen2.5:3b42057242
gemma2:2b368109271
glm-4.7-flash45423218
2026-10-14.log
9] qwen2.5:3b: PASS (3s)
[20:16:20] gemma2:2b: PASS (2s)
[20:16:27] glm-4.7-flash: PASS (7s)
[20:16:27] round 424 done: qwen2.5:3b: P371/F51/E2/H39 | gemma2:2b: P323/F99/E2/H63 | glm-4.7-flash: P402/F20/E2/H16
[20:36:28] --- round 425 task flatten ---
[20:36:30] qwen2.5:3b: PASS (2s)
[20:36:32] gemma2:2b: PASS (2s)
[20:36:44] glm-4.7-flash: PASS (12s)
[20:36:44] round 425 done: qwen2.5:3b: P372/F51/E2/H39 | gemma2:2b: P324/F99/E2/H63 | glm-4.7-flash: P403/F20/E2/H16
[20:56:45] --- round 426 task fizzbuzz_list ---
[20:56:48] qwen2.5:3b: FAIL (AssertionError) + HALLUCINATED_CLAIM (3s)
[20:56:50] gemma2:2b: PASS (2s)
[20:56:58] glm-4.7-flash: PASS (8s)
[20:56:58] round 426 done: qwen2.5:3b: P372/F52/E2/H40 | gemma2:2b: P325/F99/E2/H63 | glm-4.7-flash: P404/F20/E2/H16
[21:16:59] --- round 427 task roman_numerals ---
[21:17:01] qwen2.5:3b: FAIL (AssertionError) (3s)
[21:17:05] gemma2:2b: FAIL (KeyError: 0) + HALLUCINATED_CLAIM (4s)
[21:17:11] glm-4.7-flash: PASS (6s)
[21:17:11] round 427 done: qwen2.5:3b: P372/F53/E2/H40 | gemma2:2b: P325/F100/E2/H64 | glm-4.7-flash: P405/F20/E2/H16
[21:37:12] --- round 428 task is_leap ---
[21:37:14] qwen2.5:3b: PASS (3s)
[21:37:16] gemma2:2b: PASS (2s)
[21:37:22] glm-4.7-flash: PASS (5s)
[21:37:22] round 428 done: qwen2.5:3b: P373/F53/E2/H40 | gemma2:2b: P326/F100/E2/H64 | glm-4.7-flash: P406/F20/E2/H16
[21:57:23] --- round 429 task word_freq ---
[21:57:24] qwen2.5:3b: PASS (2s)
[21:57:27] gemma2:2b: PASS (3s)
[21:57:33] glm-4.7-flash: PASS (6s)
[21:57:33] round 429 done: qwen2.5:3b: P374/F53/E2/H40 | gemma2:2b: P327/F100/E2/H64 | glm-4.7-flash: P407/F20/E2/H16
[22:17:34] --- round 430 task flatten ---
[22:17:36] qwen2.5:3b: PASS (2s)
[22:17:37] gemma2:2b: PASS (1s)
[22:17:42] glm-4.7-flash: PASS (5s)
[22:17:42] round 430 done: qwen2.5:3b: P375/F53/E2/H40 | gemma2:2b: P328/F100/E2/H64 | glm-4.7-flash: P408/F20/E2/H16
[22:37:43] --- round 431 task fizzbuzz_list ---
[22:37:50] qwen2.5:3b: PASS (7s)
[22:38:19] gemma2:2b: PASS (28s)
[22:38:36] glm-4.7-flash: PASS (18s)
[22:38:36] round 431 done: qwen2.5:3b: P376/F53/E2/H40 | gemma2:2b: P329/F100/E2/H64 | glm-4.7-flash: P409/F20/E2/H16
[22:58:37] --- round 432 task roman_numerals ---
[22:58:49] qwen2.5:3b: FAIL (AssertionError) + HALLUCINATED_CLAIM (12s)
[22:59:00] gemma2:2b: FAIL (AssertionError) (12s)
[22:59:30] glm-4.7-flash: PASS (29s)
[22:59:30] round 432 done: qwen2.5:3b: P376/F54/E2/H41 | gemma2:2b: P329/F101/E2/H64 | glm-4.7-flash: P410/F20/E2/H16
[23:19:31] --- round 433 task is_leap ---
[23:19:39] qwen2.5:3b: PASS (9s)
[23:19:47] gemma2:2b: PASS (7s)
[23:20:08] glm-4.7-flash: PASS (22s)
[23:20:08] round 433 done: qwen2.5:3b: P377/F54/E2/H41 | gemma2:2b: P330/F101/E2/H64 | glm-4.7-flash: P411/F20/E2/H16
[23:40:09] --- round 434 task word_freq ---
[23:40:11] qwen2.5:3b: PASS (2s)
[23:40:14] gemma2:2b: PASS (3s)
[23:40:23] glm-4.7-flash: PASS (9s)
[23:40:23] round 434 done: qwen2.5:3b: P378/F54/E2/H41 | gemma2:2b: P331/F101/E2/H64 | glm-4.7-flash: P412/F20/E2/H16
2026-10-15.log
:22] gemma2:2b: PASS (3s)
[11:28:27] glm-4.7-flash: PASS (5s)
[11:28:27] round 469 done: qwen2.5:3b: P412/F55/E2/H41 | gemma2:2b: P360/F107/E2/H69 | glm-4.7-flash: P444/F23/E2/H18
[11:48:28] --- round 470 task flatten ---
[11:48:30] qwen2.5:3b: PASS (2s)
[11:48:31] gemma2:2b: PASS (1s)
[11:48:37] glm-4.7-flash: PASS (5s)
[11:48:37] round 470 done: qwen2.5:3b: P413/F55/E2/H41 | gemma2:2b: P361/F107/E2/H69 | glm-4.7-flash: P445/F23/E2/H18
[12:08:38] --- round 471 task fizzbuzz_list ---
[12:08:41] qwen2.5:3b: PASS (3s)
[12:08:43] gemma2:2b: PASS (2s)
[12:08:52] glm-4.7-flash: PASS (9s)
[12:08:52] round 471 done: qwen2.5:3b: P414/F55/E2/H41 | gemma2:2b: P362/F107/E2/H69 | glm-4.7-flash: P446/F23/E2/H18
[12:28:53] --- round 472 task roman_numerals ---
[12:28:56] qwen2.5:3b: FAIL (ValueError: not enough values to unpack (expected 4, got 2)) + HALLUCINATED_CLAIM (3s)
[12:28:58] gemma2:2b: PASS (2s)
[12:29:13] glm-4.7-flash: PASS (15s)
[12:29:13] round 472 done: qwen2.5:3b: P414/F56/E2/H42 | gemma2:2b: P363/F107/E2/H69 | glm-4.7-flash: P447/F23/E2/H18
[12:49:14] --- round 473 task is_leap ---
[12:49:16] qwen2.5:3b: PASS (2s)
[12:49:17] gemma2:2b: PASS (1s)
[12:49:33] glm-4.7-flash: PASS (16s)
[12:49:33] round 473 done: qwen2.5:3b: P415/F56/E2/H42 | gemma2:2b: P364/F107/E2/H69 | glm-4.7-flash: P448/F23/E2/H18
[13:09:34] --- round 474 task word_freq ---
[13:09:36] qwen2.5:3b: PASS (2s)
[13:09:39] gemma2:2b: FAIL (AssertionError) + HALLUCINATED_CLAIM (2s)
[13:09:49] glm-4.7-flash: PASS (11s)
[13:09:49] round 474 done: qwen2.5:3b: P416/F56/E2/H42 | gemma2:2b: P364/F108/E2/H70 | glm-4.7-flash: P449/F23/E2/H18
[13:29:58] --- round 475 task flatten ---
[13:30:00] qwen2.5:3b: PASS (2s)
[13:30:02] gemma2:2b: PASS (2s)
[13:30:10] glm-4.7-flash: PASS (7s)
[13:30:10] round 475 done: qwen2.5:3b: P417/F56/E2/H42 | gemma2:2b: P365/F108/E2/H70 | glm-4.7-flash: P450/F23/E2/H18
[13:50:10] --- round 476 task fizzbuzz_list ---
[13:50:12] qwen2.5:3b: PASS (2s)
[13:50:16] gemma2:2b: PASS (4s)
[13:50:27] glm-4.7-flash: PASS (11s)
[13:50:27] round 476 done: qwen2.5:3b: P418/F56/E2/H42 | gemma2:2b: P366/F108/E2/H70 | glm-4.7-flash: P451/F23/E2/H18
[14:10:28] --- round 477 task roman_numerals ---
[14:10:31] qwen2.5:3b: FAIL (AssertionError) (3s)
[14:10:32] gemma2:2b: FAIL (KeyError: 0) + HALLUCINATED_CLAIM (2s)
[14:10:44] glm-4.7-flash: PASS (12s)
[14:10:44] round 477 done: qwen2.5:3b: P418/F57/E2/H42 | gemma2:2b: P366/F109/E2/H71 | glm-4.7-flash: P452/F23/E2/H18
[14:30:45] --- round 478 task is_leap ---
[14:30:50] qwen2.5:3b: PASS (5s)
[14:30:52] gemma2:2b: PASS (2s)
[14:31:03] glm-4.7-flash: PASS (10s)
[14:31:03] round 478 done: qwen2.5:3b: P419/F57/E2/H42 | gemma2:2b: P367/F109/E2/H71 | glm-4.7-flash: P453/F23/E2/H18
[14:51:04] --- round 479 task word_freq ---
[14:51:09] qwen2.5:3b: PASS (6s)
[14:51:11] gemma2:2b: PASS (2s)
[14:51:21] glm-4.7-flash: PASS (10s)
[14:51:21] round 479 done: qwen2.5:3b: P420/F57/E2/H42 | gemma2:2b: P368/F109/E2/H71 | glm-4.7-flash: P454/F23/E2/H18
loop.log
:22] gemma2:2b: PASS (3s)
[11:28:27] glm-4.7-flash: PASS (5s)
[11:28:27] round 469 done: qwen2.5:3b: P412/F55/E2/H41 | gemma2:2b: P360/F107/E2/H69 | glm-4.7-flash: P444/F23/E2/H18
[11:48:28] --- round 470 task flatten ---
[11:48:30] qwen2.5:3b: PASS (2s)
[11:48:31] gemma2:2b: PASS (1s)
[11:48:37] glm-4.7-flash: PASS (5s)
[11:48:37] round 470 done: qwen2.5:3b: P413/F55/E2/H41 | gemma2:2b: P361/F107/E2/H69 | glm-4.7-flash: P445/F23/E2/H18
[12:08:38] --- round 471 task fizzbuzz_list ---
[12:08:41] qwen2.5:3b: PASS (3s)
[12:08:43] gemma2:2b: PASS (2s)
[12:08:52] glm-4.7-flash: PASS (9s)
[12:08:52] round 471 done: qwen2.5:3b: P414/F55/E2/H41 | gemma2:2b: P362/F107/E2/H69 | glm-4.7-flash: P446/F23/E2/H18
[12:28:53] --- round 472 task roman_numerals ---
[12:28:56] qwen2.5:3b: FAIL (ValueError: not enough values to unpack (expected 4, got 2)) + HALLUCINATED_CLAIM (3s)
[12:28:58] gemma2:2b: PASS (2s)
[12:29:13] glm-4.7-flash: PASS (15s)
[12:29:13] round 472 done: qwen2.5:3b: P414/F56/E2/H42 | gemma2:2b: P363/F107/E2/H69 | glm-4.7-flash: P447/F23/E2/H18
[12:49:14] --- round 473 task is_leap ---
[12:49:16] qwen2.5:3b: PASS (2s)
[12:49:17] gemma2:2b: PASS (1s)
[12:49:33] glm-4.7-flash: PASS (16s)
[12:49:33] round 473 done: qwen2.5:3b: P415/F56/E2/H42 | gemma2:2b: P364/F107/E2/H69 | glm-4.7-flash: P448/F23/E2/H18
[13:09:34] --- round 474 task word_freq ---
[13:09:36] qwen2.5:3b: PASS (2s)
[13:09:39] gemma2:2b: FAIL (AssertionError) + HALLUCINATED_CLAIM (2s)
[13:09:49] glm-4.7-flash: PASS (11s)
[13:09:49] round 474 done: qwen2.5:3b: P416/F56/E2/H42 | gemma2:2b: P364/F108/E2/H70 | glm-4.7-flash: P449/F23/E2/H18
[13:29:58] --- round 475 task flatten ---
[13:30:00] qwen2.5:3b: PASS (2s)
[13:30:02] gemma2:2b: PASS (2s)
[13:30:10] glm-4.7-flash: PASS (7s)
[13:30:10] round 475 done: qwen2.5:3b: P417/F56/E2/H42 | gemma2:2b: P365/F108/E2/H70 | glm-4.7-flash: P450/F23/E2/H18
[13:50:10] --- round 476 task fizzbuzz_list ---
[13:50:12] qwen2.5:3b: PASS (2s)
[13:50:16] gemma2:2b: PASS (4s)
[13:50:27] glm-4.7-flash: PASS (11s)
[13:50:27] round 476 done: qwen2.5:3b: P418/F56/E2/H42 | gemma2:2b: P366/F108/E2/H70 | glm-4.7-flash: P451/F23/E2/H18
[14:10:28] --- round 477 task roman_numerals ---
[14:10:31] qwen2.5:3b: FAIL (AssertionError) (3s)
[14:10:32] gemma2:2b: FAIL (KeyError: 0) + HALLUCINATED_CLAIM (2s)
[14:10:44] glm-4.7-flash: PASS (12s)
[14:10:44] round 477 done: qwen2.5:3b: P418/F57/E2/H42 | gemma2:2b: P366/F109/E2/H71 | glm-4.7-flash: P452/F23/E2/H18
[14:30:45] --- round 478 task is_leap ---
[14:30:50] qwen2.5:3b: PASS (5s)
[14:30:52] gemma2:2b: PASS (2s)
[14:31:03] glm-4.7-flash: PASS (10s)
[14:31:03] round 478 done: qwen2.5:3b: P419/F57/E2/H42 | gemma2:2b: P367/F109/E2/H71 | glm-4.7-flash: P453/F23/E2/H18
[14:51:04] --- round 479 task word_freq ---
[14:51:09] qwen2.5:3b: PASS (6s)
[14:51:11] gemma2:2b: PASS (2s)
[14:51:21] glm-4.7-flash: PASS (10s)
[14:51:21] round 479 done: qwen2.5:3b: P420/F57/E2/H42 | gemma2:2b: P368/F109/E2/H71 | glm-4.7-flash: P454/F23/E2/H18

successor to the psych-debate bench, archived Oct 8 2026. logs live on the box.