`get_rows` dequantizes as it gathers, so reading a few rows out of a large table never materializes
the rest -- which is what an embedding lookup wants. `dequantize` becomes the special case: every row
in order. CUDA has no indexed dequantize upstream, so it gathers first and unpacks after, as before.
Fail when a shipped artifact is not a loadable library
An .so committed while unsmudged stores its own LFS pointer as the object: a
~130-byte file that fails to dlopen. That happened once and was invisible,
because a kernel that will not load looks exactly like a kernel that was never
built for this platform, so every test skipped and the run was green.
test_artifacts.py checks the bytes on disk and that the build for this machine
imports, and it runs even when no kernel loads. Verified by reproducing the
corruption: two named failures instead of a green skip.
The test modules import `gguf_kernels`, which only resolved because the CUDA
variants still carry a `gguf_kernels/` shim from a pre-0.17 kernel-builder. The
Metal variants, built with 0.17, do not have it, so the suite could not be
collected against them at all -- and rebuilding CUDA will remove it there too.
Resolving through `kernels.get_local_kernel` instead exercises the same path a
consumer takes and does not depend on the layout. A platform with no build
reports skips rather than a collection error.
Metal: cover every quant type upstream's metallib has a gemv for
Marc Sun
0becb0a
The type table was four entries; the metallib always held kernels for 22. The gap was host-side:
CUDA reaches upstream's `mul_mat_vec_q_switch_type`, one function that switches over every type,
while ggml's Metal type->pipeline mapping lives in ggml-metal-device.cpp, which pulls in the whole
backend and is not compiled here. So each type has to be registered by hand.
Transcribed from ggml_metal_library_get_pipeline_mul_mv: the legacy quants, the K quants, the IQ
quants and MXFP4 -- 20 types, parity with the CUDA port apart from NVFP4/Q1_0/Q2_0, which the Metal
metallib has no kernels for. The one field not available as a macro is the grid: ggml-metal-ops.cpp
dispatches the dense types and q8_0 on (ne01 + nr0 - 1)/nr0 and everything else divides by nsg too,
so among quantized types q8_0 stays the only `mv_reduce_across_sgs`.
`GEMV_TYPES` becomes a `gemv_types()` op each backend answers for itself. It was one hardcoded CUDA
list, which Metal inherited and over-claimed by 19 types -- a …
GGUF kernels: dequantize + fused gemv over packed blocks, 9 CUDA variants
