CAPA: Convertibility-Aware Mixed-Precision Accelerator for Asymmetric GEMM in Weight-Only Quantized LLMs

Abstract

Weight-only quantized large language models (LLMs) have emerged as a standard paradigm for memory-efficient deployment, motivating mixed-precision (FP-INT) arithmetic between integer (INT) quantized weights (Qweights) and floating-point (FP) activations. While FP–INT GEMM accelerators facilitate their execution, prior designs incur high accumulation overheads, underutilize the parallelism of low-precision Qweights, and lack versatility. To address this, we propose CAPA, a convertibility-aware accelerator optimized for asymmetric GEMM, where operands exhibit significant bit-wise precision differences. We introduce Hybrid Delta Block Floating Point (HDBFP), a novel format that converts FP activations to INT while preserving accuracy, thereby maximizing the efficiency of INT-based processing elements (PEs). In addition, our data decomposition scheme significantly enhances computation parallelism for low-precision Qweights within a unified hardware datapath. Together, the CAPA architecture ensures full versatility across common FP16/BF16–INT8/INT4 combinations. Extensive evaluations show that CAPA maintains high accuracy comparable to a GPU baseline across language evaluation benchmarks, improving area efficiency by 3.98× and power efficiency by 4.38× over the FP–FP baseline.

Insu Choi

Ph.D. Candidate · AI Accelerators & Computer Architecture

My research interests include AI/ML, AI accelerators and memory reliability.