GPU Kernels, Explained: CUDA, Triton, cuBLAS, torch.compile and KernelBench

Automated excerpt

What a GPU kernel is and how it maps work onto threads; the same kernel in CUDA and Triton; the roofline model and why fusion, tiling and Tensor Cores matter; what cuBLAS, cuDNN, eager PyTorch and torch.compile each do; how KernelBench scores a kernel and how to compare its numbers. HBM: High-bandwidth memory: the GPU's main memory (80 GB on an H100). Eager PyTorch launches a kernel per operator; torch.compile keeps the library matmul and fuses the elementwise tail into a generated Triton kernel; a hand-written kernel can do everything in one launch.

Selected automatically from source text; not independently written or fact-checked. Read the original for full context.