Tensor Parallelism

Walk through how tensor parallelism shards one matmul across GPUs and what it adds to decode step time

NVLink vs PCIe: match each interconnect to its bandwidth tier in a modern multi-GPU server.

Match each multi-GPU parallelism strategy to its defining property