Host-to-GPU bandwidth on an 8x RTX 4090 server without peer-to-peer
On an eight-GPU RTX 4090 server with no GPU peer-to-peer, every byte between GPUs goes through host memory. Measured, each CPU socket carries about 25 GB/s host to GPU, however many of its GPUs copy at once.