Distributed Training

Compare data, tensor, pipeline, and sharded parallelism in terms of memory ownership, communication, and synchronization.