PyTorch is a project written in a combination of Python, C++, and CUDA which was mainly developed in Facebook’s AI research lab.
It has shared a repository with deep learning framework Caffe2 since 2018 and is one of the main competitors to Google’s TensorFlow.
Here’s a write up of a recent update that adds distributed model parallel training.
In PyTorch 1.4, distributed model parallel training has been added to accommodate the growing scale and complexity of modern models. In the case of Facebook’s RoBERTa method, the amount of parameters to take into account can be up in the billions, which not all machines can handle.