Installing Open3D-ML for 3D Computer Vision with PyTorch

In a previous post, I introduced my reasons to test Open3D-ML and the steps to install it with TensorFlow as the backend. In this post, I go over the steps to install the same library with PyTorch as the backend. Many of the steps are similar but there are some important differences. I hope this post is helpful for people interested in testing Open3D-ML.

To install Open3D-ML with PyTorch, follow the steps below. Note that my system is Ubuntu 20.04.4 LTS and I have a Cuda-enabled GPU, therefore, the instructions here presented may vary depending on your system.

Step 1: Install Conda

Using Conda is the recommended way to try anything new without risking breaking your system. To install Conda follow the official steps here.

Step 2: Create and activate a Conda environment

Make sure to replace myenv with the actual name that you want to use.

conda create --name myenv
conda activate myenv

Step 3: Install Node.js and Yarn

To install Node.js and Yarn you can follow the steps below:

curl -fsSL https://deb.nodesource.com/setup_18.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g yarn

Step 4: Install PyTorch with GPU support

conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge

Step 5: Install the cuDNN library

conda install -c anaconda cudnn

Step 6: Test PyTorch installation

To test the installation, run the following Python code. If the output is True, then all is working fine.

import torch
torch.cuda.is_available()

Step 7: Install Jupyter Lab

conda install -c conda-forge jupyterlab

Step 8: Clone Open3D

git clone https://github.com/isl-org/Open3D

Step 9: Install dependencies

cd Open3d
./util/install_deps_ubuntu.sh

Step 10: Create the build directory and clone Open3D-ML

mkdir build
cd build
git clone https://github.com/isl-org/Open3D-ML.git

Step 11: Configure the installation

This is assuming you have a Cuda-enabled GPU. Make sure to replace /path/to/your/conda-env/bin/python with the correct path to your Python. You can get this path by typing which python in a terminal. Also do not forget the two dots at the end of the command.

cmake -DBUILD_CUDA_MODULE=ON -DGLIBCXX_USE_CXX11_ABI=OFF -DBUILD_PYTORCH_OPS=ON -DBUILD_CUDA_MODULE=ON -DBUNDLE_OPEN3D_ML=ON -DOPEN3D_ML_ROOT=Open3D-ML -DBUILD_JUPYTER_EXTENSION:BOOL=ON -DBUILD_WEBRTC=ON -DPython3_ROOT=/path/to/your/conda-env/bin/python ..

If you get the following error:

CMake Error at /path/to/your/conda-env/lib/python/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:96 (message):
Your installed Caffe2 version uses cuDNN but I cannot find the cuDNN
libraries. Please set the proper cuDNN prefixes and / or install cuDNN.

Open the CMakeCache.txt file inside the build directory and edit the following lines to point to the cuDNN library. Make sure to replace /path/to/your/conda-env/ with the path to your Conda environment.

//Folder containing NVIDIA cuDNN header files
CUDNN_INCLUDE_DIR:FILEPATH=/path/to/your/conda-env/include//Path to a file.
CUDNN_INCLUDE_PATH:PATH=/path/to/your/conda-env/include//Path to the cudnn library file (e.g., libcudnn.so)
CUDNN_LIBRARY:PATH=FILEPATH=/path/to/your/conda-env/lib/libcudnn.so//Path to a library.
CUDNN_LIBRARY_PATH:FILEPATH=/path/to/your/conda-env/lib/libcudnn.so

Then run this again:

cmake -DBUILD_CUDA_MODULE=ON -DGLIBCXX_USE_CXX11_ABI=OFF -DBUILD_PYTORCH_OPS=ON -DBUILD_CUDA_MODULE=ON -DBUNDLE_OPEN3D_ML=ON -DOPEN3D_ML_ROOT=Open3D-ML -DBUILD_JUPYTER_EXTENSION:BOOL=ON -DBUILD_WEBRTC=ON -DPython3_ROOT=/path/to/your/conda-env/bin/python ..

Step 12: Build the library

make -j$(nproc)

Step 13: Install as a Python package

make install-pip-package

If you get ModuleNotFoundError: No module named ‘yapf’

pip install yapf

If you get ModuleNotFoundError: No module named ‘jupyter_packaging’

pip install jupyter-packaging

If you get ModuleNotFoundError: No module named ‘ipywidgets’

pip install ipywidgets

Then try installing again.

make install-pip-package

Step 14: Test Open3D installation

python -c "import open3d"

Step 15: Downloading and preparing a dataset

In this step, we will be downloading the SemanticKITTI dataset. This dataset is over 80 GB so make sure to have plenty of space and time. The following steps will download and prepare the dataset. Make sure to replace /path/to/save/dataset with the desired path.

cd Open3D-ML/scripts/download_datasets
./download_semantickitti.sh /path/to/save/dataset

Step 16: Loading and visualizing the dataset

In order to visualize the SemanticKITTI dataset, save the following Python code in a file and run it. Remember to replace /path/to/save/dataset/ with the path where the SemanticKITTI dataset was saved.

import open3d.ml.torch as ml3d  

# construct a dataset by specifying dataset_path
dataset = ml3d.datasets.SemanticKITTI(dataset_path='/path/to/save/dataset/')

# get the 'all' split that combines training, validation and test set
all_split = dataset.get_split('all')

# print the attributes of the first datum
print(all_split.get_attr(0))

# print the shape of the first point cloud
print(all_split.get_data(0)['point'].shape)

# show the first 340 frames using the visualizer
vis = ml3d.vis.Visualizer()
vis.visualize_dataset(dataset, 'all', indices=range(340))

Right when you run the Python script, a visualizer opens and loads the first 340 data frames. You can change the number of frames loaded in the code. Once opened, you can explore the point clouds based on intensity, but the most interesting part is to explore the point clouds based on the semantic label of each point. The videos below show two examples.

In the first video, you can see how by selecting multiple frames you can play them as an animation. Make sure to select labels as the data type from the presented options.

The second video shows how you can select a given frame and inspect the semantic objects present by activating and deactivating certain labels. When certain colors are too light and difficult to see, you can change the color to improve visibility.

And that’s it. Open3D-ML is a great tool for visualizing point cloud datasets. It can be used with both TensorFlow and PyTorch as the backend. This post explained how to install it with PyTorch.

The next step is to study the datasets to see how they were labeled. Then, I will go over training/testing 3D models with Open3D. Hopefully, this will bring me closer to performing the same operations with my custom data.