examples and docs.

Summary: This diff updates the documentation and tutorials with information about the new pulsar backend. For more information about the pulsar backend, see the release notes and the paper (https://arxiv.org/abs/2004.07484). For information on how to use the backend, see the point cloud rendering notebook and the examples in the folder docs/examples.

Reviewed By: nikhilaravi

Differential Revision: D24498129

fbshipit-source-id: e312b0169a72b13590df6e4db36bfe6190d742f9

Author: classner

INSTALL.md

@@ -14,18 +14,20 @@ The core library is written in PyTorch. Several components have underlying imple
 - gcc & g++ ≥ 4.9
 - [fvcore](https://github.com/facebookresearch/fvcore)
 - If CUDA is to be used, use at least version 9.2.
+- If CUDA is to be used, the CUB library must be available. Starting from CUDA 11, CUB is part of CUDA. If you're using an earlier CUDA version and are not using conda, download the CUB library from https://github.com/NVIDIA/cub/releases and unpack it to a folder of your choice. Define the environment variable CUB_HOME before building and point it to the directory that contains `CMakeLists.txt` for CUB.
 
-These can be installed by running:
+The dependencies can be installed by running:
 ```
 conda create -n pytorch3d python=3.8
 conda activate pytorch3d
 conda install -c pytorch pytorch=1.6.0 torchvision cudatoolkit=10.2
 conda install -c conda-forge -c fvcore fvcore
+conda install -c cub
 ```
 
 ### Tests/Linting and Demos
 
-For developing on top of PyTorch3D or contributing, you will need to run the linter and tests. If you want to run any of the notebook tutorials as `docs/tutorials` you will also need matplotlib.
+For developing on top of PyTorch3D or contributing, you will need to run the linter and tests. If you want to run any of the notebook tutorials as `docs/tutorials` or the examples in `docs/examples` you will also need matplotlib and OpenCV.
 - scikit-image
 - black
 - isort
@@ -35,12 +37,13 @@ For developing on top of PyTorch3D or contributing, you will need to run the lin
 - jupyter
 - imageio
 - plotly
+- opencv-python
 
 These can be installed by running:
 ```
-# Demos
+# Demos and examples
 conda install jupyter
-pip install scikit-image matplotlib imageio plotly
+pip install scikit-image matplotlib imageio plotly opencv-python
 
 # Tests/Linting
 pip install black 'isort<5' flake8 flake8-bugbear flake8-comprehensions
@@ -81,6 +84,8 @@ To install using the code of the released version instead of from the main branc
 pip install 'git+https://github.com/facebookresearch/pytorch3d.git@stable'
 ```
 
+For CUDA builds with versions earlier than CUDA 11, set `CUB_HOME` before building as described above.
+
 **Install from Github on macOS:**
 Some environment variables should be provided, like this.
 ```
@@ -92,7 +97,7 @@ MACOSX_DEPLOYMENT_TARGET=10.14 CC=clang CXX=clang++ pip install 'git+https://git
 git clone https://github.com/facebookresearch/pytorch3d.git
 cd pytorch3d && pip install -e .
 ```
-To rebuild after installing from a local clone run, `rm -rf build/ **/*.so` then `pip install -e .`. You often need to rebuild pytorch3d after reinstalling PyTorch.
+To rebuild after installing from a local clone run, `rm -rf build/ **/*.so` then `pip install -e .`. You often need to rebuild pytorch3d after reinstalling PyTorch. For CUDA builds with versions earlier than CUDA 11, set `CUB_HOME` before building as described above.
 
 **Install from local clone on macOS:**
 ```

README.md

@@ -106,11 +106,23 @@ If you find PyTorch3D useful in your research, please cite our tech report:
 }
 ```
 
+If you are using the pulsar backend for sphere-rendering (the `PulsarPointRenderer` or `pytorch3d.renderer.points.pulsar.Renderer`), please cite the tech report:
+
+```bibtex
+@article{lassner2020pulsar,
+    author = {Christoph Lassner},
+    title = {Fast Differentiable Raycasting for Neural Rendering using Sphere-based Representations},
+    journal = {arXiv:2004.07484},
+    year = {2020},
+}
+```
 
 ## News
 
 Please see below for a timeline of the codebase updates in reverse chronological order. We are sharing updates on the releases as well as research projects which are built with PyTorch3D. The changelogs for the releases are available under [`Releases`](https://github.com/facebookresearch/pytorch3d/releases),  and the builds can be installed using `conda` as per the instructions in [INSTALL.md](INSTALL.md).
 
+**[November 2nd 2020]:** PyTorch3D v0.3 released, integrating the pulsar backend.
+
 **[Aug 28th 2020]:**   PyTorch3D v0.2.5 released
 
 **[July 17th 2020]:**   PyTorch3D tech report published on ArXiv: https://arxiv.org/abs/2007.08501

docs/examples/pulsar_basic.py

@@ -5,18 +5,23 @@
 sphere renderer. It renders and saves an image with 10 random spheres.
 Output: basic.png.
 """
+import math
 from os import path
 
 import imageio
 import torch
 from pytorch3d.renderer.points.pulsar import Renderer
 
 
+torch.manual_seed(1)
+
 n_points = 10
 width = 1_000
 height = 1_000
 device = torch.device("cuda")
-renderer = Renderer(width, height, n_points).to(device)
+# The PyTorch3D system is right handed; in pulsar you can choose the handedness.
+# For easy reproducibility we use a right handed coordinate system here.
+renderer = Renderer(width, height, n_points, right_handed_system=True).to(device)
 # Generate sample data.
 vert_pos = torch.rand(n_points, 3, dtype=torch.float32, device=device) * 10.0
 vert_pos[:, 2] += 25.0
@@ -29,7 +34,7 @@
         0.0,
         0.0,  # Position 0, 0, 0 (x, y, z).
         0.0,
-        0.0,
+        math.pi,  # Because of the right handed system, the camera must look 'back'.
         0.0,  # Rotation 0, 0, 0 (in axis-angle format).
         5.0,  # Focal length in world size.
         2.0,  # Sensor size in world size.

docs/examples/pulsar_basic_unified.py

@@ -0,0 +1,67 @@
+#!/usr/bin/env python3
+# Copyright (c) Facebook, Inc. and its affiliates. All rights reserved.
+"""
+This example demonstrates the most trivial use of the pulsar PyTorch3D
+interface for sphere renderering. It renders and saves an image with
+10 random spheres.
+Output: basic-pt3d.png.
+"""
+from os import path
+
+import imageio
+import torch
+from pytorch3d.renderer import PerspectiveCameras  # , look_at_view_transform
+from pytorch3d.renderer import (
+    PointsRasterizationSettings,
+    PointsRasterizer,
+    PulsarPointsRenderer,
+)
+from pytorch3d.structures import Pointclouds
+
+
+torch.manual_seed(1)
+
+n_points = 10
+width = 1_000
+height = 1_000
+device = torch.device("cuda")
+
+# Generate sample data.
+vert_pos = torch.rand(n_points, 3, dtype=torch.float32, device=device) * 10.0
+vert_pos[:, 2] += 25.0
+vert_pos[:, :2] -= 5.0
+vert_col = torch.rand(n_points, 3, dtype=torch.float32, device=device)
+pcl = Pointclouds(points=vert_pos[None, ...], features=vert_col[None, ...])
+# Alternatively, you can also use the look_at_view_transform to get R and T:
+# R, T = look_at_view_transform(
+#     dist=30.0, elev=0.0, azim=180.0, at=((0.0, 0.0, 30.0),), up=((0, 1, 0),),
+# )
+cameras = PerspectiveCameras(
+    # The focal length must be double the size for PyTorch3D because of the NDC
+    # coordinates spanning a range of two - and they must be normalized by the
+    # sensor width (see the pulsar example). This means we need here
+    # 5.0 * 2.0 / 2.0 to get the equivalent results as in pulsar.
+    focal_length=(5.0 * 2.0 / 2.0,),
+    R=torch.eye(3, dtype=torch.float32, device=device)[None, ...],
+    T=torch.zeros((1, 3), dtype=torch.float32, device=device),
+    image_size=((width, height),),
+    device=device,
+)
+vert_rad = torch.rand(n_points, dtype=torch.float32, device=device)
+raster_settings = PointsRasterizationSettings(
+    image_size=(width, height),
+    radius=vert_rad,
+)
+rasterizer = PointsRasterizer(cameras=cameras, raster_settings=raster_settings)
+renderer = PulsarPointsRenderer(rasterizer=rasterizer).to(device)
+# Render.
+image = renderer(
+    pcl,
+    gamma=(1.0e-1,),  # Renderer blending parameter gamma, in [1., 1e-5].
+    znear=(1.0,),
+    zfar=(45.0,),
+    radius_world=True,
+    bg_col=torch.ones((3,), dtype=torch.float32, device=device),
+)[0]
+print("Writing image to `%s`." % (path.abspath("basic-pt3d.png")))
+imageio.imsave("basic-pt3d.png", (image.cpu().detach() * 255.0).to(torch.uint8).numpy())

docs/examples/pulsar_cam.py

@@ -7,14 +7,17 @@
 The same scene parameterization is loaded and the camera parameters
 distorted. Gradient-based optimization is used to converge towards the
 original camera parameters.
+Output: cam.gif.
 """
+import math
 from os import path
 
 import cv2
 import imageio
 import numpy as np
 import torch
 from pytorch3d.renderer.points.pulsar import Renderer
+from pytorch3d.transforms import axis_angle_to_matrix, matrix_to_rotation_6d
 from torch import nn, optim
 
 
@@ -66,19 +69,18 @@ def __init__(self):
         )
         self.register_parameter(
             "cam_rot",
+            # We're using the 6D rot. representation for better gradients.
             nn.Parameter(
-                torch.tensor(
-                    [
-                        # We're using the 6D rot. representation for better gradients.
-                        0.9995,
-                        0.0300445,
-                        -0.0098482,
-                        -0.0299445,
-                        0.9995,
-                        0.0101482,
-                    ],
-                    dtype=torch.float32,
-                ),
+                matrix_to_rotation_6d(
+                    axis_angle_to_matrix(
+                        torch.tensor(
+                            [
+                                [0.02, math.pi + 0.02, 0.01],
+                            ],
+                            dtype=torch.float32,
+                        )
+                    )
+                )[0],
                 requires_grad=True,
             ),
         )
@@ -88,7 +90,7 @@ def __init__(self):
                 torch.tensor([4.8, 1.8], dtype=torch.float32), requires_grad=True
             ),
         )
-        self.renderer = Renderer(width, height, n_points)
+        self.renderer = Renderer(width, height, n_points, right_handed_system=True)
 
     def forward(self):
         return self.renderer.forward(
@@ -106,7 +108,7 @@ def forward(self):
     torch.from_numpy(
         imageio.imread(
             "../../tests/pulsar/reference/examples_TestRenderer_test_cam.png"
-        )
+        )[:, ::-1, :].copy()
     ).to(torch.float32)
     / 255.0
 ).to(device)