Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

Mikaela Angelina Uy1          Quang-Hieu Pham2          Binh-Son Hua3          Duc Thanh Nguyen4          Sai-Kit Yeung1

1Hong Kong University of Science and Technology 2Singapore University of Technology and Design
3The University of Tokyo 4Deakin University

International Conference on Computer Vision (ICCV), 2019 (Oral)


Sample objects from our ScanObjectNN dataset. The dataset contains ~15,000 objects that are categorized into 15 categories with 2902 unique object instances. The raw objects are represented by a list of points with global and local coordinates, normals, colors attributes and semantic labels. We also provide part annotations, which to the best of our knowledge is the first on real-world data.

Abstract

Deep learning techniques for point cloud data have demonstrated great potentials in solving classical problems in 3D computer vision such as 3D object classification and segmentation. Several recent 3D object classification methods have reported state-of-the-art performance on CAD model datasets such as ModelNet40 with high accuracy ~92%. Despite such impressive results, in this paper, we argue that object classification is still a challenging task when objects are framed with real-world settings. To prove this, we introduce ScanObjectNN, a new real-world point cloud object dataset based on scanned indoor scene data. From our comprehensive benchmark, we show that our dataset poses great challenges to existing point cloud classification techniques as objects from real-world scans are often cluttered with background and/or are partial due to occlusions. We identify three key open problems for point cloud object classification, and propose new point cloud classification neural networks that achieve state-of-the-art performance on classifying objects with cluttered background.

Benchmark
Method overall acc. avg acc. bag bin box cabinet chair desk display door shelf table bed pillow sink sofa toilet
3DmFV63.058.139.862.815.065.184.436.062.385.260.666.751.861.946.772.461.2
PointNet68.263.436.169.810.562.689.050.073.093.872.667.861.867.664.276.755.3
SpiderCNN73.769.843.475.912.874.28965.374.591.47865.969.18065.890.570.6
PointNet++77.975.449.484.431.677.491.37479.485.272.672.675.58180.890.585.9
DGCNN78.173.649.482.433.183.991.863.3778979.377.464.577.17591.469.4
PointCNN78.575.157.882.933.183.692.665.378.484.884.267.4808072.591.971.8
BGA-PN++80.277.554.285.939.881.790.87684.387.678.474.473.68077.591.985.9
BGA-DGCNN79.775.748.281.930.184.492.677.380.492.480.574.172.778.179.29172.9

Download as .csv

Leaderboard shows the classification accuracy for our hardest variant, PB_T50_RS. Researchers who want to add results on our dataset to this leaderboard, please visit our 3D Scene Understanding Benchmark page and submit your predictions there.



Materials

Data coming soon! We are currently tidying up the data for public release, if an early access is needed for academic purposes, please send an email to mikacuy@gmail.com.

Citation
@inproceedings{uy-scanobjectnn-iccv19,
      title = {Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data},
      author = {Mikaela Angelina Uy and Quang-Hieu Pham and Binh-Son Hua and Duc Thanh Nguyen and Sai-Kit Yeung},
      booktitle = {International Conference on Computer Vision (ICCV)},
      year = {2019}
  }

Acknowledgements

We would like to sincerely thank Tan Sang Ha, Fan Wai Shan, Xu Ting Ting, Loh Pei Huan, Luong Van An, Ng Shi Xian Bryden, Li Jingxin and Chiz Huang for helping in the part annotations.

This research project is partially supported by an internal grant from HKUST (R9429).