ScanObjectNN

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

Mikaela Angelina Uy¹ Quang-Hieu Pham² Binh-Son Hua³ Duc Thanh Nguyen⁴ Sai-Kit Yeung¹

¹Hong Kong University of Science and Technology ²Singapore University of Technology and Design
³The University of Tokyo ⁴Deakin University

International Conference on Computer Vision (ICCV), 2019 (Oral)

Sample objects from our ScanObjectNN dataset. The dataset contains ~15,000 objects that are categorized into 15 categories with 2902 unique object instances. The raw objects are represented by a list of points with global and local coordinates, normals, colors attributes and semantic labels. We also provide part annotations, which to the best of our knowledge is the first on real-world data.

Abstract

Deep learning techniques for point cloud data have demonstrated great potentials in solving classical problems in 3D computer vision such as 3D object classification and segmentation. Several recent 3D object classification methods have reported state-of-the-art performance on CAD model datasets such as ModelNet40 with high accuracy ~92%. Despite such impressive results, in this paper, we argue that object classification is still a challenging task when objects are framed with real-world settings. To prove this, we introduce ScanObjectNN, a new real-world point cloud object dataset based on scanned indoor scene data. From our comprehensive benchmark, we show that our dataset poses great challenges to existing point cloud classification techniques as objects from real-world scans are often cluttered with background and/or are partial due to occlusions. We identify three key open problems for point cloud object classification, and propose new point cloud classification neural networks that achieve state-of-the-art performance on classifying objects with cluttered background.

Benchmark

Method	overall acc.	avg acc.	bag	bin	box	cabinet	chair	desk	display	door	shelf	table	bed	pillow	sink	sofa	toilet
3DmFV	63.0	58.1	39.8	62.8	15.0	65.1	84.4	36.0	62.3	85.2	60.6	66.7	51.8	61.9	46.7	72.4	61.2
PointNet	68.2	63.4	36.1	69.8	10.5	62.6	89.0	50.0	73.0	93.8	72.6	67.8	61.8	67.6	64.2	76.7	55.3
SpiderCNN	73.7	69.8	43.4	75.9	12.8	74.2	89	65.3	74.5	91.4	78	65.9	69.1	80	65.8	90.5	70.6
PointNet++	77.9	75.4	49.4	84.4	31.6	77.4	91.3	74	79.4	85.2	72.6	72.6	75.5	81	80.8	90.5	85.9
DGCNN	78.1	73.6	49.4	82.4	33.1	83.9	91.8	63.3	77	89	79.3	77.4	64.5	77.1	75	91.4	69.4
PointCNN	78.5	75.1	57.8	82.9	33.1	83.6	92.6	65.3	78.4	84.8	84.2	67.4	80	80	72.5	91.9	71.8
BGA-PN++	80.2	77.5	54.2	85.9	39.8	81.7	90.8	76	84.3	87.6	78.4	74.4	73.6	80	77.5	91.9	85.9
BGA-DGCNN	79.7	75.7	48.2	81.9	30.1	84.4	92.6	77.3	80.4	92.4	80.5	74.1	72.7	78.1	79.2	91	72.9

Download as .csv

Leaderboard shows the classification accuracy for our hardest variant, PB_T50_RS. Researchers who want to add results on our dataset to this leaderboard, please visit our 3D Scene Understanding Benchmark page and submit your predictions there.

Materials

Paper

Supplementary

Poster

Code

Dataset:
If you would like to download the ScanObjectNN data, please fill out an agreement to the "ScanObjectNN Terms of Use" to get the download link. It should be noted that the processed and raw data sizes are 13 GB and 197 GB, respectively.

Citation

@inproceedings{uy-scanobjectnn-iccv19,
      title = {Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data},
      author = {Mikaela Angelina Uy and Quang-Hieu Pham and Binh-Son Hua and Duc Thanh Nguyen and Sai-Kit Yeung},
      booktitle = {International Conference on Computer Vision (ICCV)},
      year = {2019}
  }

Acknowledgements

We would like to sincerely thank Tan Sang Ha, Fan Wai Shan, Xu Ting Ting, Loh Pei Huan, Luong Van An, Ng Shi Xian Bryden, Li Jingxin and Chiz Huang for helping in the part annotations.

This research project is partially supported by an internal grant from HKUST (R9429).