【Scannet V2 三维数据集下载】

数据集下载】"/>

【Scannet V2 三维数据集下载】

Scannet V2三维数据下载

Scannet V2数据介绍：“ScanNet 是一个 RGB-D 视频数据集，包含 2 多次扫描中的 5 万次观看，并带有 1500D 摄像机姿势、表面重建和实例级语义分割进行注释。 为了收集这些数据，我们设计了一个易于使用且可扩展的RGB-D捕获系统，其中包括自动表面重建和众包语义注释。 我们表明，使用这些数据有助于在多个 3D 场景理解任务上实现最先进的性能，包括 3D 对象分类、语义体素标记和 CAD 模型检索。 更多信息可以在我们的论文: link中找到。”
更多关于Scannet数据的介绍可以查看这篇CSDN博客: scannet数据集简介和下载

因为我的目的是用Scannet V2的三维数据集来进行HAIS论文复现，所以这里我只复述一下我是如何下载该三维数据集的。Scannet数据按正常流程来说，是需要签订一份协议并且向该数据团队申请的，发完邮件之后等他们通过申请然后回复下载数据的python脚本。但是等待回复的时间有长有短，我着急复现实验，所以直接上网扒代码进行下载。

电脑配置：

Ubuntu20.0
python2.7
这里需要注意，运行下载scannet数据的代码可以用python2.7也可以用python3.x。

注意事项：

因为我用的是python2.7，所以在运行时可能会出现一些问题。比如：
1.ERROR:SyntaxError: Non-ASCII character ‘\xe5’ in file
这是因为当使用中文输出或注释时运行脚本的时候，导致出现问题。因为python默认的编码文件是ASCII码，但是你的代码中使用中文或者非英语字符，第一种最最简单的方法就是去文件里把中文部分处理掉，如果不想动它，建议在每个源文件的基础上，使用文件顶部的特殊注释声明编码，使python源代码编码既可见又可更改。即在代码页的最顶端，第一行加入以下编码提示：

# coding: utf-8
#!/usr/bin/env python

或者

# -*- coding:UTF-8 -*-
#!/usr/bin/env python

注意：coding: utf-8中，coding后面不能有空格，即#coding ：utf-8会报错。
在Ubuntu系统上#coding ：utf-8需要放在代码的第一行，而在Windows系统上，#coding ：utf-8好像不需要放在第一行，只要放在开头就行。
2.在代码运行时：
会出现Press any key to countinue ,or CTRL-C to exit，这个时候，如果是Windows系统的话不用管它，也不要按任何键盘上的按键，它自己就会跟着下载数据（不知道这是否只有我一个人是这样的情况，我仅仅是把我所遇到的问题陈述出来）。如果是在Ubuntu系统上，则需要键入任意数字，然后摁下回车键即可，如果直接摁下回车键或者其他的字符都会报错。
3.在download_scannetv2.py中，如果用python2.7需要更改几行代码：
import urllib.request #(for python3) 更改成import urllib #(for python2.7);
def get_release_scans(release_file):函数中的scan_lines = urllib.request.urlopen(release_file) # python3.x用这一行更改为scan_lines = urllib.urlopen(release_file) # python2.7用这一行
4.报错：UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xe4 in position 0: ordinal not in range(128)
这是因为python2.7的str默认是ascii编码，和unicode编码冲突，就会报这个标题错误，所以需要在代码页开头加上

from imp import reload
reload(sys)
sys.setdefaultencoding('utf8')

5.报错：IOError[Errno socket error] [Errno -3] Temporary failure in name resolution
下载的时候偶尔会出现这个报错，将命令行重新运行一次就行，代码会接着下载数据集。

代码下载：

该代码从这篇CSDN博客引用: scannet数据集下载文件
下面是 download_scannetv2.py的代码：

// An highlighted block
#coding:utf-8
#!/usr/bin/env python
# Downloads ScanNet public data release
# Run with ./download-scannet.py (or python download-scannet.py on Windows)
# -*- coding: utf-8 -*-
import argparse
import os
#import urllib.request #(for python3)
import urllib #(for python2.7)
import tempfile
from imp import reload
reload(sys)
sys.setdefaultencoding('utf8')BASE_URL = '/'
TOS_URL = BASE_URL + 'ScanNet_TOS.pdf'
FILETYPES = ['.aggregation.json', '.sens', '.txt', '_vh_clean.ply', '_vh_clean_2.0.010000.segs.json', '_vh_clean_2.ply', '_vh_clean.segs.json', '_vh_clean.aggregation.json', '_vh_clean_2.labels.ply', '_2d-instance.zip', '_2d-instance-filt.zip', '_2d-label.zip', '_2d-label-filt.zip']
FILETYPES_TEST = ['.sens', '.txt', '_vh_clean.ply', '_vh_clean_2.ply']
PREPROCESSED_FRAMES_FILE = ['scannet_frames_25k.zip', '5.6GB']
TEST_FRAMES_FILE = ['scannet_frames_test.zip', '610MB']
LABEL_MAP_FILES = ['scannetv2-labelsbined.tsv', 'scannet-labelsbined.tsv']
RELEASES = ['v2/scans', 'v1/scans']
RELEASES_TASKS = ['v2/tasks', 'v1/tasks']
RELEASES_NAMES = ['v2', 'v1']
RELEASE = RELEASES[0]
RELEASE_TASKS = RELEASES_TASKS[0]
RELEASE_NAME = RELEASES_NAMES[0]
LABEL_MAP_FILE = LABEL_MAP_FILES[0]
RELEASE_SIZE = '1.2TB'
V1_IDX = 1def get_release_scans(release_file):#scan_lines = urllib.request.urlopen(release_file)  # python3.x用这一行scan_lines = urllib.urlopen(release_file)  # python2.7用这一行scans = []for scan_line in scan_lines:scan_id = scan_line.decode('utf8').rstrip('\n')scans.append(scan_id)return scansdef download_release(release_scans, out_dir, file_types, use_v1_sens):if len(release_scans) == 0:returnprint('Downloading ScanNet ' + RELEASE_NAME + ' release to ' + out_dir + '...')for scan_id in release_scans:scan_out_dir = os.path.join(out_dir, scan_id)download_scan(scan_id, scan_out_dir, file_types, use_v1_sens)print('Downloaded ScanNet ' + RELEASE_NAME + ' release.')def download_file(url, out_file):out_dir = os.path.dirname(out_file)if not os.path.isdir(out_dir):os.makedirs(out_dir)if not os.path.isfile(out_file):print('\t' + url + ' > ' + out_file)fh, out_file_tmp = tempfile.mkstemp(dir=out_dir)f = os.fdopen(fh, 'w')f.close()#urllib.request.urlretrieve(url, out_file_tmp)urllib.urlretrieve(url, out_file_tmp)os.rename(out_file_tmp, out_file)else:print('WARNING: skipping download of existing file ' + out_file)def download_scan(scan_id, out_dir, file_types, use_v1_sens):print('Downloading ScanNet ' + RELEASE_NAME + ' scan ' + scan_id + ' ...')if not os.path.isdir(out_dir):os.makedirs(out_dir)for ft in file_types:v1_sens = use_v1_sens and ft == '.sens'url = BASE_URL + RELEASE + '/' + scan_id + '/' + scan_id + ft if not v1_sens else BASE_URL + RELEASES[V1_IDX] + '/' + scan_id + '/' + scan_id + ftout_file = out_dir + '/' + scan_id + ftdownload_file(url, out_file)print('Downloaded scan ' + scan_id)def download_task_data(out_dir):print('Downloading ScanNet v1 task data...')files = [LABEL_MAP_FILES[V1_IDX], 'obj_classification/data.zip','obj_classification/trained_models.zip', 'voxel_labeling/data.zip','voxel_labeling/trained_models.zip']for file in files:url = BASE_URL + RELEASES_TASKS[V1_IDX] + '/' + filelocalpath = os.path.join(out_dir, file)localdir = os.path.dirname(localpath)if not os.path.isdir(localdir):os.makedirs(localdir)download_file(url, localpath)print('Downloaded task data.')def download_label_map(out_dir):print('Downloading ScanNet ' + RELEASE_NAME + ' label mapping file...')files = [ LABEL_MAP_FILE ]for file in files:url = BASE_URL + RELEASE_TASKS + '/' + filelocalpath = os.path.join(out_dir, file)localdir = os.path.dirname(localpath)if not os.path.isdir(localdir):os.makedirs(localdir)download_file(url, localpath)print('Downloaded ScanNet ' + RELEASE_NAME + ' label mapping file.')def main():parser = argparse.ArgumentParser(description='Downloads ScanNet public data release.')parser.add_argument('-o', '--out_dir', required=True, help='directory in which to download')parser.add_argument('--task_data', action='store_true', help='download task data (v1)')parser.add_argument('--label_map', action='store_true', help='download label map file')parser.add_argument('--v1', action='store_true', help='download ScanNet v1 instead of v2')parser.add_argument('--id', help='specific scan id to download')parser.add_argument('--preprocessed_frames', action='store_true', help='download preprocessed subset of ScanNet frames (' + PREPROCESSED_FRAMES_FILE[1] + ')')parser.add_argument('--test_frames_2d', action='store_true', help='download 2D test frames (' + TEST_FRAMES_FILE[1] + '; also included with whole dataset download)')parser.add_argument('--type', help='specific file type to download (.aggregation.json, .sens, .txt, _vh_clean.ply, _vh_clean_2.0.010000.segs.json, _vh_clean_2.ply, _vh_clean.segs.json, _vh_clean.aggregation.json, _vh_clean_2.labels.ply, _2d-instance.zip, _2d-instance-filt.zip, _2d-label.zip, _2d-label-filt.zip)')args = parser.parse_args()print('By pressing any key to continue you confirm that you have agreed to the ScanNet terms of use as described at:')print(TOS_URL)print('***')print('Press any key to continue, or CTRL-C to exit.')# key = raw_input('')if args.v1:global RELEASEglobal RELEASE_TASKSglobal RELEASE_NAMEglobal LABEL_MAP_FILERELEASE = RELEASES[V1_IDX]RELEASE_TASKS = RELEASES_TASKS[V1_IDX]RELEASE_NAME = RELEASES_NAMES[V1_IDX]LABEL_MAP_FILE = LABEL_MAP_FILES[V1_IDX]release_file = BASE_URL + RELEASE + '.txt'release_scans = get_release_scans(release_file)file_types = FILETYPES;release_test_file = BASE_URL + RELEASE + '_test.txt'release_test_scans = get_release_scans(release_test_file)file_types_test = FILETYPES_TEST;out_dir_scans = os.path.join(args.out_dir, 'scans')out_dir_test_scans = os.path.join(args.out_dir, 'scans_test')out_dir_tasks = os.path.join(args.out_dir, 'tasks')if args.type:  # download file typefile_type = args.typeif file_type not in FILETYPES:print('ERROR: Invalid file type: ' + file_type)returnfile_types = [file_type]if file_type in FILETYPES_TEST:file_types_test = [file_type]else:file_types_test = []if args.task_data:  # download task datadownload_task_data(out_dir_tasks)elif args.label_map:  # download label map filedownload_label_map(args.out_dir)elif args.preprocessed_frames:  # download preprocessed scannet_frames_25k.zip fileif args.v1:print('ERROR: Preprocessed frames only available for ScanNet v2')print('You are downloading the preprocessed subset of frames ' + PREPROCESSED_FRAMES_FILE[0] + ' which requires ' + PREPROCESSED_FRAMES_FILE[1] + ' of space.')download_file(os.path.join(BASE_URL, RELEASE_TASKS, PREPROCESSED_FRAMES_FILE[0]), os.path.join(out_dir_tasks, PREPROCESSED_FRAMES_FILE[0]))elif args.test_frames_2d:  # download test scannet_frames_test.zip fileif args.v1:print('ERROR: 2D test frames only available for ScanNet v2')print('You are downloading the 2D test set ' + TEST_FRAMES_FILE[0] + ' which requires ' + TEST_FRAMES_FILE[1] + ' of space.')download_file(os.path.join(BASE_URL, RELEASE_TASKS, TEST_FRAMES_FILE[0]), os.path.join(out_dir_tasks, TEST_FRAMES_FILE[0]))elif args.id:  # download single scanscan_id = args.idis_test_scan = scan_id in release_test_scansif scan_id not in release_scans and (not is_test_scan or args.v1):print('ERROR: Invalid scan id: ' + scan_id)else:out_dir = os.path.join(out_dir_scans, scan_id) if not is_test_scan else os.path.join(out_dir_test_scans, scan_id)scan_file_types = file_types if not is_test_scan else file_types_testuse_v1_sens = not is_test_scanif not is_test_scan and not args.v1 and '.sens' in scan_file_types:print('Note: ScanNet v2 uses the same .sens files as ScanNet v1: Press \'n\' to exclude downloading .sens files for each scan')# key = raw_input('')# if key.strip().lower() == 'n':# scan_file_types.remove('.sens')download_scan(scan_id, out_dir, scan_file_types, use_v1_sens)else:  # download entire releaseif len(file_types) == len(FILETYPES):print('WARNING: You are downloading the entire ScanNet ' + RELEASE_NAME + ' release which requires ' + RELEASE_SIZE + ' of space.')else:print('WARNING: You are downloading all ScanNet ' + RELEASE_NAME + ' scans of type ' + file_types[0])print('Note that existing scan directories will be skipped. Delete partially downloaded directories to re-download.')print('***')print('Press any key to continue, or CTRL-C to exit.')# key = raw_input('')if not args.v1 and '.sens' in file_types:print('Note: ScanNet v2 uses the same .sens files as ScanNet v1: Press \'n\' to exclude downloading .sens files for each scan')# key = raw_input('')# if key.strip().lower() == 'n':# file_types.remove('.sens')download_release(release_scans, out_dir_scans, file_types, use_v1_sens=True)if not args.v1:download_label_map(args.out_dir)download_release(release_test_scans, out_dir_test_scans, file_types_test, use_v1_sens=False)download_file(os.path.join(BASE_URL, RELEASE_TASKS, TEST_FRAMES_FILE[0]), os.path.join(out_dir_tasks, TEST_FRAMES_FILE[0]))if __name__ == "__main__": main()

运行命令：

python download_scannetv2.py -o path(存放数据集的路径) --type _vh_clean_2.ply
python download_scannetv2.py -o path(存放数据集的) --type _vh_clean_2.label.ply
#python3 download-scannetv2.py -o scannet/ --type  _vh_clean_2.labels.ply
python download_scannetv2.py -o path(存放数据集的) --type _vh_clean_2.0.010000.segs.json
python download_scannetv2.py -o path(存放数据集的) --type .aggregation.json

如果你的代码文件名字不是 download_scannetv2.py，那你也可以将 download_scannetv2.py的名字换成自己的代码名字。

下载后的目录结构如下：

├── scannet
│   ├── scans
│   │   ├── [scene_id]									
│   │   │   ├── [scene_id].aggregation.json
│   │   │   ├── [scene_id]_vh_clean_2.0.010000.segs.json
│   │   │   ├── [scene_id]_vh_clean_2.labels.ply
│   │   │   ├── [scene_id]_vh_clean_2.ply
│   ├── scans_test
│   │   ├── [scene_id]								
│   │   │   ├── [scene_id]_vh_clean_2.ply
│   ├── scannetv2-labelsbined.tsv

声明：

以上内容都只是我个人所遇到的问题，然后陈述出来，并不一定每个人都适用。而且我也是才入门实例分割这个方向不久，关于该数据集的下载也是自己上网摸索学来的，陈述内容不一定对，各位看官请自行判断。

引用文章：

1. scannet数据集简介和下载
2. UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xe4 in position 0: ordinal not in range(128)
3. scannet数据集下载文件
4. scannet v2 数据集下载
5. Python编码错误的解决办法SyntaxError: Non-ASCII character ‘\xe5‘ in file