有没有一种方法可以从HDF5数据集中删除行?

本文介绍了有没有一种方法可以从HDF5数据集中删除行?的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我创建了一个约有210万个实例的H5PY数据集.问题是我已经填满了除最后一行以外的所有行.我想删除最后一行，但不确定这样做是否可行或安全.

I have created a H5PY dataset, with around 2.1 million instances. The issue is I have filled all the rows apart from the last one. I want to remove the last row but unsure if it is feasible or safe to do.

这是如何创建数据集的摘录:

This is a snippet of how the dataset is created:

shape = (dataset_length, args.batch_size, 2048, 1, 1) with h5py.File(path, mode='a') as hdf5_file: array_40 = hdf5_file.create_dataset( f'{phase}_40x_arrays', shape, maxshape=(None, args.batch_size, 2048, 1, 1) # either new or checkpointed file exists # load file and create references to exisitng h5 datasets with h5py.File(path, mode='r+') as hdf5_file: array_40 = hdf5_file[f'{phase}_40x_arrays'] for i, (inputs40x, labels) in enumerate(dataloaders_dict): inputs40x = inputs40x.to(device) x40 = resnet(inputs40x) array_40[batch_idx, ...] = x40.cpu() hdf5_file.flush()

我不确定是否需要将所有实例复制到新数据集中.我尝试调整大小，但这没用...

I'm not really sure if I need to copy all instances to a new dataset. I tried resizing, but that didn't work...

干杯

推荐答案

这是一个非常简单的示例，用于演示一个数据集的 dataset.resize().

Here is a very simple example to demonstrate dataset.resize() for one dataset.

import numpy as np import h5py arr = np.random.rand(100).reshape(20,5) with h5py.File('SO_61487687.h5', mode='a') as h5f: h5f.create_dataset('array1', data=arr, maxshape=(None, 5) ) with h5py.File('SO_61487687.h5', mode='r+') as h5f: print ('Before:', h5f['array1'].shape) h5f['array1'].resize(10,axis=0) print ('After:', h5f['array1'].shape) h5f.flush()