我正在使用python读取大量数据并将它们分成各种文件。 我正在寻找一种方法来加快我已有的代码。 进入的数字是小端32位浮点数。 我做了几次测试。
首先测试8分钟完成：
f = open('filename','rb') #file_out is a list of many open writing files 'wb' while chunk: for i in range(self.num_files): chunk = f.read(4) file_out[i].write(chunk)这是可以接受的快速，但是当我尝试添加一些操作时，事情会急剧减慢到56分钟：
file_old = [0,0,0,...,0] f = open('filename','rb') #file_out is a list of many open writing files 'wb' while chunk: for i in range(self.num_files): chunk = f.read(4) num_chunk = numpy.fromstring(chunk, dtype = numpy.float32) file_out[i].write(num_chunk-file_old[i]) file_old[i] = num_chunk我在缩短的样本上对上面的代码运行了cProfile。 结果如下：
写= 3.457
Numpy fromstring = 2.274
读= 1.370
我怎么能加快速度呢？
I am using python to read mass amounts of data and split them into various files. I am looking for a way to speed up the code that I already have. The numbers coming in are little-endian 32bit floats. I have run several tests.
First test 8 minutes to complete:
f = open('filename','rb') #file_out is a list of many open writing files 'wb' while chunk: for i in range(self.num_files): chunk = f.read(4) file_out[i].write(chunk)This was acceptably fast, but when I try to add some operations, things slow down dramatically to 56 minutes:
file_old = [0,0,0,...,0] f = open('filename','rb') #file_out is a list of many open writing files 'wb' while chunk: for i in range(self.num_files): chunk = f.read(4) num_chunk = numpy.fromstring(chunk, dtype = numpy.float32) file_out[i].write(num_chunk-file_old[i]) file_old[i] = num_chunkI ran cProfile on the above code on a shortened sample. Here are the results:
write = 3.457
Numpy fromstring = 2.274
read = 1.370
How could I speed this up?
最满意答案
我能够使用numpy.fromfile发现一种更快速的数据读取方式。 我写了一个快速的小测试脚本，如下所示：
from os.path import join import numpy import struct from time import time def main(): #Set the path name and filename folder = join("Tone_Tests","1khz_10ns_0907153323") fn = join(folder,"Channel1.raw32") #Test 1 start = time() f = open(fn,'rb') array = read_fromstring(f) f.close() print "Test fromString = ",time()-start del array #Test 2 start = time() f = open(fn,'rb') array = read_struct(f) f.close() print "Test fromStruct = ",time()-start del array #Test 3 start = time() f = open(fn,'rb') array = read_fromfile(f) f.close() print "Test fromfile = ",time()-start del array def read_fromstring(f): #Use Numpy fromstring, read each 4 bytes, convert, store in list data = [] chunk = f.read(4) while chunk: num_chunk = numpy.fromstring(chunk, dtype = 'float32') data.append(num_chunk) chunk = f.read(4) return numpy.array(data) def read_struct(f): #Same as numpy froms string but using the struct. data = [] chunk = f.read(4) while chunk: num_chunk = struct.unpack('<f',chunk) data.append(num_chunk) chunk = f.read(4) return numpy.array(data) def read_fromfile(f): return numpy.fromfile(f, dtype = 'float32', count = -1)终端的定时输出是：
Test fromString = 4.43499994278 Test fromStruct = 2.42199993134 Test fromfile = 0.00399994850159使用python -m cProfile -s time filename.py > profile.txt显示时间是：
ncalls tottime percall cumtime percall filename:lineno(function) 1 1.456 1.456 4.272 4.272 Read_Data_tester.py:42(read_fromstring) 1 1.162 1.162 2.369 2.369 Read_Data_tester.py:56(read_struct) 1 0.000 0.000 0.005 0.005 Read_Data_tester.py:70(read_fromfile)I was able to discover a much faster way of reading in the data using numpy.fromfile. I wrote a quick little test script shown below:
from os.path import join import numpy import struct from time import time def main(): #Set the path name and filename folder = join("Tone_Tests","1khz_10ns_0907153323") fn = join(folder,"Channel1.raw32") #Test 1 start = time() f = open(fn,'rb') array = read_fromstring(f) f.close() print "Test fromString = ",time()-start del array #Test 2 start = time() f = open(fn,'rb') array = read_struct(f) f.close() print "Test fromStruct = ",time()-start del array #Test 3 start = time() f = open(fn,'rb') array = read_fromfile(f) f.close() print "Test fromfile = ",time()-start del array def read_fromstring(f): #Use Numpy fromstring, read each 4 bytes, convert, store in list data = [] chunk = f.read(4) while chunk: num_chunk = numpy.fromstring(chunk, dtype = 'float32') data.append(num_chunk) chunk = f.read(4) return numpy.array(data) def read_struct(f): #Same as numpy froms string but using the struct. data = [] chunk = f.read(4) while chunk: num_chunk = struct.unpack('<f',chunk) data.append(num_chunk) chunk = f.read(4) return numpy.array(data) def read_fromfile(f): return numpy.fromfile(f, dtype = 'float32', count = -1)The timed outputs from the terminal were:
Test fromString = 4.43499994278 Test fromStruct = 2.42199993134 Test fromfile = 0.00399994850159Using python -m cProfile -s time filename.py > profile.txt shows that the times were:
ncalls tottime percall cumtime percall filename:lineno(function) 1 1.456 1.456 4.272 4.272 Read_Data_tester.py:42(read_fromstring) 1 1.162 1.162 2.369 2.369 Read_Data_tester.py:56(read_struct) 1 0.000 0.000 0.005 0.005 Read_Data_tester.py:70(read_fromfile)更多推荐
发布评论