我使用读1 GB的文件的 FREAD 在C.我读在1MB块的文件,使用以下循环:
I am reading a 1 GB file using fread in C. I am reading the file in 1MB chunks, using the following loop:
FILE *fp; fp = fopen(filename, "rb"); unsigned char* buf; buf = malloc(CHUNK_SIZE); for(i = 0; i < NUMBER_OF_CHUNKS; ++i) { fread(buf, CHUNK_SIZE, 1, fp); //Do something with contents of buffer } fclose(fp);阅读这种方式,大约需要2秒钟的文件。
Reading the file this way takes ~2 seconds.
不过,我决定,我想分配一个大的缓冲整个文件的内容,而不是在每次迭代移动缓冲区指针在 FREAD 函数里,像这样的:
However, I decided that I wanted to allocate one big buffer for the contents of the whole file instead and "move the buffer pointer" inside the fread function at each iteration, like this:
FILE *fp; fp = fopen(filename, "rb"); unsigned char* buf; buf = malloc(CHUNK_SIZE * NUMBER_OF_CHUNKS); for(i = 0; i < NUMBER_OF_CHUNKS; ++i) { fread(&buf[i*CHUNK_SIZE], CHUNK_SIZE, 1, fp); } fclose(fp);这会减慢阅读显著,现在大约需要〜40秒。
This slows down the reading significantly, it now takes about ~40 seconds.
我的问题是:
该文件包含字母数字字符的单行。
The file consists of a single line of alphanumeric characters.
我想在第二方式读取它,这样我可以有其他线程访问那些已经读入缓冲器中的数据块,而读取线程继续填充缓冲区的其余部分。
I want to read it in the second way, so that I can have other threads access the chunks in the buffer that are already read, while the reading thread continues filling the rest of the buffer.
感谢您!
推荐答案这是可能的,你是你的机器上运行内存。 1千兆是相当多的内存来分配。您的操作系统我有交换一些数据到磁盘上,这将导致经济放缓幅度的秩序。
It's possible that you are running out of memory on your machine. A gigabyte is rather a lot of memory to allocate. Your OS my have to swap some of the data to disk, which will cause an order of magnitude slowdown.
您可以考虑单独分配每个块,并释放他们时,他们与完成。这样,你的程序的总内存使用量由工作集为界,而不是整个文件。
You could consider allocating each chunk individually, and freeing them when they are done with. This way the total memory usage of your program is bounded by the working set, rather than the entire file.
更多推荐
为什么移动缓冲区指针减慢FREAD(C编程语言)?
发布评论