快速读取非常大的表作为数据框

使用 vroom 来自 tidyverse 包 vroom 用于将数据从 csv/tab-delimited 文件直接导入 R tibble.请参阅赫克托的回答.

Using vroom from the tidyverse package vroom for importing data from csv/tab-delimited files directly into an R tibble. See Hector's answer.

使用 fread 在 data.table 用于导入将数据从 csv/tab 分隔的文件直接导入 R.请参阅 mnel 的回答.

使用 read_table 在 readr(2015 年 4 月在 CRAN 上).这很像上面的 fread.链接中的 readme 解释了这两个函数之间的区别(readr 目前声称比 data.table::fread).

Using read_table in readr (on CRAN from April 2015). This works much like fread above. The readme in the link explains the difference between the two functions (readr currently claims to be "1.5-2x slower" than data.table::fread).

read.csv.raw 来自 iotools 提供了第三个选项来快速读取 CSV 文件.

read.csv.raw from iotools provides a third option for quickly reading CSV files.

尝试在数据库而不是平面文件中存储尽可能多的数据.(作为更好的永久存储介质，数据以二进制格式传入和传出 R，速度更快.)read.csv.sql 在 sqldf 包，如 JD 中所述Long 的回答，将数据导入临时 SQLite 数据库，然后将其读入 R.另请参阅:RODBC 包，以及 DBI 包 页面.MonetDB.R 给出您是一种伪装成数据框但实际上是底层的 MonetDB 的数据类型，从而提高了性能.使用其monetdb.read.csv 功能.dplyr 允许你直接工作数据存储在多种类型的数据库中.

Trying to store as much data as you can in databases rather than flat files. (As well as being a better permanent storage medium, data is passed to and from R in a binary format, which is faster.) read.csv.sql in the sqldf package, as described in JD Long's answer, imports data into a temporary SQLite database and then reads it into R. See also: the RODBC package, and the reverse depends section of the DBI package page. MonetDB.R gives you a data type that pretends to be a data frame but is really a MonetDB underneath, increasing performance. Import data with its monetdb.read.csv function. dplyr allows you to work directly with data stored in several types of database.

以二进制格式存储数据对于提高性能也很有用.使用 saveRDS/readRDS(见下文)，h5 或 rhdf5 用于 HDF5 格式的包，或 write_fst/read_fst-project/package=fst" rel="noreferrer">fst 包.

Storing data in binary formats can also be useful for improving performance. Use saveRDS/readRDS (see below), the h5 or rhdf5 packages for HDF5 format, or write_fst/read_fst from the fst package.

原答案

无论您使用 read.table 还是 scan，都可以尝试一些简单的方法.

There are a couple of simple things to try, whether you use read.table or scan.

设置nrows=数据中的记录数(nmax in scan).

确保 comment.char="" 关闭注释解释.

Make sure that comment.char="" to turn off interpretation of comments.

使用read.table中的colClasses明确定义每一列的类.

Explicitly define the classes of each column using colClasses in read.table.

设置 multi.line=FALSE 也可以提高扫描性能.

Setting multi.line=FALSE may also improve performance in scan.

如果这些都不起作用，那么使用 分析包之一 以确定哪些线路正在减慢速度.或许你可以根据结果写出一个read.table的精简版.

If none of these thing work, then use one of the profiling packages to determine which lines are slowing things down. Perhaps you can write a cut down version of read.table based on the results.

另一种选择是在将数据读入 R 之前对其进行过滤.

The other alternative is filtering your data before you read it into R.

或者，如果问题是你必须定期读取它，那么使用这些方法一次读取数据，然后将数据帧保存为带有save saveRDS，然后下次您可以使用 load readRDS.

Or, if the problem is that you have to read it in regularly, then use these methods to read the data in once, then save the data frame as a binary blob with save saveRDS, then next time you can retrieve it faster with load readRDS.