自定义定界符CSV阅读器Spark

本文介绍了自定义定界符CSV阅读器Spark的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我想使用Apache Spark读取具有以下结构的文件.

I would like to read in a file with the following structure with Apache Spark.

628344092\t20070220\t200702\t2007\t2007.1370

定界符为\ t.在使用spark.read.csv()时如何实现呢?

The delimiter is \t. How can I implement this while using spark.read.csv()?

csv太大了，无法使用熊猫，因为它需要很长时间才能读取此文件.是否有某种类似于

The csv is much too big to use pandas because it takes ages to read this file. Is there some way which works similar to

pandas.read_csv(file, sep = '\t')

非常感谢！

推荐答案

使用spark.read.option("delimiter", "\t").csv(file)或sep代替delimiter.

如果字面上是\t，而不是制表符特殊字符，请使用双精度\:spark.read.option("delimiter", "\\t").csv(file)

If it's literally \t, not tab special character, use double \: spark.read.option("delimiter", "\\t").csv(file)