读取pySpark中的文件范围

本文介绍了读取pySpark中的文件范围的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！ 问题描述

我需要读取pySpark中的连续文件.以下对我有用.

I need to read contiguous files in pySpark. The following works for me.

from pyspark.sql import SQLContext file = "events.parquet/exportDay=2015090[1-7]" df = sqlContext.read.load(file)

我如何读取文件8-14?

How do I read files 8-14?

推荐答案

使用花括号.

file ="events.parquet/exportDay = 201509 {08,09,10,11,12,13,14}"

file = "events.parquet/exportDay=201509{08,09,10,11,12,13,14}"

这是关于堆栈溢出的类似问题: Pyspark使用正则表达式glob 选择文件子集.他们建议要么使用花括号，要么执行多次读取然后合并对象(无论是RDD还是数据帧，还是应该有某种方式).

Here's a similar question on stack overflow: Pyspark select subset of files using regex glob. They suggest either using curly braces, OR performing multiple reads and then unioning the objects (whether they are RDDs or data frames or whatever, there should be some way).