我有一个从cassandra中加载的数据集.加载此数据集后,我将从cassandra中删除一些项,但是我希望我的数据集作为下一个计算的第一项.我已经使用 persist(DISK_ONLY)来解决它,但这似乎是最大的努力.我该如何强制火花以避免重新计算?
I've a dataset which is loaded from cassandra in spark. After loading this dataset, I will remove some of the items from cassandra, but I want my dataset being as first for the next computation. I've used persist(DISK_ONLY) to solve it, but it seems to best effort. How can I force spark to avoid re-computation?
示例:
val dataset:Dataset[Int] = ??? // something from cassandra dataset.persist(StorageLevel.DISK_ONLY) // it's best effort dataset.count // = 2n dataset.persist(_ % 2 == 0).remove // remove from cassandra data.count // = n => I need orginal dataset here 推荐答案Spark cache 不能以这种方式使用.这是一种优化,即使使用最保守的 StorageLevels ( DISK_ONLY_2 ),在工作人员失败或退役的情况下,数据也可能会丢失并重新计算.
Spark cache is not intended to be used this way. It is an optimization, and even with the most conservative StorageLevels (DISK_ONLY_2), data can be lost and recomputed in case of worker failure or decommissioning.
检查点到可靠的文件系统可能是一个更好的选择,但是我怀疑可能存在某些边界情况,这可能会导致数据丢失.
Checkpoint to a reliable file system might be a better option, but I suspect there might be some border cases, which can result in the data loss.
请确保正确性我强烈建议至少将中间数据写入持久性存储,例如分布式文件系统,然后再读回:
Yo ensure correctness I would strongly recommend at least writing intermediate data to a persistent storage, like distributed file system, and reading it back:
dataset.write.format(...).save("persisted/location") ... // Remove data from the source spark.read.format(...).load("persisted/location") //reading the same again更多推荐
如何强制执行火花以避免数据集重新计算?
发布评论