当前位置: 动力学知识库 > 问答 > 编程问答 >

scala - Spark RDD persistings every time I call it

问题描述:

I have the following code

val dfTransformationToRDD = findLastActiveUsersByEvents(df).withColumn(RedshiftEventEnum.WEIGHT, FunctionsUDFUtil.weightEventUDF(df("type"))).rdd

if(dfTransformationToRDD.getStorageLevel == StorageLevel.NONE){

println("im persisting ")

dfTransformationToRDD.persist(StorageLevel.MEMORY_AND_DISK)

}

I validate rather this RDD has storageLevel so I can know its cached but every time I call this function it always gets into the if block I'm i doing something wrong or am I misunderstanding this?

分享给朋友:
您可能感兴趣的文章:
随机阅读: