Total size of serialized results of 20 tasks (1088.8 MB) is bigger than spark.driver.maxResultSize (

Total size of serialized results of 20 tasks (1088.8 MB) is bigger thanspark.driver.maxResultSize (1024.0 MB)

pypark 2.0 (spark python)  执行中出现上述错误,解决方案:

SparkContext.setSystemProperty('spark.driver.maxResultSize', '10g')



你可能感兴趣的:(Total size of serialized results of 20 tasks (1088.8 MB) is bigger than spark.driver.maxResultSize ()