org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow...

Spark异常:Kryo serialization failed: Buffer overflow.






1、问题描述

SparkSQL任务报错如下:

org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: xxx. To avoid this, increase spark.kryoserializer.buffer.max value.

根据报错信息可知,Spark在Kryo序列化时发生缓存溢出

2、问题解决

根据提示信息,可以适当增加spark.kryoserializer.buffer.max的值,例如256m或512m,该参数的默认值为128m

更多信息参考:https://study.sf.163.com/documents/read/service_support/dsc-p-a-0067


你可能感兴趣的:(#,Spark,spark,大数据)