一个神奇的 spark 报错

写 spark 跑的时候遇到一个神奇的错误，报错如下

18/11/20 16:44:44 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() for one-way message.
org.apache.spark.SparkException: Could not find CoarseGrainedScheduler.
	at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:160)
	at org.apache.spark.rpc.netty.Dispatcher.postOneWayMessage(Dispatcher.scala:140)
	at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:655)
	at org.apache.spark.network.server.TransportRequestHandler.processOneWayMessage(TransportRequestHandler.java:208)
	at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:113)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278)
	at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:962)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:485)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:399)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:371)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
	at java.lang.Thread.run(Thread.java:745)

在 spark web ui 上的报错就更加无厘头，说

Job aborted due to stage failure: Task 194 in stage 31.0 failed 4 times, most recent failure: Lost task 194.3 in stage 31.0 (TID 24324, 10.56.83.212, executor 73): UnknownReason

重新审视了一下代码，并没有发现问题，于是采用二分大法，加上日志看，同时也放狗搜，本以为这种虚无缥缈的报错是查不到啥的，结果居然有，看到这里， https://stackoverflow.com/ques… 说

 Yeah now I know the meaning of that cryptic exception, the executor got killed because it exceeds the container threshold.  There are couple of reasons that could happen but the first culprit is to check your job or try adding more nodes/executors to your cluster. 

这里 https://blog.csdn.net/u0137092… 也说

好吧，先试试加大 executor 和内存看看，其实原来的 executor 已经够多了，200 个，但是 core 是 2，内存也是 2g，这回都搞到 4 看看

。。。

依然不行

加上打日志的逻辑，前面半段是能跑完的，看来，问题就出在后面半段了，其实这里的改动并不大，就是加了一个新的表的 rdd 的 join 逻辑而已

正在一筹莫展之际，回头仔细的翻了翻抛出来的堆栈，结果赫然发现一个 null pointer exception 在列，原来根本不是这次的改动引入的，而是之前的一个改动没有做空指针保护，人参啊。。