记录一次由C3P0连接池引起的事故

事故描述

从中午11点开始服务报大量的超时，手动调用接口也不通。

事故解决

首先借用公司的CAT和PINPOIT等，查看有什么异常，在CAT上发现有大量insert异常，错误信息如下：

	java.lang.reflect.InvocationTargetException 
at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown Source) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:498) 
at org.apache.ibatis.plugin.Invocation.proceed(Invocation.java:49) 
at com.tiefan.cat.mybatis.CatMybatisPlugin.intercept(CatMybatisPlugin.java:90) 
at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:61) 
at com.sun.proxy.$Proxy120.update(Unknown Source) 
at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:170) 
at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:157) 
at sun.reflect.GeneratedMethodAccessor144.invoke(Unknown Source) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:498) 
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:358) 
at com.sun.proxy.$Proxy67.insert(Unknown Source) 
at org.mybatis.spring.SqlSessionTemplate.insert(SqlSessionTemplate.java:240) 
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:52) 
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:53) 
at com.sun.proxy.$Proxy100.insertSelective(Unknown Source) 
at com.tiefan.btc.his.Schedule.SystemLogTiming.saveSystemLog(SystemLogTiming.java:162) 
at com.tiefan.btc.his.Schedule.SystemLogTiming.access$000(SystemLogTiming.java:48) 
at com.tiefan.btc.his.Schedule.SystemLogTiming$2.perform(SystemLogTiming.java:98) 
at com.tiefan.btc.his.Schedule.SystemLogTiming$2.perform(SystemLogTiming.java:95) 
at com.tiefan.btc.his.service.parallel.Parallel$1.call(Parallel.java:33) 
at com.tiefan.btc.his.service.parallel.Parallel$1.call(Parallel.java:31) 
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745) 
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: An attempt by a client to checkout a Connection has timed out. 
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) 
at org.mybatis.spring.transaction.SpringManagedTransaction.openConnection(SpringManagedTransaction.java:81) 
at org.mybatis.spring.transaction.SpringManagedTransaction.getConnection(SpringManagedTransaction.java:67) 
at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:315) 
at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:75) 
at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:48) 
at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:115) 
at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:75) 
... 28 more 
Caused by: java.sql.SQLException: An attempt by a client to checkout a Connection has timed out. 
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118) 
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:77) 
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:690) 
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140) 
at sun.reflect.GeneratedMethodAccessor174.invoke(Unknown Source) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:498) 
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) 
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) 
at com.sun.proxy.$Proxy61.getConnection(Unknown Source) 
at com.tiefan.btc.his.base.db.SysRwDataSoure.getConnection(SysRwDataSoure.java:33) 
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) 
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) 
... 35 more 
Caused by: com.mchange.v2.resourcepool.TimeoutException: A client timed out while waiting to acquire a resource from com.mchange.v2.resourcepool.BasicResourcePool@258557ff -- timeout at awaitAvailable() 
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1467) 
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644) 
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554) 
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758) 
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685) 
... 45 more 

可以看到最后的不能获得连接池和不能从资源池获得资源的异常，进而猜测是因为数据库链接阻塞，导致线程积压，拖垮了服务。
然后dump出服务器的镜像，如下：

可以看到这个char[]是数量最多也是占用空间最多的，点开看看，

可以看到是一个一个我要插入数据库的数据，由此确认了事故的根源。

注

这里由于dump比较大，用java自带的分析工具jvisualVM打不开，采用了visualVM分析的。