虹墨空间站

2024-08-21发表Tech / BackEnd13 分钟读完 (大约1879个字)

问题现象

周末生产环境出现了一个奇怪的问题，部署在 k8s 容器中的 SpringBoot 应用连接到数据库的交换机出现了故障，交换机的故障恢复后，查询数据库的接口还是无法提供服务，试了很多次，日志中报出的异常依然是 Connection is not available，难道是网络还是没回复？

网络可能不通？那我们就 telnet 一下，网络是没问题的。最后我们使用 netstat 看一下和数据库的连接状态，状态是 ‌ESTABLISHED，这个状态表示已经成功建立连接，但是连接只有一个，与我们配置的最小活跃连接数 10 的数量不相符，此时为了业务能尽快恢复，只好重启了应用服务。

事后，我们又查看了所有通过这个故障交换机连接数据库的应用，有的没有发生问题，有的没有经过重启，分别在 20 分钟和 120 分钟之后自动恢复了，于是我们开始在测试环境想办法复现这个问题并寻找解决方案。

2020-06-04发表Tech / BackEnd7 分钟读完 (大约1065个字)

JDBC 连接 Azure Database for MySQL 报错 SSL peer shut down incorrectly 的解决

这个问题找了一天的原因，如果你也遇到了这个问题，希望能帮到你。

完整错误信息：

点击展开 >folded

03-Jun-2020 07:52:22.772 SEVERE [main] org.apache.tomcat.jdbc.pool.ConnectionPool.init Unable to create initial connections of pool.
	com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet successfully received from the server was 384 milliseconds ago.  The last packet sent successfully to the server was 376 milliseconds ago.
		at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
		at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
		at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
		at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
		at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
		at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1117)
		at com.mysql.jdbc.ExportControlled.transformSocketToSSLSocket(ExportControlled.java:105)
		at com.mysql.jdbc.MysqlIO.negotiateSSLConnection(MysqlIO.java:5126)
		at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1666)
		at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1244)
		at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2397)
		at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2430)
		at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2215)
		at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:813)
		at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
		at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
		at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
		at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
		at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
		at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
		at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:399)
		at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:334)
		at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:319)
		at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:212)
		at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:744)
		at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:676)
		at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:483)
		at org.apache.tomcat.jdbc.pool.ConnectionPool.<init>(ConnectionPool.java:154)
		at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:118)
		at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:107)
		at org.apache.tomcat.jdbc.pool.DataSourceFactory.createDataSource(DataSourceFactory.java:560)
		at org.apache.tomcat.jdbc.pool.DataSourceFactory.getObjectInstance(DataSourceFactory.java:244)
		at org.apache.naming.factory.FactoryBase.getObjectInstance(FactoryBase.java:96)
		at java.naming/javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:325)
		at org.apache.naming.NamingContext.lookup(NamingContext.java:857)
		at org.apache.naming.NamingContext.lookup(NamingContext.java:160)
		at org.apache.naming.NamingContext.lookup(NamingContext.java:843)
		at org.apache.naming.NamingContext.lookup(NamingContext.java:174)
		at org.apache.catalina.core.NamingContextListener.addResource(NamingContextListener.java:1017)
		at org.apache.catalina.core.NamingContextListener.createNamingContext(NamingContextListener.java:557)
		at org.apache.catalina.core.NamingContextListener.lifecycleEvent(NamingContextListener.java:253)
		at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
		at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:924)
		at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
		at org.apache.catalina.startup.Catalina.start(Catalina.java:633)
		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:566)
		at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:344)
		at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:475)
	Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
		at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1322)
		at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1160)
		at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063)
		at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
		at com.mysql.jdbc.ExportControlled.transformSocketToSSLSocket(ExportControlled.java:90)
		... 44 more
		Suppressed: java.net.SocketException: Broken pipe (Write failed)
			at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
			at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
			at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
			at java.base/sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81)
			at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:351)
			at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263)
			at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:405)
			... 45 more
	Caused by: java.io.EOFException: SSL peer shut down incorrectly
		at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:167)
		at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:108)
		at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152)
		... 47 more

提取关键信息：

1 2	Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake Caused by: java.io.EOFException: SSL peer shut down incorrectly

得知 SSL 握手失败，于是我找了 Azure 的示例代码（https://docs.azure.cn/zh-cn/mysql/connect-java）来测，也是同样的问题。