Nova报错：OperationalError: (_mysql_exceptions.OperationalError) (1040, 'Too many connections')
Openstack版本：Liberty
系统版本：CentOS 64bit
##########################################################
1.现象
当在Dashboard上操作一段时间后，会频繁出现报错，使用命令行操作也是出现“500 未知内部错误“，经查nova-api.log发现如下报错：
2016-04-11 14:23:22.069 99072 INFO nova.osapi_compute.wsgi.server [req-96fbde09-2796-4153-8209-ae3a22f4d35d ca23866c0b774f5ca978e50beaa61a96 80fd780584f3427797db539a06c9f60c - - -] 192.168.91.7 "POST /v2/80fd780584f3427797db539a06c9f60c/servers/5cda35ed-4211-44eb-ad82-c29a9be5c657/action HTTP/1.1" status: 200 len: 317 time: 0.1082351 2016-04-11 14:23:22.178 99072 INFO nova.osapi_compute.wsgi.server [req-0a30a628-a436-4779-9cb4-71d6aa947a50 ca23866c0b774f5ca978e50beaa61a96 80fd780584f3427797db539a06c9f60c - - -] 192.168.91.7 "POST /v2/80fd780584f3427797db539a06c9f60c/servers/d04bd833-db6d-4adf-98d7-3c37f612009b/action HTTP/1.1" status: 200 len: 317 time: 0.1045909 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions [req-b69f6cac-6905-41c9-8a0c-abf733ba4178 ca23866c0b774f5ca978e50beaa61a96 80fd780584f3427797db539a06c9f60c - - -] Unexpected exception in API method 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/remote_consoles.py", line 53, in get_vnc_console 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions instance = common.get_instance(self.compute_api, context, id) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/common.py", line 510, in get_instance 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions expected_attrs=expected_attrs) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2026, in get 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions context, instance_id, expected_attrs=expected_attrs) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 171, in wrapper 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 372, in get_by_uuid 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions use_slave=use_slave) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 645, in instance_get_by_uuid 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions columns_to_join, use_slave=use_slave) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 216, in wrapper 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1713, in instance_get_by_uuid 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions columns_to_join=columns_to_join, use_slave=use_slave) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1720, in _instance_get_by_uuid 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions use_slave=use_slave).\ 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1751, in _build_instance_get 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions project_only=True, use_slave=use_slave).\ 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 275, in model_query 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions session = get_session(use_slave=use_slave) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 182, in get_session 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions facade = _create_facade_lazily(_MAIN_FACADE, conf_group) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 164, in _create_facade_lazily 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions _ENGINE_FACADE[facade] = _create_facade(conf_group) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 156, in _create_facade 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions retry_interval=conf_group.retry_interval) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 943, in __init__ 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions slave_connection=slave_connection) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 338, in _start 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions engine_args, maker_args) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 362, in _setup_for_connection 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions sql_connection=sql_connection, **engine_kwargs) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 152, in create_engine 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions test_conn = _test_connection(engine, max_retries, retry_interval) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 326, in _test_connection 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions return engine.connect() 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2018, in connect 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions return self._connection_cls(self, **kwargs) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 72, in __init__ 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions if connection is not None else engine.raw_connection() 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2104, in raw_connection 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions self.pool.unique_connection, _connection) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions e, dialect, self) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions util.raise_from_cause(newraise, exc_info) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions reraise(type(exception), exception, tb=exc_tb) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions return fn() 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 318, in unique_connection 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions return _ConnectionFairy._checkout(self) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions fairy = _ConnectionRecord.checkout(pool) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 480, in checkout 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions rec = pool._do_get() 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1060, in _do_get 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions self._dec_overflow() 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__ 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions compat.reraise(exc_type, exc_value, exc_tb) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1057, in _do_get 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions return self._create_connection() 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 323, in _create_connection 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions return _ConnectionRecord(self) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 449, in __init__ 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions self.connection = self.__connect() 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 607, in __connect 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions connection = self.__pool._invoke_creator(self) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions return dialect.connect(*cargs, **cparams) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions return self.dbapi.connect(*cargs, **cparams) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 81, in Connect 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions return Connection(*args, **kwargs) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 187, in __init__ 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions super(Connection, self).__init__(*args, **kwargs2) 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions OperationalError: (_mysql_exceptions.OperationalError) (1040, 'Too many connections') 2016-04-11 14:23:22.628 99106 ERROR nova.api.openstack.extensions 2016-04-11 14:23:22.639 99106 INFO nova.api.openstack.wsgi [req-b69f6cac-6905-41c9-8a0c-abf733ba4178 ca23866c0b774f5ca978e50beaa61a96 80fd780584f3427797db539a06c9f60c - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'sqlalchemy.exc.OperationalError'>####################################################################
2.问题解决
根据报错提示，显然是数据库超最大连接了，调整mariadb数据库最大连接即可(nova连接数据库采用的是永久连接，永久连接数大约设在200~300之间，这块源码我还没有细看，大家可以根据需要自行调整数据库最大连接数大于300即可)。
3.注意
1.CentOS7系统调整数据库最大连接数和之前有所不同，具体可参考：