CentOS7上で動いているMariaDB 5.5サーバをMariaDB 10.6にアップデートして死にかけた。
データベースのアップデート自体は問題なかったのだが、アプリケーション側で問題発生。
/var/log/messages に下記の様なメッセージが多発
Jul 13 10:50:11 ホスト名 kernel: dict[23339]: segfault at a8 ip 00007f99c6fa9fa1 sp 00007ffc4641fe50 error 4 in libmysqlclient.so.18.0.0[7f99c6f4b000+320000]
Jul 13 10:50:11 ホスト名 mariadbd: 2021-07-13 10:50:11 7 [Warning] Aborted connection 7 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
Jul 13 10:50:11 ホスト名 kernel: dict[23340]: segfault at a8 ip 00007f3950ee4fa1 sp 00007ffe8bec4f70 error 4 in libmysqlclient.so.18.0.0[7f3950e86000+320000]
Jul 13 10:50:11 ホスト名 kernel: dict[23341]: segfault at a8 ip 00007f89b3d8efa1 sp 00007ffed464d830 error 4 in libmysqlclient.so.18.0.0[7f89b3d30000+320000]
Jul 13 10:50:11 ホスト名 mariadbd: 2021-07-13 10:50:11 8 [Warning] Aborted connection 8 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
Jul 13 10:50:33 ホスト名 kernel: auth[23362]: segfault at a8 ip 00007f197609dfa1 sp 00007ffe82dc98a0 error 4 in libmysqlclient.so.18.0.0[7f197603f000+320000]
authとdictでlibmysqlclient.so.18.0.0 のsegmentation faultが発生しているという・・・
で・・このauthとdictというのはdovecot/authとdovecot/dictで、そこで何か起こっているようだ
/var/log/dovecot/dovecot.log を見てみる
Jul 13 10:50:05 ホスト名1 dovecot: dict(23024): Error: mysql(127.0.0.1): Connect failed to database (vmail): Can't connect to MySQL s
erver on '127.0.0.1' (111 "Connection refused") - waiting for 5 seconds before retry
Jul 13 10:50:05 ホスト名1 dovecot: dict(23024): Error: mysql(127.0.0.1): Connect failed to database (vmail): Can't connect to MySQL s
erver on '127.0.0.1' (111 "Connection refused") - waiting for 5 seconds before retry
Jul 13 10:50:10 ホスト名1 dovecot: dict: Error: #007Character set 'utf8' is not a compiled character set and is not specified in the '/usr/share/mysql/charsets/Index.xml' file
Jul 13 10:50:10 ホスト名1 dovecot: dict(23024): Error: mysql(127.0.0.1): Connect failed to database (vmail): Can't initialize character set utf8 (path: /usr/share/mysql/charsets/) - waiting for 25 seconds before retry
Jul 13 10:50:10 ホスト名1 dovecot: dict: Error: #007Character set 'utf8' is not a compiled character set and is not specified in the '/usr/share/mysql/charsets/Index.xml' file
Jul 13 10:50:10 ホスト名1 dovecot: dict(23024): Error: mysql(127.0.0.1): Connect failed to database (vmail): Can't initialize character set utf8 (path: /usr/share/mysql/charsets/) - waiting for 25 seconds before retry
Jul 13 10:50:11 ホスト名1 dovecot: dict: Error: #007Character set 'utf8' is not a compiled character set and is not specified in the '/usr/share/mysql/charsets/Index.xml' file
Jul 13 10:50:11 ホスト名1 dovecot: dict(23338): Error: mysql(127.0.0.1): Connect failed to database (vmail): Can't initialize character set utf8 (path: /usr/share/mysql/charsets/) - waiting for 1 seconds before retry
mariadb 10.6で文字コード utf8の指定ができないというエラーなようだ。
/usr/share/mysql/charsets/ に utf8の定義があるか確認してみる
# ls -l /usr/share/mysql/charsets/
合計 236
-rw-r--r-- 1 root root 23531 7月 5 19:47 Index.xml
-rw-r--r-- 1 root root 1749 7月 5 19:47 README
-rw-r--r-- 1 root root 5526 7月 5 19:47 armscii8.xml
-rw-r--r-- 1 root root 5512 7月 5 19:47 ascii.xml
-rw-r--r-- 1 root root 8241 7月 5 19:47 cp1250.xml
-rw-r--r-- 1 root root 8365 7月 5 19:47 cp1251.xml
-rw-r--r-- 1 root root 5569 7月 5 19:47 cp1256.xml
-rw-r--r-- 1 root root 8902 7月 5 19:47 cp1257.xml
-rw-r--r-- 1 root root 5506 7月 5 19:47 cp850.xml
-rw-r--r-- 1 root root 5528 7月 5 19:47 cp852.xml
-rw-r--r-- 1 root root 5613 7月 5 19:47 cp866.xml
-rw-r--r-- 1 root root 6529 7月 5 19:47 dec8.xml
-rw-r--r-- 1 root root 5516 7月 5 19:47 geostd8.xml
-rw-r--r-- 1 root root 5728 7月 5 19:47 greek.xml
-rw-r--r-- 1 root root 5517 7月 5 19:47 hebrew.xml
-rw-r--r-- 1 root root 5502 7月 5 19:47 hp8.xml
-rw-r--r-- 1 root root 5529 7月 5 19:47 keybcs2.xml
-rw-r--r-- 1 root root 5510 7月 5 19:47 koi8r.xml
-rw-r--r-- 1 root root 6532 7月 5 19:47 koi8u.xml
-rw-r--r-- 1 root root 9816 7月 5 19:47 latin1.xml
-rw-r--r-- 1 root root 7238 7月 5 19:47 latin2.xml
-rw-r--r-- 1 root root 5515 7月 5 19:47 latin5.xml
-rw-r--r-- 1 root root 7438 7月 5 19:47 latin7.xml
-rw-r--r-- 1 root root 8047 7月 5 19:47 macce.xml
-rw-r--r-- 1 root root 8058 7月 5 19:47 macroman.xml
-rw-r--r-- 1 root root 6530 7月 5 19:47 swe7.xml
# grep utf8 /usr/share/mysql/charsets/Index.xml
<charset name="utf8mb3">
<collation name="utf8mb3_general_ci" id="33">
<collation name="utf8mb3_bin" id="83">
#
なんと無い。
/usr/share/mysql/charsets/Index.xml のutf8関連記述を見てみる
<charset name="utf8mb3">
<family>Unicode</family>
<description>UTF-8 Unicode</description>
<alias>utf-8</alias>
<collation name="utf8mb3_general_ci" id="33">
<flag>primary</flag>
<flag>compiled</flag>
</collation>
<collation name="utf8mb3_bin" id="83">
<flag>binary</flag>
<flag>compiled</flag>
</collation>
</charset>
おやぁ？？？
以前mariadb 10.5にアップデートしたサーバの/usr/share/mysql/charsets/Index.xmlの該当箇所と比較すると、いままでは utf8だったエントリが utf8mb3 に置き換わっているようだ。
<charset name="utf8">
<family>Unicode</family>
<description>UTF-8 Unicode</description>
<alias>utf-8</alias>
<collation name="utf8_general_ci" id="33">
<flag>primary</flag>
<flag>compiled</flag>
</collation>
<collation name="utf8_bin" id="83">
<flag>binary</flag>
<flag>compiled</flag>
</collation>
</charset>
で・・・ /etc/my.cnf の設定で言語に関する設定は下記の様になっていた
[mysqld]
＜略＞
# character-set-server に関する値は未設定
＜略＞
[client]
default-character-set=utf8
クライアントからのアクセス時のデフォルトがutf8と設定されていた。
そこで/etc/my.cnf の文字コード指定を utf8mb3 に変更
[mysqld]
＜略＞
character-set-server=utf8mb3
＜略＞
[client]
default-character-set=utf8mb3
これで接続できない、という事象はなくなった。
なくなったが、/var/log/messages には下記のメッセージが出続けている。
Jul 13 13:10:01 ホスト名 mariadbd: 2021-07-13 13:10:01 925 [Warning] Aborted connection 925 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
Jul 13 13:10:02 ホスト名 mariadbd: 2021-07-13 13:10:02 934 [Warning] Aborted connection 934 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
Jul 13 13:10:06 ホスト名 mariadbd: 2021-07-13 13:10:06 935 [Warning] Aborted connection 935 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
Jul 13 13:10:06 ホスト名 mariadbd: 2021-07-13 13:10:06 936 [Warning] Aborted connection 936 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
Jul 13 13:10:22 ホスト名 mariadbd: 2021-07-13 13:10:22 811 [Warning] Aborted connection 811 to db: 'amavisd' user: 'amavisd' host: 'localhost' (Got timeout reading communication packets)
Jul 13 13:11:01 ホスト名 mariadbd: 2021-07-13 13:11:01 786 [Warning] Aborted connection 786 to db: 'amavisd' user: 'amavisd' host: 'localhost' (Got timeout reading communication packets)
Jul 13 13:12:26 ホスト名 mariadbd: 2021-07-13 13:12:26 839 [Warning] Aborted connection 839 to db: 'amavisd' user: 'amavisd' host: 'localhost' (Got timeout reading communication packets)
Jul 13 13:12:26 ホスト名 mariadbd: 2021-07-13 13:12:26 828 [Warning] Aborted connection 828 to db: 'amavisd' user: 'amavisd' host: 'localhost' (Got timeout reading communication packets)
iRedMailフォーラム「mysqld [Warning] Aborted connection」によると、ログレベル下げろ、とのこと
mysqlを起動しようとしたら問題が・・・
# mysql -u root
ERROR 2019 (00000): Can't initialize character set utf8mb3 (path: compiled_in)
#
とりあえず /etc/my.cnf の最後に下記を追加
[mysql]
default-character-set=utf8
警告はでるのもの操作は可能（なお、default-character-set=utf8mb3 だと起動できなかった)
# mysql -u root
mysql: Warning: Charset id '33' csname 'utf8mb3' trying to replace existing csname 'utf8'
mysql: Warning: Charset id '83' csname 'utf8mb3' trying to replace existing csname 'utf8'
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 1103
Server version: 10.6.3-MariaDB-log MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]>
起動したのでフォーラムにあった設定を入れる。
MariaDB [(none)]> show global variables like '%log_warnings%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| log_warnings | 2 |
+---------------+-------+
1 row in set (0.002 sec)
MariaDB [(none)]> set global log_warnings=1;
Query OK, 0 rows affected (0.000 sec)
MariaDB [(none)]> show global variables like '%log_warnings%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| log_warnings | 1 |
+---------------+-------+
1 row in set (0.002 sec)
MariaDB [(none)]> quit
Bye
#
しばらく放置・・・
Jul 13 13:40:01 ホスト名 mariadbd: 2021-07-13 13:40:01 313 [Warning] Aborted connection 313 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
Jul 13 13:40:02 ホスト名 mariadbd: 2021-07-13 13:40:02 314 [Warning] Aborted connection 314 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
Jul 13 13:40:04 ホスト名 mariadbd: 2021-07-13 13:40:04 315 [Warning] Aborted connection 315 to db: 'unconnected' user: 'unauthenticated' host: 'localhost' (This connection closed normally without authentication)
Jul 13 13:42:53 ホスト名 mariadbd: 2021-07-13 13:42:53 359 [Warning] Aborted connection 359 to db: 'vmail' user: 'vmail' host: 'localhost' (Got an error reading communication packets)
Jul 13 13:42:53 ホスト名 mariadbd: 2021-07-13 13:42:53 352 [Warning] Aborted connection 352 to db: 'vmail' user: 'vmail' host: 'localhost' (Got an error reading communication packets)
Jul 13 13:42:53 ホスト名 mariadbd: 2021-07-13 13:42:53 357 [Warning] Aborted connection 357 to db: 'vmail' user: 'vmail' host: 'localhost' (Got an error reading communication packets)
Jul 13 13:42:53 ホスト名 mariadbd: 2021-07-13 13:42:53 351 [Warning] Aborted connection 351 to db: 'vmail' user: 'vmail' host: 'localhost' (Got an error reading communication packets)
うーん？
あれ？反映されてない
# mysql -u root
mysql: Warning: Charset id '33' csname 'utf8mb3' trying to replace existing csname 'utf8'
mysql: Warning: Charset id '83' csname 'utf8mb3' trying to replace existing csname 'utf8'
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 474
Server version: 10.6.3-MariaDB-log MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> show global variables like '%log_warnings%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| log_warnings | 2 |
+---------------+-------+
1 row in set (0.002 sec)
MariaDB [(none)]> set global log_warnings=1;
Query OK, 0 rows aff
MariaDB [(none)]>
なんだろ？とりあえず再設定・・・
今度は大丈夫か？
…2時間経過
大丈夫そうだな
2021/07/15追記
iredmailでは/usr/local/bin/fail2ban_banned_dbがcronにより毎分実行されているのだが、これがmysqlコマンドを使ってデータベース操作をしているようで、下記のメールが届くようになってしまった・・・
mysql: Warning: Charset id '33' csname 'utf8mb3' trying to replace existing csname 'utf8'
mysql: Warning: Charset id '83' csname 'utf8mb3' trying to replace existing csname 'utf8'
mysqlコマンドを実行した際のメッセージをなんとしてでも消さなければならないようだ。
utf8がutf8mb3になった変更について「MariaDB Server/MDEV-8334/Rename utf8 to utf8mb3」
MariaDBの「OLD_MODE」という設定
# mysql -u root
mysql: Warning: Charset id '33' csname 'utf8mb3' trying to replace existing csname 'utf8'
mysql: Warning: Charset id '83' csname 'utf8mb3' trying to replace existing csname 'utf8'
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 38510
Server version: 10.6.3-MariaDB-log MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> SELECT REPLACE(@@OLD_MODE, ',', '\n');
+--------------------------------+
| REPLACE(@@OLD_MODE, ',', '\n') |
+--------------------------------+
| UTF8_IS_UTF8MB3 |
+--------------------------------+
1 row in set (0.023 sec)
MariaDB [(none)]>
現状は「UTF8_IS_UTF8MB3」が設定されているので、UTF8=UTF8MB3扱いになるらしいが、設定されていない場合は「UTF8=UTF8MB4」となるそうな。
だったら、「utf8」でアクセスした場合でも勝手に読み替えて欲しいもんなんだけど、どうなっているのか？？？
MariaDB [(none)]> show variables like 'char%';
+--------------------------+----------------------------+
| Variable_name | Value |
+--------------------------+----------------------------+
| character_set_client | utf8mb3 |
| character_set_connection | utf8mb3 |
| character_set_database | utf8mb3 |
| character_set_filesystem | binary |
| character_set_results | utf8mb3 |
| character_set_server | utf8mb3 |
| character_set_system | utf8mb3 |
| character_sets_dir | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.002 sec)
MariaDB [(none)]>
MariaDB [(none)]> show global variables like '%character_set%';
+--------------------------+----------------------------+
| Variable_name | Value |
+--------------------------+----------------------------+
| character_set_client | utf8mb3 |
| character_set_connection | utf8mb3 |
| character_set_database | utf8mb3 |
| character_set_filesystem | binary |
| character_set_results | utf8mb3 |
| character_set_server | utf8mb3 |
| character_set_system | utf8mb3 |
| character_sets_dir | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.002 sec)
MariaDB [(none)]> show global variables like 'collation%';
+----------------------+--------------------+
| Variable_name | Value |
+----------------------+--------------------+
| collation_connection | utf8mb3_general_ci |
| collation_database | utf8mb3_general_ci |
| collation_server | utf8mb3_general_ci |
+----------------------+--------------------+
3 rows in set (0.002 sec)
MariaDB [(none)]>
ん？？？？？
トラブル生じた時に、MariaDB10.5にしなかったっけ？と
# rpm -qa | grep MariaDB
MariaDB-client-10.5.11-1.el7.centos.x86_64
MariaDB-compat-10.6.3-1.el7.centos.x86_64
MariaDB-common-10.6.3-1.el7.centos.x86_64
MariaDB-server-10.6.3-1.el7.centos.x86_64
#
あー・・・MariaDB-clientとMariaDB-serverのバージョンがずれていました。
そんなわけで、MariaDB-clientの方も 10.6.3に揃えて再実行
# mysql -u root
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 405
Server version: 10.6.3-MariaDB-log MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> exit
Bye
#
というわけで、解決しました。
