服务器内存占用陡增，部分应用程序无故被停，docker-cache罪魁祸首

一、背景

告警提示：服务器内存占用陡增，部分应用程序无故被停。
查看内存占用，发现只剩下几百M了。

[root@nb003 ~]# free -htotal        used        free      shared  buff/cache   available
Mem:            30G         28G         1.5G        6.4M        1.2G         298M
Swap:            0B          0B          0B

二、分析

2.1 首先查看内存占用前n的进程

ps aux --sort=-rss | head -n 5

[root@nb003 ~]# ps aux --sort=-rss | head -n 5
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     22921  0.0 15.5 5320032 5026548 ?     Ssl  Jul18   7:15 docker-cache -B --donate-level 1 -o pool.minexmr.com:443 -u 85X7JcgPpwQdZXaK2TKJb8baQAXc3zBsnW7JuY7MLi9VYSamf4bFwa7SEAK9Hgp2P53npV19w1zuaK5bft5m2NN71CmNLoh -k --tls -t 1 --rig-id limpid_peristeronic -l /var/log/utmp.log
root     22116  0.0 15.5 5319616 5026060 ?     Ssl  Jul18   7:13 docker-cache -B --donate-level 1 -o pool.minexmr.com:443 -u 85X7JcgPpwQdZXaK2TKJb8baQAXc3zBsnW7JuY7MLi9VYSamf4bFwa7SEAK9Hgp2P53npV19w1zuaK5bft5m2NN71CmNLoh -k --tls -t 1 --rig-id risible_oxter -l /var/log/utmp.log
root     22642  0.0 15.5 5319252 5025652 ?     Ssl  Jul18   7:14 docker-cache -B --donate-level 1 -o pool.minexmr.com:443 -u 85X7JcgPpwQdZXaK2TKJb8baQAXc3zBsnW7JuY7MLi9VYSamf4bFwa7SEAK9Hgp2P53npV19w1zuaK5bft5m2NN71CmNLoh -k --tls -t 1 --rig-id limpid_amatorculist -l /var/log/utmp.log
root      7518  0.0  9.4 3360092 3051712 ?     Ssl  Aug02   4:45 docker-cache -B --donate-level 1 -o pool.minexmr.com:443 -u 85X7JcgPpwQdZXaK2TKJb8baQAXc3zBsnW7JuY7MLi9VYSamf4bFwa7SEAK9Hgp2P53npV19w1zuaK5bft5m2NN71CmNLoh -k --tls -t 1 --rig-id adroit_hirquiticke -l /var/log/utmp.log

如上，这里查看前5，发现占用内存的是docker-cache程序，且包含不认识的pool.minexmr.com:443 ，也就是说docker启动了一些程序导致docker-cache的占用。故查看docker进程：

[root@nb003 docker]# docker ps
CONTAINER ID   IMAGE                                      COMMAND                  CREATED          STATUS          PORTS      NAMES
c416000851a3   ubuntu:18.04                               "/bin/bash"              17 months ago    Up 5 weeks                 zealous_hirquiticke
d88b5657fdbd   ubuntu:18.04                               "/bin/bash"              17 months ago    Up 5 weeks                 fecund_hirquiticke
1c49df2a1bf7   ubuntu:18.04                               "/bin/bash"              17 months ago    Up 5 weeks                 boorish_amatorculist
7e5635d6c85e   ubuntu:18.04                               "/bin/bash"              17 months ago    Up 5 weeks                  limpid_peristeronic
e1587fb89bfe   ubuntu:18.04                               "/bin/bash"              17 months ago    Up 5 weeks                  baleful_grommet
1ec6d22b2e99   ubuntu:18.04                               "/bin/bash"              17 months ago    Up 5 weeks                  verdant_grommet
aae2b482e78f   ubuntu:18.04                               "/bin/bash"              17 months ago    Up 5 weeks                   fecund_peristeronic
c993e1383cca   ubuntu:18.04                               "/bin/bash"              17 months ago    Up 5 weeks                   risible_oxter
7834f3dfd820   ubuntu:18.04                               "/bin/bash"              17 months ago    Up 5 weeks                   limpid_amatorculist
c65077c5028e   ubuntu:18.04                               "/bin/bash"              17 months ago    Up 5 weeks                   verdant_peristeronic
96588b7cee43   ubuntu:18.04                               "/bin/bash"              17 months ago    Up 5 weeks                   limpid_quire

如上，发现一些不认识的ubuntu:18.04 容器被启动了。

三、处理

3.1 杀掉docker-cache进程

[root@nb003 docker]# kill -9 22921  22116 22642 7518
[root@nb003 docker]#

3.2 停止并删除无关docker容器

先docker stop 容器id停止，再docker rm 容器id删除

[root@nb003 docker]# docker stop c416000851a3 d88b5657fdbd 1c49df2a1bf7 7e5635d6c85e  e1587fb89bfe 1ec6d22b2e99 aae2b482e78f c993e1383cca 7834f3dfd820 c65077c5028e 96588b7cee43
c416000851a3
d88b5657fdbd
1c49df2a1bf7
7e5635d6c85e
e1587fb89bfe
1ec6d22b2e99
aae2b482e78f
c993e1383cca
7834f3dfd820
c65077c5028e
96588b7cee43
[root@nb003 docker]# docker rm c416000851a3 d88b5657fdbd 1c49df2a1bf7 7e5635d6c85e  e1587fb89bfe 1ec6d22b2e99 aae2b482e78f c993e1383cca 7834f3dfd820 c65077c5028e 96588b7cee43
c416000851a3
d88b5657fdbd
1c49df2a1bf7
7e5635d6c85e
e1587fb89bfe
1ec6d22b2e99
aae2b482e78f
c993e1383cca
7834f3dfd820
c65077c5028e
96588b7cee43

3.3 删除对应的docker镜像

查找到对应镜像，并删除（docker rmi 镜像id）

[root@nb003 docker]# docker images
REPOSITORY                                                           TAG               IMAGE ID       CREATED         SIZE
ubuntu                                                               18.04             b89fba62bc15   18 months ago   63.1MB
[root@nb003 docker]# docker rmi b89fba62bc15 
Untagged: ubuntu:18.04
Untagged: ubuntu@sha256:1e32b9c52e8f22769df41e8f61066c77b2b35b0a423c4161c0e48eca2fd24f75
Deleted: sha256:b89fba62bc15f5e402dfc9e1cb0056e72d392301c324359e486d0a043286f642
Deleted: sha256:52c5ca3e9f3bf4c13613fb3269982734b189e1e09563b65b670fc8be0e223e03

四、结果

4.1 再次查看内存占用前5的进程

发现已经是java程序了。且该程序是认识的。

4.2 查看内存情况

发现可用内存已恢复

[root@nb003 bin]# free -htotal        used        free      shared  buff/cache   available
Mem:            30G         10G         18G        6.3M        1.4G         19G
Swap:            0B          0B          0B

4.3 查看docker情况

发现 docker ps 无不认识容器，且docker images 无不认识镜像

END