zabbix serverがダウン
サーバがダウンしたのではなくて、zabbix_serverのプロセスがダウン。手動で起動してもNG。
ログを見るとこんなのが。
10299:20180823:121010.333 __mem_malloc: skipped 0 asked 72 skip_min 4294967295 skip_max 0 10299:20180823:121010.335 [file:dbcache.c,line:3407] zbx_mem_malloc(): out of memory (requested 72 bytes) 10299:20180823:121010.335 [file:dbcache.c,line:3407] zbx_mem_malloc(): please increase TrendCacheSize configuration parameter 10288:20180823:121010.371 One child process died (PID:10299,exitcode/signal:1). Exiting ... 10288:20180823:121012.379 syncing history data... 10288:20180823:121012.390 __mem_malloc: skipped 0 asked 72 skip_min 4294967295 skip_max 0 10288:20180823:121012.390 [file:dbcache.c,line:3407] zbx_mem_malloc(): out of memory (requested 72 bytes) 10288:20180823:121012.390 [file:dbcache.c,line:3407] zbx_mem_malloc(): please increase TrendCacheSize configuration parameter zabbix_server [22477]: cannot open log: cannot create semaphore set: [28] No space left on device zabbix_server [22493]: cannot open log: cannot create semaphore set: [28] No space left on device zabbix_server [22507]: cannot open log: cannot create semaphore set: [28] No space left on device zabbix_server [22519]: cannot open log: cannot create semaphore set: [28] No space left on device zabbix_server [22531]: cannot open log: cannot create semaphore set: [28] No space left on device zabbix_server [22547]: cannot open log: cannot create semaphore set: [28] No space left on device
以下セマフォが確保できないとずらっと。
大昔にapacheでsemaphoreがたくさんで起動できない事象がでたことを思い出して、同様の処置。
# ipcs -s | grep zabbix | awk '{print $2}' | xargs ipcrm sem