zabbix serverがダウン

サーバがダウンしたのではなくて、zabbix_serverのプロセスがダウン。手動で起動してもNG。
ログを見るとこんなのが。

 10299:20180823:121010.333 __mem_malloc: skipped 0 asked 72 skip_min 4294967295 skip_max 0
 10299:20180823:121010.335 [file:dbcache.c,line:3407] zbx_mem_malloc(): out of memory (requested 72 bytes)
 10299:20180823:121010.335 [file:dbcache.c,line:3407] zbx_mem_malloc(): please increase TrendCacheSize configuration parameter
 10288:20180823:121010.371 One child process died (PID:10299,exitcode/signal:1). Exiting ...
 10288:20180823:121012.379 syncing history data...
 10288:20180823:121012.390 __mem_malloc: skipped 0 asked 72 skip_min 4294967295 skip_max 0
 10288:20180823:121012.390 [file:dbcache.c,line:3407] zbx_mem_malloc(): out of memory (requested 72 bytes)
 10288:20180823:121012.390 [file:dbcache.c,line:3407] zbx_mem_malloc(): please increase TrendCacheSize configuration parameter
zabbix_server [22477]: cannot open log: cannot create semaphore set: [28] No space left on device
zabbix_server [22493]: cannot open log: cannot create semaphore set: [28] No space left on device
zabbix_server [22507]: cannot open log: cannot create semaphore set: [28] No space left on device
zabbix_server [22519]: cannot open log: cannot create semaphore set: [28] No space left on device
zabbix_server [22531]: cannot open log: cannot create semaphore set: [28] No space left on device
zabbix_server [22547]: cannot open log: cannot create semaphore set: [28] No space left on device

以下セマフォが確保できないとずらっと。
大昔にapacheでsemaphoreがたくさんで起動できない事象がでたことを思い出して、同様の処置。

# ipcs -s | grep zabbix | awk '{print $2}' | xargs ipcrm sem