Free up disk space from deleted files under running processes.

A lot of the time a large log file will grow and need removed,  most the time these files cannot actually be “deleted” or “cleared” until the service releases its file descriptor.

 

Identify the file:

List files recently deleted that have not been released.

 

root@osc-1015 #> lsof -a +L1
COMMAND      PID USER   FD   TYPE DEVICE   SIZE/OFF NLINK   NODE NAME
systemd-j   1059 root  txt    REG  253,4     278808     0  11628 /usr/lib/systemd/systemd-journald;570ba957 (deleted)
systemd-l   1451 root  txt    REG  253,4     584560     0  33117 /usr/lib/systemd/systemd-logind;570ba957 (deleted)
monitor     1617 root    5w   REG  253,4        500     0 261586 /var/log/openvswitch/ovsdb-server.log-20160404 (deleted)
monitor     1617 root    7u   REG  253,4        141     0     17 /tmp/tmpfsSX0WX (deleted)
ovsdb-ser   1619 root    7u   REG  253,4        141     0     17 /tmp/tmpfsSX0WX (deleted)
monitor     1722 root    3w   REG  253,4     474455     0 261589 /var/log/openvswitch/ovs-vswitchd.log-20160404 (deleted)
ceph-osd   20462 root  txt    REG  253,4   11589728     0  33573 /usr/bin/ceph-osd;570b9a69 (deleted)
ceph-osd   20686 root  txt    REG  253,4   11589728     0  33573 /usr/bin/ceph-osd;570b9a69 (deleted)
qemu-kvm  107850 qemu    8w   REG  253,4 2207794598     0 261623 /var/lib/nova/instances/8921a9ef-81c4-4a06-be00-7cad86bd6a1c/console.log (deleted)

in this instance I need to clear the console.log file

 

Release the kernel lock:

Now we will release its lock in the kernel.  The key parts here are the PID and FD,   We remove the write flag from the FD and use its ID.

 root@osc-1015#> : > "/proc/107850/fd/8"

 

Once ran the file is released and can be relocked by the process if it begins writing again.

Leave a Reply

Your email address will not be published. Required fields are marked *