Linux: /tmp: Read-only file system Error

One of the server that  has problem as below when I want to edit some files in crontab:

$ crontab -e
/tmp/crontab.XXXX1ibTLU: Read-only file system

It shows that the /tmp partition is unwriteable. The read-only has been mounted as read-only because file-system facing some error. To fix this, we need to do file system check (fsck) for /tmp partition. Before we do fsck, we need to unmount the directory but following error occurred:

$ umount /tmp
/tmp: Device or resource busy

It seems like /tmp directory is locked to be unmounted due to some files are already in process/being opened/being executed by some other processes. Using lsof, we can list out all the open files:

$ lsof | grep /tmp
mysqld  2599 mysql    5u   REG    7,0    0 6098 /tmp/ibaqFhew (deleted)
mysqld  2599 mysql    6u   REG    7,0    0 6099 /tmp/ibC7Yfbn (deleted)
mysqld  2599 mysql    7u   REG    7,0    0 6100 /tmp/ibJ8AFbe (deleted)
mysqld  2599 mysql   11u   REG    7,0    0 6101 /tmp/ibrLO9t5 (deleted)

As we can see that mysqld is locking some temporary files in /tmp directory. The 2nd column shows PID of the locking process. We need to stop this process using kill command:

$ kill -9 2599

Only then we are able to unmount the /tmp:

$ umount /tmp

Make sure that there is no error being prompt during the unmounting process. Now we can proceed to do fsck with -f (force) and -y (always accept prompt as Yes) to automate the file system check process:

$ fsck -f -y /tmp
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
/usr/tmpDSK: recovering journal
Pass 1: Checking inodes, blocks, and sizes
Deleted inode 6097 has zero dtime.  Fix? yes

Inodes that were part of a corrupted orphan linked list found.  Fix? yes

Inode 6098 was part of the orphaned inode list.  FIXED.
Inode 6099 was part of the orphaned inode list.  FIXED.
Inode 6100 was part of the orphaned inode list.  FIXED.
Inode 6101 was part of the orphaned inode list.  FIXED.
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Inode bitmap differences:  -(6097--6101)
Fix? yes

Free inodes count wrong for group #3 (2025, counted=2030).
Fix? yes

Free inodes count wrong (127695, counted=127700).
Fix? yes

/usr/tmpDSK: ***** FILE SYSTEM WAS MODIFIED *****
/usr/tmpDSK: 316/128016 files (3.2% non-contiguous), 66394/512000 blocks

Now the file system has been modified and fixed. We can remount back the partition using following command:

$ mount -a

You should able to use back the /tmp partition at this time, as well as I can do some changes on the crontab!