Friday, August 29, 2014

Datastore unmount error: Cannot unmount volume because file system is busy

Just this week ran into this issue while trying to unmount some stale datastore from vCenter server.


SSH into the ESXi host and browse the datastore.
On the datastore I found the following files listed with following names:

Found KB 2069171 specifying this problem where the VSAN modules stores the traces needed to debug VSAN related problems which is keeping a lock on the datastore.


Since we are not using VSAN I had to disabled the VSAN services on the ESXi host to remove the datastore.

  • To verify open files for vsantrace:
    • lsof | grep vsantraced | grep volumes
  • Run command to stop services:
    • /etc/init.d/vsantraced stop
  • Perform a Refresh for Storage
  • Unmount the datastore
  • Run command to start the services: 
    • /etc/init.d/vsantraced start

If you are however using VSAN then you need to change the trace location with the following command:
esxcli vsan trace set -p <datastore_name>

To verify open files for vsantrace
lsof | grep vsantraced | grep volumes

On a side note:

Initially the unmount of the datastore still did not exceed and this was due to my SSH sessions which has the datastore open :) Close the SSH session before trying to unmount.

You could potentially still have problem with unmount when following scenario exists: 
  • Scratch for host is configured on the problem datastore. Modify the ScratchConfig value and change it to another datastore. For more information, see Creating a persistent scratch location for ESXi 4.x and 5.x (1033696).
  • Coredump is configured to write to a file in the problem datastore in an ESXi 5.5 host. For more information see, Move the coredump to a File from the VMware vSphere 5.5 documentation Center.
    • Run following command to check for paths:
      • esxcli system coredump file list
    • Verify the UUID and compare to datastore name.
    • Run following command to remove the coredump files. 
      • esxcli system coredump file remove --force