Datastore unmount error: Cannot unmount volume because file system is busy

Just this week ran into this issue while trying to unmount some stale datastore from vCenter server.

Troubleshooting:

SSH into the ESXi host and browse the datastore.
On the datastore I found the following files listed with following names:
vsantraces*

Found KB 2069171 specifying this problem where the VSAN modules stores the traces needed to debug VSAN related problems which is keeping a lock on the datastore.



Resolution:

Since we are not using VSAN I had to disabled the VSAN services on the ESXi host to remove the datastore.

• To verify open files for vsantrace:
• lsof | grep vsantraced | grep volumes
• Run command to stop services:
• /etc/init.d/vsantraced stop
• Perform a Refresh for Storage
• Unmount the datastore
• Run command to start the services: 
• /etc/init.d/vsantraced start

If you are however using VSAN then you need to change the trace location with the following command:

esxcli vsan trace set -p <datastore_name>

To verify open files for vsantrace
lsof | grep vsantraced | grep volumes



On a side note:

Initially the unmount of the datastore still did not exceed and this was due to my SSH sessions which has the datastore open :) Close the SSH session before trying to unmount.

You could potentially still have problem with unmount when following scenario exists: 

• Scratch for host is configured on the problem datastore. Modify the ScratchConfig value and change it to another datastore. For more information, see Creating a persistent scratch location for ESXi 4.x and 5.x (1033696).
• Coredump is configured to write to a file in the problem datastore in an ESXi 5.5 host. For more information see, Move the coredump to a File from the VMware vSphere 5.5 documentation Center.
• Run following command to check for paths:
• esxcli system coredump file list
• Verify the UUID and compare to datastore name.
• Run following command to remove the coredump files. 
• esxcli system coredump file remove --force
• 
  

Links:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2069171

http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1033696