Tuesday, December 31, 2013

Error: Virtual disk 'Hard disk x' is a mapped direct-access LUN that is not accessible

Our exchange server 2010 VM has RDMs for each database. Overnight the volumes filled up after commvault snapshots of database volumes which took the volume offline and stopped the VM.
Reading online it seems this problemalso occur when users try to vMotion a VM but that has not happened in my case.

The VM was rebooted by the Exchange admin but during start up the VM gave the following error message:

Virtual disk 'Hard disk x' is a mapped direct-access LUN that is not accessible

Troubleshooting:

On vsphere client when selecting the Hard disk and selecting manage paths you get errors "there is no multipath configuration for this LUN"


Identify if the Raw Data Mapping (RDM) LUN signature vml.############ assigned to the VM no longer matches the physical LUN assigned to the ESXi Host.
  • On VM record the physical LUN mapping vml.########
  • On ESXi host where VM resides run command to find the device name entry naa.############ for the vml identifier of VM.  The "Other UIDs:" displays the vml identifier.
  • On each ESXi host within the cluster verify that the device name has the same vml entry as what is associate to virtual machine hard disk.
    • ls -alh /vmfs/devices/disks

Solution 1:

This was my first try without consulting online and did not want to remove the Mapped Raw LUN from the VM so to fix this I did the following:

  1. I disabled the problem LUN on the back-end storage (our case Netapp) and then on each host i re-scanned the storage so that the devices was not showing anymore.
  2. I enabled the problem LUN again and rescanned the hosts.
  3. Make sure the device is showing again.
  4. Verify that the datastore mapping file (vml.######) was identical on VM to that on each ESXi host in the cluster for the particular problem hard drive.
  5. Started up the VM and problem was resolved.
If this solution does not work for you, then i would recommend using solution 2 which is pretty much straight from the VMware KB 1016210

Solution 2:

Identify the problem hard drive and record the following settings on virtual machine:
  • Mapped Raw LUN's - physical LUN and datastore mapping file location. (vml.############)
  • SCSI ID
  • Associated LUN ID
    • to get this information you need to run a few commands on ESXi host where VM currently resides
      • ls -alh /vmfs/devices/disks (this will give device indentifier from RDM vml address.
      • esxcli storage core path list (shows device identifier with LUN information) 
  • Remove the RDM LUN
  • Map the RDM LUN again within the VM.  This will be done by snapdrive agent on server.
  • Power on the VM.


Links:


Sunday, December 29, 2013

vCloud Director upgrade from 5.1 to 5.5

Current Version:

5.1.2 Build 1068441 vmware-vcloud-director-5.1.2-1968441.bin

New Version:

5.5.0 Build 1323688  vmware-vcloud-director-5.5.0-1323688.bin

I run a RHEL virtual machine with vcloud director installed, so before starting the upgrade I fully patch the RHEL environment.

PRE-CHECKLIST:

VUM errror during remediation of ESXi 5.1 to 5.5 host upgade: "Cannot run upgrade script on host" - resolved


VUM error during remediation of ESXi host upgrade from 5.1 to 5.5:
"vmware update manager 5.5 Cannot run upgrade script on host"

Debugging the problem:

Troubleshooting this problem led to a few discoveries online of users experiencing the same error message but different log entries regarding problem. This can be read in kb articles below:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2007163
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2014084

vCenter Server upgrade from 5.1.1 to 5.5.0

Current version:

vCenter server:       5.1.0 Build 1063329
ESXi hosts:           5.1.0 Build 1117900


New version:

5.5.0b

ESXi software file:  VMware-VMvisor-Installer-5.5.0-1331820.x86_64-Dell_Customized_A01   (we have Dell servers so using the latest Dell provided 5.5 installer)
VCenter server:      VMware-VIMSetup-all-5.5.0-1476387-20131201 (5.5.0b)

Friday, December 6, 2013

Change VM disk from thick to thin with standalone ESX host and no vcenter server connection

Just got an interesting request from user regarding a datastore that ran out of space on a standalone ESXi host.  This is a Dell R720 server with local attached storage and all drives filled.

Debugging the problem:

Upon further investigate i found they create a disk on VM with thick provisioning when they not even using half of the disk space within the vmdk.

However since ESX host is not attached to VCenter server we cannot perform a clone or migration from the console, what to do?

Resolution:

Using The Command Line (SSH)
If you don't have any space available on current datastore like in my situation then you would have to add a  temporary datastore. NFS easiest to quickly configure.

SSH to host and run the following command:

vmkfstools –i “<ThickDiskName.vmdk” -d thin “NewDiskName.vmdk”

After this is completed you have to remove the thick disk and add the newly create thin disk to the VM.
For removal options i would recommend selecting "remove from virtual machine" only and NOT yet permanently delete the disk just yet.  Wait until you have new disk configured within the VM operating system and verify it works as expected.