Wednesday, March 26, 2014

Vcenter Update manager error: "The host returns esxupdate error code:99"

Just out of blue started receiving the following error after trying to scan my ESXi hosts with baseline group:

"The host returns esxup error code:99. An unhandled exception was encountered. Check the update manager log files and esxupdate log files for more details"

well the /var/log/esxupdate.log on ESXi host showed the following error, but not much else:
Metadata.pyc: INFO: Unrecognized file vendor-index.xml in Metadata file

Wednesday, March 12, 2014

Powercli - Create alarm actions triggers for existing alarm definitions

Here is a script i created to add the following actions for triggers on alarms:
- Send a notification email from specified alarm state and email address
- Send a notification trap from specified alarm state and email address

I recommend just testing with a single Alarm definition in CSV file to make sure it works correctly for you. Only run this once otherwise you will create duplicate triggers.

Connect-VIServer <servername>
$alarmlist = Import-CSV C:\vcenter-alarm.csv

foreach ($item in $alarmlist) {

$alarmname = $item.alarmname
$alarmstartstatus = $item.alarmstartstatus
$alarmendstatus = $item.alarmendstatus
$alarmemail = $item.alarmemail

Get-AlarmDefinition -Name $alarmname | New-AlarmAction -Email -To $alarmemail | New-AlarmActionTrigger -StartStatus $alarmstartstatus -EndStatus $alarmendstatus
Get-AlarmDefinition -Name $alarmname | New-AlarmAction -Snmp | New-AlarmActionTrigger -StartStatus $alarmstartstatus -EndStatus $alarmendstatus

CSV file has the following columns: (no quotes necessary for text)

alarmname alarmstartstatus alarmendstatus alarmemail

Please use this script at your own risk and test it out in your test lab first before using it in production.

How to retrieve a VM log file through vSphere Web client

In the web client there is an easier way instead or using outside methods for instance:
SSH into host and upload log files to ftp site
GUI interface apps like WinSCP.

These normally have issues with copying of files due to file being locked while VM is powered on.

Firstly need to note the folder file name for VM:

  1. Select VM
  2. Select Summary tab
  3. On VM hardware window and select "settings"
  4. Expand the Hard disks and note the Disk File name provided.

To retrieve the log files following these instructions:

  1. Select VM
  2. Select Related Objects tab
  3. Select Datastores
  4. Right click on databastore and select "Browse files"
  5. Double click the folder associated to VM which was retrieved in beginning.
  6. Highlight each vmware*.log file individually
  7. Right click and select "Download from datastore"
  8. Select local folder location
  9. done!
If your server is unresponsive or crashed you can also export all the log files.  However beware this option will suspend or crash your VM so please use with caution!(KB 1007802)

  1. Select VM
  2. Right click VM and select "all vCenter Actions"
  3. Select "export system logs"
  4. Select system logs to export
  5. Select finish
  6. done

Monday, March 10, 2014

vSphere web client login error: "Client is not authenticated to VMware Inventory Service"

This week received the following error when trying to login to vSphere web client.
"Client is not authenticated to VMware Inventory Service"


Was able to login with local user "administrator@vsphere.local".
Problem seems to be only related to domain user authentication.

Looked at all services on vcenter server, which was running.
Service health on vSphere web client also all green without any errors.

Tuesday, March 4, 2014

Commvault - Backup Copy of snapshot for Exchange 2010

Been running into an interesting scenario with a backup process which I hope will help someone else.

I have an exchange backup of DAG with intellisnap which creates a snapshot of each of the exchange database volume on Netapp.  The snapshot is then mounted and copied via SAN transport on the backend fiber connection to the storage library on commvault agent.

This has a storage policy to perform a backup copy of the latest snap via SAN transport weekly on Thursdays.

With commvault I find that sometimes the snapshots are not copied to the commvault storage library, without any type of error.
In viewing the jobs for the primary copy of snaphost i just shows a status of "to be copied", but no failure.

This however causes problems since if you do not pay attention you can sit with weeks of uncopied backup copies of the snapshots, and if you snapshots are set for retention of a few days then you loose all those copies and have to mark them bad! (not good!)

Have not find a way to resolve this, even with the help of Commvault support, but here are a few tips to make sure you are at least inform in more detail regarding this process:

  • A good indication that the backup copy is not taking place is when the snapshots on netapp volume is not being deleted.
  • Also make sure you setup a very detailed report for the day after the snapshot copy was to take place and make sure the backup copy detailed information is also included in this report.

If however you do run into this problem, and your snapshots have already been rotated on the volumes on storage you would have to perform the following tasks to continue the backup copy process.
(I recommend contacting Commvault support if you are not familiar with your storage, snapshots and intellisnap, the steps below are provided for use at your own risk, you will loose backup data with these steps)
  • Verify you oldest snapshot date which are still on volumes for databases
  • Select storage policy
  • Right click and primary copy and select view -> jobs
  • this will list all the backup copies and show the ones with status "to be copied".
  • Select all jobs up to before the date you have for snapshots on Netapp volume.
  • Right click and select "Mark job bad" (i know not my favorite step)
  • Right click on the storage policy again and select All Tasks -> Run Backup copy.
  • This will start the backup copy of your latest snapshot which resides on backup storage volume
Another solution is to manually add snapshots taken during the week to be added for backup copies.

  • Right click on the storage policy again and select View -> Jobs
  • Right click on the snap you want to create backup copy off.
  • Select "Pick for backup copy"
  • As a note, you cannot create a backup copy of snapshot which is older than your latest backup copy. so make sure you backup your oldest snapshot first . 
Anybody else running into the same problem would love to hear from you.

Dell DRAC and Internet Explorer 11

I don't think i have ever had more problems that i have had with my IE browser and Dell DRAC interface :)

Seems that there is always something broken and can either never get logged in, or show me the turning circle of death or just a blank white screen.

I am still a big IE browser user, yes yes I know, mainly due to compatibility but I think it is getting worse with newer versions and the whole implementation of their compatibility view is just crazy. 
Already downgraded from IE 11 back to IE 10, hence why i am writing this email since for the life of me I could not get the DELL DRACs to work. Alas I am back on IE 11 and have this working..