VirtualRealization

Wednesday, March 26, 2014

Vcenter Update manager error: "The host returns esxupdate error code:99"

Just out of blue started receiving the following error after trying to scan my ESXi hosts with baseline group:

"The host returns esxup error code:99. An unhandled exception was encountered. Check the update manager log files and esxupdate log files for more details"

well the /var/log/esxupdate.log on ESXi host showed the following error, but not much else:
Metadata.pyc: INFO: Unrecognized file vendor-index.xml in Metadata file

Powercli - Create alarm actions triggers for existing alarm definitions

Here is a script i created to add the following actions for triggers on alarms:
- Send a notification email from specified alarm state and email address
- Send a notification trap from specified alarm state and email address

I recommend just testing with a single Alarm definition in CSV file to make sure it works correctly for you. Only run this once otherwise you will create duplicate triggers.

Connect-VIServer <servername>
$alarmlist = Import-CSV C:\vcenter-alarm.csv

foreach ($item in $alarmlist) {

$alarmname = $item.alarmname
$alarmstartstatus = $item.alarmstartstatus
$alarmendstatus = $item.alarmendstatus
$alarmemail = $item.alarmemail

Get-AlarmDefinition -Name $alarmname | New-AlarmAction -Email -To $alarmemail | New-AlarmActionTrigger -StartStatus $alarmstartstatus -EndStatus $alarmendstatus
Get-AlarmDefinition -Name $alarmname | New-AlarmAction -Snmp | New-AlarmActionTrigger -StartStatus $alarmstartstatus -EndStatus $alarmendstatus
}

CSV file has the following columns: (no quotes necessary for text)

alarmname alarmstartstatus alarmendstatus alarmemail

Disclaimer:
Please use this script at your own risk and test it out in your test lab first before using it in production.

How to retrieve a VM log file through vSphere Web client

In the web client there is an easier way instead or using outside methods for instance:
SSH into host and upload log files to ftp site
GUI interface apps like WinSCP.

These normally have issues with copying of files due to file being locked while VM is powered on.

Firstly need to note the folder file name for VM:

Select VM
Select Summary tab
On VM hardware window and select "settings"
Expand the Hard disks and note the Disk File name provided.

To retrieve the log files following these instructions:

Select VM
Select Related Objects tab
Select Datastores
Right click on databastore and select "Browse files"
Double click the folder associated to VM which was retrieved in beginning.
Highlight each vmware*.log file individually
Right click and select "Download from datastore"
Select local folder location
done!

If your server is unresponsive or crashed you can also export all the log files. However beware this option will suspend or crash your VM so please use with caution!(KB 1007802)

Select VM
Right click VM and select "all vCenter Actions"
Select "export system logs"
Select system logs to export
Select finish
done

Monday, March 10, 2014

vSphere web client login error: "Client is not authenticated to VMware Inventory Service"

This week received the following error when trying to login to vSphere web client.
"Client is not authenticated to VMware Inventory Service"

Troubleshooting:

Was able to login with local user "administrator@vsphere.local".
Problem seems to be only related to domain user authentication.

Looked at all services on vcenter server, which was running.
Service health on vSphere web client also all green without any errors.

Commvault - Backup Copy of snapshot for Exchange 2010

Been running into an interesting scenario with a backup process which I hope will help someone else.

I have an exchange backup of DAG with intellisnap which creates a snapshot of each of the exchange database volume on Netapp. The snapshot is then mounted and copied via SAN transport on the backend fiber connection to the storage library on commvault agent.

This has a storage policy to perform a backup copy of the latest snap via SAN transport weekly on Thursdays.

With commvault I find that sometimes the snapshots are not copied to the commvault storage library, without any type of error.

In viewing the jobs for the primary copy of snaphost i just shows a status of "to be copied", but no failure.

This however causes problems since if you do not pay attention you can sit with weeks of uncopied backup copies of the snapshots, and if you snapshots are set for retention of a few days then you loose all those copies and have to mark them bad! (not good!)

Have not find a way to resolve this, even with the help of Commvault support, but here are a few tips to make sure you are at least inform in more detail regarding this process:

A good indication that the backup copy is not taking place is when the snapshots on netapp volume is not being deleted.
Also make sure you setup a very detailed report for the day after the snapshot copy was to take place and make sure the backup copy detailed information is also included in this report.

If however you do run into this problem, and your snapshots have already been rotated on the volumes on storage you would have to perform the following tasks to continue the backup copy process.

(I recommend contacting Commvault support if you are not familiar with your storage, snapshots and intellisnap, the steps below are provided for use at your own risk, you will loose backup data with these steps)

Verify you oldest snapshot date which are still on volumes for databases
Select storage policy
Right click and primary copy and select view -> jobs
this will list all the backup copies and show the ones with status "to be copied".
Select all jobs up to before the date you have for snapshots on Netapp volume.
Right click and select "Mark job bad" (i know not my favorite step)
Right click on the storage policy again and select All Tasks -> Run Backup copy.
This will start the backup copy of your latest snapshot which resides on backup storage volume

Another solution is to manually add snapshots taken during the week to be added for backup copies.

Right click on the storage policy again and select View -> Jobs
Right click on the snap you want to create backup copy off.
Select "Pick for backup copy"
As a note, you cannot create a backup copy of snapshot which is older than your latest backup copy. so make sure you backup your oldest snapshot first .

Anybody else running into the same problem would love to hear from you.

Dell DRAC and Internet Explorer 11

I don't think i have ever had more problems that i have had with my IE browser and Dell DRAC interface :)

Seems that there is always something broken and can either never get logged in, or show me the turning circle of death or just a blank white screen.

I am still a big IE browser user, yes yes I know, mainly due to compatibility but I think it is getting worse with newer versions and the whole implementation of their compatibility view is just crazy.

Already downgraded from IE 11 back to IE 10, hence why i am writing this email since for the life of me I could not get the DELL DRACs to work. Alas I am back on IE 11 and have this working..

vCloud Director: Error on provider VDC "No datastores accessible to VDC "

This normally happens when a new datastore is added to the storage cluster which is associate to a storage policy for the vCloud Director Organization. The rule-set (storage capability) was not properly set to the datastore before it was added to the datastore cluster.

All the datastores, datastore clusters and policies disappear and you get the following errors:

Error on provider VDC:
No datastores accessible to VDC

On Provider VDC storage policies tab:
This storage policy is not backed by any usable datastores accessible from this VDC.
There is no usable datastore accessible from this VDC that supports this storage policy. Ensure that this storage policy is backed by at least one datastore in vCenter Server, this datastore is enabled in vCloud Director, and the datastore(s) are accessible by the hosts in the clusters used by this VDC.

vSphere web client 5.5: Add new datastore to storage cluster with associate storage profile.

Adding a new datastore to vCloud storage profile has change just a little bit with vSphere Web client.
refer to my previous post to see the difference:
http://virtualrealization.blogspot.com/2014/01/assigning-storage-profilepolicy.html

Information to know up front:
Storage policy/profile name
Storage capability (Rule-Set 1 Tag) - vSphere Web migrated the capabilities to Tags

Steps:

Backend storage:

This is normally handled by storage team which will provide the LUN.
Since I am also the storage engineer I created the LUN from my Dynamic Pool on the Hitachi AMS 2100.

Within vCenter server web client:

Right click the ESXi host from vCloud datastore
Select "all vCenter actions" from menu -> Rescan storage
Only need to check the Scan for new Storage devices.
Rick click the ESXi host again
Select "New datastore".

Type - VFMS or NFS depending on block or file level storage.
Datastore name
Select version
Partition size

Go to vSphere web client home page.
Select VM storage policies which will be associated to this datastore.
Edit the VM storage policy, select Rule-set and write down the Tag name
Right click the newly added datastore
Select "Assign Tag"
Add the Tag to datastore for association to VM Storage policy.
Now you can add the new datastore to the Datastore cluster.

Assigning Storage Profile/Policy: Difference between vSphere client and web client

The vSphere client makes use of the following terminology:
Storage profile
Storage capabilities

The vSphere web client makes use of the following terminology:
Storage policy
Rule-Set through TAGS

The main thing to take note of here is that Storage capabilities created in vSphere client is migrated over to TAGS within vSphere Web client. These tags are then assigned to the VM Storage Policy through the rule-set.

vCloud director 5.5: Catalogs problem #3 - Assigning catalog media files to different VDC within an ORG?

We have a public catalog which is shared to all ORGs within vCloud. This also contains some ISO media files for users to mount to cdrom of virtual machines.
However in order for a user to make use of a media file from a public catalog, that media file firstly needs to be copy to the local catalog of that ORG.

Within this ORG we have multiple vDCs. This is where things get interesting.

Scenario:

When a vAPP is deployed from catalog you assign this vAPP to a specific vDC.
When you then try to mount an ISO from the local catalog media files, you are unable to do so since this media file is associate to a different vDC...mmm

Troubleshooting:

I believe the problem lies with how the Catalogs are done in 5.5. When creating or copying files into a Catalog, there is no method to accurately control where the item will be stored. This is more critical when there are multiple Org vDCs, and a file could be placed in an Org vDC to which the file is not intended for.

This is a departure from vCloud Director 5.1, where importing media allowed you to select a specific Organization vDC. Even when you could select a vDC, you still needed to have the ISO image in the vDC where the VM resides to be allowed to insert it.

I had a lot remote sessions and discussions with support on this issue since we could not find a way to assign a media file to specific VDC.

Workarounds:

Option 1 - Temporarily disable storage profiles before doing an ISO import. This would eliminate certain vDCs as an option, and force the selection process to go to the remaining vDC. this would be my recommendation.

Option 2 - Set the catalog to use a specific Storage Profile, from the desired Organization vDC and import the media again. This should send the imported files to the correct ORG vDC. You will need to import Media files multiple times to accomplish this. This only possible if you have multiple storage profiles which I currently do not have.

Option 3 - Have a catalog specific to an individual ORG vDC, name the catalog relative to the ORG vDC, and select the Storage Class/Profile from the same ORG vDC. This means that any ISOs will be sent up to the correct vDC when you import to a given catalog. On this option here are the steps to setup more catalog for more than one vDC:
1. Create a "master" catalog in the ORG on the Storage Profile vDC A
2. Publish the Catalog created in Step 1 externally
3. Create a "slave" catalog in the ORG on the Storage Profile vDC B, and have this catalog sync from the catalog created in step 1.
4. Repeat this to have Catalog C on Storage Profile C, and sync to Catalog A.
5. Whenever you add a new ISO image to the Master Catalog, it will automatically update the others in 24 hours or when a sync is requested.