vCenter Configuration Manager (VCM): jobs fail to start

Currently having some issue with VCM so wanted to provide some quick information on this particular problem.

Trying to install Agent to a Windows Machine with DCOM but when i start the process the task does not appear in the jobs window.   I can however manually install the agent on the windows machine after which i can successfully connect through VCM.

Troubleshooting:

Verified all of the following

• Collector service is running.  
• My user account has VCM or SQL permissions to run these jobs.
• Network authority account has local admin to windows machine
• DNS and reverse DNS not set up properly for the target marchine or VCM.
• Use nslookup on VCM to validate NetBIOS, FQDN and reverse IP lookup for the target machine.
• DCOM not disabled by GPO on domain
• Firewall not enabled on VMC or client windows machine

Viewing of log files for VCM:

RDP to the VCM server and look at (VCM default install location) C:\Program Files (x86)\VMware\VCM\Tools\EcmDebugEventViewer.

Launch EcmDebugEventViewer.exe and Press F5. This tool will display all the Warnings, Errors, and Exceptions and may reveal some additional conditions we'll need to troubleshoot this issue.

Resolution:

Circled back to agent install task not showing in jobs window at all and traced problem down to broker services not running on my VCM databases.

• Open SSMS for SQL and connect to database server
• Run this query to verify which database has broker enabled:
• select name, is_broker_enabled from sys.databases
• Note: The broker value should be 1 for all databases, except master and model.
• For each database on which the broker is disabled, run this SQL command:
• alter database [Database Name] SET ENABLE_BROKER

What we did find is that if this service has been down for a very long time you could possible receive SQL deadlock errors in the log files and will also be unable to login to environment.  In SQL you will see alot of delete commands by running script "exec_sp3".  Just let this run out and complete.  Could take a couple of hours.

VMworld 2014 recap

Sitting at airport on way back from a very successful VMworld week and thought i would put some my thoughts into a quick post.  I am just going to briefly mention my highlights.  Some of these are existing products and nothing new to most people and then a few new things too.

vCAC

• It is like they say the future and the future is looking good.
• vCloud director only available as service provide offering in near future...

Application director 

• If you ever get a chance then watch this session:
• https://vmworld2014.activeevents.com/connect/sessionDetail.ww?SESSION_ID=2609

RaaS (Recovery as a service) 

• Cloud based disaster recovery with very good RPO & RTO
• http://www.vmware.com/products/vcloud-hybrid-service/

vCloud Air everything

• vRealize Air Automation

EVO:RAIL

• hyper-converged infrastructure appliance.  This is Nutanix territory but excited to see what EVO:RAIL will bring in future.
• http://www.vmware.com/products/evorail

NSX

• Virtualize the network and all the new features including 3de party integration, but the most exciting is the micro-segmentation from a security perspective.

Hands on Lab

• This was amazing and a great success.  Look forward to running though all those lab sessions in the future.

AirWatch

• 8th most taken lab at VMworld and received a lot of attention!

vCOPS alert: analytics resource: number of resources exceeds supported limit

Received the following admin alert in vCOPS:

analytics resource: number of resources exceeds supported limit

This is due to the default max number of resources set for vCOPS is 10 000.
This was not enough of our environment and therefore we had to increase this max number.  The way vCOPS handles resources also contributed to this alert since it does not by default delete non-existant resources.  In another post I provide information on how to change this behaviour:
http://virtualrealization.blogspot.com/2014/08/vcops-admin-alerts-for-vin-adapter.html


Solution:

Add the following line to the file on the analytics server:
"/usr/lib/vmware-vcops/user/conf/analytics/advanced.properties

maxNumberOfResourcesSupported= 999999 (can set your own number here)

restart the analytics server

• SSH into analytics server and login as admin user
• vcops-admin restart

vCOPS: admin alerts for VIN adapter: resources do not receive data from this adapter resource

I was getting a lot of admin alarms within vCOPS for VIN adapters not able to receive data for resources.

VIN adapter instance: 200 resources do not receive data from this adapter resource

I believe the primary reason for this is because my vCloud environment is so dynamic with users deleting and creating new VM's that vCOPS and VIN are not keeping up and non-existent resources are causing the alerts.

By default vCOPS does not delete non-existant resources but I found the following KB 2020638 which provides information on changing the schedule for when deletion of old objects will take place from vCOPS.

Snapshot consolidation failed with error

Virtual machine was showing following alert:

Virtual machine disk consolidation is needed.  Virtual machine consolidation needed status

After trying to consolidate the virtual machine snapshots I received the following error:

Status: An error occurred while consolidating disks: Could not open/create change tracking file

Cannot complete the operation because the file or folder ds:///vmfs/volumes/*.vmdk already exists

Troubleshooting:

SSH to ESXi host and browse the volume where the VM resides on.

cd into the VM folder and found flat files for each of the vmdk disks but the VM does not show that any snapshots exists.

Datastore unmount error: Cannot unmount volume because file system is busy

Just this week ran into this issue while trying to unmount some stale datastore from vCenter server.

Troubleshooting:

SSH into the ESXi host and browse the datastore.
On the datastore I found the following files listed with following names:
vsantraces*

Found KB 2069171 specifying this problem where the VSAN modules stores the traces needed to debug VSAN related problems which is keeping a lock on the datastore.

VIN guest operating system management: limited permissions for vcenter server

vCenter Infrastructure Navigator has a single account which it uses to access VMs.
This is set within vCenter Server's infrastructure navigator screen.

I wanted to limit which datacenters inside vCenter Server VIN would be able to see as the functions this account can perform.

Resolution:

Create a guest operating system management role within vCenter Server with minimal privileges for VIN:

• Navigate to administration
• Select roles
• Click add roles
• Enable the following with checkbox:
• Virtual machine > Interaction -> Guest operating system management by VIX API 
• Virtual machine > Interaction -> Console interaction
• Provide role name
• 
  
• On the vCenter entity root level click manage tab
• Select permissions and add
• Select user and assign the newly create limited VIN role.

This will provide the necessary privileges to enable the discovery process for user selected.

In my case I also did not want this service account to view my vCloud director datacenter so I added the user to the datacenter permissions with "no access" role.

From inventory menu select Infrastructure navigator.
Select settings tab
Here you can now set the new user account.
Make sure to enable access to VMs

vCOPS - Custom UI LDAP error - "One or more users already exist and haven't been imported"

The regular vcops-vshere WEB GUI was easy and straight forward to configure for LDAP authentication since it uses the vCenter privileges.

However vCOPS custom GUI web interface uses a custom configuration for LDAP authentication. I am also using sAMAccountName for Username Field on LDAP settings.  This is not available and have to type in manually.

After setting up LDAP I tried to import the same users but receive the following error message:

"One or more users already exist and haven't been imported"

Java problems with vCenter Orchestrator

All applets and web start java applications has defaulted to high security since Update 11.
The security context that is used by vCO Client is set to high so some changes are needed within the Java control panel.

Resolution:

• Open the Java Control Panel
• Go to the Security tab. 
• At the bottom of the dialog you will see the current Exception Site List. 
• Click the Edit Site List button.
• In the exception entry dialog, enter the URL for your vCO Server

VNX MnR: Not showing SAN data after upgrade

Recently ran the upgrade of VNX monitoring and reporting from version 1.2 to 2.0.

Upgrade completed successfully but after logging in and viewing the data we were unable to view the file storage information.

Resolution:

• Verify NaviSECCLI path is correct in VNX MnR Config 

• If the above is correctly configured, attempt updating NaviSecCLI to latest Version (Found on EMC Support Site under downloads if you search for NAVISECCLI)
• Once Installed issue any command to accept the certificate from VNX This is only required if NAVICLI version is updated 
• Open Cmd Prompt and issue any NAVI CLI Command to be prompted to accept certificate

vCOPS: admin alert "number of resources exceeds supported limit"

We recently started seeing the following alert in our vCops environment.

Resolution:

The problem is that the maximum of 1000 supported resources has been reached.
To fix this you have to update the advanced properties configuration file for analytic.

• Open SSH session to Analytic server.
• browse to /usr/lib/vmware-vcops/user/conf/analytics/
• open the file advanced.properties (vi advanced.properties)
• add the following line to the file:
• "maxNumberOfResourcesSupported=<#>"
• make sure the number added is larger than 1000 since this is the default
• save the file
• Restart the analytic services within vCops.
• "service vcops restart analytics"

vCenter Log Insight - VMware - vSphere content pack v 2.0 reinstall

After upgrading to vCenter Log Insight v 2.0 I wanted to install the latest content packs so I downloaded the VMware - vSphere content pack from VMware Solution Exchange.

I imported the content pack and I was prompted if i wanted to upgrade.
Not paying attention I actually downgraded the content pack to v1.5 which is the latest which seems to be available on Solution Exchange.

Here are the steps to revert back to latest v 2.0 content pack for VMware - vSphere:

1. Download 7zip.
2. Open the archive PAK file which is downloaded for the upgrade of Log Insight.
3. Browse to \usr\lib\loginsight\application\etc\content-packs\vsphere\
4. Extract the file content.json.
5. Open Log Insight.
6. Select Content Packs
7. Select "Import Content Pack"
8. Browse for the content.json file.
9. Upload

This will now again install the latest version 2.0.

vCenter Log Insight 1.0.4 upgrade to v2.0.3? where to now?

So I had vCenter Log Insight v 1.0.4 installed in our environment and just never had the time to upgrade this until recently.

The problem that exists in that in the old releases the rpm file was used to upgrade through CLI.
In the new releases since 1.5G they make use of PAK files for upgrades through the web interface which is must more user friendly.

Well since I only had v1 installed and uses RPM it caused a bit of issues since only the v2 PAK file is available on download site.

To fix this i did the following:

1. Download the v1.5 file from following location:
• https://my.vmware.com/group/vmware/details?downloadGroup=STRATA15&productId=386&rPId=4787
2. Backups / snapshot your VM appliance.
3. Copy the rpm to the VM, i used the \tmp folder.
4. Upgrade Log Insight through the CLI by running the following commands:
1. service loginsight stop
2. rpm -Uvh loginsight-cloudvm-1.5.0-1435442.x86_64.rpm
3. service loginsight start
5. Now login to the appliance.
1. Go to administration
2. Under management select Appliance
3. upload the new PAK file for v2
4. Click upgrade.

This should be it and you should have the new version ready.

Links:

Upgrade log insight installed versions:
http://pubs.vmware.com/log-insight-15/index.jsp?topic=%2Fcom.vmware.log-insight.administration.doc%2FGUID-5933433F-4A17-4DF0-A05F-49DF2D104974.html&__utma=207178772.1598018981.1404147381.1404147381.1404147381.1&__utmb=207178772.0.10.1404147381&__utmc=207178772&__utmx=-&__utmz=207178772.1404147381.1.1.utmcsr=bing|utmccn=(organic)|utmcmd=organic|utmctr=vcenter%20log%20insight%20upgrade&__utmv=-&__utmk=81845013

VMWare OpenSSL fixes: what to do

VMware:

Vulnerable Products:

• ESXi hosts
• Windows version of vCenter Server (only Atlanta)
• vCenter Server appliance
• VMware Client Integration Plug-in
• vCenter Single Sign-On VMware Directory Service

Fixes:

• ESXi hosts
• Verify you version installed and only upgrade to the patch specified in the following KB:
• http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2076665
• Running following commands to re-issue the certificate:
• cd /etc/vmware/ssl and ls -l
• mv rui.crt /vmfs/volumes/ATL01PRDCS01_WINBUILD_01_01/VMH04orig.rui.crt
• mv rui.key /vmfs/volumes/ATL01PRDCS01_WINBUILD_01_01/VMH04orig.rui.key
• /sbin/generate-certificates
• chmod +t rui.crt
• chmod +t rui.key
• Change root password
• Reboot the server
• Reconnect host again to vCenter
• Windows version of vCenter Server
• Verify you version installed and only upgrade to the patch specified in the following KB:
• http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2076692
• In my instance I upgraded vCenter server from 5.5 build 1476327 to 5.5.0c build 1750597
• Perform steps as listed below for SSO certificate recreate
• Now you can upgrade to latest 5.5 Update 1a build 1750787?  (KB: 2076692)
• Verify log on as accounts for VMware services after upgrade.  For some reason mine changed to the user as was logged in as so some plugins did not start.

• vCenter Server appliance
• Upgrade to latest release to patch Client Integration Plug-in.

• vCenter Single Sign-On VMware Directory Service
• recreate certificates (reference KB 2076692)
• change password

• VMware Client Integration Plug-in
• Download from webpage and upgrade for all users accessing vCenter web client.

Good KB to subscribe too for updates on this issue:

http://kb.vmware.com/kb/2079783

vCloud 5.5.1: Create new storage policy, add storage, migrate vApps.

We recently purchased new storage so the new datastores had to be added to vCloud director, and all the vApps migrated over.  There are many ways to go about doing this but the best solution I found was to create a new storage cluster and put all datastores in it and also create a new storage policy and profile.  I have been told it is also possible to just perform a storage vmotion on the VM's but i had mixed results with this approach.  Here is my information which I hope might be useful to someone.

In Vcenter server:

1. Create tag category (I called mine "storage policies" to keep it simple", but description can also be used to describe the function)
2. Create tag and add to category
3. Open VM storage policies
1. Create new VM storage policy
2. Select vCenter
3. Enter a policy name
4. Provide description
5. Add tag-based rule and assign newly create tag from tag category.
4. I ALWAYS FORGET to enable the policy so here is not to you doing the same :)
1. select the new policy name
2. 2nd icon from left is "Enable VM storage policies per compute resource.

1. Assign the newly created tag to datastore cluster
2. Assign the newly created tag to each datastore within the cluster.

In vCloud director:

1. Under Manage & Monitor tab select vCenters from vSphere resources.
2. Right click the vCenter server name and select  "refresh storage policies

After you get the storage profile to show in storage policies you have to assign it:

1. Go to Provider VDC and open it
2. Select storage policies tab
3. Add new storage policy! 
4. There is a BUG in 5.5.1 which does not refresh or show the storage policy and an outage is actually necessary to clean up the database with a script (not very nice but works)


Error on 5.5.1 (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2075366)

1. Go to Organizations and open the one which will be using the new storage
2. Select Virtual datacenter tab
3. Select Storage policies tab
4. Add the new storage policy
5. If you want the new policy to be the default - Right click the new storage policy and select "Set as default"

Migrating vAPPS to new storage profile:

• The best method I found was to shutdown the vAPP and right click select "move to" and then each individual VM select the new storage policy.
• You can also do this with a powered on VM and go to properties and change the storage polic

Vcenter Update manager error: "The host returns esxupdate error code:99"

Just out of blue started receiving the following error after trying to scan my ESXi hosts with baseline group:

"The host returns esxup error code:99. An unhandled exception was encountered. Check the update manager log files and esxupdate log files for more details"

well the /var/log/esxupdate.log on ESXi host showed the following error, but not much else:
Metadata.pyc: INFO: Unrecognized file vendor-index.xml in Metadata file

Powercli - Create alarm actions triggers for existing alarm definitions

Here is a script i created to add the following actions for triggers on alarms:
- Send a notification email from specified alarm state and email address
- Send a notification trap from specified alarm state and email address

I recommend just testing with a single Alarm definition in CSV file to make sure it works correctly for you. Only run this once otherwise you will create duplicate triggers.


Connect-VIServer <servername>
$alarmlist = Import-CSV C:\vcenter-alarm.csv

foreach ($item in $alarmlist) {

$alarmname = $item.alarmname
$alarmstartstatus = $item.alarmstartstatus
$alarmendstatus = $item.alarmendstatus
$alarmemail = $item.alarmemail

Get-AlarmDefinition -Name $alarmname | New-AlarmAction -Email -To $alarmemail | New-AlarmActionTrigger -StartStatus $alarmstartstatus -EndStatus $alarmendstatus
Get-AlarmDefinition -Name $alarmname | New-AlarmAction -Snmp | New-AlarmActionTrigger -StartStatus $alarmstartstatus -EndStatus $alarmendstatus
}


CSV file has the following columns: (no quotes necessary for text)

alarmname alarmstartstatus alarmendstatus alarmemail



Disclaimer:
Please use this script at your own risk and test it out in your test lab first before using it in production.

How to retrieve a VM log file through vSphere Web client

In the web client there is an easier way instead or using outside methods for instance:
SSH into host and upload log files to ftp site
GUI interface apps like WinSCP.

These normally have issues with copying of files due to file being locked while VM is powered on.


Firstly need to note the folder file name for VM:

1. Select VM
2. Select Summary tab
3. On VM hardware window and select "settings"
4. Expand the Hard disks and note the Disk File name provided.

To retrieve the log files following these instructions:

1. Select VM
2. Select Related Objects tab
3. Select Datastores
4. Right click on databastore and select "Browse files"
5. Double click the folder associated to VM which was retrieved in beginning.
6. Highlight each vmware*.log file individually
7. Right click and select "Download from datastore"
8. Select local folder location
9. done!

If your server is unresponsive or crashed you can also export all the log files.  However beware this option will suspend or crash your VM so please use with caution!(KB 1007802)

1. Select VM
2. Right click VM and select "all vCenter Actions"
3. Select "export system logs"
4. Select system logs to export
5. Select finish
6. done

vSphere web client login error: "Client is not authenticated to VMware Inventory Service"

This week received the following error when trying to login to vSphere web client.
"Client is not authenticated to VMware Inventory Service"

Troubleshooting:

Was able to login with local user "administrator@vsphere.local".
Problem seems to be only related to domain user authentication.

Looked at all services on vcenter server, which was running.
Service health on vSphere web client also all green without any errors.

Commvault - Backup Copy of snapshot for Exchange 2010

Been running into an interesting scenario with a backup process which I hope will help someone else.

I have an exchange backup of DAG with intellisnap which creates a snapshot of each of the exchange database volume on Netapp.  The snapshot is then mounted and copied via SAN transport on the backend fiber connection to the storage library on commvault agent.

This has a storage policy to perform a backup copy of the latest snap via SAN transport weekly on Thursdays.

With commvault I find that sometimes the snapshots are not copied to the commvault storage library, without any type of error.

In viewing the jobs for the primary copy of snaphost i just shows a status of "to be copied", but no failure.

This however causes problems since if you do not pay attention you can sit with weeks of uncopied backup copies of the snapshots, and if you snapshots are set for retention of a few days then you loose all those copies and have to mark them bad! (not good!)

Have not find a way to resolve this, even with the help of Commvault support, but here are a few tips to make sure you are at least inform in more detail regarding this process:

• A good indication that the backup copy is not taking place is when the snapshots on netapp volume is not being deleted.
• Also make sure you setup a very detailed report for the day after the snapshot copy was to take place and make sure the backup copy detailed information is also included in this report.
• 
  

If however you do run into this problem, and your snapshots have already been rotated on the volumes on storage you would have to perform the following tasks to continue the backup copy process.

(I recommend contacting Commvault support if you are not familiar with your storage, snapshots and intellisnap, the steps below are provided for use at your own risk, you will loose backup data with these steps)

• Verify you oldest snapshot date which are still on volumes for databases
• Select storage policy
• Right click and primary copy and select view -> jobs
• this will list all the backup copies and show the ones with status "to be copied".
• Select all jobs up to before the date you have for snapshots on Netapp volume.
• Right click and select "Mark job bad" (i know not my favorite step)
• Right click on the storage policy again and select All Tasks -> Run Backup copy.
• This will start the backup copy of your latest snapshot which resides on backup storage volume

Another solution is to manually add snapshots taken during the week to be added for backup copies.

• Right click on the storage policy again and select View -> Jobs
• Right click on the snap you want to create backup copy off.
• Select "Pick for backup copy"
• As a note, you cannot create a backup copy of snapshot which is older than your latest backup copy. so make sure you backup your oldest snapshot first . 

Anybody else running into the same problem would love to hear from you.

Dell DRAC and Internet Explorer 11

I don't think i have ever had more problems that i have had with my IE browser and Dell DRAC interface :)

Seems that there is always something broken and can either never get logged in, or show me the turning circle of death or just a blank white screen.

I am still a big IE browser user, yes yes I know, mainly due to compatibility but I think it is getting worse with newer versions and the whole implementation of their compatibility view is just crazy. 

Already downgraded from IE 11 back to IE 10, hence why i am writing this email since for the life of me I could not get the DELL DRACs to work. Alas I am back on IE 11 and have this working..

vCloud Director: Error on provider VDC "No datastores accessible to VDC "

This normally happens when a new datastore is added to the storage cluster which is associate to a storage policy for the vCloud Director Organization.  The rule-set (storage capability) was not properly set to the datastore before it was added to the datastore cluster.

All the datastores, datastore clusters and policies disappear and you get the following errors:

Error on provider VDC:
No datastores accessible to VDC

On Provider VDC storage policies tab:
This storage policy is not backed by any usable datastores accessible from this VDC.
There is no usable datastore accessible from this VDC that supports this storage policy. Ensure that this storage policy is backed by at least one datastore in vCenter Server, this datastore is enabled in vCloud Director, and the datastore(s) are accessible by the hosts in the clusters used by this VDC.

vSphere web client 5.5: Add new datastore to storage cluster with associate storage profile.

Adding a new datastore to vCloud storage profile has change just a little bit with vSphere Web client.
refer to my previous post to see the difference:
http://virtualrealization.blogspot.com/2014/01/assigning-storage-profilepolicy.html



Information to know up front:
Storage policy/profile name
Storage capability (Rule-Set 1 Tag) - vSphere Web migrated the capabilities to Tags

Steps:

Backend storage:

1. This is normally handled by storage team which will provide the LUN.
2. Since I am also the storage engineer I created the LUN from my Dynamic Pool on the Hitachi AMS 2100.

Within vCenter server web client:

1. Right click the ESXi host from vCloud datastore
2. Select "all vCenter actions"  from menu -> Rescan storage
3. Only need to check the Scan for new Storage devices.
4. Rick click the ESXi host again
5. Select "New datastore".
1. Type - VFMS or NFS depending on block or file level storage.
2. Datastore name
3. Select version
4. Partition size
6. Go to vSphere web client home page.
7. Select VM storage policies which will be associated to this datastore.
8. Edit the VM storage policy, select Rule-set and write down the Tag name
9. Right click the newly added datastore 
10. Select "Assign Tag"
11. Add the Tag to datastore for association to VM Storage policy.
12. Now you can add the new datastore to the Datastore cluster.

Assigning Storage Profile/Policy: Difference between vSphere client and web client

The vSphere client makes use of the following terminology:
Storage profile
Storage capabilities

The vSphere web client makes use of the following terminology:
Storage policy
Rule-Set through TAGS

The main thing to take note of here is that Storage capabilities created in vSphere client is migrated over to TAGS within vSphere Web client.  These tags are then assigned to the VM Storage Policy through the rule-set.

.

vCloud director 5.5: Catalogs problem #3 - Assigning catalog media files to different VDC within an ORG?

We have a public catalog which is shared to all ORGs within vCloud.  This also contains some ISO media files for users to mount to cdrom of virtual machines.
However in order for a user to make use of a media file from a public catalog, that media file firstly needs to be copy to the local catalog of that ORG.

Within this ORG we have multiple vDCs.  This is where things get interesting.

Scenario:

When a vAPP is deployed from catalog you assign this vAPP to a specific vDC.
When you then try to mount an ISO from the local catalog media files, you are unable to do so since this media file is associate to a different vDC...mmm

Troubleshooting:

I believe the problem lies with how the Catalogs are done in 5.5.  When creating or copying files into a Catalog, there is no method to accurately control where the item will be stored. This is more critical when there are multiple Org vDCs, and a file could be placed in an Org vDC to which the file is not intended for.

This is a departure from vCloud Director 5.1, where importing media allowed you to select a specific Organization vDC. Even when you could select a vDC, you still needed to have the ISO image in the vDC where the VM resides to be allowed to insert it.

I had a lot remote sessions and discussions with support on this issue since we could not find a way to assign a media file to specific VDC.

Workarounds:

Option 1 -  Temporarily disable storage profiles before doing an ISO import. This would eliminate certain vDCs as an option, and force the selection process to go to the remaining vDC.  this would be my recommendation.

Option 2 - Set the catalog to use a specific Storage Profile, from the desired Organization vDC and import the media again. This should send the imported files to the correct ORG vDC. You will need to import Media files multiple times to accomplish this.  This only possible if you have multiple storage profiles which I currently do not have.

Option 3 - Have a catalog specific to an individual ORG vDC, name the catalog relative to the ORG vDC, and select the Storage Class/Profile from the same ORG vDC. This means that any ISOs will be sent up to the correct vDC when you import to a given catalog.  On this option here are the steps to setup more catalog for more than one vDC:
1. Create a "master" catalog in the ORG on the Storage Profile vDC A
2. Publish the Catalog created in Step 1 externally
3. Create a "slave" catalog in the ORG on the Storage Profile vDC B, and have this catalog sync from the catalog created in step 1.
4. Repeat this to have Catalog C on Storage Profile C, and sync to Catalog A.
5. Whenever you add a new ISO image to the Master Catalog, it will automatically update the others in 24 hours or when a sync is requested.

vCloud director 5.5: Catalogs problem #2 - Error mounting catalog media ISO files to CDROM - "This media type is incorrect. Media type ISO was expected." (known issue)

Catalogs has had quiet a big revamp in vCloud 5.5 with some enhancements.
This can be read in the "whats's new" pdf:
http://www.vmware.com/files/pdf/products/vCloud/Whats-New-VMware-vCloud-Director-55-Technical-Whitepaper.pdf

We have a public catalog which is shared to all ORGs within vCloud.  This also contains some ISO media files for users to mount to cdrom of virtual machines.
However in order for a user to make use of a media file from a public catalog, that media file firstly needs to be copy to the local catalog of that ORG.

Problem:

A users copies the media ISO file to local catalog however the file is not copied over correctly.

When users tries to attach the newly copied file from media to CDROM, the following error is observed:

"This media type is incorrect. Media type ISO was expected."

vCloud director 5.5: Catalogs problem #1 - browser crashes on "copy to catalog" (resolved)

Catalogs has had quiet a big revamp in vCloud 5.5 with some enhancements.
This can be read in the "whats's new" pdf:
http://www.vmware.com/files/pdf/products/vCloud/Whats-New-VMware-vCloud-Director-55-Technical-Whitepaper.pdf

We have a public catalog which is shared to all ORGs within vCloud.  This also contains some ISO media files for users to mount to cdrom of virtual machines.
However in order for a user to make use of a media file from a public catalog, that media file firstly needs to be copy to the local catalog of that ORG.

Problem:

When a user log in directly to their respective ORG
Select the public catalog and go to media tab
Right click on media file select to "copy to catalog"
The browser crashes and restarts without any error messages.

Troubleshoot:

This was tested on MAC and Windows with each of the following browsers including Chrome, Firefox and Internet explorer.
Same problem appears when logged in as ORG or system admin.

However it works fine when I login as system admin to main vCloud Director login page.  Only seems to crash browser when logged in direclty to any ORG site.

Solution:

Do not share the catalog with the radio button "all organizations", but rather select all the ORGs individually from the list.

I do not experience the same results as listed the following KB but same solution solves both problems:
http://kb.vmware.com/kb/2063431

Pretty Exciting news: VMware acquires AirWatch!

Being an AirWatch employee and also working with VMware products for 6 years I am very excited to be joining VMware and cannot wait to see what the future holds.  This is a great fit for both companies and a privilege to be part of this.

VMware to Acquire AirWatch for $1.54 Billion http://ow.ly/sPEX1 via @WSJ

AirWatch is thrilled to be joining VMware as our companies take a giant step forward in leading the Mobile-Cloud industry http://ow.ly/sPGj7

VMware 5.5 error - Invalid datastore path ''. - When creating new Virtual Machine

A new problem I found in VMware 5.5 is when you try to create a new Virtual Machine.

Debugging the problem:

I have never had any issues with creating new Virtual machines from scratch.  However i had to create a new OS template and started the process but received error:  Invalid datastore path.

Looking online I did seem to find this error popping up alot but not related really to my problem.
I did test the following:
•Verify that datastore mentioned in the error is connected and accessible.
•Verify that the file exists in the datastore.

•Ensure that the file name does not contain non-Basic Latin characters.

Kind of figure the problem storage or permission from error message so  I went through my custom VM steps and under storage selection I always choose the Storage cluster and keep storage DRS enable so VMware make the recommendation of where to place the disks.  This is where the problem lies for some reason....

Resolution:

When you create the VM and under storage selection if you choose the storage cluster, make sure you disable Storage DRS for the VM and select the datastore manually.

This seems to have fix my issues and I was able to create the VM successfully.

Migration to host failed with error Already Disconnected (195887150)

This is the first time i ever ran into this problem after installing a new ESXi host.  Added the host to our current vDS, moved into cluster and disabled maintenance mode.
Thought might be related to our newly upgrade 5.5, however seems a lot of people ran into same problem.

ERROR:

Migration to host <10.33.29.51> failed with error Already disconnected (195887150).
vMotion migration [169942327:1389120124781890] failed writing stream completion: Already disconnected
vMotion migration [169942327:1389120124781890] failed to flush stream buffer: Already disconnected

vMotion migration [169942327:1389120124781890] socket connected returned: Already disconnected

VMware KB found on problem:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1030267

I tried all the regular fixes:

• vmkping to and from the vmkernel port used for vmotion
• verify DNS
• restart the management agent on host given error.
• /etc/init.d/hostd restart
• /etc/init.d/vpxa restart
• Verified time on the both hosts
• Turned off Enable Migration in the Advanced Settings of the host and re-enabled it.
• vSphere web client -> Manage tab -> Settings tab -> Advanced System Settings
• Search for "migration"
• Find Migrate.Enabled and set to 0(disabled)
• Unchecked vmotion on the vmkernel port and re-enabled it.

Still this did not resolve my problem.

RESOLUTION:

Change the IP address of the MVkernel adapter for vMotion traffic.  This seems to fix the problem.  I then  also changed the IP back to original and problem was still resolved.

The vMotion VMkernel adpater IP address was previously used but not on this server since new server.

Cannot use vSphere Client to edit the settings of virtual machine of version 10 or higher

After upgrading to VMware 5.5.0 and you are also considering upgrade the Virtual machine compatibility from v9 to v10 I do recommend performing a snapshot or backup of the VM before hand:

You are no longer able to edit your Virtual machines through the vSphere client in v10!

If you still have users/employees connecting through the regular vSphere client, you might want to consider this or have them access the web client going forward.

Here are the hardware version features to consider:
http://pubs.vmware.com/vsphere-55/index.jsp?topic=%2Fcom.vmware.vsphere.vm_admin.doc%2FGUID-789C3913-1053-4850-A0F0-E29C3D32B6DA.html

http://www.vmware.com/files/pdf/vsphere/VMware-vSphere-Platform-Whats-New.pdf

Links:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2061336

vCloud Director 5.5 - Replace current self-signed certificates with signed certificates

Initially I had very little time to get vCloud Director up and and running and just created self signed certificates.
After upgrading from vCloud 5.1 to 5.5, we found a known problem with VMware where MAC desktops cannot access VM console when using self-signed certificates.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2058496

This provided the perfect opportunity to replacing initial self signed certificates with newly created signed certificates. (internal root CA server)