vCenter Configuration Manager (VCM): jobs fail to start

Currently having some issue with VCM so wanted to provide some quick information on this particular problem.

Trying to install Agent to a Windows Machine with DCOM but when i start the process the task does not appear in the jobs window.   I can however manually install the agent on the windows machine after which i can successfully connect through VCM.

Troubleshooting:

Verified all of the following

• Collector service is running.  
• My user account has VCM or SQL permissions to run these jobs.
• Network authority account has local admin to windows machine
• DNS and reverse DNS not set up properly for the target marchine or VCM.
• Use nslookup on VCM to validate NetBIOS, FQDN and reverse IP lookup for the target machine.
• DCOM not disabled by GPO on domain
• Firewall not enabled on VMC or client windows machine

Viewing of log files for VCM:

RDP to the VCM server and look at (VCM default install location) C:\Program Files (x86)\VMware\VCM\Tools\EcmDebugEventViewer.

Launch EcmDebugEventViewer.exe and Press F5. This tool will display all the Warnings, Errors, and Exceptions and may reveal some additional conditions we'll need to troubleshoot this issue.

Resolution:

Circled back to agent install task not showing in jobs window at all and traced problem down to broker services not running on my VCM databases.

• Open SSMS for SQL and connect to database server
• Run this query to verify which database has broker enabled:
• select name, is_broker_enabled from sys.databases
• Note: The broker value should be 1 for all databases, except master and model.
• For each database on which the broker is disabled, run this SQL command:
• alter database [Database Name] SET ENABLE_BROKER

What we did find is that if this service has been down for a very long time you could possible receive SQL deadlock errors in the log files and will also be unable to login to environment.  In SQL you will see alot of delete commands by running script "exec_sp3".  Just let this run out and complete.  Could take a couple of hours.

VMworld 2014 recap

Sitting at airport on way back from a very successful VMworld week and thought i would put some my thoughts into a quick post.  I am just going to briefly mention my highlights.  Some of these are existing products and nothing new to most people and then a few new things too.

vCAC

• It is like they say the future and the future is looking good.
• vCloud director only available as service provide offering in near future...

Application director 

• If you ever get a chance then watch this session:
• https://vmworld2014.activeevents.com/connect/sessionDetail.ww?SESSION_ID=2609

RaaS (Recovery as a service) 

• Cloud based disaster recovery with very good RPO & RTO
• http://www.vmware.com/products/vcloud-hybrid-service/

vCloud Air everything

• vRealize Air Automation

EVO:RAIL

• hyper-converged infrastructure appliance.  This is Nutanix territory but excited to see what EVO:RAIL will bring in future.
• http://www.vmware.com/products/evorail

NSX

• Virtualize the network and all the new features including 3de party integration, but the most exciting is the micro-segmentation from a security perspective.

Hands on Lab

• This was amazing and a great success.  Look forward to running though all those lab sessions in the future.

AirWatch

• 8th most taken lab at VMworld and received a lot of attention!

vCOPS alert: analytics resource: number of resources exceeds supported limit

Received the following admin alert in vCOPS:

analytics resource: number of resources exceeds supported limit

This is due to the default max number of resources set for vCOPS is 10 000.
This was not enough of our environment and therefore we had to increase this max number.  The way vCOPS handles resources also contributed to this alert since it does not by default delete non-existant resources.  In another post I provide information on how to change this behaviour:
http://virtualrealization.blogspot.com/2014/08/vcops-admin-alerts-for-vin-adapter.html


Solution:

Add the following line to the file on the analytics server:
"/usr/lib/vmware-vcops/user/conf/analytics/advanced.properties

maxNumberOfResourcesSupported= 999999 (can set your own number here)

restart the analytics server

• SSH into analytics server and login as admin user
• vcops-admin restart

vCOPS: admin alerts for VIN adapter: resources do not receive data from this adapter resource

I was getting a lot of admin alarms within vCOPS for VIN adapters not able to receive data for resources.

VIN adapter instance: 200 resources do not receive data from this adapter resource

I believe the primary reason for this is because my vCloud environment is so dynamic with users deleting and creating new VM's that vCOPS and VIN are not keeping up and non-existent resources are causing the alerts.

By default vCOPS does not delete non-existant resources but I found the following KB 2020638 which provides information on changing the schedule for when deletion of old objects will take place from vCOPS.

Snapshot consolidation failed with error

Virtual machine was showing following alert:

Virtual machine disk consolidation is needed.  Virtual machine consolidation needed status

After trying to consolidate the virtual machine snapshots I received the following error:

Status: An error occurred while consolidating disks: Could not open/create change tracking file

Cannot complete the operation because the file or folder ds:///vmfs/volumes/*.vmdk already exists

Troubleshooting:

SSH to ESXi host and browse the volume where the VM resides on.

cd into the VM folder and found flat files for each of the vmdk disks but the VM does not show that any snapshots exists.

Datastore unmount error: Cannot unmount volume because file system is busy

Just this week ran into this issue while trying to unmount some stale datastore from vCenter server.

Troubleshooting:

SSH into the ESXi host and browse the datastore.
On the datastore I found the following files listed with following names:
vsantraces*

Found KB 2069171 specifying this problem where the VSAN modules stores the traces needed to debug VSAN related problems which is keeping a lock on the datastore.

VIN guest operating system management: limited permissions for vcenter server

vCenter Infrastructure Navigator has a single account which it uses to access VMs.
This is set within vCenter Server's infrastructure navigator screen.

I wanted to limit which datacenters inside vCenter Server VIN would be able to see as the functions this account can perform.

Resolution:

Create a guest operating system management role within vCenter Server with minimal privileges for VIN:

• Navigate to administration
• Select roles
• Click add roles
• Enable the following with checkbox:
• Virtual machine > Interaction -> Guest operating system management by VIX API 
• Virtual machine > Interaction -> Console interaction
• Provide role name
• 
  
• On the vCenter entity root level click manage tab
• Select permissions and add
• Select user and assign the newly create limited VIN role.

This will provide the necessary privileges to enable the discovery process for user selected.

In my case I also did not want this service account to view my vCloud director datacenter so I added the user to the datacenter permissions with "no access" role.

From inventory menu select Infrastructure navigator.
Select settings tab
Here you can now set the new user account.
Make sure to enable access to VMs

vCOPS - Custom UI LDAP error - "One or more users already exist and haven't been imported"

The regular vcops-vshere WEB GUI was easy and straight forward to configure for LDAP authentication since it uses the vCenter privileges.

However vCOPS custom GUI web interface uses a custom configuration for LDAP authentication. I am also using sAMAccountName for Username Field on LDAP settings.  This is not available and have to type in manually.

After setting up LDAP I tried to import the same users but receive the following error message:

"One or more users already exist and haven't been imported"

Java problems with vCenter Orchestrator

All applets and web start java applications has defaulted to high security since Update 11.
The security context that is used by vCO Client is set to high so some changes are needed within the Java control panel.

Resolution:

• Open the Java Control Panel
• Go to the Security tab. 
• At the bottom of the dialog you will see the current Exception Site List. 
• Click the Edit Site List button.
• In the exception entry dialog, enter the URL for your vCO Server

VNX MnR: Not showing SAN data after upgrade

Recently ran the upgrade of VNX monitoring and reporting from version 1.2 to 2.0.

Upgrade completed successfully but after logging in and viewing the data we were unable to view the file storage information.

Resolution:

• Verify NaviSECCLI path is correct in VNX MnR Config 

• If the above is correctly configured, attempt updating NaviSecCLI to latest Version (Found on EMC Support Site under downloads if you search for NAVISECCLI)
• Once Installed issue any command to accept the certificate from VNX This is only required if NAVICLI version is updated 
• Open Cmd Prompt and issue any NAVI CLI Command to be prompted to accept certificate