PSC : Firstboot script execution error

I installed a test PSC today and right at the end of the installation an error popped up "Firstboot script execution error".

After looking through the log files I found the following:

VMware Appliance Configuration...\", \n        \"translatable\": \"Starting %(0)s...\"\n    }, \n    \"warning\": [], \n    \"error\": {\n        \"resolution\": {\n            \"id\": \"install.ciscommon.validatePNID.resolution\", \n            \"localized\": \"If the supplied system name is a FQDN, then make sure the DNS forward lookup results in at least one valid IP address in the system. If the supplied system name is an IP address, then it should be one of the valid IP address(es) in the system.\", \n            \"translatable\": \"If the supplied system name is a FQDN, then make sure the DNS forward lookup results in at least one valid IP address in the system. If the supplied system name is an IP address, then it should be one of the valid IP address(es) in the system.\"\n        }, \n        \"detail\": [\n            {\n                \"args\": [\n                    \"jpsctest01.sovsystems.com\"\n                ], \n                \"id\": \"install.ciscommon.validatePNID.error\", \n                \"localized\": \"The supplied System Name jpsctest01.sovsystems.com is not valid.\", \n                \"translatable\": \"The supplied System Name %(0)s is not valid.\"\n            }\n        ], \n        \"componentKey\": \"visl-integration\", \n        \"problemId\": \"install.ciscommon.validatePNID\"\n    }, \n    \"progress\":0\n}","isFinal":"true"}
2016-02-22 16:52:16.814728 Progress Controller: [VCSA ERROR] - First Boot error


Solution:


In my haste for testing i forgot to setup the A-records in DNS for the new PSC appliance.
The problem can also be related to providing the wrong DNS name during the installation wizard.

SRM 5.8: Synchronize storage freezes at 90%

SRM 5.8 with storage array replication VNX mirrorview.

Scenario:
Run a recovery and once completed run reprotect.
During the reprotect the storage synchronization gets stuck at 90%.

No real information from SRM on the status or errors so had to do some digging.  

Solution:
On the storage array reviewed the replicated LUN for the specific recovery plan and found that the the secondary image was showing "waiting for administrator to start synchronization".

By default SRM queries an ongoing synchronization every 30 seconds to report status so after selecting synchronization and its completion did the SRM status also update and completed.

This setting is adjustable in the SRM advanced settings per site:  storage.querySyncStatusPollingInterval.

vCD-SP 8: vCenter Server status error

Had issue this week come up where vCloud Director is unable to communicate with vCenter server.  I had this many years ago but seems to be still relevant in vCD-SP8 so thought just write up about it with reference to good information:

In my environment this caused template deployments to fail with "Cannot retrieve list of the supported Operating Systems from Provider VDC "PVDC". Please ensure vCenter is connected and available."

Solution:

After reviewing vCenter Server vSphere resource under Manage and Monitor I found that vCenter is not connected.
KB 1035506 from VMware documents this issue but request that you call support to get this resolved.

The issue seems to be related to quart table and I am not going to get into much details here since Jason Boche wrote a nice detailed article how the problem comes about and how to resolve it.

http://www.boche.net/blog/index.php/2011/12/16/vcloud-director-and-vcenter-proxy-service-failure/

vSphere Web client 6.0 missing license UI

Found that on our recently upgrade vCSA 6.0U1 the license UI was missing.

Found a detailed KB article from VMware on this but they reference this happens when you have a proxy in place, which we don't so seems this problem can potentially affect more environments.

Solution for vCSA:

Stop the vSphere Web Client service by running:
service vsphere-client stop

Remove the contents of the vSphere Web Client work directory by running:
rm -rf /usr/lib/vmware-vsphere-client/server/work/*

Remove the contents of the pickup directory by running:
rm /usr/lib/vmware-virgo/server/pickup/*


Back up the following files that are located in /usr/lib/vmware-vsphere-client/plugin-packages/vsphere-client/plugins/:

• telemetry-service-6.0.0.jar
• telemetry-ui-war-6.0.0.war
• phonehome-collector-ui-war-6.0.0.war
• cis-data-service-cmc-6.0.0.jar
  

Remove the following files that are located in /usr/lib/vmware-vsphere-client/plugin-packages/vsphere-client/plugins/:

• telemetry-service-6.0.0.jar
• telemetry-ui-war-6.0.0.war
• phonehome-collector-ui-war-6.0.0.war
• cis-data-service-cmc-6.0.0.jar
  

Start the following vCenter service by running:
service vsphere-client start

 

Links:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2135810

vCD-SP 8: Blank disconnected console or shows unavailable error

Verify that you have your Public Addresses entered in vCloud director:

System -> Administration

Under system settings select Public addresses

Fill in the necessary information, especially the FQDN for vCD public console proxy address.

vCD8-SP: Wildcard certificate

I have a wildcard.pfx certificate which I need to use in my vCD instance.  Here are the steps to convert the certificate and import into vCD:

1. Convert the pfx to pem:
1. openssl pkcs12 -in certificate.pfx -out certificate.cer –nodes
2. Extract the private key from the certificate.cer.
1. copy from -----BEGIN PRIVATE CERTIFICATE----- 
2. to end of -----END PRIVATE CERTIFICATE-----
3. Create new file called certificate.key and paste 
3. Recreate pfx and set alias for http
1. openssl pkcs12 -export -in certificate.cer -inkey certificate.key -name http -passout pass:yourpassword -out http.pfx
4. Recreate pfx and set alias for consoleproxy
1. openssl pkcs12 -export -in certificate.cer -inkey certificate.key -name consoleproxy -passout pass:yourpassword -out consoleproxy.pfx
5. Import the 2 PKCS12 keystores into Java keystore using keystore
1. ./keytool -importkeystore -srckeystore http.pfx -srcstoretype PKCS12 -destkeystore certificate.ks -deststoretype JCEKS -deststorepass yourpassword -srcalias http -destalias http -srcstorepass yourpassword
2. ./keytool -importkeystore -srckeystore consoleproxy.pfx -srcstoretype PKCS12 -destkeystore certificate.ks -deststoretype JCEKS -deststorepass yourpassword -srcalias consoleproxy -destalias consoleproxy -srcstorepass yourpassword
6. Now import the root and intermediate certificates (if any) to the same keystore
1. ./keytool -importcert -alias root -file GeoTrust_Global_CA.cer -storetype JCEKS -keystore certificate.ks -storepass yourpassword
2. keytool -importcert -alias intermediate -file RapidSSL.cer -storetype JCEKS -keystore certificate.ks -storepass yourpassword

To replace it your can read my previous blog post

vCD-SP 8: Replace certificates

I currently have some self signed certificates on my vCloud Director 8 installation and want to update them with new certificates.  Here are the simplified steps to get this accomplished:

Firstly you need to create 2 certificates for each member of the group (cell) and import the certificates into host keystores.  Each vCD has 2 IP address which allows support for 2 different SSL endpoints(http and consoleproxy).  Each endpoint requires its own SSL certificate.

Requirements for cert include an X.500 distinguished name, while Subject Alternative Name is not necessary.

Replace certificate using vCD configuration script:  (this does not work in 8.10 anymore)
this process will also validate the db connection and prompt for SSL certificate and skips all other.

Before doing any work, take a snapshot of your VCD Cells and backup your database.

1. SSH to vCD cell
2. Stop the vCD services
1. service vmware-vcd stop
3. Run the configuration
1. /opt/vmware/vcloud-director/bin/configure
2. Specify full path to java keystore that holds the new certificates
3. Provide keystore and certificate password

This will replace the certificates and restart the vCD services.

Replace certificates using the certificates command:

Certificates command of the cell management tool automates process replace certificates in JCEKS keystore.

1. # cd /opt/vmware/vcloud-director/bin
2. # ./cell-management-tool certificates -j -k /tmp/<certificate-file-name>.ks -w keystorepassword
3. Restart the cell for changes to take affect.
1. # service vmware-vcd restart

vCD-SP 8 - what's new

I finally got around to installing vCD-SP 8 and here are my finding on what's new and upcoming.

• Support for vCenter 6.0
• ESXi and vCSA 6.0 U1 is a big advantage to 5.5 and allow SP's to take advantage of new features and optimization provided with 6.
• Support for NSX 6.2 and 6.2.1!
• This allows connection to a NSX manager instead of a vShield Manager appliance in backward compatibility mode.
• When using NSX manager the VSE is deployed with version 5.5.4-2673026
• vDC template
• Create Organization specific virtual datacenter templates with pre-set resources delegation rule-set.  Have not tested this but provide some feedback later. 
• vApp enhancements
• Reconfigure of VMs within a vApp.
• Reconfigure of networking connectivity and capabilities during creation.
• I have not seen anything from a UI perspective on this but seems to be more under the cover work through API.  
• oAuth for identify source
• tokens now supported.
• Tenant throttling
• This prevents noisy tenants from consuming all the resources of a single instance.
• Helps ensure fairness of execution and scheduling of among tenants
•  
  

As mentioned in previous blog VMware did post about new UI changes coming to vCD-SP in 2016 and the following advanced networking selections seems to indicate this.  I did again not find anything in documentation on these menu selections but hopefully seem some of vCloud Air features soon:
https://www.vmware.com/au/cloud-services/infrastructure/vcloud-air-advanced-networking-services

Links:

http://pubs.vmware.com/vcd-80/index.jsp#com.vmware.vcloud.api.sp.doc_90/GUID-375148E9-601D-4BFA-9C10-4D2595D96D95.html

http://pubs.vmware.com/vcd-80/index.jsp#com.vmware.vcloud.api.sp.doc_90/GUID-AA9C56E3-B7AA-42E3-9AE0-46233E1FDD2A.html

http://pubs.vmware.com/vcd-80/index.jsp#com.vmware.vcloud.api.sp.doc_90/GUID-74126E65-F99E-4128-A023-0CCF88B04832.html

https://www.vmware.com/au/cloud-services/infrastructure/vcloud-air-advanced-networking-services

vCD-SP 8 - Failed to create external network as port group dvportgroup-XXX or its VLAN ID XXX is in use

I just received the following error in my lab when i tried to create a new external network.  This seems to be a known issue in vCD 5.1 & 5.5 but has not been resolved in v8.

I did not delete any port groups or VSE's directly from vCenter server so assume the database was not successfully delete when i removed external networks.

VMware provides a comprehensive KB article on this so not going to get into to much details but solution is to remove the stale entries from Resource_Assignment table in the database.

KB: 2042474


Links:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2042474

vCSA 6 postgreSQL - connect externally through pgAdmin

I recently had to query the VCDB database of vCenter Server 6 but had an appliance deployed and since not using a MS SQL database server I had to do some digging to figure out how i can get access.

My favorite tool to query the database I found to be pgAdmin III but this is installed on my jump server so here are the steps i following to allow pgAdmin to connect to the internal postgresql database on vCSA 6!
http://www.pgadmin.org/

SOLUTION:

1. Enable SSH for vCSA.
2. Login as root
1. shell.set --enabled True
2. shell
3. View following 2 files for information on database installation
4. /etc/vmware-vpx/embedded_db.cfg
1. General server information and password for superuser. 
5. /etc/vmware-vpx/vcdb.properties
1. Stores connection information for vCenter server database VCDB (password for vc user
6. Edit /storage/db/vpostgres/pg_hba.conf
1. Add following lines to add your own subnets to be able to connect to PG
2. Host                   all              all                  <ip address range>/<subnet>           md5
7. Edit /storage/db/vpostgres/postgresql.conf
1. Add the line to end of file:   listen_addresses = '*'
2. Restart posgresql /etc/init.d/vmware-vpostgres restart
8. Open port on vcsa appliance firewall:
1. iptables -A INPUT -p tcp -m tcp --dport 5432 -j ACCEPT

This should do it and allow you to connect externally through the nice pgAdmin GUI to your database.