Just to lot you all know virtual-portal.com will no longer be active and I will continue all of my blogging activity over at stevenbryen.co.uk

Thank you for the continued support and visits to this blog. I hope to be more active over at the new blog and hopefully write some entertaining stuff that will keep you reading.

Thanks

Steve

I can confirm that today I sat the Cloud Architect Specialist Exam (Virtualised Infrastructure) and Passed

I want to talk a bit about my preparation for the Exam and why I decided I wanted this Certification.

So Why?
Some of you may have noticed that a few weeks ago I passed the EMCISA exam. This is required for the Cloud Architect Cert and I have actually been studying for this exam on and off for a few months. It was something I wanted to complete before joining starting my new job, however due to a busy handover period at my last place it kinda slipped. Anyway, I eventually got it out the way and as I had some previous experience in using Enterprise storage this was a good refresher for me on storage technologies.

I decided to do this as a stepping stone to get to the Cloud Architect Exam. My Previous Role was working for a Cloud Service Provider, so I felt that I had a good idea of business drivers, available technologies and cloud concepts. The certification hopefully would be a nice way to test myself on this.

Preperation
The main thing I did to prepare for this course was to attend the instructor Led training from EMC. I would recommend this to anybody who is looking to achieve the certification. It is not a very technical course, however it seemed it was designed to generate various debates and discussions within the class. I was fortunate to be on the course with some really experienced people from both EMC and External companies, so we had some great discussions around VDC and Cloud.

Another thing to mention is that this course is worth attending just for the course manual. It has all areas covered that will pop up in the Exam and was my main Prep resource after completing the course.

Finally I used the EMC online practice tests to prepare for the Exam and as usual these are a great reflection of the types of questions you will have to answer. Obviously these are not brain dumps of the actual Exam questions, so it is probably not worth digging really deep into the specific answer for the questions that you got wrong on the exam, but it is really good to see which modules you need to study more.

The Exam
For obvious reasons I cannot talk too much about the actual exam content, but I would just like to mention that the exam was a lot tougher than I expected after the course. Just make sure that you really study the course material, it may glaze over a technology product or feature on one slide or two, and it could so happen that 2 or 3 questions in your exam are based on that technology. So please do make sure that you cover all areas of the book. Unfortunately I can’t say no more except for Good luck to anybody who is taking the exam after reading this post .

As of next month I will be joining VCE as a Senior Solutions Engineer, EMEA. I am very happy and want to thank VCE for giving me such a great opportunity. The decision wasn’t a hard one, VCE are a forward thinking company, backed by Industry giants and using best of breed technologies across the board. I truly cannot wait to get started.

That said, I have had a great time working for Carrenza and had exposure to a huge ammount of technologies and People. I would like to thank the guys at Carrenza for understanding my decision, they are a brilliant company, and i would definietly reccomend them to anyone.

Steve

p.s. Don’t worry… My blog and Twitter feeds will both remain active

“We’re pleased to designate you as a vExpert 2011 as recognition of your contributions to the VMware, virtualization, and cloud computing communities. You’ve done work above and beyond, and we’re delighted to communicate more closely, to share resources, and to offer other opportunities for greater interaction throughout the year as we continue to grow knowledge and success in the community of IT professionals. Welcome to the vExpert 2011 Program!”

And I must say I am so Happy to be included in this Program. There are some ridiculously smart people that have been selected for the Program this year and I am looking forward to being a part of it.

I would like to thank John Troyer and his vExpert Team for picking me during the selection process, and also anybody who nominated me.

I would like to say congratulations to all the other vExpert’s for 2011. And Finally a special Congratulations to all of the other London VMUG vExperts, you have all contributed to the User Group masively and It has been a pleasure to be a member.

Although a small message is shown when you change the settings (shown below), these settings are not propogated down to the existing vApp VM’s until the vApps have been restarted.

Message:
“These settings only take effect when you start the vApps in this Organization’s vDCs.”

This is rather annoying as I am sure you all know that reservations can be changed whilst a VM is powered on via the vSphere client/PowerCLI etc. And this is still true with vCloud Managed VM’s. So the way to get around this is to change the Allocation % in the ORg VDC Settings and then also go into each VM in that VDC from the vSphere client or PowerCLI (May write script and update later) and change the reservation manually also.

There is a workaround for this explained on the following KB Article 1024500

The main issue with this is that when upgrading to ESX 4.1 Update 1 unless you have performed the workaround, the issue still persists. This is a bit of a pain, as the quickest option is probably to rebuild your ESXi hosts, But i have wrote a powershell script that will perform the workaround for you, so that you can continue to run as normal. it doesnt take very long at all to run around 10 seconds maximum per ESX host and will perform all of the changes needed.

This script uses plink.exe to connect to ESXi Hosts using ssh (link further down), If you have not connected to this ESX host using ssh before then you will need to accept the host keys first. Here is a link to a batch file that i wrote that will allow you to ssh to list of servers from the command prompt and accept the host keys one by one to speed things up (also requires plink):

Batch Script – plink

The Steps that thsi script takes are:

1. Connect to vCenter and get an array of ESX hosts running version 4.1
2. Enable Remote Tech Support on All those hosts
3. Disconnect from the vCenter Server once data collected
4. Loop through ESX Hosts and perform changes to console files using plink.exe
5. Connect directly to Each ESX host and change password, this will change encryption back to MD5
6. Connect Back to vCenter Server
7. Disable Remote Tech Support Mode
8. Disconnect from vCenter

Things to Note:

1. You will require plink.exe. Downloadable in executable form putty site… here
2. This has only been tested on ESxi not Classic ESX
3. Output will be stored in text file specified in $outfile variable
4. Unfortunately when running plink.exe in powershell there are lots of nasty error looking outputs, you can ignore most of these, please check back through the console to see if there are any issues.
5. Script assumes vCenter Access is via Windows Credentials.

NOTE: Currently there is no validation, If a host has already been fixed this could cause issues.

The Script:

#Specify Variables
$outfile = "C:\Temp\plinkoutput.txt"
$vcenterServer = "vcenterName"
$User = "root"
$Pswd = "password"
$Computer = ($vmh.Name)
$plink = "C:\Temp\plink.exe"
$plinkoptions = " -v -batch -pw $Pswd"

echo "Starting Script" > $outfile
# Connect to virtual Center, Gather Hosts and Enable TSM-SSH Service on All Hosts
connect-viserver $vcenterServer
$vmhosts = Get-VMHost | where {$_.ApiVersion -eq "4.1"}

foreach ($VMHost in $vmhosts) {
	echo "Enabling SSH service on $VMHost" >> $outfile
   Get-VMHostService -VMHost $VMHost | where {$_.Key -eq "TSM-SSH"} | Start-VMHostService
  }

#Disconnect from vCenter to allow individual host connections later
Disconnect-VIServer $vcenterServer -Confirm:$false -Force:$true

#Loop through ESXi Hosts and perform checks
foreach ($vmh in $vmhosts) {
	echo "Fixing Security Issue on on $vmh" >> $outfile

	#Remote Commands
	$Command1 = "chmod 777 /etc/pam.d/system-auth"
	$Command2 = "sed -e '/password.*pam_unix.so/s/$/ md5/' -i /etc/pam.d/system-auth"
	$Command3 = "echo `" sed -e '/password.*pam_unix.so/s/$/ md5/' -i /etc/pam.d/system-auth`" >> /etc/rc.local"
	$Command4 = "chmod 444 /etc/pam.d/system-auth"

	#Run Command 1 - Change File Permissions on /etc/pam.d/system-auth
	$command = $plink + " " + $plinkoptions + " " + $User + "@" + $computer + " " + "`"" + $Command1 + "`""
	$msg = Invoke-Expression -command $command -ErrorAction SilentlyContinue

	#Run Command 2 - Add md5 to system-authd file
	$command = $plink + " " + $plinkoptions + " " + $User + "@" + $computer + " " + "`"" + $Command2 + "`""
	$msg = Invoke-Expression -command $command -ErrorAction SilentlyContinue

	#Run Command 3 - add the sed command from above to /etc/rc.local
	$command = $plink + " " + $plinkoptions + " " + $User + "@" + $computer + " " + "`"" + $Command3 + "`""
	$msg = Invoke-Expression -command $command -ErrorAction SilentlyContinue

	#Run Command 4 - Change File Permissions on /etc/pam.d/system-auth
	$command = $plink + " " + $plinkoptions + " " + $User + "@" + $computer + " " + "`"" + $Command4 + "`""
	$msg = Invoke-Expression -command $command -ErrorAction SilentlyContinue

	#Connect Directly to ESX host to change password
	connect-viserver $vmh -user $User -password $Pswd -ErrorAction SilentlyContinue | Out-Null
	Set-VMHostAccount -UserAccount $User -Password $Pswd | Out-Null
	Disconnect-VIServer ($vmh.Name) -Confirm:$false -Force:$true
}
# Connect Back to virtual Center and Disable TSM-SSH Service on All Hosts
connect-viserver $vcenterServer
foreach ($VMHost in $vmhosts) {
   Get-VMHostService -VMHost $VMHost | where {$_.Key -eq "TSM-SSH"} | Stop-VMHostService -Confirm:$false
  }
Disconnect-VIServer $vcenterServer -Confirm:$false -Force:$true

Don’t get me wrong, It is really cool that companies like VMware, Dynamic Ops, Abiquo etc are deveoping tools that allow me to automate the deployment of Virtual Machines t my customers. They have created self service portals and request mechanisms that all fit nicely together and are compatible with my existing infrastructure, which is a good step in the right direction when it comes to cloud computing.

But say that I am a large corporate, or a Service Provider and I would like to build a Cloud (Private or Public) there is still a huge task involved in provisioning Hardware so that it is ready for me to even start considering deploying a vApp or Template. Yes, you got it the gap I feel is in the hardware provisioning stage of the process. I feel that there are not as many tools out there to assist with this. If i want to build a private cloud, I need to configure my Storage, Network, Compute and Security devices all before I can even look at a tool like VMwares vCloud Request Manager.

How do I configure my Cisco Switches for Layer 2 Networking? How do I install ESXi onto my HP Blades? and how to I then configure my MDS Switches and 3PAR Array to present LUNs to this new hypervizor?

Maybe I am being a bit ambitious today, but I would love to see a solution where I could order a rack of hardware, go to a Web App, fill in a few fields and have a rack configured with Storage, Hypervizor, Networking, and then even move on to automate the installation of a vCloud Cell for that Rack of hardware, all ready with new Clusters and pVDCs. OK ill wake up now…. but honestly I don’t see why this isn’t achievable over the next 6-12 months.

I have heard people say great things about what the VCE Coallition are doing behind doors when it comes to this, because I have been asking myself for a long time “When I order a vBlock pre configured, does that just mean that some other guy is sitting at his laptop mounting ESXi images to blades and telnetting to Cisco switches instead of me?”. Maybe some VCE members can help me out here but when i thought hard about it they seemed like the only people who would be able to offer such a solution today(HP in future since 3PAR Acquisition).

This is due to one of the main reasons with developing such a tool, You really need to know which specific hardware you are going to be buying. Unless some company has the resource to develop a deployment app which covers all vendors products in all 4 markets? This i think will lead to cloud providers buying whole in rack hardware solutions like vBlocks in the future. Unless they develop an in house tool to manage their own custom hardware stack.

The point I am getting at here is that for people looking to build a Cloud, and if they are looking to grow it, there is a huge amount of operational work involved. And today i believe only VCE (possibly EMC UIM, which i have no experience with) and HP (with 3PAr/c7000 etc) are capable of creating such tools.

I really hope this is something that we see coming out of these big vendors in the near future, I mean its nice to have a vBlock arrive preconfigured, but wouldnt it be nice to have a tool to configure it yourself, or even nice for EMC to save money on guys sitting there building hypervisors all day (if thats how its done).

If anybody reading knows of any companies that are doing this or looking t doing this then please do leave a comment, I would be more than happy to explore the product.

Error creating Shield network appliance.
- vCloud-Shield edge error: Creating/configuring the VR failed:
vsmHandle.initializeEdge() net:104950669/network-6874 vse:vm-7424 VSM IP:192.168.1.10 failed.
- HTTP/1.1 400 Bad Request
Code : 70508, Description : vShield Edge is already installed.

Internal Server Error
- Unable to start vApp “vApp_test2″.
- Unable to start virtual machines in resource pool “1955899465-Organisation1-oVDC1″.
- Error creating Shield network appliance.vClould-Shield edge error:
Creating/configuring the VR failed: vsmHandle.initializeEdge()
net:1669311005/network-6874 vse:vm-7270 VSM IP:192.168.1.10 failed.
- HTTP/1.1 400 Bad Request
Code : 70508, Description : vShield Edge is already installed

Both of these error messages were very similar and both presented the smae error code, so we decided to approach these both as one issue. My initial thought was that there was anothe cached entry somewhere that was telling vCloud Director that the Edge Device was already installed. After contacting VMware support it seems that this is a known issue with their engineering team. To solve this issue follow the steps below.

1. Firstly Log into the vShield Manager Web Itnerface as an Administrator and from the drop down menu in the top left corner select Secured Port Groups.

SCREEN001

2. Select the portgroup that the vshield edge device is trying to route between.

NOTE: You may have to do some digging to figure this out. Look at the Portgroups assigned to your Network Pools and you may even need to go into vCenter and find currently deployed VSE devices and check which Portgroups are legitimately in use.

3. Once you have found the affected port group select it in the vShield Manager GUI and select the Edge tab from the right hand pane. Please ensure that there are no Virtual Machines in this portgroup and it is definitely not in use before proceeding.

4. If you see a blank screen then this is vShield Manager playing up, hit the refresh link in the top right corner and wait for the information to reload. (This took around 40-60 seconds on an affected portgroup)

5. Once the page has refreshed if you see a screen like the one below then the vShield manage can see an entry for the Edge device but cannot find it in vCenter.

SCREEN002

6. Press the uninstall button and wait for the Task to complete sucessfully.

Once this has been completed on all affected VLANs, then go back to the vCloud Director GUI and attempt to create a Routed Network of any kind once again.

I hope this helped

SCREEN001

My Initial thought was to try and unprepare and then prepare the host again, so that the agent was re-installed. This wasnt as successful as i hopes and I was told that the host was pending an upgrade, when I tried to unprepare it in vCD. (Screen Below)

SCREEN002

Ok, so there is an upgrade option, this looks like the right thing to click, But again no luck. I saw the following error.

SCREEN003

I was now stuck in a loop. I couldnt get this host to unprepare which was a bit worrying. So i spoke to VMware support and they helped me massively. This is a new product so we had some fun going through and experiementing with different possibilities and I learnt a log about vCloud Director during this time.  The final solution is outlined here.

It seemed that the host thought that its agent needed to be upgraded, and aparently this was down to some kind of cached entry in the vCenter database that would be reloaded by removing the host from Virtual Center and re-adding it. The steps below outline how to ensure that this is done gracefully.

First thing was to log onto the ESXi host and check to see if the agent had actually installed at all. This was not the case on 3 of our affected hosts, but on one host it seemed to have some of the files in place. To check if the agent is installed, log onto your ESXi hosts console and do the following:

ls /opt/vmware/

If you see the directory vslad then the Agent is installed in some way shape or form. To remove the agent complete the following:

SCREEN004

Once the uninstall is complete, Check the agent has been uninstalled correctly by checking the vslad.log file as shown below. If it shows the last final messages then we are good to move on.

SCREEN005

Ok, so now you are sure that there is no agent installed on the affected ESXi host we need to complete the following steps on the host:

1. Enter Host into Maintenance Mode
2. Disconnect Host from Virtual Center
3. Remove Host from Virtual Center
4. Go back to the vCloud Director GUI and right click on the affected host and select Unprepare. This should remove it from the list.
5. Return to Virtual Center and add the host back into the cluster
6. Exit Maintenance Mode on the Host (Must wait for this to complete 100% before moving on)
7. Refresh the hosts page in the vCloud Director GUI. You should now see your affected host back in the list.
8. Right click on the host and Select the Prepare Host option. You should see a screen like the one below.. Enter the hosts username and password and click OK

SCREEN006

You should now see your host in the Hosts view with lots of lovely Green coloured ticks next to its name.

SCREEN007

The Installer runs fine without any of the funny unmounting and re-mounting of the ESXi .ISO image.

You can download the Firmware from the following link:

iLO3 Firmware 1.15