Recently a new power9 950 was purchased, so it has to be brought into the environment and setup. The specific 950 in question (9040-MR9) is a bit longer than previous versions. This means that things can get quite tight in the back of the server rack, possibly, interfering with the proper closing of the rear doors. There are other server racks out there that accomodate this, and some are basically mod kids to help hack the back of your server to allow an addition 6-12 inches of distance for the cables.
Once this was addressed, the server was racked using the included rack rail kit, then power connected and hooked up to the UPS. Once this was accomplished, there were a number of cables run to support fibre, networking, etc. Proper cable management is a must here, as failure to do so can cause outages, or failure to get support personelle access to the server when required.
Next little task was to get the server hooked up and connected into the Hardware Management Console (HMC). However, the version of the current HMCs were too old. In this case, a physical HMC was used (actually two). These are a dying breed. Thus, both HMCs had to get an upgrade to a new enough version to support the 950. They had to be updated to at least V9R1M940. The next HMC iteration will be novalink, which is a virtual HMC. This is typically coupled with powerVC which can be used to streamline the SAN procurement, partition creation, software networks, etc.
Next piece, login to the HMC. This complained about the ASMI default username / password. So a quick login to that, and setting the ASMI password and system password fixed that. Next in the ASMI, go to system settings and configure the date / time of day. Once the server was booted, it was giving an error that was being captured in the HMC. The HMC error code was referring to an incorrect language layout. This was corrected by hooking up a serial console to a laptop, then set the keyboard and language.
Next we move on to downloading and installing the firmware. This can be accomplished using the system partition. This URL helped with setting that up, and then removing the system partition afterwords.
Power cycle the managed system, otherwise it’ll show as “Resources unavailable”. Next piece was to use the HMC’s template to deploy 2 x vios servers. This is accomplished by deploying the ‘test plan’, using 2 vios & 1 virtual network, indicating the 950 server as the target to deploy to. Providing host names of the vio servers, and the LPARs are created. Then the installation wizard continues. This part gets tricky as it wants to install to a hard drive. As this server was ordered without hard drives, that added a wrinkle into the rollout.
Next step was to install the VIOS, however, w/o a hard drive the idea was to use a SAN boot device. So activate the newly created LPARs and open a console session to both. In each console session, boot into the SMS menus, and determine the location codes for the HBA adapters. Next, reboot the systems, and this time go into the “open firmware prompt”. At the prompt type:
0> DISPLAY_ADAPTER_WWPN [address] where “[address]” is the location code of the found HBA adapter. Once this is completed, you will have the World Wide Name (WWN) of the hba adapter ports. Provide this information to the SAN team (unless you set these things up yourself) to have some storage carved up and presented back.
Next you will want to download the PowerVM VIOS image from IBMs ESS site. Once downloaded, you will use the HMC enhanced+ interface, to upload the ISO (templates and images). Then proceed to do an installios from the HMC. The installios installation through the GUI didn’t work. So, moved on to attempting it via a remote console session from the HMC. So ssh padmin@hmcs_fqdn. Then run: installios and specify some details to proceed with the installation. Details to provide will be where the ISO files are, as well as the networking adapter to use to perform the installation. By default, the ISO image is uploaded to:
/extra/viosimages/<iso_name_you_provided>/dvdimage.v1.iso
If you encounter some errors about unable to install due to an existing label, you’ll have to remove that first. So from the HMC console, run: installios -F -e -R default3 assuming default3 was the one it was complaining on. After this completes, you should be able to launch installios again and get the system installed properly. Usually setting the network settings to auto will work fine.
Nim Server Install
If the installios command above didn’t work, you could try an installation using a NIM server. If so, the best bet is to download and use the powervm vios flash image (if available). The flash image is typically a newer image that has the two ISO images (usually DVD1 and DVD2) combined into one ISO file (and it’s usually a newer SP). Once you have the ISO image downloaded, you’ll want to extract the mksysb_image and the bosinst_data from the ISO and place on your NIM server. Then define some resources. You’ll need a mksysb, spot and bosinst_data resource. Example:
- Download Virtual_IO_Server_3.1.0.10_flash.iso
- scp file to NIM server @ /nim/cd_images/
- mount on nimserver:
loopmount -i /nim/cd_images/Virtual_IO_Server_3.1.0.10_flash.iso -m /mnt -i "-V udfs -o ro" - Copy mksysb file:
mkdir /nim/mymksysb && cp /mnt/usr/sys/inst.images/mksysb_image /nim/mymksysb/ - Define mksysb resource:
nim -o define -t mksysb -a server=master -a location=/nim/mymksysb/mksysb_image vios_31010_mksysb - Define a spot:
mkdir /nim/spot && nim -o define -t spot -a server=master -a location=/nim/spot -a source=vios_31010_mksysb vios_31010_spot - Copy the bosinst_data:
mkdir -f /mnt/bosinst_data/vios_31010_bosinst && cp /mnt/bosinst.data /nim/bosinst_data/vios_31010_bosinst/ - Define bosinst resource:
nim -o define -t bosinst_data -a server=master -a location=/nim/bosinst_data/vios_31010_bosinst/bosinst.data vios_30101_bosinst_data - Add the nim network (if required):
nim -o define -t ent -a net_addr=192.168.5.0 -a snm=255.255.255.0 -a routing1="default 192.168.5.254" -a comments="test_network" 192_168_5_0 - Define a machine for the VIOS:
nim -o define -t standalone -a platform=chrp -a if1="192_168_5_0 vio1 macaddr" -a cable_type1=tp -a netboot_kernel=mp -a connect=shell vio1#Replacing macaddr with the actual macaddress of the VIO server
Install VIOS using the NIM server
First allocate some resources:
nim -o allocate -a bosinst_data=vios_31010_bosinst_data vio1nim -o allocate -a mksysb=vios_31010_mksysb vio1nim -o allocate -a spot=vios_31010_spot vio1
Perform a VIO installation: nim -o bos_inst -a source=mksysb -a boot_client=no -a accept_licenses=yes vio1
Check the exports: showmount -e
Show the spot, bosinst_data and mksysb exported to the vio: ls -l /tftpboot
Show line for vio including its IP, subnetmask and GW: tail /etc/bootptab
OK, all set. Boot the vio1 client and do the install:
- Activate vio1 LPAR from HMC
- Enter SMS menu (option 1)
- Setup Remote IPL (Inital Program Load)
- Interpartition Logical LAN (confirm the MAC address)
- Set IPv4 address format (unless using ipv6)
- bootp
- Set IP Paramaters:
- Client IP address = 192.168.5.X
- Server IP address = x.x.x.x
- Gateway IP address = 192.168.5.254
- Subnet Mask = 255.255.255.0
Replacing the GW, subnet mask as required.
Next perform a Ping test. If successful, move on to booting it normally for the bos installation.
M (hit M to go to main menu in the SMS) NOTE: exact numbers may not match.
(5) Boot options ## number could be different
(1) Install Device
(4) Network
(1) Bootp
(1) Interpartition Logical LAN
(2) Normal Boot Mode
(1) yes, exit SMS
In the end, I ordered some hard drives and done the installation using a powervm VIOS image. It was prepared by punting the image to an USB flash drive using DD. This had to be done on-site, due to physically installing the USB flash drive. The USB drive with PowerVM VIOS was inserted into the USB port on the Power950 (not the HMC). Then the first VIO’s profile was modified, and the USB controller added. Next, the vio was booted up, and using a virtual remote console, accessed the SMS menus. Configured the system to boot from the USB Flash drive. This allowed the installation of the powervm VIO. Once installed, the first task was to put it on the network, then install the propriarty disk drivers. Next get inventory scout downloaded and configured (as per here).
From there, we know which firmware to download. Navigate to IBM’s fixcentral, doing specific search for the 9040-MR9 system. Download the latest applicable firmware, and install them in the appropriate order. Next move on to upgrading the powerVM VIO server to a newer version.
OK, now that the VIOS is setup on the network, it is time to setup the network switches for redundancy, and configure etherchannels / Shared Ethernet Adapters so that shared virtual networks can be utilized.
With the hardware purchased and configured, the following was going to be setup.
- 2 x VIO servers
- 2 fibre ports per vio (at least 8GB)
- 2 x 10GB+ network etherchanneled (data etherchannel)
- 2 x 1GB network etherchannelled (for VIO management IP)
- 1 SAS (mirrored raid) per VIO for redundancy
Knowing the layout, one can safely setup what should be configured and how to go about that, as you’ll require redundancy at a switch level, as well as at a hardware level.
For storage, using HBAs one could purchase 2 x 2-port HBA adapters. This will give you redundancy at the storage level. If one of the hba adapters fail, the vios will be unhappy at the lost storage. However, the 2nd vio will continue to push out the IO.
Networking
As it was laid out that 2 x etherchannels are required per vio, that would require a minimum of 8 network connections. For redundancy, 4 network adapters (minimum of 2 port) would be required. However, for scalability, more network adapters may be advisable. So in this specific scenario, the following was setup: (per vio)
- 2 x 10G ports (seperate adapters)
- 2 x 1G network ports (seperate adapters)
With the version of PowerVM VIOs installed, by default all of the hba / ethernet adapters are there (and their children). However, no virtual I/O Ethernet adapters are present. This can pose a problem for setting up the VIO etherchannel (if configuring as a Shared Ethernet Adapter), as a requirement for create a SEA is to have an available Virtual I/O Ethernet adapter.
If you don’t need the SEA, you can etherchannel the 2 x 1G network ports together, and then configure the IP address to the etherchannel (this may be against best practices).
If that isn’t an option, you will need to use a SEA (Shared Ethernet Adapter). In the event that you boot the VIO and you have the appropriate Virtual I/O Ethernet adapters, great. If not, and it’s for your VIO IP, you’ll have to create the Virtual I/O Ethernet adapter. This can be accomplished by editing the VIO profile. Then adding a Virtual Ethernet (under virtual adapters). When creating this virtual ethernet adapter, you will want to know if it’s going to be used for VLANs or not, as that will be factored in during the creation phase. Assuming it is not used (and you’ll be using VLANs from the 2nd SEA), you can create it as:
- adapter_ID: use the next available one that is automatically calculated
- VLAN_ID: 1 (best practices to use 1 here)
- Is required for server activation (usually checked here if urgent to have this)
- IEEE_8021.q .. Checked only if you are configuring VLANs here
- Network Bridge: yes (1st VIO set priority = 1, 2nd vio priority = 2)
If creating additional virtual ethernet adapters, ensure you mirror it to the 2nd vio. Typically if creating multiple etherchannels/SEAs, just set the 1st vio’s priority = 1 and 2nd vio = 2.
As this was done at the “profile” level, the way to activate it would be to power off the VIO serer. Then “activate it”, indicating you want to activate it using the profile (as the current configuration would be old information).
Once the system is booted, you should notice that you have the required “Virtual I/O Ethernet Adapters” now present. This is a requirement to create a Shared Ethernet Adapter.
So first, you’ll want to determine what ethernet devices are available. Login to the vio server (usually as padmin) and run: lsdev |grep en and that should display the relevant ethernet devices.
For this example, let’s say your ethernet 1G ports plugged in are at:
C3 T1 and C5 T1. You will first have to determine which ethernet device maps to those specific adapter & ports. Assuming you have 2 network quad port adapters, by default you will have the devices: en0 – en7 (and the children entX and etX). So an easy way to see the device name and mapping would be to run:
for i in 0 1 2 3 4 5 6 7 ; do lsdev -dev ent$i -vpd|grep ent ; done
example:
$
for i in 0 1 2 3 4 5 6 7 ; do lsdev -dev ent$i -vpd|grep ent ; doneent0 U7814.XH1.CSA3801-P1-C3-T1 PCIe2 4-Port Adapter (1GbE rj45) (exxxx100)
ent1 U7814.XH1.CSA3801-P1-C3-T2 PCIe2 4-Port Adapter (1GbE rj45) (exxxx100)
ent2 U7814.XH1.CSA3801-P1-C3-T3 PCIe2 4-Port Adapter (10GbE SFP+) (exxxx100)
ent3 U7814.XH1.CSA3801-P1-C3-T4 PCIe2 4-Port Adapter (10GbE SFP+) (exxxx100)
ent4 U7814.XH1.CSA3801-P1-C5-T1 PCIe2 4-Port Adapter (1GbE rj45) (exxxx100)
ent5 U7814.XH1.CSA3801-P1-C5-T2 PCIe2 4-Port Adapter (1GbE rj45) (exxxx100)
ent6 U7814.XH1.CSA3801-P1-C5-T3 PCIe2 4-Port Adapter (10GbE SFP+) (exxxx100)
ent7 U7814.XH1.CSA3801-P1-C5-T4 PCIe2 4-Port Adapter (10GbE SFP+) (exxxx100)