When dealing with an AIX Operating System, there comes a time to update the device firmware. Within the o/s there is adapter firmware and system firmware. The system firmware is updated similarly, however, a reboot of the system is a standard part of the process. Note: there is typically two sides to the system firmware. A temporary side, and a permanent side.
When one updates the system firmware, it is the temp side that gets affected. This way, if anything goes wrong you can reboot the machine from the other side to for recovery. Additionally, whatever is on the current temp side is automatically moved to the permanent side.
Both system firmware and adapter firmware can be updated with “diag”. For the adapter firmware, you can determine what level it is in diag, or with the lsmcode command. So for example, if you wanted to get a list of all of the firmware for the hba (fibre) cards in the system you can use this one liner.
lsdev | grep fcs | awk {‘print “lsmcode -cd ” $1’} | sh
To update the firmware for a hba card, you would typically want to remove any of it’s paths first. Use lspath for this. Additionally, one typically updates the adapter firmware one at a time. Note: when updating the firmware of a dual-port card it is advisable to disable paths on both sides (whenever possible).
Typically one would download the update for the firmware from ibm fix central. In the case of an external entity (like qlogic) the qlogic fibre adapter device software can be downloaded directly from qlogic. The first thing you need to know is the type of adapter. Do a lscfg -vpl fcs0 for example to get the info. Download the firmware for that type of card. Once downloaded, transfer it to the aix machine (via ftp, scp, etc). Then you can copy it to /etc/microcode.
Open “diag” as root, choose task selections (usually 3rd choice), enter microcode tasks, select the adapter (using enter) and commit with F7. Then tell it to update the firmware. Test functionality afterwords. Repeat for each of the fibre adapters.
It is advisable to vary on each of the hba adapters when you are done with them for increased functionality (maximum fail-over tolerance).
Typically one varies off the paths for one port (or dual if possible), updates the firmware on the port(s), then brings the port(s) back online after some functionality testing.
wtf:
Apparently it may not be as easy to update the qlogic 8GB CIOv fibre card as it used to be. However, this depends on the firmware you need installed to it. The steps above would work for the typical firmware, however, they have an EDC firmware update now. This EDC update requires the system be booted off of an .iso image, and the firmware updated that way. This way of course will require a production outage.
Update on HBA Firmware Update:
To perform the firmware update steps, it is now a two-step process. First run the EDC Kit update (using the .iso method). Then afterwords, one would run the typical microcode firmware update. However, there are some steps which should probably be done prior to attempting this process. First, ensure that your system is NOT configured to boot from one of your 8GB fibre cards. As the documentation plainly states that interferring with the boot process (after flashing the EDC firmware) can damage the hba adapter permanently, requiring a HBA adapter replacement. This can be checked in AIX with bootlist -m normal -o. This will typically point to hdiskX of your rootvg. It may NOT list a specific pathid, which means it will boot from whichever it sees first. Note: more than one path can be listed as well. To fix this, you can force it to use a specific pathid. To get a list of your pathids for your rootvg disk it can be accomplished in a couple of steps.
Step 1 — Get a list of all of your fibre cards (and their speeds) with lsdev |grep fcs
Step 2 — Using lspath, get a list of all of the pathids. For this example, I’ll assume the boot disk is hdisk5. lspath -F"name parent connection path_id path_status" | sort | grep hdisk5
Now that you have a list of all of your hba adapters (and the type 4G/8G etc) and a listing of the pathids, you can now change the bootlist (for sms) so that your system won’t boot from one of your 8GB adapters. You can always change this back after the firmware updates are done!
To change it to use pathid’s of 0 and 4 (and assuming pathid 0 and 4 point to your 4GB hba adapters).
bootlist -m normal hdisk5 pathid=0,4
One last recommendation. Log all of your steps when performing the update. Something like this will work (assuming you are connecting from a nix system).
* Assign the media tray in the AMM/HMC to the proper blade
* Power-down the blade in question.
* Insert the burned CD into the media tray for the blade chassis
* run “script edc.log” (give it any name you like here, as it will log all text output to this file)
* ssh [email protected]
* Boot into SMS, and go through the steps of changing the boot order so the system will boot from the CD. Typically (5. Select Boot Options, 1. Select Install/Boot Device, 3. CD/DVD, 6. USB, 1. USB CD-ROM, 2. Normal Boot Mode, X. Exit System Management, Are you sue you want to exit System Management Services ? 1. Yes)
== Interesting Side Effect ==
Below are two seperate incidents from attempting to update the firmware adapter within a power6 blade (js23).
#1 — After running the EDC Kit firmware the system would hang at the “Welcome to VIOS screen” and never boot. After 24 hours, still sat at the same screen. Powering off and on the blade or performing typical reboots would NOT rememedy the situation. The Fix: Power off the blade, login to the AMM, to a “env -T system:blade[x], power -on -c .. Once you see the IBM IBM Logo, hit 8 for Open Firmware. Then type “boot -s verbose”. The system boots normally (with excessive logging). Power down the blade, and power up as normal (as system is functional again).
NOTE: For a power10 system, the key to open the restricted firmware is “9”. It’s a more locked down version that uses macros. The pdf for the OF is available here. So to boot a disk in verbose boot mode, you first need to discover the disk with:
display_boot_devices #disk## as we know it’s a disk, not network etc.. it’s find SAN LUN too!Example output:
check /pci@80000002000000a/pci1014,0339@0/sata/disk@60000
Device Current Device
Number Position Name
1. 2 SAS 136 GB Harddisk ( loc=U78AA.001.WIH002F–P2–D1 )
2. – SAS 136 GB Harddisk ( loc=U78AA.001.WIH002F–P2–D6 )So to boot that disk in verbose boot mode at the open firmware prompt:
0 > boot_from_device #disk U78AA.001.WIH002F–P2–D1 -s verbose
#2 — Replaced HBA adapters in a js43 blade, then performed EDC Kit firmware update. Once in the O/S, done a microcode update. Shortly afterwords noted that the IVM Webpage didn’t start. Upon attempting to start normally, got errors. Rebooting the VIOS did NOT fix the problem. IBM recommended installing VIOS, then updating one (or more times) to get to the latest VIOS level. The fix: Same as above. Power off the blade, enable verbose boot logging. System is functional again (with webpage accessible). Power off the blade, then power it back on (w/o verbose logging).
== IBM Recommendations on alleviating this matter =
1. Do a regular shutdown/reboot to ensure that there is no underlying issue and that we can boot fine prior to flash the edc and the rom bios
2. Shutdown the blade , and set the port on the switch offline to make sure it will not see the San
3. Open blade server chassis, and remove any 4Gb Qlogic Cards, then remove one of the 8Gb cards (if dual-wide blade)
4. Boot system from the EDC kit cd (from burned vt.boot.1.21.iso) and select 3 (List – view the current versions on the card)
5. Then select option 1 to update EDC code for Qlogic, you should have the following version if successful EFI 2.38 BIOS 2.13 FCcode 3.17 FIRMWARE 5.03.17
6. Once update is completed reboot to the SMS menu , then power off the blade.
7. Remove the Qlogic 8GB ciov and install the 2nd one and update the EDC reboot once to SMS then power off again.
8. Put back both Qlogic and using standalone diagnostic flash the boot rom code on both Qlogic adapter this will bring the firmware at 5.06.03 which contain support setting for the EDC code. Available at: http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=migr-5091308
9. Reboot blade and go into “Open Firmware” prompt. Verify bios levels with “set-edc-config” (available at: http://filedownloads.qlogic.com/files/Driver/82457/AppNote_IBM_8Gb_FC_ExpCard_EDC_FW_Update_CU0251401-00C.pdf )
10. Set port online on the switch for the blade and set fix speed 8GB
11. reboot VIOS
= switch config =
For the above setup to work properly, a blade server would have their hba firmware updated and the qlogic fibre switch updated.
This of course can cause issues for the blade servers which are not updated during the same outage window. So one should decide what is the best scenario.
Systems updated:
Hba is hard-coded to “auto-negotiate” link speed (8Gb adapters) and the port(s) on the qlogic fibre switch are set to 8Gb.
All of the systems updated won’t exhibit any errors like before. However, any systems NOT patched will continue with the “path failures”, “path recoveries”, “temporary disk errors” or “link failures”. The unpatched systems can contain some or all of the errors due to the fibre time-outs.
If one changes the hba link_speed=4G and the port(s) on the switch to =auto, the unpatched systems will stop complaining. However, the patched systems will start to flag “link errors”.
In my experiences, no outages have been experienced with either case. The ideal situation would merit updating the qlogic fibre switch(es) and ALL of the blade servers in the same outage window. However, that may not be feasible.