The sequence to extend the filesystem is as follow.
(1) Add the new disk. In my case I was running Red Hat Enterprise 5 Linux as a guest operating system in Vmware ESX server. I simply had to increase the disk size using the Vmware Vcenter interface, under edit setting, disk provisioning. You will need to reboot the guest OS after to recognize the new disk size.

(2) Partition the new disk space using fdisk.

(3) Initialize the new partition, using the command pvcreate. Run partprobe to update the kernel.

(4) Extend the existing volume group with the new partition using the command vgextend.

(5) Extend the existing logical volume group using lvextend.

(6) Grow the filesystem with the command resize2fs.

Take a snapshot of the existing disks and to understand how they are partitioned.
The fdisk command will display the configuration of the existing disks. In this example there are two disks, a 10.7 GB /dev/sda and /dev/sdb which is 21.4 GB. Disk sda has two partitions, sda1 and sda2. The second disk sdb has only one partition sdb1.

[root@linux_server ~]# fdisk -l

Disk /dev/sda: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

ice Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 1305 10377990 8e Linux LVM

Disk /dev/sdb: 21.4 GB, 21475491840 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

ice Boot Start End Blocks Id System
/dev/sdb1 1 2610 20964793+ 8e Linux LVM

You also need to take a snapshot of how the partitions are allocated to the volume groups. In the below example, the result of pvscan indicates that sda2 is mapped to the volume group VolGroup00. But where is sda1? Well sda1 is mapped to the /boot filesystem, you will see it when you issue the mount command. You can also use the command pvs or pvdisplay instead of pvscan.

[root@linux_server ~]# pvscan
PV /dev/sdb1 VG gfs_vg lvm2 [19.99 GB / 8.99 GB free]
PV /dev/sda2 VG VolGroup00 lvm2 [9.88 GB / 0 free]
Total: 2 [29.84 GB] / in use: 2 [29.84 GB] / in no VG: 0 [0 ]

The mount command will show you that /dev/sda1 is mounted as /boot. In this example we will be extending the / (root) filesystem.

[root@linux_server ~]# mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts typepts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
none on /sys/kernel/config type configfs (rw)
/dev/mapper/gfs_vg-gfs_lv on /mnt/gfs type gfs2 (rw,noatime,nodiratime,hostdata=jid=0:id=196609:first=0)

Add the new disk or extend the existing disk in Vmware.
Expand the disk sda using the VMware Vcenter management console. Under the summary tab of the guest server, go to edit settings – disk provisioning. Simply increase the disk size and then reboot the guest OS.

Partition the disk
After the guest server has been rebooted it should see the size increment in sda. You now need to partition the new disk space using the command fdisk.

[root@linux_server ~]# fdisk /dev/sda

The number of cylinders for this disk is set to 2610.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): m (select m to display the help screen)
Command action
a toggle a bootable flag
b edit bsd disklabel
c toggle the dos compatibility flag
d delete a partition
l list known partition types
m print this menu
n add a new partition
o create a new empty DOS partition table
p print the partition table
q quit without saving changes
s create a new empty Sun disklabel
t change a partition’s system id
u change display/entry units
v verify the partition table
w write table to disk and exit
x extra functionality (experts only)

Command (m for help): n (select n to add a new partition)

Command action
e extended
p primary partition (1-4)
p (select p to create a primary partition)
Partition number (1-4): 3 (enter 3 for partition number, we already know from the output of fdisk -l that sda1 and sda2 already exist, therefore the new partition will be sda3)

First cylinder (1306-2610, default 1306): (hit the enter key to accept the default)
Using default value 1306
Last cylinder or +size or +sizeM or +sizeK (1306-2610, default 2610): (press the enter key to accept the default. You can also select something lower than 2610, if you do not want to use all available free disk space)
Using default value 2610

Command (m for help): t (select t to change the system id)
Partition number (1-4): 3 (select the new partition number)
Hex code (type L to list codes): 8e (Enter 8e which is the code for Linux LVM)
Changed system type of partition 3 to 8e (Linux LVM)

Command (m for help): p (this will print the partition table, verify the changes you made)

Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

ice Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 1305 10377990 8e Linux LVM
/dev/sda3 1306 2610 10482412+ 8e Linux LVM

Command (m for help): w (this will write the table tto disk committing it)
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16:ice or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

Note: (May have to hit enter to get command prompt back)

Initialize the physical volume
Initialize the new physical volume with the command pvcreate /dev/sda3. If you have not rebooted the operating system after creating the new partition, then you will need to run partprobe for the kernel to recognize the new partition. Or else you will get an error.

[root@linux_server ~]# pvcreate /dev/sda3
ice /dev/sda3 not found (or ignored by filtering).

Run partrobe to update the kernel.

[root@linux_server]# partprobe -s
/dev/sda: msdos partitions 1 2 3
/dev/sdb: msdos partitions 1

Initialize the partition.

[root@linux_server]# pvcreate /dev/sda3
Physical volume “/dev/sda3″ successfully created

Add new physical volume to the volume group VolGroup00
The command vgdisplay will show you the specifics of the existing volume groups. Note the VG size is 9.88 GB. You can also use vgs and vgscan to display a different view of all volume groups.

[root@linux_server ~]# vgdisplay
— Volume group —
VG Name gfs_vg
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 7
VG Access read/write
VG Status resizable
Clustered yes
Shared no
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size 19.99 GB
PE Size 4.00 MB
Total PE 5118
Alloc PE / Size 2816 / 11.00 GB
Free PE / Size 2302 / 8.99 GB
VG UUID ws74W5-bejg-UUqe-ReFJ-ip6i-4Ixy-CAKkDC

— Volume group —
VG Name VolGroup00
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size 9.88 GB
PE Size 32.00 MB
Total PE 316
Alloc PE / Size 316 / 9.88 GB
Free PE / Size 0 / 0
VG UUID KxKjSV-yOQl-T0gR-vse1-Z0RO-nJfM-FQxMZ0

Add sda3 to VolGroup00

[root@linux_server]# vgextend /dev/VolGroup00 /dev/sda3
Volume group “VolGroup00″ successfully extended

Note that the VG size has increased to 19.84 GB.
[root@linux_server]# vgdisplay
— Volume group —
VG Name gfs_vg
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 7
VG Access read/write
VG Status resizable
Clustered yes
Shared no
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size 19.99 GB
PE Size 4.00 MB
Total PE 5118
Alloc PE / Size 2816 / 11.00 GB
Free PE / Size 2302 / 8.99 GB
VG UUID ws74W5-bejg-UUqe-ReFJ-ip6i-4Ixy-CAKkDC

— Volume group —
VG Name VolGroup00
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 4
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 2
Act PV 2
VG Size 19.84 GB
PE Size 32.00 MB
Total PE 635
Alloc PE / Size 316 / 9.88 GB
Free PE / Size 319 / 9.97 GB
VG UUID KxKjSV-yOQl-T0gR-vse1-Z0RO-nJfM-FQxMZ0

Extend the Logical Volume Group /dev/VolGroup00/LogVol00 which is mounted as the root filesystem.
Display the specifics for all logical volume with lvdisplay. You can also use lvscan or lvs to display a summarized view.

[root@linux_server]# lvdisplay
— Logical volume —
LV Name /dev/gfs_vg/gfs_lv
VG Name gfs_vg
LV UUID NZrcfq-6Shf-z5cZ-fWJM-fQVi-wOTa-kPhUVG
LV Write Access read/write
LV Status available
# open 1
LV Size 10.00 GB
Current LE 2560
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 256
Blockice 253:2

— Logical volume —
LV Name /dev/gfs_vg/quorum_lv
VG Name gfs_vg
LV UUID Pr27bq-X1AM-U6WC-ceJd-dlIu-AipK-pFRfAE
LV Write Access read/write
LV Status available
# open 1
LV Size 1.00 GB
Current LE 256
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 256
Blockice 253:3

— Logical volume —
LV Name /dev/VolGroup00/LogVol00
VG Name VolGroup00
LV UUID n4QGnw-R9Wj-RHXU-Q1QO-X2aa-vKdx-Js3fwp
LV Write Access read/write
LV Status available
# open 1
LV Size 8.88 GB
Current LE 284
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 256
Blockice 253:0

— Logical volume —
LV Name /dev/VolGroup00/LogVol01
VG Name VolGroup00
LV UUID qqyXcr-nys3-K7MO-LsbU-5CL8-d2hR-MyliKF
LV Write Access read/write
LV Status available
# open 1
LV Size 1.00 GB
Current LE 32
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 256
Blockice 253:1

Extend /dev/VolGroup00/LogVol00 using lvextend. The option -l +100%FREE will use up all available disk space. You can also specify the size with the option -L18G, which will extend the logical volume to 18 GB. The option -L+1G will add another gigabyte to the logical volume.

[root@linux_server]# lvextend -l +100%FREE /dev/VolGroup00/LogVol00
Extending logical volume LogVol00 to 18.84 GB
Logical volume LogVol00 successfully resized

Extend the filesystem
We are almost done, the last step is to extend the filesystem.

Record the size of the existing filesystems with the df -h command.

[root@linux_server]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
8.6G 8.0G 205M 98% /
/dev/sda1 99M 19M 75M 21% /boot
tmpfs 490M 0 490M 0% /dev/shm
/dev/mapper/gfs_vg-gfs_lv
9.8G 584M 9.2G 6% /mnt/gfs

Expand the filesystem /dev/mapper/VolGroup00-LogVol00 which is mounted on /. The command resize2fs can be used to enlarge or shrink ext2 or ext3 filesystems that are unmounted. On Linux kernel version that is equal or greater than 2.6, resize2fs can expand a mounted ext3 filesystem only.

[root@linux_server]# resize2fs /dev/mapper/VolGroup00-LogVol00
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/mapper/VolGroup00-LogVol00 is mounted on /; on-line resizing required
Performing an on-line resize of /dev/mapper/VolGroup00-LogVol00 to 4939776 (4k) blocks.
The filesystem on /dev/mapper/VolGroup00-LogVol00 is now 4939776 blocks long.

Verify that /dev/mapper/VolGroup00-LogVol00 has increased in size.

[root@linux_server]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
19G 8.0G 9.4G 46% /
/dev/sda1 99M 19M 75M 21% /boot
tmpfs 490M 0 490M 0% /dev/shm
/dev/mapper/gfs_vg-gfs_lv
9.8G 584M 9.2G 6% /mnt/gfs

It seems that I was able to login successfully into the server, but within few seconds it would kick me out. Just before it disconnected my ssh session, I saw the message ‘The server is going down..’. For some strange reason that message was stuck in the operating system, it was the message displayed just before the scheduled reboot. I had pushed out the reboot command from the Red Hat Satellite server. All the other servers rebooted fine.

I knew that clearing this message would resolve my dilemma, but I had no idea which file held it. After much searching I found the culprit, it was /etc/nologin. Due to reason I did not want to try and figure out as it would take too much time, this file did not get automatically deleted after the reboot. I deleted /etc/nologin, and then was able to successfully login via ssh to the server.

Before you delete or rename the root account it is important that you create a new account with the same privileges as root. This account should have the same capabilities as root, a new superuser account. I will explain the steps to create a new superuser account in Redhat Linux. The steps are similar for Centos, Fedora and most other variants of Linux.

Let’s begin by checking the characteristics of the root account.

# id root
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel)

uid=0(root), this indicates that root has a user id of 0.
gid=0, the group id is 0.
groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel), these are the groups root belongs to. The numbers signifies the group id numbers, e.g. disk has a group id of 6.

To create a new superuser, it must have the same uid, gid and belong to the same groups as the root account.

You can create a new superuser account using the adduser command.

# adduser -u 0 -o -g 0 -G 0,1,2,3,4,6,10 -M andrew

-u 0 –o, this sets the uid to 0.
-g 0, sets the gid to 0.
-G 0,1,2,3,4,6,10, sets the group membership to 0 = root, 1 = bin, 2 = daemon, 3 = sys, 4 = adm, 6 = disk and 10 = wheel.
-M, do not create a home directory.
andrew, new user account name.

Create a password for the new account.

# passwd andrew
Changing password for user andrew:
New UNIX password:
Retype new UNIX password:
Passwd: all authentication tokens updated successfully.

What if you want to promote an existing user to superuser? Then you will need to edit two files, /etc/passwd and /etc/group.

Open /etc/passwd with Vi editor and change the gid and uid to 0 for andrew.

andrew:x:0:0::/home/andrew:/bin/bash

Edit /etc/group and add andrew to the groups root, bin, daemon, sys, adm, disk and wheel.

root:x:0:root,andrew
bin:x:1:root,bin,daemon,andrew
daemon:x:2:root,bin,daemon,andrew
sys:x:3:root,bin,adm,andrew
adm:x:4:root,adm,daemon,andrew
disk:x:6:root,andrew
wheel:x:10:root,andrew

If the user account andrew has and existing home directory such as /home/andrew, then you will need to change the ownership to reflect root.

# chown root:root /home/andrew

Otherwise you will receive the following error message, upon logging in.

User’s $HOME/.dmrc file is being ignored. This prevents the default session and language from being saved. File should be owned by user and have 644 permission. User $HOME directory must be owned by user and not writable by other users.

This is because the uid and gid for andrew has been changed to 0, this is the same as root. Whereas /home/andrew still belongs to the username andrew, which had a uid and gid of 500 or higher number.

I received and error while trying to update the Redhat Enterprise 5 server kernel using yum update. Looking at the output of yum update it seems that the dependency mkinitrd was missing.

# yum update
Loaded plugins: rhnplugin, security
Skipping security plugin, no data
Setting up Update Process
Resolving Dependencies
Skipping security plugin, no data
–> Running transaction check
—> Package kernel.i686 0:2.6.18-238.5.1.el5 set to be installed
–> Processing Dependency: mkinitrd >= 4.2.21-1 for package: kernel
–> Finished Dependency Resolution
kernel-2.6.18-238.5.1.el5.i686 from rhel-i386-server-5 has depsolving problems
–> Missing Dependency: mkinitrd >= 4.2.21-1 is needed by package kernel-2.6.18-238.5.1.el5.i686 (rhel-i386-server-5)
Error: Missing Dependency: mkinitrd >= 4.2.21-1 is needed by package kernel-2.6.18-238.5.1.el5.i686 (rhel-i386-server-5)
You could try using –skip-broken to work around the problem
You could try running: package-cleanup –problems
package-cleanup –dupes
rpm -Va –nofiles –nodigest

I tried the suggested fix by running the commands package-cleanup. But that did not resolve the problem. I then tried clear all yum cache and to re-synch with the Red Hat Network Satellite server. But yum update failed with the same error.

# rm -rf /var/cache/yum/*
# yum clean all
# rhn-profile-sync
# yum update

The next logical thing to do was to check and see if mkinitrd was installed, and it was.

# rpm -q mkinitrd
mkinitrd-5.1.19.6-68.el5

Now I was really puzzled, if the dependency was installed then was it healthy? Using the rpm –Va command I found out that mkinitrd was missing the dependency nash.

# rpm -Va mkinitrd
Unsatisfied dependencies for mkinitrd-5.1.19.6-68.el5.i386: config(mkinitrd) = 5.1.19.6-68.el5, nash = 5.1.19.6-68.el5, mkinitrd

Following the trail, I checked to see if nash was installed, and it was.

# rpm –q nash
nash-5.1.19.6-68.el5

Is nash healthy? No result means no fault found.

# rpm -Va nash

If mkinitrd claims that it is dependent on nash, then checking to see what requires nash should result in mkinitrd. But I did not receive any result back on the check.

# rpm -q –whatrequires nash

Now I was stumped, if mkinitrd says that it was dependant on nash, then why is nash not reporting that mkinitrd requires it? Using the rpm –q –whatrequires I checked to see what required mkinitrd.

# rpm -q –whatrequires mkinitrd
mkbootdisk-1.5.3-2.1
system-config-kdump-1.0.14-4.el5
kernel-2.6.18-194.17.1.el5

I compared the result of rpm –q –whatrequires nash to different installation of Redhat. Below is what the result should be.

# rpm -q –whatrequires nash
mkinitrd-5.1.19.6-68.el5

To sum it up, the kernel cannot be updated because it is missing the dependency mkinitrd. Mkinitrd is installed but not healthy because it is missing the dependency nash. Nash is installed and healthy which contradicts the former statement. Therefore there is nothing wrong with mkinitrd or nash, the problem then lies with the yum cache or rpm database. I already ran yum clean all to clear all yum cache and headers, it did not resolve the issue. The logical answer is that the rpm database is corrupted and requires a rebuild.

It may be beneficial to backup the current rpm databases before deleting. The rpm database are in /var/lib/rpm. It is also recommended that rebuidling be done in single user mode, although I did it successfully in init 3 run level.

Delete the lock files.

# rm –f /var/lib/rpm/__db*

Rebuild the database

# rpm –rebuilddb

You can also use the -vv option for verbose mode.

# rpm -vv –rebuilddb

The rescue mode is provided for troubleshooting and repair purposes. This is really handy when the OS is broken and you are unable to access the system drive. Using rescue mode you can access the files stored on the system’s hard drive.

Boot the system from the installation CD-ROM, or you cab also use the iso file in the case of Vmware. When presented with the boot: prompt, type the following command:
linux rescue.

You are prompted to answer a few basic questions, such as language to use, and keyboard type. Select do not start network interface, unless you have a reason to.

When prompted with the message “The rescue environment will attempt to find the Linux installation and mount it under the directory /mnt/sysimage…..”, select continue. You should then see a message “Your system had been mounted under /mnt/sysimage…”. Select OK, you are now is single-user mode.

The default root partition while in rescue mode is a temporary root partition, not the root partition of the file system used during normal user mode (runlevel 3 or 5). If you selected to mount your file system and it mounted successfully, you can change the root partition of the rescue mode environment to the root partition of your file system by executing the following command:
Chroot /mnt/sysimage

This is useful if you need to run commands such as rpm that require your root partition to be mounted as /. To exit the chroot environment, type exit, and you will return to the prompt.

You can also mount a partition manually inside rescue mode by creating a directory such as /foo, and typing the following command:
mount -t ext3 /dev/hda5 /foo
In the above command, /foo is a directory that you have created and /dev/hda5 is the partition you want to mount. If the partition is of type ext2, replace ext3 with ext2.

If you do not know the names of your partitions, use the following command to list them:
fdisk –l

From the prompt, you can run many useful commands such as
list-harddrives to list the hard drives in the system
ssh, scp, and ping if the network is started
dump and restore for users with tape drives
parted and fdisk for managing partitions
rpm for installing or upgrading software
joe for editing configuration files (If you try to start other popular editors such as emacs, pico, or vi, the joe editor will be started.)

# uname -r
2.6.18-194.32.1.el5

The –r option will print the kernel release.

# uname -s
Linux

-s prints the kernel name.

# uname -n
mycomputer.wiivil.com

Print the node name or hostname.

# uname -p
i686

Print the processor type, the i686 signifies a 32 bit processor. If it was a 64 processor it would display x86_64 instead.

# uname -a
Linux myserver.wiivil.com 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:53:09 EST 2011 i686 i686 i386 GNU/Linux

Print all the information except omit –o and –I if unknown.

# cat /etc/redhat-release
CentOS release 5.5 (Final)

This will give you the version of RedHat Linux you are running.

# cat /proc/version
Linux version 2.6.18-194.32.1.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Wed Jan 5 17:53:09 EST 2011

This will give you who and when compiled the kernel and what gcc compiler was used to build it.

# rpm -q kernel
kernel-2.6.18-194.11.3.el5
kernel-2.6.18-194.26.1.el5
kernel-2.6.18-194.32.1.el5

This will list all the kernels installed on your system.

If the new kernel causes issues then you have the option to revert to the older version. You may have to boot the system with the boot media and re-configure the boot loader. The AMD and Intel architecture uses GRUB boot loader. You will need to make changes to /boot/grub/grub.conf to activate a different default kernel.

Please not that when the kernel is installed using rpm, the kernel package creates an entry in the boot loader configuration file for the new kernel. But the rpm does not configure the new kernel to boot as the default kernel, this must be done manually.

Below is an example of /boot/grub/grub.conf. The directive default=0 means that default boot kernel is 0, which is the first stanza that starts with the title entry.

So in this case title CentOS (2.6.18-194.32.1.el5) is stanza 0. When the system boots this kernel will be used. The title contains the kernel version number 2.6.18-194.32.1.el5, which must match with the version number in the kernel /vmlinuz-2.6.18-194.32.1.el5 (this is the line below root.

If a separate /boot/ partition was created, the paths to the kernel and the initramfs image are relative to /boot/. This is the case in Example 23.2, “/boot/grub/grub.conf”, above. Therefore the initrd /initramfs-2.6.32-22.el6.x86_64.img line in the first kernel stanza means that the initramfs image is actually located at /boot/initramfs-2.6.32-22.el6.x86_64.img when the root file system is mounted, and likewise for the kernel path (for example: kernel /vmlinuz-2.6.32-22.el6.x86_64) in each stanza of grub.conf

To clarify further, stanza 0 starts with title CentOS (2.6.18-194.32.1.el5)
Stanza 1 starts with title CentOS (2.6.18-194.26.1.el5)
Stanza 2 is title CentOS (2.6.18-194.11.3.el5)

To change the default boot kernel to stanza 2 simply change the line default=0 to default=2.

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/VolGroup00/LogVol00
# initrd /initrd-version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-194.32.1.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-194.32.1.el5 ro root=/dev/VolGroup00/LogVol00 rhgb quiet
initrd /initrd-2.6.18-194.32.1.el5.img
title CentOS (2.6.18-194.26.1.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-194.26.1.el5 ro root=/dev/VolGroup00/LogVol00 rhgb quiet
initrd /initrd-2.6.18-194.26.1.el5.img
title CentOS (2.6.18-194.11.3.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-194.11.3.el5 ro root=/dev/VolGroup00/LogVol00 rhgb quiet
initrd /initrd-2.6.18-194.11.3.el5.img

Here is what my /etc/resolv.conf looks like, the two DNS servers are defined correctly as it the search domain.
domain andrewlin.com
namesserver 10.10.10.11
namesserver 10.10.11.13
search andrewlin.com
The gateway was defined in /etc/defaultrouter.
10.10.0.1
It seems that ping did not know how to resolve an address. I was also not able to surf the internet with the Firefox browser in the Sun Java Desktop System for Soalris. All symptoms indicated that the nsswitch.conf did not have the correct environment defined. The /etc/nsswitch.conf file defines which services are to used to retrieve information such as hostnames, password files, and group files. So in this case for the command ping and firefox which wants to resolve and address, it will check the nsswitch.conf file to see what services to use.

Below are the lines I changed in /etc/nsswitch.conf, by adding dns to the end of each lines.
hosts: files dns
ipnodes: files dns
I am now able to ping and surf the internet without any issues.

I then checked the log files for more information, and below are the messages I saw.
[root@esx4-server /]# cat /var/log/messages
Oct 26 10:32:30 esx4-server vobd: Oct 26 10:32:30.936: 159470608100us: [esx.problem.vmfs.nfs.mount.connect.failed] Failed to mount to server 192.20.66.88 mount point /VMLibrary/. Error: Unable to connect to NFS server.
Oct 26 11:36:50 esx4-server kernel: [162970.448490] portmap: server localhost not responding, timed out
Oct 26 11:36:50 esx4-server kernel: [162970.453525] RPC: failed to contact portmap (errno -5).
From the log file I was able to determine that the pormap service was not started. It is required to map to a remote NFS directory. I then proceeded with starting the protmap service.
service portmap start

To esure that the service starts automatically when the server reboots use the below command.
chkconfig portmap on
You may also have to stop the firewall service on the esx server with the command service firewall stop. To prevent it from starting after a reboot, chkconfig firewall off.

The procedure to identify the path for the RDM (raw disk mapping) to LUN (disk on SAN) on Vmware ESX 4 is different from Vmware Esx 3. There are a few more steps in version 4. I will explain below how to determine the path for RDM to LUN in ESX 4.

Login to the console with root permission.

# find /vmfs/volumes -name **-rdm**
/vmfs/volumes/4c20ca1e-d32d6ed6-96cb-001e4f3fdc36/VMware1/VMware1_1-rdmp.vmdk

Find all RDM file in .vmfs/volumes. Remove -rdmp from the result and that is the path you need for the next step.

# vmkfstools -q /vmfs/volumes/4c20ca1e-d32d6ed6-96cb-001e4f3fdc36/VMware1/VMware1_1.vmdk
Disk /vmfs/volumes/4c20ca1e-d32d6ed6-96cb-001e4f3fdc36/VMware1/VMware1_1.vmdk is a Passthrough Raw Device Mapping
Maps to: vml.02000a000060060e80058c7b0000008c7b0000310b4f50454e2d56

Use the vmkfstools -q command to find the vml id of the LUN, it is vml.02000a000060060e80058c7b0000008c7b0000310b4f50454e2d56.

# esxcfg-scsidevs -u | grep vml.02000a000060060e80058c7b0000008c7b0000310b4f50454e2d56
naa.60060e80058c7b0000008c7b0000310b vml.02000a000060060e80058c7b0000008c7b0000310b4f50454e2d56

Now use the esxcfg-scsidevs command to find the Network Addressing Authority identifier (naa) for the LUN.

# esxcfg-mpath -l –device=naa.60060e80058c7b0000008c7b0000310b
fc.2001001b3232c093:2101001b3232c093-fc.50060e80058c7b55:50060e80058c7b55-naa.60060e80058c7b0000008c7b0000310b
Runtime Name: vmhba2:C0:T0:L10
Device: naa.60060e80058c7b0000008c7b0000310b
Device Display Name: HITACHI Fibre Channel Disk (naa.60060e80058c7b0000008c7b0000310b)
Adapter: vmhba2 Channel: 0 Target: 0 LUN: 10
Adapter Identifier: fc.3001001b3232c093:2101001b3232c093
Target Identifier: fc.60060e80058c7b55:50060e80058c7b55
Plugin: NMP
State: active
Transport: fc
Adapter Transport Details: WWNN: 22:01:00:1b:32:32:c0:93 WWPN: 23:01:00:1b:32:32:c0:93
Target Transport Details: WWNN: 55:06:0e:80:05:8c:7b:55 WWPN: 56:06:0e:80:05:8c:7b:55

fc.2000001b3212c093:2100001b3212c093-fc.50060e80058c7b45:50060e80058c7b45-naa.60060e80058c7b0000008c7b0000310b
Runtime Name: vmhba1:C0:T0:L10
Device: naa.60060e80058c7b0000008c7b0000310b
Device Display Name: HITACHI Fibre Channel Disk (naa.60060e80058c7b0000008c7b0000310b)
Adapter: vmhba1 Channel: 0 Target: 0 LUN: 10
Adapter Identifier: fc.3000001b3212c093:2100001b3212c093
Target Identifier: fc.60060e80058c7b45:50060e80058c7b45
Plugin: NMP
State: active
Transport: fc
Adapter Transport Details: WWNN: 22:00:00:1b:32:12:c0:93 WWPN: 23:00:00:1b:32:12:c0:93
Target Transport Details: WWNN: 55:06:0e:80:05:8c:7b:45 WWPN: 56:06:0e:80:05:8c:7b:45

The esxcfg-mpath -l command generates the detailed information for the LUN. Because there are redundant path to the LUN on the SAN, the result is listed twice with different fiber channel HBA interfaces.

Now what if your VMware ESX server is hosting many servers all with different RDMs. It would be very tedious to manually compile a report for RDM to LUN mapping. I create a script that will automatically do all the above steps and will output the result to screen or you can redirect the result to a file. This will script will search for all the RDMs and create the mappings. Below is the script and explanation of how it works.

#!/bin/bash
# Andrew Lin
# August 4, 2010
# Pay Andrew Lin lot’s of money before you
# can use this script

date
hostname

find /vmfs/volumes -name **-rdm** | sed ‘s/-rdmp.vmdk/.vmdk/g’ | sed ‘s/-rdm.vmdk/.vmdk/g’ | sed ‘s/^/vmkfstools -q /g’ >> andrew-disk-map |
chmod 645 andrew-disk-map

file=andrew-disk-map
while read line
do
$line
$line | grep vml | sed ‘s/.*to:/esxcfg-scsidevs -u | grep/g’ > andrew_lin1

chmod 755 andrew_lin1
./andrew_lin1 | sed ‘s/ .*//g’
./andrew_lin1 | sed ‘s/ .*//g’ | sed ‘s/^/esxcfg-mpath -L –device=/g’ >andrew_lin2
# rm andrew_lin1

chmod 755 andrew_lin2
./andrew_lin2
# rm andrew_lin2

echo -e “\n”
done <$file

I will explain below what the above script does.

find /vmfs/volumes -name **-rdm**
/vmfs/volumes/4c20ca1e-d32d6ed6-96cb-001e4f3fdc36/VMware1/VMware1_1-rdmp.vmdk

Search for all RDM (raw disk mapping), VMware1_1-rdmp.vmdk is found.

sed ‘s/-rdmp.vmdk/.vmdk/g’

Change the characters -rdmp.vmdk to .vmdk. The s/ means to substitute, and /g means globally apply the changes for all matches. Some RDM have the -rdmp.vmdk extension (not shown in the above example).

sed ‘s/-rdm.vmdk/.vmdk/g’

Change -rdm.vmdk to .vmdk for all matches.

sed ‘s/^/vmkfstools -q /g’

Add the command vmkfstools -q to the beginning of each line.

>> andrew-disk-map | chmod 645 andrew-disk-map

Redirect the output to the file andrew-disk-map. Change the file permission to 645 to make it executable.

Below is the result of the above command stored in andrew-disk-map.

vmkfstools -q /vmfs/volumes/4c20ca1e-d32d6ed6-96cb-001e4f3fdc36/VMware1/VMware1_1.vmdk

file=andrew-disk-map

Define the variable called file which contains the name of the file andrew-disk-map.

while read line
do
..
…
….
done <$file

The while loop will read the contents of the file defined in the variable $file (which is andrew-disk-map). The file is read one line at a time and the commands defined within the while loop are executed for each line read.

$line

Execute the line read from the file andrew-disk-map and send the output to screen.

Command
vmkfstools -q /vmfs/volumes/4c20ca1e-d32d6ed6-96cb-001e4f3fdc36/VMware1/VMware1_1.vmdk
Output
Disk /vmfs/volumes/4c20ca1e-d32d6ed6-96cb-001e4f3fdc36/VMware1/VMware1_1.vmdk is a Passthrough Raw Device Mapping
Maps to: vml.02000a000060060e80058c7b0000008c7b0000310b4f50454e2d56

$line | grep vml | sed ‘s/.*to:/esxcfg-scsidevs -u | grep/g’ > andrew_lin1

Grep will find the line that contains the characters vml, this is the unique LUN id. Replace all characters before and upto the characters to: with esxcfg-scsidevs -u | grep. The result will look like the below line which is redirected to the file andrew_lin1.

esxcfg-scsidevs -u | grep vml.02000a000060060e80058c7b0000008c7b0000310b4f50454e2d56

chmod 755 andrew_lin1

Chmod 755 will make the file andrew_lin1 executable.

./andrew_lin1 | sed ‘s/ .*//g’

Execute the file andrew_lin1. Sed ‘s/ .*//g’ will delete everything after the first space found, otherwise the output will look like the below.

Command executed (this is the content of the file andrew_lin1).
esxcfg-scsidevs -u | grep vml.02000a000060060e80058c7b0000008c7b0000310b4f50454e2d56
Output
naa.60060e80058c7b0000008c7b0000310b vml.02000a000060060e80058c7b0000008c7b0000310b4f50454e2d56

./andrew_lin1 | sed ‘s/ .*//g’ | sed ‘s/^/esxcfg-mpath -l –device=/g’ >andrew_lin2

Execute the file andrew_lin1, from the result remove everything after the LUN ID number (naa.60060e80058c7b0000008c7b0000310b). Then add the command esxcfg-mpath -L –device= in front of the LUN ID number, see below example. The output is redirected to the file andrew_lin2.
esxcfg-mpath -l –device=naa.60060e80058c7b0000008c7b0000310b

# rm andrew_lin1

You are done with file andrew_lin1 and can delete it. If you want to save the file for troubleshooting then comment out the above line with the # sign.

chmod 755 andrew_lin2

Make the file andrew_lin2 executable.

./andrew_lin2

Execute andrew_lin2. Which contains the command esxcfg-mpath -l –device=naa.60060e80058c7b0000008c7b0000310b. The output below is displayed on screen. Notice that there are redundant paths to the LUN on the SAN.

fc.2001001b3232c093:2101001b3232c093-fc.50060e80058c7b55:50060e80058c7b55-naa.60060e80058c7b0000008c7b0000310b
Runtime Name: vmhba2:C0:T0:L10
Device: naa.60060e80058c7b0000008c7b0000310b
Device Display Name: HITACHI Fibre Channel Disk (naa.60060e80058c7b0000008c7b0000310b)
Adapter: vmhba2 Channel: 0 Target: 0 LUN: 10
Adapter Identifier: fc.3001001b3232c093:2101001b3232c093
Target Identifier: fc.60060e80058c7b55:50060e80058c7b55
Plugin: NMP
State: active
Transport: fc
Adapter Transport Details: WWNN: 22:01:00:1b:32:32:c0:93 WWPN: 23:01:00:1b:32:32:c0:93
Target Transport Details: WWNN: 55:06:0e:80:05:8c:7b:55 WWPN: 56:06:0e:80:05:8c:7b:55

fc.2000001b3212c093:2100001b3212c093-fc.50060e80058c7b45:50060e80058c7b45-naa.60060e80058c7b0000008c7b0000310b
Runtime Name: vmhba1:C0:T0:L10
Device: naa.60060e80058c7b0000008c7b0000310b
Device Display Name: HITACHI Fibre Channel Disk (naa.60060e80058c7b0000008c7b0000310b)
Adapter: vmhba1 Channel: 0 Target: 0 LUN: 10
Adapter Identifier: fc.3000001b3212c093:2100001b3212c093
Target Identifier: fc.60060e80058c7b45:50060e80058c7b45
Plugin: NMP
State: active
Transport: fc
Adapter Transport Details: WWNN: 22:00:00:1b:32:12:c0:93 WWPN: 23:00:00:1b:32:12:c0:93
Target Transport Details: WWNN: 55:06:0e:80:05:8c:7b:45 WWPN: 56:06:0e:80:05:8c:7b:45

# rm andrew_lin2

Delete the file andrew_lin2 by removing the # comment.

echo -e “\n”

Enter a line space.

I will explain how to recover a Solaris 10 server from hardware replicated SAN disk. It took me sometime to figure out how to boot up from the replicated SAN LUN (disk), and many more hours to understand why the steps I applied works.

In this example I have a SUN SPARC M3000 server with two Qlogic fiber channel cards (HBA) installed in the PCI solts. The HBAs were already configured to connect to the SAN disk (LUN). This LUN contained the replicated copy of a production Slaris 10 server. The production server had two ZFS pool residing in a single LUN.

Using the Solaris 10 installation CD boot up the Sparc server into single user mode.

boot cdrom -s

The first thing you need to do is to see if the HBAs are working. The connected status indictes that communication between the server and switch is working. There are two HBAs installed for redundancy, both connected to the same LUN.

# luxadm -e port
/devices/pci@0,600000/pci@0/pci@8/SUNW,qlc@0/fp@0,0:devctl CONNECTED
/devices/pci@1,700000/pci@0/pci@0/SUNW,qlc@0/fp@0,0:devctl CONNECTED

Now you need to find out if the SAN disk is visible from the server. Even though both HBAs are connected to the same SAN disk, you will see two separate SAN disks in the results below. It just means there are two paths to the SAN.

# luxadm probe
No Network Array enclosures found in /dev/es

Found Fibre Channel device(s):
Node WWN:50060e80058c7b10 Device Type:Disk device
Logical Path:/dev/rdsk/c1t50060E80058C7B10d1s2
Node WWN:50060e80058c7b00 Device Type:Disk device
Logical Path:/dev/rdsk/c2t50060E80058C7B00d1s2

In the above example the first LUN is c1t50060E80058C7B10d1s2. This is the logical device name which is a symbolic link to the physical device name stored in the /devices directory. Logical device names contain the controller number(c2), target number (t50060E80058C7B10), disk number (d1), and slice number (s2).

The next step is to find out how the disk is partitioned, the format command will give you that information. You need this information to understand how to boot up the disk.

# format
Searching for disks…done

AVAILABLE DISK SELECTIONS:
0. c1t50060E80058C7B10d1 1066>/pci@0,600000/pci@0/pci@8/SUNW,qlc@0/fp@0,0/ssd@w50060e80058c7b10,1

1. c2t50060E80058C7B00d1 1066>/pci@1,700000/pci@0/pci@0/SUNW,qlc@0/fp@0,0/ssd@w50060e80058c7b00,1

Select the first disk 0.

Specify disk (enter its number): 0
selecting c1t50060E80058C7B10d1
[disk formatted]

FORMAT MENU:
disk – select a disk
type – select (define) a disk type
partition – select (define) a partition table
current – describe the current disk
format – format and analyze the disk
repair – repair a defective sector
label – write label to the disk
analyze – surface analysis
defect – defect list management
backup – search for backup labels
verify – read and display labels
save – save new disk/partition definitions
inquiry – show vendor, product and revision
volname – set 8-character volume name
! – execute , then return
quit

Display the labels and slices (partitions). In Solaris each slice is treated as a separate physical disk. In the below example you can tell that the disk is labeled as VTOC (Volume Table of Contents) because you can see the cylinders. VTOC is also known as SMI label. If the disk was labeled with EFI (Extensible Firmware Interface), then you would see sectors instead of cylinders. Partition 0 (slice 0) holds the operating system files, the boot disk. Please note that you cannot boot from a disk with EFI label. Slice 2 is the entire physical disk because it contains all cylinders, 0 – 65532.

format> verify

Primary label contents:

Volume name = <
ascii name =
pcyl = 65535
ncyl = 65533
acyl = 2
nhead = 15
nsect = 1066
Part Tag Flag Cylinders Size Blocks
0 root wm 0 – 3356 25.60GB (3357/0/0) 53678430
1 unassigned wm 0 0 (0/0/0) 0
2 backup wm 0 – 65532 499.66GB (65533/0/0) 1047872670
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 3357 – 65532 474.07GB (62176/0/0) 994194240

Here is what we know so far, we know that the disk name is c1t50060E80058C7B10d1. Slice 0 on this disk contains the boot files. The physical path for this disk is /pci@0,600000/pci@0/pci@8/SUNW,qlc@0/fp@0,0/ssd@w50060e80058c7b10,1. Now we need to find out the physical path for Slice 0.

I know that the disk contains ZFS filesystems because it is a replica of the production disk. When a ZFS filesystem is moved to a different SPARC server it must first be imported because the hostid is different.

List the ZFS pool contained on the disk using the zpool import command. There are two ZFS pools in the below example, epool and rpool. Take a note of the status and action.

# zpool import
pool: epool
id: 16865366839830765202
state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and the ‘-f’ flag.
see: http://www.sun.com/msg/ZFS-8000-EY
config:

epool ONLINE
c2t50060E80058C7B00d1s7 ONLINE

pool: rpool
id: 10594898920105832331
state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and the ‘-f’ flag.
see: http://www.sun.com/msg/ZFS-8000-EY
config:

rpool ONLINE
c2t50060E80058C7B00d1s0 ONLINE

Import the pool with the zpool import command. The options -a will import all the ZFS pools it can find, the -f option will force the import. If you do not specify the force option, then the import may fail with the error “cannot import ‘rpool’: pool may be in use from other system, it was last accessed by server name (hostid: 123456)”. Ignore the error message about failed to create mountpoint.

# zpool import -af
cannot mount ‘/epool’: failed to create mountpoint
cannot mount ‘/rpool’: failed to create mountpoint

List the imported ZFS pools.

# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
epool 472G 260G 212G 55% ONLINE -
rpool 25.5G 5.75G 19.8G 22% ONLINE -

List the ZFS filesystems. In the example below notice the mountpoint / is mounted on the zfs filesystem rpool/ROOT/zfsboot. This is the boot partition, it resides in rpool.

# zfs list
NAME USED AVAIL REFER MOUNTPOINT
epool 260G 205G 21K /epool
rpool 7.74G 17.4G 99K /rpool
rpool/ROOT 4.74G 17.4G 21K legacy
rpool/ROOT/zfsboot 4.74G 17.4G 4.48G /
rpool/ROOT/zfsboot/var 139M 17.4G 105M /var
rpool/dump 1.00G 17.4G 1.00G -
rpool/swap 2.00G 19.4G 16K -

Change the mountpoint for rpoo/ROOT/zfsboot to /mnt so you can mount it to read the contents.

# zfs set mountpoint=/mnt rpool/ROOT/zfsboot

Confirm that the mountpoint was changed.

# zfs list
NAME USED AVAIL REFER MOUNTPOINT
epool 260G 205G 21K /epool
rpool 7.74G 17.4G 99K /rpool
rpool/ROOT 4.74G 17.4G 21K legacy
rpool/ROOT/zfsboot 4.74G 17.4G 4.48G /mnt
rpool/ROOT/zfsboot/var 139M 17.4G 105M /mnt/var
rpool/dump 1.00G 17.4G 1.00G -
rpool/swap 2.00G 19.4G 16K -

Now mount rpool/ROOT/zfsboot.

# zfs mount rpool/ROOT/zfsboot

List the logical disks.

# cd /dev/dsk
# ls
c1t50060E80058C7B10d1s0 c2t50060E80058C7B00d1s0
c1t50060E80058C7B10d1s1 c2t50060E80058C7B00d1s1
c1t50060E80058C7B10d1s2 c2t50060E80058C7B00d1s2
c1t50060E80058C7B10d1s3 c2t50060E80058C7B00d1s3
c1t50060E80058C7B10d1s4 c2t50060E80058C7B00d1s4
c1t50060E80058C7B10d1s5 c2t50060E80058C7B00d1s5
c1t50060E80058C7B10d1s6 c2t50060E80058C7B00d1s6
c1t50060E80058C7B10d1s7 c2t50060E80058C7B00d1s7

As stated earlier the physical path for the disk we are looking for is /pci@0,600000/pci@0/pci@8/SUNW,qlc@0/fp@0,0/ssd@w50060e80058c7b10,1. The boot slice is 0. We can derive from the physical path that the disk name is 50060e80058c7b10. We also know from the output of the format command that the physical disk 50060e80058c7b10 maps to the logical disk c1t50060E80058C7B10d1. Therefore we can derive that the logical boot disk is c1t50060E80058C7B10d1s0.

Now find out what physical path c1t50060E80058C7B10d1s0 is a symbolic link for and that is your complete boot path. In the below example the boot path starts at the first slash (/) right after /devices. It is /pci@0,600000/pci@0/pci@8/SUNW,qlc@0/fp@0,0/ssd@w50060e80058c7b10,1:a. You need to replace ssd@ with disk@ when entering the path into EEPROM.

# ls -l c1t50060E80058C7B10d1s0
lrwxrwxrwx 1 root root 82 Jul 5 10:21 c1t50060E80058C7B10d1s0 ->
../../devices/pci@0,600000/pci@0/pci@8/SUNW,qlc@0/fp@0,0/ssd@w50060e80058c7b10,1:a

If you have more than one ZFS pool the non root pool may not get mounted upon booting up the server, as in this example. You may get the below error.

SUNW-MSG-ID: ZFS-8000-D3, TYPE: Fault, VER: 1, SEVERITY: Major
EVENT-TIME: Mon Jul 5 11:54:14 EDT 2010
PLATFORM: SUNW,SPARC-Enterprise, CSN: PX654321, HOSTNAME: Andrew-Lin
SOURCE: zfs-diagnosis, REV: 1.0
EVENT-ID: 33e5a9f1-49ac-6ebc-f2a9-dff25dea6b86
DESC: A ZFS device failed. Refer to http://sun.com/msg/ZFS-8000-D3 for more information.
AUTO-RESPONSE: No automated response will occur.
IMPACT: Fault tolerance of the pool may be compromised.
REC-ACTION: Run ‘zpool status -x’ and replace the bad device.

http://sun.com/msg/ZFS-8000-D3fameserverq9{root}: zpool status -x
pool: epool
state: UNAVAIL
status: One or more devices could not be opened. There are insufficient
replicas for the pool to continue functioning.
action: Attach the missing device and online it using ‘zpool online’.
see: http://www.sun.com/msg/ZFS-8000-3C
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
epool UNAVAIL 0 0 0 insufficient replicas
c3t60060E8005652C000000652C00002100d0s7 UNAVAIL 0 0 0 cannot open

The above error is caused by the zpool.cache file. This file contains the old paths of the disks from the previous server. The default behavior of Solaris 10 is to read the path from the zpool.cache file to speed up the boot sequence. You should delete this file and the system will recreate a fresh one during the boot up sequence.

Below are the steps to rename the zpool.cache file.

# cd /mnt/etc/zfs
# ls
zpool.cache
# mv zpool.cache zpool.cache.old

Now you need to reverse the changed you applied to the mountpoint earlier. Make sure that you change directory out of /mnt to /, otherwise the set mountpoint command will fail with the error device busy. Ignore the cannot mount ‘/’: directory is not empty message.

# zfs set mountpoint=/ rpool/ROOT/zfsboot
cannot mount ‘/’: directory is not empty
property may be set but unable to remount filesystem

Confirm that the mount points were changed.

# zfs list
NAME USED AVAIL REFER MOUNTPOINT
epool 260G 205G 21K /epool
rpool 7.74G 17.4G 99K /rpool
rpool/ROOT 4.74G 17.4G 21K legacy
rpool/ROOT/zfsboot 4.74G 17.4G 4.48G /
rpool/ROOT/zfsboot/var 139M 17.4G 105M /var
rpool/dump 1.00G 17.4G 1.00G -
rpool/swap 2.00G 19.4G 16K -

Shutdown the server.

# init 0

Now set the boot device in EEPROM.

setenv boot-device /pci@0,600000/pci@0/pci@8/SUNW,qlc@0/fp@0,0/disk@w50060e80058c7b10,1:a

The server is ready to be booted with the boot command.

{0} ok boot