{"id":815,"date":"2010-07-09T15:23:40","date_gmt":"2010-07-09T20:23:40","guid":{"rendered":"http:\/\/www.gamescheat.ca\/?p=815"},"modified":"2019-09-04T19:57:39","modified_gmt":"2019-09-05T00:57:39","slug":"booting-up-solaris-10-from-a-san-replicated-lun-on-a-different-sun-sparc-server","status":"publish","type":"post","link":"http:\/\/www.gamescheat.ca\/?p=815","title":{"rendered":"Booting up Solaris 10 from a SAN replicated LUN on a different Sun SPARC server"},"content":{"rendered":"<p>The quickest way to recover from a total disaster is to have some sort of replication implemented.  There are two different methods of  real-time replication, hardware and software.  My experiences with software replication such as Symantec Veritas Volume replicator for AIX was  not pleasing.  It required constant maintenance and troubleshooting. The best is hardware replication if you can afford it.  A lot of  organizations pick software replication as it generally cost a lot less up front, but the cost of maintenance eventually adds up.<\/p>\n<p>I will explain how to recover a Solaris 10 server from hardware replicated SAN disk.  It took me sometime to figure out how to boot up from  the replicated SAN LUN (disk), and many more hours to understand why the steps I applied works.<\/p>\n<p>In this example I have a SUN SPARC M3000 server with two Qlogic fiber channel cards (HBA) installed in the PCI solts.  The HBAs were already  configured to connect to the SAN disk (LUN).  This LUN contained the replicated copy of a production Slaris 10 server.  The production server had two ZFS pool residing in a single LUN.<\/p>\n<p>Using the Solaris 10 installation CD boot up the Sparc server into single user mode. <\/p>\n<blockquote><p>boot cdrom -s<\/p><\/blockquote>\n<p>The first thing you need to do is to see if the HBAs are working.  The connected status indictes that communication between the server and  switch is working.  There are two HBAs installed for redundancy, both connected to the same LUN.<\/p>\n<blockquote><p># luxadm -e port<br \/>\n\/devices\/pci@0,600000\/pci@0\/pci@8\/SUNW,qlc@0\/fp@0,0:devctl CONNECTED<br \/>\n\/devices\/pci@1,700000\/pci@0\/pci@0\/SUNW,qlc@0\/fp@0,0:devctl CONNECTED<\/p><\/blockquote>\n<p>Now you need to find out if the SAN disk is visible from the server. Even though both HBAs are connected to the same SAN disk, you will see  two separate SAN disks in the results below.  It just means there are two paths to the SAN.<\/p>\n<blockquote><p># luxadm probe<br \/>\nNo Network Array enclosures found in \/dev\/es<\/p>\n<p>Found Fibre Channel device(s):<br \/>\n  Node WWN:50060e80058c7b10  Device Type:Disk device<br \/>\n    Logical Path:\/dev\/rdsk\/c1t50060E80058C7B10d1s2<br \/>\n  Node WWN:50060e80058c7b00  Device Type:Disk device<br \/>\n    Logical Path:\/dev\/rdsk\/c2t50060E80058C7B00d1s2<\/p><\/blockquote>\n<p>In the above example the first LUN is c1t50060E80058C7B10d1s2. This is the logical device name which is a symbolic link to the physical  device name stored in the \/devices directory. Logical device names contain the controller number(c2), target number (t50060E80058C7B10), disk number (d1), and slice number (s2).<\/p>\n<p>The next step is to find out how the disk is partitioned, the format command will give you that information. You need this information to  understand how to boot up the disk.<\/p>\n<blockquote><p># format<br \/>\nSearching for disks\u2026done<\/p>\n<p>AVAILABLE DISK SELECTIONS:<br \/>\n0. c1t50060E80058C7B10d1 <HITACHI-OPEN-V-SUN-6006 cyl 65533 alt 2 hd 15 sec  \n1066>\/pci@0,600000\/pci@0\/pci@8\/SUNW,qlc@0\/fp@0,0\/ssd@w50060e80058c7b10,1<\/p>\n<p>1. c2t50060E80058C7B00d1 <HITACHI-OPEN-V-SUN-6006 cyl 65533 alt 2 hd 15 sec  \n1066>\/pci@1,700000\/pci@0\/pci@0\/SUNW,qlc@0\/fp@0,0\/ssd@w50060e80058c7b00,1<\/p><\/blockquote>\n<p>Select the first disk 0.<\/p>\n<blockquote><p>Specify disk (enter its number): 0<br \/>\nselecting c1t50060E80058C7B10d1<br \/>\n[disk formatted]<\/p>\n<p>FORMAT MENU:<br \/>\n        disk       \u2013 select a disk<br \/>\n        type       \u2013 select (define) a disk type<br \/>\n        partition  \u2013 select (define) a partition table<br \/>\n        current    \u2013 describe the current disk<br \/>\n        format     \u2013 format and analyze the disk<br \/>\n        repair     \u2013 repair a defective sector<br \/>\n        label      \u2013 write label to the disk<br \/>\n        analyze    \u2013 surface analysis<br \/>\n        defect     \u2013 defect list management<br \/>\n        backup     \u2013 search for backup labels<br \/>\n        verify     \u2013 read and display labels<br \/>\n        save       \u2013 save new disk\/partition definitions<br \/>\n        inquiry    \u2013 show vendor, product and revision<br \/>\n        volname    \u2013 set 8-character volume name<br \/>\n        !<cmd>     \u2013 execute <cmd>, then return<br \/>\n        quit<\/p><\/blockquote>\n<div style=\"float: left;\"><div style=\"margin: 15px 15px 15px 15px\";><script type=\"text\/javascript\"><!--\ngoogle_ad_client = \"pub-3319935785736004\";\ngoogle_alternate_color = \"FFFFFF\";\ngoogle_ad_width = 468;\ngoogle_ad_height = 60;\ngoogle_ad_format = \"468x60_as\";\ngoogle_ad_type = \"text_image\";\ngoogle_ad_channel =\"\";\ngoogle_color_border = \"cccccc\";\ngoogle_color_link = \"cc0000\";\ngoogle_color_bg = \"ffffff\";\ngoogle_color_text = \"000000\";\ngoogle_color_url = \"008000\";\n\/\/--><\/script>\n<script type=\"text\/javascript\"\n  src=\"http:\/\/pagead2.googlesyndication.com\/pagead\/show_ads.js\">\n<\/script><\/div><\/div><p>Display the labels and slices (partitions).  In Solaris each slice is treated as a separate physical disk.  In the below example you can tell that the disk is labeled as VTOC (Volume Table of Contents) because you can see the cylinders.  VTOC is also known as SMI label.  If the disk was labeled with EFI (Extensible Firmware Interface), then you would see sectors instead of cylinders.  Partition 0 (slice 0) holds the  operating system files, the boot disk.  Please note that you cannot boot from a disk with EFI label.  Slice 2 is the entire physical disk  because it contains all cylinders, 0 \u2013 65532.<\/p>\n<blockquote><p>format> verify<\/p>\n<p>Primary label contents:<\/p>\n<p>Volume name = <\nascii name  = <HITACHI-OPEN-V-SUN-6006 cyl 65533 alt 2 hd 15 sec 1066><br \/>\npcyl        = 65535<br \/>\nncyl        = 65533<br \/>\nacyl        =    2<br \/>\nnhead       =   15<br \/>\nnsect       = 1066<br \/>\nPart      Tag    Flag     Cylinders         Size            Blocks<br \/>\n  0       root    wm       0 \u2013  3356       25.60GB    (3357\/0\/0)    53678430<br \/>\n  1 unassigned    wm       0                0         (0\/0\/0)              0<br \/>\n  2     backup    wm       0 \u2013 65532      499.66GB    (65533\/0\/0) 1047872670<br \/>\n  3 unassigned    wm       0                0         (0\/0\/0)              0<br \/>\n  4 unassigned    wm       0                0         (0\/0\/0)              0<br \/>\n  5 unassigned    wm       0                0         (0\/0\/0)              0<br \/>\n  6 unassigned    wm       0                0         (0\/0\/0)              0<br \/>\n  7 unassigned    wm    3357 \u2013 65532      474.07GB    (62176\/0\/0)  994194240\n    <\/p><\/blockquote>\n<p>Here is what we know so far, we know that the disk name is c1t50060E80058C7B10d1. Slice 0 on this disk contains the boot files.  The physical path for this disk is \/pci@0,600000\/pci@0\/pci@8\/SUNW,qlc@0\/fp@0,0\/ssd@w50060e80058c7b10,1.  Now we need to find out the physical path for Slice 0.<\/p>\n<p>I know that the disk contains ZFS filesystems because it is a replica of the production disk.  When a ZFS filesystem is moved to a different  SPARC server it must first be imported because the hostid is different.<\/p>\n<p>List the ZFS pool contained on the disk using the zpool import command.  There are two ZFS pools in the below example, epool and rpool.  Take a note of the status and action.<\/p>\n<blockquote><p># zpool import<br \/>\n  pool: epool<br \/>\n    id: 16865366839830765202<br \/>\n state: ONLINE<br \/>\nstatus: The pool was last accessed by another system.<br \/>\naction: The pool can be imported using its name or numeric identifier and the \u2018-f\u2019 flag.<br \/>\n   see: http:\/\/www.sun.com\/msg\/ZFS-8000-EY<br \/>\nconfig:<\/p>\n<p>        epool                      ONLINE<br \/>\n          c2t50060E80058C7B00d1s7  ONLINE<\/p>\n<p>  pool: rpool<br \/>\n    id: 10594898920105832331<br \/>\n state: ONLINE<br \/>\nstatus: The pool was last accessed by another system.<br \/>\naction: The pool can be imported using its name or numeric identifier and the \u2018-f\u2019 flag.<br \/>\n   see: http:\/\/www.sun.com\/msg\/ZFS-8000-EY<br \/>\nconfig:<\/p>\n<p>        rpool                      ONLINE<br \/>\n          c2t50060E80058C7B00d1s0  ONLINE<\/p><\/blockquote>\n<p>Import the pool with the zpool import command.  The options -a will import all the ZFS pools it can find, the -f option will force the  import.  If you do not specify the force option, then the import may fail with the error \u201ccannot import \u2018rpool\u2019: pool may be in use from other system, it was last accessed by server name (hostid: 123456)\u201d.  Ignore the error message about failed to create mountpoint.<\/p>\n<blockquote><p># zpool import -af<br \/>\ncannot mount \u2018\/epool\u2019: failed to create mountpoint<br \/>\ncannot mount \u2018\/rpool\u2019: failed to create mountpoint<\/p><\/blockquote>\n<p>List the imported ZFS pools.<\/p>\n<blockquote><p># zpool list<br \/>\nNAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT<br \/>\nepool   472G   260G   212G    55%  ONLINE  \u2013<br \/>\nrpool  25.5G  5.75G  19.8G    22%  ONLINE  \u2013<\/p><\/blockquote>\n<p>List the ZFS filesystems. In the example below notice the mountpoint \/ is mounted on the zfs filesystem rpool\/ROOT\/zfsboot.  This is the boot partition, it resides in rpool.<\/p>\n<blockquote><p># zfs list<br \/>\nNAME                                     USED  AVAIL  REFER  MOUNTPOINT<br \/>\nepool                                    260G   205G    21K  \/epool<br \/>\nrpool                                   7.74G  17.4G    99K  \/rpool<br \/>\nrpool\/ROOT                              4.74G  17.4G    21K  legacy<br \/>\nrpool\/ROOT\/zfsboot                      4.74G  17.4G  4.48G  \/<br \/>\nrpool\/ROOT\/zfsboot\/var                   139M  17.4G   105M  \/var<br \/>\nrpool\/dump                              1.00G  17.4G  1.00G  \u2013<br \/>\nrpool\/swap                              2.00G  19.4G    16K  \u2013<\/p><\/blockquote>\n<p>Change the mountpoint for rpoo\/ROOT\/zfsboot to \/mnt so you can mount it to read the contents.<\/p>\n<blockquote><p># zfs set mountpoint=\/mnt rpool\/ROOT\/zfsboot<\/p><\/blockquote>\n<p>Confirm that the mountpoint was changed.<\/p>\n<blockquote><p># zfs list<br \/>\nNAME                                     USED  AVAIL  REFER  MOUNTPOINT<br \/>\nepool                                    260G   205G    21K  \/epool<br \/>\nrpool                                   7.74G  17.4G    99K  \/rpool<br \/>\nrpool\/ROOT                              4.74G  17.4G    21K  legacy<br \/>\nrpool\/ROOT\/zfsboot                      4.74G  17.4G  4.48G  \/mnt<br \/>\nrpool\/ROOT\/zfsboot\/var                   139M  17.4G   105M  \/mnt\/var<br \/>\nrpool\/dump                              1.00G  17.4G  1.00G  \u2013<br \/>\nrpool\/swap                              2.00G  19.4G    16K  \u2013\n           <\/p><\/blockquote>\n<p>Now mount rpool\/ROOT\/zfsboot.<\/p>\n<blockquote><p># zfs mount rpool\/ROOT\/zfsboot<\/p><\/blockquote>\n<p>List the logical disks.<\/p>\n<blockquote><p># cd \/dev\/dsk<br \/>\n# ls<br \/>\nc1t50060E80058C7B10d1s0  c2t50060E80058C7B00d1s0<br \/>\nc1t50060E80058C7B10d1s1  c2t50060E80058C7B00d1s1<br \/>\nc1t50060E80058C7B10d1s2  c2t50060E80058C7B00d1s2<br \/>\nc1t50060E80058C7B10d1s3  c2t50060E80058C7B00d1s3<br \/>\nc1t50060E80058C7B10d1s4  c2t50060E80058C7B00d1s4<br \/>\nc1t50060E80058C7B10d1s5  c2t50060E80058C7B00d1s5<br \/>\nc1t50060E80058C7B10d1s6  c2t50060E80058C7B00d1s6<br \/>\nc1t50060E80058C7B10d1s7  c2t50060E80058C7B00d1s7<\/p><\/blockquote>\n<p>As stated earlier the physical path for the disk we are looking for is \/pci@0,600000\/pci@0\/pci@8\/SUNW,qlc@0\/fp@0,0\/ssd@w50060e80058c7b10,1.   The boot slice is 0.  We can derive from the physical path that the disk name is 50060e80058c7b10.  We also know from the output of the format command that the physical disk 50060e80058c7b10 maps to the logical disk c1t50060E80058C7B10d1.  Therefore we can derive that the  logical boot disk is c1t50060E80058C7B10d1s0.<\/p>\n<p>Now find out what physical path c1t50060E80058C7B10d1s0 is a symbolic link for and that is your complete boot path.  In the below example the boot path starts at the first slash (\/) right after \/devices. It is \/pci@0,600000\/pci@0\/pci@8\/SUNW,qlc@0\/fp@0,0\/ssd@w50060e80058c7b10,1:a.  You need to replace ssd@ with disk@ when entering the path into EEPROM.<\/p>\n<blockquote><p># ls -l c1t50060E80058C7B10d1s0<br \/>\nlrwxrwxrwx   1 root     root          82 Jul  5 10:21 c1t50060E80058C7B10d1s0 -><br \/>\n ..\/..\/devices\/pci@0,600000\/pci@0\/pci@8\/SUNW,qlc@0\/fp@0,0\/ssd@w50060e80058c7b10,1:a<\/p><\/blockquote>\n<p>If you have more than one ZFS pool the non root pool may not get mounted upon booting up the server, as in this example.  You may get the  below error.<\/p>\n<blockquote><p>SUNW-MSG-ID: ZFS-8000-D3, TYPE: Fault, VER: 1, SEVERITY: Major<br \/>\nEVENT-TIME: Mon Jul  5 11:54:14 EDT 2010<br \/>\nPLATFORM: SUNW,SPARC-Enterprise, CSN: PX654321, HOSTNAME: Andrew-Lin<br \/>\nSOURCE: zfs-diagnosis, REV: 1.0<br \/>\nEVENT-ID: 33e5a9f1-49ac-6ebc-f2a9-dff25dea6b86<br \/>\nDESC: A ZFS device failed.  Refer to http:\/\/sun.com\/msg\/ZFS-8000-D3 for more information.<br \/>\nAUTO-RESPONSE: No automated response will occur.<br \/>\nIMPACT: Fault tolerance of the pool may be compromised.<br \/>\nREC-ACTION: Run \u2018zpool status -x\u2019 and replace the bad device.<\/p>\n<p>http:\/\/sun.com\/msg\/ZFS-8000-D3fameserverq9{root}: zpool status -x<br \/>\n  pool: epool<br \/>\n state: UNAVAIL<br \/>\nstatus: One or more devices could not be opened.  There are insufficient<br \/>\n        replicas for the pool to continue functioning.<br \/>\naction: Attach the missing device and online it using \u2018zpool online\u2019.<br \/>\n   see: http:\/\/www.sun.com\/msg\/ZFS-8000-3C<br \/>\n scrub: none requested<br \/>\nconfig:<\/p>\n<p>        NAME                                       STATE     READ WRITE CKSUM<br \/>\n        epool                                      UNAVAIL      0     0     0  insufficient replicas<br \/>\n          c3t60060E8005652C000000652C00002100d0s7  UNAVAIL      0     0     0  cannot open<\/p><\/blockquote>\n<p>The above error is caused by the zpool.cache file.  This file contains the old paths of the disks from the previous server. The default  behavior of Solaris 10 is to read the path from the zpool.cache file to speed up the boot sequence.  You should delete this file and the  system will recreate a fresh one during the boot up sequence.<\/p>\n<p>Below are the steps to rename the zpool.cache file.<\/p>\n<blockquote><p># cd \/mnt\/etc\/zfs<br \/>\n# ls<br \/>\nzpool.cache<br \/>\n# mv zpool.cache zpool.cache.old<\/p><\/blockquote>\n<p>Now you need to reverse the changed you applied to the mountpoint earlier. Make sure that you change directory out of \/mnt to \/, otherwise  the set mountpoint command will fail with the error device busy.  Ignore the cannot mount \u2018\/\u2019: directory is not empty message.<\/p>\n<blockquote><p># zfs set mountpoint=\/ rpool\/ROOT\/zfsboot<br \/>\ncannot mount \u2018\/\u2019: directory is not empty<br \/>\nproperty may be set but unable to remount filesystem                            <\/p><\/blockquote>\n<p>Confirm that the mount points were changed.<\/p>\n<blockquote><p># zfs list<br \/>\nNAME                                     USED  AVAIL  REFER  MOUNTPOINT<br \/>\nepool                                    260G   205G    21K  \/epool<br \/>\nrpool                                   7.74G  17.4G    99K  \/rpool<br \/>\nrpool\/ROOT                              4.74G  17.4G    21K  legacy<br \/>\nrpool\/ROOT\/zfsboot                      4.74G  17.4G  4.48G  \/<br \/>\nrpool\/ROOT\/zfsboot\/var                   139M  17.4G   105M  \/var<br \/>\nrpool\/dump                              1.00G  17.4G  1.00G  \u2013<br \/>\nrpool\/swap                              2.00G  19.4G    16K  \u2013<\/p><\/blockquote>\n<p>Shutdown the server.<\/p>\n<blockquote><p># init 0\n                            <\/p><\/blockquote>\n<p>Now set the boot device in EEPROM.<\/p>\n<blockquote><p>setenv boot-device \/pci@0,600000\/pci@0\/pci@8\/SUNW,qlc@0\/fp@0,0\/disk@w50060e80058c7b10,1:a<\/p><\/blockquote>\n<p>The server is ready to be booted with the boot command.<\/p>\n<blockquote><p>{0} ok boot<\/p><\/blockquote>\n","protected":false},"excerpt":{"rendered":"<p>The quickest way to recover from a total disaster is to have some sort of replication implemented. There are two different methods of real-time replication, hardware and software. My experiences with software replication such as Symantec Veritas Volume replicator for&hellip;<\/p>\n<p class=\"more-link-p\"><a class=\"more-link\" href=\"http:\/\/www.gamescheat.ca\/?p=815\">Read more &rarr;<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[65,19],"tags":[33,64,75,35],"_links":{"self":[{"href":"http:\/\/www.gamescheat.ca\/index.php?rest_route=\/wp\/v2\/posts\/815"}],"collection":[{"href":"http:\/\/www.gamescheat.ca\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.gamescheat.ca\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.gamescheat.ca\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.gamescheat.ca\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=815"}],"version-history":[{"count":7,"href":"http:\/\/www.gamescheat.ca\/index.php?rest_route=\/wp\/v2\/posts\/815\/revisions"}],"predecessor-version":[{"id":822,"href":"http:\/\/www.gamescheat.ca\/index.php?rest_route=\/wp\/v2\/posts\/815\/revisions\/822"}],"wp:attachment":[{"href":"http:\/\/www.gamescheat.ca\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=815"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.gamescheat.ca\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=815"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.gamescheat.ca\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=815"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}