It seems that I was able to login successfully into the server, but within few seconds it would kick me out. Just before it disconnected my ssh session, I saw the message ‘The server is going down..’. For some strange reason that message was stuck in the operating system, it was the message displayed just before the scheduled reboot. I had pushed out the reboot command from the Red Hat Satellite server. All the other servers rebooted fine.

I knew that clearing this message would resolve my dilemma, but I had no idea which file held it. After much searching I found the culprit, it was /etc/nologin. Due to reason I did not want to try and figure out as it would take too much time, this file did not get automatically deleted after the reboot. I deleted /etc/nologin, and then was able to successfully login via ssh to the server.

Here is what my /etc/resolv.conf looks like, the two DNS servers are defined correctly as it the search domain.
domain andrewlin.com
namesserver 10.10.10.11
namesserver 10.10.11.13
search andrewlin.com
The gateway was defined in /etc/defaultrouter.
10.10.0.1
It seems that ping did not know how to resolve an address. I was also not able to surf the internet with the Firefox browser in the Sun Java Desktop System for Soalris. All symptoms indicated that the nsswitch.conf did not have the correct environment defined. The /etc/nsswitch.conf file defines which services are to used to retrieve information such as hostnames, password files, and group files. So in this case for the command ping and firefox which wants to resolve and address, it will check the nsswitch.conf file to see what services to use.

Below are the lines I changed in /etc/nsswitch.conf, by adding dns to the end of each lines.
hosts: files dns
ipnodes: files dns
I am now able to ping and surf the internet without any issues.

# swap -l
swapfile dev swaplo blocks free
dev/dsk/c0t0d0s1 32,9 16 8425712 8425712

The output has five columns:
path
The path name for the swap area.

dev
The major/minor device number in decimal if it is a block special device; zeroes otherwise.

swaplo
The swaplow value for the area in 512-byte blocks.

blocks
The swaplen value for the area in 512-byte blocks.

free
The number of 512-byte blocks in this area that are not currently allocated.

To find the status of total swap usage use the -s option.

# swap -s
total: 180928k bytes allocated + 28456k reserved = 209384k used, 10691424k available

List the status of all the swap areas. The output has five columns:

path
The path name for the swap area.

dev
The major/minor device number in decimal if it is a block special device; zeroes otherwise.

swaplo
The swaplow value for the area in 512-byte blocks.

blocks
The swaplen value for the area in 512-byte blocks.

free
The number of 512-byte blocks in this area that are not currently allocated.

The total amount of swap space in bytes currently allocated for use as backing store.

reserved
The total amount of swap space in bytes not currently allocated, but claimed by memory mappings for possible future use.

used
The total amount of swap space in bytes that is either allocated or reserved.

available
The total swap space in bytes that is currently available for future reservation and allocation.

These numbers include swap space from all configured swap areas as listed by the -l option, as well swap space in the form of physical memory.

Here is an example, if you wanted to find out which file in the directory /home contains the word groundhog.

grep –i “groundhog” /home/*
/home/shadow:Today is groundhog day

The –i option means grep will ignore upper/lower case distinction during comparisons.
The word to search for is enclosed in double quotes, “groundhog”.
/home is the directory to search in and the * signifies all files.
The file /home/shadow contains the word groundhog, the content of the line containing the word groundhog is displayed on screen.

But what if you wanted to search for a string of words. Grep only searches for a single word. Egrep will search all files for a string of words. Here is an excerpt from the man pages of Solaris 10.

The egrep (expression grep) utility searches files for a
pattern of characters and prints all lines that contain that
pattern. egrep uses full regular expressions (expressions
that has string values that use the full set of
alphanumeric and special characters) to match the patterns.
It uses a fast deterministic algorithm that sometimes needs
exponential space.

Below is an example of egrep.

egrep –i “Apple announces the release of the ipad” /home/*
/home/obama: Today Apple announces the release of the ipad

#!/bin/sh
# This script archive directories and create a log file with the time and date stamp.
# Andrew Lin
# www.GamesCheat.ca
# 27 January, 2010

cd /backup/archive
echo “Backup start: ” > backup.log 2>&1
date >> backup.log 2>&1

cd /export/home

# tar cvf /backup/archive/backup.tar dir1 dir2 >> /backup/archive/backup.log 2>&1

tar cvf /backup/archive/backup.tar dir1 dir2 2>>/backup/archive/backup.log

echo “Backup Stop: ” >> /backup/archive/backup.log 2>&1
date >> /backup/archive/backup.log 2>&1

#!/bin/sh
This indicates that the script is written to be executed in the Bash shell.

The # sign indicates the line is a comment.

cd /backup/archive
Change the path /backup/archive to where this script is located on your machine.

echo “Backup start: ” > backup.log 2>&1
Create a file backup.log. The text Backup start: is inserted into this file. The file will be created in the directory
/backup/archive.

date >> backup.log 2>&1
The date will be appended into the file backup.log. This is useful if you wanted to determine the time of execution of the script.

cd /export/home
Go to the directory /export/home. This directory contains the files or directories you wish to backup.

# tar cvf /backup/archive/backup.tar dir1 dir2 >> /backup/archive/backup.log 2>&1
This line is commented out with the # sign. Uncomment this line if you want a detailed log of the tar command. Success and failure will be recorded in the log. Depending on the amount of files being archived this log could become very long.

tar cvf /backup/archive/backup.tar dir1 dir2 2>>/backup/archive/backup.log
This command will copy the directories dir1 and dir2. The copies will be stored in /backup/archive/backup.tar. Any error will be recorded in the file backup.log. If you wanted both error and success recorded then comment this line out and uncomment the line above it.

echo “Backup Stop: ” >> /backup/archive/backup.log 2>&1
After the completion of the tar command, insert the line Backup Stop: into the file backup.log.

date >> /backup/archive/backup.log 2>&1
Insert the date and time into the file backup.log. This is useful for determining the completion time/date of the script.

Save this file with the name of backup-script. Make sure that you make the file executable, with the chmod command, e.g.
chmod 777 backup-script.log

ls –b
scritps010/010/010/010

I tried every command I could think of, rmdir “scripts”, rmdir “scripts010/010/010/010”. The only way to remove this directory was to delete it using the inode number.

To determine the inode use the -i option with the ls command.
ls –i
184538 scritps

Below is the command syntax to delete the directory.
find . –inum 184538 –exec rmdir {} \;

If you wanted to delete a stubborn file then simply substitute rmdir with rm.
find . –inum 184538 –exec rm {} \;

Here is an excerpt from IBM, “Although some systems administrators still use the term “maintenance level” when discussing their AIX version, the term is now reserved for legacy AIX systems. The new IBM methodology dictates two TL releases per year. The first TL includes hardware features, enablement, and software services. The second includes software features in the release, which means the second release is larger and more comprehensive. Finally, there is also now support for new hardware on older TLs.”

To determine the machine type of an IBM AIX server use the uname command.

uname -MuL
IBM,9113-550 IBM,0210A463E 1 10-A463E

The option -M gives the machine type and model.
-u gives the plant code and machine identifier.
-L show the LPAR number and name.

The Machine type is 9113
The Model is 550
OF prefixm IBM, 0210A463E
Plant Code, 1
Sequence number, 10-A463E

To determine the AIX OS level and maintenance level use the instfix command with the option -i.

instfix -i | grep AIX_ML
All filesets for 5.3.0.0_AIX_ML were found.
All filesets for 5300-01_AIX_ML were found.
All filesets for 5300-02_AIX_ML were found.
Not all filesets for 5300-03_AIX_ML were found.
Not all filesets for 5300-04_AIX_ML were found.
Not all filesets for 5300-05_AIX_ML were found.IBM,9113-550 IBM,0210A463E 1 10-A463E

You can also use the command oslevel to determine the current AIX version, the -r option determines the highest recommended maintenance level.

oslevel -r
5300-02

Use the lsdev command to list the fiber channel cards (HBA) installed in the Aix server. Depending on the version of aix one or all of the commands below should work.

:/root > lsdev -C | grep fcs
fcs0 Available 04-08 FC Adapter
fcs1 Available 07-08 FC Adapter
fcs2 Available 0A-08 FC Adapter
fcs3 Available 0B-08 FC Adapter

OR

:/root > lscfg -vp | grep fcs
fcs2 U787B.001.DNW19B5-P1-C1-T1
FC Adapter
fcs3 U787B.001.DNW19B5-P1-C2-T1
FC Adapter
fcs0 U787B.001.DNW19B5-P1-C3-T1
FC Adapter
fcs1 U787B.001.DNW19B5-P1-C5-T1
FC Adapter

OR

itsm:/root > lsdev -Ccadapter | grep fcs
fcs0 Available 04-08 FC Adapter
fcs1 Available 07-08 FC Adapter
fcs2 Available 0A-08 FC Adapter
fcs3 Available 0B-08 FC Adapter

To find the WWN info for the Fc interface use the lscfg command. The Network Address is the WWN, e.g. Network Address………….10000000C942E65C.
The firmware level is Device Specific.(Z9)……..TS1.91A5.

:/root > lscfg -vl fcs0
fcs0 U787B.001.DNW19B5-P1-C3-T1 FC Adapter

Part Number……………..80P4543
EC Level………………..A
Serial Number……………1F4500AC78
Manufacturer…………….001F
Feature Code/Marketing ID…280B
FRU Number……………… 80P4544
Device Specific.(ZM)……..3
Network Address………….10000000C942E65C
ROS Level and ID…………02881955
Device Specific.(Z0)……..1001206D
Device Specific.(Z1)……..00000000
Device Specific.(Z2)……..00000000
Device Specific.(Z3)……..03000909
Device Specific.(Z4)……..FF801413
Device Specific.(Z5)……..02881955
Device Specific.(Z6)……..06831955
Device Specific.(Z7)……..07831955
Device Specific.(Z8)……..20000000C942E65C
Device Specific.(Z9)……..TS1.91A5
Device Specific.(ZA)……..T1D1.91A5
Device Specific.(ZB)……..T2D1.91A5
Device Specific.(YL)……..U787B.001.DNW19B5-P1-C3-T1

The command lsmcode will display the firmware and microcode level.
:/root > lsmcode -d fcs0

Below is the command to list the fileset.

:/root > lslpp -l | grep “FC Adapter”
devices.pci.77101223.com 5.3.0.51 APPLIED PCI FC Adapter (77101223)
5.3.0.50 APPLIED PCI FC Adapter (77101223)
devices.pci.77101223.rte 5.3.0.10 COMMITTED PCI FC Adapter (77101223)
devices.pci.77102224.com 5.3.0.51 APPLIED PCI-X FC Adapter (77102224)
devices.pci.77102224.diag 5.3.0.0 COMMITTED PCI-X FC Adapter (77102224)
devices.pci.77102224.rte 5.3.0.50 APPLIED PCI-X FC Adapter (77102224)
devices.pci.df1000f7.com 5.3.0.54 APPLIED Common PCI FC Adapter Device
5.3.0.50 APPLIED PCI FC Adapter Device
devices.pci.df1000f7.rte 5.3.0.30 APPLIED PCI FC Adapter Device Software
devices.pci.df1000f9.diag 5.3.0.0 COMMITTED 64-bit PCI FC Adapter Device
devices.pci.df1000f9.rte 5.3.0.30 APPLIED 64-bit PCI FC Adapter Device
devices.pci.df1000fd.rte 5.3.0.50 APPLIED 4Gb PCI-X FC Adapter Device
devices.pci.df1080f9.diag 5.3.0.0 COMMITTED PCI-X FC Adapter Device
devices.pci.df1080f9.rte 5.3.0.30 APPLIED PCI-X FC Adapter Device
devices.pci.77101223.com 5.3.0.0 COMMITTED PCI FC Adapter (77101223)
devices.pci.77102224.com 5.3.0.0 COMMITTED PCI-X FC Adapter (77102224)
devices.pci.df1000f7.com 5.3.0.54 APPLIED Common PCI FC Adapter Device
devices.pci.df1000f7.diag 5.3.0.0 COMMITTED PCI FC Adapter Device
devices.pci.df1000f7.rte 5.3.0.0 COMMITTED PCI FC Adapter Device Software

Cron jobs are generally stored in the default path of /var/spool/cron/crontabs. This path is common for most Unix and Linux versions. The easiest way to determine if a cron job is running is to goto /var/spool/cron/crontabs and list the files. You can cat the file to see the scheduled job.

I was only concerned about the scheduled jobs for all active userids and wanted to make sure that I captued them all.

The first thing I did was to list all active user accounts using the listusers command and redirecting the output to a file all_user_names. See the example below.

# listusers > all_user_names

You can view the file one page at a time using more, I copied the file to my desktop and inported it into Excel. This way I could then take notes.

# more all_user_names

You need to login as each user to see if they have any cron job. Since I did not know the password for each user accounts I simply used to su – user_name command to logon as the user. This will not prompt for password if you are already logged in as root or superuser.

# su – andrew

After loggin in as the user, run the command crontab -l to display any scheduled job.

server_promtp% crontab -l
# This command helps clean up user accounts.
1 0 * * 0 rm /home/andrew/*.log > /dev/null 2>&1

In the above example the # indicates a comment follows. All files with the extnsion .log is deleted very Sunday at 12:01 am.

If you get the message crontab: can’t open your crontab file, then it means that there are no schedule job for the the user.

Top can be download from unixtop.org, the documentation are also avaiable from this site. With Top you can also find out which account is responsible for utilizing the most cpu resources, and for how long the process has been running. It provides a real time snapshot of the running processes.

Microsoft Windows has the graphical version which is called Task Manager. To access task manager right click on the task bar, and select task manager.

Below is a snapshot of the Top display.

Mem: 2074120k total, 1290556k used, 783564k free, 91940k buffers
Swap: 4064312k total, 0k used, 4064312k free, 650100k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3093 root 15 0 154m 52m 19m S 6.3 2.6 2:08.97 firefox-bin
13423 apache 15 0 37636 20m 9m S 6.0 1.0 0:01.89 httpd
2586 mysql 16 0 138m 28m 4020 S 4.7 1.4 69:12.06 mysqld