Common boot problems

Can't Boot?

Watch the system closely as it boots, and take note of any error messages that appear. If the system complains that it is unable to mount the root filesystem, for example, this can be for any of several reasons:


* The BIOS cannot find the boot loader. This sometimes happens after you've installed Linux to dual-boot with Windows, but - out of concern
to not misconfigure the system - have asked the install program to place the boot loader in the Linux root (or /boot) filesystem.
The problem is that the BIOS can't see it there, unless you make that the active partition. The simplest fix is to reinstall
Linux and this time, let it place the LILO or GRUB boot loader into the Master Boot Record - don't worry, the Linux boot loaders are
automatically set up to let you choose Linux or Windows at boot time. It is possible to perform a more complex fix, for example by
copying the Linux boot loader sector into a file, and setting up the Windows NT/2K/XP boot loader to chain to it - but that is too
complex to describe here
(see http://www.lesbell.com.au/Home.nsf/web/Using+the+NT+Boot+Loader+to+Boot+Linux?OpenDocument where you'll find a longer article
describing how to use the NT boot loader to boot Linux).
* The kernel doesn't have a device driver to access the hard drive (e.g. a SCSI drive). Fix this by using the mkinitrd script to build a new
initrd file that contains the correct drivers, or recompile the kernel to include the driver code. This usually happens because you've
built a new kernel and slightly messed up the configuration.
* The kernel doesn't have a filesystem driver to access the root partition. For example, if the root filesystem is formatted with ext3,
then you will need the ext3 and jbd modules in the initrd or compiled into the kernel. Fix as for the previous problem. Again, this
usually happens after building a new kernel.
* The partition table has been modified, for example, by the installation of another operating system. In this case, edit the kernel
command line (in /ec/lilo.conf or /boot/grub/menu.lst) and the contents of /etc/fstab to contain the correct entries.
* Filesystems are corrupted, due to a power failure or system crash. Generally, after a system crash or power outage (what? No UPS?),
the system will come up and repair itself. If you are using a journalling filesystem like ext3fs, jfs, xfs or resiserfs, it will usually
perform a roll-forward recovery from its journal file and carry on. Even with the older ext2fs, the system usually runs an fsck
(file system check) on the various file systems and repairs them automatically. However, just occasionally manual intervention is
required - ; you might have to answer 'Y' to a string of questions (answering 'N' will get you nowhere unless you intend to
perform really low-level repairs yourself in a last-ditch attempt to avoid data loss). In the worst case, you might have to reboot from
rescue media and manuall run the e2fsck (or similar) command against each filesystem in turn. For example:

# e2fsck -p /dev/hda7

If the program complains that the superblock - the master block that links to everything else - is corrupted, it is useful to
remember that the superblock is so critical that it is duplicated every 8192 blocks through the filesystem and you can tell e2fsck to
use one of the backups:

# e2fsck -b 8193 /dev/hda7
* One or more filesystems cannot be found and mounted: Check the contents of /etc/fstab - in making quick alterations here, typographical
errors are common. You can use the e2label command to view the label of each filesystem: some distributions set these to the mount point
so you can figure out what is what.

Exploring Hardware Devices

A snapshot of all connected devices is maintained by HAL: Hardware Abstraction Layer
hal-device lists all devices in text mode
hal-device-manager displays all devices on a graphical window
lspci and lsusb list devices connected to the PCI and USB buses, respectively
The /proc and /sys filesystems also contain bus and device specific information


Hardware devices can be monitored in a number of different ways. The /proc filesystem has historically been the main method, with files such as /proc/devices, /proc/interrupts, /proc/iomem and /proc/ ioports. Buses such as the PCI and USB bus are also exposed through the /proc/bus/ directory.

To make the /proc entries more readable, utilities such as lspci and lsusb are also provided.

More recently, however, a new layer has been provided to expose hardware information: HAL (Hardware Abstraction Layer). HAL continuously maintains a snapshot of all hardware devices currently connected to the system. This snapshot may be monitored in text mode using the hal-device command, or in graphical mode with the hal-device- manager application.
# hal-device
# lspci
# lsusb

Much of the information provided through HAL can also be accessed from the /sys filesystem.

Possible commands for troubleshooting sound on linux

1. Could you please try disabling the onboard sound from BIOS settings. If the disabling of onboard sound doesn't help, please provide us the following command output,
#aplay -l
#cat /proc/asound/cards
#cat /proc/asound/modules
#cat /proc/asound/devices

2. Please run the following command,
# alsactl -f alsactl.out store 0

This will generate alsactl.out file for card ID 0. Provide us the alsactl.out file from your system to check for different settings.

3. Also, provide us the screenshot of the following command,
# alsamixer -V all

4. This is to test the speakers
# speaker-test -Dhw:0,0 -c2 -t wav

Why is the output of du and df inconsistant for some partitions

The output of du and df command may differ in some situations . For example:
# du -sh /opt
1.5G /opt

# df -hP /opt
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol05 6.0G 5.1G 594M 90% /opt

Here du shows 1.5G utilizations , whereas df shows 5.1G utilization

This behaviour could happen sometimes when a big file is deleted from the /opt partition. The inodes for the file could be reserved by a process still in memory. Restarting the related service would solve this issue.

For confirming, check the output of:
# lsof | grep '(deleted)'

In the left side of the related output entry, you could see the program which is holding the file and its pid.

You can restart the the program if possible. If there is any problem in restarting the program, you would have to restart the system.

Why does my file system gets readonly ?

The kernel will change a filesystem to read-only when it detects an I/O error against that file system. This can happen for a number of reasons:

- Your connection to your SAN dropped;

- Your hard drive(s) are dying;

- You have significant data corruption;

- and on and on...


Except for the first reason I listed, all of the other reasons are Real Bad.

If you're lucky, you've got some minor data corruption that caused the kernel to try to write beyond the end of the drive or something like that; you should try running fsck on the filesystem first. Be warned, though, that if you have significant data corruption, fsck may completely hose the filesystem, so get as good a backup as you can first.

You should check /var/log/messages for kernel messages about this. If it happens again, dmesg will also have useful information (at least, it will until you reboot).

If the problem is transient, a simple userspace mount call will fix it:
# mount -o remount,rw,usrquota /home

How to manually clean up shared memory / semaphors

This procedure may be useful for something like Oracle database, SAP or Peoplesoft when application was shutdown but shared memory hasn't been clened up for some reason.

Get the list of all semaphors for a user:
# ipcs | grep [username]

Remove (clean up) semaphor:
# ipcrm -s [semaphor id]

You can also use the command below to generate a script:
# ipcs | grep [username] | awk '{printf("ipcrm -s %s\n", $2);}'

How do you find whether the application leaks memory or not

You have a powerful tool called valgrind. Get the customer to install the appropriate debug packages. If it is net-snmp, install the correct net-snmp debuginfo packages

I will show two examples here, one for net-snmp and httpd

net-snmp
# valgrind -v --leak-check=full --show-reachable=yes --log-file=snmpd-memchk /usr/sbin/snmpd -f -Lsd -Lf /dev/null -p /var/run/snmpd.pid -a


httpd
# valgrind -v --leak-check=full --show-reachable=yes --log-file=httpd-memchk /usr/sbin/httpd -X


Allow the process/daemons to run for a day or more. Stop the valgrind process using CTRL-C and then attach the resultant file. The information would be collected in httpd-memchk. file. What you see here is basically the leak summary


Valgrind output
==3623== LEAK SUMMARY:
==3623== definitely lost: 1,754,671 bytes in 4,074 blocks.
==3623== indirectly lost: 1,321,068,200 bytes in 38,549 blocks.
==3623== possibly lost: 12,860,528 bytes in 251 blocks.
==3623== still reachable: 18,011,981 bytes in 133,360 blocks.
==3623== suppressed: 0 bytes in 0 blocks.

before you take any decision, Please consult with a senior engineer