Steps to delete a defunct nagios host from centreon mysql database

If you have deleted a Centreon host from GUI but still is shows in Centreon monitoring service status, you may need to clean up host and related services from MySQl DB.

Below is steps to delete a dead host. If select statement do not return any record, obviously, corresponding delete is not required.


# mysqld
myysql> connect centreon_status;

myysql> SELECT host_id        FROM nagios_hosts WHERE display_name LIKE '%deadsystem01%';
myysql> SELECT host_object_id FROM nagios_hosts WHERE display_name= 'deadsystem01';
myysql> SELECT *              FROM nagios_hosts WHERE display_name = 'deadsystem01';

myysql> DELETE from nagios_hosts WHERE  display_name = 'deadsystem01';


myysql> SELECT * FROM nagios_objects WHERE name1 = 'deadsystem01';
myysql> DELETE   FROM nagios_objects WHERE name1 = 'deadsystem01';


myysql> SELECT * FROM nagios_hostgroup_members WHERE host_object_id = '123456';
myysql> DELETE   FROM nagios_hostgroup_members WHERE host_object_id = '123456';


myysql> SELECT * FROM nagios_hosts WHERE host_object_id = '123456';
myysql> DELETE   FROM nagios_hosts WHERE host_object_id = '123456';


myysql> SELECT * FROM nagios_hoststatus WHERE host_object_id = '123456';
myysql> DELETE   FROM nagios_hoststatus WHERE host_object_id = '123456';


myysql> SELECT * FROM nagios_objects WHERE name1 = 'deadsystem01';
myysql> DELETE   FROM nagios_objects WHERE name1 = 'deadsystem01';


myysql> SELECT service_object_id FROM nagios_services WHERE host_object_id = '123456';
myysql> DELETE FROM nagios_services WHERE host_object_id = '123456';


myysql> SELECT * FROM nagios_servicestatus WHERE service_object_id = '$above_id';
myysql> DELETE   FROM nagios_servicestatus WHERE service_object_id = '$above_id';

Puppet : how to append a string as new item into existing array ?

What is hiera?

Hiera is simple Lightweight Pluggable Hierarchical Database. It is a key/value lookup tool for hiera_lookupiguration data. Key/value may be string, array or hash.

Using Hiera

Suppose you have a a_hiera_hash in your yaml. It has a_hiera_key in two different yaml. At one place it is string ( hiera/location/dc1.yaml ) and at another place it is array (hiera/location/dc2.yaml)

==> hiera/location/dc1.yaml

a_hiera_hash:
    a_hiera_key: item_string


==> hiera/location/dc2.yaml

a_hiera_hash:
    a_hiera_key:
        - item_one
        - item_two
        - item_three

Adding a string in a string or array returned by hiera_hash

Now, if you want to add another string 'item_string_2' at the end of a_hiera_hash, there are 2 cases:

Case-1:

system1.dc1.com - a system in DC1 - For this system, a_hiera_key will be evaluated as string.

hiera_lookup = hiera_hash('a_hiera_hash', {})
hiera_lookup['a_hiera_key'] = 'item_string'  << this is what we will get

So we need to add 'item_string_2' at the end of above string. Puppet code will look like below.

        $a_hiera_key = $hiera_lookup['a_hiera_key']
        $a_hiera_key_new = "${a_hiera_key} item_string_2"
        $a_hiera_key = $a_hiera_key_new

Case-1:

system2.dc2.com - a system in DC2 - For this system, a_hiera_key will be evaluated as array.

hiera_lookup = hiera_hash('a_hiera_hash', {})
a_hiera_hash['a_hiera_key'] = [ 'item_one', 'item_two', 'item_three' ] << this is what we will get

So, now we need to add 'item_string_2' string as a new item of above array ! (type mismatch)


There is no straight way to add string as a new item of array item ! 

1- Convert array item into space seprated string using join library function - 'item_one item_two item_three'

        $a_hiera_key = join($hiera_lookup['a_hiera_key'], ' ')


2- Add 'item_string_2' at the end of string - 'item_one item_two item_three item_string_2'

        $a_hiera_key_new = "${a_hiera_key} item_string_2"

3- Split string using space a separator to make an array using split function- [ 'item_one', 'item_two', 'item_three', 'item_string_2' ]

         $a_hiera_key = split($a_hiera_key_new, ' ')



So, puppet code will look like something this.

hiera_lookup = hiera_hash('a_hiera_hash', {})

      if is_array($hiera_lookup['a_hiera_key']) {
        $a_hiera_key = join($hiera_lookup['a_hiera_key'], ' ')
        $a_hiera_key_new = "${a_hiera_key} item_string_2"
        $a_hiera_key = split($a_hiera_key_new, ' ')
      } else {
        $a_hiera_key = $hiera_lookup['a_hiera_key']
        $a_hiera_key_new = "${a_hiera_key} item_string_2"
        $a_hiera_key = $a_hiera_key_new
      }


NOTE:
If dc1.yaml also has array (similar to dc2.yaml), hiera_has will automatically merge 2 arrays. For eacmple, if dc1.yaml array is following

a_hiera_hash:
    a_hiera_key:
        - item_dc1


Then, a_hiera_key will be merge as below.

hiera_lookup = hiera_hash('a_hiera_hash', {})
a_hiera_hash['a_hiera_key'] = [ 'item_one', 'item_two', 'item_three', 'item_dc1'] << this is what we will get

References:
https://github.com/puppetlabs/hiera (hiera)
https://docs.puppetlabs.com/hiera/1/lookup_types.html (hiera_hash)
https://docs.puppetlabs.com/references/latest/function.html (split)
https://forge.puppetlabs.com/puppetlabs/stdlib/readme (join)

Veritas storage foundation and replication manula stop and start steps

If we need to stop and start Veritas Storage Foundation and Veritas Replication to avoid reboot, below steps should work.

Steps to stop

umount /some/vxfs/filesystems
service vxfs_replication stop
service vxodm stop
service vxdbdctrl stop
service vxpbx_exchanged stop
service vxrsyncd.sh stop
service vxnm-vxnetd stop
service vxfs stop
service vxvm-boot stop

Steps to start

service vxfs start
service vxvm-boot start
service vxpbx_exchanged start
service vxnm-vxnetd start
service vxdbdctrl start
service vxvm-reconfig start
service vxvm-recover start
service vxodm start
service vxrsyncd.sh start
service vxfs_replication start
mount /some/vxfs/filesystems

A typical .gitconfig that make my life easier when working with git

This is what make my life working with git easier. alisa save thousands of keystrokes every weeks !


$cat .gitconfig
[user]
        name = Nasimuddin Ansari
        email = Nasimuddin Ansari@example.com
[push]
        default = matching
[branch]
        autosetupbase = always
        autosetuprebase = always
[color]
        ui = true
        status = auto
        branch = auto
[core]
        editor = vi
        excludesfile = ~/.gitignore

[alias]
        bd = branch -d
        l = log --graph --decorate --oneline --all -n15
        rbm = rebase master
        co = checkout
        cm = checkout master
        ss = status -s
        s = status
        b = branch -v
        a = add .
        c = commit -a
        ca = commit --amend
        d = diff
        p = pull
        dc = diff --cached
        dh = diff HEAD
        dhh = diff HEAD^
        cb = checkout -b
        r = review -v
        rn = review -n

[help]
        autocorrect = 3

Ref: https://gist.github.com/pksunkara/988716

Veritas VVR - removing a volume from VVR and deleting it from disk group

Follow below steps to remove a volume  from VVR replication and remove it from disk group.


1- Unmount filesystem using volume ( say app_vol)

2- Run below command on Primary site to delete volume from VVR

  vradmin -g app_dg -f delvol app_rvg app_vol

3- Confirm volume has been removed from VVR

 vradmin -g app_dg -l repstatus app_rvg
 vradmin printvol

4- Now remove, volume from Primary as well as secondary systems

  vxvol -g app_dg stop app_vol
 vxassist -g app_dg remove volume app_vol

5- Confirm, volume has been removed from Disk Group and space has been freed up

vxprint -v
vxdg -g app_dg -u h free

6-If you want to use freed up space in another replicated volume, follow.

Extending Veritas Volume Replication Group

Reset (unbind and bind) a HBA without rebooting Linux system

One of path of SAN LUN is in 'failed faulty running' state. I reset HBA and it has brought path in active state.


>>Below shows that one path of LUN was failed.

# multipath -ll
mpath999 (3600a0b80000b5c9c0000044d3b667c19) dm-0 STORAGE,VV
size=200G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 4:0:0:0 sdb 8:16 failed faulty running
  `- 1:0:0:0 sdc 8:32 active ready  running



>> 'fdisk -l' did not list anything on faulty path but works fine on working one.

# fdisk -l /dev/sdb <
#
# fdisk -l /dev/sdc

Disk /dev/sdc: 214.7 GB, 214748364800 bytes
255 heads, 63 sectors/track, 26108 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 16384 bytes / 16777216 bytes
Disk identifier: 0x00000000



>> 'dmsetup status' also shows path is in in F state ( 8:16 F )

# dmsetup status mpath999
0 419430400 multipath 2 0 0 0 1 1 A 0 2 0 8:16 F 1 8:32 A 0



>> Rescanning HBA, restarting multipathd and resetting - none of bring faulty path online.

# for scsi_bus in $(ls /sys/class/scsi_host/); do echo '- - -' > /sys/class/scsi_host/$scsi_bus/scan; done
# /etc/init.d/multipathd restart
# echo '1' >/sys/class/fc_host/host4/issue_lip



>> On this system, 2 FC hosts are online, hence each SAN disk should have 2 paths. Also take note of host4 device ID - 0000:21:00.1

# cd /sys/class/fc_host/
# ls -l
total 0
lrwxrwxrwx 1 root root 0 Jun 21  2014 host1 -> ../../devices/pci0000:00/0000:00:03.0/0000:05:00.0/host1/fc_host/host1
lrwxrwxrwx 1 root root 0 Jun 21  2014 host2 -> ../../devices/pci0000:00/0000:00:03.0/0000:05:00.1/host2/fc_host/host2
lrwxrwxrwx 1 root root 0 Jun 21  2014 host3 -> ../../devices/pci0000:20/0000:20:03.0/0000:21:00.0/host3/fc_host/host3
lrwxrwxrwx 1 root root 0 Jun 21  2014 host4 -> ../../devices/pci0000:20/0000:20:03.0/0000:21:00.1/host4/fc_host/host4
# cat host1/port_state
Online
# cat host?/port_state
Online
Linkdown
Linkdown
Online



>> Check what are devices connected with problematic HBA host

# ls -l /dev/disk/by-path|grep 0000:21:00.1
lrwxrwxrwx 1 root root  9 Apr 13 10:11 pci-0000:21:00.1-fc-0x22220002ac004714-lun-0 -> ../../sdb



>>Force all the paths attached to that HBA to fail, and remove all the device entries from the system

# faulty_hba=0000:21:00.1
# for d in $(ls -l /dev/disk/by-path/| grep $faulty_hba |awk -F'/' '{print $NF}') 
    echo "Dealing with $d" 
    echo "fail path $d" | multipathd -k 
    echo "del path $d" | multipathd -k 
    echo 1 > /sys/block/$d/device/delete 
done 



>> failed path should disappear from multipath

# multipath -ll
mpath999 (3600a0b80000b5c9c0000044d3b667c19) dm-0 STORAGE,VV
size=200G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 1:0:0:0 sdc 8:32 active ready  running



>> unbind (off line) problematic HBA- take note that host4 ID is 0000:21:00.1 - it will disappear from driver list.

# cd /sys/bus/pci/drivers/qla2xxx/
# ll
total 0
lrwxrwxrwx 1 root root    0 Apr 13 10:09 0000:05:00.0 -> ../../../../devices/pci0000:00/0000:00:03.0/0000:05:00.0
lrwxrwxrwx 1 root root    0 Apr 13 10:09 0000:05:00.1 -> ../../../../devices/pci0000:00/0000:00:03.0/0000:05:00.1
lrwxrwxrwx 1 root root    0 Apr 13 10:09 0000:21:00.0 -> ../../../../devices/pci0000:20/0000:20:03.0/0000:21:00.0
lrwxrwxrwx 1 root root    0 Apr 13 10:09 0000:21:00.1 -> ../../../../devices/pci0000:20/0000:20:03.0/0000:21:00.1
--w------- 1 root root 4096 Apr 13 10:09 bind
lrwxrwxrwx 1 root root    0 Apr 13 10:09 module -> ../../../../module/qla2xxx
--w------- 1 root root 4096 Apr 13 10:09 new_id
--w------- 1 root root 4096 Apr 13 10:09 remove_id
--w------- 1 root root 4096 Jun 21  2014 uevent
--w------- 1 root root 4096 Apr 13 10:09 unbind
# echo '0000:21:00.1' > unbind
# ll
total 0
lrwxrwxrwx 1 root root    0 Apr 13 10:09 0000:05:00.0 -> ../../../../devices/pci0000:00/0000:00:03.0/0000:05:00.0
lrwxrwxrwx 1 root root    0 Apr 13 10:09 0000:05:00.1 -> ../../../../devices/pci0000:00/0000:00:03.0/0000:05:00.1
lrwxrwxrwx 1 root root    0 Apr 13 10:09 0000:21:00.0 -> ../../../../devices/pci0000:20/0000:20:03.0/0000:21:00.0
--w------- 1 root root 4096 Apr 13 10:09 bind
lrwxrwxrwx 1 root root    0 Apr 13 10:09 module -> ../../../../module/qla2xxx
--w------- 1 root root 4096 Apr 13 10:09 new_id
--w------- 1 root root 4096 Apr 13 10:09 remove_id
--w------- 1 root root 4096 Jun 21  2014 uevent
--w------- 1 root root 4096 Apr 13 10:11 unbind


>> bind (on line) host4 HBA again - corresponding entry should be available

# echo '0000:21:00.1' > bind
# ll
total 0
lrwxrwxrwx 1 root root    0 Apr 13 10:09 0000:05:00.0 -> ../../../../devices/pci0000:00/0000:00:03.0/0000:05:00.0
lrwxrwxrwx 1 root root    0 Apr 13 10:09 0000:05:00.1 -> ../../../../devices/pci0000:00/0000:00:03.0/0000:05:00.1
lrwxrwxrwx 1 root root    0 Apr 13 10:09 0000:21:00.0 -> ../../../../devices/pci0000:20/0000:20:03.0/0000:21:00.0
lrwxrwxrwx 1 root root    0 Apr 13 10:11 0000:21:00.1 -> ../../../../devices/pci0000:20/0000:20:03.0/0000:21:00.1
--w------- 1 root root 4096 Apr 13 10:11 bind
lrwxrwxrwx 1 root root    0 Apr 13 10:09 module -> ../../../../module/qla2xxx
--w------- 1 root root 4096 Apr 13 10:09 new_id
--w------- 1 root root 4096 Apr 13 10:09 remove_id
--w------- 1 root root 4096 Jun 21  2014 uevent
--w------- 1 root root 4096 Apr 13 10:11 unbind



>> HBA port may come up with different name - host7 ( instead of host4)

# ll /sys/class/fc_host/
total 0
lrwxrwxrwx 1 root root 0 Jun 21  2014 host1 -> ../../devices/pci0000:00/0000:00:03.0/0000:05:00.0/host1/fc_host/host1
lrwxrwxrwx 1 root root 0 Jun 21  2014 host2 -> ../../devices/pci0000:00/0000:00:03.0/0000:05:00.1/host2/fc_host/host2
lrwxrwxrwx 1 root root 0 Jun 21  2014 host3 -> ../../devices/pci0000:20/0000:20:03.0/0000:21:00.0/host3/fc_host/host3
lrwxrwxrwx 1 root root 0 Apr 13 10:12 host7 -> ../../devices/pci0000:20/0000:20:03.0/0000:21:00.1/host7/fc_host/host7



>> And, all faulty path of LUN should be active and ready (Assuming there is no other problem of FC cables/SAN network, Storage host port etc)!
# multipath -ll
mpath999 (3600a0b80000b5c9c0000044d3b667c19) dm-0 STORAGE,VV
size=200G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 1:0:0:0 sdc 8:32 active ready running
  `- 7:0:0:0 sdb 8:16 active ready running



Reference: https://access.redhat.com/solutions/287303

Linux kernel - crash utility to analysis vmcore

What is crash ?

A tool to interactively analyzing the state of the crash generated by kdump, netdump, diskdump, LKCD, xendump or kvmdump

What are packages required?

yum install crash kernel-debuginfo-$(uname -r) kernel-debuginfo-common-x86_64-$(uname -r) 

How to check created kernel crash file vmcore is valid?

If below command will show date when kernel crash was created and exit - then vmcore is created.

# crash -st /usr/lib/debug/lib/modules/$(uname -r)/vmlinux vmcore
Tue Mar 10 04:02:55 2015

Is vmcore created by crash is valid?

It will take couple of minutes to load kernel and show you crash prompt if crash is valid and you are using correct debug package. It will show number of cpu, system name, memory size, time when crash was created, load of system, date when system was crashed.

# crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux vmcore
crash 5.1.8-2.el5_9
Copyright (C) 2002-2011  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

      KERNEL: /usr/lib/debug/lib/modules/2.6.18-308.8.2.el5/vmlinux
    DUMPFILE: vmcore
        CPUS: 2
        DATE: Tue Mar 10 04:02:55 2015
      UPTIME: 1 days, 00:09:41
LOAD AVERAGE: 9.29, 4.35, 2.80
       TASKS: 205
    NODENAME: dev001.example.com
     RELEASE: 2.6.18-308.8.2.el5
     VERSION: #1 SMP Tue May 29 11:54:17 EDT 2012
     MACHINE: x86_64  (2699 Mhz)
      MEMORY: 3.9 GB
       PANIC: "Kernel panic - not syncing: out of memory. panic_on_oom is selected"
         PID: 3419
     COMMAND: "sshd"
        TASK: ffff81013b2e5860  [THREAD_INFO: ffff81012f594000]
         CPU: 1
       STATE: TASK_RUNNING (PANIC)

How to use crash commands ?

-log: dump lernel log_buf contents in chronological order.Most interesting information will be at the end of log file. kernel thread dump, memory dump, swap usages etc.

crash>log
.....
lowmem_reserve[]: 0 3000 4010 4010
Node 0 DMA32 free:10008kB min:6052kB low:7564kB high:9076kB active:1483204kB inactive:1420176kB present:3072160kB pages_scanned:10951420 all_unreclaimable? yes
lowmem_reserve[]: 0 0 1010 1010
Node 0 Normal free:1980kB min:2036kB low:2544kB high:3052kB active:442232kB inactive:458364kB present:1034240kB pages_scanned:1716084 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 4*4kB 3*8kB 4*16kB 1*32kB 3*64kB 4*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10056kB
Node 0 DMA32: 0*4kB 5*8kB 1*16kB 1*32kB 1*64kB 1*128kB 4*256kB 1*512kB 0*1024kB 0*2048kB 2*4096kB = 10008kB
Node 0 Normal: 5*4kB 5*8kB 0*16kB 0*32kB 2*64kB 4*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1980kB
Node 0 HighMem: empty
10029 pagecache pages
Swap cache: add 127866954, delete 127858677, find 188220155/188834341, race 0+1692
Free swap  = 0kB
Total swap = 2031608kB
Free swap:            0kB
1310720 pages of RAM
332619 reserved pages
20496 pages shared
8412 pages swap cached
Kernel panic - not syncing: out of memory. panic_on_oom is selected


INFO: task kjournald:602 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kjournald     D ffff810009004420     0   602     49           631   578 (L-TLB)

INFO: task nagios:20831 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
nagios        D ffff810009004420     0 20831   3763               20729 (NOTLB)



-ps : process status at time of crash. Active tasks have a preceding >

crash>ps
   PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
      0      0   0  ffffffff80319b60  RU   0.0       0      0  [swapper]
      0      1   1  ffff8101047460c0  RU   0.0       0      0  [swapper]
      1      0   1  ffff8101047347a0  IN   0.0   10372    484  init
      2      1   0  ffff810104734040  IN   0.0       0      0  [migration/0]
  2979      1   1  ffff81013f5827a0  IN   0.0   28664    356  restorecond
>  3419      1   1  ffff81013b2e5860  RU   0.0   60808    456  sshd
   2988  19639   0  ffff810130580820  IN   0.0    5876    388  vmstat
   2990      1   1  ffff81013cb267a0  IN   0.0    5932    472  syslogd
crash>



-bt : display a kernel stack backtrace. bt -a - stack tradce of all active tasks

crash> bt
PID: 3419   TASK: ffff81013b2e5860  CPU: 1   COMMAND: "sshd"
 #0 [ffff81012f595aa0] crash_kexec at ffffffff800b099c
 #1 [ffff81012f595b60] panic at ffffffff80093989
 #2 [ffff81012f595c50] out_of_memory at ffffffff800caa5d
 #3 [ffff81012f595ca0] __alloc_pages at ffffffff8000f612
 #4 [ffff81012f595d10] read_swap_cache_async at ffffffff80032415
 #5 [ffff81012f595d50] swapin_readahead at ffffffff800d0777
 #6 [ffff81012f595da0] __handle_mm_fault at ffffffff800092d9
 #7 [ffff81012f595e60] do_page_fault at ffffffff80067202
 #8 [ffff81012f595f50] error_exit at ffffffff8005dde9
    RIP: 00002b3e2f66a25a  RSP: 00007fff86964840  RFLAGS: 00010206
    RAX: 0000000000000000  RBX: 00002b3e2f9261d8  RCX: 00002b3e2f9261d8
    RDX: 00002b3e4a2363e0  RSI: 000000000000000a  RDI: 0000000000000001
    RBP: 00007fff86964860   R8: 00002b3e4a2363e0   R9: 000000000000000a
    R10: 00007fff86964e10  R11: 0000000000000246  R12: 00002b3e4a236570
    R13: 00007fff86964e50  R14: 0000000000000001  R15: 00002b3e2cd76078
    ORIG_RAX: ffffffffffffffff  CS: 0033  SS: 002b
crash>


-bt : to backtrace a PID

crash > bt
crash > bt -f



-bt -f : display all stack data contained in a frame; this option can be used to determine the arguments passed to each function

crash>bt -f
 #0 [ffff81012f595aa0] crash_kexec at ffffffff800b099c
 #1 [ffff81012f595b60] panic at ffffffff80093989
 #2 [ffff81012f595c50] out_of_memory at ffffffff800caa5d



-kmem : details of kernel memory location e.g. For #2 above

crash> kmem ffffffff800caa5d
ffffffff800caa5d (T) out_of_memory+593 ../debug/kernel-2.6.18/linux-2.6.18-308.8.2.el5.x86_64/mm/oom_kill.c: 506

      PAGE       PHYSICAL      MAPPING       INDEX CNT FLAGS
ffff810100009c30   2ca000                0        0  1 400
crash>




-whatis : search symbol table for data or type information

crash> whatis crash_kexec
void crash_kexec(struct pt_regs *);

crash> whatis panic
void panic(const char *, ...);

crash> whatis out_of_memory
void out_of_memory(struct zonelist *, gfp_t, int, int);



-some_more_commands

crash> sys|egrep -i "cpu|date|uptime|load|tasks|node|release|machine|memory|panic"
    DUMPFILE: /cores/retrace/tasks/152431766/crash/vmcore  [PARTIAL DUMP]
        CPUS: 24
        DATE: Wed Sep  9 07:54:53 2015
      UPTIME: 322 days, 11:00:25
LOAD AVERAGE: 0.13, 0.13, 0.13
       TASKS: 755
    NODENAME: ms00456
     RELEASE: 2.6.32-358.el6.x86_64
     MACHINE: x86_64  (2494 Mhz)
      MEMORY: 192 GB
       PANIC: "Kernel panic - not syncing: An NMI occurred, please see the Integrated Management Log for details."

crash> rd -a 0xffffffff8201c001 100
ffffffff8201c000:  HP
ffffffff8201c004:  P70
ffffffff8201c008:  2.8
ffffffff8201c010:  12/20/2013  <<<
ffffffff8201c01c:  HP
ffffffff8201c020:  ProLiant DL380p Gen8


crash> dis -rl 0xffffffffa00574ca
0xffffffffa00574c8 : callq  0xffffffff8150cf21
/usr/src/debug/kernel-2.6.32-358.el6/linux-2.6.32-358.el6.x86_64/drivers/watchdog/hpwdt.c: 495


crash> mod |grep -E "NAME|"
     MODULE       NAME                   SIZE  OBJECT FILE
ffffffffa00581a0  hpwdt                  7094  /cores/retrace/repos/kernel/x86_64/usr/lib/debug/lib/modules/2.6.32-358.el6.x86_64/kernel/drivers/watchdog/hpwdt.ko.debug 


crash> px ((struct module *)0xffffffffa00581a0)->name
$2 = "hpwdt\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"

crash> px ((struct module *)0xffffffffa00581a0)->version
$3 = 0xffff88301782cb20 "1.3.0"


crash> px ((struct module *)0xffffffffa00581a0)->srcversion
$4 = 0xffff8830194fe4c0 "87D39D97B9E0A6F667C8671"


crash> px ((struct module *)0xffffffffa00581a0)->gpgsig_ok
$5 = 0x1


crash> px notify_die
notify_die = $6 = 
 {int (enum die_val, const char *, struct pt_regs *, long, int, int)} 0xffffffff8109cbd0

crash> eval -b 00000000000000f1

-Other crash command

help : list all supported crash command
help : to know available option and details of a crash command e.g 'help bt'
net : show network interfaces and ip address configured
mount : show mount point and mount option at time of crash
sys : system specific information excatly what you see when crash was started
swap : swap usages
task : task structure of running task. In above example, it is sshd with PID 3419
runq : displays the tasks on the run queues of each cpu
set -v : display the current state of internal crash variables.

How to run crash analysis unattended and save in a text file ?

# cat > inputfile.txt
sys
mount
net
swap
log
ps
runq
bt
bt -a
bt -g
bt -t
bt -al
bt -f
foreach bt
task
kmem
kmem -i
kmem -S
exit

# crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux vmcore -i inputfile.txt > crash-analysys.txt
# less crash-analysys.txt

- References
http://people.redhat.com/anderson/crash_whitepaper/
http://www.dedoimedo.com/computers/crash-book.html
https://codeascraft.com/2012/03/30/kernel-debugging-101/