How to change IO scheduler and LUN IO queue depth

To change scheduler to cfq for a LUN

# echo cfq > /sys/block/sdxx/queue/scheduler
# cat /sys/block/sdxx/queue/scheduler
  noop anticipatory deadline [cfq]


To change schedule to deadline on for all paths of all multiapth devices

# multipath -l|awk '$3~/^sd/ {print $3}'|while read dev; do echo deadline > /sys/block/$dev/queue/scheduler; done 
# multipath -l| awk '$3~/^sd/ {print $3}'|while read dev; do echo -n "$dev "; cat /sys/block/$dev/queue/scheduler; done 

Add elevator=deadline in grub.conf to retain it over reboot.

To change queue depth from 16 to 32 for Q-Logic HBA

# cat /sys/module/qla2xxx/parameters/ql2xmaxqdepth
16
# echo 32 > /sys/module/qla2xxx/parameters/ql2xmaxqdepth
# cat /sys/module/qla2xxx/parameters/ql2xmaxqdepth

Modify /etc/modprobe.conf to retain it over reboot
options qla2xxx ql2xmaxqdepth=32 qlport_down_retry=14 ql2xloginretrycount=30


To monitor LUN queue

# cat  $(echo /proc/scsi/sg/device{_hdr,s})|more
host   chan    id      lun     type    opens   qdepth  busy   online
3       0       0       12      0       1       32     5       1

If busy is reaching near to qdepth, we need to increase LUN qdepth. We should not increase qdepth too much !

The column headers in 'device_hdr' are given below. If the device is not present (and one is present after it) then a line of "-1" entries is output. Each entry is separated by a whitespace (currently a tab):

host            host number (indexes 'hosts' table, origin 0)
chan            channel number of device
id              SCSI id of device
lun             Logical Unit number of device
type            SCSI type (e.g. 0->disk, 5->cdrom, 6->scanner)
opens           number of opens (by sd, sr, sr and sg) at this time
depth           maximum queue depth supported by device
busy            number of commands being processed by host for this device
online          1 indicates device is in normal online state, 0->offline

Using vsftpd under Veritas VCS Application agent

Add a group

haconf -makerw
hagrp -add vsftpd_grp
hagrp -modify vsftpd_grp SystemList Node01 0 Node02 1
hagrp -autoenable vsftpd_grp -sys Node01
hagrp -autoenable vsftpd_grp -sys Node02
hagrp -modify vsftpd_grp OnlineRetryLimit 3
# start group on both nodes
hagrp -modify vsftpd_grp AutoFailOver 0
hagrp -modify vsftpd_grp Parallel 1

Add Resource

hares -add vsftpd_res Application vsftpd_grp
hares -modify vsftpd_res StartProgram "/etc/init.d/vsftpd start"
hares -modify vsftpd_res StopProgram  "/etc/init.d/vsftpd stop"
# if you want to run as root - ignore this line

hares -modify vsftpd_res User a_user_name

# command will be executed as su - a_user_name
hares -modify vsftpd_res UseSUDash 1                  
# Monitor process - whole string as appears in ps -ef output
hares -modify vsftpd_res MonitorProcesses \ 
   "/usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf"
# not tested
hares -modify vsftpd_res MonitorProgram "/etc/init.d/vsftpd status"
# does not write a pid file - ignore it
hares -modify vsftpd_res PidFiles "/var/run/vsftpd.pid"         
hares -modify vsftpd_res Enabled 1
# Make group dependendt on cluster filesystem (or any other group you wish)
hagrp -link vsftpd_grp CFS_Group online local firm
haconf -dump -makero

Bring vsftpd online

hagrp -online vsftpd_grp -sys Node01
hagrp -online vsftpd_grp -sys Node02

Snip from config file

group mq_group (
        SystemList = { Node01 = 0, Node02 = 1 }
        AutoFailOver = 0
        Parallel = 1
        AutoStartList = { Node01, Node02 }
        OnlineRetryLimit = 3
        )

        Application activemq (
                StartProgram = "/etc/init.d/vsftpd start"
                MonitorProcesses = { "/usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf" }
                StopProgram = "/etc/init.d/vsftpd stop"
                )

        requires group CFS_Group online local firm

Using bond interface for Veritas VCS LLT link

If you need to use bonded interface (say bond0) in Veritas VCS on linux, you may be getting jeopardy entries in gabconfig -a.


# gabconfig -a  ( on a three node cluster)
GAB Port Memberships
===============================================================
Port a gen   5afa01 membership 01
Port a gen   5afa01   jeopardy ;1
Port b gen   5afa04 membership 01
Port b gen   5afa04   jeopardy ;1
Port d gen   5afa07 membership 01
Port d gen   5afa07   jeopardy ;1
Port f gen   5afa13 membership 01
Port f gen   5afa13   jeopardy ;1
Port h gen   5afa09 membership 01
Port h gen   5afa09   jeopardy ;1
Port u gen   5afa11 membership 01
Port u gen   5afa11   jeopardy ;1
Port v gen   5afa0c membership 01
Port v gen   5afa0c   jeopardy ;1
Port w gen   5afa0e membership 01
Port w gen   5afa0e   jeopardy ;1
Port y gen   5afa0b membership 01
Port y gen   5afa0b   jeopardy ;1

You need to change llttab like bellow ( see set-dbg-minlinks 2) and it will go off !

# cat /etc/llttab

set-node

set-cluster 59195

link bond0 bond0 - ether - -

set-verbose 1

set-dbg-minlinks 2

A good working gabconfig will loook like:

# gabconfig -a

GAB Port Memberships

===============================================================

Port a gen   495401 membership 012

Port b gen   495405 membership 012

Port d gen   495408 membership 012

Port f gen   495417 membership 012

Port h gen   49540c membership 012

Port u gen   495414 membership 012

Port v gen   49540f membership 012

Port w gen   495411 membership 012

Port y gen   49540e membership 012

VxVM vxdg Error - cannot be added to a CDS disk group ERROR V-5-1-6478

Issue

Not able to add a disk on shared disk group.
VxVM vxdg ERROR V-5-1-6478 Device disk_1 cannot be added to a CDS disk group


Solution:

You are not executing command on master node !

- Find which node is mater node
vxdctl -c mode

- Initialize disk
vxdisksetup -i disk_1

- Add disk is diskgroup

vxdg -g diskgroup_name adddisk   a_good_disk_name=disk_1


Isn't simple ?

Rehat satellite satellite-sync fails with ORA-00001: unique constraint (RHNSAT.RHN_PACKAGE_FILE_PID_CID_UQ) violated

Problem

Redhat satellite 5.4.1 satellite-sync fails while syncing of one software channel with below erroe

10:00:00    Importing *relevant* package metadata: rhel-some-channel-name (99)

SYNC ERROR: unhandled exception occurred:

(Check logs/email for potentially more detail)


(54, 'ORA-00001: unique constraint (RHNSAT.RHN_PACKAGE_FILE_PID_CID_UQ) violated\n', '\n     Package Upload Failed due to uniqueness constraint violation.\n     Make sure the package does not have any duplicate dependencies or\n     does not already exists on the server\n     ')

Solution

1- Enable debug in rhn.conf

# echo 'debug = 7' >> /etc/rhn/rhn.conf


2- Run satellite sync for failed software channel

# satellite-sync -c rhel-some-channel-name


3- See package ID causing issue in log - say package-id causing problem is 12345

# less /var/log/rhn/rhn_server_satellite.log


4- Take DB backup and delete package_id

sqlplus $(spacewalk-cfg-get default_db)
SQL > delete from rhnPackageFile where package_id=12345 ;
SQL > commit;
SQl> quit


5- Repeat steps 2,3,4 until satellite-sync for rhel-some-channel-name is complete.


6- Run full satellite sync

satellite-sync


It's rare issue. Did it help you?

Error V-3-20005 and V-3-24996 while mounting VVR replicated filesystem as read only on secondary

Are you getting below error when trying to mount Veritas VVR replicated volume on secondary in read only mode?

root@secondary ~]# mount -t vxfs -o ro /dev/vx/dsk/product_dg  /mnt
UX:vxfs mount.vxfs: ERROR: V-3-20005: read of super-block on /dev/vx/dsk/product_dg/product_vol failed: Input/output error
UX:vxfs mount.vxfs: ERROR: V-3-24996: Unable to get disk layout version


Solution

Error is misleading !

Check RVG status. Most probably it is in DISABLE state. volume group is an object of replicated RVG. If RVG is disable, even volumes are started (ENABLED), it is not usable.

root@secondary ~]# vxprint -rt |grep ^rv
rv product_rvg      1            DISABLED CLEAN    secondary 3        srl_vol

Actions

1- Start RVG

vxrvg -g product_dg start product_rvg


2- Verify if rvg is enabled

root@secondary ~]# vxprint -rt |grep ^rv
rv product_rvg      1            ENABLED CLEAN    secondary 3        srl_vol


3-Mount filesystem as read only . NOTE- it is not recommended to mount replicated volume in even read-only on secondary )

root@secondary ~]# mount -t vxfs -o ro /dev/vx/dsk/product_dg  /mnt
root@secondary ~]#


Did this post help you ?

Rename Veitas Shared Disk Group

How do you rename a shared disk group cfs_group_name configured under SFCFSHA ?

Assumptions:

- cfs_group_name is VCS cluster group importing DG on multiple nodes at same time.
- Shred filesystem is mounted on all cluster nodes in read-write mode

• Bring cfs_group_name offline on all nodes. Repeat below for each node

         hagrp -offline cfs_group_name -sys node1

• Confirm cfs_group_name is offline on all nodes

          hastatus -sum

• Fine out which node is master node

         vxdctl -c mode

• From master node, deport DG on one node. It will deport on on all other node

        vxdg deport old_dg_name

•     Import DG on master node with new name

    vxdg -n app_data_dg -s import old_dg_name ## -s == shared

• Update Veritas config file

           haconf -makerw
     hares -modify cvmvoldg1 CVMDiskGroup app_data_dg
     hares -modify cfsmount1 BlockDevice /dev/vx/dsk/app_data_dg/appvol
     haconf -dump -makero  ## it will update config on all nodes

• Bring cfs_group_name online on all nodes (repeat below on each node)

          hagrp -online cfs_group_name -sys node1

• Verify shared filesystem has been monted sucessfully and there is no error in log

           df -hP ; tailf /var/VRTSvcs/log/engine_A.log

• On related note : How to rename volume of shared disk group ?

   Offline colume resource, rename volume, change cluster config using hares -modify and bring resource online.


     /usr/sbin/vxedit -g  diskgroup rename oldname newname