Ansari's Technical Notes: 2016

Friday, September 23, 2016

Manually installing UEFI and grub2 on USB already have Ubuntu

You might have installed Ubuntu on USB drive ( using steps similar to my other blog) but UEFI and grub is not installed on USB drive hence you can not use same USB drive to boot Ubuntu on other system.

A Little Theory

1-UEFI industry standard boot procedure for personal computers from 2006. Firmware of your computer hardware knows that it need to look far 'Unifed Extensible Firmware Interface' install on available storage.

2-System native firmware scans all storage devices for ESP ( EFI System Partition - a variant of vfat32 partition with boot flag enabled). gparted shows boot and esp flags for this partition. Each storage device must have only on ESP.

3-Firmware mounts this (a variant of vfat32) partition to /boot/efi and locates /boot/efi/EFI/

/*..efi file and execute it. ( remember - filename on vfat32 partition is case insensitive). It triggers grub. This file has location of grub. See this for more detail.

4-UEFI does not rely on boot sector. Instead it invoke boot manager like - grub. In turn, grub menu will appear. Grub configuration have /boot partition information from where kernel and initrd will be loaded, uncopressed and executed. they will mount root flesystem and trigger user space process init ( pid=1 ).

Installing UEFI and grub2 on a same USB from which Ubuntu is booted

You are able to boot Ubuntu installed on an USB via a chainloder boot loader and want to install EFI and grub2 on this. Follow below seps to install EFI and grub.

1-Ensure you have an ESP partition of USB Flash drive - sdX and it is mount on /boot/efi. Use gparted to check flag of this partition - boot, esp
df -PhalT /boot/efi Filesystem Type Size Used Avail Use% Mounted on /dev/sdb2 vfat 286M 132K 286M 1% /boot/efi

2-Move old stuff somewhere and install EFI and grub

mkdir /var/tmp/boot_efi_backup
mv /boot/efi/* /var/tmp/boot_efi_backup
apt-get install --reinstall grub-efi-amd64
grub-install --removable --boot-directory=/boot --efi-directory=/boot/efi /dev/sdX
( above command use whole disk as last argument - do not partition /dev/sdX1 )
find /boot/efi/ /boot/efi/ /boot/efi/EFI /boot/efi/EFI/BOOT /boot/efi/EFI/BOOT/BOOTX64.EFI

3-Look for some interesting thing it has.

strings /boot/efi/EFI/BOOT/BOOTX64.EFI |grep -Ei "grub|hd|setuuid|root|boot|set"|less
ghex /boot/efi/EFI/BOOT/BOOTX64.EFI

4-Reboot with UEFI ready USB

Installing EFI and grub2 on a USB having Ubuntu while booted from other Ubuntu

If you want to install EFI and grub2 on a USB having an Ubuntu by mounting it on another instance of Ubuntu, Installer Media OS, Rescue CD, Live DVD etc., follow below steps. Creates ESp partition on USB if it dows not have. If you can not create it, you can not install UEFI !

1-Mount root, boot and EFI partition

mount /dev/sdYn /mnt

mount /dev/sdYn /mnt/boot

mount /dev/sdYp /mnt/boot/efi

2-Take backup, chroot, install EFI and grub2

mkdir /mnt/boot_efi_backup

mv /mnt/boot/efi/* /mnt/boot_efi_backup/

chroot /mnt

apt-get install --reinstall grub-efi-amd64

grub-install --removable --target=x86_64-efi --boot-directory=/mnt/boot --efi-directory=/mnt/boot/efi /dev/sdY

4-Reboot with UEFI ready USB

Installing grub2 in BIOS mode.

If your system do not support UEFI mode, you need to install grub2 in legacy BIOS mode. It will install grub on MBR of disk sdX

apt-get install grub-pc-bin grub-install --removable --target=i386-pc --debug --boot-directory=/boot /dev/sdX

Install and use Ubuntu from a Removable USB Flash Drive

Ubuntu is fantastic Linux distribution. It takes jut 600 MB RAM on fresh book. It is much faster than other propitiatory desktop operating systems.

Normally, a home compute have SATA hard disk ( 300 Mbps for SATA 2.0 and 600 MBps for SATA3.0. Now a days, USB 3.0 comes with 150 MBps and upto 200GB !

My Objective

1-Being Linux professional, use Ubuntu Linux for all daily work I do at home. So, install and run desktop OS from a USB Flash drive and Propitiatory OS only when firmware need to be upgraded ( most vendor do not release firmware for Linux !). I have achieved it !
2-Bring your operating system in where ever you go ! Use your USB to run your own Personal Desktop OS at school, office, Cafe, hotel etc.

Let me share steps.

What you need

1-A 2 GB USB drive to be used as Latest Ubuntu installer media

2-At least 8 GB Usb drive to install Ubuntu . I use ScanDisk Ultra Fit USB 3.0 Flash drive - 32 GB (SGD 16)

Prepare Install Media

1- Download latest Ubuntu iso : http://www.ubuntu.com/download

2- Download Rufus . Use it to burn Ubuntu iso on USB flash drive you are intended to use as installer : http://www.ubuntu.com/download/desktop/create-a-usb-stick-on-windows

If you are already on one of Ubuntu system, use Startup Disk Creator to burn iso on USB Srive

3-Boot Ubuntu from installer USB. You may need to change boot order in setup. Usually accessed by F2 key when pressed during POST. F12 - one time boot menu. Turn off Secure Boot if your BIOS has this option.

This installer USB is a FAT32 filesystem and it will not save any changes you will made on this as it similar to live CD. Though, there is work-around to save your changes (if you need) - search for casper-rw. Though, this is not our interest at this moment.

4- (skip this if you can boot with USB installer) If you can not boot from installer USB with all your efforts, possibly, your system does not support USB boot , does not support USB 3.0 or you need to upgrade bios etc. You may try re-burning iso once more. Obviously, if you can not boot installer from USB, you can not install and run Ubuntu from USB too. Stop here !

5- If above fails and you wish to install Ubuntu on free space of existing disk or on second disk :
a- Burn iso on a DVD ( you have bootable DVD, right !)
b- Download macrorit disk partition expert portable version
c- Reduce your disk ( BTW can be use dor USB too) partition to free up around 8 GB
d- Follow steps somewhat similar to below to install Ubuntu ( You do not need BIOS Boot and EFI System partition ESP)

Ubuntu Installation on USB Drive

1-Boot from USB installer media prepared in above step. Open an terminal and check if installer has booted in UEFI mode:

[ -d /sys/firmware/efi ] && echo "EFI boot on HDD" || echo "Legacy boot on HDD" ]

2- Select "Install Ubuntu". Refer : http://www.ubuntu.com/download/desktop/install-ubuntu-desktop . Select language, contact to wifi, Check both check boxes - to install update and third party update.

3- Plug in USB Flash drive on which you are going to install Ubuntu. Identify USB correctly and destroy partition table and create GPT disklabel. [ CAUTION: identify USB correctly, this operation is destructibve ]
         dd if=/dev/zero of=/dev/sdX count=100
   eject /dev/sdX
         ( re-insert USB to re-read partition
         parted /dev/sdX mklabel gpt

4-You must select "Something Else" in installation type. It will allow you to manual partition USB Drive

5- USB Partition : You must have minimum 3 partition. swap partition (point c) can be ignored, though it is recommended.
Note : If you do not see BIOS Boot and ESP partition option in below steps, installer has not booted in UEFI mode. So, you will not be installed Ubuntu in UEFI mode. But you can install it in Legacy mode. It mean, your flash drive will not appear as boot device on other system that support UEFI. Enable UEFI in system setup and again if you need to install it in UEFI mode.

a- A 10 MB partition "BIOS boot" partition - Unformulated, no filesystem on it, no mount point, Set bios_grub flag on it ( it is required by GPT partition type - ignoring details discussion here)

b- A 100 MB flag "EFI System Partition - ESP". DO NOT select mount poin /boot/efi and DO NOT select to format it as vfat32 ( ESP is a variant of vfat32 for not exact vfat32). Selecting ESP partition will automatically will set boot and esp flags on it. Leave mount point blank as installer know what to do with this. More detail is here.
c- 1 or 2 GB Swap partition of "Linux swap" type

d - All remaining space as - ext4 Linux FIlesystem, format it

Important Note #1 :You MUST select your USB drive in "Device for boot loader installation"

6- Select Download updates while installing and Install this third-party software now. It will install many 3rd party packages that Ubuntu repository do not have because of Licensing issues.

7- Go ahead and kick off installation and grab a coffee you deserve ! Once you will be back, Installation might be completed. Take out installation media, label it, keep it safe ( Trust me, you need it in future). Reboot system. If USB is your first boot device, your Ubuntu will be on your screen in 2 minutes. If required, bring one time boot menu, select USB flash drive, boot it.

8-Is your newly installed Ubuntu is portable and you can use it on any other systems ? Hint : If you see UEFI : USB 3.0 ScanDisk (32GB), [ see this image]. It depends how much attention you paid to Important Note #1. Try to boot some other system to boot from USB.

Important Note #2 :If you do not see a boot option similar to above, it indicate that boot loader was installed on some other existing ( first hard disk) disk that already had a UEFI ESP. Follow steps on my next blog.

Important Note #2 : There is fantastic tool boot-repair is you have messed up you windows boot manager or installer USB boot record.

Also see https://www.ventoy.net/en/index.html to boot multiple Linux from USB

Install and use Ubuntu from a Removable USB Flash Drive

My Objective

What you need

1-A 2 GB USB drive to be used as Latest Ubuntu installer media

2-At least 8 GB Usb drive to install Ubuntu . I use ScanDisk Ultra Fit USB 3.0 Flash drive - 32 GB (SGD 16)

Prepare Install Media

1- Download latest Ubuntu iso : http://www.ubuntu.com/download

2- Download Rufus . Use it to burn Ubuntu iso on USB flash drive you are intended to use as installer : http://www.ubuntu.com/download/desktop/create-a-usb-stick-on-windows

3-Boot Ubuntu from installer USB. You may need to change boot order in setup. Usually accessed by F2 key when pressed during POST. F12 - one time boot menu. Turn off Secure Boot if your BIOS has this option.

This installer USB is a FAT32 filesystem and it will not save any changes you will made on this as it similar to live CD. Though, there is work-around to save your changes (if you need) - search for casper-rw. Though, this is not our interest at this moment.

4- (skip this if you can boot with USB installer) If you can not boot from installer USB with all your efforts, possibly, your system does not support USB boot , does not support USB 3.0 or you need to upgrade bios etc. You may try re-burning iso once more. Obviously, if you can not boot installer from USB, you can not install and run Ubuntu from USB too. Stop here !

5- If above fails and you wish to install Ubuntu on free space of existing disk or on second disk :
a- Burn iso on a DVD ( you have bootable DVD, right !)
b- Download macrorit disk partition expert portable version
c- Reduce your disk ( BTW can be use dor USB too) partition to free up around 8 GB
d- Follow steps somewhat similar to below to install Ubuntu ( You do not need BIOS Boot and EFI System partition ESP)

Ubuntu Installation on USB Drive

1-Boot from USB installer media prepared in above step

2- Select "Install Ubuntu". Refer : http://www.ubuntu.com/download/desktop/install-ubuntu-desktop

3- Plug in USB Flash drive on which you are going to install Ubuntu

4-You must select "Something Else" in installation type. It will allow you to manual partition USB Drive

5- USB Partition : You must have minimum 3 partition. swap partition (point c) can be ignored, though it is recommended.

a- A 10 MB partition "BIOS boot" partition - Unformulated, no filesystem on it, no mount point, Set bios_grub flag on it ( it is required by GPT partition type - ignoring details discussion here)

b- A 200 MB fat32 partition, flag "EFI System Partition - ESP". DO NOT select mount poin /boot/efi . Leave mount point blank as installer know what to do with this. More detail is here.

c- 1 or 2 GB Swap partition of "Linux swap" type

d - All remaining space as - ext4 Linux FIlesystem, format it

Important Note #1 :You MUST select your usb drive in "Device for boot loader installation"

6- Select Download updates while installing and Install this third-party software now. It will install many 3rd party packages that Ubuntu repository do not have because of Licensing issues.

7- Go ahead and kick off installation and grab a coffee you deserve ! Once you will be back, Installation might be completed. Take out installation media, label it, keep it safe ( Trust me, you need it in future). Reboot system. If USB is your first boot device, your Ubuntu will be on your screen in 2 minutes. If required, bring one time boot menu, select USB flash drive, boot it.

8-Is your newly installed Ubuntu is portable and you can use it on any other systems ? Hint : If you see UEFI : USB 3.0 ScanDisk (32GB), [ see this image]. It depends how much attention you paid to Important Note #1. Try to boot some other system to boot from USB.

Important Note #2 :Is you do not see a boot option similar to above, it indicate that boot loader was installed on some other existing ( first hard disk) disk that already had a UEFI ESP. Follow steps on my next blog.

Friday, February 12, 2016

A note of kernel parameter min_free_kbytes

min_free_kbytes

- This controls the amount of memory that is kept free for use by special reserves including “atomic” allocations (those which cannot wait for reclaim)
- The minimum number of kilobytes to keep free across the system
- Default is min_free_kbytes = 4 * sqrt(lowmem_kbytes)
- kernel use all available RAM to cache but RAM equal to min_free_kbytes will not be used for cache
- Be cautious when setting this parameter, as both too-low and too-high values can be damaging. [1]
- setting this parameter to a value that is too high (5-10% of total system memory) will cause your system to become out-of-memory immediately. [3]
- Setting this too high will OOM your machine instantly.[2]
- lowmem_kbytes is complicated. Take it as size of RAM for approximation [4]

[1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/s-memory-tunables.html
[2] https://www.kernel.org/doc/Documentation/sysctl/vm.txt
[3] https://access.redhat.com/solutions/336033
[4] http://www.codeskulptor.org/#user41_u8JqnFK759_0.py

Thursday, January 14, 2016

Dealing with github error: The requested URL returned error: 403 while accessing

Above error means -I have tried to access a git project and I am not authorized .
I will show you one case to get this error and what is fix.

1-Create a github repository

-I have logded in to https://github.com/ using my github using github credential
-I have created a repository nagios_plugins (with default options)

2-Create local repository

$ mkdir nagios_plugins
$ cd nagios_plugins
$ echo "# nagios_plugins" >> README.md
$ git init
$ git add README.md
$ git commit -m "first commit"
$ git remote add origin https://github.com/nansari/nagios_plugins.git
$ git push -u origin master

--and I get following error !!

error: The requested URL returned error: 403 while accessing https://github.com/nansari/nagios_plugins.git/info/refs

fatal: HTTP request failed

3-How to fix

--Now, we know, a case where we see this error. Let us fix it. Since url is https - it need some kind of authentication - that we have not configured or provided so for.

--https access to github requires username and password. here is how we can configure git to use a username for each push and pull command.

--First, I checked what is remote configuration.

$ git remote -v

origin https://github.com/nansari/nagios_plugins.git (fetch)

origin https://github.com/nansari/nagios_plugins.git (push)

--Okay, I will modify origin to use my github username.

$ git remote set-url origin https://nansari@github.com/nansari/nagios_plugins.git

$ git remote -v

origin https://nansari@github.com/nansari/nagios_plugins.git (fetch)

origin https://nansari@github.com/nansari/nagios_plugins.git (push)
$ cat .git/config

--And, all done ! I am successfully able to push my local branch to remote.

$ git push -u origin master

Password: xxxxxx

Counting objects: 6, done.

Delta compression using up to 8 threads.

Compressing objects: 100% (3/3), done.

Writing objects: 100% (6/6), 511 bytes, done.

Total 6 (delta 0), reused 0 (delta 0)

To https://nansari@github.com/nansari/nagios_plugins.git

* [new branch] master -> master

Branch master set up to track remote branch master from origin by rebasing.

3-A step ahead - pushing local branch to remote

--If you are working in local branch other than master branch, you need to commit change on your local branch and then push your local branch to remote master

$ git push origin local_branch:master

Password: xxxxxx
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 320 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
To https://nansari@github.com/nansari/nagios_plugins.git 12b060a..fc3936b local_branch -> master

Saturday, January 9, 2016

SCSI driver Error Handaling (EH) - how multipath behave when few paths of a device fail

Question?

It is challenge even for most seasoned system admin to explain how the IO will behave when few paths of device fail but some paths are still active. What is your answer ….?

Most people answer this – IO will continue via remaining active paths.

And…….. that is wrong !!

Let me explain, it with extreme case – say one of remote storage port (rport) of storage frame failed. So if, there were 2 active paths of device, now device have only one good path and one failed path.

As soon as Linux kernel scsi (sc) driver will detect a failed path, it will quiesce HBA (yes, complete HBA) and wait for all outstanding IO to complete or timeout [1]. Then SCSI layer will activate error handaler. NO IO WILL BE SUBMITTED until error recovery completes – even though one path is failed and still one good paths available. This design is there to avoid any data corruption.

Device Recovery Steps

11- Abort the command after specified scsi device timeout value defined in /sys/block//device/timeout [1]

When the error handler is triggered, it attempts the following operations in order (until one successfully executes or all options exhausts):

22- Wait several seconds to hope that remote port become online (if device is Fiber Channel Device – not applicable for SCSI device)

33- Activate Error Handler and do following in sequence

a. Reset the device

b. Reset the bus

c. Reset the Host

First, scsi driver try to reset device and then bus. If it is not successful, and adapter firmware and device drive decide that adapter has not completed full recovery, adapter will be hard reset. It means, all paths of a disk via that adapter will be unavailable for few moment – irrespective, if they have failed are healthy. Hard reset happens when the I/O is black-holed with NOP response in the fabric. Since, IO had been frozen by scsi drive, there is no change of IO request drop or data corruption.

Case-1: If all of above will fail, device will be set to offline. It means, complete device is not available via any path. It need manual intervention to look at system/storage logs, find problem, fix it , scan HBA to detect active paths and make device online/running state.

Case-2: If recovery succeed, path check heuristics of multipath will mark dead paths as failed. Now, IO to device will continue via remaining active path.

Reduce device recovery time

To reduce overall recovery time, upgrade kernel (to version 2.6.18-371.6.1 or higher) and device_mapper (to version 0.4.7-63 or higher) to latest release to leverage on time related parameters such as :

11- scsi driver Error Handaling (EH) timeout – eh_timeout (from default 10 second to 5 seconds) [4]

22- HBA port reset time - eh_deadline ( from disable/0 to 5 seconds) [5]

33- Adpater reset time e.g. Qlogic reset time [2]

Add the following to /etc/modprobe.conf an recreate initrd

options qla2xxx ql2xextended_error_logging=1 qlport_down_retry=10 ql2xloginretrycount=10

44- Multipath check_timeout (reduce to 10 seconds from default 60 seconds) [3]

Reference

[1] Controlling the SCSI Command Timer and Device Status

[2] Qlogic Driver Parameter

[3] /usr/share/doc/device-mapper-multipath-0.4.7/multipath.conf.annotated

[4] http://comments.gmane.org/gmane.linux.scsi/81881

[5] Configurable Timeout for Unresponsive Devices ⁠Configurable Timeout for Unresponsive Devices and this

Additional Reference

1- http://events.linuxfoundation.org/sites/events/files/slides/SCSI-EH.pdf ( very good presentation)

2- http://www.faqs.org/docs/Linux-HOWTO/SCSI-Generic-HOWTO.html#AEN215

3- Redhat KB this and this

Tuesday, January 5, 2016

Kodi - Open Source Home Theatre Software - more I explore, more I get surprised

KODI ( Earlier name XBMC) is Open Source Home Theatre Software

- It runs on Linux, OS X, Windows, iOS, Android - I run OpenElect on Raspberrry Pi

- Have web based, and android / iOS software remote controls - Kore
- 'Play to Kodi' Chrome browser plugin can send you tube video link to Kodi to play it independently

- Third party configuration tools - one such excellent tool is http://www.tvaddons.ag/

- Have tons of plugin - video, audio, configuration, setting, programs etc.

- http://addons.tvaddons.ag/

- http://fusion.tvaddons.ag/ [1]
- Some of indian TV /Movie channels provided by 'aj add-ons (new look)' included in above are liste on below pages.
- http://www.desirulez.me/forums/41-Tv-Shows
- http://www.desitvbox.me/
- http://www.movierulz.to/

- Play entertain media from your Media Center, Desktop, gDrive etc.

- Record live show ( Beaware of legal implication of you storing / download media)

- Support all popular media format of Audio and Video

- UPnP - Universal Plug and Play [2]- Supported by all modern home appliance

[1] Enabling fusion and AJ India addon & Tutorials on installing fusion

[2] https://en.wikipedia.org/wiki/Universal_Plug_and_Play