Saturday, June 22, 2013

Making a Bootable USB Disk with an NTFS Partition and GRUB

Introduction

There may be many reasons to create a bootable USB flash drive with an installed bootloader. Not all live CD's of popular distros contain hybrid MBR/El-Torito bootable sectors. So if you are trying to write the ISO image directly into the flash drive using: then it won't help. The flash drive won't boot, because BIOS sees it as a hard disk and requires bootstrap code in the Master Boot Record (MBR) of the drive. That's because the ISO image written to the drive most probably has so called "no-emulation" mode boot sector of the El Torito bootable CD specification somewhere at position 17 (sector size is 2048 bytes). On contrary, GRUB bootloader, when properly tuned, is capable of loading just few Linux distributions directly from the ISO image stored on the drive. It cannot boot all ISO files, but at least ArchLinux and Ubuntu seem to be bootable.
In addition, with the help of memdisk utility of the SYSLINUX bootloader it can load any ISO/floppy disk image of sufficiently small size directly to RAM and chainload the boot process. This method can be used to boot exotic operating systems such as Kolibri, MenuetOS, DixOS which are written in assembly language. Booting small Linux-based distros such as Tinfoil Hat Linux and Tiny Core Linux in the RAM disk is also possible. The only prerequisite is a sufficiently large RAM size of the computer.
Also it should be mentioned that GRUB provides a minimal shell environment, and acts like a small operating system. It is capable of reading data from many operating systems and even displaying graphic images of the JPEG and PNG format. Moreover, a flash drive formatted with NTFS file system can be used as a USB stick for storing and transferring data between computers (especially which are running Windows OS). This is actually the main function of the flash drive in the daily life.

Prerequisites

An IBM-compatible PC, the Arch Linux operating system, a flash drive recognized as /dev/sdb (do not accidentally overwrite your main hard disk), QEMU virual machine (optional).

Installing Necessary Packages

First of all back up your data on your flash drive. Install necessary packages.

Partitioning the Drive

Now you can partition the drive despite all warnings and non-optimal behavior of small 512 byte long sectors. But who cares? Using NTFS/EXT3 or any other popular file system is also a bad idea. Those file system do not take into consideration wear leveling when a single sector is repeatedly overwritten. So ideally it would be nice to use JFFS or some other flash file system.
Here intention is different. First of all, a lot of people have ever wondered how do the 446 bytes of the bootstrap code found in the MBR can magically locate the partition and load some files from the file system of that partition? The whole file system input/output is needed! And this requires several kilo-bytes of memory. Usually bootloaders store this code in the MS-DOS compatibility region (62 sectors of size 512 bytes each located right after the MBR, i.e. of total size 31 KB). So what happens if we create a partition table without MS-DOS compatibility region? GRUB will be able to load its core image directly using only MBR's bootstrap code without any file system driver! This method is called "blocklists" method. Here is the proof. We will create a primary partition starting right after the MBR:

Formatting the Partition

Format the first partition with NTFS file system and label it "BootableUSB":

Checking the MBR

Now check the MBR for the "55 AA" word at the end of the sector:

Checking the VBR

Similarly, check the Volume Boot Record (VBR) of the first partition for the "55 AA" word at the end of the sector. Note that VBR is located right after the MBR (option "skip=1" of the "dd" command): Note that should give the same result.

Installing GRUB into the MBR

Mount the first partition of the flash memory and install GRUB into the MBR using blocklists method: If you have a 64-bit system, change the target value "--target=x86_64-pc".

Configuring the GRUB

Now create GRUB configuration file: and paste the following content:

Installing the Memdisk Module

Simply copy the memdisk file to your GRUB folder:

Downloading the ISO/Floppy Images

Now you can download the ISO/floppy images of the operating systems listed in "/mnt/USB/grub/grub.conf" (i.e. ArchLinux, Ubuntu, Tinfoil Hat Linux, Micro-Core Linux and G4L partitioning utility) and put them under "/mnt/USB/System/" folder.

Testing the Bootable USB Disk

Now you can unmount the USB disk: and test it in the QEMU virtual machine: Instead of "qemu-system-i386" you can use "qemu-system-x86_64".
Hopefully you will get the following screen:



Now unplug the flash drive and test it on the real hardware.

Saturday, August 4, 2012

Zotero Data Server Installation

This tutorial contains installation instructions for setting up Zotero data server on the computer running Arch Linux.

Retrieving Sources

Change directory to "/srv/http/": Dowload Zotero data server source code from github repository: Rename the directory (important!): Download Zend framework: Decompress part of the archive ("ZendFramework-1.11.12/library/Zend") to the "include" directory ("/srv/http/ZoteroDataServer/include/Zend"):

Configuring Apache HTTP Server and PHP Engine

Install Apache server: Create a virtual host for Zotero data server. To do that, append the following configuration into "/etc/httpd/conf/extra/httpd-vhosts.conf". The directory for data server is "/srv/http/ZoteroDataServer/". Make sure that the following modules are loaded in "/etc/httpd/conf/httpd.conf" (uncomment these lines if necessary): Check whether additional configuration files are included. If not, place this at the end of the "Include" list ( "/etc/httpd/conf/httpd.conf") Make sure that the following line is uncommented in "/etc/httpd/conf/httpd.conf" in the section/(after the line) <IfModule mime_module> Make Apache server listen to port number 85 which points to Zotero virtual host by adding the following line to "/etc/httpd/conf/httpd.conf": Start Apache server: Add or uncomment the following lines in "/etc/php/php.ini": and check whether thsese libraries (shared objects) exist in "/usr/lib/php/modules/".

Configuring MySQL

Install MySQL: Zotero data server is configured to use "SecurePassword" as a root password. You have to reset root password to "SecurePassword" or modify source codes of the data server. To reset the root password, stop the MySQL daemon if it is running: Restart MySQL daemon and bypass authentication: Connect to the MySQL server: Change root password: Change MySQL time zone to UTC by modifying the following line in "/etc/mysql/my.cnf": Install PHPMyAdmin to control databases using WEB interface:

Configuring Zotero Data Server

Main settings are stored in "/srv/http/ZoteroDataServer/include/config/config.inc.php" file: Sync domain is configured to localhost so it won't accept connections from other IPs. But the goal is to make data server work at least on local host. If you want to change the data server root directory by editting $CLI_DOCUMENT_ROOT do not forget the trailing slash "/" at the end of the path. Interaction with MySQL database through PHP is done using credentials in "/srv/http/ZoteroDataServer/include/config/config.inc.php" file. You can change default root password ("SecurePassword") only by editting this file.

Setting Up Zotero Data Server

Start MySQL daemon first: Change directory to "/srv/http/ZoteroDataServer/misc" Run "test_reset" script which deletes all existing Zotero data server databases (if they exist) and creates new databases from scratch. Now you can run "test_setup" script which adds some items to the "zoterotest1" database: If you see "Test setup is successfull." this means items were successfully added to the database. To check this, log into MySQL: Now check for the created databases: Now it is possible to open three terminals and start upload, download and error daemons separately in foreground:
In the 1st terminal: In the 2nd terminal: In the 3rd terminal:

Testing Authentication on Zotero Data Server

To test authentication on the server, post the following request: It should return something like: Now using provided "sessionID" you can post request to "updated" action: The answer will be similar to:

Installing Zotero Attachment Server

In order to be able to upload attachment files you need to provide your Zotero client with a WebDAV server. Distributed authoring and versioning (WebDAV). For example, you can use YandexDisk with WebDAV protcocol. It provides 5 GB disc space for free! Files are also accessible through its e-mail web-client. If you are paranoid enough not to trust anyone, you can set up a WebDAV server by yourself. Uncomment (or add) the following lines in "/etc/httpd/conf/httpd.conf" Now create WebDAV lock directory and file: Create directory to store uploaded Zotero attachments: Append or add the following configuration to "/etc/httpd/conf/extra/httpd-dav.conf" Now create an MD5 hash for user "testuser" in realm "WebDAV" authorized by some password and store it in "/etc/httpd/conf/extra/AuthWebDAV.passwd" : Restart Apache server: In order to test your WebDAV server you have to install "cadaver" package: Now connect to your WebDAV server using "cadaver" program:

Installing Zotero Client

Right now only Zotero Firefox extension is available. Zotero standalone will be available soon.

Zotero Firefox Extension

Install Firefox extension from "/srv/http/ZoteroDataServer/zotero-3.0.8-patched.xpi". It is patched such that you can add custom data servers as shown in the figure:

Testing Zotero Data Server: Synchronization Process

Now open your Firefox and click on the "sync" button with an round green arrow on it:

Debugging

You can see what is going by capturing the traffic between the client and the server or by enabling debugging in the client.

Capturing Traffic

You can capture the traffic using Wireshark. Select loopback interface (lo) to filter all other IP addresses except localhost (127.0.0.1). You should change HTTP port preferences in Wireshark to be able to analyze traffic. In Wireshark -> Preferences -> Protocols -> HTTP -> TCP Ports add port 85. In the capture filter field, enter "http". Turn on capture.

Enabling Debugging in the Client 

Enable debugging by setting Zotero -> Preferences -> Advanced -> Debug Output Logging to "Enable" to see what is going on.

Friday, August 3, 2012

Adding SyntaxHighlighter Support to Blogger.com

To add Alex Gorbatchev's SyntaxHighlighter script just insert the following HTML code befre the </head> tag. If you are placing HTML code inside SyntaxHighlighter Displaying listing is easy: just place your code as shown below. Be careful with HTML code. To display HTML code inside an HTML page regular characters should be translated into escape characters first. This can be done by using HTML-encoder, e.g. http://www.string-functions.com/htmlencode.aspx. Output code will be difficult to read: <br> will be encoded to &lt;br&gt; To display you should post