Organizing My Thoughts

Saturday, June 22, 2013

Making a Bootable USB Disk with an NTFS Partition and GRUB

Introduction

There may be many reasons to create a bootable USB flash drive with an installed bootloader. Not all live CD's of popular distros contain hybrid MBR/El-Torito bootable sectors. So if you are trying to write the ISO image directly into the flash drive using:

1	`$` `sudo` `dd` `if=Ubuntu-10.04-desktop-i386.iso of=/dev/sdb`

then it won't help. The flash drive won't boot, because BIOS sees it as a hard disk and requires bootstrap code in the Master Boot Record (MBR) of the drive. That's because the ISO image written to the drive most probably has so called "no-emulation" mode boot sector of the El Torito bootable CD specification somewhere at position 17 (sector size is 2048 bytes). On contrary, GRUB bootloader, when properly tuned, is capable of loading just few Linux distributions directly from the ISO image stored on the drive. It cannot boot all ISO files, but at least ArchLinux and Ubuntu seem to be bootable.
In addition, with the help of memdisk utility of the SYSLINUX bootloader it can load any ISO/floppy disk image of sufficiently small size directly to RAM and chainload the boot process. This method can be used to boot exotic operating systems such as Kolibri, MenuetOS, DixOS which are written in assembly language. Booting small Linux-based distros such as Tinfoil Hat Linux and Tiny Core Linux in the RAM disk is also possible. The only prerequisite is a sufficiently large RAM size of the computer.
Also it should be mentioned that GRUB provides a minimal shell environment, and acts like a small operating system. It is capable of reading data from many operating systems and even displaying graphic images of the JPEG and PNG format. Moreover, a flash drive formatted with NTFS file system can be used as a USB stick for storing and transferring data between computers (especially which are running Windows OS). This is actually the main function of the flash drive in the daily life.

Prerequisites

An IBM-compatible PC, the Arch Linux operating system, a flash drive recognized as /dev/sdb (do not accidentally overwrite your main hard disk), QEMU virual machine (optional).

Installing Necessary Packages

First of all back up your data on your flash drive. Install necessary packages.

1	`$` `sudo` `pacman -S parted syslinux grub-bios ntfs-3g qemu`

Partitioning the Drive

Now you can partition the drive despite all warnings and non-optimal behavior of small 512 byte long sectors. But who cares? Using NTFS/EXT3 or any other popular file system is also a bad idea. Those file system do not take into consideration wear leveling when a single sector is repeatedly overwritten. So ideally it would be nice to use JFFS or some other flash file system.
Here intention is different. First of all, a lot of people have ever wondered how do the 446 bytes of the bootstrap code found in the MBR can magically locate the partition and load some files from the file system of that partition? The whole file system input/output is needed! And this requires several kilo-bytes of memory. Usually bootloaders store this code in the MS-DOS compatibility region (62 sectors of size 512 bytes each located right after the MBR, i.e. of total size 31 KB). So what happens if we create a partition table without MS-DOS compatibility region? GRUB will be able to load its core image directly using only MBR's bootstrap code without any file system driver! This method is called "blocklists" method. Here is the proof. We will create a primary partition starting right after the MBR:

$ sudo parted /dev/sdb
(parted) rm 1 # remove partition 1 if such exists
(parted) mktable msdos # make MS-DOS partition table
(parted) help mkpart   
(parted) print devices
(parted) print free
(parted) mkpart primary ntfs 1s -1s # make a primary partition starting at sector 1 (right after the MBR) and ending at the very last sector
Warning: The resulting partition is not properly aligned for best performance.
Ignore/Cancel? Ignore # type "Ignore"
(parted) set 1 boot on # set bootable flag (is it necessary?)
(parted) quit

Formatting the Partition

Format the first partition with NTFS file system and label it "BootableUSB":

1	`$` `sudo` `mkfs.ntfs -v` `-Q -L` `"BootableUSB"` `/dev/sdb1`

Checking the MBR

Now check the MBR for the "55 AA" word at the end of the sector:

$ sudo dd if=/dev/sdb bs=512 count=1|hexdump -C
1+0 records in
1+0 records out
512 bytes (512 B) copied, 3.8609e-05 s, 13.3 MB/s
00000000  eb 63 90 10 8e d0 bc 00  b0 b8 00 00 8e d8 8e c0  |.c..............|
00000010  fb be 00 7c bf 00 06 b9  00 02 f3 a4 ea 21 06 00  |...|.........!..|
00000020  00 be be 07 38 04 75 0b  83 c6 10 81 fe fe 07 75  |....8.u........u|
00000030  f3 eb 16 b4 02 b0 01 bb  00 7c b2 80 8a 74 01 8b  |.........|...t..|
00000040  4c 02 cd 13 ea 00 7c 00  00 eb fe 00 00 00 00 00  |L.....|.........|
00000050  00 00 00 00 00 00 00 00  00 00 00 80 09 e5 53 00  |..............S.|
00000060  00 00 00 00 ff fa 90 90  f6 c2 80 74 05 f6 c2 70  |...........t...p|
00000070  74 02 b2 80 ea 79 7c 00  00 31 c0 8e d8 8e d0 bc  |t....y|..1......|
00000080  00 20 fb a0 64 7c 3c ff  74 02 88 c2 52 be 80 7d  |. ..d|<.t...R..}|
00000090  e8 17 01 be 05 7c b4 41  bb aa 55 cd 13 5a 52 72  |.....|.A..U..ZRr|
000000a0  3d 81 fb 55 aa 75 37 83  e1 01 74 32 31 c0 89 44  |=..U.u7...t21..D|
000000b0  04 40 88 44 ff 89 44 02  c7 04 10 00 66 8b 1e 5c  |.@.D..D.....f..\|
000000c0  7c 66 89 5c 08 66 8b 1e  60 7c 66 89 5c 0c c7 44  ||f.\.f..`|f.\..D|
000000d0  06 00 70 b4 42 cd 13 72  05 bb 00 70 eb 76 b4 08  |..p.B..r...p.v..|
000000e0  cd 13 73 0d f6 c2 80 0f  84 d8 00 be 8b 7d e9 82  |..s..........}..|
000000f0  00 66 0f b6 c6 88 64 ff  40 66 89 44 04 0f b6 d1  |.f....d.@f.D....|
00000100  c1 e2 02 88 e8 88 f4 40  89 44 08 0f b6 c2 c0 e8  |.......@.D......|
00000110  02 66 89 04 66 a1 60 7c  66 09 c0 75 4e 66 a1 5c  |.f..f.`|f..uNf.\|
00000120  7c 66 31 d2 66 f7 34 88  d1 31 d2 66 f7 74 04 3b  ||f1.f.4..1.f.t.;|
00000130  44 08 7d 37 fe c1 88 c5  30 c0 c1 e8 02 08 c1 88  |D.}7....0.......|
00000140  d0 5a 88 c6 bb 00 70 8e  c3 31 db b8 01 02 cd 13  |.Z....p..1......|
00000150  72 1e 8c c3 60 1e b9 00  01 8e db 31 f6 bf 00 80  |r...`......1....|
00000160  8e c6 fc f3 a5 1f 61 ff  26 5a 7c be 86 7d eb 03  |......a.&Z|..}..|
00000170  be 95 7d e8 34 00 be 9a  7d e8 2e 00 cd 18 eb fe  |..}.4...}.......|
00000180  47 52 55 42 20 00 47 65  6f 6d 00 48 61 72 64 20  |GRUB .Geom.Hard |
00000190  44 69 73 6b 00 52 65 61  64 00 20 45 72 72 6f 72  |Disk.Read. Error|
000001a0  0d 0a 00 bb 01 00 b4 0e  cd 10 ac 3c 00 75 f4 c3  |...........<.u..|
000001b0  00 00 00 00 00 00 00 00  0c 61 0e 00 00 00 00 00  |.........a......|
000001c0  02 00 07 de 77 e7 01 00  00 00 ff 97 77 00 00 00  |....w.......w...|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|

Checking the VBR

Similarly, check the Volume Boot Record (VBR) of the first partition for the "55 AA" word at the end of the sector. Note that VBR is located right after the MBR (option "skip=1" of the "dd" command):

$ dd if=/dev/sdb bs=512 count=1 skip=1|hexdump -C
1+0 records in
1+0 records out
512 bytes (512 B) copied, 3.7795e-05 s, 13.5 MB/s
00000000  eb 52 90 4e 54 46 53 20  20 20 20 00 02 08 00 00  |.R.NTFS    .....|
00000010  00 00 00 00 00 f8 00 00  3e 00 7c 00 01 00 00 00  |........>.|.....|
00000020  00 00 00 00 80 00 80 00  fe 97 77 00 00 00 00 00  |..........w.....|
00000030  04 00 00 00 00 00 00 00  7f 79 07 00 00 00 00 00  |.........y......|
00000040  f6 00 00 00 01 00 00 00  5a 50 80 73 94 60 16 3a  |........ZP.s.`.:|
00000050  00 00 00 00 fa 33 c0 8e  d0 bc 00 7c fb 68 c0 07  |.....3.....|.h..|
00000060  1f 1e 68 66 00 cb 88 16  0e 00 66 81 3e 03 00 4e  |..hf......f.>..N|
00000070  54 46 53 75 15 b4 41 bb  aa 55 cd 13 72 0c 81 fb  |TFSu..A..U..r...|
00000080  55 aa 75 06 f7 c1 01 00  75 03 e9 d2 00 1e 83 ec  |U.u.....u.......|
00000090  18 68 1a 00 b4 48 8a 16  0e 00 8b f4 16 1f cd 13  |.h...H..........|
000000a0  9f 83 c4 18 9e 58 1f 72  e1 3b 06 0b 00 75 db a3  |.....X.r.;...u..|
000000b0  0f 00 c1 2e 0f 00 04 1e  5a 33 db b9 00 20 2b c8  |........Z3... +.|
000000c0  66 ff 06 11 00 03 16 0f  00 8e c2 ff 06 16 00 e8  |f...............|
000000d0  40 00 2b c8 77 ef b8 00  bb cd 1a 66 23 c0 75 2d  |@.+.w......f#.u-|
000000e0  66 81 fb 54 43 50 41 75  24 81 f9 02 01 72 1e 16  |f..TCPAu$....r..|
000000f0  68 07 bb 16 68 70 0e 16  68 09 00 66 53 66 53 66  |h...hp..h..fSfSf|
00000100  55 16 16 16 68 b8 01 66  61 0e 07 cd 1a e9 6a 01  |U...h..fa.....j.|
00000110  90 90 66 60 1e 06 66 a1  11 00 66 03 06 1c 00 1e  |..f`..f...f.....|
00000120  66 68 00 00 00 00 66 50  06 53 68 01 00 68 10 00  |fh....fP.Sh..h..|
00000130  b4 42 8a 16 0e 00 16 1f  8b f4 cd 13 66 59 5b 5a  |.B..........fY[Z|
00000140  66 59 66 59 1f 0f 82 16  00 66 ff 06 11 00 03 16  |fYfY.....f......|
00000150  0f 00 8e c2 ff 0e 16 00  75 bc 07 1f 66 61 c3 a0  |........u...fa..|
00000160  f8 01 e8 08 00 a0 fb 01  e8 02 00 eb fe b4 01 8b  |................|
00000170  f0 ac 3c 00 74 09 b4 0e  bb 07 00 cd 10 eb f2 c3  |..<.t...........|
00000180  0d 0a 41 20 64 69 73 6b  20 72 65 61 64 20 65 72  |..A disk read er|
00000190  72 6f 72 20 6f 63 63 75  72 72 65 64 00 0d 0a 42  |ror occurred...B|
000001a0  4f 4f 54 4d 47 52 20 69  73 20 6d 69 73 73 69 6e  |OOTMGR is missin|
000001b0  67 00 0d 0a 42 4f 4f 54  4d 47 52 20 69 73 20 63  |g...BOOTMGR is c|
000001c0  6f 6d 70 72 65 73 73 65  64 00 0d 0a 50 72 65 73  |ompressed...Pres|
000001d0  73 20 43 74 72 6c 2b 41  6c 74 2b 44 65 6c 20 74  |s Ctrl+Alt+Del t|
000001e0  6f 20 72 65 73 74 61 72  74 0d 0a 00 00 00 00 00  |o restart.......|
000001f0  00 00 00 00 00 00 00 00  80 9d b2 ca 00 00 55 aa  |..............U.|

Note that

1	`$` `sudo` `dd` `if=/dev/sdb1` `bs=512 count=1\|hexdump -C`

should give the same result.

Installing GRUB into the MBR

Mount the first partition of the flash memory and install GRUB into the MBR using blocklists method:

$ sudo mkdir /mnt/USB
$ sudo mount /dev/sdb1 /mnt/USB
$ sudo grub-install --target=i386-pc --recheck --force --no-floppy --boot-directory=/mnt/USB /dev/sdb
 
/usr/bin/grub-bios-setup: warning: this msdos-style partition label has no post-MBR gap; embedding won't be possible.
/usr/bin/grub-bios-setup: warning: Embedding is not possible.  GRUB can only be installed in this setup by using blocklists.  However, blocklists are UNRELIABLE and their use is discouraged..
Installation finished. No error reported.

If you have a 64-bit system, change the target value "--target=x86_64-pc".

Configuring the GRUB

Now create GRUB configuration file:

1	`$` `sudo` `nano` `/mnt/USB/grub/grub.cfg`

and paste the following content:

insmod part_msdos
insmod ntfs
 
set timeout=10
set default=1
 
function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}
 
if loadfont unicode ; then
  set gfxmode="800x600x32"
  load_video
  insmod gfxterm
  terminal_input console
  terminal_output gfxterm
fi
 
insmod gfxmenu
#loadfont /grub/themes/starfield/dejavu_10.pf2
#loadfont /grub/themes/starfield/dejavu_12.pf2
#loadfont /grub/themes/starfield/dejavu_14.pf2
#loadfont /grub/themes/starfield/dejavu_16.pf2
#loadfont /grub/themes/starfield/dejavu_bold_14.pf2
 
#insmod jpeg
#background_image /grub/image.jpeg
#set theme=/grub/themes/starfield/theme.txt
#export theme
 
terminal_input at_keyboard
#keymap /grub/tr.gkb
insmod memdisk
 
set pager=1
 
menuentry "ArchLinux x86_64, June 2013" { 
  set isofile="/System/archlinux-2013.06.01-dual.iso"
  loopback loop $isofile
  linux (loop)/arch/boot/x86_64/vmlinuz isofrom=$isofile archisolabel="ARCH_201306" img_dev=/dev/disk/by-label/BootableUSB img_loop=$isofile
  initrd (loop)/arch/boot/x86_64/archiso.img
}
 
menuentry "ArchLinux i686, June 2013" { 
  set isofile="/System/archlinux-2013.06.01-dual.iso"
  loopback loop $isofile
  linux (loop)/arch/boot/i686/vmlinuz isofrom=$isofile archisolabel="ARCH_201306" img_dev=/dev/disk/by-label/BootableUSB img_loop=$isofile
  initrd (loop)/arch/boot/i686/archiso.img
}
 
menuentry "Ubuntu-13.04-desktop-amd64.iso" {
  set isofile="/System/ubuntu-13.04-desktop-amd64.iso"
  loopback loop $isofile
  linux (loop)/casper/vmlinuz.efi boot=casper iso-scan/filename=$isofile quiet noeject noprompt splash --
  initrd (loop)/casper/initrd.lz
}
 
menuentry "Tinfoil Hat Linux (using memdisk)" { 
  linux16 /grub/memdisk iso
  initrd16 "/System/tinfoil.img"
}
 
menuentry "Micro-Core Linux (using memdisk)" { 
  linux16 /grub/memdisk iso
  initrd16 "/System/microcore-current.iso"
}
 
menuentry "G4L Hard Disk and Partition Imaging and Cloning tool  (using memdisk)" { 
  linux16 /grub/memdisk iso
  initrd16 "/System/g4l-v0.43.iso"
}

Installing the Memdisk Module

Simply copy the memdisk file to your GRUB folder:

1	`$ sudo cp /usr/lib/syslinux/memdisk /mnt/USB/grub`

Downloading the ISO/Floppy Images

Now you can download the ISO/floppy images of the operating systems listed in "/mnt/USB/grub/grub.conf" (i.e. ArchLinux, Ubuntu, Tinfoil Hat Linux, Micro-Core Linux and G4L partitioning utility) and put them under "/mnt/USB/System/" folder.

1	`$ sudo mkdir /mnt/USB/System`

Testing the Bootable USB Disk

Now you can unmount the USB disk:

1	`$ sudo umount /mnt/USB`

and test it in the QEMU virtual machine:

1	`$` `sudo` `qemu-system-i386 -usb` `/dev/sdb`

Instead of "qemu-system-i386" you can use "qemu-system-x86_64".
Hopefully you will get the following screen:

Now unplug the flash drive and test it on the real hardware.

Saturday, August 4, 2012

Zotero Data Server Installation

This tutorial contains installation instructions for setting up Zotero data server on the computer running Arch Linux.

Retrieving Sources

Change directory to "/srv/http/":

1	`$` `cd` `/srv/http/`

Dowload Zotero data server source code from github repository:

1	`$` `sudo` `git clone https://github.com/Panzerkampfwagen/dataserver.git`

Rename the directory (important!):

1	`$` `sudo` `mv` `dataserver ZoteroDataServer`

Download Zend framework:

1	`$` `sudo` `wget -vc` `"http://framework.zend.com/releases/ZendFramework-1.11.12/ZendFramework-1.11.12.tar.gz"`

Decompress part of the archive ("ZendFramework-1.11.12/library/Zend") to the "include" directory ("/srv/http/ZoteroDataServer/include/Zend"):

1	`$` `sudo` `tar` `-xvf ZendFramework-1.11.12.tar.gz --strip=3 -C` `"/srv/http/ZoteroDataServer/include/Zend"` `"ZendFramework-1.11.12/library/Zend"`

Configuring Apache HTTP Server and PHP Engine

Install Apache server:

1	`$` `sudo` `pacman -S apache php php-apache php-mcrypt`

Create a virtual host for Zotero data server. To do that, append the following configuration into "/etc/httpd/conf/extra/httpd-vhosts.conf". The directory for data server is "/srv/http/ZoteroDataServer/".

NameVirtualHost *:85
<VirtualHost *:85>
ServerName *:85
       DocumentRoot "/srv/http/ZoteroDataServer/htdocs"
       ErrorLog "/var/log/httpd/Zotero-Data-Server-Error.log"
       CustomLog "/var/log/httpd/Zotero-Data-Server-Access.log" common
       <Directory "/srv/http/ZoteroDataServer/htdocs">
              Options FollowSymLinks MultiViews
              AllowOverride All
              Order allow,deny
              Allow from all
       </Directory>
       AllowEncodedSlashes On
</VirtualHost>

Make sure that the following modules are loaded in "/etc/httpd/conf/httpd.conf" (uncomment these lines if necessary):

LoadModule rewrite_module modules/mod_rewrite.so
LoadModule php5_module modules/libphp5.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so

Check whether additional configuration files are included. If not, place this at the end of the "Include" list ( "/etc/httpd/conf/httpd.conf")

1 2	`Include conf/extra/php5_module.conf` `Include conf/extra/httpd-vhosts.conf`

Make sure that the following line is uncommented in "/etc/httpd/conf/httpd.conf" in the section/(after the line) <IfModule mime_module>

1	`TypesConfig conf/mime.types`

Make Apache server listen to port number 85 which points to Zotero virtual host by adding the following line to "/etc/httpd/conf/httpd.conf":

1	`Listen 85`

Start Apache server:

1	`$` `sudo` `/etc/rc.d/httpd` `start`

Add or uncomment the following lines in "/etc/php/php.ini":

extension=sockets.so
extension=mcrypt.so
extension=mysql.so
extension=mysqli.so

and check whether thsese libraries (shared objects) exist in "/usr/lib/php/modules/".

Configuring MySQL

Install MySQL:

1	`$` `sudo` `pacman -S mysql`

Zotero data server is configured to use "SecurePassword" as a root password. You have to reset root password to "SecurePassword" or modify source codes of the data server. To reset the root password, stop the MySQL daemon if it is running:

1	`$` `sudo` `/etc/rc.d/mysqld` `stop`

Restart MySQL daemon and bypass authentication:

1	`$` `sudo` `mysqld_safe --skip-grant-tables &`

Connect to the MySQL server:

1	`$` `sudo` `mysql -u root mysql`

Change root password:

mysql> UPDATE mysql.user SET Password=PASSWORD('SecurePassword') WHERE User='root';
mysql> FLUSH PRIVILEGES;
mysql> EXIT

Change MySQL time zone to UTC by modifying the following line in "/etc/mysql/my.cnf":

[mysqld]
...
default-time-zone = '+0:00'

Install PHPMyAdmin to control databases using WEB interface:

1	`$` `sudo` `pacman -S phpmyadmin`

Configuring Zotero Data Server

Main settings are stored in "/srv/http/ZoteroDataServer/include/config/config.inc.php" file:

...
public static $SYNC_DOMAIN = '127.0.0.1:85';
...
public static $CLI_PHP_PATH = '/usr/bin/php';
public static $CLI_DOCUMENT_ROOT = "/srv/http/ZoteroDataServer/";
...

Sync domain is configured to localhost so it won't accept connections from other IPs. But the goal is to make data server work at least on local host. If you want to change the data server root directory by editting $CLI_DOCUMENT_ROOT do not forget the trailing slash "/" at the end of the path. Interaction with MySQL database through PHP is done using credentials in "/srv/http/ZoteroDataServer/include/config/config.inc.php" file. You can change default root password ("SecurePassword") only by editting this file.

Setting Up Zotero Data Server

Start MySQL daemon first:

1	`$` `sudo` `/etc/rc.d/mysqld` `start`

Change directory to "/srv/http/ZoteroDataServer/misc"

1	`$` `cd` `/srv/http/ZoteroDataServer/misc`

Run "test_reset" script which deletes all existing Zotero data server databases (if they exist) and creates new databases from scratch.

$ ./test_reset
Deleting databases
Creating databases
Deleting users
Creating users
Updating user privileges
Filling databases with default fields
Reset is successfull. Now run ./test_setup

Now you can run "test_setup" script which adds some items to the "zoterotest1" database:

1	`$ ./test_setup`

If you see "Test setup is successfull." this means items were successfully added to the database. To check this, log into MySQL:

1	`$ mysql -u root -pSecurePassword`

Now check for the created databases:

$ mysql> SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| mysql              |
| zotero_cache       |
| zotero_www         |
| zotero_www_test    |
| zoterotest1        |
| zoterotest2        |
| zoterotest_ids     |
| zoterotest_master  |
+--------------------+
8 rows in set (0.01 sec)
 
mysql> SELECT * FROM items;
+--------+-----------+------------+---------------------+---------------------+----------+---------------------+---------+
| itemID | libraryID | itemTypeID | dateAdded           | dateModified        | key      | serverDateModified  | version |
+--------+-----------+------------+---------------------+---------------------+----------+---------------------+---------+
|      1 |         1 |          2 | 2012-10-09 20:26:11 | 2012-10-09 20:26:11 | AAAA2222 | 2012-10-09 20:26:11 |       0 |
|      2 |         2 |          2 | 2012-10-09 20:26:13 | 2012-10-09 20:26:13 | BBBB2222 | 2012-10-09 20:26:13 |       0 |
|      3 |         2 |         14 | 2012-10-09 20:26:15 | 2012-10-09 20:26:15 | CCCC4444 | 2012-10-09 20:26:15 |       0 |
|      4 |         3 |          2 | 2012-10-09 20:26:16 | 2012-10-09 20:26:16 | CCCC2222 | 2012-10-09 20:26:16 |       0 |
|      5 |         3 |         14 | 2012-10-09 20:26:16 | 2012-10-09 20:26:16 | CCCC3333 | 2012-10-09 20:26:16 |       0 |
+--------+-----------+------------+---------------------+---------------------+----------+---------------------+---------+
5 rows in set (0.00 sec)

Now it is possible to open three terminals and start upload, download and error daemons separately in foreground:
In the 1st terminal:

1 2	`$` `cd` `/srv/http/ZoteroDataServer/processor/upload/` `$ php daemon.php`

In the 2nd terminal:

1 2	`$` `cd` `/srv/http/ZoteroDataServer/processor/download/` `$ php daemon.php`

In the 3rd terminal:

1 2	`$` `cd` `/srv/http/ZoteroDataServer/processor/error/` `$ php daemon.php`

Testing Authentication on Zotero Data Server

To test authentication on the server, post the following request:

1	`$ curl -X POST -d` `"version=9&username=testuser&password=testuser"` `http://127.0.0.1:85/login`

It should return something like:

<?xml version="1.0"?>
<response version="9" timestamp="1341134959">
   <sessionID>
      da802280ce0bfc2e90cb1ad0747ff642
   </sessionID>
</response>

Now using provided "sessionID" you can post request to "updated" action:

1	`$ curl -X POST -d` `"version=9&sessionid=da802280ce0bfc2e90cb1ad0747ff642&lastsync=1"` `http://127.0.0.1:85/updated`

The answer will be similar to:

<?xml version="1.0"?>
<response version="9" 
      timestamp="1341135305" 
      userID="1"
      defaultLibraryID="1" 
      updateKey="43d4eaa497ab8cbfc8f4d201d955fd70"
      earliest="1341131740">
<updated/>
</response>

Installing Zotero Attachment Server

In order to be able to upload attachment files you need to provide your Zotero client with a WebDAV server. Distributed authoring and versioning (WebDAV). For example, you can use YandexDisk with WebDAV protcocol. It provides 5 GB disc space for free! Files are also accessible through its e-mail web-client. If you are paranoid enough not to trust anyone, you can set up a WebDAV server by yourself. Uncomment (or add) the following lines in "/etc/httpd/conf/httpd.conf"

LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule dav_module modules/mod_dav.so
LoadModule dav_fs_module modules/mod_dav_fs.so
Include conf/extra/httpd-dav.conf

Now create WebDAV lock directory and file:

$ sudo mkdir /srv/http/DAVLock
$ sudo chmod -R 777 /srv/http/DAVLock
$ sudo chown -R nobody:nobody /srv/http/DAVLock
$ sudo touch /srv/http/DAVLock/DAVLockDB

Create directory to store uploaded Zotero attachments:

$ sudo mkdir /srv/http/zotero
$ sudo chown -R http:http /srv/http/zotero
$ sudo chmod -R 777 /srv/http/zotero

Append or add the following configuration to "/etc/httpd/conf/extra/httpd-dav.conf"

DavLockDB "/srv/http/DAVLock/DAVLockDB"
<Directory "/srv/http/zotero">
     Dav On
     Order Allow,Deny
     Allow from all
     AllowOverride None
     AuthType Digest
     AuthName "WebDAV"
     AuthUserFile "/etc/httpd/conf/extra/AuthWebDAV.passwd"
     AuthDigestProvider file
     Require user "testuser"
</Directory>

Now create an MD5 hash for user "testuser" in realm "WebDAV" authorized by some password and store it in "/etc/httpd/conf/extra/AuthWebDAV.passwd" :

$ sudo htdigest -c /etc/httpd/conf/extra/AuthWebDAV.passwd WebDAV
testuser
Adding password for testuser in realm WebDAV.
New password:
Re-type new password:

Restart Apache server:

1	`$` `sudo` `/etc/rc.d/httpd` `restart`

In order to test your WebDAV server you have to install "cadaver" package:

1	`$` `sudo` `pacman -S cadaver`

Now connect to your WebDAV server using "cadaver" program:

$ cadaver http://127.0.0.1/zotero
Authentication required for WebDAV on server `127.0.0.1':
Username: testuser
Password:
dav:/zotero/> mkcol SomeCollection
Creating `SomeCollection': succeeded.
dav:/zotero/>

Installing Zotero Client

Right now only Zotero Firefox extension is available. Zotero standalone will be available soon.

Zotero Firefox Extension

Install Firefox extension from "/srv/http/ZoteroDataServer/zotero-3.0.8-patched.xpi". It is patched such that you can add custom data servers as shown in the figure:

Testing Zotero Data Server: Synchronization Process

Now open your Firefox and click on the "sync" button with an round green arrow on it:

$ cadaver http://127.0.0.1/zotero
Authentication required for WebDAV on server `127.0.0.1':
Username: testuser
Password:
dav:/zotero/> ls
Listing collection `/zotero/': succeeded.
3QBQSD38.prop    117 Aug 12 13:21
3QBQSD38.zip  465149 Aug 12 13:21
5BA5I3IP.prop    117 Aug 12 13:21
5BA5I3IP.zip     809 Aug 12 13:21
8IF963XB.prop    117 Aug 12 13:21
8IF963XB.zip  860702 Aug 12 13:21
lastsync           1 Aug 12 13:21
dav:/zotero/>

Debugging

You can see what is going by capturing the traffic between the client and the server or by enabling debugging in the client.

Capturing Traffic

You can capture the traffic using Wireshark. Select loopback interface (lo) to filter all other IP addresses except localhost (127.0.0.1). You should change HTTP port preferences in Wireshark to be able to analyze traffic. In Wireshark -> Preferences -> Protocols -> HTTP -> TCP Ports add port 85. In the capture filter field, enter "http". Turn on capture.

Enabling Debugging in the Client

Enable debugging by setting Zotero -> Preferences -> Advanced -> Debug Output Logging to "Enable" to see what is going on.

Friday, August 3, 2012

Adding SyntaxHighlighter Support to Blogger.com

To add Alex Gorbatchev's SyntaxHighlighter script just insert the following HTML code befre the </head> tag. If you are placing HTML code inside SyntaxHighlighter

<!--SYNTAX HIGHLIGHTER BEGINS-->
<!-- MAIN JS and CSS Files -->
<link href='https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/styles/shCore.css' rel='stylesheet' type='text/css'/>
<link href='https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/styles/shThemeDefault.css' rel='stylesheet' type='text/css'/>
<script language='javascript' src='https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shCore.js'/>
<!-- END OF MAIN JS and CSS Files -->
 
<!-- LANGUAGE FILES -->
<script language='javascript' src='https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shBrushPhp.js'/>
<script language='javascript' src='https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shBrushCSharp.js'/>
<script language='javascript' src='https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shBrushVb.js'/>
<script language='javascript' src='https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shBrushBash.js'/>
<script language='javascript' src='https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shBrushPlain.js'/>
<script language='javascript' src='https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shBrushXml.js'/>
<!-- END OF LANGUAGE FILES -->
 
<!-- SETTINGS -->
<script src='http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js' type='text/javascript'/>
<script type='text/javascript'>
$(document).ready( function() {
SyntaxHighlighter.config.bloggerMode = true;
SyntaxHighlighter.config.clipboardSwf = &#39;https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/clipboard.swf&#39;;
SyntaxHighlighter.defaults[&#39;wrap-lines&#39;] = false;
SyntaxHighlighter.config.strings.expandSource = &quot;Expand&quot;;
SyntaxHighlighter.config.strings.viewSource = &quot;Raw source&quot;;
SyntaxHighlighter.config.strings.copyToClipboard = &quot;Copy to clipboard&quot;;
SyntaxHighlighter.config.strings.print = &quot;Print&quot;;
SyntaxHighlighter.config.strings.help = &quot;Help&quot;;
SyntaxHighlighter.config.strings.alert = &quot;http://signalverarbeitung.blogspot.com\n&quot;;
SyntaxHighlighter.all();
});
</script>
<!-- ENF OF SETTINGS -->
<!--SYNTAX HIGHLIGHTER ENDS-->

Displaying listing is easy: just place your code as shown below.

<script type="syntaxhighlighter" class="brush: plain"><![CDATA[
 Some code here...
]]></script>

Be careful with HTML code. To display HTML code inside an HTML page regular characters should be translated into escape characters first. This can be done by using HTML-encoder, e.g. http://www.string-functions.com/htmlencode.aspx. Output code will be difficult to read: <br> will be encoded to <br> To display

<tr>
 <th>Symbol</th>
 <th>Code</th>
 <th>Entity Name</th>
</tr>

you should post

<script type="syntaxhighlighter" class="brush: html"><![CDATA[
 &lt;tr&gt;
  &lt;th&gt;Symbol&lt;/th&gt;
  &lt;th&gt;Code&lt;/th&gt;
  &lt;th&gt;Entity Name&lt;/th&gt;
 &lt;/tr&gt;
]]></script>