Organizing My Thoughts: August 2012

Saturday, August 4, 2012

Zotero Data Server Installation

This tutorial contains installation instructions for setting up Zotero data server on the computer running Arch Linux.

Retrieving Sources

Change directory to "/srv/http/":

1	`$` `cd` `/srv/http/`

Dowload Zotero data server source code from github repository:

1	`$` `sudo` `git clone https://github.com/Panzerkampfwagen/dataserver.git`

Rename the directory (important!):

1	`$` `sudo` `mv` `dataserver ZoteroDataServer`

Download Zend framework:

1	`$` `sudo` `wget -vc` `"http://framework.zend.com/releases/ZendFramework-1.11.12/ZendFramework-1.11.12.tar.gz"`

Decompress part of the archive ("ZendFramework-1.11.12/library/Zend") to the "include" directory ("/srv/http/ZoteroDataServer/include/Zend"):

1	`$` `sudo` `tar` `-xvf ZendFramework-1.11.12.tar.gz --strip=3 -C` `"/srv/http/ZoteroDataServer/include/Zend"` `"ZendFramework-1.11.12/library/Zend"`

Configuring Apache HTTP Server and PHP Engine

Install Apache server:

1	`$` `sudo` `pacman -S apache php php-apache php-mcrypt`

Create a virtual host for Zotero data server. To do that, append the following configuration into "/etc/httpd/conf/extra/httpd-vhosts.conf". The directory for data server is "/srv/http/ZoteroDataServer/".

NameVirtualHost *:85
<VirtualHost *:85>
ServerName *:85
       DocumentRoot "/srv/http/ZoteroDataServer/htdocs"
       ErrorLog "/var/log/httpd/Zotero-Data-Server-Error.log"
       CustomLog "/var/log/httpd/Zotero-Data-Server-Access.log" common
       <Directory "/srv/http/ZoteroDataServer/htdocs">
              Options FollowSymLinks MultiViews
              AllowOverride All
              Order allow,deny
              Allow from all
       </Directory>
       AllowEncodedSlashes On
</VirtualHost>

Make sure that the following modules are loaded in "/etc/httpd/conf/httpd.conf" (uncomment these lines if necessary):

LoadModule rewrite_module modules/mod_rewrite.so
LoadModule php5_module modules/libphp5.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so

Check whether additional configuration files are included. If not, place this at the end of the "Include" list ( "/etc/httpd/conf/httpd.conf")

1 2	`Include conf/extra/php5_module.conf` `Include conf/extra/httpd-vhosts.conf`

Make sure that the following line is uncommented in "/etc/httpd/conf/httpd.conf" in the section/(after the line) <IfModule mime_module>

1	`TypesConfig conf/mime.types`

Make Apache server listen to port number 85 which points to Zotero virtual host by adding the following line to "/etc/httpd/conf/httpd.conf":

1	`Listen 85`

Start Apache server:

1	`$` `sudo` `/etc/rc.d/httpd` `start`

Add or uncomment the following lines in "/etc/php/php.ini":

extension=sockets.so
extension=mcrypt.so
extension=mysql.so
extension=mysqli.so

and check whether thsese libraries (shared objects) exist in "/usr/lib/php/modules/".

Configuring MySQL

Install MySQL:

1	`$` `sudo` `pacman -S mysql`

Zotero data server is configured to use "SecurePassword" as a root password. You have to reset root password to "SecurePassword" or modify source codes of the data server. To reset the root password, stop the MySQL daemon if it is running:

1	`$` `sudo` `/etc/rc.d/mysqld` `stop`

Restart MySQL daemon and bypass authentication:

1	`$` `sudo` `mysqld_safe --skip-grant-tables &`

Connect to the MySQL server:

1	`$` `sudo` `mysql -u root mysql`

Change root password:

mysql> UPDATE mysql.user SET Password=PASSWORD('SecurePassword') WHERE User='root';
mysql> FLUSH PRIVILEGES;
mysql> EXIT

Change MySQL time zone to UTC by modifying the following line in "/etc/mysql/my.cnf":

[mysqld]
...
default-time-zone = '+0:00'

Install PHPMyAdmin to control databases using WEB interface:

1	`$` `sudo` `pacman -S phpmyadmin`

Configuring Zotero Data Server

Main settings are stored in "/srv/http/ZoteroDataServer/include/config/config.inc.php" file:

...
public static $SYNC_DOMAIN = '127.0.0.1:85';
...
public static $CLI_PHP_PATH = '/usr/bin/php';
public static $CLI_DOCUMENT_ROOT = "/srv/http/ZoteroDataServer/";
...

Sync domain is configured to localhost so it won't accept connections from other IPs. But the goal is to make data server work at least on local host. If you want to change the data server root directory by editting $CLI_DOCUMENT_ROOT do not forget the trailing slash "/" at the end of the path. Interaction with MySQL database through PHP is done using credentials in "/srv/http/ZoteroDataServer/include/config/config.inc.php" file. You can change default root password ("SecurePassword") only by editting this file.

Setting Up Zotero Data Server

Start MySQL daemon first:

1	`$` `sudo` `/etc/rc.d/mysqld` `start`

Change directory to "/srv/http/ZoteroDataServer/misc"

1	`$` `cd` `/srv/http/ZoteroDataServer/misc`

Run "test_reset" script which deletes all existing Zotero data server databases (if they exist) and creates new databases from scratch.

$ ./test_reset
Deleting databases
Creating databases
Deleting users
Creating users
Updating user privileges
Filling databases with default fields
Reset is successfull. Now run ./test_setup

Now you can run "test_setup" script which adds some items to the "zoterotest1" database:

1	`$ ./test_setup`

If you see "Test setup is successfull." this means items were successfully added to the database. To check this, log into MySQL:

1	`$ mysql -u root -pSecurePassword`

Now check for the created databases:

$ mysql> SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| mysql              |
| zotero_cache       |
| zotero_www         |
| zotero_www_test    |
| zoterotest1        |
| zoterotest2        |
| zoterotest_ids     |
| zoterotest_master  |
+--------------------+
8 rows in set (0.01 sec)
 
mysql> SELECT * FROM items;
+--------+-----------+------------+---------------------+---------------------+----------+---------------------+---------+
| itemID | libraryID | itemTypeID | dateAdded           | dateModified        | key      | serverDateModified  | version |
+--------+-----------+------------+---------------------+---------------------+----------+---------------------+---------+
|      1 |         1 |          2 | 2012-10-09 20:26:11 | 2012-10-09 20:26:11 | AAAA2222 | 2012-10-09 20:26:11 |       0 |
|      2 |         2 |          2 | 2012-10-09 20:26:13 | 2012-10-09 20:26:13 | BBBB2222 | 2012-10-09 20:26:13 |       0 |
|      3 |         2 |         14 | 2012-10-09 20:26:15 | 2012-10-09 20:26:15 | CCCC4444 | 2012-10-09 20:26:15 |       0 |
|      4 |         3 |          2 | 2012-10-09 20:26:16 | 2012-10-09 20:26:16 | CCCC2222 | 2012-10-09 20:26:16 |       0 |
|      5 |         3 |         14 | 2012-10-09 20:26:16 | 2012-10-09 20:26:16 | CCCC3333 | 2012-10-09 20:26:16 |       0 |
+--------+-----------+------------+---------------------+---------------------+----------+---------------------+---------+
5 rows in set (0.00 sec)

Now it is possible to open three terminals and start upload, download and error daemons separately in foreground:
In the 1st terminal:

1 2	`$` `cd` `/srv/http/ZoteroDataServer/processor/upload/` `$ php daemon.php`

In the 2nd terminal:

1 2	`$` `cd` `/srv/http/ZoteroDataServer/processor/download/` `$ php daemon.php`

In the 3rd terminal:

1 2	`$` `cd` `/srv/http/ZoteroDataServer/processor/error/` `$ php daemon.php`

Testing Authentication on Zotero Data Server

To test authentication on the server, post the following request:

1	`$ curl -X POST -d` `"version=9&username=testuser&password=testuser"` `http://127.0.0.1:85/login`

It should return something like:

<?xml version="1.0"?>
<response version="9" timestamp="1341134959">
   <sessionID>
      da802280ce0bfc2e90cb1ad0747ff642
   </sessionID>
</response>

Now using provided "sessionID" you can post request to "updated" action:

1	`$ curl -X POST -d` `"version=9&sessionid=da802280ce0bfc2e90cb1ad0747ff642&lastsync=1"` `http://127.0.0.1:85/updated`

The answer will be similar to:

<?xml version="1.0"?>
<response version="9" 
      timestamp="1341135305" 
      userID="1"
      defaultLibraryID="1" 
      updateKey="43d4eaa497ab8cbfc8f4d201d955fd70"
      earliest="1341131740">
<updated/>
</response>

Installing Zotero Attachment Server

In order to be able to upload attachment files you need to provide your Zotero client with a WebDAV server. Distributed authoring and versioning (WebDAV). For example, you can use YandexDisk with WebDAV protcocol. It provides 5 GB disc space for free! Files are also accessible through its e-mail web-client. If you are paranoid enough not to trust anyone, you can set up a WebDAV server by yourself. Uncomment (or add) the following lines in "/etc/httpd/conf/httpd.conf"

LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule dav_module modules/mod_dav.so
LoadModule dav_fs_module modules/mod_dav_fs.so
Include conf/extra/httpd-dav.conf

Now create WebDAV lock directory and file:

$ sudo mkdir /srv/http/DAVLock
$ sudo chmod -R 777 /srv/http/DAVLock
$ sudo chown -R nobody:nobody /srv/http/DAVLock
$ sudo touch /srv/http/DAVLock/DAVLockDB

Create directory to store uploaded Zotero attachments:

$ sudo mkdir /srv/http/zotero
$ sudo chown -R http:http /srv/http/zotero
$ sudo chmod -R 777 /srv/http/zotero

Append or add the following configuration to "/etc/httpd/conf/extra/httpd-dav.conf"

DavLockDB "/srv/http/DAVLock/DAVLockDB"
<Directory "/srv/http/zotero">
     Dav On
     Order Allow,Deny
     Allow from all
     AllowOverride None
     AuthType Digest
     AuthName "WebDAV"
     AuthUserFile "/etc/httpd/conf/extra/AuthWebDAV.passwd"
     AuthDigestProvider file
     Require user "testuser"
</Directory>

Now create an MD5 hash for user "testuser" in realm "WebDAV" authorized by some password and store it in "/etc/httpd/conf/extra/AuthWebDAV.passwd" :

$ sudo htdigest -c /etc/httpd/conf/extra/AuthWebDAV.passwd WebDAV
testuser
Adding password for testuser in realm WebDAV.
New password:
Re-type new password:

Restart Apache server:

1	`$` `sudo` `/etc/rc.d/httpd` `restart`

In order to test your WebDAV server you have to install "cadaver" package:

1	`$` `sudo` `pacman -S cadaver`

Now connect to your WebDAV server using "cadaver" program:

$ cadaver http://127.0.0.1/zotero
Authentication required for WebDAV on server `127.0.0.1':
Username: testuser
Password:
dav:/zotero/> mkcol SomeCollection
Creating `SomeCollection': succeeded.
dav:/zotero/>

Installing Zotero Client

Right now only Zotero Firefox extension is available. Zotero standalone will be available soon.

Zotero Firefox Extension

Install Firefox extension from "/srv/http/ZoteroDataServer/zotero-3.0.8-patched.xpi". It is patched such that you can add custom data servers as shown in the figure:

Testing Zotero Data Server: Synchronization Process

Now open your Firefox and click on the "sync" button with an round green arrow on it:

$ cadaver http://127.0.0.1/zotero
Authentication required for WebDAV on server `127.0.0.1':
Username: testuser
Password:
dav:/zotero/> ls
Listing collection `/zotero/': succeeded.
3QBQSD38.prop    117 Aug 12 13:21
3QBQSD38.zip  465149 Aug 12 13:21
5BA5I3IP.prop    117 Aug 12 13:21
5BA5I3IP.zip     809 Aug 12 13:21
8IF963XB.prop    117 Aug 12 13:21
8IF963XB.zip  860702 Aug 12 13:21
lastsync           1 Aug 12 13:21
dav:/zotero/>

Debugging

You can see what is going by capturing the traffic between the client and the server or by enabling debugging in the client.

Capturing Traffic

You can capture the traffic using Wireshark. Select loopback interface (lo) to filter all other IP addresses except localhost (127.0.0.1). You should change HTTP port preferences in Wireshark to be able to analyze traffic. In Wireshark -> Preferences -> Protocols -> HTTP -> TCP Ports add port 85. In the capture filter field, enter "http". Turn on capture.

Enabling Debugging in the Client

Enable debugging by setting Zotero -> Preferences -> Advanced -> Debug Output Logging to "Enable" to see what is going on.

Friday, August 3, 2012

Adding SyntaxHighlighter Support to Blogger.com

To add Alex Gorbatchev's SyntaxHighlighter script just insert the following HTML code befre the </head> tag. If you are placing HTML code inside SyntaxHighlighter

<!--SYNTAX HIGHLIGHTER BEGINS-->
<!-- MAIN JS and CSS Files -->
<link href='https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/styles/shCore.css' rel='stylesheet' type='text/css'/>
<link href='https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/styles/shThemeDefault.css' rel='stylesheet' type='text/css'/>
<script language='javascript' src='https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shCore.js'/>
<!-- END OF MAIN JS and CSS Files -->
 
<!-- LANGUAGE FILES -->
<script language='javascript' src='https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shBrushPhp.js'/>
<script language='javascript' src='https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shBrushCSharp.js'/>
<script language='javascript' src='https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shBrushVb.js'/>
<script language='javascript' src='https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shBrushBash.js'/>
<script language='javascript' src='https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shBrushPlain.js'/>
<script language='javascript' src='https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/shBrushXml.js'/>
<!-- END OF LANGUAGE FILES -->
 
<!-- SETTINGS -->
<script src='http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js' type='text/javascript'/>
<script type='text/javascript'>
$(document).ready( function() {
SyntaxHighlighter.config.bloggerMode = true;
SyntaxHighlighter.config.clipboardSwf = &#39;https://cdnjs.cloudflare.com/ajax/libs/SyntaxHighlighter/3.0.83/scripts/clipboard.swf&#39;;
SyntaxHighlighter.defaults[&#39;wrap-lines&#39;] = false;
SyntaxHighlighter.config.strings.expandSource = &quot;Expand&quot;;
SyntaxHighlighter.config.strings.viewSource = &quot;Raw source&quot;;
SyntaxHighlighter.config.strings.copyToClipboard = &quot;Copy to clipboard&quot;;
SyntaxHighlighter.config.strings.print = &quot;Print&quot;;
SyntaxHighlighter.config.strings.help = &quot;Help&quot;;
SyntaxHighlighter.config.strings.alert = &quot;http://signalverarbeitung.blogspot.com\n&quot;;
SyntaxHighlighter.all();
});
</script>
<!-- ENF OF SETTINGS -->
<!--SYNTAX HIGHLIGHTER ENDS-->

Displaying listing is easy: just place your code as shown below.

<script type="syntaxhighlighter" class="brush: plain"><![CDATA[
 Some code here...
]]></script>

Be careful with HTML code. To display HTML code inside an HTML page regular characters should be translated into escape characters first. This can be done by using HTML-encoder, e.g. http://www.string-functions.com/htmlencode.aspx. Output code will be difficult to read: <br> will be encoded to <br> To display

<tr>
 <th>Symbol</th>
 <th>Code</th>
 <th>Entity Name</th>
</tr>

you should post

<script type="syntaxhighlighter" class="brush: html"><![CDATA[
 &lt;tr&gt;
  &lt;th&gt;Symbol&lt;/th&gt;
  &lt;th&gt;Code&lt;/th&gt;
  &lt;th&gt;Entity Name&lt;/th&gt;
 &lt;/tr&gt;
]]></script>