Retrieving Sources
Change directory to "/srv/http/":1 | $ cd /srv/http/ |
1 | $ sudo git clone https: //github .com /Panzerkampfwagen/dataserver .git |
1 | $ sudo mv dataserver ZoteroDataServer |
Decompress part of the archive ("ZendFramework-1.11.12/library/Zend") to the "include" directory ("/srv/http/ZoteroDataServer/include/Zend"):
1 | $ sudo tar -xvf ZendFramework-1.11.12. tar .gz --strip=3 -C "/srv/http/ZoteroDataServer/include/Zend" "ZendFramework-1.11.12/library/Zend" |
Configuring Apache HTTP Server and PHP Engine
Install Apache server:1 | $ sudo pacman -S apache php php-apache php-mcrypt |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | NameVirtualHost *:85 < VirtualHost *:85> ServerName *:85 DocumentRoot "/srv/http/ZoteroDataServer/htdocs" ErrorLog "/var/log/httpd/Zotero-Data-Server-Error.log" CustomLog "/var/log/httpd/Zotero-Data-Server-Access.log" common < Directory "/srv/http/ZoteroDataServer/htdocs"> Options FollowSymLinks MultiViews AllowOverride All Order allow,deny Allow from all </ Directory > AllowEncodedSlashes On </ VirtualHost > |
1 2 3 | LoadModule rewrite_module modules/mod_rewrite.so LoadModule php5_module modules/libphp5.so LoadModule vhost_alias_module modules/mod_vhost_alias.so |
1 2 | Include conf/extra/php5_module.conf Include conf/extra/httpd-vhosts.conf |
1 | TypesConfig conf/mime.types |
1 | Listen 85 |
1 | $ sudo /etc/rc .d /httpd start |
1 2 3 4 | extension=sockets.so extension=mcrypt.so extension=mysql.so extension=mysqli.so |
Configuring MySQL
Install MySQL:1 | $ sudo pacman -S mysql |
1 | $ sudo /etc/rc .d /mysqld stop |
1 | $ sudo mysqld_safe --skip-grant-tables & |
1 | $ sudo mysql -u root mysql |
1 2 3 | mysql> UPDATE mysql.user SET Password=PASSWORD( 'SecurePassword' ) WHERE User= 'root' ; mysql> FLUSH PRIVILEGES; mysql> EXIT |
1 2 3 | [mysqld] ... default- time -zone = '+0:00' |
1 | $ sudo pacman -S phpmyadmin |
Configuring Zotero Data Server
Main settings are stored in "/srv/http/ZoteroDataServer/include/config/config.inc.php" file:1 2 3 4 5 6 | ... public static $SYNC_DOMAIN = '127.0.0.1:85' ; ... public static $CLI_PHP_PATH = '/usr/bin/php' ; public static $CLI_DOCUMENT_ROOT = "/srv/http/ZoteroDataServer/" ; ... |
Setting Up Zotero Data Server
Start MySQL daemon first:1 | $ sudo /etc/rc .d /mysqld start |
1 | $ cd /srv/http/ZoteroDataServer/misc |
1 2 3 4 5 6 7 8 | $ . /test_reset Deleting databases Creating databases Deleting users Creating users Updating user privileges Filling databases with default fields Reset is successfull. Now run . /test_setup |
1 | $ . /test_setup |
1 | $ mysql -u root -pSecurePassword |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | $ mysql> SHOW DATABASES; +--------------------+ | Database | +--------------------+ | mysql | | zotero_cache | | zotero_www | | zotero_www_test | | zoterotest1 | | zoterotest2 | | zoterotest_ids | | zoterotest_master | +--------------------+ 8 rows in set (0.01 sec) mysql> SELECT * FROM items; +--------+-----------+------------+---------------------+---------------------+----------+---------------------+---------+ | itemID | libraryID | itemTypeID | dateAdded | dateModified | key | serverDateModified | version | +--------+-----------+------------+---------------------+---------------------+----------+---------------------+---------+ | 1 | 1 | 2 | 2012-10-09 20:26:11 | 2012-10-09 20:26:11 | AAAA2222 | 2012-10-09 20:26:11 | 0 | | 2 | 2 | 2 | 2012-10-09 20:26:13 | 2012-10-09 20:26:13 | BBBB2222 | 2012-10-09 20:26:13 | 0 | | 3 | 2 | 14 | 2012-10-09 20:26:15 | 2012-10-09 20:26:15 | CCCC4444 | 2012-10-09 20:26:15 | 0 | | 4 | 3 | 2 | 2012-10-09 20:26:16 | 2012-10-09 20:26:16 | CCCC2222 | 2012-10-09 20:26:16 | 0 | | 5 | 3 | 14 | 2012-10-09 20:26:16 | 2012-10-09 20:26:16 | CCCC3333 | 2012-10-09 20:26:16 | 0 | +--------+-----------+------------+---------------------+---------------------+----------+---------------------+---------+ 5 rows in set (0.00 sec) |
In the 1st terminal:
1 2 | $ cd /srv/http/ZoteroDataServer/processor/upload/ $ php daemon.php |
1 2 | $ cd /srv/http/ZoteroDataServer/processor/download/ $ php daemon.php |
1 2 | $ cd /srv/http/ZoteroDataServer/processor/error/ $ php daemon.php |
Testing Authentication on Zotero Data Server
To test authentication on the server, post the following request:1 | $ curl -X POST -d "version=9&username=testuser&password=testuser" http: //127 .0.0.1:85 /login |
1 2 3 4 5 6 | <? xml version="1.0"?> < response version="9" timestamp="1341134959"> < sessionID > da802280ce0bfc2e90cb1ad0747ff642 </ sessionID > </ response > |
1 | $ curl -X POST -d "version=9&sessionid=da802280ce0bfc2e90cb1ad0747ff642&lastsync=1" http: //127 .0.0.1:85 /updated |
1 2 3 4 5 6 7 8 9 | <? xml version="1.0"?> < response version="9" timestamp="1341135305" userID="1" defaultLibraryID="1" updateKey="43d4eaa497ab8cbfc8f4d201d955fd70" earliest="1341131740"> < updated /> </ response > |
Installing Zotero Attachment Server
In order to be able to upload attachment files you need to provide your Zotero client with a WebDAV server. Distributed authoring and versioning (WebDAV). For example, you can use YandexDisk with WebDAV protcocol. It provides 5 GB disc space for free! Files are also accessible through its e-mail web-client. If you are paranoid enough not to trust anyone, you can set up a WebDAV server by yourself. Uncomment (or add) the following lines in "/etc/httpd/conf/httpd.conf"1 2 3 4 | LoadModule auth_digest_module modules/mod_auth_digest.so LoadModule dav_module modules/mod_dav.so LoadModule dav_fs_module modules/mod_dav_fs.so Include conf/extra/httpd-dav.conf |
1 2 3 4 | $ sudo mkdir /srv/http/DAVLock $ sudo chmod -R 777 /srv/http/DAVLock $ sudo chown -R nobody:nobody /srv/http/DAVLock $ sudo touch /srv/http/DAVLock/DAVLockDB |
1 2 3 | $ sudo mkdir /srv/http/zotero $ sudo chown -R http:http /srv/http/zotero $ sudo chmod -R 777 /srv/http/zotero |
1 2 3 4 5 6 7 8 9 10 11 12 | DavLockDB "/srv/http/DAVLock/DAVLockDB" < Directory "/srv/http/zotero"> Dav On Order Allow,Deny Allow from all AllowOverride None AuthType Digest AuthName "WebDAV" AuthUserFile "/etc/httpd/conf/extra/AuthWebDAV.passwd" AuthDigestProvider file Require user "testuser" </ Directory > |
1 2 3 4 5 | $ sudo htdigest -c /etc/httpd/conf/extra/AuthWebDAV . passwd WebDAV testuser Adding password for testuser in realm WebDAV. New password: Re- type new password: |
1 | $ sudo /etc/rc .d /httpd restart |
1 | $ sudo pacman -S cadaver |
1 2 3 4 5 6 7 | $ cadaver http: //127 .0.0.1 /zotero Authentication required for WebDAV on server `127.0.0.1': Username: testuser Password: dav: /zotero/ > mkcol SomeCollection Creating `SomeCollection': succeeded. dav: /zotero/ > |
Installing Zotero Client
Right now only Zotero Firefox extension is available. Zotero standalone will be available soon.Zotero Firefox Extension
Install Firefox extension from "/srv/http/ZoteroDataServer/zotero-3.0.8-patched.xpi". It is patched such that you can add custom data servers as shown in the figure:Testing Zotero Data Server: Synchronization Process
Now open your Firefox and click on the "sync" button with an round green arrow on it:1 2 3 4 5 6 7 8 9 10 11 12 13 14 | $ cadaver http: //127 .0.0.1 /zotero Authentication required for WebDAV on server `127.0.0.1': Username: testuser Password: dav: /zotero/ > ls Listing collection ` /zotero/ ': succeeded. 3QBQSD38.prop 117 Aug 12 13:21 3QBQSD38.zip 465149 Aug 12 13:21 5BA5I3IP.prop 117 Aug 12 13:21 5BA5I3IP.zip 809 Aug 12 13:21 8IF963XB.prop 117 Aug 12 13:21 8IF963XB.zip 860702 Aug 12 13:21 lastsync 1 Aug 12 13:21 dav: /zotero/ > |
Hi, many thanks, very useful guide.
ReplyDeleteI am trying to install the dataserver on Archlinux as of 20th February 2013 and made some adaptations, which I wanted to share:
1) the sudo /etc/rc.d/daemon start are now replaced with sudo systemctl enable daemon.service (and sudo systemctl start)
2) I needed to install and activate memcached
3) I had to use sudo to run the test_reset and test_setup scripts
4) Before SELECT * FROM items; you need to enter USE zerotest1;
5) Before starting the php daemon.php commands, I needed to do a sudo chmod 777 /srv/http/ZoteroDataServer/tmp
6) When creating the MD5 hash for testuser, testuser should be at the end of the sudo htdigest command (line 1), not on line 2
7) At this stage, the curl commands work perfect, I can get the session ID and request updated action.
8) Unfortunately, from there I have a problem accessing dav:/zotero/ with cadaver:
$ cadaver http://127.0.0.1/zotero
Could not open collection:
403 Forbidden
I have not found a cure yet. Also, I will need to recompile the Firefox plugin for latest Firefox.
Hi. This tutorial is old, but not obsolete. Arch Linux is now using "systemd", Zotero data server code has also changed. Does it use memcached? I am not sure.
ReplyDeleteI should update the tutorial in the near future.
Now regarding your problems. WebDAV should work fine, just check your directory and file permissions:
$ sudo mkdir /srv/http/DAVLock
$ sudo chmod -R 777 /srv/http/DAVLock
$ sudo chown -R nobody:nobody /srv/http/DAVLock
$ sudo touch /srv/http/DAVLock/DAVLockDB
If you want to reassemble Zotero Firefox extension, here are some steps.
Assuming that you want to change my custom Zotero client ("zotero-3.0.8-patched.xpi"):
$ mkdir zotero-3.0.8-patched
$ unzip zotero-3.0.8-patched.xpi -d ./zotero-3.0.8-patched
$ cd zotero-3.0.8-patched/chrome # Notice only single file here: zotero.jar
$ mkdir tmp
$ unzip zotero.jar -d ./tmp
$ mv zotero.jar zotero.jar.original #Back up original file
$ cd ./tmp
Three new directories will appear. The source is in "content/zotero".
Now you can perform some changes in the source code.
You can search for "zotero.org" string using "find" and "grep" commands:
$ find ./content/zotero -type f -exec grep -l "zotero.org" {} \;
This will output all file containing this string inside. This is useful for changing default host name www.zotero.org to your custom server IP.
GUI components are in *.xul files (actually text files). You can look at my commits to see the changes in the GUI.
Assuming that you are in "zotero-3.0.8-patched/chrome/tmp" directory (referring to the root), to reassemble the client, run:
$ zip -0 -r ../zotero.jar *
$ cd ../../
$ zip -rq ../zotero-3.0.8-custom.xpi * -x ./chrome/tmp/\* ./chrome/zotero.jar.original
Wuala! You have your custom Zotero client.
Hi,
ReplyDeleteMany thanks for your advice.
memcached was necessary to avoid a warning message to run the test_reset and test_setup scripts. I do not think it was required though.
I have rebuilt the DAVLock directory and files and permissions, still with the same issue. I double-checked httpd.conf and httpd-dav.conf, and cannot find what is wrong with them. Weird. I am trying this in a Virtualbox Arch Linux install, but that should not be a problem (and the curl commands work perfect).
For the Zotero program, I shall try rebuild the standalone version, which will be easier to deploy for my users. But thanks for the procedure, this is very helpful.
Finally, I shall try on Debian 7 (Wheezy), the RC1 has just been released. I love Arch, but I think having the procedure in a stable distribution such as Debian may be useful in a work environement. I'll return to you with new developments, and we could update the Zotero wiki. As far as I understand, the Zotero team obfuscate a bit the dataserver install procedure. I understand they want to sell their cloud service, which is very legitimate, but sometimes this is not practical for companies, that may not be able to put data on external servers for legal reasons.
A question regarding WebDAV: should the http server listen both ports 80 and 85? When using cadaver http://127.0.0.1/zotero, that is the default 80 port, isn't it?
ReplyDeleteYes, it is. Plain HTTP with port number 80.
ReplyDeletehttps://wiki.archlinux.org/index.php/WebDAV
Well I have the same issue following the procedure on the Arch wiki you mentioned. Thus I have a config problem or something has changed in Arch.
ReplyDeleteStandalone Zotero client most probably will give a segmentation fault. It's better to use Firefox extension. Arch is very unstable nowadays, constant kernel panics, changes in mkinitcpio scripts.
ReplyDeleteAbout WebDAV problem. Do these permissions work?
$ sudo mkdir /srv/http/zotero
$ sudo chown -R http:http /srv/http/zotero
$ sudo chmod -R 777 /srv/http/zotero
As I understood, the authentication for WebDAV works fine, but the server cannot create/open a collection. Most likely the problem in your Apache configuration files.
About the Zotero data server itself. It was completely useless and buggy, however I managed to update MySQL table structure and it worked. Now I can upload the metadata to the data server and the attachments to the WebDAV server. In further sync sessions some problem occurs and I couldn't solve it. It won't synchronize. Debugging requires a lot of knowledge, JavaScript at the client side, PHP at the server side, the synchronization protocol itself. I am not good at all these stuff, so help is appreciated. Now they are planning to release Zotero 3.1 with new features in the beginning of April, this means that table structure of MySQL database will change.
Well, considering they commercialise the server offer, they have no interest in de-obfuscating their code. Anyway, have you tried to upgrade to the latest Zotero code? Or is it as obscure as before?
ReplyDeleteBesides, I have no issues with Arch at the moment but I guess that a stable Debian would be very suitable here.
Hi,
ReplyDeleteI have made a lot of progress, and have understood the WebDAV configuration. I have been able to adapt your tutorial and run it on Debian. I'll have to redo it properly and will be happy to share the procedure.
It is very similar to your procedure, except the paths of config files and the Apache user is www-data.
I still have a question: in the Zotero Sync Server section of the plugin, which user is that? It is not the testuser of the WebDAV I suppose?
Disregard my silly question above, this is the testuser/testuser the script ./test_setup creates.
ReplyDeleteAlmost there, everything looks fine, but when I click on the Sync button, it rotates forever. cadaver does not show any new collection in zotero (but mkcol works). I am wondering if the problem is with the plugin version?
ReplyDeleteThat exactly what I was talking about. It's a bug. And I don't know how to fix it. May be it is already fixed with the new versions of the server and client.
ReplyDeleteFirstly, enable debugging by setting "Zotero" -> "Preferences" -> "Advanced" -> "Debug Output Logging" to "Enable". Then set the debug level to maximum by checking "Zotero" -> "Preferences" -> "Advanced" -> "Miscellaneous" -> "Open about:config". Search for "extensions.zotero.debug.level". Set it to 5.
If you want to upload (which corresponds to synchronization at the first time) the metadata and attachments, uncheck "Sync automatically" option in Zotero and try to sync manually by pressing the "sync" button. If it doesn't work, there is another temporary solution.
Try to check the "Zotero -> Preferences -> Sync -> Reset -> Restore to Zotero Server" option in the client, then press the "Reset..." button. It should work.
Another advice is to capture the traffic in Wireshark on the loopback interface. Authentication, metadata upload and WebDAV packets will be visible there.
Hi,
ReplyDeleteYes I get this error in the logs:
(5)(+0002001): SELECT version FROM version WHERE schema='lastremotesync'
(5)(+0000000): SELECT version FROM version WHERE schema='lastlocalsync'
(3)(+0000000): HTTP POST version=9&sessionid=********&lastsync=1&upload=1 to http://127.0.0.1:85/updated
(3)(+0000099):
(3)(+0000000): Associated libraries are locked -- waiting 2000ms before next check
(4)(+0001238): Unregistering observer in notifier with hash 'Kt'
I don't know why the associated libraries are locked. I found this: http://forums.zotero.org/discussion/25370/no-longer-syncing-with-zotero-server-associated-files-are-locked/ reporting the same bug.
After I reset, I can sync an empty document base, but I get this error in my logs:
(3)(+0000000): HTTP OPTIONS for http://testuser:********@127.0.0.1/zotero/
(3)(+0000006): HTTP GET http://testuser:********@127.0.0.1/zotero/lastsync
(3)(+0000003):
404 Not Found
Not Found
The requested URL /zotero/lastsync was not found on this server.
Apache/2.2.22 (Debian) Server at 127.0.0.1 Port 80
(3)(+0000000): ===>404<===(number)
However, after creating a document, syncing is locked once again.
I am going to create the missing directories and will report.
Can you give me your mail address? I will write the answer to it.
ReplyDeleteThis comment has been removed by a blog administrator.
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteI keep getting the same loop erros with the upload processor saying:
ReplyDelete2013-04-02 18:01:55.9634 exception 'Exception' with message 'Mysqli statement execute error : Lock wait timeout exceeded; try restarting transaction
in /home/zotero/ZoteroDataServer/include/DB.inc.php:979
Stack trace:
#0 /home/zotero/ZoteroDataServer/include/DB.inc.php(401): Zotero_DB::error(Object(Zend_Db_Statement_Mysqli_Exception), 'INSERT INTO syn...', Array, 1)
#1 /home/zotero/ZoteroDataServer/model/DataObjects.inc.php(671): Zotero_DB::query('INSERT INTO syn...', Array, 1)
#2 /home/zotero/ZoteroDataServer/model/DataObjects.inc.php(709): Zotero_DataObjects::delete(1, '2NDIS8CP')
#3 /home/zotero/ZoteroDataServer/model/Sync.inc.php(1630): Zotero_DataObjects::deleteFromXML(Object(SimpleXMLElement))
#4 /home/zotero/ZoteroDataServer/model/Sync.inc.php(428): Zotero_Sync::processUploadInternal(1, Object(SimpleXMLElement), 1160342154, '2034610610')
#5 /home/zotero/ZoteroDataServer/model/Processor.inc.php(128): Zotero_Sync::processUploadFromQueue('2034610610')
#6 /home/zotero/ZoteroDataServer/model/Processor.inc.php(51): Zotero_Upload_Processor->processFromQueue()
#7 /home/zotero/ZoteroDataServer/processor/upload/processor.php(19): Zotero_Processor->run('2034610610')
#8 {main}
2013-04-02 18:01:55.9652 [2034610610] Exiting on lock error
2013-04-02 18:01:55.9658 LOCK received — waiting 2 seconds
2013-04-02 18:01:57.9662 Purged 0 lost processors
2013-04-02 18:01:57.9675 Purged 0 old processes
2013-04-02 18:01:57.9681 0 processors, 1 queued process
2013-04-02 18:01:57.9682 Starting 1 new processor
2013-04-02 18:01:58.0482 [982339177] Starting sync processor
have you come across this issue?
Hi. Regarding custom Zotero clients.
ReplyDeleteI patched version 3.0.14 of Zotero Firefox extension. It can be found here:
https://github.com/Panzerkampfwagen/dataserver/blob/master/misc/zotero-3.0.14-patched.xpi
I you want to reassemble your own client, take a look at my four commits (by Panzerkampfwagen) here:
https://github.com/Panzerkampfwagen/zotero/commits/master
Instructions for creating an "*.xpi" is somewhere above.
Installation documentation ("*.pdf" file) can be found in
https://github.com/Panzerkampfwagen/dataserver/tree/master/misc
Eric and I tested Zotero 3.0.14 with the latest data server (March 2013) and it worked fine. He had some similar issues with MySQL connector.
https://groups.google.com/forum/?fromgroups=#!topic/zotero-dev/7js-wbCUfME
It is suggested to increase:
"wait_timeout" and "max_allowed_packet" in "/etc/mysql/my.cnf"
http://www.socialengineforum.com/index.php?/topic/11257-zend-db-statement-mysqli-exception/
Hi Dave,
ReplyDeleteI have not got this error so far. The change in "wait_timeout" corrected the "MySQL has gone away" error, but for your error, it seems to be rather an adjustment innodb_lock_wait_timeout (http://dev.mysql.com/doc/refman/5.0/en/innodb-parameters.html#sysvar_innodb_lock_wait_timeout).
Now is your server very busy? What are your versions of the dataserver, of MySQL and of Zend framework in your dataserver?
Look at the install documentation, I have made a few changes in the repository and the procedure, which work fine. Today, I also uploaded the zotero-4.0.3 plugin for Firefox 20, as the version 3 is incompatible.
Hi Dave,
ReplyDeleteHave you been able to fix your problem? If I am not mistaken, errors of type "Mysqli statement execute error" are not directly returned by Zotero, but by the Zend framework code. These can be fixed by changing MySQL settings, such as the lock timeouts.
Hi there!
ReplyDeleteI managed to install the server and - as far as I know - it's running smoothly. However: The Firefox plugin (4.0.3) in the git doesn't seem to work anymore. Even without any login credentials it causes Firefox (v25.0.1) to create a hell of a load (debugging enabled). Neither sync is working...
Is there already a patched edition of a newer plugin? Or is the standanlone-edition finally patched?
Yours Jonas
Confirmed that. In Firefox v20 everything works. And an additional question: Is there any documentation on the structure of the database? I'd really like to add users and them to groups.
DeleteAs far as I understand, users can be added into the _www/users and groups to _master/groups. In _master/groupusers these two are brought together. The library-id in groups refers to _master/libraryies. Though that all seems to fit the group doesn't show up.
Any ideas?
Yours
Jonas
Sorry for the late answer. I am no longer maintaining the data server code in my repository because I don't use the server. The last time Eric added some modifications to the installation instructions in the
Delete"dataserver/misc/Zotero_Data_Server_Installation_Archlinux.pdf" file.
Regarding the database. I haven't seen any documentation on the data server or the database. You can inspect the database structure using PhpMyAdmin GUI during debugging.
About the Zotero Firefox client: you have to follow the guide for assembling Firefox extension posted above on February 21, 2013 at 2:17 AM by me. You can find the difference between files according to the commits in the
"https://github.com/Panzerkampfwagen/zotero/commits/master" repository by Panzerkampfwagen applied to "sync.js", "preferences.xul", "zotero.js" files.
Also some threads regarding the data server zotero-dev gmail group ("https://groups.google.com/d/forum/zotero-dev") may be helpful. Da gibt es meine Antworten. Offensichtlich der Spitzname der einen anderen Waffenname enthält gehört zu mir.
Hi, I have not worked on this for a long time, but managed to add the latest client (4.0.25.2) today. It works with the latest Firefox (v35).
DeleteOn the server side, I did not find the time to update to master (last sync was 20 June 2013). I tried on a local machine a more recent version, and from memory identified the switch from S3 to AWS being a problem, which needs some work. The switch was committed on the 11 March 2014. So the most recent version might be 9 March 2014. If someone has the time to look into it, that would be great indeed.
Hi, could Zotero data server be installed on shared hosting Cpanel without root access?
ReplyDelete