This tutorial contains installation instructions for setting up Zotero data server on the computer running Arch Linux.
Retrieving Sources
Change directory to
"/srv/http/":
Dowload Zotero data server source code from github repository:
Rename the directory (important!):
Download Zend framework:
Decompress part of the archive (
"ZendFramework-1.11.12/library/Zend") to the "include" directory (
"/srv/http/ZoteroDataServer/include/Zend"):
Configuring Apache HTTP Server and PHP Engine
Install Apache server:
Create a virtual host for Zotero data server. To do that, append the following configuration into
"/etc/httpd/conf/extra/httpd-vhosts.conf".
The directory for data server is
"/srv/http/ZoteroDataServer/".
Make sure that the following modules are loaded in
"/etc/httpd/conf/httpd.conf"
(uncomment these lines if necessary):
Check whether additional configuration files are included. If not, place this at the end of the
"Include" list (
"/etc/httpd/conf/httpd.conf")
Make sure that the following line is uncommented in
"/etc/httpd/conf/httpd.conf" in the
section/(after the line)
<IfModule mime_module>
Make Apache server listen to port number 85 which points to Zotero virtual host by adding the following line to
"/etc/httpd/conf/httpd.conf":
Start Apache server:
Add or uncomment the following lines in
"/etc/php/php.ini":
and check whether thsese libraries (shared objects) exist in
"/usr/lib/php/modules/".
Configuring MySQL
Install MySQL:
Zotero data server is configured to use "SecurePassword" as a root password. You have to reset root password to "SecurePassword" or modify source codes of the data server. To reset the root password, stop the MySQL daemon if it is running:
Restart MySQL daemon and bypass authentication:
Connect to the MySQL server:
Change root password:
Change MySQL time zone to UTC by modifying the following line in
"/etc/mysql/my.cnf":
Install PHPMyAdmin to control databases using WEB interface:
Configuring Zotero Data Server
Main settings are stored in
"/srv/http/ZoteroDataServer/include/config/config.inc.php" file:
Sync domain is configured to localhost so it won't accept connections from other IPs. But the goal is to make data server work at least on local host.
If you want to change the data server root directory by editting
$CLI_DOCUMENT_ROOT do not forget the trailing slash "/" at the end of the path.
Interaction with MySQL database through PHP is done using credentials in
"/srv/http/ZoteroDataServer/include/config/config.inc.php" file. You can change default root password ("SecurePassword") only by editting this file.
Setting Up Zotero Data Server
Start MySQL daemon first:
Change directory to
"/srv/http/ZoteroDataServer/misc"
Run
"test_reset" script which deletes all existing Zotero data server databases (if they exist)
and creates new databases from scratch.
Now you can run
"test_setup" script which adds some items to the
"zoterotest1" database:
If you see
"Test setup is successfull." this means items were successfully added to the database. To check this, log into MySQL:
Now check for the created databases:
Now it is possible to open three terminals and start upload, download and error daemons separately in foreground:
In the 1st terminal:
In the 2nd terminal:
In the 3rd terminal:
Testing Authentication on Zotero Data Server
To test authentication on the server, post the following request:
It should return something like:
Now using provided "sessionID" you can post request to "updated" action:
The answer will be similar to:
Installing Zotero Attachment Server
In order to be able to upload attachment files you need to provide your Zotero client with a WebDAV server. Distributed authoring and versioning (WebDAV). For example, you can use YandexDisk with WebDAV protcocol. It provides 5 GB disc space for free! Files are also accessible through its e-mail web-client. If you are paranoid enough not to trust anyone, you can set up a WebDAV server by yourself.
Uncomment (or add) the following lines in
"/etc/httpd/conf/httpd.conf"
Now create WebDAV lock directory and file:
Create directory to store uploaded Zotero attachments:
Append or add the following configuration to
"/etc/httpd/conf/extra/httpd-dav.conf"
Now create an MD5 hash for user "testuser" in realm "WebDAV" authorized by some password and store it in
"/etc/httpd/conf/extra/AuthWebDAV.passwd" :
Restart Apache server:
In order to test your WebDAV server you have to install "cadaver" package:
Now connect to your WebDAV server using "cadaver" program:
Installing Zotero Client
Right now only Zotero Firefox extension is available. Zotero standalone will be available soon.
Zotero Firefox Extension
Install Firefox extension from
"/srv/http/ZoteroDataServer/zotero-3.0.8-patched.xpi". It is patched such that you can add custom data servers as shown in the figure:
Testing Zotero Data Server: Synchronization Process
Now open your Firefox and click on the "sync" button with an round green arrow on it:
Debugging
You can see what is going by capturing the traffic between the client and the server or by enabling debugging in the client.
Capturing Traffic
You can capture the traffic using Wireshark. Select loopback interface (lo) to filter all other IP addresses except localhost (127.0.0.1). You should change HTTP port preferences in Wireshark to be able to analyze traffic. In
Wireshark ->
Preferences ->
Protocols ->
HTTP ->
TCP Ports add port 85. In the capture filter field, enter "
http". Turn on capture.
Enabling Debugging in the Client
Enable debugging by setting
Zotero ->
Preferences ->
Advanced ->
Debug Output Logging to "Enable" to see what is going on.
Hi, many thanks, very useful guide.
ReplyDeleteI am trying to install the dataserver on Archlinux as of 20th February 2013 and made some adaptations, which I wanted to share:
1) the sudo /etc/rc.d/daemon start are now replaced with sudo systemctl enable daemon.service (and sudo systemctl start)
2) I needed to install and activate memcached
3) I had to use sudo to run the test_reset and test_setup scripts
4) Before SELECT * FROM items; you need to enter USE zerotest1;
5) Before starting the php daemon.php commands, I needed to do a sudo chmod 777 /srv/http/ZoteroDataServer/tmp
6) When creating the MD5 hash for testuser, testuser should be at the end of the sudo htdigest command (line 1), not on line 2
7) At this stage, the curl commands work perfect, I can get the session ID and request updated action.
8) Unfortunately, from there I have a problem accessing dav:/zotero/ with cadaver:
$ cadaver http://127.0.0.1/zotero
Could not open collection:
403 Forbidden
I have not found a cure yet. Also, I will need to recompile the Firefox plugin for latest Firefox.
Hi. This tutorial is old, but not obsolete. Arch Linux is now using "systemd", Zotero data server code has also changed. Does it use memcached? I am not sure.
ReplyDeleteI should update the tutorial in the near future.
Now regarding your problems. WebDAV should work fine, just check your directory and file permissions:
$ sudo mkdir /srv/http/DAVLock
$ sudo chmod -R 777 /srv/http/DAVLock
$ sudo chown -R nobody:nobody /srv/http/DAVLock
$ sudo touch /srv/http/DAVLock/DAVLockDB
If you want to reassemble Zotero Firefox extension, here are some steps.
Assuming that you want to change my custom Zotero client ("zotero-3.0.8-patched.xpi"):
$ mkdir zotero-3.0.8-patched
$ unzip zotero-3.0.8-patched.xpi -d ./zotero-3.0.8-patched
$ cd zotero-3.0.8-patched/chrome # Notice only single file here: zotero.jar
$ mkdir tmp
$ unzip zotero.jar -d ./tmp
$ mv zotero.jar zotero.jar.original #Back up original file
$ cd ./tmp
Three new directories will appear. The source is in "content/zotero".
Now you can perform some changes in the source code.
You can search for "zotero.org" string using "find" and "grep" commands:
$ find ./content/zotero -type f -exec grep -l "zotero.org" {} \;
This will output all file containing this string inside. This is useful for changing default host name www.zotero.org to your custom server IP.
GUI components are in *.xul files (actually text files). You can look at my commits to see the changes in the GUI.
Assuming that you are in "zotero-3.0.8-patched/chrome/tmp" directory (referring to the root), to reassemble the client, run:
$ zip -0 -r ../zotero.jar *
$ cd ../../
$ zip -rq ../zotero-3.0.8-custom.xpi * -x ./chrome/tmp/\* ./chrome/zotero.jar.original
Wuala! You have your custom Zotero client.
Hi,
ReplyDeleteMany thanks for your advice.
memcached was necessary to avoid a warning message to run the test_reset and test_setup scripts. I do not think it was required though.
I have rebuilt the DAVLock directory and files and permissions, still with the same issue. I double-checked httpd.conf and httpd-dav.conf, and cannot find what is wrong with them. Weird. I am trying this in a Virtualbox Arch Linux install, but that should not be a problem (and the curl commands work perfect).
For the Zotero program, I shall try rebuild the standalone version, which will be easier to deploy for my users. But thanks for the procedure, this is very helpful.
Finally, I shall try on Debian 7 (Wheezy), the RC1 has just been released. I love Arch, but I think having the procedure in a stable distribution such as Debian may be useful in a work environement. I'll return to you with new developments, and we could update the Zotero wiki. As far as I understand, the Zotero team obfuscate a bit the dataserver install procedure. I understand they want to sell their cloud service, which is very legitimate, but sometimes this is not practical for companies, that may not be able to put data on external servers for legal reasons.
A question regarding WebDAV: should the http server listen both ports 80 and 85? When using cadaver http://127.0.0.1/zotero, that is the default 80 port, isn't it?
ReplyDeleteYes, it is. Plain HTTP with port number 80.
ReplyDeletehttps://wiki.archlinux.org/index.php/WebDAV
Well I have the same issue following the procedure on the Arch wiki you mentioned. Thus I have a config problem or something has changed in Arch.
ReplyDeleteStandalone Zotero client most probably will give a segmentation fault. It's better to use Firefox extension. Arch is very unstable nowadays, constant kernel panics, changes in mkinitcpio scripts.
ReplyDeleteAbout WebDAV problem. Do these permissions work?
$ sudo mkdir /srv/http/zotero
$ sudo chown -R http:http /srv/http/zotero
$ sudo chmod -R 777 /srv/http/zotero
As I understood, the authentication for WebDAV works fine, but the server cannot create/open a collection. Most likely the problem in your Apache configuration files.
About the Zotero data server itself. It was completely useless and buggy, however I managed to update MySQL table structure and it worked. Now I can upload the metadata to the data server and the attachments to the WebDAV server. In further sync sessions some problem occurs and I couldn't solve it. It won't synchronize. Debugging requires a lot of knowledge, JavaScript at the client side, PHP at the server side, the synchronization protocol itself. I am not good at all these stuff, so help is appreciated. Now they are planning to release Zotero 3.1 with new features in the beginning of April, this means that table structure of MySQL database will change.
Well, considering they commercialise the server offer, they have no interest in de-obfuscating their code. Anyway, have you tried to upgrade to the latest Zotero code? Or is it as obscure as before?
ReplyDeleteBesides, I have no issues with Arch at the moment but I guess that a stable Debian would be very suitable here.
Hi,
ReplyDeleteI have made a lot of progress, and have understood the WebDAV configuration. I have been able to adapt your tutorial and run it on Debian. I'll have to redo it properly and will be happy to share the procedure.
It is very similar to your procedure, except the paths of config files and the Apache user is www-data.
I still have a question: in the Zotero Sync Server section of the plugin, which user is that? It is not the testuser of the WebDAV I suppose?
Disregard my silly question above, this is the testuser/testuser the script ./test_setup creates.
ReplyDeleteAlmost there, everything looks fine, but when I click on the Sync button, it rotates forever. cadaver does not show any new collection in zotero (but mkcol works). I am wondering if the problem is with the plugin version?
ReplyDeleteThat exactly what I was talking about. It's a bug. And I don't know how to fix it. May be it is already fixed with the new versions of the server and client.
ReplyDeleteFirstly, enable debugging by setting "Zotero" -> "Preferences" -> "Advanced" -> "Debug Output Logging" to "Enable". Then set the debug level to maximum by checking "Zotero" -> "Preferences" -> "Advanced" -> "Miscellaneous" -> "Open about:config". Search for "extensions.zotero.debug.level". Set it to 5.
If you want to upload (which corresponds to synchronization at the first time) the metadata and attachments, uncheck "Sync automatically" option in Zotero and try to sync manually by pressing the "sync" button. If it doesn't work, there is another temporary solution.
Try to check the "Zotero -> Preferences -> Sync -> Reset -> Restore to Zotero Server" option in the client, then press the "Reset..." button. It should work.
Another advice is to capture the traffic in Wireshark on the loopback interface. Authentication, metadata upload and WebDAV packets will be visible there.
Hi,
ReplyDeleteYes I get this error in the logs:
(5)(+0002001): SELECT version FROM version WHERE schema='lastremotesync'
(5)(+0000000): SELECT version FROM version WHERE schema='lastlocalsync'
(3)(+0000000): HTTP POST version=9&sessionid=********&lastsync=1&upload=1 to http://127.0.0.1:85/updated
(3)(+0000099):
(3)(+0000000): Associated libraries are locked -- waiting 2000ms before next check
(4)(+0001238): Unregistering observer in notifier with hash 'Kt'
I don't know why the associated libraries are locked. I found this: http://forums.zotero.org/discussion/25370/no-longer-syncing-with-zotero-server-associated-files-are-locked/ reporting the same bug.
After I reset, I can sync an empty document base, but I get this error in my logs:
(3)(+0000000): HTTP OPTIONS for http://testuser:********@127.0.0.1/zotero/
(3)(+0000006): HTTP GET http://testuser:********@127.0.0.1/zotero/lastsync
(3)(+0000003):
404 Not Found
Not Found
The requested URL /zotero/lastsync was not found on this server.
Apache/2.2.22 (Debian) Server at 127.0.0.1 Port 80
(3)(+0000000): ===>404<===(number)
However, after creating a document, syncing is locked once again.
I am going to create the missing directories and will report.
Can you give me your mail address? I will write the answer to it.
ReplyDeleteThis comment has been removed by a blog administrator.
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteI keep getting the same loop erros with the upload processor saying:
ReplyDelete2013-04-02 18:01:55.9634 exception 'Exception' with message 'Mysqli statement execute error : Lock wait timeout exceeded; try restarting transaction
in /home/zotero/ZoteroDataServer/include/DB.inc.php:979
Stack trace:
#0 /home/zotero/ZoteroDataServer/include/DB.inc.php(401): Zotero_DB::error(Object(Zend_Db_Statement_Mysqli_Exception), 'INSERT INTO syn...', Array, 1)
#1 /home/zotero/ZoteroDataServer/model/DataObjects.inc.php(671): Zotero_DB::query('INSERT INTO syn...', Array, 1)
#2 /home/zotero/ZoteroDataServer/model/DataObjects.inc.php(709): Zotero_DataObjects::delete(1, '2NDIS8CP')
#3 /home/zotero/ZoteroDataServer/model/Sync.inc.php(1630): Zotero_DataObjects::deleteFromXML(Object(SimpleXMLElement))
#4 /home/zotero/ZoteroDataServer/model/Sync.inc.php(428): Zotero_Sync::processUploadInternal(1, Object(SimpleXMLElement), 1160342154, '2034610610')
#5 /home/zotero/ZoteroDataServer/model/Processor.inc.php(128): Zotero_Sync::processUploadFromQueue('2034610610')
#6 /home/zotero/ZoteroDataServer/model/Processor.inc.php(51): Zotero_Upload_Processor->processFromQueue()
#7 /home/zotero/ZoteroDataServer/processor/upload/processor.php(19): Zotero_Processor->run('2034610610')
#8 {main}
2013-04-02 18:01:55.9652 [2034610610] Exiting on lock error
2013-04-02 18:01:55.9658 LOCK received — waiting 2 seconds
2013-04-02 18:01:57.9662 Purged 0 lost processors
2013-04-02 18:01:57.9675 Purged 0 old processes
2013-04-02 18:01:57.9681 0 processors, 1 queued process
2013-04-02 18:01:57.9682 Starting 1 new processor
2013-04-02 18:01:58.0482 [982339177] Starting sync processor
have you come across this issue?
Hi. Regarding custom Zotero clients.
ReplyDeleteI patched version 3.0.14 of Zotero Firefox extension. It can be found here:
https://github.com/Panzerkampfwagen/dataserver/blob/master/misc/zotero-3.0.14-patched.xpi
I you want to reassemble your own client, take a look at my four commits (by Panzerkampfwagen) here:
https://github.com/Panzerkampfwagen/zotero/commits/master
Instructions for creating an "*.xpi" is somewhere above.
Installation documentation ("*.pdf" file) can be found in
https://github.com/Panzerkampfwagen/dataserver/tree/master/misc
Eric and I tested Zotero 3.0.14 with the latest data server (March 2013) and it worked fine. He had some similar issues with MySQL connector.
https://groups.google.com/forum/?fromgroups=#!topic/zotero-dev/7js-wbCUfME
It is suggested to increase:
"wait_timeout" and "max_allowed_packet" in "/etc/mysql/my.cnf"
http://www.socialengineforum.com/index.php?/topic/11257-zend-db-statement-mysqli-exception/
Hi Dave,
ReplyDeleteI have not got this error so far. The change in "wait_timeout" corrected the "MySQL has gone away" error, but for your error, it seems to be rather an adjustment innodb_lock_wait_timeout (http://dev.mysql.com/doc/refman/5.0/en/innodb-parameters.html#sysvar_innodb_lock_wait_timeout).
Now is your server very busy? What are your versions of the dataserver, of MySQL and of Zend framework in your dataserver?
Look at the install documentation, I have made a few changes in the repository and the procedure, which work fine. Today, I also uploaded the zotero-4.0.3 plugin for Firefox 20, as the version 3 is incompatible.
Hi Dave,
ReplyDeleteHave you been able to fix your problem? If I am not mistaken, errors of type "Mysqli statement execute error" are not directly returned by Zotero, but by the Zend framework code. These can be fixed by changing MySQL settings, such as the lock timeouts.
Hi there!
ReplyDeleteI managed to install the server and - as far as I know - it's running smoothly. However: The Firefox plugin (4.0.3) in the git doesn't seem to work anymore. Even without any login credentials it causes Firefox (v25.0.1) to create a hell of a load (debugging enabled). Neither sync is working...
Is there already a patched edition of a newer plugin? Or is the standanlone-edition finally patched?
Yours Jonas
Confirmed that. In Firefox v20 everything works. And an additional question: Is there any documentation on the structure of the database? I'd really like to add users and them to groups.
DeleteAs far as I understand, users can be added into the _www/users and groups to _master/groups. In _master/groupusers these two are brought together. The library-id in groups refers to _master/libraryies. Though that all seems to fit the group doesn't show up.
Any ideas?
Yours
Jonas
Sorry for the late answer. I am no longer maintaining the data server code in my repository because I don't use the server. The last time Eric added some modifications to the installation instructions in the
Delete"dataserver/misc/Zotero_Data_Server_Installation_Archlinux.pdf" file.
Regarding the database. I haven't seen any documentation on the data server or the database. You can inspect the database structure using PhpMyAdmin GUI during debugging.
About the Zotero Firefox client: you have to follow the guide for assembling Firefox extension posted above on February 21, 2013 at 2:17 AM by me. You can find the difference between files according to the commits in the
"https://github.com/Panzerkampfwagen/zotero/commits/master" repository by Panzerkampfwagen applied to "sync.js", "preferences.xul", "zotero.js" files.
Also some threads regarding the data server zotero-dev gmail group ("https://groups.google.com/d/forum/zotero-dev") may be helpful. Da gibt es meine Antworten. Offensichtlich der Spitzname der einen anderen Waffenname enthält gehört zu mir.
Hi, I have not worked on this for a long time, but managed to add the latest client (4.0.25.2) today. It works with the latest Firefox (v35).
DeleteOn the server side, I did not find the time to update to master (last sync was 20 June 2013). I tried on a local machine a more recent version, and from memory identified the switch from S3 to AWS being a problem, which needs some work. The switch was committed on the 11 March 2014. So the most recent version might be 9 March 2014. If someone has the time to look into it, that would be great indeed.
Hi, could Zotero data server be installed on shared hosting Cpanel without root access?
ReplyDelete