Troubleshooting Zenodo

This small guide will help resolve some of the common issues with running Zenodo locally, both installation (see the documentaion and the development setup, which boils down to two things:

1. How to find the answer in already reported problems
2. How to report a new problem effectively

NOTE: A lot of errors are often related to:

• Not using docker for development, but installing the services (DB, ES, etc.) system-wide (with different versions available for your OS, a million other things, which can complicate debugging process). We highly recommend spending some time to set up docker properly - this way we are all on the same page (more or less). You will gain this time back later on by not having to deal with a lot of nasty problems. Also, if the problem will look like it's specific to your OS, we won't be able to help you out, as we won't be able to reproduce it ourselves.
• Not using virtualenvwrapper correctly (just virtualenv is not enough) and not working in a virtual environment as you install Zenodo and other related modules. Follow the installation guide of virtualenvwrapper HERE.
• Installing something with sudo (unless mentioned in Zenodo installation guide) which shouldn't be installed with sudo. Running some step with sudo might work if something doesn't work for the first time, but it will produce errors later along the way. If some installation command is not working it's usually because you're not in a virtual environment. The only exception is that some fresh installations of docker need to be initially run with sudo (but this also has been solved: How can I use docker without sudo)

How to find the answer in already reported problems

There are to places where the installation and local development problem are reported and they are reported in GitHub Issues tagged as installation:

• Open Issues
• Closed Issues

Another place is the section below titled List of known issues.

How to report a new problem effectively

If you haven't found your problem in the places mentioned above, it is important to create an Issue on Zenodo with the accurate description of your problem, and then asking on Gitter (in that order! ;) ). This way we can help you faster and more effective. Gitter is great for quick general questions, but awful for helping you debug a 3-page Flask stacktrace. Moreover, your question can be lost in the avalanche of other messages before we can see it. After you've created an Issue on Zenodo, you can ask on Gitter to take a look at it and paste the link to the Issue, this way it's more persistent.

When submitting an issue it is crucial to provide the following:

• Your OS (Linux, OSX, Windows?)
• Whether you use docker for DB, ES, MQ, Cache etc. or do you run those services "on metal".
• If you are following the Installation Guide (http://zenodo.readthedocs.io/en/latest/) mention the step at which it happens.
• Stacktrace if you are getting some on the terminal, screenshot if it's some error on the webpage.
• If relevant to the error, provide the version of an service/application/package that is failing:
  • NPM error? What's you NPM version?
  • An error saying something about "Invenio-Deposit"? What's the version? (include output of pip show invenio-deposit)?
  • Elasticsearch error? ..you guessed it! :)
• If the error occurs at some point during development after you've already installed Zenodo:
  • What is the state of code at the moment?
  • Did you modify anything? What's the top commit hash (git log) and what changes are in the stage (git status).
  • Did you install any of the invenio packages locally?

List of known issues

Q: A build in Travis grid fails with the following error:

E           ConnectionTimeout: ConnectionTimeout caused by - ReadTimeout(HTTPConnectionPool(host='localhost', port=9200): Read timed out. (read timeout=10))

Answer: It's not an issue with your code but a known issue with elasticsearch timing out on our test setup. Just ping us (@krzysztof or @slint) on the PR that travis fails with ES timeout and we will restart the build (it should pass)..

Q: Error during zenodo run:

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) fe_sendauth: no password supplied

Answer: You should be getting this only if you don't use docker for development (which is recommended). This is because you need to set the configuration for your database (take a look at the variable SQLALCHEMY_DATABASE_URI in the config). The easiest way to do that is to set it in your shell like this:

export APP_SQLALCHEMY_DATABASE_URI="postgresql://admin:zenodo@localhost/zenodo"

NOTE: Depending on where the PostgreSQL is installed in your system, the URI of the DB might be slightly different. Nonetheless, it should follow the pattern postgresql://user:password@localhost:port/database_name

Q: I've tried to create an ES index and now I have the following error(s):

elasticsearch.exceptions.NotFoundError: TransportError(404, 'index_not_found_exception', 'no such index')

...or...

elasticsearch.exceptions.RequestError: TransportError(400, 'mapper_parsing_exception', 'No handler for type [attachment] declared on field [_attachment]')

First of all, if you get this error:

elasticsearch.exceptions.NotFoundError: TransportError(404, 'index_not_found_exception', 'no such index')

...and you're run Zenodo successfully before, it's possible that ES got scrambled (most likely because you ran a unit test, which was using ES). Try to execute the following to fix that:

zenodo index destroy --force
zenodo index init
zenodo migration reindex -t od_lic
zenodo migration reindex -t recid
zenodo migration reindex -t depid
zenodo index run

otherwise, read along:

Answer: Make sure you are using Elasticsearch in version 2.x. Elasticsearch 5.x is not yet supported. Another requirement is the Elasticsearch Mapper Attachments Plugin, used for file content indexing, which can be installed by running

/usr/share/elasticsearch/bin/plugin install -b mapper-attachments

Don't forget to restart your Elasticsearch instance with

sudo service elasticsearch restart (also wait for a couple of seconds because Elasticsearch takes a while to start).

After executing the zenodo index destroy command, if you get the same error, it means that there was an error with the previous initialization of the index and now not all of indices are properly created. You can fix that by force-removing it zenodo index destroy --force and retrying again with zenodo index init.

Q: NPM errors while running ./scripts/setup-assets.sh:

(...)
npm ERR! Cloning into bare repository '/user/.npm/_git-remotes/git-github-com-webcanvas-angular-schema-form-ckeditor-git-b213fa934759a18b1436e23bfcbd9f0f730f1296-b1e2a8ac'...
npm ERR! fatal: unable to connect to github.com:
npm ERR! github.com[0: 192.30.253.112]: errno=Operation timed out
npm ERR! github.com[1: 192.30.253.113]: errno=Operation timed out
(...)

Answer: Multiple users have already reported this to be a networking issue. If you're behind a proxy or on a university connection which might be blocking some sites, please try change networks or take a look at possible solution here: https://jjasonclark.com/how-to-setup-node-behind-web-proxy/

Q: Error during docker-compose build:

  Cannot initiate the connection to deb.debian.org:80 (..) - connect (101: Network is unreachable)
W: Failed to fetch http://deb.debian.org/debian/dists/jessie/InRelease
W: Failed to fetch http://deb.debian.org/debian/dists/jessie-updates/InRelease
W: Failed to fetch http://security.debian.org/debian/dists/updates/InRelease
(...)
W: Some index files failed to download. They have been ignored, or old ones used instead.

Answer: It's also related to the network you're in and/or proxy settings, please take a look at this SO question: http://stackoverflow.com/questions/23111631/cannot-download-docker-images-behind-a-proxy/28093517#28093517 and this docker guide: https://docs.docker.com/engine/admin/systemd/#/http-proxy

In addition to that, if you get the following (or similar) error with the DNS:

fatal: unable to access 'https://github.com/zenodo/zenodo-migrator.git/': Could not resolve host: github.com

You need to follow the guide here: https://robinwinslow.uk/2016/06/23/fix-docker-networking-dns/

If still getting problems related to npm try out using different connections instead of your university connection or see some solutions here:

1. http://stackoverflow.com/questions/7559648/is-there-a-way-to-make-npm-install-the-command-to-work-behind-proxy
2. http://stackoverflow.com/questions/25660936/using-npm-behind-corporate-proxy-pac
3. https://github.com/angular/angular-phonecat/issues/235

Q: Changes to Zenodo take several minutes to show up on the webpage!

Answer: This can be realted to two things:

1. If you are modifying anything on the Zenodo frontpage, know that it is cached with a 10-minute timeout (see frontpage/views.py#L62). For development you can comment-out this decorator (just make sure not to commit this changes to your PR).
2. You need to restart the application and sometimes the celery worker (if you have modified any of the code in celery tasks) for the code to be picked up. Alternatively run Zenodo in DEBUG mode, which should at least restart the application by itself (see HERE).

Q: OSX and PostgreSQL/psycopg2 installation issues.

Usually it manifests itself like this:

ImportError: dlopen(/Users/youruser/Envs/zenodo/lib/python2.7/site-packages/psycopg2/_psycopg.so, 2): Library not loaded: @rpath/libssl.1.0.0.dylib
  Referenced from: /Users/youruser/Envs/zenodo/lib/python2.7/site-packages/psycopg2/_psycopg.so
  Reason: image not found

This was reported as an issue on OS X Sierra specifically. Try to follow this guide to solve it: https://medium.com/pixel-heart/os-x-sierra-postgresql-and-psycopg2-42c0c95acb23#.gu6gvniyj