[enh] version 0.12.0

[fix] fix duckduckgo's offset

.gitignore

@@ -1,28 +1,18 @@
 .coverage
+coverage/
 .installed.cfg
 engines.cfg
 env
 robot_log.html
 robot_output.xml
 robot_report.html
+test_basic/
 setup.cfg
 
 *.pyc
 */*.pyc
 *~
 
-bin/
-build/
-coverage/
-develop-eggs/
-dist/
-eggs/
-include/
-lib/
-local/
-parts/
-searx.egg-info/
-var/
 node_modules/
 
 .tx/

.travis.yml

@@ -4,25 +4,30 @@ cache:
   - npm
   - directories:
     - $HOME/.cache/pip
+addons:
+  firefox: "latest"
 language: python
 python:
   - "2.7"
+  - "3.6"
 before_install:
   - "export DISPLAY=:99.0"
   - "sh -e /etc/init.d/xvfb start"
-  - npm install -g less grunt-cli
-  - ( cd searx/static/themes/oscar;npm install )
+  - npm install less grunt-cli
+  - ( cd searx/static/themes/oscar;npm install; cd - )
+  - mkdir -p ~/drivers; export PATH=~/drivers:$PATH;
+  - GECKODRIVER_URL="https://github.com/mozilla/geckodriver/releases/download/v0.11.1/geckodriver-v0.11.1-linux64.tar.gz";
+  - FILE=`mktemp`; wget "$GECKODRIVER_URL" -qO $FILE && tar xz -C ~/drivers -f $FILE geckodriver; rm $FILE; chmod 777 ~/drivers/geckodriver;
 install:
-  - "make"
+  - ./manage.sh update_dev_packages
   - pip install coveralls
 script:
-  - "make flake8"
-  - "make robot"
-  - "make styles"
-  - "make grunt"
-  - make coverage
+  - ./manage.sh styles
+  - ./manage.sh grunt_build
+  - ./manage.sh tests
 after_success:
-  coveralls
+  - ./manage.sh py_test_coverage
+  - coveralls
 notifications:
   irc:
     channels:

AUTHORS.rst

@@ -1,4 +1,4 @@
-Searx was created by Adam Tauber and is maintained by Adam Tauber and Alexandre Flament.
+Searx was created by Adam Tauber and is maintained by Adam Tauber, Alexandre Flament and Noémi Ványi.
 
 Major contributing authors:
 
@@ -7,6 +7,8 @@ Major contributing authors:
 - Thomas Pointhuber
 - Alexandre Flament `@dalf <https://github.com/dalf>`_
 - @Cqoicebordel
+- Noémi Ványi
+- Marc Abonce Seguin @a01200356
 
 People who have submitted patches/translates, reported bugs, consulted features or
 generally made searx better:
@@ -39,5 +41,29 @@ generally made searx better:
 - @underr
 - Emmanuel Benazera
 - @GreenLunar
-- Noemi Vanyi
 - Kang-min Liu
+- Kirill Isakov
+- Guilhem Bonnefille
+- @jibe-b
+- Christian Pietsch @pietsch
+- @Maxqia
+- Ashutosh Das @pyprism
+- YuLun Shih @imZack
+- Dmitry Mikhirev @mikhirev
+- David A Roberts `@davidar <https://github.com/davidar>`_
+- Jan Verbeek @blyxxyz
+- Ammar Najjar @ammarnajjar
+- @stepshal
+- François Revol @mmuman
+- Harry Wood @harry-wood
+- Thomas Renard @threnard
+- Pydo `<https://github.com/pydo>`_
+- Athemis `<https://github.com/Athemis>`_
+- Stefan Antoni `<http://stefan.antoni.io>`
+- @firebovine
+- Lorenzo J. Lucchini @luccoj
+- @eig8phei
+- Joachim Cherqui
+- @maxigas
+- Jannik Winkel @kiney
+- @juanitobananas

CHANGELOG.rst

@@ -1,3 +1,129 @@
+0.12.0 2017.06.04
+=================
+
+- Python3 compatibility
+- New engines
+
+  - 1337x.to (files, music, video)
+  - Semantic Scholar (science)
+  - Library Genesis (general)
+  - Framalibre (IT)
+  - Free Software Directory (IT)
+- More compact result UI (oscar theme)
+- Configurable static content and template path
+- Spelling suggestions
+- Multiple engine fixes (duckduckgo, bing, swisscows, yahoo news, bing news, twitter, bing images)
+- Reduced static image size
+- Docker updates
+- Translation updates
+
+
+Special thanks to `NLNet <https://nlnet.nl>`__ for sponsoring multiple features of this release.
+
+
+0.11.0 2017.01.10
+=================
+
+- New engines
+
+  - Protein Data Bank Europe (science)
+  - Voat.co (general, social media)
+  - Online Etimology Dictionary (science)
+  - CCC tv (video, it)
+  - Searx (all categories - can rotate multiple other instances)
+- Answerer functionality (see answerer section on /preferences)
+- Local answerers
+
+  - Statistical functions
+  - Random value generator
+- Result proxy support (with `morty <https://github.com/asciimoo/morty>`__)
+- Extended time range filter
+- Improved search language support
+- Multiple engine fixes (digbt, 500px, google news, ixquick, bing, kickass, google play movies, habrahabr, yandex)
+- Minor UI improvements
+- Suggestion support for JSON engine
+- Result and query escaping fixes
+- Configurable HTTP server version
+- More robust search error handling
+- Faster webapp initialization in debug mode
+- Search module refactor
+- Translation updates
+
+
+0.10.0 2016.09.06
+=================
+
+- New engines
+
+  - Archive.is (general)
+  - INA (videos)
+  - Scanr (science)
+  - Google Scholar (science)
+  - Crossref (science)
+  - Openrepos (files)
+  - Microsoft Academic Search Engine (science)
+  - Hoogle (it)
+  - Diggbt (files)
+  - Dictzone (general - dictionary)
+  - Translated (general - translation)
+- New Plugins
+
+  - Infinite scroll on results page
+  - DOAI rewrite
+- Full theme redesign
+- Display the number of results
+- Filter searches by date range
+- Instance config API endpoint
+- Dependency version updates
+- Socks proxy support for outgoing requests
+- 404 page
+
+
+News
+~~~~
+
+@kvch joined the maintainer team
+
+
+0.9.0 2016.05.24
+================
+
+- New search category: science
+- New engines
+
+  - Wolframalpha (science)
+  - Frinkiac (images)
+  - Arch Linux (it)
+  - BASE - Bielefeld Academic Search Engine (science)
+  - Dokuwiki (general)
+  - Nyaa.se (files, images, music, video)
+  - Reddit (general, images, news, social media)
+  - Torrentz.eu (files, music, video)
+  - Tokyo Toshokan (files, music, video)
+  - F-Droid (files)
+  - Erowid (general)
+  - Bitbucket (it)
+  - GitLab (it)
+  - Geektimes (it)
+  - Habrahabr (it)
+- New plugins
+
+  - Open links in new tab
+  - Vim hotkeys for better navigation
+- Wikipedia/Mediawiki engine improvements
+- Configurable instance name
+- Configurable connection pool size
+- Fixed broken google engine
+- Better docker image
+- Images in standard results
+- Fixed and refactored user settings (Warning: backward incompatibility - you have to reset your custom engine preferences)
+- Suspending engines on errors
+- Simplified development/deployment tooling
+- Translation updates
+- Multilingual autocompleter
+- Qwant autocompleter backend
+
+
 0.8.1 2015.12.22
 ================
 

Dockerfile

@@ -1,22 +1,53 @@
-FROM python:2.7-slim
+FROM alpine:3.5
+MAINTAINER searx <https://github.com/asciimoo/searx>
+LABEL description "A privacy-respecting, hackable metasearch engine."
 
-WORKDIR /app
+ENV BASE_URL=False IMAGE_PROXY=False
+EXPOSE 8888
+WORKDIR /usr/local/searx
+CMD ["/sbin/tini","--","/usr/local/searx/run.sh"]
 
-RUN useradd searx
+RUN adduser -D -h /usr/local/searx -s /bin/sh searx searx \
+ && echo '#!/bin/sh' >> run.sh \
+ && echo 'sed -i "s|base_url : False|base_url : $BASE_URL|g" searx/settings.yml' >> run.sh \
+ && echo 'sed -i "s/image_proxy : False/image_proxy : $IMAGE_PROXY/g" searx/settings.yml' >> run.sh \
+ && echo 'sed -i "s/ultrasecretkey/`openssl rand -hex 16`/g" searx/settings.yml' >> run.sh \
+ && echo 'python searx/webapp.py' >> run.sh \
+ && chmod +x run.sh
 
-EXPOSE 5000
-CMD ["/usr/local/bin/uwsgi", "--uid", "searx", "--gid", "searx", "--http", ":5000", "-w",  "searx.webapp"]
+COPY requirements.txt ./requirements.txt
 
-RUN apt-get update && \
-    apt-get install -y --no-install-recommends \
-            zlib1g-dev libxml2-dev libxslt1-dev libffi-dev build-essential \
-            libssl-dev openssl && \
-    rm -rf /var/lib/apt/lists/*
+RUN echo "@commuedge http://nl.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories \
+ && apk -U add \
+    build-base \
+    python \
+    python-dev \
+    py-pip \
+    libxml2 \
+    libxml2-dev \
+    libxslt \
+    libxslt-dev \
+    libffi-dev \
+    openssl \
+    openssl-dev \
+    ca-certificates \
+    tini@commuedge \
+ && pip install --no-cache -r requirements.txt \
+ && apk del \
+    build-base \
+    python-dev \
+    libffi-dev \
+    openssl-dev \
+    libxslt-dev \
+    libxml2-dev \
+    openssl-dev \
+    ca-certificates \
+ && rm -f /var/cache/apk/*
 
-RUN pip install --no-cache uwsgi
+COPY . .
 
-COPY requirements.txt /app/requirements.txt
-RUN pip install --no-cache -r requirements.txt
+RUN chown -R searx:searx *
 
-COPY . /app
-RUN sed -i -e "s/ultrasecretkey/`openssl rand -hex 16`/g" searx/settings.yml
+USER searx
+
+RUN sed -i "s/127.0.0.1/0.0.0.0/g" searx/settings.yml

Makefile

@@ -1,66 +0,0 @@
-# convenience makefile to boostrap & run buildout
-# use `make options=-v` to run buildout with extra options
-
-version = 2.7
-python = bin/python
-options =
-
-all: .installed.cfg
-
-.installed.cfg: bin/buildout buildout.cfg setup.py
-	bin/buildout $(options)
-
-bin/buildout: $(python) buildout.cfg bootstrap.py
-	$(python) bootstrap.py
-	@touch $@
-
-$(python):
-	virtualenv -p python$(version) --no-site-packages .
-	@touch $@
-
-robot: .installed.cfg
-	@bin/robot
-
-flake8: .installed.cfg
-	@bin/flake8 setup.py
-	@bin/flake8 ./searx/
-
-tests: .installed.cfg flake8
-	@bin/test
-	@grunt test --gruntfile searx/static/themes/oscar/gruntfile.js
-
-coverage: .installed.cfg
-	@bin/coverage run bin/test
-	@bin/coverage report
-	@bin/coverage html
-
-production: bin/buildout production.cfg setup.py
-	bin/buildout -c production.cfg $(options)
-	@echo "* Please modify `readlink --canonicalize-missing ./searx/settings.py`"
-	@echo "* Hint 1: on production, disable debug mode and change secret_key"
-	@echo "* Hint 2: searx will be executed at server startup by crontab"
-	@echo "* Hint 3: to run immediatley, execute 'bin/supervisord'"
-
-minimal: bin/buildout minimal.cfg setup.py
-	bin/buildout -c minimal.cfg $(options)
-
-styles:
-	@lessc -x searx/static/themes/default/less/style.less > searx/static/themes/default/css/style.css
-	@lessc -x searx/static/themes/default/less/style-rtl.less > searx/static/themes/default/css/style-rtl.css
-	@lessc -x searx/static/themes/courgette/less/style.less > searx/static/themes/courgette/css/style.css
-	@lessc -x searx/static/themes/courgette/less/style-rtl.less > searx/static/themes/courgette/css/style-rtl.css
-	@lessc -x searx/static/less/bootstrap/bootstrap.less > searx/static/css/bootstrap.min.css
-	@lessc -x searx/static/themes/oscar/less/oscar/oscar.less > searx/static/themes/oscar/css/oscar.min.css
-	@lessc -x searx/static/themes/pix-art/less/style.less > searx/static/themes/pix-art/css/style.css
-
-grunt:
-	@grunt --gruntfile searx/static/themes/oscar/gruntfile.js
-
-locales:
-	@pybabel compile -d searx/translations
-
-clean:
-	@rm -rf .installed.cfg .mr.developer.cfg bin parts develop-eggs eggs \
-		searx.egg-info lib include .coverage coverage
-
-.PHONY: all tests robot flake8 coverage production minimal styles locales clean

README.rst

@@ -7,137 +7,24 @@ engine <https://en.wikipedia.org/wiki/Metasearch_engine>`__.
 List of `running
 instances <https://github.com/asciimoo/searx/wiki/Searx-instances>`__.
 
-See the `wiki <https://github.com/asciimoo/searx/wiki>`__ for more information.
+See the `documentation <https://asciimoo.github.io/searx>`__ and the `wiki <https://github.com/asciimoo/searx/wiki>`__ for more information.
 
 |Flattr searx|
 
-Features
-~~~~~~~~
-
-- Tracking free
-- Supports multiple output formats
-
-  - json ``curl https://searx.me/?format=json&q=[query]``
-  - csv ``curl https://searx.me/?format=csv&q=[query]``
-  - opensearch/rss ``curl https://searx.me/?format=rss&q=[query]``
-- Opensearch support (you can set as default search engine)
-- Configurable search engines/categories
-- Different search languages
-- Duckduckgo like !bang functionality with engine shortcuts
-- Parallel queries - relatively fast
-
 Installation
 ~~~~~~~~~~~~
 
 -  clone source:
-   ``git clone git@github.com:asciimoo/searx.git && cd searx``
--  install dependencies: ``pip install -r requirements.txt``
+   ``git clone https://github.com/asciimoo/searx.git && cd searx``
+-  install dependencies: ``./manage.sh update_packages``
 -  edit your
-   `settings.yml <https://github.com/asciimoo/searx/blob/master/settings.yml>`__
+   `settings.yml <https://github.com/asciimoo/searx/blob/master/searx/settings.yml>`__
    (set your ``secret_key``!)
 -  run ``python searx/webapp.py`` to start the application
 
 For all the details, follow this `step by step
 installation <https://github.com/asciimoo/searx/wiki/Installation>`__
 
-Alternative (Recommended) Installation
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
--  clone source:
-   ``git clone git@github.com:asciimoo/searx.git && cd searx``
--  build in current folder: ``make minimal``
--  run ``bin/searx-run`` to start the application
-
-Development
-~~~~~~~~~~~
-
-Just run ``make``. Versions of dependencies are pinned down inside
-``versions.cfg`` to produce most stable build. Also remember, NO make
-command should be run as root, not even ``make production``
-
-Deployment
-~~~~~~~~~~
-
--  clone source:
-   ``git clone git@github.com:asciimoo/searx.git && cd searx``
--  build in current folder: ``make production``
--  run ``bin/supervisord`` to start the application
-
-Upgrading
-~~~~~~~~~
-
--  inside previously cloned searx directory run: ``git stash`` to
-   temporarily save any changes you have made
--  pull source: ``git pull origin master``
--  re-build in current folder: ``make production``
--  run ``bin/supervisorctl stop searx`` to stop searx, if it does not,
-   then run ``fuser -k 8888/tcp``
--  run ``bin/supervisorctl reload`` to re-read supervisor config and
-   start searx
-
-Command make
-~~~~~~~~~~~~
-
-``make``
-''''''''
-
-Builds development environment with testing support.
-
-``make tests``
-''''''''''''''
-
-Runs tests. You can write tests
-`here <https://github.com/asciimoo/searx/tree/master/searx/tests>`__ and
-remember 'untested code is broken code'.
-
-``make robot``
-''''''''''''''
-
-Runs robot (Selenium) tests, you must have ``firefox`` installed because
-this functional tests actually run the browser and perform operations on
-it. Also searx is executed with
-`settings\_robot <https://github.com/asciimoo/searx/blob/master/searx/settings_robot.yml>`__.
-
-``make flake8``
-'''''''''''''''
-
-'pep8 is a tool to check your Python code against some of the style
-conventions in `PEP 8 <http://www.python.org/dev/peps/pep-0008/>`__.'
-
-``make coverage``
-'''''''''''''''''
-
-Checks coverage of tests, after running this, execute this:
-``firefox ./coverage/index.html``
-
-``make production``
-'''''''''''''''''''
-
-Used to make co-called production environment - without tests (you
-should ran tests before deploying searx on the server). This installs
-supervisord, so if searx crashes, it will try to pick itself up again.
-And crontab entry is added to start supervisord at server boot.
-
-``make minimal``
-''''''''''''''''
-
-Minimal build - without test frameworks, the quickest build option.
-
-``make clean``
-''''''''''''''
-
-Deletes several folders and files (see ``Makefile`` for more), so that
-next time you run any other ``make`` command it will rebuild everithing.
-
-TODO
-~~~~
-
-- Moar engines
-- Better ui
-- Browser integration
-- Documentation
-- Tests
-
 Bugs
 ~~~~
 

base.cfg

@@ -1,17 +0,0 @@
-[buildout]
-extends = versions.cfg
-unzip = true
-newest = false
-prefer-final = true
-develop = .
-
-eggs =
-    searx
-
-parts =
-    omelette
-
-
-[omelette]
-recipe = collective.recipe.omelette
-eggs = ${buildout:eggs}

bootstrap.py

@@ -1,210 +0,0 @@
-##############################################################################
-#
-# Copyright (c) 2006 Zope Foundation and Contributors.
-# All Rights Reserved.
-#
-# This software is subject to the provisions of the Zope Public License,
-# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
-# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
-# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
-# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
-# FOR A PARTICULAR PURPOSE.
-#
-##############################################################################
-"""Bootstrap a buildout-based project
-
-Simply run this script in a directory containing a buildout.cfg.
-The script accepts buildout command-line options, so you can
-use the -c option to specify an alternate configuration file.
-"""
-
-import os
-import shutil
-import sys
-import tempfile
-
-from optparse import OptionParser
-
-__version__ = '2015-07-01'
-# See zc.buildout's changelog if this version is up to date.
-
-tmpeggs = tempfile.mkdtemp(prefix='bootstrap-')
-
-usage = '''\
-[DESIRED PYTHON FOR BUILDOUT] bootstrap.py [options]
-
-Bootstraps a buildout-based project.
-
-Simply run this script in a directory containing a buildout.cfg, using the
-Python that you want bin/buildout to use.
-
-Note that by using --find-links to point to local resources, you can keep
-this script from going over the network.
-'''
-
-parser = OptionParser(usage=usage)
-parser.add_option("--version",
-                  action="store_true", default=False,
-                  help=("Return bootstrap.py version."))
-parser.add_option("-t", "--accept-buildout-test-releases",
-                  dest='accept_buildout_test_releases',
-                  action="store_true", default=False,
-                  help=("Normally, if you do not specify a --version, the "
-                        "bootstrap script and buildout gets the newest "
-                        "*final* versions of zc.buildout and its recipes and "
-                        "extensions for you.  If you use this flag, "
-                        "bootstrap and buildout will get the newest releases "
-                        "even if they are alphas or betas."))
-parser.add_option("-c", "--config-file",
-                  help=("Specify the path to the buildout configuration "
-                        "file to be used."))
-parser.add_option("-f", "--find-links",
-                  help=("Specify a URL to search for buildout releases"))
-parser.add_option("--allow-site-packages",
-                  action="store_true", default=False,
-                  help=("Let bootstrap.py use existing site packages"))
-parser.add_option("--buildout-version",
-                  help="Use a specific zc.buildout version")
-parser.add_option("--setuptools-version",
-                  help="Use a specific setuptools version")
-parser.add_option("--setuptools-to-dir",
-                  help=("Allow for re-use of existing directory of "
-                        "setuptools versions"))
-
-options, args = parser.parse_args()
-if options.version:
-    print("bootstrap.py version %s" % __version__)
-    sys.exit(0)
-
-
-######################################################################
-# load/install setuptools
-
-try:
-    from urllib.request import urlopen
-except ImportError:
-    from urllib2 import urlopen
-
-ez = {}
-if os.path.exists('ez_setup.py'):
-    exec(open('ez_setup.py').read(), ez)
-else:
-    exec(urlopen('https://bootstrap.pypa.io/ez_setup.py').read(), ez)
-
-if not options.allow_site_packages:
-    # ez_setup imports site, which adds site packages
-    # this will remove them from the path to ensure that incompatible versions
-    # of setuptools are not in the path
-    import site
-    # inside a virtualenv, there is no 'getsitepackages'.
-    # We can't remove these reliably
-    if hasattr(site, 'getsitepackages'):
-        for sitepackage_path in site.getsitepackages():
-            # Strip all site-packages directories from sys.path that
-            # are not sys.prefix; this is because on Windows
-            # sys.prefix is a site-package directory.
-            if sitepackage_path != sys.prefix:
-                sys.path[:] = [x for x in sys.path
-                               if sitepackage_path not in x]
-
-setup_args = dict(to_dir=tmpeggs, download_delay=0)
-
-if options.setuptools_version is not None:
-    setup_args['version'] = options.setuptools_version
-if options.setuptools_to_dir is not None:
-    setup_args['to_dir'] = options.setuptools_to_dir
-
-ez['use_setuptools'](**setup_args)
-import setuptools
-import pkg_resources
-
-# This does not (always?) update the default working set.  We will
-# do it.
-for path in sys.path:
-    if path not in pkg_resources.working_set.entries:
-        pkg_resources.working_set.add_entry(path)
-
-######################################################################
-# Install buildout
-
-ws = pkg_resources.working_set
-
-setuptools_path = ws.find(
-    pkg_resources.Requirement.parse('setuptools')).location
-
-# Fix sys.path here as easy_install.pth added before PYTHONPATH
-cmd = [sys.executable, '-c',
-       'import sys; sys.path[0:0] = [%r]; ' % setuptools_path +
-       'from setuptools.command.easy_install import main; main()',
-       '-mZqNxd', tmpeggs]
-
-find_links = os.environ.get(
-    'bootstrap-testing-find-links',
-    options.find_links or
-    ('http://downloads.buildout.org/'
-     if options.accept_buildout_test_releases else None)
-    )
-if find_links:
-    cmd.extend(['-f', find_links])
-
-requirement = 'zc.buildout'
-version = options.buildout_version
-if version is None and not options.accept_buildout_test_releases:
-    # Figure out the most recent final version of zc.buildout.
-    import setuptools.package_index
-    _final_parts = '*final-', '*final'
-
-    def _final_version(parsed_version):
-        try:
-            return not parsed_version.is_prerelease
-        except AttributeError:
-            # Older setuptools
-            for part in parsed_version:
-                if (part[:1] == '*') and (part not in _final_parts):
-                    return False
-            return True
-
-    index = setuptools.package_index.PackageIndex(
-        search_path=[setuptools_path])
-    if find_links:
-        index.add_find_links((find_links,))
-    req = pkg_resources.Requirement.parse(requirement)
-    if index.obtain(req) is not None:
-        best = []
-        bestv = None
-        for dist in index[req.project_name]:
-            distv = dist.parsed_version
-            if _final_version(distv):
-                if bestv is None or distv > bestv:
-                    best = [dist]
-                    bestv = distv
-                elif distv == bestv:
-                    best.append(dist)
-        if best:
-            best.sort()
-            version = best[-1].version
-if version:
-    requirement = '=='.join((requirement, version))
-cmd.append(requirement)
-
-import subprocess
-if subprocess.call(cmd) != 0:
-    raise Exception(
-        "Failed to execute command:\n%s" % repr(cmd)[1:-1])
-
-######################################################################
-# Import and run buildout
-
-ws.add_entry(tmpeggs)
-ws.require(requirement)
-import zc.buildout.buildout
-
-if not [a for a in args if '=' not in a]:
-    args.append('bootstrap')
-
-# if -c was provided, we push it back into args for buildout' main function
-if options.config_file is not None:
-    args[0:0] = ['-c', options.config_file]
-
-zc.buildout.buildout.main(args)
-shutil.rmtree(tmpeggs)

buildout.cfg

@@ -1,30 +0,0 @@
-[buildout]
-extends = base.cfg
-develop = .
-
-eggs =
-    searx [test]
-
-parts +=
-    pyscripts
-    robot
-    test
-
-
-[pyscripts]
-recipe = zc.recipe.egg:script
-eggs = ${buildout:eggs}
-interpreter = py
-dependent-scripts = true
-
-
-[robot]
-recipe = zc.recipe.testrunner
-eggs = ${buildout:eggs}
-defaults = ['--color', '--auto-progress', '--layer', 'SearxRobotLayer']
-
-
-[test]
-recipe = zc.recipe.testrunner
-eggs = ${buildout:eggs}
-defaults = ['--color', '--auto-progress', '--layer', 'SearxTestLayer', '--layer', '!SearxRobotLayer']

examples/basic_engine.py

@@ -1,5 +1,6 @@
 
-categories = ['general'] # optional
+categories = ['general']  # optional
+
 
 def request(query, params):
     '''pre-request callback
@@ -22,4 +23,3 @@ def response(resp):
     resp: requests response object
     '''
     return [{'url': '', 'title': '', 'content': ''}]
-

manage.sh

@@ -0,0 +1,128 @@
+#!/bin/sh
+
+BASE_DIR=$(dirname "`readlink -f "$0"`")
+PYTHONPATH=$BASE_DIR
+SEARX_DIR="$BASE_DIR/searx"
+ACTION=$1
+
+update_packages() {
+    pip install -r "$BASE_DIR/requirements.txt"
+}
+
+update_dev_packages() {
+    update_packages
+    pip install -r "$BASE_DIR/requirements-dev.txt"
+}
+
+check_geckodriver() {
+    echo '[!] Checking geckodriver'
+    set -e
+    geckodriver -V 2>1 > /dev/null || NOTFOUND=1
+    set +e
+    if [ -z $NOTFOUND ]; then
+	return
+    fi
+    GECKODRIVER_VERSION="v0.11.1"
+    PLATFORM=`python -c "import platform; print platform.system().lower(), platform.architecture()[0]"`
+    case $PLATFORM in
+	"linux 32bit" | "linux2 32bit") ARCH="linux32";;
+	"linux 64bit" | "linux2 64bit") ARCH="linux64";;
+	"windows 32 bit") ARCH="win32";;
+	"windows 64 bit") ARCH="win64";;
+	"mac 64bit") ARCH="macos";;
+    esac
+    GECKODRIVER_URL="https://github.com/mozilla/geckodriver/releases/download/$GECKODRIVER_VERSION/geckodriver-$GECKODRIVER_VERSION-$ARCH.tar.gz";
+    if [ -z "$VIRTUAL_ENV" ]; then
+	echo "geckodriver can't be installed because VIRTUAL_ENV is not set, you should download it from\n  $GECKODRIVER_URL"
+	exit
+    else
+	echo "Installing $VIRTUAL_ENV from\n  $GECKODRIVER_URL"
+	FILE=`mktemp`
+	wget "$GECKODRIVER_URL" -qO $FILE && tar xz -C $VIRTUAL_ENV/bin/ -f $FILE geckodriver
+	rm $FILE
+	chmod 777 $VIRTUAL_ENV/bin/geckodriver
+    fi
+}
+
+pep8_check() {
+    echo '[!] Running pep8 check'
+    # ignored rules:
+    #  E402 module level import not at top of file
+    #  W503 line break before binary operator
+    pep8 --max-line-length=120 --ignore "E402,W503" "$SEARX_DIR" "$BASE_DIR/tests"
+}
+
+unit_tests() {
+    echo '[!] Running unit tests'
+    python -m nose2 -s "$BASE_DIR/tests/unit"
+}
+
+py_test_coverage() {
+    echo '[!] Running python test coverage'
+    PYTHONPATH=`pwd` python -m nose2 -C --coverage "$SEARX_DIR" -s "$BASE_DIR/tests/unit"
+    coverage report
+    coverage html
+}
+
+robot_tests() {
+    echo '[!] Running robot tests'
+    PYTHONPATH=`pwd` python "$SEARX_DIR/testing.py" robot
+}
+
+tests() {
+    set -e
+    pep8_check
+    unit_tests
+    check_geckodriver
+    robot_tests
+    set +e
+}
+
+build_style() {
+    lessc -x "$BASE_DIR/searx/static/$1" "$BASE_DIR/searx/static/$2"
+}
+
+styles() {
+    echo '[!] Building styles'
+	build_style themes/legacy/less/style.less themes/legacy/css/style.css
+	build_style themes/legacy/less/style-rtl.less themes/legacy/css/style-rtl.css
+	build_style themes/courgette/less/style.less themes/courgette/css/style.css
+	build_style themes/courgette/less/style-rtl.less themes/courgette/css/style-rtl.css
+	build_style less/bootstrap/bootstrap.less css/bootstrap.min.css
+	build_style themes/oscar/less/pointhi/oscar.less themes/oscar/css/pointhi.min.css
+	build_style themes/oscar/less/logicodev/oscar.less themes/oscar/css/logicodev.min.css
+	build_style themes/pix-art/less/style.less themes/pix-art/css/style.css
+}
+
+grunt_build() {
+	grunt --gruntfile "$SEARX_DIR/static/themes/oscar/gruntfile.js"
+}
+
+locales() {
+	pybabel compile -d "$SEARX_DIR/translations"
+}
+
+help() {
+    [ -z "$1" ] || printf "Error: $1\n"
+    echo "Searx manage.sh help
+
+Commands
+========
+    grunt_build          - Build js files
+    help                 - This text
+    locales              - Compile locales
+    pep8_check           - Pep8 validation
+    py_test_coverage     - Unit test coverage
+    robot_tests          - Run selenium tests
+    styles               - Build less files
+    tests                - Run all python tests (pep8, unit, robot)
+    unit_tests           - Run unit tests
+    update_dev_packages  - Check & update development and production dependency changes
+    update_packages      - Check & update dependency changes
+    check_geckodriver    - Check & download geckodriver (required for robot_tests)
+"
+}
+
+[ "$(command -V "$ACTION" | grep ' function$')" = "" ] \
+    && help "action not found" \
+    || $ACTION

minimal.cfg

@@ -1,15 +0,0 @@
-[buildout]
-extends = base.cfg
-develop = .
-
-eggs =
-    searx
-
-parts +=
-    pyscripts
-
-
-[pyscripts]
-recipe = zc.recipe.egg:script
-eggs = ${buildout:eggs}
-interpreter = py

production.cfg

@@ -1,34 +0,0 @@
-[buildout]
-extends = base.cfg
-develop = .
-
-eggs =
-    searx
-
-parts +=
-    pyscripts
-    supervisor
-    crontab_reboot
-
-
-[pyscripts]
-recipe = zc.recipe.egg:script
-eggs = ${buildout:eggs}
-interpreter = py
-
-
-[supervisor]
-recipe = collective.recipe.supervisor
-http-socket = unix
-user = searxer
-password = ohpleasedochangeme
-file = /tmp/supervisor.sock
-chmod = 0700
-programs =
-    50 searx    ${buildout:bin-directory}/searx-run
-
-
-[crontab_reboot]
-recipe = z3c.recipe.usercrontab
-times = @reboot
-command = ${buildout:bin-directory}/supervisord

requirements-dev.txt

@@ -0,0 +1,10 @@
+babel==2.3.4
+mock==2.0.0
+nose2[coverage-plugin]
+pep8==1.7.0
+plone.testing==5.0.0
+splinter==0.7.5
+transifex-client==0.12.2
+unittest2==1.1.0
+zope.testrunner==4.5.1
+selenium==3.0.1

requirements.txt

@@ -1,12 +1,10 @@
-flask
-flask-babel
-requests
-lxml
-pyyaml
-pygments
-python-dateutil
-ndg-httpsclient
-pyopenssl
-pyasn1
-pyasn1-modules
-certifi
+certifi==2017.1.23
+flask==0.12
+flask-babel==0.11.1
+lxml==3.7.3
+idna==2.5
+pygments==2.1.3
+pyopenssl==16.2.0
+python-dateutil==2.6.0
+pyyaml==3.12
+requests[socks]==2.13.0

searx/__init__.py

@@ -15,9 +15,11 @@ along with searx. If not, see < http://www.gnu.org/licenses/ >.
 (C) 2013- by Adam Tauber, <asciimoo@gmail.com>
 '''
 
+import certifi
 import logging
 from os import environ
-from os.path import realpath, dirname, join, abspath
+from os.path import realpath, dirname, join, abspath, isfile
+from ssl import OPENSSL_VERSION_INFO, OPENSSL_VERSION
 try:
     from yaml import load
 except:
@@ -28,23 +30,60 @@ except:
 searx_dir = abspath(dirname(__file__))
 engine_dir = dirname(realpath(__file__))
 
-# if possible set path to settings using the
-# enviroment variable SEARX_SETTINGS_PATH
+
+def check_settings_yml(file_name):
+    if isfile(file_name):
+        return file_name
+    else:
+        return None
+
+# find location of settings.yml
 if 'SEARX_SETTINGS_PATH' in environ:
-    settings_path = environ['SEARX_SETTINGS_PATH']
-# otherwise using default path
+    # if possible set path to settings using the
+    # enviroment variable SEARX_SETTINGS_PATH
+    settings_path = check_settings_yml(environ['SEARX_SETTINGS_PATH'])
 else:
-    settings_path = join(searx_dir, 'settings.yml')
+    # if not, get it from searx code base or last solution from /etc/searx
+    settings_path = check_settings_yml(join(searx_dir, 'settings.yml')) or check_settings_yml('/etc/searx/settings.yml')
+
+if not settings_path:
+    raise Exception('settings.yml not found')
 
 # load settings
 with open(settings_path) as settings_yaml:
     settings = load(settings_yaml)
 
-if settings.get('general', {}).get('debug'):
+'''
+enable debug if
+the environnement variable SEARX_DEBUG is 1 or true
+(whatever the value in settings.yml)
+or general.debug=True in settings.yml
+
+disable debug if
+the environnement variable SEARX_DEBUG is 0 or false
+(whatever the value in settings.yml)
+or general.debug=False in settings.yml
+'''
+searx_debug_env = environ.get('SEARX_DEBUG', '').lower()
+if searx_debug_env == 'true' or searx_debug_env == '1':
+    searx_debug = True
+elif searx_debug_env == 'false' or searx_debug_env == '0':
+    searx_debug = False
+else:
+    searx_debug = settings.get('general', {}).get('debug')
+
+if searx_debug:
     logging.basicConfig(level=logging.DEBUG)
 else:
     logging.basicConfig(level=logging.WARNING)
 
 logger = logging.getLogger('searx')
+logger.debug('read configuration from %s', settings_path)
+# Workaround for openssl versions <1.0.2
+# https://github.com/certifi/python-certifi/issues/26
+if OPENSSL_VERSION_INFO[0:3] < (1, 0, 2):
+    if hasattr(certifi, 'old_where'):
+        environ['REQUESTS_CA_BUNDLE'] = certifi.old_where()
+    logger.warning('You are using an old openssl version({0}), please upgrade above 1.0.2!'.format(OPENSSL_VERSION))
 
 logger.info('Initialisation done')

searx/answerers/__init__.py

@@ -0,0 +1,50 @@
+from os import listdir
+from os.path import realpath, dirname, join, isdir
+from sys import version_info
+from searx.utils import load_module
+from collections import defaultdict
+
+if version_info[0] == 3:
+    unicode = str
+
+
+answerers_dir = dirname(realpath(__file__))
+
+
+def load_answerers():
+    answerers = []
+    for filename in listdir(answerers_dir):
+        if not isdir(join(answerers_dir, filename)) or filename.startswith('_'):
+            continue
+        module = load_module('answerer.py', join(answerers_dir, filename))
+        if not hasattr(module, 'keywords') or not isinstance(module.keywords, tuple) or not len(module.keywords):
+            exit(2)
+        answerers.append(module)
+    return answerers
+
+
+def get_answerers_by_keywords(answerers):
+    by_keyword = defaultdict(list)
+    for answerer in answerers:
+        for keyword in answerer.keywords:
+            for keyword in answerer.keywords:
+                by_keyword[keyword].append(answerer.answer)
+    return by_keyword
+
+
+def ask(query):
+    results = []
+    query_parts = list(filter(None, query.query.split()))
+
+    if query_parts[0].decode('utf-8') not in answerers_by_keywords:
+        return results
+
+    for answerer in answerers_by_keywords[query_parts[0].decode('utf-8')]:
+        result = answerer(query)
+        if result:
+            results.append(result)
+    return results
+
+
+answerers = load_answerers()
+answerers_by_keywords = get_answerers_by_keywords(answerers)

searx/answerers/random/answerer.py

@@ -0,0 +1,55 @@
+import random
+import string
+import sys
+from flask_babel import gettext
+
+# required answerer attribute
+# specifies which search query keywords triggers this answerer
+keywords = ('random',)
+
+random_int_max = 2**31
+
+if sys.version_info[0] == 2:
+    random_string_letters = string.lowercase + string.digits + string.uppercase
+else:
+    unicode = str
+    random_string_letters = string.ascii_lowercase + string.digits + string.ascii_uppercase
+
+
+def random_string():
+    return u''.join(random.choice(random_string_letters)
+                    for _ in range(random.randint(8, 32)))
+
+
+def random_float():
+    return unicode(random.random())
+
+
+def random_int():
+    return unicode(random.randint(-random_int_max, random_int_max))
+
+
+random_types = {b'string': random_string,
+                b'int': random_int,
+                b'float': random_float}
+
+
+# required answerer function
+# can return a list of results (any result type) for a given query
+def answer(query):
+    parts = query.query.split()
+    if len(parts) != 2:
+        return []
+
+    if parts[1] not in random_types:
+        return []
+
+    return [{'answer': random_types[parts[1]]()}]
+
+
+# required answerer function
+# returns information about the answerer
+def self_info():
+    return {'name': gettext('Random value generator'),
+            'description': gettext('Generate different random values'),
+            'examples': [u'random {}'.format(x) for x in random_types]}

searx/answerers/statistics/answerer.py

@@ -0,0 +1,55 @@
+from sys import version_info
+from functools import reduce
+from operator import mul
+
+from flask_babel import gettext
+
+if version_info[0] == 3:
+    unicode = str
+
+keywords = ('min',
+            'max',
+            'avg',
+            'sum',
+            'prod')
+
+
+# required answerer function
+# can return a list of results (any result type) for a given query
+def answer(query):
+    parts = query.query.split()
+
+    if len(parts) < 2:
+        return []
+
+    try:
+        args = list(map(float, parts[1:]))
+    except:
+        return []
+
+    func = parts[0]
+    answer = None
+
+    if func == b'min':
+        answer = min(args)
+    elif func == b'max':
+        answer = max(args)
+    elif func == b'avg':
+        answer = sum(args) / len(args)
+    elif func == b'sum':
+        answer = sum(args)
+    elif func == b'prod':
+        answer = reduce(mul, args, 1)
+
+    if answer is None:
+        return []
+
+    return [{'answer': unicode(answer)}]
+
+
+# required answerer function
+# returns information about the answerer
+def self_info():
+    return {'name': gettext('Statistics functions'),
+            'description': gettext('Compute {functions} of the arguments').format(functions='/'.join(keywords)),
+            'examples': ['avg 123 548 2.04 24.2']}

searx/autocomplete.py

@@ -18,7 +18,6 @@ along with searx. If not, see < http://www.gnu.org/licenses/ >.
 
 from lxml import etree
 from json import loads
-from urllib import urlencode
 from searx import settings
 from searx.languages import language_codes
 from searx.engines import (
@@ -26,6 +25,11 @@ from searx.engines import (
 )
 from searx.poolrequests import get as http_get
 
+try:
+    from urllib import urlencode
+except:
+    from urllib.parse import urlencode
+
 
 def get(*args, **kwargs):
     if 'timeout' not in kwargs:
@@ -81,22 +85,22 @@ def searx_bang(full_query):
             engine_query = full_query.getSearchQuery()[1:]
 
             for lc in language_codes:
-                lang_id, lang_name, country = map(str.lower, lc)
+                lang_id, lang_name, country, english_name = map(unicode.lower, lc)
 
                 # check if query starts with language-id
                 if lang_id.startswith(engine_query):
                     if len(engine_query) <= 2:
-                        results.append(':{lang_id}'.format(lang_id=lang_id.split('_')[0]))
+                        results.append(u':{lang_id}'.format(lang_id=lang_id.split('-')[0]))
                     else:
-                        results.append(':{lang_id}'.format(lang_id=lang_id))
+                        results.append(u':{lang_id}'.format(lang_id=lang_id))
 
                 # check if query starts with language name
-                if lang_name.startswith(engine_query):
-                    results.append(':{lang_name}'.format(lang_name=lang_name))
+                if lang_name.startswith(engine_query) or english_name.startswith(engine_query):
+                    results.append(u':{lang_name}'.format(lang_name=lang_name))
 
                 # check if query starts with country
                 if country.startswith(engine_query.replace('_', ' ')):
-                    results.append(':{country}'.format(country=country.replace(' ', '_')))
+                    results.append(u':{country}'.format(country=country.replace(' ', '_')))
 
     # remove duplicates
     result_set = set(results)
@@ -110,12 +114,11 @@ def searx_bang(full_query):
     return list(result_set)
 
 
-def dbpedia(query):
+def dbpedia(query, lang):
     # dbpedia autocompleter, no HTTPS
     autocomplete_url = 'http://lookup.dbpedia.org/api/search.asmx/KeywordSearch?'
 
-    response = get(autocomplete_url
-                   + urlencode(dict(QueryString=query)))
+    response = get(autocomplete_url + urlencode(dict(QueryString=query)))
 
     results = []
 
@@ -127,7 +130,7 @@ def dbpedia(query):
     return results
 
 
-def duckduckgo(query):
+def duckduckgo(query, lang):
     # duckduckgo autocompleter
     url = 'https://ac.duckduckgo.com/ac/?{0}&type=list'
 
@@ -137,12 +140,11 @@ def duckduckgo(query):
     return []
 
 
-def google(query):
+def google(query, lang):
     # google autocompleter
     autocomplete_url = 'https://suggestqueries.google.com/complete/search?client=toolbar&'
 
-    response = get(autocomplete_url
-                   + urlencode(dict(q=query)))
+    response = get(autocomplete_url + urlencode(dict(hl=lang, q=query)))
 
     results = []
 
@@ -153,8 +155,8 @@ def google(query):
     return results
 
 
-def startpage(query):
-    # wikipedia autocompleter
+def startpage(query, lang):
+    # startpage autocompleter
     url = 'https://startpage.com/do/suggest?{query}'
 
     resp = get(url.format(query=urlencode({'query': query}))).text.split('\n')
@@ -163,9 +165,26 @@ def startpage(query):
     return []
 
 
-def wikipedia(query):
+def qwant(query, lang):
+    # qwant autocompleter (additional parameter : lang=en_en&count=xxx )
+    url = 'https://api.qwant.com/api/suggest?{query}'
+
+    resp = get(url.format(query=urlencode({'q': query, 'lang': lang})))
+
+    results = []
+
+    if resp.ok:
+        data = loads(resp.text)
+        if data['status'] == 'success':
+            for item in data['data']['items']:
+                results.append(item['value'])
+
+    return results
+
+
+def wikipedia(query, lang):
     # wikipedia autocompleter
-    url = 'https://en.wikipedia.org/w/api.php?action=opensearch&{0}&limit=10&namespace=0&format=json'
+    url = 'https://' + lang + '.wikipedia.org/w/api.php?action=opensearch&{0}&limit=10&namespace=0&format=json'
 
     resp = loads(get(url.format(urlencode(dict(search=query)))).text)
     if len(resp) > 1:
@@ -177,5 +196,6 @@ backends = {'dbpedia': dbpedia,
             'duckduckgo': duckduckgo,
             'google': google,
             'startpage': startpage,
+            'qwant': qwant,
             'wikipedia': wikipedia
             }

searx/engines/1337x.py

@@ -0,0 +1,39 @@
+from lxml import html
+from searx.engines.xpath import extract_text
+from searx.utils import get_torrent_size
+from searx.url_utils import quote, urljoin
+
+url = 'https://1337x.to/'
+search_url = url + 'search/{search_term}/{pageno}/'
+categories = ['videos']
+paging = True
+
+
+def request(query, params):
+    params['url'] = search_url.format(search_term=quote(query), pageno=params['pageno'])
+
+    return params
+
+
+def response(resp):
+    results = []
+
+    dom = html.fromstring(resp.text)
+
+    for result in dom.xpath('//table[contains(@class, "table-list")]/tbody//tr'):
+        href = urljoin(url, result.xpath('./td[contains(@class, "name")]/a[2]/@href')[0])
+        title = extract_text(result.xpath('./td[contains(@class, "name")]/a[2]'))
+        seed = extract_text(result.xpath('.//td[contains(@class, "seeds")]'))
+        leech = extract_text(result.xpath('.//td[contains(@class, "leeches")]'))
+        filesize_info = extract_text(result.xpath('.//td[contains(@class, "size")]/text()'))
+        filesize, filesize_multiplier = filesize_info.split()
+        filesize = get_torrent_size(filesize, filesize_multiplier)
+
+        results.append({'url': href,
+                        'title': title,
+                        'seed': seed,
+                        'leech': leech,
+                        'filesize': filesize,
+                        'template': 'torrent.html'})
+
+    return results

searx/engines/__init__.py

@@ -16,13 +16,15 @@ along with searx. If not, see < http://www.gnu.org/licenses/ >.
 (C) 2013- by Adam Tauber, <asciimoo@gmail.com>
 '''
 
-from os.path import realpath, dirname, splitext, join
+from os.path import realpath, dirname
 import sys
-from imp import load_source
-from flask.ext.babel import gettext
+from flask_babel import gettext
 from operator import itemgetter
+from json import loads
+from requests import get
 from searx import settings
 from searx import logger
+from searx.utils import load_module
 
 
 logger = logger.getChild('engines')
@@ -33,22 +35,35 @@ engines = {}
 
 categories = {'general': []}
 
+languages = loads(open(engine_dir + '/../data/engines_languages.json').read())
+
 engine_shortcuts = {}
-
-
-def load_module(filename):
-    modname = splitext(filename)[0]
-    if modname in sys.modules:
-        del sys.modules[modname]
-    filepath = join(engine_dir, filename)
-    module = load_source(modname, filepath)
-    module.name = modname
-    return module
+engine_default_args = {'paging': False,
+                       'categories': ['general'],
+                       'language_support': True,
+                       'supported_languages': [],
+                       'safesearch': False,
+                       'timeout': settings['outgoing']['request_timeout'],
+                       'shortcut': '-',
+                       'disabled': False,
+                       'suspend_end_time': 0,
+                       'continuous_errors': 0,
+                       'time_range_support': False}
 
 
 def load_engine(engine_data):
-    engine_name = engine_data['engine']
-    engine = load_module(engine_name + '.py')
+
+    if '_' in engine_data['name']:
+        logger.error('Engine name conains underscore: "{}"'.format(engine_data['name']))
+        sys.exit(1)
+
+    engine_module = engine_data['engine']
+
+    try:
+        engine = load_module(engine_module + '.py', engine_dir)
+    except:
+        logger.exception('Cannot load engine "{}"'.format(engine_module))
+        return None
 
     for param_name in engine_data:
         if param_name == 'engine':
@@ -57,31 +72,13 @@ def load_engine(engine_data):
             if engine_data['categories'] == 'none':
                 engine.categories = []
             else:
-                engine.categories = map(
-                    str.strip, engine_data['categories'].split(','))
+                engine.categories = list(map(str.strip, engine_data['categories'].split(',')))
             continue
         setattr(engine, param_name, engine_data[param_name])
 
-    if not hasattr(engine, 'paging'):
-        engine.paging = False
-
-    if not hasattr(engine, 'categories'):
-        engine.categories = ['general']
-
-    if not hasattr(engine, 'language_support'):
-        engine.language_support = True
-
-    if not hasattr(engine, 'safesearch'):
-        engine.safesearch = False
-
-    if not hasattr(engine, 'timeout'):
-        engine.timeout = settings['outgoing']['request_timeout']
-
-    if not hasattr(engine, 'shortcut'):
-        engine.shortcut = ''
-
-    if not hasattr(engine, 'disabled'):
-        engine.disabled = False
+    for arg_name, arg_value in engine_default_args.items():
+        if not hasattr(engine, arg_name):
+            setattr(engine, arg_name, arg_value)
 
     # checking required variables
     for engine_attr in dir(engine):
@@ -92,55 +89,88 @@ def load_engine(engine_data):
                          .format(engine.name, engine_attr))
             sys.exit(1)
 
+    # assign supported languages from json file
+    if engine_data['name'] in languages:
+        setattr(engine, 'supported_languages', languages[engine_data['name']])
+
+    # assign language fetching method if auxiliary method exists
+    if hasattr(engine, '_fetch_supported_languages'):
+        setattr(engine, 'fetch_supported_languages',
+                lambda: engine._fetch_supported_languages(get(engine.supported_languages_url)))
+
     engine.stats = {
         'result_count': 0,
         'search_count': 0,
         'page_load_time': 0,
+        'page_load_count': 0,
+        'engine_time': 0,
+        'engine_time_count': 0,
         'score_count': 0,
         'errors': 0
     }
 
-    if hasattr(engine, 'categories'):
-        for category_name in engine.categories:
-            categories.setdefault(category_name, []).append(engine)
-    else:
-        categories['general'].append(engine)
+    for category_name in engine.categories:
+        categories.setdefault(category_name, []).append(engine)
+
+    if engine.shortcut in engine_shortcuts:
+        logger.error('Engine config error: ambigious shortcut: {0}'.format(engine.shortcut))
+        sys.exit(1)
+
+    engine_shortcuts[engine.shortcut] = engine.name
 
-    if engine.shortcut:
-        if engine.shortcut in engine_shortcuts:
-            logger.error('Engine config error: ambigious shortcut: {0}'
-                         .format(engine.shortcut))
-            sys.exit(1)
-        engine_shortcuts[engine.shortcut] = engine.name
     return engine
 
 
+def to_percentage(stats, maxvalue):
+    for engine_stat in stats:
+        if maxvalue:
+            engine_stat['percentage'] = int(engine_stat['avg'] / maxvalue * 100)
+        else:
+            engine_stat['percentage'] = 0
+    return stats
+
+
 def get_engines_stats():
     # TODO refactor
     pageloads = []
+    engine_times = []
     results = []
     scores = []
     errors = []
     scores_per_result = []
 
-    max_pageload = max_results = max_score = max_errors = max_score_per_result = 0  # noqa
+    max_pageload = max_engine_times = max_results = max_score = max_errors = max_score_per_result = 0  # noqa
     for engine in engines.values():
         if engine.stats['search_count'] == 0:
             continue
         results_num = \
             engine.stats['result_count'] / float(engine.stats['search_count'])
-        load_times = engine.stats['page_load_time'] / float(engine.stats['search_count'])  # noqa
+
+        if engine.stats['page_load_count'] != 0:
+            load_times = engine.stats['page_load_time'] / float(engine.stats['page_load_count'])  # noqa
+        else:
+            load_times = 0
+
+        if engine.stats['engine_time_count'] != 0:
+            this_engine_time = engine.stats['engine_time'] / float(engine.stats['engine_time_count'])  # noqa
+        else:
+            this_engine_time = 0
+
         if results_num:
             score = engine.stats['score_count'] / float(engine.stats['search_count'])  # noqa
             score_per_result = score / results_num
         else:
             score = score_per_result = 0.0
-        max_results = max(results_num, max_results)
+
         max_pageload = max(load_times, max_pageload)
+        max_engine_times = max(this_engine_time, max_engine_times)
+        max_results = max(results_num, max_results)
         max_score = max(score, max_score)
         max_score_per_result = max(score_per_result, max_score_per_result)
         max_errors = max(max_errors, engine.stats['errors'])
+
         pageloads.append({'avg': load_times, 'name': engine.name})
+        engine_times.append({'avg': this_engine_time, 'name': engine.name})
         results.append({'avg': results_num, 'name': engine.name})
         scores.append({'avg': score, 'name': engine.name})
         errors.append({'avg': engine.stats['errors'], 'name': engine.name})
@@ -149,38 +179,18 @@ def get_engines_stats():
             'name': engine.name
         })
 
-    for engine in pageloads:
-        if max_pageload:
-            engine['percentage'] = int(engine['avg'] / max_pageload * 100)
-        else:
-            engine['percentage'] = 0
-
-    for engine in results:
-        if max_results:
-            engine['percentage'] = int(engine['avg'] / max_results * 100)
-        else:
-            engine['percentage'] = 0
-
-    for engine in scores:
-        if max_score:
-            engine['percentage'] = int(engine['avg'] / max_score * 100)
-        else:
-            engine['percentage'] = 0
-
-    for engine in scores_per_result:
-        if max_score_per_result:
-            engine['percentage'] = int(engine['avg']
-                                       / max_score_per_result * 100)
-        else:
-            engine['percentage'] = 0
-
-    for engine in errors:
-        if max_errors:
-            engine['percentage'] = int(float(engine['avg']) / max_errors * 100)
-        else:
-            engine['percentage'] = 0
+    pageloads = to_percentage(pageloads, max_pageload)
+    engine_times = to_percentage(engine_times, max_engine_times)
+    results = to_percentage(results, max_results)
+    scores = to_percentage(scores, max_score)
+    scores_per_result = to_percentage(scores_per_result, max_score_per_result)
+    erros = to_percentage(errors, max_errors)
 
     return [
+        (
+            gettext('Engine time (sec)'),
+            sorted(engine_times, key=itemgetter('avg'))
+        ),
         (
             gettext('Page loads (sec)'),
             sorted(pageloads, key=itemgetter('avg'))
@@ -204,10 +214,8 @@ def get_engines_stats():
     ]
 
 
-if 'engines' not in settings or not settings['engines']:
-    logger.error('No engines found. Edit your settings.yml')
-    exit(2)
-
-for engine_data in settings['engines']:
-    engine = load_engine(engine_data)
-    engines[engine.name] = engine
+def initialize_engines(engine_list):
+    for engine_data in engine_list:
+        engine = load_engine(engine_data)
+        if engine is not None:
+            engines[engine.name] = engine

searx/engines/archlinux.py

@@ -0,0 +1,142 @@
+# -*- coding: utf-8 -*-
+
+"""
+ Arch Linux Wiki
+
+ @website      https://wiki.archlinux.org
+ @provide-api  no (Mediawiki provides API, but Arch Wiki blocks access to it
+ @using-api    no
+ @results      HTML
+ @stable       no (HTML can change)
+ @parse        url, title
+"""
+
+from lxml import html
+from searx.engines.xpath import extract_text
+from searx.url_utils import urlencode, urljoin
+
+# engine dependent config
+categories = ['it']
+language_support = True
+paging = True
+base_url = 'https://wiki.archlinux.org'
+
+# xpath queries
+xpath_results = '//ul[@class="mw-search-results"]/li'
+xpath_link = './/div[@class="mw-search-result-heading"]/a'
+
+
+# cut 'en' from 'en_US', 'de' from 'de_CH', and so on
+def locale_to_lang_code(locale):
+    if locale.find('-') >= 0:
+        locale = locale.split('-')[0]
+    return locale
+
+
+# wikis for some languages were moved off from the main site, we need to make
+# requests to correct URLs to be able to get results in those languages
+lang_urls = {
+    'all': {
+        'base': 'https://wiki.archlinux.org',
+        'search': '/index.php?title=Special:Search&offset={offset}&{query}'
+    },
+    'de': {
+        'base': 'https://wiki.archlinux.de',
+        'search': '/index.php?title=Spezial:Suche&offset={offset}&{query}'
+    },
+    'fr': {
+        'base': 'https://wiki.archlinux.fr',
+        'search': '/index.php?title=Spécial:Recherche&offset={offset}&{query}'
+    },
+    'ja': {
+        'base': 'https://wiki.archlinuxjp.org',
+        'search': '/index.php?title=特別:検索&offset={offset}&{query}'
+    },
+    'ro': {
+        'base': 'http://wiki.archlinux.ro',
+        'search': '/index.php?title=Special:Căutare&offset={offset}&{query}'
+    },
+    'tr': {
+        'base': 'http://archtr.org/wiki',
+        'search': '/index.php?title=Özel:Ara&offset={offset}&{query}'
+    }
+}
+
+
+# get base & search URLs for selected language
+def get_lang_urls(language):
+    if language in lang_urls:
+        return lang_urls[language]
+    return lang_urls['all']
+
+
+# Language names to build search requests for
+# those languages which are hosted on the main site.
+main_langs = {
+    'ar': 'العربية',
+    'bg': 'Български',
+    'cs': 'Česky',
+    'da': 'Dansk',
+    'el': 'Ελληνικά',
+    'es': 'Español',
+    'he': 'עברית',
+    'hr': 'Hrvatski',
+    'hu': 'Magyar',
+    'it': 'Italiano',
+    'ko': '한국어',
+    'lt': 'Lietuviškai',
+    'nl': 'Nederlands',
+    'pl': 'Polski',
+    'pt': 'Português',
+    'ru': 'Русский',
+    'sl': 'Slovenský',
+    'th': 'ไทย',
+    'uk': 'Українська',
+    'zh': '简体中文'
+}
+supported_languages = dict(lang_urls, **main_langs)
+
+
+# do search-request
+def request(query, params):
+    # translate the locale (e.g. 'en_US') to language code ('en')
+    language = locale_to_lang_code(params['language'])
+
+    # if our language is hosted on the main site, we need to add its name
+    # to the query in order to narrow the results to that language
+    if language in main_langs:
+        query += '(' + main_langs[language] + ')'
+
+    # prepare the request parameters
+    query = urlencode({'search': query})
+    offset = (params['pageno'] - 1) * 20
+
+    # get request URLs for our language of choice
+    urls = get_lang_urls(language)
+    search_url = urls['base'] + urls['search']
+
+    params['url'] = search_url.format(query=query, offset=offset)
+
+    return params
+
+
+# get response from search-request
+def response(resp):
+    # get the base URL for the language in which request was made
+    language = locale_to_lang_code(resp.search_params['language'])
+    base_url = get_lang_urls(language)['base']
+
+    results = []
+
+    dom = html.fromstring(resp.text)
+
+    # parse results
+    for result in dom.xpath(xpath_results):
+        link = result.xpath(xpath_link)[0]
+        href = urljoin(base_url, link.attrib.get('href'))
+        title = extract_text(link)
+
+        results.append({'url': href,
+                        'title': title})
+
+    return results

searx/engines/base.py

@@ -0,0 +1,121 @@
+#!/usr/bin/env python
+
+"""
+ BASE (Scholar publications)
+
+ @website     https://base-search.net
+ @provide-api yes with authorization (https://api.base-search.net/)
+
+ @using-api   yes
+ @results     XML
+ @stable      ?
+ @parse       url, title, publishedDate, content
+ More info on api: http://base-search.net/about/download/base_interface.pdf
+"""
+
+from lxml import etree
+from datetime import datetime
+import re
+from searx.url_utils import urlencode
+from searx.utils import searx_useragent
+
+
+categories = ['science']
+
+base_url = 'https://api.base-search.net/cgi-bin/BaseHttpSearchInterface.fcgi'\
+           + '?func=PerformSearch&{query}&boost=oa&hits={hits}&offset={offset}'
+
+# engine dependent config
+paging = True
+number_of_results = 10
+
+# shortcuts for advanced search
+shorcut_dict = {
+    # user-friendly keywords
+    'format:': 'dcformat:',
+    'author:': 'dccreator:',
+    'collection:': 'dccollection:',
+    'hdate:': 'dchdate:',
+    'contributor:': 'dccontributor:',
+    'coverage:': 'dccoverage:',
+    'date:': 'dcdate:',
+    'abstract:': 'dcdescription:',
+    'urls:': 'dcidentifier:',
+    'language:': 'dclanguage:',
+    'publisher:': 'dcpublisher:',
+    'relation:': 'dcrelation:',
+    'rights:': 'dcrights:',
+    'source:': 'dcsource:',
+    'subject:': 'dcsubject:',
+    'title:': 'dctitle:',
+    'type:': 'dcdctype:'
+}
+
+
+def request(query, params):
+    # replace shortcuts with API advanced search keywords
+    for key in shorcut_dict.keys():
+        query = re.sub(str(key), str(shorcut_dict[key]), query)
+
+    # basic search
+    offset = (params['pageno'] - 1) * number_of_results
+
+    string_args = dict(query=urlencode({'query': query}),
+                       offset=offset,
+                       hits=number_of_results)
+
+    params['url'] = base_url.format(**string_args)
+
+    params['headers']['User-Agent'] = searx_useragent()
+    return params
+
+
+def response(resp):
+    results = []
+
+    search_results = etree.XML(resp.text)
+
+    for entry in search_results.xpath('./result/doc'):
+        content = "No description available"
+
+        date = datetime.now()  # needed in case no dcdate is available for an item
+        for item in entry:
+            if item.attrib["name"] == "dchdate":
+                harvestDate = item.text
+
+            elif item.attrib["name"] == "dcdate":
+                date = item.text
+
+            elif item.attrib["name"] == "dctitle":
+                title = item.text
+
+            elif item.attrib["name"] == "dclink":
+                url = item.text
+
+            elif item.attrib["name"] == "dcdescription":
+                content = item.text[:300]
+                if len(item.text) > 300:
+                    content += "..."
+
+# dates returned by the BASE API are not several formats
+        publishedDate = None
+        for date_format in ['%Y-%m-%dT%H:%M:%SZ', '%Y-%m-%d', '%Y-%m', '%Y']:
+            try:
+                publishedDate = datetime.strptime(date, date_format)
+                break
+            except:
+                pass
+
+        if publishedDate is not None:
+            res_dict = {'url': url,
+                        'title': title,
+                        'publishedDate': publishedDate,
+                        'content': content}
+        else:
+            res_dict = {'url': url,
+                        'title': title,
+                        'content': content}
+
+        results.append(res_dict)
+
+    return results

searx/engines/bing.py

@@ -13,15 +13,15 @@
  @todo        publishedDate
 """
 
-from urllib import urlencode
-from cgi import escape
 from lxml import html
 from searx.engines.xpath import extract_text
+from searx.url_utils import urlencode
 
 # engine dependent config
 categories = ['general']
 paging = True
 language_support = True
+supported_languages_url = 'https://www.bing.com/account/general'
 
 # search-url
 base_url = 'https://www.bing.com/'
@@ -32,18 +32,17 @@ search_string = 'search?{query}&first={offset}'
 def request(query, params):
     offset = (params['pageno'] - 1) * 10 + 1
 
-    if params['language'] == 'all':
-        language = 'en-US'
+    if params['language'] != 'all':
+        lang = params['language'].split('-')[0].upper()
     else:
-        language = params['language'].replace('_', '-')
+        lang = 'EN'
+
+    query = u'language:{} {}'.format(lang, query.decode('utf-8')).encode('utf-8')
 
     search_path = search_string.format(
-        query=urlencode({'q': query, 'setmkt': language}),
+        query=urlencode({'q': query}),
         offset=offset)
 
-    params['cookies']['SRCHHPGUSR'] = \
-        'NEWWND=0&NRSLT=-1&SRCHLANG=' + language.split('-')[0]
-
     params['url'] = base_url + search_path
     return params
 
@@ -54,28 +53,30 @@ def response(resp):
 
     dom = html.fromstring(resp.text)
 
+    try:
+        results.append({'number_of_results': int(dom.xpath('//span[@class="sb_count"]/text()')[0]
+                                                 .split()[0].replace(',', ''))})
+    except:
+        pass
+
     # parse results
     for result in dom.xpath('//div[@class="sa_cc"]'):
         link = result.xpath('.//h3/a')[0]
         url = link.attrib.get('href')
         title = extract_text(link)
-        content = escape(extract_text(result.xpath('.//p')))
+        content = extract_text(result.xpath('.//p'))
 
         # append result
         results.append({'url': url,
                         'title': title,
                         'content': content})
 
-    # return results if something is found
-    if results:
-        return results
-
     # parse results again if nothing is found yet
     for result in dom.xpath('//li[@class="b_algo"]'):
         link = result.xpath('.//h2/a')[0]
         url = link.attrib.get('href')
         title = extract_text(link)
-        content = escape(extract_text(result.xpath('.//p')))
+        content = extract_text(result.xpath('.//p'))
 
         # append result
         results.append({'url': url,
@@ -84,3 +85,17 @@ def response(resp):
 
     # return results
     return results
+
+
+# get supported languages from their site
+def _fetch_supported_languages(resp):
+    supported_languages = []
+    dom = html.fromstring(resp.text)
+    options = dom.xpath('//div[@id="limit-languages"]//input')
+    for option in options:
+        code = option.xpath('./@id')[0].replace('_', '-')
+        if code == 'nb':
+            code = 'no'
+        supported_languages.append(code)
+
+    return supported_languages

searx/engines/bing_images.py

@@ -15,20 +15,26 @@
               limited response to 10 images
 """
 
-from urllib import urlencode
 from lxml import html
-from yaml import load
+from json import loads
 import re
+from searx.engines.bing import _fetch_supported_languages, supported_languages_url
+from searx.url_utils import urlencode
 
 # engine dependent config
 categories = ['images']
 paging = True
 safesearch = True
+time_range_support = True
 
 # search-url
 base_url = 'https://www.bing.com/'
 search_string = 'images/search?{query}&count=10&first={offset}'
-thumb_url = "https://www.bing.com/th?id={ihk}"
+time_range_string = '&qft=+filterui:age-lt{interval}'
+time_range_dict = {'day': '1440',
+                   'week': '10080',
+                   'month': '43200',
+                   'year': '525600'}
 
 # safesearch definitions
 safesearch_types = {2: 'STRICT',
@@ -36,6 +42,9 @@ safesearch_types = {2: 'STRICT',
                     0: 'OFF'}
 
 
+_quote_keys_regex = re.compile('({|,)([a-z][a-z0-9]*):(")', re.I | re.U)
+
+
 # do search-request
 def request(query, params):
     offset = (params['pageno'] - 1) * 10 + 1
@@ -44,7 +53,7 @@ def request(query, params):
     if params['language'] == 'all':
         language = 'en-US'
     else:
-        language = params['language'].replace('_', '-')
+        language = params['language']
 
     search_path = search_string.format(
         query=urlencode({'q': query}),
@@ -55,6 +64,8 @@ def request(query, params):
         '&ADLT=' + safesearch_types.get(params['safesearch'], 'DEMOTE')
 
     params['url'] = base_url + search_path
+    if params['time_range'] in time_range_dict:
+        params['url'] += time_range_string.format(interval=time_range_dict[params['time_range']])
 
     return params
 
@@ -65,34 +76,33 @@ def response(resp):
 
     dom = html.fromstring(resp.text)
 
-    # init regex for yaml-parsing
-    p = re.compile('({|,)([a-z]+):(")')
-
     # parse results
-    for result in dom.xpath('//div[@class="dg_u"]'):
+    for result in dom.xpath('//div[@id="mmComponent_images_1"]/ul/li/div/div[@class="imgpt"]'):
         link = result.xpath('./a')[0]
 
-        # parse yaml-data (it is required to add a space, to make it parsable)
-        yaml_data = load(p.sub(r'\1\2: \3', link.attrib.get('m')))
+        # TODO find actual title
+        title = link.xpath('.//img/@alt')[0]
 
-        title = link.attrib.get('t1')
-        ihk = link.attrib.get('ihk')
+        # parse json-data (it is required to add a space, to make it parsable)
+        json_data = loads(_quote_keys_regex.sub(r'\1"\2": \3', link.attrib.get('m')))
 
-        # url = 'http://' + link.attrib.get('t3')
-        url = yaml_data.get('surl')
-        img_src = yaml_data.get('imgurl')
+        url = json_data.get('purl')
+        img_src = json_data.get('murl')
+
+        thumb_json_data = loads(_quote_keys_regex.sub(r'\1"\2": \3', link.attrib.get('mad')))
+        thumbnail = thumb_json_data.get('turl')
 
         # append result
         results.append({'template': 'images.html',
                         'url': url,
                         'title': title,
                         'content': '',
-                        'thumbnail_src': thumb_url.format(ihk=ihk),
+                        'thumbnail_src': thumbnail,
                         'img_src': img_src})
 
         # TODO stop parsing if 10 images are found
-        if len(results) >= 10:
-            break
+        # if len(results) >= 10:
+        #     break
 
     # return results
     return results

searx/engines/bing_news.py

@@ -11,21 +11,26 @@
  @parse       url, title, content, publishedDate, thumbnail
 """
 
-from urllib import urlencode
-from urlparse import urlparse, parse_qsl
 from datetime import datetime
 from dateutil import parser
 from lxml import etree
 from searx.utils import list_get
+from searx.engines.bing import _fetch_supported_languages, supported_languages_url
+from searx.url_utils import urlencode, urlparse, parse_qsl
 
 # engine dependent config
 categories = ['news']
 paging = True
 language_support = True
+time_range_support = True
 
 # search-url
 base_url = 'https://www.bing.com/'
 search_string = 'news/search?{query}&first={offset}&format=RSS'
+search_string_with_time = 'news/search?{query}&first={offset}&qft=interval%3d"{interval}"&format=RSS'
+time_range_dict = {'day': '7',
+                   'week': '8',
+                   'month': '9'}
 
 
 # remove click
@@ -46,20 +51,32 @@ def image_url_cleanup(url_string):
     return url_string
 
 
+def _get_url(query, language, offset, time_range):
+    if time_range in time_range_dict:
+        search_path = search_string_with_time.format(
+            query=urlencode({'q': query, 'setmkt': language}),
+            offset=offset,
+            interval=time_range_dict[time_range])
+    else:
+        search_path = search_string.format(
+            query=urlencode({'q': query, 'setmkt': language}),
+            offset=offset)
+    return base_url + search_path
+
+
 # do search-request
 def request(query, params):
+    if params['time_range'] and params['time_range'] not in time_range_dict:
+        return params
+
     offset = (params['pageno'] - 1) * 10 + 1
 
     if params['language'] == 'all':
         language = 'en-US'
     else:
-        language = params['language'].replace('_', '-')
+        language = params['language']
 
-    search_path = search_string.format(
-        query=urlencode({'q': query, 'setmkt': language}),
-        offset=offset)
-
-    params['url'] = base_url + search_path
+    params['url'] = _get_url(query, language, offset, params['time_range'])
 
     return params
 
@@ -95,12 +112,11 @@ def response(resp):
 
         # append result
         if thumbnail is not None:
-            results.append({'template': 'videos.html',
-                            'url': url,
+            results.append({'url': url,
                             'title': title,
                             'publishedDate': publishedDate,
                             'content': content,
-                            'thumbnail': thumbnail})
+                            'img_src': thumbnail})
         else:
             results.append({'url': url,
                             'title': title,

searx/engines/blekko_images.py

@@ -11,7 +11,7 @@
 """
 
 from json import loads
-from urllib import urlencode
+from searx.url_utils import urlencode
 
 # engine dependent config
 categories = ['images']
@@ -37,7 +37,7 @@ def request(query, params):
                           c=c)
 
     if params['pageno'] != 1:
-        params['url'] += '&page={pageno}'.format(pageno=(params['pageno']-1))
+        params['url'] += '&page={pageno}'.format(pageno=(params['pageno'] - 1))
 
     # let Blekko know we wan't have profiling
     params['cookies']['tag_lesslogging'] = '1'

searx/engines/btdigg.py

@@ -10,12 +10,11 @@
  @parse       url, title, content, seed, leech, magnetlink
 """
 
-from urlparse import urljoin
-from cgi import escape
-from urllib import quote
 from lxml import html
 from operator import itemgetter
 from searx.engines.xpath import extract_text
+from searx.url_utils import quote, urljoin
+from searx.utils import get_torrent_size
 
 # engine dependent config
 categories = ['videos', 'music', 'files']
@@ -29,7 +28,7 @@ search_url = url + '/search?q={search_term}&p={pageno}'
 # do search-request
 def request(query, params):
     params['url'] = search_url.format(search_term=quote(query),
-                                      pageno=params['pageno']-1)
+                                      pageno=params['pageno'] - 1)
 
     return params
 
@@ -38,7 +37,7 @@ def request(query, params):
 def response(resp):
     results = []
 
-    dom = html.fromstring(resp.content)
+    dom = html.fromstring(resp.text)
 
     search_res = dom.xpath('//div[@id="search_res"]/table/tr')
 
@@ -50,8 +49,8 @@ def response(resp):
     for result in search_res:
         link = result.xpath('.//td[@class="torrent_name"]//a')[0]
         href = urljoin(url, link.attrib.get('href'))
-        title = escape(extract_text(link))
-        content = escape(extract_text(result.xpath('.//pre[@class="snippet"]')[0]))
+        title = extract_text(link)
+        content = extract_text(result.xpath('.//pre[@class="snippet"]')[0])
         content = "<br />".join(content.split("\n"))
 
         filesize = result.xpath('.//span[@class="attr_val"]/text()')[0].split()[0]
@@ -68,20 +67,7 @@ def response(resp):
         leech = 0
 
         # convert filesize to byte if possible
-        try:
-            filesize = float(filesize)
-
-            # convert filesize to byte
-            if filesize_multiplier == 'TB':
-                filesize = int(filesize * 1024 * 1024 * 1024 * 1024)
-            elif filesize_multiplier == 'GB':
-                filesize = int(filesize * 1024 * 1024 * 1024)
-            elif filesize_multiplier == 'MB':
-                filesize = int(filesize * 1024 * 1024)
-            elif filesize_multiplier == 'KB':
-                filesize = int(filesize * 1024)
-        except:
-            filesize = None
+        filesize = get_torrent_size(filesize, filesize_multiplier)
 
         # convert files to int if possible
         if files.isdigit():

searx/engines/currency_convert.py

@@ -1,21 +1,25 @@
-from datetime import datetime
+import json
 import re
 import os
-import json
+import sys
 import unicodedata
 
+from datetime import datetime
+
+if sys.version_info[0] == 3:
+    unicode = str
 
 categories = []
 url = 'https://download.finance.yahoo.com/d/quotes.csv?e=.csv&f=sl1d1t1&s={query}=X'
 weight = 100
 
-parser_re = re.compile(u'^\W*(\d+(?:\.\d+)?)\W*([^.0-9].+)\W+in?\W+([^\.]+)\W*$', re.I)  # noqa
+parser_re = re.compile(b'.*?(\\d+(?:\\.\\d+)?) ([^.0-9]+) (?:in|to) ([^.0-9]+)', re.I)
 
 db = 1
 
 
 def normalize_name(name):
-    name = name.lower().replace('-', ' ')
+    name = name.decode('utf-8').lower().replace('-', ' ').rstrip('s')
     name = re.sub(' +', ' ', name)
     return unicodedata.normalize('NFKD', name).lower()
 
@@ -35,7 +39,7 @@ def iso4217_to_name(iso4217, language):
 
 
 def request(query, params):
-    m = parser_re.match(unicode(query, 'utf8'))
+    m = parser_re.match(query)
     if not m:
         # wrong query
         return params

searx/engines/dailymotion.py

@@ -12,10 +12,9 @@
  @todo        set content-parameter with correct data
 """
 
-from urllib import urlencode
 from json import loads
-from cgi import escape
 from datetime import datetime
+from searx.url_utils import urlencode
 
 # engine dependent config
 categories = ['videos']
@@ -28,6 +27,8 @@ search_url = 'https://api.dailymotion.com/videos?fields=created_time,title,descr
 embedded_url = '<iframe frameborder="0" width="540" height="304" ' +\
     'data-src="//www.dailymotion.com/embed/video/{videoid}" allowfullscreen></iframe>'
 
+supported_languages_url = 'https://api.dailymotion.com/languages'
+
 
 # do search-request
 def request(query, params):
@@ -57,7 +58,7 @@ def response(resp):
     for res in search_res['list']:
         title = res['title']
         url = res['url']
-        content = escape(res['description'])
+        content = res['description']
         thumbnail = res['thumbnail_360_url']
         publishedDate = datetime.fromtimestamp(res['created_time'], None)
         embedded = embedded_url.format(videoid=res['id'])
@@ -75,3 +76,22 @@ def response(resp):
 
     # return results
     return results
+
+
+# get supported languages from their site
+def _fetch_supported_languages(resp):
+    supported_languages = {}
+
+    response_json = loads(resp.text)
+
+    for language in response_json['list']:
+        supported_languages[language['code']] = {}
+
+        name = language['native_name']
+        if name:
+            supported_languages[language['code']]['name'] = name
+        english_name = language['name']
+        if english_name:
+            supported_languages[language['code']]['english_name'] = english_name
+
+    return supported_languages

searx/engines/deezer.py

@@ -11,7 +11,7 @@
 """
 
 from json import loads
-from urllib import urlencode
+from searx.url_utils import urlencode
 
 # engine dependent config
 categories = ['music']
@@ -30,8 +30,7 @@ embedded_url = '<iframe scrolling="no" frameborder="0" allowTransparency="true"
 def request(query, params):
     offset = (params['pageno'] - 1) * 25
 
-    params['url'] = search_url.format(query=urlencode({'q': query}),
-                                      offset=offset)
+    params['url'] = search_url.format(query=urlencode({'q': query}), offset=offset)
 
     return params
 
@@ -51,10 +50,11 @@ def response(resp):
             if url.startswith('http://'):
                 url = 'https' + url[4:]
 
-            content = result['artist']['name'] +\
-                " &bull; " +\
-                result['album']['title'] +\
-                " &bull; " + result['title']
+            content = u'{} - {} - {}'.format(
+                result['artist']['name'],
+                result['album']['title'],
+                result['title'])
+
             embedded = embedded_url.format(audioid=result['id'])
 
             # append result

searx/engines/deviantart.py

@@ -12,27 +12,37 @@
  @todo        rewrite to api
 """
 
-from urllib import urlencode
-from urlparse import urljoin
 from lxml import html
 import re
 from searx.engines.xpath import extract_text
+from searx.url_utils import urlencode
 
 # engine dependent config
 categories = ['images']
 paging = True
+time_range_support = True
 
 # search-url
 base_url = 'https://www.deviantart.com/'
-search_url = base_url+'browse/all/?offset={offset}&{query}'
+search_url = base_url + 'browse/all/?offset={offset}&{query}'
+time_range_url = '&order={range}'
+
+time_range_dict = {'day': 11,
+                   'week': 14,
+                   'month': 15}
 
 
 # do search-request
 def request(query, params):
+    if params['time_range'] and params['time_range'] not in time_range_dict:
+        return params
+
     offset = (params['pageno'] - 1) * 24
 
     params['url'] = search_url.format(offset=offset,
                                       query=urlencode({'q': query}))
+    if params['time_range'] in time_range_dict:
+        params['url'] += time_range_url.format(range=time_range_dict[params['time_range']])
 
     return params
 
@@ -47,14 +57,13 @@ def response(resp):
 
     dom = html.fromstring(resp.text)
 
-    regex = re.compile('\/200H\/')
+    regex = re.compile(r'\/200H\/')
 
     # parse results
-    for result in dom.xpath('//div[contains(@class, "tt-a tt-fh")]'):
-        link = result.xpath('.//a[contains(@class, "thumb")]')[0]
-        url = urljoin(base_url, link.attrib.get('href'))
-        title_links = result.xpath('.//span[@class="details"]//a[contains(@class, "t")]')
-        title = extract_text(title_links[0])
+    for result in dom.xpath('.//span[@class="thumb wide"]'):
+        link = result.xpath('.//a[@class="torpedo-thumb-link"]')[0]
+        url = link.attrib.get('href')
+        title = extract_text(result.xpath('.//span[@class="title"]'))
         thumbnail_src = link.xpath('.//img')[0].attrib.get('src')
         img_src = regex.sub('/', thumbnail_src)
 

searx/engines/dictzone.py

@@ -0,0 +1,68 @@
+"""
+ Dictzone
+
+ @website     https://dictzone.com/
+ @provide-api no
+ @using-api   no
+ @results     HTML (using search portal)
+ @stable      no (HTML can change)
+ @parse       url, title, content
+"""
+
+import re
+from lxml import html
+from searx.utils import is_valid_lang
+from searx.url_utils import urljoin
+
+categories = ['general']
+url = u'http://dictzone.com/{from_lang}-{to_lang}-dictionary/{query}'
+weight = 100
+
+parser_re = re.compile(b'.*?([a-z]+)-([a-z]+) ([^ ]+)$', re.I)
+results_xpath = './/table[@id="r"]/tr'
+
+
+def request(query, params):
+    m = parser_re.match(query)
+    if not m:
+        return params
+
+    from_lang, to_lang, query = m.groups()
+
+    from_lang = is_valid_lang(from_lang)
+    to_lang = is_valid_lang(to_lang)
+
+    if not from_lang or not to_lang:
+        return params
+
+    params['url'] = url.format(from_lang=from_lang[2],
+                               to_lang=to_lang[2],
+                               query=query)
+
+    return params
+
+
+def response(resp):
+    results = []
+
+    dom = html.fromstring(resp.text)
+
+    for k, result in enumerate(dom.xpath(results_xpath)[1:]):
+        try:
+            from_result, to_results_raw = result.xpath('./td')
+        except:
+            continue
+
+        to_results = []
+        for to_result in to_results_raw.xpath('./p/a'):
+            t = to_result.text_content()
+            if t.strip():
+                to_results.append(to_result.text_content())
+
+        results.append({
+            'url': urljoin(resp.url, '?%d' % k),
+            'title': from_result.text_content(),
+            'content': '; '.join(to_results)
+        })
+
+    return results

searx/engines/digbt.py

@@ -0,0 +1,62 @@
+"""
+ DigBT (Videos, Music, Files)
+
+ @website     https://digbt.org
+ @provide-api no
+
+ @using-api   no
+ @results     HTML (using search portal)
+ @stable      no (HTML can change)
+ @parse       url, title, content, magnetlink
+"""
+
+from sys import version_info
+from lxml import html
+from searx.engines.xpath import extract_text
+from searx.utils import get_torrent_size
+from searx.url_utils import urljoin
+
+if version_info[0] == 3:
+    unicode = str
+
+categories = ['videos', 'music', 'files']
+paging = True
+
+URL = 'https://digbt.org'
+SEARCH_URL = URL + '/search/{query}-time-{pageno}'
+FILESIZE = 3
+FILESIZE_MULTIPLIER = 4
+
+
+def request(query, params):
+    params['url'] = SEARCH_URL.format(query=query, pageno=params['pageno'])
+
+    return params
+
+
+def response(resp):
+    dom = html.fromstring(resp.text)
+    search_res = dom.xpath('.//td[@class="x-item"]')
+
+    if not search_res:
+        return list()
+
+    results = list()
+    for result in search_res:
+        url = urljoin(URL, result.xpath('.//a[@title]/@href')[0])
+        title = extract_text(result.xpath('.//a[@title]'))
+        content = extract_text(result.xpath('.//div[@class="files"]'))
+        files_data = extract_text(result.xpath('.//div[@class="tail"]')).split()
+        filesize = get_torrent_size(files_data[FILESIZE], files_data[FILESIZE_MULTIPLIER])
+        magnetlink = result.xpath('.//div[@class="tail"]//a[@class="title"]/@href')[0]
+
+        results.append({'url': url,
+                        'title': title,
+                        'content': content,
+                        'filesize': filesize,
+                        'magnetlink': magnetlink,
+                        'seed': 'N/A',
+                        'leech': 'N/A',
+                        'template': 'torrent.html'})
+
+    return results

searx/engines/digg.py

@@ -10,11 +10,10 @@
  @parse       url, title, content, publishedDate, thumbnail
 """
 
-from urllib import quote_plus
+from dateutil import parser
 from json import loads
 from lxml import html
-from cgi import escape
-from dateutil import parser
+from searx.url_utils import quote_plus
 
 # engine dependent config
 categories = ['news', 'social media']
@@ -22,7 +21,7 @@ paging = True
 
 # search-url
 base_url = 'https://digg.com/'
-search_url = base_url+'api/search/{query}.json?position={position}&format=html'
+search_url = base_url + 'api/search/{query}.json?position={position}&format=html'
 
 # specific xpath variables
 results_xpath = '//article'
@@ -56,7 +55,7 @@ def response(resp):
         url = result.attrib.get('data-contenturl')
         thumbnail = result.xpath('.//img')[0].attrib.get('src')
         title = ''.join(result.xpath(title_xpath))
-        content = escape(''.join(result.xpath(content_xpath)))
+        content = ''.join(result.xpath(content_xpath))
         pubdate = result.xpath(pubdate_xpath)[0].attrib.get('datetime')
         publishedDate = parser.parse(pubdate)
 

searx/engines/doku.py

@@ -0,0 +1,84 @@
+# Doku Wiki
+#
+# @website     https://www.dokuwiki.org/
+# @provide-api yes
+#              (https://www.dokuwiki.org/devel:xmlrpc)
+#
+# @using-api   no
+# @results     HTML
+# @stable      yes
+# @parse       (general)    url, title, content
+
+from lxml.html import fromstring
+from searx.engines.xpath import extract_text
+from searx.url_utils import urlencode
+
+# engine dependent config
+categories = ['general']  # TODO , 'images', 'music', 'videos', 'files'
+paging = False
+language_support = False
+number_of_results = 5
+
+# search-url
+# Doku is OpenSearch compatible
+base_url = 'http://localhost:8090'
+search_url = '/?do=search'\
+             '&{query}'
+# TODO             '&startRecord={offset}'\
+# TODO             '&maximumRecords={limit}'\
+
+
+# do search-request
+def request(query, params):
+
+    params['url'] = base_url +\
+        search_url.format(query=urlencode({'id': query}))
+
+    return params
+
+
+# get response from search-request
+def response(resp):
+    results = []
+
+    doc = fromstring(resp.text)
+
+    # parse results
+    # Quickhits
+    for r in doc.xpath('//div[@class="search_quickresult"]/ul/li'):
+        try:
+            res_url = r.xpath('.//a[@class="wikilink1"]/@href')[-1]
+        except:
+            continue
+
+        if not res_url:
+            continue
+
+        title = extract_text(r.xpath('.//a[@class="wikilink1"]/@title'))
+
+        # append result
+        results.append({'title': title,
+                        'content': "",
+                        'url': base_url + res_url})
+
+    # Search results
+    for r in doc.xpath('//dl[@class="search_results"]/*'):
+        try:
+            if r.tag == "dt":
+                res_url = r.xpath('.//a[@class="wikilink1"]/@href')[-1]
+                title = extract_text(r.xpath('.//a[@class="wikilink1"]/@title'))
+            elif r.tag == "dd":
+                content = extract_text(r.xpath('.'))
+
+                # append result
+                results.append({'title': title,
+                                'content': content,
+                                'url': base_url + res_url})
+        except:
+            continue
+
+        if not res_url:
+            continue
+
+    # return results
+    return results

searx/engines/duckduckgo.py

@@ -11,41 +11,86 @@
  @parse       url, title, content
 
  @todo        rewrite to api
- @todo        language support
-              (the current used site does not support language-change)
 """
 
-from urllib import urlencode
 from lxml.html import fromstring
+from requests import get
+from json import loads
 from searx.engines.xpath import extract_text
+from searx.url_utils import urlencode
 
 # engine dependent config
 categories = ['general']
 paging = True
 language_support = True
+supported_languages_url = 'https://duckduckgo.com/d2030.js'
+time_range_support = True
 
 # search-url
-url = 'https://duckduckgo.com/html?{query}&s={offset}'
+url = 'https://duckduckgo.com/html?{query}&s={offset}&api=/d.js&o=json&dc={dc_param}'
+time_range_url = '&df={range}'
+
+time_range_dict = {'day': 'd',
+                   'week': 'w',
+                   'month': 'm'}
 
 # specific xpath variables
-result_xpath = '//div[@class="results_links results_links_deep web-result"]'  # noqa
-url_xpath = './/a[@class="large"]/@href'
-title_xpath = './/a[@class="large"]'
-content_xpath = './/div[@class="snippet"]'
+result_xpath = '//div[@class="result results_links results_links_deep web-result "]'  # noqa
+url_xpath = './/a[@class="result__a"]/@href'
+title_xpath = './/a[@class="result__a"]'
+content_xpath = './/a[@class="result__snippet"]'
+
+
+# match query's language to a region code that duckduckgo will accept
+def get_region_code(lang):
+    # custom fixes for languages
+    if lang == 'all':
+        region_code = None
+    elif lang[:2] == 'ja':
+        region_code = 'jp-jp'
+    elif lang[:2] == 'sl':
+        region_code = 'sl-sl'
+    elif lang == 'zh-TW':
+        region_code = 'tw-tzh'
+    elif lang == 'zh-HK':
+        region_code = 'hk-tzh'
+    elif lang[-2:] == 'SA':
+        region_code = 'xa-' + lang.split('-')[0]
+    elif lang[-2:] == 'GB':
+        region_code = 'uk-' + lang.split('-')[0]
+    else:
+        region_code = lang.split('-')
+        if len(region_code) == 2:
+            # country code goes first
+            region_code = region_code[1].lower() + '-' + region_code[0].lower()
+        else:
+            # tries to get a country code from language
+            region_code = region_code[0].lower()
+            for lc in supported_languages:
+                lc = lc.split('-')
+                if region_code == lc[0]:
+                    region_code = lc[1].lower() + '-' + lc[0].lower()
+                    break
+    return region_code
 
 
 # do search-request
 def request(query, params):
+    if params['time_range'] and params['time_range'] not in time_range_dict:
+        return params
+
     offset = (params['pageno'] - 1) * 30
 
-    if params['language'] == 'all':
-        locale = 'en-us'
+    region_code = get_region_code(params['language'])
+    if region_code:
+        params['url'] = url.format(
+            query=urlencode({'q': query, 'kl': region_code}), offset=offset, dc_param=offset)
     else:
-        locale = params['language'].replace('_', '-').lower()
+        params['url'] = url.format(
+            query=urlencode({'q': query}), offset=offset, dc_param=offset)
 
-    params['url'] = url.format(
-        query=urlencode({'q': query, 'kl': locale}),
-        offset=offset)
+    if params['time_range'] in time_range_dict:
+        params['url'] += time_range_url.format(range=time_range_dict[params['time_range']])
 
     return params
 
@@ -76,3 +121,17 @@ def response(resp):
 
     # return results
     return results
+
+
+# get supported languages from their site
+def _fetch_supported_languages(resp):
+
+    # response is a js file with regions as an embedded object
+    response_page = resp.text
+    response_page = response_page[response_page.find('regions:{') + 8:]
+    response_page = response_page[:response_page.find('}') + 1]
+
+    regions_json = loads(response_page)
+    supported_languages = map((lambda x: x[3:] + '-' + x[:2].upper()), regions_json.keys())
+
+    return supported_languages

searx/engines/duckduckgo_definitions.py

@@ -1,12 +1,16 @@
 import json
-from urllib import urlencode
 from lxml import html
-from searx.utils import html_to_text
+from re import compile
 from searx.engines.xpath import extract_text
+from searx.engines.duckduckgo import _fetch_supported_languages, supported_languages_url
+from searx.url_utils import urlencode
+from searx.utils import html_to_text
 
 url = 'https://api.duckduckgo.com/'\
     + '?{query}&format=json&pretty=0&no_redirect=1&d=1'
 
+http_regex = compile(r'^http:')
+
 
 def result_to_text(url, text, htmlResult):
     # TODO : remove result ending with "Meaning" or "Category"
@@ -19,8 +23,8 @@ def result_to_text(url, text, htmlResult):
 
 
 def request(query, params):
-    # TODO add kl={locale}
     params['url'] = url.format(query=urlencode({'q': query}))
+    params['headers']['Accept-Language'] = params['language'].split('-')[0]
     return params
 
 
@@ -103,6 +107,10 @@ def response(resp):
         urls.append({'title': search_res.get('DefinitionSource'),
                     'url': definitionURL})
 
+    # to merge with wikidata's infobox
+    if infobox_id:
+        infobox_id = http_regex.sub('https:', infobox_id)
+
     # entity
     entity = search_res.get('Entity', None)
     # TODO continent / country / department / location / waterfall /

searx/engines/duckduckgo_images.py

@@ -0,0 +1,91 @@
+"""
+ DuckDuckGo (Images)
+
+ @website     https://duckduckgo.com/
+ @provide-api yes (https://duckduckgo.com/api),
+              but images are not supported
+
+ @using-api   no
+ @results     JSON (site requires js to get images)
+ @stable      no (JSON can change)
+ @parse       url, title, img_src
+
+ @todo        avoid extra request
+"""
+
+from requests import get
+from json import loads
+from searx.engines.xpath import extract_text
+from searx.engines.duckduckgo import _fetch_supported_languages, supported_languages_url, get_region_code
+from searx.url_utils import urlencode
+
+# engine dependent config
+categories = ['images']
+paging = True
+language_support = True
+safesearch = True
+
+# search-url
+images_url = 'https://duckduckgo.com/i.js?{query}&s={offset}&p={safesearch}&o=json&vqd={vqd}'
+site_url = 'https://duckduckgo.com/?{query}&iar=images&iax=1&ia=images'
+
+
+# run query in site to get vqd number needed for requesting images
+# TODO: find a way to get this number without an extra request (is it a hash of the query?)
+def get_vqd(query):
+    res = get(site_url.format(query=urlencode({'q': query})))
+    content = res.text
+    vqd = content[content.find('vqd=\'') + 5:]
+    vqd = vqd[:vqd.find('\'')]
+    return vqd
+
+
+# do search-request
+def request(query, params):
+    # to avoid running actual external requests when testing
+    if 'is_test' not in params:
+        vqd = get_vqd(query)
+    else:
+        vqd = '12345'
+
+    offset = (params['pageno'] - 1) * 50
+
+    safesearch = params['safesearch'] - 1
+
+    region_code = get_region_code(params['language'])
+    if region_code:
+        params['url'] = images_url.format(
+            query=urlencode({'q': query, 'l': region_code}), offset=offset, safesearch=safesearch, vqd=vqd)
+    else:
+        params['url'] = images_url.format(
+            query=urlencode({'q': query}), offset=offset, safesearch=safesearch, vqd=vqd)
+
+    return params
+
+
+# get response from search-request
+def response(resp):
+    results = []
+
+    content = resp.text
+    try:
+        res_json = loads(content)
+    except:
+        return []
+
+    # parse results
+    for result in res_json['results']:
+        title = result['title']
+        url = result['url']
+        thumbnail = result['thumbnail']
+        image = result['image']
+
+        # append result
+        results.append({'template': 'images.html',
+                        'title': title,
+                        'content': '',
+                        'thumbnail_src': thumbnail,
+                        'img_src': image,
+                        'url': url})
+
+    return results

searx/engines/faroo.py

@@ -10,10 +10,10 @@
  @parse       url, title, content, publishedDate, img_src
 """
 
-from urllib import urlencode
 from json import loads
 import datetime
 from searx.utils import searx_useragent
+from searx.url_utils import urlencode
 
 # engine dependent config
 categories = ['general', 'news']
@@ -88,7 +88,7 @@ def response(resp):
     for result in search_res['results']:
         if result['news']:
             # timestamp (milliseconds since 1970)
-            publishedDate = datetime.datetime.fromtimestamp(result['date']/1000.0)  # noqa
+            publishedDate = datetime.datetime.fromtimestamp(result['date'] / 1000.0)  # noqa
 
             # append news result
             results.append({'url': result['url'],

searx/engines/fdroid.py

@@ -0,0 +1,51 @@
+"""
+ F-Droid (a repository of FOSS applications for Android)
+
+ @website      https://f-droid.org/
+ @provide-api  no
+ @using-api    no
+ @results      HTML
+ @stable       no (HTML can change)
+ @parse        url, title, content
+"""
+
+from lxml import html
+from searx.engines.xpath import extract_text
+from searx.url_utils import urlencode
+
+# engine dependent config
+categories = ['files']
+paging = True
+
+# search-url
+base_url = 'https://f-droid.org/'
+search_url = base_url + 'repository/browse/?{query}'
+
+
+# do search-request
+def request(query, params):
+    query = urlencode({'fdfilter': query, 'fdpage': params['pageno']})
+    params['url'] = search_url.format(query=query)
+    return params
+
+
+# get response from search-request
+def response(resp):
+    results = []
+
+    dom = html.fromstring(resp.text)
+
+    for app in dom.xpath('//div[@id="appheader"]'):
+        url = app.xpath('./ancestor::a/@href')[0]
+        title = app.xpath('./p/span/text()')[0]
+        img_src = app.xpath('.//img/@src')[0]
+
+        content = extract_text(app.xpath('./p')[0])
+        content = content.replace(title, '', 1).strip()
+
+        results.append({'url': url,
+                        'title': title,
+                        'content': content,
+                        'img_src': img_src})
+
+    return results

searx/engines/filecrop.py

@@ -1,5 +1,9 @@
-from urllib import urlencode
-from HTMLParser import HTMLParser
+from searx.url_utils import urlencode
+
+try:
+    from HTMLParser import HTMLParser
+except:
+    from html.parser import HTMLParser
 
 url = 'http://www.filecrop.com/'
 search_url = url + '/search.php?{query}&size_i=0&size_f=100000000&engine_r=1&engine_d=1&engine_e=1&engine_4=1&engine_m=1&pos={index}'  # noqa
@@ -8,6 +12,7 @@ paging = True
 
 
 class FilecropResultParser(HTMLParser):
+
     def __init__(self):
         HTMLParser.__init__(self)
         self.__start_processing = False
@@ -72,8 +77,7 @@ class FilecropResultParser(HTMLParser):
 
 def request(query, params):
     index = 1 + (params['pageno'] - 1) * 30
-    params['url'] = search_url.format(query=urlencode({'w': query}),
-                                      index=index)
+    params['url'] = search_url.format(query=urlencode({'w': query}), index=index)
     return params
 
 

searx/engines/flickr.py

@@ -13,8 +13,8 @@
  More info on api-key : https://www.flickr.com/services/apps/create/
 """
 
-from urllib import urlencode
 from json import loads
+from searx.url_utils import urlencode
 
 categories = ['images']
 
@@ -77,21 +77,13 @@ def response(resp):
 
         url = build_flickr_url(photo['owner'], photo['id'])
 
-        title = photo['title']
-
-        content = '<span class="photo-author">' +\
-                  photo['ownername'] +\
-                  '</span><br />' +\
-                  '<span class="description">' +\
-                  photo['description']['_content'] +\
-                  '</span>'
-
         # append result
         results.append({'url': url,
-                        'title': title,
+                        'title': photo['title'],
                         'img_src': img_src,
                         'thumbnail_src': thumbnail_src,
-                        'content': content,
+                        'content': photo['description']['_content'],
+                        'author': photo['ownername'],
                         'template': 'images.html'})
 
     # return results

searx/engines/flickr_noapi.py

@@ -12,10 +12,11 @@
  @parse       url, title, thumbnail, img_src
 """
 
-from urllib import urlencode
 from json import loads
+from time import time
 import re
 from searx.engines import logger
+from searx.url_utils import urlencode
 
 
 logger = logger.getChild('flickr-noapi')
@@ -24,21 +25,32 @@ categories = ['images']
 
 url = 'https://www.flickr.com/'
 search_url = url + 'search?{query}&page={page}'
+time_range_url = '&min_upload_date={start}&max_upload_date={end}'
 photo_url = 'https://www.flickr.com/photos/{userid}/{photoid}'
 regex = re.compile(r"\"search-photos-lite-models\",\"photos\":(.*}),\"totalItems\":", re.DOTALL)
 image_sizes = ('o', 'k', 'h', 'b', 'c', 'z', 'n', 'm', 't', 'q', 's')
 
 paging = True
+time_range_support = True
+time_range_dict = {'day': 60 * 60 * 24,
+                   'week': 60 * 60 * 24 * 7,
+                   'month': 60 * 60 * 24 * 7 * 4,
+                   'year': 60 * 60 * 24 * 7 * 52}
 
 
 def build_flickr_url(user_id, photo_id):
     return photo_url.format(userid=user_id, photoid=photo_id)
 
 
-def request(query, params):
-    params['url'] = search_url.format(query=urlencode({'text': query}),
-                                      page=params['pageno'])
+def _get_time_range_url(time_range):
+    if time_range in time_range_dict:
+        return time_range_url.format(start=time(), end=str(int(time()) - time_range_dict[time_range]))
+    return ''
 
+
+def request(query, params):
+    params['url'] = (search_url.format(query=urlencode({'text': query}), page=params['pageno'])
+                     + _get_time_range_url(params['time_range']))
     return params
 
 
@@ -91,16 +103,15 @@ def response(resp):
 
         title = photo.get('title', '')
 
-        content = '<span class="photo-author">' +\
-                  photo['username'] +\
-                  '</span><br />'
+        author = photo['username']
 
         # append result
         results.append({'url': url,
                         'title': title,
                         'img_src': img_src,
                         'thumbnail_src': thumbnail_src,
-                        'content': content,
+                        'content': '',
+                        'author': author,
                         'template': 'images.html'})
 
     return results

searx/engines/framalibre.py

@@ -0,0 +1,69 @@
+"""
+ FramaLibre (It)
+
+ @website     https://framalibre.org/
+ @provide-api no
+
+ @using-api   no
+ @results     HTML
+ @stable      no (HTML can change)
+ @parse       url, title, content, thumbnail, img_src
+"""
+
+from cgi import escape
+from lxml import html
+from searx.engines.xpath import extract_text
+from searx.url_utils import urljoin, urlencode
+
+# engine dependent config
+categories = ['it']
+paging = True
+
+# search-url
+base_url = 'https://framalibre.org/'
+search_url = base_url + 'recherche-par-crit-res?{query}&page={offset}'
+
+# specific xpath variables
+results_xpath = '//div[@class="nodes-list-row"]/div[contains(@typeof,"sioc:Item")]'
+link_xpath = './/h3[@class="node-title"]/a[@href]'
+thumbnail_xpath = './/img[@class="media-object img-responsive"]/@src'
+content_xpath = './/div[@class="content"]//p'
+
+
+# do search-request
+def request(query, params):
+    offset = (params['pageno'] - 1)
+    params['url'] = search_url.format(query=urlencode({'keys': query}),
+                                      offset=offset)
+
+    return params
+
+
+# get response from search-request
+def response(resp):
+    results = []
+
+    dom = html.fromstring(resp.text)
+
+    # parse results
+    for result in dom.xpath(results_xpath):
+        link = result.xpath(link_xpath)[0]
+        href = urljoin(base_url, link.attrib.get('href'))
+        # there's also a span (class="rdf-meta element-hidden" property="dc:title")'s content property for this...
+        title = escape(extract_text(link))
+        thumbnail_tags = result.xpath(thumbnail_xpath)
+        thumbnail = None
+        if len(thumbnail_tags) > 0:
+            thumbnail = extract_text(thumbnail_tags[0])
+            if thumbnail[0] == '/':
+                thumbnail = base_url + thumbnail
+        content = escape(extract_text(result.xpath(content_xpath)))
+
+        # append result
+        results.append({'url': href,
+                        'title': title,
+                        'img_src': thumbnail,
+                        'content': content})
+
+    # return results
+    return results

searx/engines/frinkiac.py

@@ -0,0 +1,44 @@
+"""
+Frinkiac (Images)
+
+@website     https://www.frinkiac.com
+@provide-api no
+@using-api   no
+@results     JSON
+@stable      no
+@parse       url, title, img_src
+"""
+
+from json import loads
+from searx.url_utils import urlencode
+
+categories = ['images']
+
+BASE = 'https://frinkiac.com/'
+SEARCH_URL = '{base}api/search?{query}'
+RESULT_URL = '{base}?{query}'
+THUMB_URL = '{base}img/{episode}/{timestamp}/medium.jpg'
+IMAGE_URL = '{base}img/{episode}/{timestamp}.jpg'
+
+
+def request(query, params):
+    params['url'] = SEARCH_URL.format(base=BASE, query=urlencode({'q': query}))
+    return params
+
+
+def response(resp):
+    results = []
+    response_data = loads(resp.text)
+    for result in response_data:
+        episode = result['Episode']
+        timestamp = result['Timestamp']
+
+        results.append({'template': 'images.html',
+                        'url': RESULT_URL.format(base=BASE,
+                                                 query=urlencode({'p': 'caption', 'e': episode, 't': timestamp})),
+                        'title': episode,
+                        'content': '',
+                        'thumbnail_src': THUMB_URL.format(base=BASE, episode=episode, timestamp=timestamp),
+                        'img_src': IMAGE_URL.format(base=BASE, episode=episode, timestamp=timestamp)})
+
+    return results

searx/engines/gigablast.py

@@ -10,20 +10,29 @@
  @parse       url, title, content
 """
 
-from urllib import urlencode
-from cgi import escape
-from lxml import etree
-from random import randint
+from json import loads
 from time import time
+from lxml.html import fromstring
+from searx.url_utils import urlencode
 
 # engine dependent config
 categories = ['general']
 paging = True
-number_of_results = 5
+number_of_results = 10
+language_support = True
+safesearch = True
 
-# search-url, invalid HTTPS certificate
+# search-url
 base_url = 'https://gigablast.com/'
-search_string = 'search?{query}&n={number_of_results}&s={offset}&format=xml&qh=0&rxiyd={rxiyd}&rand={rand}'
+search_string = 'search?{query}'\
+    '&n={number_of_results}'\
+    '&c=main'\
+    '&s={offset}'\
+    '&format=json'\
+    '&qh=0'\
+    '&qlang={lang}'\
+    '&ff={safesearch}'\
+    '&rxikd={rxikd}'  # random number - 9 digits
 
 # specific xpath variables
 results_xpath = '//response//result'
@@ -31,17 +40,31 @@ url_xpath = './/url'
 title_xpath = './/title'
 content_xpath = './/sum'
 
+supported_languages_url = 'https://gigablast.com/search?&rxikd=1'
+
 
 # do search-request
 def request(query, params):
     offset = (params['pageno'] - 1) * number_of_results
 
-    search_path = search_string.format(
-        query=urlencode({'q': query}),
-        offset=offset,
-        number_of_results=number_of_results,
-        rxiyd=randint(10000, 10000000),
-        rand=int(time()))
+    if params['language'] == 'all':
+        language = 'xx'
+    else:
+        language = params['language'].replace('-', '_').lower()
+        if language.split('-')[0] != 'zh':
+            language = language.split('-')[0]
+
+    if params['safesearch'] >= 1:
+        safesearch = 1
+    else:
+        safesearch = 0
+
+    search_path = search_string.format(query=urlencode({'q': query}),
+                                       offset=offset,
+                                       number_of_results=number_of_results,
+                                       rxikd=str(time())[:9],
+                                       lang=language,
+                                       safesearch=safesearch)
 
     params['url'] = base_url + search_path
 
@@ -52,18 +75,32 @@ def request(query, params):
 def response(resp):
     results = []
 
-    dom = etree.fromstring(resp.content)
-
     # parse results
-    for result in dom.xpath(results_xpath):
-        url = result.xpath(url_xpath)[0].text
-        title = result.xpath(title_xpath)[0].text
-        content = escape(result.xpath(content_xpath)[0].text)
+    response_json = loads(resp.text)
 
+    for result in response_json['results']:
         # append result
-        results.append({'url': url,
-                        'title': title,
-                        'content': content})
+        results.append({'url': result['url'],
+                        'title': result['title'],
+                        'content': result['sum']})
 
     # return results
     return results
+
+
+# get supported languages from their site
+def _fetch_supported_languages(resp):
+    supported_languages = []
+    dom = fromstring(resp.text)
+    links = dom.xpath('//span[@id="menu2"]/a')
+    for link in links:
+        href = link.xpath('./@href')[0].split('lang%3A')
+        if len(href) == 2:
+            code = href[1].split('_')
+            if len(code) == 2:
+                code = code[0] + '-' + code[1].upper()
+            else:
+                code = code[0]
+            supported_languages.append(code)
+
+    return supported_languages

searx/engines/github.py

@@ -10,9 +10,8 @@
  @parse       url, title, content
 """
 
-from urllib import urlencode
 from json import loads
-from cgi import escape
+from searx.url_utils import urlencode
 
 # engine dependent config
 categories = ['it']
@@ -48,7 +47,7 @@ def response(resp):
         url = res['html_url']
 
         if res['description']:
-            content = escape(res['description'][:500])
+            content = res['description'][:500]
         else:
             content = ''
 

searx/engines/google.py

@@ -9,12 +9,10 @@
 # @parse       url, title, content, suggestion
 
 import re
-from cgi import escape
-from urllib import urlencode
-from urlparse import urlparse, parse_qsl
 from lxml import html, etree
 from searx.engines.xpath import extract_text, extract_url
-from searx.search import logger
+from searx import logger
+from searx.url_utils import urlencode, urlparse, parse_qsl
 
 logger = logger.getChild('google engine')
 
@@ -24,6 +22,7 @@ categories = ['general']
 paging = True
 language_support = True
 use_locale_domain = True
+time_range_support = True
 
 # based on https://en.wikipedia.org/wiki/List_of_Google_domains and tests
 default_hostname = 'www.google.com'
@@ -46,11 +45,11 @@ country_to_hostname = {
     'NZ': 'www.google.co.nz',  # New Zealand
     'PH': 'www.google.com.ph',  # Philippines
     'SG': 'www.google.com.sg',  # Singapore
-    # 'US': 'www.google.us',  # United State, redirect to .com
+    # 'US': 'www.google.us',  # United States, redirect to .com
     'ZA': 'www.google.co.za',  # South Africa
     'AR': 'www.google.com.ar',  # Argentina
     'CL': 'www.google.cl',  # Chile
-    'ES': 'www.google.es',  # Span
+    'ES': 'www.google.es',  # Spain
     'MX': 'www.google.com.mx',  # Mexico
     'EE': 'www.google.ee',  # Estonia
     'FI': 'www.google.fi',  # Finland
@@ -61,7 +60,7 @@ country_to_hostname = {
     'HU': 'www.google.hu',  # Hungary
     'IT': 'www.google.it',  # Italy
     'JP': 'www.google.co.jp',  # Japan
-    'KR': 'www.google.co.kr',  # South Korean
+    'KR': 'www.google.co.kr',  # South Korea
     'LT': 'www.google.lt',  # Lithuania
     'LV': 'www.google.lv',  # Latvia
     'NO': 'www.google.no',  # Norway
@@ -76,9 +75,9 @@ country_to_hostname = {
     'SE': 'www.google.se',  # Sweden
     'TH': 'www.google.co.th',  # Thailand
     'TR': 'www.google.com.tr',  # Turkey
-    'UA': 'www.google.com.ua',  # Ikraine
-    # 'CN': 'www.google.cn',  # China, only from china ?
-    'HK': 'www.google.com.hk',  # Hong kong
+    'UA': 'www.google.com.ua',  # Ukraine
+    # 'CN': 'www.google.cn',  # China, only from China ?
+    'HK': 'www.google.com.hk',  # Hong Kong
     'TW': 'www.google.com.tw'  # Taiwan
 }
 
@@ -90,21 +89,29 @@ url_map = 'https://www.openstreetmap.org/'\
 search_path = '/search'
 search_url = ('https://{hostname}' +
               search_path +
-              '?{query}&start={offset}&gbv=1&gws_rd=cr')
+              '?{query}&start={offset}&gws_rd=cr&gbv=1&lr={lang}&ei=x')
+
+time_range_search = "&tbs=qdr:{range}"
+time_range_dict = {'day': 'd',
+                   'week': 'w',
+                   'month': 'm',
+                   'year': 'y'}
 
 # other URLs
 map_hostname_start = 'maps.google.'
 maps_path = '/maps'
 redirect_path = '/url'
 images_path = '/images'
+supported_languages_url = 'https://www.google.com/preferences?#languages'
 
 # specific xpath variables
-results_xpath = '//li[@class="g"]'
+results_xpath = '//div[@class="g"]'
 url_xpath = './/h3/a/@href'
 title_xpath = './/h3'
 content_xpath = './/span[@class="st"]'
 content_misc_xpath = './/div[@class="f slp"]'
 suggestion_xpath = '//p[@class="_Bmc"]'
+spelling_suggestion_xpath = '//a[@class="spell"]'
 
 # map : detail location
 map_address_xpath = './/div[@class="s"]//table//td[2]/span/text()'
@@ -149,7 +156,7 @@ def parse_url(url_string, google_hostname):
 def extract_text_from_dom(result, xpath):
     r = result.xpath(xpath)
     if len(r) > 0:
-        return escape(extract_text(r[0]))
+        return extract_text(r[0])
     return None
 
 
@@ -160,13 +167,19 @@ def request(query, params):
     if params['language'] == 'all':
         language = 'en'
         country = 'US'
+        url_lang = ''
+    elif params['language'][:2] == 'jv':
+        language = 'jw'
+        country = 'ID'
+        url_lang = 'lang_jw'
     else:
-        language_array = params['language'].lower().split('_')
+        language_array = params['language'].lower().split('-')
         if len(language_array) == 2:
             country = language_array[1]
         else:
             country = 'US'
         language = language_array[0] + ',' + language_array[0] + '-' + country
+        url_lang = 'lang_' + language_array[0]
 
     if use_locale_domain:
         google_hostname = country_to_hostname.get(country.upper(), default_hostname)
@@ -175,7 +188,10 @@ def request(query, params):
 
     params['url'] = search_url.format(offset=offset,
                                       query=urlencode({'q': query}),
-                                      hostname=google_hostname)
+                                      hostname=google_hostname,
+                                      lang=url_lang)
+    if params['time_range'] in time_range_dict:
+        params['url'] += time_range_search.format(range=time_range_dict[params['time_range']])
 
     params['headers']['Accept-Language'] = language
     params['headers']['Accept'] = 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
@@ -201,6 +217,16 @@ def response(resp):
     # convert the text to dom
     dom = html.fromstring(resp.text)
 
+    instant_answer = dom.xpath('//div[@id="_vBb"]//text()')
+    if instant_answer:
+        results.append({'answer': u' '.join(instant_answer)})
+    try:
+        results_num = int(dom.xpath('//div[@id="resultStats"]//text()')[0]
+                          .split()[1].replace(',', ''))
+        results.append({'number_of_results': results_num})
+    except:
+        pass
+
     # parse results
     for result in dom.xpath(results_xpath):
         try:
@@ -209,29 +235,29 @@ def response(resp):
             parsed_url = urlparse(url, google_hostname)
 
             # map result
-            if ((parsed_url.netloc == google_hostname and parsed_url.path.startswith(maps_path))
-               or (parsed_url.netloc.startswith(map_hostname_start))):
-                x = result.xpath(map_near)
-                if len(x) > 0:
-                    # map : near the location
-                    results = results + parse_map_near(parsed_url, x, google_hostname)
-                else:
-                    # map : detail about a location
-                    results = results + parse_map_detail(parsed_url, result, google_hostname)
+            if parsed_url.netloc == google_hostname:
+                # TODO fix inside links
+                continue
+                # if parsed_url.path.startswith(maps_path) or parsed_url.netloc.startswith(map_hostname_start):
+                #     print "yooooo"*30
+                #     x = result.xpath(map_near)
+                #     if len(x) > 0:
+                #         # map : near the location
+                #         results = results + parse_map_near(parsed_url, x, google_hostname)
+                #     else:
+                #         # map : detail about a location
+                #         results = results + parse_map_detail(parsed_url, result, google_hostname)
+                # # google news
+                # elif parsed_url.path == search_path:
+                #     # skipping news results
+                #     pass
 
-            # google news
-            elif (parsed_url.netloc == google_hostname
-                  and parsed_url.path == search_path):
-                # skipping news results
-                pass
-
-            # images result
-            elif (parsed_url.netloc == google_hostname
-                  and parsed_url.path == images_path):
-                # only thumbnail image provided,
-                # so skipping image results
-                # results = results + parse_images(result, google_hostname)
-                pass
+                # # images result
+                # elif parsed_url.path == images_path:
+                #     # only thumbnail image provided,
+                #     # so skipping image results
+                #     # results = results + parse_images(result, google_hostname)
+                #     pass
 
             else:
                 # normal result
@@ -253,7 +279,10 @@ def response(resp):
     # parse suggestion
     for suggestion in dom.xpath(suggestion_xpath):
         # append suggestion
-        results.append({'suggestion': escape(extract_text(suggestion))})
+        results.append({'suggestion': extract_text(suggestion)})
+
+    for correction in dom.xpath(spelling_suggestion_xpath):
+        results.append({'correction': extract_text(correction)})
 
     # return results
     return results
@@ -297,9 +326,9 @@ def parse_map_detail(parsed_url, result, google_hostname):
     results = []
 
     # try to parse the geoloc
-    m = re.search('@([0-9\.]+),([0-9\.]+),([0-9]+)', parsed_url.path)
+    m = re.search(r'@([0-9\.]+),([0-9\.]+),([0-9]+)', parsed_url.path)
     if m is None:
-        m = re.search('ll\=([0-9\.]+),([0-9\.]+)\&z\=([0-9]+)', parsed_url.query)
+        m = re.search(r'll\=([0-9\.]+),([0-9\.]+)\&z\=([0-9]+)', parsed_url.query)
 
     if m is not None:
         # geoloc found (ignored)
@@ -344,3 +373,16 @@ def attributes_to_html(attributes):
         retval = retval + '<tr><th>' + a.get('label') + '</th><td>' + value + '</td></tr>'
     retval = retval + '</table>'
     return retval
+
+
+# get supported languages from their site
+def _fetch_supported_languages(resp):
+    supported_languages = {}
+    dom = html.fromstring(resp.text)
+    options = dom.xpath('//table//td/font/label/span')
+    for option in options:
+        code = option.xpath('./@id')[0][1:]
+        name = option.text.title()
+        supported_languages[code] = {"name": name}
+
+    return supported_languages

searx/engines/google_images.py

@@ -10,33 +10,54 @@
  @parse       url, title, img_src
 """
 
-from urllib import urlencode
-from urlparse import parse_qs
+from datetime import date, timedelta
 from json import loads
 from lxml import html
+from searx.url_utils import urlencode
+
 
 # engine dependent config
 categories = ['images']
 paging = True
 safesearch = True
+time_range_support = True
+number_of_results = 100
 
 search_url = 'https://www.google.com/search'\
     '?{query}'\
+    '&asearch=ichunk'\
+    '&async=_id:rg_s,_pms:s'\
     '&tbm=isch'\
-    '&ijn=1'\
-    '&start={offset}'
+    '&yv=2'\
+    '&{search_options}'
+time_range_attr = "qdr:{range}"
+time_range_custom_attr = "cdr:1,cd_min:{start},cd_max{end}"
+time_range_dict = {'day': 'd',
+                   'week': 'w',
+                   'month': 'm'}
 
 
 # do search-request
 def request(query, params):
-    offset = (params['pageno'] - 1) * 100
+    search_options = {
+        'ijn': params['pageno'] - 1,
+        'start': (params['pageno'] - 1) * number_of_results
+    }
 
-    params['url'] = search_url.format(query=urlencode({'q': query}),
-                                      offset=offset,
-                                      safesearch=safesearch)
+    if params['time_range'] in time_range_dict:
+        search_options['tbs'] = time_range_attr.format(range=time_range_dict[params['time_range']])
+    elif params['time_range'] == 'year':
+        now = date.today()
+        then = now - timedelta(days=365)
+        start = then.strftime('%m/%d/%Y')
+        end = now.strftime('%m/%d/%Y')
+        search_options['tbs'] = time_range_custom_attr.format(start=start, end=end)
 
     if safesearch and params['safesearch']:
-        params['url'] += '&' + urlencode({'safe': 'active'})
+        search_options['safe'] = 'on'
+
+    params['url'] = search_url.format(query=urlencode({'q': query}),
+                                      search_options=urlencode(search_options))
 
     return params
 
@@ -45,14 +66,17 @@ def request(query, params):
 def response(resp):
     results = []
 
-    dom = html.fromstring(resp.text)
+    g_result = loads(resp.text)
+
+    dom = html.fromstring(g_result[1][1])
 
     # parse results
     for result in dom.xpath('//div[@data-ved]'):
-        data_url = result.xpath('./a/@href')[0]
-        data_query = {k: v[0] for k, v in parse_qs(data_url.split('?', 1)[1]).iteritems()}
 
-        metadata = loads(result.xpath('./div[@class="rg_meta"]/text()')[0])
+        try:
+            metadata = loads(''.join(result.xpath('./div[@class="rg_meta"]/text()')))
+        except:
+            continue
 
         thumbnail_src = metadata['tu']
 
@@ -60,11 +84,11 @@ def response(resp):
         thumbnail_src = thumbnail_src.replace("http://", "https://")
 
         # append result
-        results.append({'url': data_query['imgrefurl'],
+        results.append({'url': metadata['ru'],
                         'title': metadata['pt'],
                         'content': metadata['s'],
-                        'thumbnail_src': metadata['tu'],
-                        'img_src': data_query['imgurl'],
+                        'thumbnail_src': thumbnail_src,
+                        'img_src': metadata['ou'],
                         'template': 'images.html'})
 
     # return results

searx/engines/google_news.py

@@ -1,41 +1,58 @@
 """
  Google (News)
 
- @website     https://www.google.com
- @provide-api yes (https://developers.google.com/web-search/docs/),
-              deprecated!
+ @website     https://news.google.com
+ @provide-api no
 
- @using-api   yes
- @results     JSON
- @stable      yes (but deprecated)
+ @using-api   no
+ @results     HTML
+ @stable      no
  @parse       url, title, content, publishedDate
 """
 
-from urllib import urlencode
-from json import loads
-from dateutil import parser
+from lxml import html
+from searx.engines.google import _fetch_supported_languages, supported_languages_url
+from searx.url_utils import urlencode
 
 # search-url
 categories = ['news']
 paging = True
 language_support = True
+safesearch = True
+time_range_support = True
+number_of_results = 10
 
-# engine dependent config
-url = 'https://ajax.googleapis.com/'
-search_url = url + 'ajax/services/search/news?v=2.0&start={offset}&rsz=large&safe=off&filter=off&{query}&hl={lang}'
+search_url = 'https://www.google.com/search'\
+    '?{query}'\
+    '&tbm=nws'\
+    '&gws_rd=cr'\
+    '&{search_options}'
+time_range_attr = "qdr:{range}"
+time_range_dict = {'day': 'd',
+                   'week': 'w',
+                   'month': 'm',
+                   'year': 'y'}
 
 
 # do search-request
 def request(query, params):
-    offset = (params['pageno'] - 1) * 8
 
-    language = 'en-US'
+    search_options = {
+        'start': (params['pageno'] - 1) * number_of_results
+    }
+
+    if params['time_range'] in time_range_dict:
+        search_options['tbs'] = time_range_attr.format(range=time_range_dict[params['time_range']])
+
+    if safesearch and params['safesearch']:
+        search_options['safe'] = 'on'
+
+    params['url'] = search_url.format(query=urlencode({'q': query}),
+                                      search_options=urlencode(search_options))
+
     if params['language'] != 'all':
-        language = params['language'].replace('_', '-')
-
-    params['url'] = search_url.format(offset=offset,
-                                      query=urlencode({'q': query}),
-                                      lang=language)
+        language_array = params['language'].lower().split('-')
+        params['url'] += '&lr=lang_' + language_array[0]
 
     return params
 
@@ -44,24 +61,24 @@ def request(query, params):
 def response(resp):
     results = []
 
-    search_res = loads(resp.text)
-
-    # return empty array if there are no results
-    if not search_res.get('responseData', {}).get('results'):
-        return []
+    dom = html.fromstring(resp.text)
 
     # parse results
-    for result in search_res['responseData']['results']:
-        # parse publishedDate
-        publishedDate = parser.parse(result['publishedDate'])
-        if 'url' not in result:
+    for result in dom.xpath('//div[@class="g"]|//div[@class="g _cy"]'):
+        try:
+            r = {
+                'url': result.xpath('.//div[@class="_cnc"]//a/@href')[0],
+                'title': ''.join(result.xpath('.//div[@class="_cnc"]//h3//text()')),
+                'content': ''.join(result.xpath('.//div[@class="st"]//text()')),
+            }
+        except:
             continue
 
-        # append result
-        results.append({'url': result['unescapedUrl'],
-                        'title': result['titleNoFormatting'],
-                        'publishedDate': publishedDate,
-                        'content': result['content']})
+        imgs = result.xpath('.//img/@src')
+        if len(imgs) and not imgs[0].startswith('data'):
+            r['img_src'] = imgs[0]
+
+        results.append(r)
 
     # return results
     return results

searx/engines/ina.py

@@ -0,0 +1,87 @@
+#  INA (Videos)
+#
+# @website     https://www.ina.fr/
+# @provide-api no
+#
+# @using-api   no
+# @results     HTML (using search portal)
+# @stable      no (HTML can change)
+# @parse       url, title, content, publishedDate, thumbnail
+#
+# @todo        set content-parameter with correct data
+# @todo        embedded (needs some md5 from video page)
+
+from json import loads
+from lxml import html
+from dateutil import parser
+from searx.engines.xpath import extract_text
+from searx.url_utils import urlencode
+
+try:
+    from HTMLParser import HTMLParser
+except:
+    from html.parser import HTMLParser
+
+# engine dependent config
+categories = ['videos']
+paging = True
+page_size = 48
+
+# search-url
+base_url = 'https://www.ina.fr'
+search_url = base_url + '/layout/set/ajax/recherche/result?autopromote=&hf={ps}&b={start}&type=Video&r=&{query}'
+
+# specific xpath variables
+results_xpath = '//div[contains(@class,"search-results--list")]/div[@class="media"]'
+url_xpath = './/a/@href'
+title_xpath = './/h3[@class="h3--title media-heading"]'
+thumbnail_xpath = './/img/@src'
+publishedDate_xpath = './/span[@class="broadcast"]'
+content_xpath = './/p[@class="media-body__summary"]'
+
+
+# do search-request
+def request(query, params):
+    params['url'] = search_url.format(ps=page_size,
+                                      start=params['pageno'] * page_size,
+                                      query=urlencode({'q': query}))
+
+    return params
+
+
+# get response from search-request
+def response(resp):
+    results = []
+
+    # we get html in a JSON container...
+    response = loads(resp.text)
+    if "content" not in response:
+        return []
+    dom = html.fromstring(response["content"])
+    p = HTMLParser()
+
+    # parse results
+    for result in dom.xpath(results_xpath):
+        videoid = result.xpath(url_xpath)[0]
+        url = base_url + videoid
+        title = p.unescape(extract_text(result.xpath(title_xpath)))
+        thumbnail = extract_text(result.xpath(thumbnail_xpath)[0])
+        if thumbnail[0] == '/':
+            thumbnail = base_url + thumbnail
+        d = extract_text(result.xpath(publishedDate_xpath)[0])
+        d = d.split('/')
+        # force ISO date to avoid wrong parsing
+        d = "%s-%s-%s" % (d[2], d[1], d[0])
+        publishedDate = parser.parse(d)
+        content = extract_text(result.xpath(content_xpath))
+
+        # append result
+        results.append({'url': url,
+                        'title': title,
+                        'content': content,
+                        'template': 'videos.html',
+                        'publishedDate': publishedDate,
+                        'thumbnail': thumbnail})
+
+    # return results
+    return results

searx/engines/json_engine.py

@@ -1,17 +1,31 @@
-from urllib import urlencode
-from json import loads
 from collections import Iterable
+from json import loads
+from sys import version_info
+from searx.url_utils import urlencode
+
+if version_info[0] == 3:
+    unicode = str
 
 search_url = None
 url_query = None
 content_query = None
 title_query = None
-# suggestion_xpath = ''
+paging = False
+suggestion_query = ''
+results_query = ''
+
+# parameters for engines with paging support
+#
+# number of results on each page
+# (only needed if the site requires not a page number, but an offset)
+page_size = 1
+# number of the first page (usually 0 or 1)
+first_page_num = 1
 
 
 def iterate(iterable):
     if type(iterable) == dict:
-        it = iterable.iteritems()
+        it = iterable.items()
 
     else:
         it = enumerate(iterable)
@@ -69,19 +83,36 @@ def query(data, query_string):
 
 def request(query, params):
     query = urlencode({'q': query})[2:]
-    params['url'] = search_url.format(query=query)
+
+    fp = {'query': query}
+    if paging and search_url.find('{pageno}') >= 0:
+        fp['pageno'] = (params['pageno'] - 1) * page_size + first_page_num
+
+    params['url'] = search_url.format(**fp)
     params['query'] = query
+
     return params
 
 
 def response(resp):
     results = []
-
     json = loads(resp.text)
+    if results_query:
+        for result in query(json, results_query)[0]:
+            url = query(result, url_query)[0]
+            title = query(result, title_query)[0]
+            content = query(result, content_query)[0]
+            results.append({'url': url, 'title': title, 'content': content})
+    else:
+        for url, title, content in zip(
+            query(json, url_query),
+            query(json, title_query),
+            query(json, content_query)
+        ):
+            results.append({'url': url, 'title': title, 'content': content})
 
-    urls = query(json, url_query)
-    contents = query(json, content_query)
-    titles = query(json, title_query)
-    for url, title, content in zip(urls, titles, contents):
-        results.append({'url': url, 'title': title, 'content': content})
+    if not suggestion_query:
+        return results
+    for suggestion in query(json, suggestion_query):
+        results.append({'suggestion': suggestion})
     return results

searx/engines/kickass.py

@@ -10,19 +10,18 @@
  @parse       url, title, content, seed, leech, magnetlink
 """
 
-from urlparse import urljoin
-from cgi import escape
-from urllib import quote
 from lxml import html
 from operator import itemgetter
 from searx.engines.xpath import extract_text
+from searx.utils import get_torrent_size, convert_str_to_int
+from searx.url_utils import quote, urljoin
 
 # engine dependent config
 categories = ['videos', 'music', 'files']
 paging = True
 
 # search-url
-url = 'https://kickass.to/'
+url = 'https://kickass.cd/'
 search_url = url + 'search/{search_term}/{pageno}/'
 
 # specific xpath variables
@@ -56,42 +55,17 @@ def response(resp):
         link = result.xpath('.//a[@class="cellMainLink"]')[0]
         href = urljoin(url, link.attrib['href'])
         title = extract_text(link)
-        content = escape(extract_text(result.xpath(content_xpath)))
-        seed = result.xpath('.//td[contains(@class, "green")]/text()')[0]
-        leech = result.xpath('.//td[contains(@class, "red")]/text()')[0]
-        filesize = result.xpath('.//td[contains(@class, "nobr")]/text()')[0]
-        filesize_multiplier = result.xpath('.//td[contains(@class, "nobr")]//span/text()')[0]
-        files = result.xpath('.//td[contains(@class, "center")][2]/text()')[0]
+        content = extract_text(result.xpath(content_xpath))
+        seed = extract_text(result.xpath('.//td[contains(@class, "green")]'))
+        leech = extract_text(result.xpath('.//td[contains(@class, "red")]'))
+        filesize_info = extract_text(result.xpath('.//td[contains(@class, "nobr")]'))
+        files = extract_text(result.xpath('.//td[contains(@class, "center")][2]'))
 
-        # convert seed to int if possible
-        if seed.isdigit():
-            seed = int(seed)
-        else:
-            seed = 0
+        seed = convert_str_to_int(seed)
+        leech = convert_str_to_int(leech)
 
-        # convert leech to int if possible
-        if leech.isdigit():
-            leech = int(leech)
-        else:
-            leech = 0
-
-        # convert filesize to byte if possible
-        try:
-            filesize = float(filesize)
-
-            # convert filesize to byte
-            if filesize_multiplier == 'TB':
-                filesize = int(filesize * 1024 * 1024 * 1024 * 1024)
-            elif filesize_multiplier == 'GB':
-                filesize = int(filesize * 1024 * 1024 * 1024)
-            elif filesize_multiplier == 'MB':
-                filesize = int(filesize * 1024 * 1024)
-            elif filesize_multiplier == 'KB':
-                filesize = int(filesize * 1024)
-        except:
-            filesize = None
-
-        # convert files to int if possible
+        filesize, filesize_multiplier = filesize_info.split()
+        filesize = get_torrent_size(filesize, filesize_multiplier)
         if files.isdigit():
             files = int(files)
         else:

searx/engines/mediawiki.py

@@ -14,23 +14,24 @@
 
 from json import loads
 from string import Formatter
-from urllib import urlencode, quote
+from searx.url_utils import urlencode, quote
 
 # engine dependent config
 categories = ['general']
 language_support = True
 paging = True
 number_of_results = 1
+search_type = 'nearmatch'  # possible values: title, text, nearmatch
 
 # search-url
 base_url = 'https://{language}.wikipedia.org/'
-search_url = base_url + 'w/api.php?action=query'\
-                                 '&list=search'\
-                                 '&{query}'\
-                                 '&srprop=timestamp'\
-                                 '&format=json'\
-                                 '&sroffset={offset}'\
-                                 '&srlimit={limit}'     # noqa
+search_postfix = 'w/api.php?action=query'\
+    '&list=search'\
+    '&{query}'\
+    '&format=json'\
+    '&sroffset={offset}'\
+    '&srlimit={limit}'\
+    '&srwhat={searchtype}'
 
 
 # do search-request
@@ -39,21 +40,25 @@ def request(query, params):
 
     string_args = dict(query=urlencode({'srsearch': query}),
                        offset=offset,
-                       limit=number_of_results)
+                       limit=number_of_results,
+                       searchtype=search_type)
 
     format_strings = list(Formatter().parse(base_url))
 
     if params['language'] == 'all':
         language = 'en'
     else:
-        language = params['language'].split('_')[0]
+        language = params['language'].split('-')[0]
 
-    if len(format_strings) > 1:
+    # format_string [('https://', 'language', '', None), ('.wikipedia.org/', None, None, None)]
+    if any(x[1] == 'language' for x in format_strings):
         string_args['language'] = language
 
     # write search-language back to params, required in response
     params['language'] = language
 
+    search_url = base_url + search_postfix
+
     params['url'] = search_url.format(**string_args)
 
     return params
@@ -71,6 +76,8 @@ def response(resp):
 
     # parse results
     for result in search_results['query']['search']:
+        if result.get('snippet', '').startswith('#REDIRECT'):
+            continue
         url = base_url.format(language=resp.search_params['language']) +\
             'wiki/' + quote(result['title'].replace(' ', '_').encode('utf-8'))
 

searx/engines/mixcloud.py

@@ -11,8 +11,8 @@
 """
 
 from json import loads
-from urllib import urlencode
 from dateutil import parser
+from searx.url_utils import urlencode
 
 # engine dependent config
 categories = ['music']

searx/engines/nyaa.py

@@ -0,0 +1,117 @@
+"""
+ Nyaa.se (Anime Bittorrent tracker)
+
+ @website      http://www.nyaa.se/
+ @provide-api  no
+ @using-api    no
+ @results      HTML
+ @stable       no (HTML can change)
+ @parse        url, title, content, seed, leech, torrentfile
+"""
+
+from lxml import html
+from searx.engines.xpath import extract_text
+from searx.url_utils import urlencode
+
+# engine dependent config
+categories = ['files', 'images', 'videos', 'music']
+paging = True
+
+# search-url
+base_url = 'http://www.nyaa.se/'
+search_url = base_url + '?page=search&{query}&offset={offset}'
+
+# xpath queries
+xpath_results = '//table[@class="tlist"]//tr[contains(@class, "tlistrow")]'
+xpath_category = './/td[@class="tlisticon"]/a'
+xpath_title = './/td[@class="tlistname"]/a'
+xpath_torrent_file = './/td[@class="tlistdownload"]/a'
+xpath_filesize = './/td[@class="tlistsize"]/text()'
+xpath_seeds = './/td[@class="tlistsn"]/text()'
+xpath_leeches = './/td[@class="tlistln"]/text()'
+xpath_downloads = './/td[@class="tlistdn"]/text()'
+
+
+# convert a variable to integer or return 0 if it's not a number
+def int_or_zero(num):
+    if isinstance(num, list):
+        if len(num) < 1:
+            return 0
+        num = num[0]
+    if num.isdigit():
+        return int(num)
+    return 0
+
+
+# get multiplier to convert torrent size to bytes
+def get_filesize_mul(suffix):
+    return {
+        'KB': 1024,
+        'MB': 1024 ** 2,
+        'GB': 1024 ** 3,
+        'TB': 1024 ** 4,
+
+        'KIB': 1024,
+        'MIB': 1024 ** 2,
+        'GIB': 1024 ** 3,
+        'TIB': 1024 ** 4
+    }[str(suffix).upper()]
+
+
+# do search-request
+def request(query, params):
+    query = urlencode({'term': query})
+    params['url'] = search_url.format(query=query, offset=params['pageno'])
+    return params
+
+
+# get response from search-request
+def response(resp):
+    results = []
+
+    dom = html.fromstring(resp.text)
+
+    for result in dom.xpath(xpath_results):
+        # category in which our torrent belongs
+        category = result.xpath(xpath_category)[0].attrib.get('title')
+
+        # torrent title
+        page_a = result.xpath(xpath_title)[0]
+        title = extract_text(page_a)
+
+        # link to the page
+        href = page_a.attrib.get('href')
+
+        # link to the torrent file
+        torrent_link = result.xpath(xpath_torrent_file)[0].attrib.get('href')
+
+        # torrent size
+        try:
+            file_size, suffix = result.xpath(xpath_filesize)[0].split(' ')
+            file_size = int(float(file_size) * get_filesize_mul(suffix))
+        except:
+            file_size = None
+
+        # seed count
+        seed = int_or_zero(result.xpath(xpath_seeds))
+
+        # leech count
+        leech = int_or_zero(result.xpath(xpath_leeches))
+
+        # torrent downloads count
+        downloads = int_or_zero(result.xpath(xpath_downloads))
+
+        # content string contains all information not included into template
+        content = 'Category: "{category}". Downloaded {downloads} times.'
+        content = content.format(category=category, downloads=downloads)
+
+        results.append({'url': href,
+                        'title': title,
+                        'content': content,
+                        'seed': seed,
+                        'leech': leech,
+                        'filesize': file_size,
+                        'torrentfile': torrent_link,
+                        'template': 'torrent.html'})
+
+    return results

searx/engines/openstreetmap.py

@@ -11,7 +11,6 @@
 """
 
 from json import loads
-from searx.utils import searx_useragent
 
 # engine dependent config
 categories = ['map']
@@ -27,9 +26,6 @@ result_base_url = 'https://openstreetmap.org/{osm_type}/{osm_id}'
 def request(query, params):
     params['url'] = base_url + search_string.format(query=query)
 
-    # using searx User-Agent
-    params['headers']['User-Agent'] = searx_useragent()
-
     return params
 
 
@@ -43,7 +39,7 @@ def response(resp):
         if 'display_name' not in r:
             continue
 
-        title = r['display_name']
+        title = r['display_name'] or u''
         osm_type = r.get('osm_type', r.get('type'))
         url = result_base_url.format(osm_type=osm_type,
                                      osm_id=r['osm_id'])

searx/engines/pdbe.py

@@ -0,0 +1,109 @@
+"""
+ PDBe (Protein Data Bank in Europe)
+
+ @website       https://www.ebi.ac.uk/pdbe
+ @provide-api   yes (https://www.ebi.ac.uk/pdbe/api/doc/search.html),
+                unlimited
+ @using-api     yes
+ @results       python dictionary (from json)
+ @stable        yes
+ @parse         url, title, content, img_src
+"""
+
+from json import loads
+from flask_babel import gettext
+
+categories = ['science']
+
+hide_obsolete = False
+
+# status codes of unpublished entries
+pdb_unpublished_codes = ['HPUB', 'HOLD', 'PROC', 'WAIT', 'AUTH', 'AUCO', 'REPL', 'POLC', 'REFI', 'TRSF', 'WDRN']
+# url for api query
+pdbe_solr_url = 'https://www.ebi.ac.uk/pdbe/search/pdb/select?'
+# base url for results
+pdbe_entry_url = 'https://www.ebi.ac.uk/pdbe/entry/pdb/{pdb_id}'
+# link to preview image of structure
+pdbe_preview_url = 'https://www.ebi.ac.uk/pdbe/static/entry/{pdb_id}_deposited_chain_front_image-200x200.png'
+
+
+def request(query, params):
+
+    params['url'] = pdbe_solr_url
+    params['method'] = 'POST'
+    params['data'] = {
+        'q': query,
+        'wt': "json"  # request response in parsable format
+    }
+    return params
+
+
+def construct_body(result):
+    # set title
+    title = result['title']
+
+    # construct content body
+    content = """{title}<br />{authors} {journal} <strong>{volume}</strong>&nbsp;{page} ({year})"""
+
+    # replace placeholders with actual content
+    try:
+        if result['journal']:
+            content = content.format(
+                title=result['citation_title'],
+                authors=result['entry_author_list'][0], journal=result['journal'], volume=result['journal_volume'],
+                page=result['journal_page'], year=result['citation_year'])
+        else:
+            content = content.format(
+                title=result['citation_title'],
+                authors=result['entry_author_list'][0], journal='', volume='', page='', year=result['release_year'])
+        img_src = pdbe_preview_url.format(pdb_id=result['pdb_id'])
+    except (KeyError):
+        content = None
+        img_src = None
+
+    # construct url for preview image
+    try:
+        img_src = pdbe_preview_url.format(pdb_id=result['pdb_id'])
+    except (KeyError):
+        img_src = None
+
+    return [title, content, img_src]
+
+
+def response(resp):
+
+    results = []
+    json = loads(resp.text)['response']['docs']
+
+    # parse results
+    for result in json:
+        # catch obsolete entries and mark them accordingly
+        if result['status'] in pdb_unpublished_codes:
+            continue
+        if hide_obsolete:
+            continue
+        if result['status'] == 'OBS':
+            # expand title to add some sort of warning message
+            title = gettext('{title}&nbsp;(OBSOLETE)').format(title=result['title'])
+            superseded_url = pdbe_entry_url.format(pdb_id=result['superseded_by'])
+
+            # since we can't construct a proper body from the response, we'll make up our own
+            msg_superseded = gettext("This entry has been superseded by")
+            content = '<em>{msg_superseded} \<a href="{url}">{pdb_id}</a></em>'.format(
+                msg_superseded=msg_superseded,
+                url=superseded_url,
+                pdb_id=result['superseded_by'], )
+
+            # obsoleted entries don't have preview images
+            img_src = None
+        else:
+            title, content, img_src = construct_body(result)
+
+        results.append({
+            'url': pdbe_entry_url.format(pdb_id=result['pdb_id']),
+            'title': title,
+            'content': content,
+            'img_src': img_src
+        })
+
+    return results

searx/engines/photon.py

@@ -10,9 +10,9 @@
  @parse       url, title
 """
 
-from urllib import urlencode
 from json import loads
 from searx.utils import searx_useragent
+from searx.url_utils import urlencode
 
 # engine dependent config
 categories = ['map']
@@ -26,7 +26,7 @@ search_string = 'api/?{query}&limit={limit}'
 result_base_url = 'https://openstreetmap.org/{osm_type}/{osm_id}'
 
 # list of supported languages
-allowed_languages = ['de', 'en', 'fr', 'it']
+supported_languages = ['de', 'en', 'fr', 'it']
 
 
 # do search-request
@@ -37,7 +37,7 @@ def request(query, params):
 
     if params['language'] != 'all':
         language = params['language'].split('_')[0]
-        if language in allowed_languages:
+        if language in supported_languages:
             params['url'] = params['url'] + "&lang=" + language
 
     # using searx User-Agent

searx/engines/piratebay.py

@@ -8,12 +8,10 @@
 # @stable      yes (HTML can change)
 # @parse       url, title, content, seed, leech, magnetlink
 
-from urlparse import urljoin
-from cgi import escape
-from urllib import quote
 from lxml import html
 from operator import itemgetter
 from searx.engines.xpath import extract_text
+from searx.url_utils import quote, urljoin
 
 # engine dependent config
 categories = ['videos', 'music', 'files']
@@ -62,7 +60,7 @@ def response(resp):
         link = result.xpath('.//div[@class="detName"]//a')[0]
         href = urljoin(url, link.attrib.get('href'))
         title = extract_text(link)
-        content = escape(extract_text(result.xpath(content_xpath)))
+        content = extract_text(result.xpath(content_xpath))
         seed, leech = result.xpath('.//td[@align="right"]/text()')[:2]
 
         # convert seed to int if possible

searx/engines/qwant.py

@@ -10,14 +10,16 @@
  @parse       url, title, content
 """
 
-from urllib import urlencode
-from json import loads
 from datetime import datetime
+from json import loads
+from searx.utils import html_to_text
+from searx.url_utils import urlencode
 
 # engine dependent config
 categories = None
 paging = True
 language_support = True
+supported_languages_url = 'https://qwant.com/region'
 
 category_to_keyword = {'general': 'web',
                        'images': 'images',
@@ -44,7 +46,16 @@ def request(query, params):
 
     # add language tag if specified
     if params['language'] != 'all':
-        params['url'] += '&locale=' + params['language'].lower()
+        if params['language'] == 'no' or params['language'].startswith('no-'):
+            params['language'] = params['language'].replace('no', 'nb', 1)
+        if params['language'].find('-') < 0:
+            # tries to get a country code from language
+            for lang in supported_languages:
+                lc = lang.split('-')
+                if params['language'] == lc[0]:
+                    params['language'] = lang
+                    break
+        params['url'] += '&locale=' + params['language'].replace('-', '_').lower()
 
     return params
 
@@ -66,9 +77,9 @@ def response(resp):
     # parse results
     for result in res.get('items', {}):
 
-        title = result['title']
+        title = html_to_text(result['title'])
         res_url = result['url']
-        content = result['desc']
+        content = html_to_text(result['desc'])
 
         if category_to_keyword.get(categories[0], '') == 'web':
             results.append({'title': title,
@@ -85,14 +96,45 @@ def response(resp):
                             'thumbnail_src': thumbnail_src,
                             'img_src': img_src})
 
-        elif (category_to_keyword.get(categories[0], '') == 'news' or
-              category_to_keyword.get(categories[0], '') == 'social'):
+        elif category_to_keyword.get(categories[0], '') == 'social':
             published_date = datetime.fromtimestamp(result['date'], None)
-
+            img_src = result.get('img', None)
             results.append({'url': res_url,
                             'title': title,
                             'publishedDate': published_date,
-                            'content': content})
+                            'content': content,
+                            'img_src': img_src})
+
+        elif category_to_keyword.get(categories[0], '') == 'news':
+            published_date = datetime.fromtimestamp(result['date'], None)
+            media = result.get('media', [])
+            if len(media) > 0:
+                img_src = media[0].get('pict', {}).get('url', None)
+            else:
+                img_src = None
+            results.append({'url': res_url,
+                            'title': title,
+                            'publishedDate': published_date,
+                            'content': content,
+                            'img_src': img_src})
 
-    # return results
     return results
+
+
+# get supported languages from their site
+def _fetch_supported_languages(resp):
+    # list of regions is embedded in page as a js object
+    response_text = resp.text
+    response_text = response_text[response_text.find('regionalisation'):]
+    response_text = response_text[response_text.find('{'):response_text.find(');')]
+
+    regions_json = loads(response_text)
+
+    supported_languages = []
+    for lang in regions_json['languages'].values():
+        if lang['code'] == 'nb':
+            lang['code'] = 'no'
+        for country in lang['countries']:
+            supported_languages.append(lang['code'] + '-' + country)
+
+    return supported_languages

searx/engines/reddit.py

@@ -0,0 +1,76 @@
+"""
+ Reddit
+
+ @website      https://www.reddit.com/
+ @provide-api  yes (https://www.reddit.com/dev/api)
+
+ @using-api    yes
+ @results      JSON
+ @stable       yes
+ @parse        url, title, content, thumbnail, publishedDate
+"""
+
+import json
+from datetime import datetime
+from searx.url_utils import urlencode, urljoin, urlparse
+
+# engine dependent config
+categories = ['general', 'images', 'news', 'social media']
+page_size = 25
+
+# search-url
+base_url = 'https://www.reddit.com/'
+search_url = base_url + 'search.json?{query}'
+
+
+# do search-request
+def request(query, params):
+    query = urlencode({'q': query, 'limit': page_size})
+    params['url'] = search_url.format(query=query)
+
+    return params
+
+
+# get response from search-request
+def response(resp):
+    img_results = []
+    text_results = []
+
+    search_results = json.loads(resp.text)
+
+    # return empty array if there are no results
+    if 'data' not in search_results:
+        return []
+
+    posts = search_results.get('data', {}).get('children', [])
+
+    # process results
+    for post in posts:
+        data = post['data']
+
+        # extract post information
+        params = {
+            'url': urljoin(base_url, data['permalink']),
+            'title': data['title']
+        }
+
+        # if thumbnail field contains a valid URL, we need to change template
+        thumbnail = data['thumbnail']
+        url_info = urlparse(thumbnail)
+        # netloc & path
+        if url_info[1] != '' and url_info[2] != '':
+            params['img_src'] = data['url']
+            params['thumbnail_src'] = thumbnail
+            params['template'] = 'images.html'
+            img_results.append(params)
+        else:
+            created = datetime.fromtimestamp(data['created_utc'])
+            content = data['selftext']
+            if len(content) > 500:
+                content = content[:500] + '...'
+            params['content'] = content
+            params['publishedDate'] = created
+            text_results.append(params)
+
+    # show images first and text results second
+    return img_results + text_results

searx/engines/scanr_structures.py

@@ -0,0 +1,76 @@
+"""
+ ScanR Structures (Science)
+
+ @website     https://scanr.enseignementsup-recherche.gouv.fr
+ @provide-api yes (https://scanr.enseignementsup-recherche.gouv.fr/api/swagger-ui.html)
+
+ @using-api   yes
+ @results     JSON
+ @stable      yes
+ @parse       url, title, content, img_src
+"""
+
+from json import loads, dumps
+from searx.utils import html_to_text
+
+# engine dependent config
+categories = ['science']
+paging = True
+page_size = 20
+
+# search-url
+url = 'https://scanr.enseignementsup-recherche.gouv.fr/'
+search_url = url + 'api/structures/search'
+
+
+# do search-request
+def request(query, params):
+
+    params['url'] = search_url
+    params['method'] = 'POST'
+    params['headers']['Content-type'] = "application/json"
+    params['data'] = dumps({"query": query,
+                            "searchField": "ALL",
+                            "sortDirection": "ASC",
+                            "sortOrder": "RELEVANCY",
+                            "page": params['pageno'],
+                            "pageSize": page_size})
+
+    return params
+
+
+# get response from search-request
+def response(resp):
+    results = []
+
+    search_res = loads(resp.text)
+
+    # return empty array if there are no results
+    if search_res.get('total', 0) < 1:
+        return []
+
+    # parse results
+    for result in search_res['results']:
+        if 'id' not in result:
+            continue
+
+        # is it thumbnail or img_src??
+        thumbnail = None
+        if 'logo' in result:
+            thumbnail = result['logo']
+            if thumbnail[0] == '/':
+                thumbnail = url + thumbnail
+
+        content = None
+        if 'highlights' in result:
+            content = result['highlights'][0]['value']
+
+        # append result
+        results.append({'url': url + 'structure/' + result['id'],
+                        'title': result['label'],
+                        # 'thumbnail': thumbnail,
+                        'img_src': thumbnail,
+                        'content': html_to_text(content)})
+
+    # return results
+    return results

searx/engines/searchcode_code.py

@@ -10,8 +10,8 @@
  @parse       url, title, content
 """
 
-from urllib import urlencode
 from json import loads
+from searx.url_utils import urlencode
 
 
 # engine dependent config
@@ -20,7 +20,7 @@ paging = True
 
 # search-url
 url = 'https://searchcode.com/'
-search_url = url+'api/codesearch_I/?{query}&p={pageno}'
+search_url = url + 'api/codesearch_I/?{query}&p={pageno}'
 
 # special code-endings which are not recognised by the file ending
 code_endings = {'cs': 'c#',
@@ -31,13 +31,7 @@ code_endings = {'cs': 'c#',
 
 # do search-request
 def request(query, params):
-    params['url'] = search_url.format(query=urlencode({'q': query}),
-                                      pageno=params['pageno']-1)
-
-    # Disable SSL verification
-    # error: (60) SSL certificate problem: unable to get local issuer
-    # certificate
-    params['verify'] = False
+    params['url'] = search_url.format(query=urlencode({'q': query}), pageno=params['pageno'] - 1)
 
     return params
 

searx/engines/searchcode_doc.py

@@ -10,8 +10,8 @@
  @parse       url, title, content
 """
 
-from urllib import urlencode
 from json import loads
+from searx.url_utils import urlencode
 
 # engine dependent config
 categories = ['it']
@@ -19,18 +19,12 @@ paging = True
 
 # search-url
 url = 'https://searchcode.com/'
-search_url = url+'api/search_IV/?{query}&p={pageno}'
+search_url = url + 'api/search_IV/?{query}&p={pageno}'
 
 
 # do search-request
 def request(query, params):
-    params['url'] = search_url.format(query=urlencode({'q': query}),
-                                      pageno=params['pageno']-1)
-
-    # Disable SSL verification
-    # error: (60) SSL certificate problem: unable to get local issuer
-    # certificate
-    params['verify'] = False
+    params['url'] = search_url.format(query=urlencode({'q': query}), pageno=params['pageno'] - 1)
 
     return params
 
@@ -44,20 +38,12 @@ def response(resp):
     # parse results
     for result in search_results.get('results', []):
         href = result['url']
-        title = "[" + result['type'] + "] " +\
-                result['namespace'] +\
-                " " + result['name']
-        content = '<span class="highlight">[' +\
-                  result['type'] + "] " +\
-                  result['name'] + " " +\
-                  result['synopsis'] +\
-                  "</span><br />" +\
-                  result['description']
+        title = "[{}] {} {}".format(result['type'], result['namespace'], result['name'])
 
         # append result
         results.append({'url': href,
                         'title': title,
-                        'content': content})
+                        'content': result['description']})
 
     # return results
     return results

searx/engines/searx_engine.py

@@ -0,0 +1,57 @@
+"""
+ Searx (all)
+
+ @website     https://github.com/asciimoo/searx
+ @provide-api yes (https://asciimoo.ithub.io/searx/dev/search_api.html)
+
+ @using-api   yes
+ @results     JSON
+ @stable      yes (using api)
+ @parse       url, title, content
+"""
+
+from json import loads
+from searx.engines import categories as searx_categories
+
+
+categories = searx_categories.keys()
+
+# search-url
+instance_urls = []
+instance_index = 0
+
+
+# do search-request
+def request(query, params):
+    global instance_index
+    params['url'] = instance_urls[instance_index % len(instance_urls)]
+    params['method'] = 'POST'
+
+    instance_index += 1
+
+    params['data'] = {
+        'q': query,
+        'pageno': params['pageno'],
+        'language': params['language'],
+        'time_range': params['time_range'],
+        'category': params['category'],
+        'format': 'json'
+    }
+
+    return params
+
+
+# get response from search-request
+def response(resp):
+
+    response_json = loads(resp.text)
+    results = response_json['results']
+
+    for i in ('answers', 'infoboxes'):
+        results.extend(response_json[i])
+
+    results.extend({'suggestion': s} for s in response_json['suggestions'])
+
+    results.append({'number_of_results': response_json['number_of_results']})
+
+    return results

searx/engines/seedpeer.py

@@ -0,0 +1,75 @@
+#  Seedpeer (Videos, Music, Files)
+#
+# @website     http://seedpeer.eu
+# @provide-api no (nothing found)
+#
+# @using-api   no
+# @results     HTML (using search portal)
+# @stable      yes (HTML can change)
+# @parse       url, title, content, seed, leech, magnetlink
+
+from lxml import html
+from operator import itemgetter
+from searx.url_utils import quote, urljoin
+
+
+url = 'http://www.seedpeer.eu/'
+search_url = url + 'search/{search_term}/7/{page_no}.html'
+# specific xpath variables
+torrent_xpath = '//*[@id="body"]/center/center/table[2]/tr/td/a'
+alternative_torrent_xpath = '//*[@id="body"]/center/center/table[1]/tr/td/a'
+title_xpath = '//*[@id="body"]/center/center/table[2]/tr/td/a/text()'
+alternative_title_xpath = '//*[@id="body"]/center/center/table/tr/td/a'
+seeds_xpath = '//*[@id="body"]/center/center/table[2]/tr/td[4]/font/text()'
+alternative_seeds_xpath = '//*[@id="body"]/center/center/table/tr/td[4]/font/text()'
+peers_xpath = '//*[@id="body"]/center/center/table[2]/tr/td[5]/font/text()'
+alternative_peers_xpath = '//*[@id="body"]/center/center/table/tr/td[5]/font/text()'
+age_xpath = '//*[@id="body"]/center/center/table[2]/tr/td[2]/text()'
+alternative_age_xpath = '//*[@id="body"]/center/center/table/tr/td[2]/text()'
+size_xpath = '//*[@id="body"]/center/center/table[2]/tr/td[3]/text()'
+alternative_size_xpath = '//*[@id="body"]/center/center/table/tr/td[3]/text()'
+
+
+# do search-request
+def request(query, params):
+    params['url'] = search_url.format(search_term=quote(query),
+                                      page_no=params['pageno'] - 1)
+    return params
+
+
+# get response from search-request
+def response(resp):
+    results = []
+    dom = html.fromstring(resp.text)
+    torrent_links = dom.xpath(torrent_xpath)
+    if len(torrent_links) > 0:
+        seeds = dom.xpath(seeds_xpath)
+        peers = dom.xpath(peers_xpath)
+        titles = dom.xpath(title_xpath)
+        sizes = dom.xpath(size_xpath)
+        ages = dom.xpath(age_xpath)
+    else:  # under ~5 results uses a different xpath
+        torrent_links = dom.xpath(alternative_torrent_xpath)
+        seeds = dom.xpath(alternative_seeds_xpath)
+        peers = dom.xpath(alternative_peers_xpath)
+        titles = dom.xpath(alternative_title_xpath)
+        sizes = dom.xpath(alternative_size_xpath)
+        ages = dom.xpath(alternative_age_xpath)
+    # return empty array if nothing is found
+    if not torrent_links:
+        return []
+
+    # parse results
+    for index, result in enumerate(torrent_links):
+        link = result.attrib.get('href')
+        href = urljoin(url, link)
+        results.append({'url': href,
+                        'title': titles[index].text_content(),
+                        'content': '{}, {}'.format(sizes[index], ages[index]),
+                        'seed': seeds[index],
+                        'leech': peers[index],
+
+                        'template': 'torrent.html'})
+
+    # return results sorted by seeder
+    return sorted(results, key=itemgetter('seed'), reverse=True)

searx/engines/soundcloud.py

@@ -10,17 +10,23 @@
  @parse       url, title, content, publishedDate, embedded
 """
 
+import re
 from json import loads
-from urllib import urlencode, quote_plus
+from lxml import html
 from dateutil import parser
+from searx import logger
+from searx.poolrequests import get as http_get
+from searx.url_utils import quote_plus, urlencode
+
+try:
+    from cStringIO import StringIO
+except:
+    from io import StringIO
 
 # engine dependent config
 categories = ['music']
 paging = True
 
-# api-key
-guest_client_id = 'b45b1aa10f1ac2941910a7f0d10f8e28'
-
 # search-url
 url = 'https://api.soundcloud.com/'
 search_url = url + 'search?{query}'\
@@ -34,6 +40,32 @@ embedded_url = '<iframe width="100%" height="166" ' +\
     'scrolling="no" frameborder="no" ' +\
     'data-src="https://w.soundcloud.com/player/?url={uri}"></iframe>'
 
+cid_re = re.compile(r'client_id:"([^"]*)"', re.I | re.U)
+
+
+def get_client_id():
+    response = http_get("https://soundcloud.com")
+
+    if response.ok:
+        tree = html.fromstring(response.content)
+        script_tags = tree.xpath("//script[contains(@src, '/assets/app')]")
+        app_js_urls = [script_tag.get('src') for script_tag in script_tags if script_tag is not None]
+
+        # extracts valid app_js urls from soundcloud.com content
+        for app_js_url in app_js_urls:
+            # gets app_js and searches for the clientid
+            response = http_get(app_js_url)
+            if response.ok:
+                cids = cid_re.search(response.text)
+                if cids is not None and len(cids.groups()):
+                    return cids.groups()[0]
+    logger.warning("Unable to fetch guest client_id from SoundCloud, check parser!")
+    return ""
+
+
+# api-key
+guest_client_id = get_client_id()
+
 
 # do search-request
 def request(query, params):

searx/engines/spotify.py

@@ -11,7 +11,7 @@
 """
 
 from json import loads
-from urllib import urlencode
+from searx.url_utils import urlencode
 
 # engine dependent config
 categories = ['music']
@@ -29,8 +29,7 @@ embedded_url = '<iframe data-src="https://embed.spotify.com/?uri=spotify:track:{
 def request(query, params):
     offset = (params['pageno'] - 1) * 20
 
-    params['url'] = search_url.format(query=urlencode({'q': query}),
-                                      offset=offset)
+    params['url'] = search_url.format(query=urlencode({'q': query}), offset=offset)
 
     return params
 
@@ -46,10 +45,11 @@ def response(resp):
         if result['type'] == 'track':
             title = result['name']
             url = result['external_urls']['spotify']
-            content = result['artists'][0]['name'] +\
-                " &bull; " +\
-                result['album']['name'] +\
-                " &bull; " + result['name']
+            content = u'{} - {} - {}'.format(
+                result['artists'][0]['name'],
+                result['album']['name'],
+                result['name'])
+
             embedded = embedded_url.format(audioid=result['id'])
 
             # append result

searx/engines/stackoverflow.py

@@ -10,11 +10,9 @@
  @parse       url, title, content
 """
 
-from urlparse import urljoin
-from cgi import escape
-from urllib import urlencode
 from lxml import html
 from searx.engines.xpath import extract_text
+from searx.url_utils import urlencode, urljoin
 
 # engine dependent config
 categories = ['it']
@@ -22,7 +20,7 @@ paging = True
 
 # search-url
 url = 'https://stackoverflow.com/'
-search_url = url+'search?{query}&page={pageno}'
+search_url = url + 'search?{query}&page={pageno}'
 
 # specific xpath variables
 results_xpath = '//div[contains(@class,"question-summary")]'
@@ -32,8 +30,7 @@ content_xpath = './/div[@class="excerpt"]'
 
 # do search-request
 def request(query, params):
-    params['url'] = search_url.format(query=urlencode({'q': query}),
-                                      pageno=params['pageno'])
+    params['url'] = search_url.format(query=urlencode({'q': query}), pageno=params['pageno'])
 
     return params
 
@@ -48,8 +45,8 @@ def response(resp):
     for result in dom.xpath(results_xpath):
         link = result.xpath(link_xpath)[0]
         href = urljoin(url, link.attrib.get('href'))
-        title = escape(extract_text(link))
-        content = escape(extract_text(result.xpath(content_xpath)))
+        title = extract_text(link)
+        content = extract_text(result.xpath(content_xpath))
 
         # append result
         results.append({'url': href,

searx/engines/startpage.py

@@ -11,7 +11,6 @@
 # @todo        paging
 
 from lxml import html
-from cgi import escape
 from dateutil import parser
 from datetime import datetime, timedelta
 import re
@@ -48,7 +47,7 @@ def request(query, params):
 
     # set language if specified
     if params['language'] != 'all':
-        params['data']['with_language'] = ('lang_' + params['language'].split('_')[0])
+        params['data']['with_language'] = ('lang_' + params['language'].split('-')[0])
 
     return params
 
@@ -57,7 +56,7 @@ def request(query, params):
 def response(resp):
     results = []
 
-    dom = html.fromstring(resp.content)
+    dom = html.fromstring(resp.text)
 
     # parse results
     for result in dom.xpath(results_xpath):
@@ -68,39 +67,39 @@ def response(resp):
         url = link.attrib.get('href')
 
         # block google-ad url's
-        if re.match("^http(s|)://(www\.)?google\.[a-z]+/aclk.*$", url):
+        if re.match(r"^http(s|)://(www\.)?google\.[a-z]+/aclk.*$", url):
             continue
 
         # block startpage search url's
-        if re.match("^http(s|)://(www\.)?startpage\.com/do/search\?.*$", url):
+        if re.match(r"^http(s|)://(www\.)?startpage\.com/do/search\?.*$", url):
             continue
 
         # block ixquick search url's
-        if re.match("^http(s|)://(www\.)?ixquick\.com/do/search\?.*$", url):
+        if re.match(r"^http(s|)://(www\.)?ixquick\.com/do/search\?.*$", url):
             continue
 
-        title = escape(extract_text(link))
+        title = extract_text(link)
 
         if result.xpath('./p[@class="desc clk"]'):
-            content = escape(extract_text(result.xpath('./p[@class="desc clk"]')))
+            content = extract_text(result.xpath('./p[@class="desc clk"]'))
         else:
             content = ''
 
         published_date = None
 
         # check if search result starts with something like: "2 Sep 2014 ... "
-        if re.match("^([1-9]|[1-2][0-9]|3[0-1]) [A-Z][a-z]{2} [0-9]{4} \.\.\. ", content):
-            date_pos = content.find('...')+4
-            date_string = content[0:date_pos-5]
+        if re.match(r"^([1-9]|[1-2][0-9]|3[0-1]) [A-Z][a-z]{2} [0-9]{4} \.\.\. ", content):
+            date_pos = content.find('...') + 4
+            date_string = content[0:date_pos - 5]
             published_date = parser.parse(date_string, dayfirst=True)
 
             # fix content string
             content = content[date_pos:]
 
         # check if search result starts with something like: "5 days ago ... "
-        elif re.match("^[0-9]+ days? ago \.\.\. ", content):
-            date_pos = content.find('...')+4
-            date_string = content[0:date_pos-5]
+        elif re.match(r"^[0-9]+ days? ago \.\.\. ", content):
+            date_pos = content.find('...') + 4
+            date_string = content[0:date_pos - 5]
 
             # calculate datetime
             published_date = datetime.now() - timedelta(days=int(re.match(r'\d+', date_string).group()))

searx/engines/subtitleseeker.py

@@ -10,11 +10,10 @@
  @parse       url, title, content
 """
 
-from cgi import escape
-from urllib import quote_plus
 from lxml import html
 from searx.languages import language_codes
 from searx.engines.xpath import extract_text
+from searx.url_utils import quote_plus
 
 # engine dependent config
 categories = ['videos']
@@ -23,7 +22,7 @@ language = ""
 
 # search-url
 url = 'http://www.subtitleseeker.com/'
-search_url = url + 'search/TITLES/{query}&p={pageno}'
+search_url = url + 'search/TITLES/{query}?p={pageno}'
 
 # specific xpath variables
 results_xpath = '//div[@class="boxRows"]'
@@ -44,10 +43,16 @@ def response(resp):
 
     search_lang = ""
 
-    if resp.search_params['language'] != 'all':
-        search_lang = [lc[1]
+    # dirty fix for languages named differenly in their site
+    if resp.search_params['language'][:2] == 'fa':
+        search_lang = 'Farsi'
+    elif resp.search_params['language'] == 'pt-BR':
+        search_lang = 'Brazilian'
+    elif resp.search_params['language'] != 'all':
+        search_lang = [lc[3]
                        for lc in language_codes
-                       if lc[0][:2] == resp.search_params['language'].split('_')[0]][0]
+                       if lc[0].split('-')[0] == resp.search_params['language'].split('-')[0]]
+        search_lang = search_lang[0].split(' (')[0]
 
     # parse results
     for result in dom.xpath(results_xpath):
@@ -59,7 +64,7 @@ def response(resp):
         elif search_lang:
             href = href + search_lang + '/'
 
-        title = escape(extract_text(link))
+        title = extract_text(link)
 
         content = extract_text(result.xpath('.//div[contains(@class,"red")]'))
         content = content + " - "
@@ -75,7 +80,7 @@ def response(resp):
         # append result
         results.append({'url': href,
                         'title': title,
-                        'content': escape(content)})
+                        'content': content})
 
     # return results
     return results

searx/engines/swisscows.py

@@ -11,8 +11,9 @@
 """
 
 from json import loads
-from urllib import urlencode, unquote
 import re
+from lxml.html import fromstring
+from searx.url_utils import unquote, urlencode
 
 # engine dependent config
 categories = ['general', 'images']
@@ -23,11 +24,13 @@ language_support = True
 base_url = 'https://swisscows.ch/'
 search_string = '?{query}&page={page}'
 
+supported_languages_url = base_url
+
 # regex
-regex_json = re.compile('initialData: {"Request":(.|\n)*},\s*environment')
-regex_json_remove_start = re.compile('^initialData:\s*')
-regex_json_remove_end = re.compile(',\s*environment$')
-regex_img_url_remove_start = re.compile('^https?://i\.swisscows\.ch/\?link=')
+regex_json = re.compile(b'initialData: {"Request":(.|\n)*},\s*environment')
+regex_json_remove_start = re.compile(b'^initialData:\s*')
+regex_json_remove_end = re.compile(b',\s*environment$')
+regex_img_url_remove_start = re.compile(b'^https?://i\.swisscows\.ch/\?link=')
 
 
 # do search-request
@@ -35,15 +38,16 @@ def request(query, params):
     if params['language'] == 'all':
         ui_language = 'browser'
         region = 'browser'
+    elif params['language'].split('-')[0] == 'no':
+        region = 'nb-NO'
     else:
-        region = params['language'].replace('_', '-')
-        ui_language = params['language'].split('_')[0]
+        region = params['language']
+        ui_language = params['language'].split('-')[0]
 
     search_path = search_string.format(
-        query=urlencode({'query': query,
-                         'uiLanguage': ui_language,
-                         'region': region}),
-        page=params['pageno'])
+        query=urlencode({'query': query, 'uiLanguage': ui_language, 'region': region}),
+        page=params['pageno']
+    )
 
     # image search query is something like 'image?{query}&page={page}'
     if params['category'] == 'images':
@@ -58,14 +62,14 @@ def request(query, params):
 def response(resp):
     results = []
 
-    json_regex = regex_json.search(resp.content)
+    json_regex = regex_json.search(resp.text)
 
     # check if results are returned
     if not json_regex:
         return []
 
-    json_raw = regex_json_remove_end.sub('', regex_json_remove_start.sub('', json_regex.group()))
-    json = loads(json_raw)
+    json_raw = regex_json_remove_end.sub(b'', regex_json_remove_start.sub(b'', json_regex.group()))
+    json = loads(json_raw.decode('utf-8'))
 
     # parse results
     for result in json['Results'].get('items', []):
@@ -73,7 +77,7 @@ def response(resp):
 
         # parse image results
         if result.get('ContentType', '').startswith('image'):
-            img_url = unquote(regex_img_url_remove_start.sub('', result['Url']))
+            img_url = unquote(regex_img_url_remove_start.sub(b'', result['Url'].encode('utf-8')).decode('utf-8'))
 
             # append result
             results.append({'url': result['SourceUrl'],
@@ -95,7 +99,7 @@ def response(resp):
     # parse images
     for result in json.get('Images', []):
         # decode image url
-        img_url = unquote(regex_img_url_remove_start.sub('', result['Url']))
+        img_url = unquote(regex_img_url_remove_start.sub(b'', result['Url'].encode('utf-8')).decode('utf-8'))
 
         # append result
         results.append({'url': result['SourceUrl'],
@@ -106,3 +110,17 @@ def response(resp):
 
     # return results
     return results
+
+
+# get supported languages from their site
+def _fetch_supported_languages(resp):
+    supported_languages = []
+    dom = fromstring(resp.text)
+    options = dom.xpath('//div[@id="regions-popup"]//ul/li/a')
+    for option in options:
+        code = option.xpath('./@data-val')[0]
+        if code.startswith('nb-'):
+            code = code.replace('nb', 'no', 1)
+        supported_languages.append(code)
+
+    return supported_languages

searx/engines/tokyotoshokan.py

@@ -0,0 +1,100 @@
+"""
+ Tokyo Toshokan (A BitTorrent Library for Japanese Media)
+
+ @website      https://www.tokyotosho.info/
+ @provide-api  no
+ @using-api    no
+ @results      HTML
+ @stable       no (HTML can change)
+ @parse        url, title, publishedDate, seed, leech,
+               filesize, magnetlink, content
+"""
+
+import re
+from lxml import html
+from searx.engines.xpath import extract_text
+from datetime import datetime
+from searx.engines.nyaa import int_or_zero, get_filesize_mul
+from searx.url_utils import urlencode
+
+# engine dependent config
+categories = ['files', 'videos', 'music']
+paging = True
+
+# search-url
+base_url = 'https://www.tokyotosho.info/'
+search_url = base_url + 'search.php?{query}'
+
+
+# do search-request
+def request(query, params):
+    query = urlencode({'page': params['pageno'], 'terms': query})
+    params['url'] = search_url.format(query=query)
+    return params
+
+
+# get response from search-request
+def response(resp):
+    results = []
+
+    dom = html.fromstring(resp.text)
+    rows = dom.xpath('//table[@class="listing"]//tr[contains(@class, "category_0")]')
+
+    # check if there are no results or page layout was changed so we cannot parse it
+    # currently there are two rows for each result, so total count must be even
+    if len(rows) == 0 or len(rows) % 2 != 0:
+        return []
+
+    # regular expression for parsing torrent size strings
+    size_re = re.compile(r'Size:\s*([\d.]+)(TB|GB|MB|B)', re.IGNORECASE)
+
+    # processing the results, two rows at a time
+    for i in range(0, len(rows), 2):
+        # parse the first row
+        name_row = rows[i]
+
+        links = name_row.xpath('./td[@class="desc-top"]/a')
+        params = {
+            'template': 'torrent.html',
+            'url': links[-1].attrib.get('href'),
+            'title': extract_text(links[-1])
+        }
+        # I have not yet seen any torrents without magnet links, but
+        # it's better to be prepared to stumble upon one some day
+        if len(links) == 2:
+            magnet = links[0].attrib.get('href')
+            if magnet.startswith('magnet'):
+                # okay, we have a valid magnet link, let's add it to the result
+                params['magnetlink'] = magnet
+
+        # no more info in the first row, start parsing the second one
+        info_row = rows[i + 1]
+        desc = extract_text(info_row.xpath('./td[@class="desc-bot"]')[0])
+        for item in desc.split('|'):
+            item = item.strip()
+            if item.startswith('Size:'):
+                try:
+                    # ('1.228', 'GB')
+                    groups = size_re.match(item).groups()
+                    multiplier = get_filesize_mul(groups[1])
+                    params['filesize'] = int(multiplier * float(groups[0]))
+                except:
+                    pass
+            elif item.startswith('Date:'):
+                try:
+                    # Date: 2016-02-21 21:44 UTC
+                    date = datetime.strptime(item, 'Date: %Y-%m-%d %H:%M UTC')
+                    params['publishedDate'] = date
+                except:
+                    pass
+            elif item.startswith('Comment:'):
+                params['content'] = item
+        stats = info_row.xpath('./td[@class="stats"]/span')
+        # has the layout not changed yet?
+        if len(stats) == 3:
+            params['seed'] = int_or_zero(extract_text(stats[0]))
+            params['leech'] = int_or_zero(extract_text(stats[1]))
+
+        results.append(params)
+
+    return results

searx/engines/torrentz.py

@@ -0,0 +1,92 @@
+"""
+ Torrentz.eu (BitTorrent meta-search engine)
+
+ @website      https://torrentz.eu/
+ @provide-api  no
+
+ @using-api    no
+ @results      HTML
+ @stable       no (HTML can change, although unlikely,
+                   see https://torrentz.eu/torrentz.btsearch)
+ @parse        url, title, publishedDate, seed, leech, filesize, magnetlink
+"""
+
+import re
+from lxml import html
+from datetime import datetime
+from searx.engines.nyaa import int_or_zero, get_filesize_mul
+from searx.engines.xpath import extract_text
+from searx.url_utils import urlencode
+
+# engine dependent config
+categories = ['files', 'videos', 'music']
+paging = True
+
+# search-url
+# https://torrentz.eu/search?f=EXAMPLE&p=6
+base_url = 'https://torrentz.eu/'
+search_url = base_url + 'search?{query}'
+
+
+# do search-request
+def request(query, params):
+    page = params['pageno'] - 1
+    query = urlencode({'q': query, 'p': page})
+    params['url'] = search_url.format(query=query)
+    return params
+
+
+# get response from search-request
+def response(resp):
+    results = []
+
+    dom = html.fromstring(resp.text)
+
+    for result in dom.xpath('//div[@class="results"]/dl'):
+        name_cell = result.xpath('./dt')[0]
+        title = extract_text(name_cell)
+
+        # skip rows that do not contain a link to a torrent
+        links = name_cell.xpath('./a')
+        if len(links) != 1:
+            continue
+
+        # extract url and remove a slash in the beginning
+        link = links[0].attrib.get('href').lstrip('/')
+
+        seed = result.xpath('./dd/span[@class="u"]/text()')[0].replace(',', '')
+        leech = result.xpath('./dd/span[@class="d"]/text()')[0].replace(',', '')
+
+        params = {
+            'url': base_url + link,
+            'title': title,
+            'seed': int_or_zero(seed),
+            'leech': int_or_zero(leech),
+            'template': 'torrent.html'
+        }
+
+        # let's try to calculate the torrent size
+        try:
+            size_str = result.xpath('./dd/span[@class="s"]/text()')[0]
+            size, suffix = size_str.split()
+            params['filesize'] = int(size) * get_filesize_mul(suffix)
+        except:
+            pass
+
+        # does our link contain a valid SHA1 sum?
+        if re.compile('[0-9a-fA-F]{40}').match(link):
+            # add a magnet link to the result
+            params['magnetlink'] = 'magnet:?xt=urn:btih:' + link
+
+        # extract and convert creation date
+        try:
+            date_str = result.xpath('./dd/span[@class="a"]/span')[0].attrib.get('title')
+            # Fri, 25 Mar 2016 16:29:01
+            date = datetime.strptime(date_str, '%a, %d %b %Y %H:%M:%S')
+            params['publishedDate'] = date
+        except:
+            pass
+
+        results.append(params)
+
+    return results

searx/engines/translated.py

@@ -0,0 +1,68 @@
+"""
+ MyMemory Translated
+
+ @website     https://mymemory.translated.net/
+ @provide-api yes (https://mymemory.translated.net/doc/spec.php)
+ @using-api   yes
+ @results     JSON
+ @stable      yes
+ @parse       url, title, content
+"""
+import re
+from sys import version_info
+from searx.utils import is_valid_lang
+
+if version_info[0] == 3:
+    unicode = str
+
+categories = ['general']
+url = u'http://api.mymemory.translated.net/get?q={query}&langpair={from_lang}|{to_lang}{key}'
+web_url = u'http://mymemory.translated.net/en/{from_lang}/{to_lang}/{query}'
+weight = 100
+
+parser_re = re.compile(u'.*?([a-z]+)-([a-z]+) (.{2,})$', re.I)
+api_key = ''
+
+
+def request(query, params):
+    m = parser_re.match(unicode(query, 'utf8'))
+    if not m:
+        return params
+
+    from_lang, to_lang, query = m.groups()
+
+    from_lang = is_valid_lang(from_lang)
+    to_lang = is_valid_lang(to_lang)
+
+    if not from_lang or not to_lang:
+        return params
+
+    if api_key:
+        key_form = '&key=' + api_key
+    else:
+        key_form = ''
+    params['url'] = url.format(from_lang=from_lang[1],
+                               to_lang=to_lang[1],
+                               query=query,
+                               key=key_form)
+    params['query'] = query
+    params['from_lang'] = from_lang
+    params['to_lang'] = to_lang
+
+    return params
+
+
+def response(resp):
+    results = []
+    results.append({
+        'url': web_url.format(
+            from_lang=resp.search_params['from_lang'][2],
+            to_lang=resp.search_params['to_lang'][2],
+            query=resp.search_params['query']),
+        'title': '[{0}-{1}] {2}'.format(
+            resp.search_params['from_lang'][1],
+            resp.search_params['to_lang'][1],
+            resp.search_params['query']),
+        'content': resp.json()['responseData']['translatedText']
+    })
+    return results

searx/engines/twitter.py

@@ -12,11 +12,10 @@
  @todo        publishedDate
 """
 
-from urlparse import urljoin
-from urllib import urlencode
 from lxml import html
 from datetime import datetime
 from searx.engines.xpath import extract_text
+from searx.url_utils import urlencode, urljoin
 
 # engine dependent config
 categories = ['social media']
@@ -28,6 +27,7 @@ search_url = base_url + 'search?'
 
 # specific xpath variables
 results_xpath = '//li[@data-item-type="tweet"]'
+avatar_xpath = './/img[contains(@class, "avatar")]/@src'
 link_xpath = './/small[@class="time"]//a'
 title_xpath = './/span[contains(@class, "username")]'
 content_xpath = './/p[contains(@class, "tweet-text")]'
@@ -40,7 +40,7 @@ def request(query, params):
 
     # set language if specified
     if params['language'] != 'all':
-        params['cookies']['lang'] = params['language'].split('_')[0]
+        params['cookies']['lang'] = params['language'].split('-')[0]
     else:
         params['cookies']['lang'] = 'en'
 
@@ -58,6 +58,8 @@ def response(resp):
         try:
             link = tweet.xpath(link_xpath)[0]
             content = extract_text(tweet.xpath(content_xpath)[0])
+            img_src = tweet.xpath(avatar_xpath)[0]
+            img_src = img_src.replace('_bigger', '_normal')
         except Exception:
             continue
 
@@ -72,12 +74,14 @@ def response(resp):
             results.append({'url': url,
                             'title': title,
                             'content': content,
+                            'img_src': img_src,
                             'publishedDate': publishedDate})
         else:
             # append result
             results.append({'url': url,
                             'title': title,
-                            'content': content})
+                            'content': content,
+                            'img_src': img_src})
 
     # return results
     return results

searx/engines/vimeo.py

@@ -12,28 +12,19 @@
 # @todo        rewrite to api
 # @todo        set content-parameter with correct data
 
-from urllib import urlencode
-from lxml import html
-from HTMLParser import HTMLParser
-from searx.engines.xpath import extract_text
+from json import loads
 from dateutil import parser
+from searx.url_utils import urlencode
 
 # engine dependent config
 categories = ['videos']
 paging = True
 
 # search-url
-base_url = 'https://vimeo.com'
+base_url = 'https://vimeo.com/'
 search_url = base_url + '/search/page:{pageno}?{query}'
 
-# specific xpath variables
-results_xpath = '//div[contains(@class,"results_grid")]/ul/li'
-url_xpath = './/a/@href'
-title_xpath = './/span[@class="title"]'
-thumbnail_xpath = './/img[@class="js-clip_thumbnail_image"]/@src'
-publishedDate_xpath = './/time/attribute::datetime'
-
-embedded_url = '<iframe data-src="//player.vimeo.com/video{videoid}" ' +\
+embedded_url = '<iframe data-src="//player.vimeo.com/video/{videoid}" ' +\
     'width="540" height="304" frameborder="0" ' +\
     'webkitallowfullscreen mozallowfullscreen allowfullscreen></iframe>'
 
@@ -49,17 +40,18 @@ def request(query, params):
 # get response from search-request
 def response(resp):
     results = []
-
-    dom = html.fromstring(resp.text)
-    p = HTMLParser()
+    data_start_pos = resp.text.find('{"filtered"')
+    data_end_pos = resp.text.find(';\n', data_start_pos + 1)
+    data = loads(resp.text[data_start_pos:data_end_pos])
 
     # parse results
-    for result in dom.xpath(results_xpath):
-        videoid = result.xpath(url_xpath)[0]
+    for result in data['filtered']['data']:
+        result = result[result['type']]
+        videoid = result['uri'].split('/')[-1]
         url = base_url + videoid
-        title = p.unescape(extract_text(result.xpath(title_xpath)))
-        thumbnail = extract_text(result.xpath(thumbnail_xpath)[0])
-        publishedDate = parser.parse(extract_text(result.xpath(publishedDate_xpath)[0]))
+        title = result['name']
+        thumbnail = result['pictures']['sizes'][-1]['link']
+        publishedDate = parser.parse(result['created_time'])
         embedded = embedded_url.format(videoid=videoid)
 
         # append result

searx/engines/wikidata.py

@@ -1,56 +1,85 @@
-import json
+# -*- coding: utf-8 -*-
+"""
+ Wikidata
+
+ @website     https://wikidata.org
+ @provide-api yes (https://wikidata.org/w/api.php)
+
+ @using-api   partially (most things require scraping)
+ @results     JSON, HTML
+ @stable      no (html can change)
+ @parse       url, infobox
+"""
 
 from searx import logger
 from searx.poolrequests import get
-from searx.utils import format_date_by_locale
-
-from datetime import datetime
-from dateutil.parser import parse as dateutil_parse
-from urllib import urlencode
+from searx.engines.xpath import extract_text
+from searx.engines.wikipedia import _fetch_supported_languages, supported_languages_url
+from searx.url_utils import urlencode
 
+from json import loads
+from lxml.html import fromstring
 
 logger = logger.getChild('wikidata')
 result_count = 1
+
+# urls
 wikidata_host = 'https://www.wikidata.org'
+url_search = wikidata_host \
+    + '/wiki/Special:ItemDisambiguation?{query}'
+
 wikidata_api = wikidata_host + '/w/api.php'
-url_search = wikidata_api \
-    + '?action=query&list=search&format=json'\
-    + '&srnamespace=0&srprop=sectiontitle&{query}'
 url_detail = wikidata_api\
-    + '?action=wbgetentities&format=json'\
-    + '&props=labels%7Cinfo%7Csitelinks'\
-    + '%7Csitelinks%2Furls%7Cdescriptions%7Cclaims'\
-    + '&{query}'
+    + '?action=parse&format=json&{query}'\
+    + '&redirects=1&prop=text%7Cdisplaytitle%7Clanglinks%7Crevid'\
+    + '&disableeditsection=1&disabletidy=1&preview=1&sectionpreview=1&disabletoc=1&utf8=1&formatversion=2'
+
 url_map = 'https://www.openstreetmap.org/'\
     + '?lat={latitude}&lon={longitude}&zoom={zoom}&layers=M'
+url_image = 'https://commons.wikimedia.org/wiki/Special:FilePath/{filename}?width=500&height=400'
+
+# xpaths
+wikidata_ids_xpath = '//div/ul[@class="wikibase-disambiguation"]/li/a/@title'
+title_xpath = '//*[contains(@class,"wikibase-title-label")]'
+description_xpath = '//div[contains(@class,"wikibase-entitytermsview-heading-description")]'
+property_xpath = '//div[@id="{propertyid}"]'
+label_xpath = './/div[contains(@class,"wikibase-statementgroupview-property-label")]/a'
+url_xpath = './/a[contains(@class,"external free") or contains(@class, "wb-external-id")]'
+wikilink_xpath = './/ul[contains(@class,"wikibase-sitelinklistview-listview")]'\
+    + '/li[contains(@data-wb-siteid,"{wikiid}")]//a/@href'
+property_row_xpath = './/div[contains(@class,"wikibase-statementview")]'
+preferred_rank_xpath = './/span[contains(@class,"wikibase-rankselector-preferred")]'
+value_xpath = './/div[contains(@class,"wikibase-statementview-mainsnak")]'\
+    + '/*/div[contains(@class,"wikibase-snakview-value")]'
+language_fallback_xpath = '//sup[contains(@class,"wb-language-fallback-indicator")]'
+calendar_name_xpath = './/sup[contains(@class,"wb-calendar-name")]'
 
 
 def request(query, params):
+    language = params['language'].split('-')[0]
+    if language == 'all':
+        language = 'en'
+
     params['url'] = url_search.format(
-        query=urlencode({'srsearch': query,
-                        'srlimit': result_count}))
+        query=urlencode({'label': query, 'language': language}))
     return params
 
 
 def response(resp):
     results = []
-    search_res = json.loads(resp.text)
+    html = fromstring(resp.text)
+    wikidata_ids = html.xpath(wikidata_ids_xpath)
 
-    wikidata_ids = set()
-    for r in search_res.get('query', {}).get('search', {}):
-        wikidata_ids.add(r.get('title', ''))
-
-    language = resp.search_params['language'].split('_')[0]
+    language = resp.search_params['language'].split('-')[0]
     if language == 'all':
         language = 'en'
 
-    url = url_detail.format(query=urlencode({'ids': '|'.join(wikidata_ids),
-                                            'languages': language + '|en'}))
-
-    htmlresponse = get(url)
-    jsonresponse = json.loads(htmlresponse.content)
-    for wikidata_id in wikidata_ids:
-        results = results + getDetail(jsonresponse, wikidata_id, language, resp.search_params['language'])
+    # TODO: make requests asynchronous to avoid timeout when result_count > 1
+    for wikidata_id in wikidata_ids[:result_count]:
+        url = url_detail.format(query=urlencode({'page': wikidata_id, 'uselang': language}))
+        htmlresponse = get(url)
+        jsonresponse = loads(htmlresponse.text)
+        results += getDetail(jsonresponse, wikidata_id, language, resp.search_params['language'])
 
     return results
 
@@ -60,124 +89,206 @@ def getDetail(jsonresponse, wikidata_id, language, locale):
     urls = []
     attributes = []
 
-    result = jsonresponse.get('entities', {}).get(wikidata_id, {})
+    title = jsonresponse.get('parse', {}).get('displaytitle', {})
+    result = jsonresponse.get('parse', {}).get('text', {})
 
-    title = result.get('labels', {}).get(language, {}).get('value', None)
-    if title is None:
-        title = result.get('labels', {}).get('en', {}).get('value', None)
-    if title is None:
+    if not title or not result:
         return results
 
-    description = result\
-        .get('descriptions', {})\
-        .get(language, {})\
-        .get('value', None)
+    title = fromstring(title)
+    for elem in title.xpath(language_fallback_xpath):
+        elem.getparent().remove(elem)
+    title = extract_text(title.xpath(title_xpath))
 
-    if description is None:
-        description = result\
-            .get('descriptions', {})\
-            .get('en', {})\
-            .get('value', '')
+    result = fromstring(result)
+    for elem in result.xpath(language_fallback_xpath):
+        elem.getparent().remove(elem)
 
-    claims = result.get('claims', {})
-    official_website = get_string(claims, 'P856', None)
-    if official_website is not None:
-        urls.append({'title': 'Official site', 'url': official_website})
-        results.append({'title': title, 'url': official_website})
+    description = extract_text(result.xpath(description_xpath))
 
+    # URLS
+
+    # official website
+    add_url(urls, result, 'P856', results=results)
+
+    # wikipedia
     wikipedia_link_count = 0
-    if language != 'en':
-        wikipedia_link_count += add_url(urls,
-                                        'Wikipedia (' + language + ')',
-                                        get_wikilink(result, language +
-                                                     'wiki'))
-    wikipedia_en_link = get_wikilink(result, 'enwiki')
-    wikipedia_link_count += add_url(urls,
-                                    'Wikipedia (en)',
-                                    wikipedia_en_link)
-    if wikipedia_link_count == 0:
-        misc_language = get_wiki_firstlanguage(result, 'wiki')
-        if misc_language is not None:
-            add_url(urls,
-                    'Wikipedia (' + misc_language + ')',
-                    get_wikilink(result, misc_language + 'wiki'))
+    wikipedia_link = get_wikilink(result, language + 'wiki')
+    if wikipedia_link:
+        wikipedia_link_count += 1
+        urls.append({'title': 'Wikipedia (' + language + ')',
+                     'url': wikipedia_link})
 
     if language != 'en':
-        add_url(urls,
-                'Wiki voyage (' + language + ')',
-                get_wikilink(result, language + 'wikivoyage'))
+        wikipedia_en_link = get_wikilink(result, 'enwiki')
+        if wikipedia_en_link:
+            wikipedia_link_count += 1
+            urls.append({'title': 'Wikipedia (en)',
+                         'url': wikipedia_en_link})
 
-    add_url(urls,
-            'Wiki voyage (en)',
-            get_wikilink(result, 'enwikivoyage'))
+    # TODO: get_wiki_firstlanguage
+    # if wikipedia_link_count == 0:
 
-    if language != 'en':
-        add_url(urls,
-                'Wikiquote (' + language + ')',
-                get_wikilink(result, language + 'wikiquote'))
+    # more wikis
+    add_url(urls, result, default_label='Wikivoyage (' + language + ')', link_type=language + 'wikivoyage')
+    add_url(urls, result, default_label='Wikiquote (' + language + ')', link_type=language + 'wikiquote')
+    add_url(urls, result, default_label='Wikimedia Commons', link_type='commonswiki')
 
-    add_url(urls,
-            'Wikiquote (en)',
-            get_wikilink(result, 'enwikiquote'))
+    add_url(urls, result, 'P625', 'OpenStreetMap', link_type='geo')
 
-    add_url(urls,
-            'Commons wiki',
-            get_wikilink(result, 'commonswiki'))
+    # musicbrainz
+    add_url(urls, result, 'P434', 'MusicBrainz', 'http://musicbrainz.org/artist/')
+    add_url(urls, result, 'P435', 'MusicBrainz', 'http://musicbrainz.org/work/')
+    add_url(urls, result, 'P436', 'MusicBrainz', 'http://musicbrainz.org/release-group/')
+    add_url(urls, result, 'P966', 'MusicBrainz', 'http://musicbrainz.org/label/')
 
-    add_url(urls,
-            'Location',
-            get_geolink(claims, 'P625', None))
+    # IMDb
+    add_url(urls, result, 'P345', 'IMDb', 'https://www.imdb.com/', link_type='imdb')
+    # source code repository
+    add_url(urls, result, 'P1324')
+    # blog
+    add_url(urls, result, 'P1581')
+    # social media links
+    add_url(urls, result, 'P2397', 'YouTube', 'https://www.youtube.com/channel/')
+    add_url(urls, result, 'P1651', 'YouTube', 'https://www.youtube.com/watch?v=')
+    add_url(urls, result, 'P2002', 'Twitter', 'https://twitter.com/')
+    add_url(urls, result, 'P2013', 'Facebook', 'https://facebook.com/')
+    add_url(urls, result, 'P2003', 'Instagram', 'https://instagram.com/')
 
-    add_url(urls,
-            'Wikidata',
-            'https://www.wikidata.org/wiki/'
-            + wikidata_id + '?uselang=' + language)
+    urls.append({'title': 'Wikidata',
+                 'url': 'https://www.wikidata.org/wiki/'
+                 + wikidata_id + '?uselang=' + language})
 
-    musicbrainz_work_id = get_string(claims, 'P435')
-    if musicbrainz_work_id is not None:
-        add_url(urls,
-                'MusicBrainz',
-                'http://musicbrainz.org/work/'
-                + musicbrainz_work_id)
+    # INFOBOX ATTRIBUTES (ROWS)
 
-    musicbrainz_artist_id = get_string(claims, 'P434')
-    if musicbrainz_artist_id is not None:
-        add_url(urls,
-                'MusicBrainz',
-                'http://musicbrainz.org/artist/'
-                + musicbrainz_artist_id)
+    # DATES
+    # inception date
+    add_attribute(attributes, result, 'P571', date=True)
+    # dissolution date
+    add_attribute(attributes, result, 'P576', date=True)
+    # start date
+    add_attribute(attributes, result, 'P580', date=True)
+    # end date
+    add_attribute(attributes, result, 'P582', date=True)
+    # date of birth
+    add_attribute(attributes, result, 'P569', date=True)
+    # date of death
+    add_attribute(attributes, result, 'P570', date=True)
+    # date of spacecraft launch
+    add_attribute(attributes, result, 'P619', date=True)
+    # date of spacecraft landing
+    add_attribute(attributes, result, 'P620', date=True)
 
-    musicbrainz_release_group_id = get_string(claims, 'P436')
-    if musicbrainz_release_group_id is not None:
-        add_url(urls,
-                'MusicBrainz',
-                'http://musicbrainz.org/release-group/'
-                + musicbrainz_release_group_id)
+    # nationality
+    add_attribute(attributes, result, 'P27')
+    # country of origin
+    add_attribute(attributes, result, 'P495')
+    # country
+    add_attribute(attributes, result, 'P17')
+    # headquarters
+    add_attribute(attributes, result, 'Q180')
 
-    musicbrainz_label_id = get_string(claims, 'P966')
-    if musicbrainz_label_id is not None:
-        add_url(urls,
-                'MusicBrainz',
-                'http://musicbrainz.org/label/'
-                + musicbrainz_label_id)
+    # PLACES
+    # capital
+    add_attribute(attributes, result, 'P36', trim=True)
+    # head of state
+    add_attribute(attributes, result, 'P35', trim=True)
+    # head of government
+    add_attribute(attributes, result, 'P6', trim=True)
+    # type of government
+    add_attribute(attributes, result, 'P122')
+    # official language
+    add_attribute(attributes, result, 'P37')
+    # population
+    add_attribute(attributes, result, 'P1082', trim=True)
+    # area
+    add_attribute(attributes, result, 'P2046')
+    # currency
+    add_attribute(attributes, result, 'P38', trim=True)
+    # heigth (building)
+    add_attribute(attributes, result, 'P2048')
 
-    # musicbrainz_area_id = get_string(claims, 'P982')
-    # P1407 MusicBrainz series ID
-    # P1004 MusicBrainz place ID
-    # P1330 MusicBrainz instrument ID
-    # P1407 MusicBrainz series ID
+    # MEDIA
+    # platform (videogames)
+    add_attribute(attributes, result, 'P400')
+    # author
+    add_attribute(attributes, result, 'P50')
+    # creator
+    add_attribute(attributes, result, 'P170')
+    # director
+    add_attribute(attributes, result, 'P57')
+    # performer
+    add_attribute(attributes, result, 'P175')
+    # developer
+    add_attribute(attributes, result, 'P178')
+    # producer
+    add_attribute(attributes, result, 'P162')
+    # manufacturer
+    add_attribute(attributes, result, 'P176')
+    # screenwriter
+    add_attribute(attributes, result, 'P58')
+    # production company
+    add_attribute(attributes, result, 'P272')
+    # record label
+    add_attribute(attributes, result, 'P264')
+    # publisher
+    add_attribute(attributes, result, 'P123')
+    # original network
+    add_attribute(attributes, result, 'P449')
+    # distributor
+    add_attribute(attributes, result, 'P750')
+    # composer
+    add_attribute(attributes, result, 'P86')
+    # publication date
+    add_attribute(attributes, result, 'P577', date=True)
+    # genre
+    add_attribute(attributes, result, 'P136')
+    # original language
+    add_attribute(attributes, result, 'P364')
+    # isbn
+    add_attribute(attributes, result, 'Q33057')
+    # software license
+    add_attribute(attributes, result, 'P275')
+    # programming language
+    add_attribute(attributes, result, 'P277')
+    # version
+    add_attribute(attributes, result, 'P348', trim=True)
+    # narrative location
+    add_attribute(attributes, result, 'P840')
 
-    postal_code = get_string(claims, 'P281', None)
-    if postal_code is not None:
-        attributes.append({'label': 'Postal code(s)', 'value': postal_code})
+    # LANGUAGES
+    # number of speakers
+    add_attribute(attributes, result, 'P1098')
+    # writing system
+    add_attribute(attributes, result, 'P282')
+    # regulatory body
+    add_attribute(attributes, result, 'P1018')
+    # language code
+    add_attribute(attributes, result, 'P218')
 
-    date_of_birth = get_time(claims, 'P569', locale, None)
-    if date_of_birth is not None:
-        attributes.append({'label': 'Date of birth', 'value': date_of_birth})
+    # OTHER
+    # ceo
+    add_attribute(attributes, result, 'P169', trim=True)
+    # founder
+    add_attribute(attributes, result, 'P112')
+    # legal form (company/organization)
+    add_attribute(attributes, result, 'P1454')
+    # operator
+    add_attribute(attributes, result, 'P137')
+    # crew members (tripulation)
+    add_attribute(attributes, result, 'P1029')
+    # taxon
+    add_attribute(attributes, result, 'P225')
+    # chemical formula
+    add_attribute(attributes, result, 'P274')
+    # winner (sports/contests)
+    add_attribute(attributes, result, 'P1346')
+    # number of deaths
+    add_attribute(attributes, result, 'P1120')
+    # currency code
+    add_attribute(attributes, result, 'P498')
 
-    date_of_death = get_time(claims, 'P570', locale, None)
-    if date_of_death is not None:
-        attributes.append({'label': 'Date of death', 'value': date_of_death})
+    image = add_image(result)
 
     if len(attributes) == 0 and len(urls) == 2 and len(description) == 0:
         results.append({
@@ -188,8 +299,9 @@ def getDetail(jsonresponse, wikidata_id, language, locale):
     else:
         results.append({
                        'infobox': title,
-                       'id': wikipedia_en_link,
+                       'id': wikipedia_link,
                        'content': description,
+                       'img_src': image,
                        'attributes': attributes,
                        'urls': urls
                        })
@@ -197,92 +309,151 @@ def getDetail(jsonresponse, wikidata_id, language, locale):
     return results
 
 
-def add_url(urls, title, url):
-    if url is not None:
-        urls.append({'title': title, 'url': url})
-        return 1
+# only returns first match
+def add_image(result):
+    # P15: route map, P242: locator map, P154: logo, P18: image, P242: map, P41: flag, P2716: collage, P2910: icon
+    property_ids = ['P15', 'P242', 'P154', 'P18', 'P242', 'P41', 'P2716', 'P2910']
+
+    for property_id in property_ids:
+        image = result.xpath(property_xpath.replace('{propertyid}', property_id))
+        if image:
+            image_name = image[0].xpath(value_xpath)
+            image_src = url_image.replace('{filename}', extract_text(image_name[0]))
+            return image_src
+
+
+# setting trim will only returned high ranked rows OR the first row
+def add_attribute(attributes, result, property_id, default_label=None, date=False, trim=False):
+    attribute = result.xpath(property_xpath.replace('{propertyid}', property_id))
+    if attribute:
+
+        if default_label:
+            label = default_label
+        else:
+            label = extract_text(attribute[0].xpath(label_xpath))
+            label = label[0].upper() + label[1:]
+
+        if date:
+            trim = True
+            # remove calendar name
+            calendar_name = attribute[0].xpath(calendar_name_xpath)
+            for calendar in calendar_name:
+                calendar.getparent().remove(calendar)
+
+        concat_values = ""
+        values = []
+        first_value = None
+        for row in attribute[0].xpath(property_row_xpath):
+            if not first_value or not trim or row.xpath(preferred_rank_xpath):
+
+                value = row.xpath(value_xpath)
+                if not value:
+                    continue
+                value = extract_text(value)
+
+                # save first value in case no ranked row is found
+                if trim and not first_value:
+                    first_value = value
+                else:
+                    # to avoid duplicate values
+                    if value not in values:
+                        concat_values += value + ", "
+                        values.append(value)
+
+        if trim and not values:
+            attributes.append({'label': label,
+                               'value': first_value})
+        else:
+            attributes.append({'label': label,
+                               'value': concat_values[:-2]})
+
+
+# requires property_id unless it's a wiki link (defined in link_type)
+def add_url(urls, result, property_id=None, default_label=None, url_prefix=None, results=None, link_type=None):
+    links = []
+
+    # wiki links don't have property in wikidata page
+    if link_type and 'wiki' in link_type:
+            links.append(get_wikilink(result, link_type))
     else:
-        return 0
+        dom_element = result.xpath(property_xpath.replace('{propertyid}', property_id))
+        if dom_element:
+            dom_element = dom_element[0]
+            if not default_label:
+                label = extract_text(dom_element.xpath(label_xpath))
+                label = label[0].upper() + label[1:]
+
+            if link_type == 'geo':
+                links.append(get_geolink(dom_element))
+
+            elif link_type == 'imdb':
+                links.append(get_imdblink(dom_element, url_prefix))
+
+            else:
+                url_results = dom_element.xpath(url_xpath)
+                for link in url_results:
+                    if link is not None:
+                        if url_prefix:
+                            link = url_prefix + extract_text(link)
+                        else:
+                            link = extract_text(link)
+                        links.append(link)
+
+    # append urls
+    for url in links:
+        if url is not None:
+            urls.append({'title': default_label or label,
+                         'url': url})
+            if results is not None:
+                results.append({'title': default_label or label,
+                                'url': url})
 
 
-def get_mainsnak(claims, propertyName):
-    propValue = claims.get(propertyName, {})
-    if len(propValue) == 0:
+def get_imdblink(result, url_prefix):
+    imdb_id = result.xpath(value_xpath)
+    if imdb_id:
+        imdb_id = extract_text(imdb_id)
+        id_prefix = imdb_id[:2]
+        if id_prefix == 'tt':
+            url = url_prefix + 'title/' + imdb_id
+        elif id_prefix == 'nm':
+            url = url_prefix + 'name/' + imdb_id
+        elif id_prefix == 'ch':
+            url = url_prefix + 'character/' + imdb_id
+        elif id_prefix == 'co':
+            url = url_prefix + 'company/' + imdb_id
+        elif id_prefix == 'ev':
+            url = url_prefix + 'event/' + imdb_id
+        else:
+            url = None
+        return url
+
+
+def get_geolink(result):
+    coordinates = result.xpath(value_xpath)
+    if not coordinates:
         return None
+    coordinates = extract_text(coordinates[0])
+    latitude, longitude = coordinates.split(',')
 
-    propValue = propValue[0].get('mainsnak', None)
-    return propValue
-
-
-def get_string(claims, propertyName, defaultValue=None):
-    propValue = claims.get(propertyName, {})
-    if len(propValue) == 0:
-        return defaultValue
-
-    result = []
-    for e in propValue:
-        mainsnak = e.get('mainsnak', {})
-
-        datavalue = mainsnak.get('datavalue', {})
-        if datavalue is not None:
-            result.append(datavalue.get('value', ''))
-
-    if len(result) == 0:
-        return defaultValue
-    else:
-        # TODO handle multiple urls
-        return result[0]
-
-
-def get_time(claims, propertyName, locale, defaultValue=None):
-    propValue = claims.get(propertyName, {})
-    if len(propValue) == 0:
-        return defaultValue
-
-    result = []
-    for e in propValue:
-        mainsnak = e.get('mainsnak', {})
-
-        datavalue = mainsnak.get('datavalue', {})
-        if datavalue is not None:
-            value = datavalue.get('value', '')
-            result.append(value.get('time', ''))
-
-    if len(result) == 0:
-        date_string = defaultValue
-    else:
-        date_string = ', '.join(result)
-
-    try:
-        parsed_date = datetime.strptime(date_string, "+%Y-%m-%dT%H:%M:%SZ")
-    except:
-        if date_string.startswith('-'):
-            return date_string.split('T')[0]
-        try:
-            parsed_date = dateutil_parse(date_string, fuzzy=False, default=False)
-        except:
-            logger.debug('could not parse date %s', date_string)
-            return date_string.split('T')[0]
-
-    return format_date_by_locale(parsed_date, locale)
-
-
-def get_geolink(claims, propertyName, defaultValue=''):
-    mainsnak = get_mainsnak(claims, propertyName)
-
-    if mainsnak is None:
-        return defaultValue
-
-    datatype = mainsnak.get('datatype', '')
-    datavalue = mainsnak.get('datavalue', {})
-
-    if datatype != 'globe-coordinate':
-        return defaultValue
-
-    value = datavalue.get('value', {})
-
-    precision = value.get('precision', 0.0002)
+    # convert to decimal
+    lat = int(latitude[:latitude.find(u'°')])
+    if latitude.find('\'') >= 0:
+        lat += int(latitude[latitude.find(u'°') + 1:latitude.find('\'')] or 0) / 60.0
+    if latitude.find('"') >= 0:
+        lat += float(latitude[latitude.find('\'') + 1:latitude.find('"')] or 0) / 3600.0
+    if latitude.find('S') >= 0:
+        lat *= -1
+    lon = int(longitude[:longitude.find(u'°')])
+    if longitude.find('\'') >= 0:
+        lon += int(longitude[longitude.find(u'°') + 1:longitude.find('\'')] or 0) / 60.0
+    if longitude.find('"') >= 0:
+        lon += float(longitude[longitude.find('\'') + 1:longitude.find('"')] or 0) / 3600.0
+    if longitude.find('W') >= 0:
+        lon *= -1
 
+    # TODO: get precision
+    precision = 0.0002
     # there is no zoom information, deduce from precision (error prone)
     # samples :
     # 13 --> 5
@@ -295,29 +466,23 @@ def get_geolink(claims, propertyName, defaultValue=''):
     if precision < 0.0003:
         zoom = 19
     else:
-        zoom = int(15 - precision*8.8322 + precision*precision*0.625447)
+        zoom = int(15 - precision * 8.8322 + precision * precision * 0.625447)
 
     url = url_map\
-        .replace('{latitude}', str(value.get('latitude', 0)))\
-        .replace('{longitude}', str(value.get('longitude', 0)))\
+        .replace('{latitude}', str(lat))\
+        .replace('{longitude}', str(lon))\
         .replace('{zoom}', str(zoom))
 
     return url
 
 
 def get_wikilink(result, wikiid):
-    url = result.get('sitelinks', {}).get(wikiid, {}).get('url', None)
-    if url is None:
-        return url
-    elif url.startswith('http://'):
+    url = result.xpath(wikilink_xpath.replace('{wikiid}', wikiid))
+    if not url:
+        return None
+    url = url[0]
+    if url.startswith('http://'):
         url = url.replace('http://', 'https://')
     elif url.startswith('//'):
         url = 'https:' + url
     return url
-
-
-def get_wiki_firstlanguage(result, wikipatternid):
-    for k in result.get('sitelinks', {}).keys():
-        if k.endswith(wikipatternid) and len(k) == (2+len(wikipatternid)):
-            return k[0:2]
-    return None

searx/engines/wikipedia.py

@@ -0,0 +1,135 @@
+"""
+ Wikipedia (Web)
+
+ @website     https://{language}.wikipedia.org
+ @provide-api yes
+
+ @using-api   yes
+ @results     JSON
+ @stable      yes
+ @parse       url, infobox
+"""
+
+from json import loads
+from lxml.html import fromstring
+from searx.url_utils import quote, urlencode
+
+# search-url
+base_url = u'https://{language}.wikipedia.org/'
+search_url = base_url + u'w/api.php?'\
+    'action=query'\
+    '&format=json'\
+    '&{query}'\
+    '&prop=extracts|pageimages'\
+    '&exintro'\
+    '&explaintext'\
+    '&pithumbsize=300'\
+    '&redirects'
+supported_languages_url = 'https://meta.wikimedia.org/wiki/List_of_Wikipedias'
+
+
+# set language in base_url
+def url_lang(lang):
+    lang = lang.split('-')[0]
+    if lang == 'all' or lang not in supported_languages:
+        language = 'en'
+    else:
+        language = lang
+
+    return language
+
+
+# do search-request
+def request(query, params):
+    if query.islower():
+        query = u'{0}|{1}'.format(query.decode('utf-8'), query.decode('utf-8').title()).encode('utf-8')
+
+    params['url'] = search_url.format(query=urlencode({'titles': query}),
+                                      language=url_lang(params['language']))
+
+    return params
+
+
+# get first meaningful paragraph
+# this should filter out disambiguation pages and notes above first paragraph
+# "magic numbers" were obtained by fine tuning
+def extract_first_paragraph(content, title, image):
+    first_paragraph = None
+
+    failed_attempts = 0
+    for paragraph in content.split('\n'):
+
+        starts_with_title = paragraph.lower().find(title.lower(), 0, len(title) + 35)
+        length = len(paragraph)
+
+        if length >= 200 or (starts_with_title >= 0 and (image or length >= 150)):
+            first_paragraph = paragraph
+            break
+
+        failed_attempts += 1
+        if failed_attempts > 3:
+            return None
+
+    return first_paragraph
+
+
+# get response from search-request
+def response(resp):
+    results = []
+
+    search_result = loads(resp.text)
+
+    # wikipedia article's unique id
+    # first valid id is assumed to be the requested article
+    for article_id in search_result['query']['pages']:
+        page = search_result['query']['pages'][article_id]
+        if int(article_id) > 0:
+            break
+
+    if int(article_id) < 0:
+        return []
+
+    title = page.get('title')
+
+    image = page.get('thumbnail')
+    if image:
+        image = image.get('source')
+
+    extract = page.get('extract')
+
+    summary = extract_first_paragraph(extract, title, image)
+
+    # link to wikipedia article
+    wikipedia_link = base_url.format(language=url_lang(resp.search_params['language'])) \
+        + 'wiki/' + quote(title.replace(' ', '_').encode('utf8'))
+
+    results.append({'url': wikipedia_link, 'title': title})
+
+    results.append({'infobox': title,
+                    'id': wikipedia_link,
+                    'content': summary,
+                    'img_src': image,
+                    'urls': [{'title': 'Wikipedia', 'url': wikipedia_link}]})
+
+    return results
+
+
+# get supported languages from their site
+def _fetch_supported_languages(resp):
+    supported_languages = {}
+    dom = fromstring(resp.text)
+    tables = dom.xpath('//table[contains(@class,"sortable")]')
+    for table in tables:
+        # exclude header row
+        trs = table.xpath('.//tr')[1:]
+        for tr in trs:
+            td = tr.xpath('./td')
+            code = td[3].xpath('./a')[0].text
+            name = td[2].xpath('./a')[0].text
+            english_name = td[1].xpath('./a')[0].text
+            articles = int(td[4].xpath('./a/b')[0].text.replace(',', ''))
+            # exclude languages with too few articles
+            if articles >= 100:
+                supported_languages[code] = {"name": name, "english_name": english_name, "articles": articles}
+
+    return supported_languages

searx/engines/wolframalpha_api.py

@@ -0,0 +1,129 @@
+# Wolfram Alpha (Science)
+#
+# @website     https://www.wolframalpha.com
+# @provide-api yes (https://api.wolframalpha.com/v2/)
+#
+# @using-api   yes
+# @results     XML
+# @stable      yes
+# @parse       url, infobox
+
+from lxml import etree
+from searx.url_utils import urlencode
+
+# search-url
+search_url = 'https://api.wolframalpha.com/v2/query?appid={api_key}&{query}'
+site_url = 'https://www.wolframalpha.com/input/?{query}'
+api_key = ''  # defined in settings.yml
+
+# xpath variables
+failure_xpath = '/queryresult[attribute::success="false"]'
+input_xpath = '//pod[starts-with(attribute::id, "Input")]/subpod/plaintext'
+pods_xpath = '//pod'
+subpods_xpath = './subpod'
+pod_primary_xpath = './@primary'
+pod_id_xpath = './@id'
+pod_title_xpath = './@title'
+plaintext_xpath = './plaintext'
+image_xpath = './img'
+img_src_xpath = './@src'
+img_alt_xpath = './@alt'
+
+# pods to display as image in infobox
+# this pods do return a plaintext, but they look better and are more useful as images
+image_pods = {'VisualRepresentation',
+              'Illustration'}
+
+
+# do search-request
+def request(query, params):
+    params['url'] = search_url.format(query=urlencode({'input': query}), api_key=api_key)
+    params['headers']['Referer'] = site_url.format(query=urlencode({'i': query}))
+
+    return params
+
+
+# replace private user area characters to make text legible
+def replace_pua_chars(text):
+    pua_chars = {u'\uf522': u'\u2192',  # rigth arrow
+                 u'\uf7b1': u'\u2115',  # set of natural numbers
+                 u'\uf7b4': u'\u211a',  # set of rational numbers
+                 u'\uf7b5': u'\u211d',  # set of real numbers
+                 u'\uf7bd': u'\u2124',  # set of integer numbers
+                 u'\uf74c': 'd',        # differential
+                 u'\uf74d': u'\u212f',  # euler's number
+                 u'\uf74e': 'i',        # imaginary number
+                 u'\uf7d9': '='}        # equals sign
+
+    for k, v in pua_chars.items():
+        text = text.replace(k, v)
+
+    return text
+
+
+# get response from search-request
+def response(resp):
+    results = []
+
+    search_results = etree.XML(resp.text)
+
+    # return empty array if there are no results
+    if search_results.xpath(failure_xpath):
+        return []
+
+    try:
+        infobox_title = search_results.xpath(input_xpath)[0].text
+    except:
+        infobox_title = ""
+
+    pods = search_results.xpath(pods_xpath)
+    result_chunks = []
+    result_content = ""
+    for pod in pods:
+        pod_id = pod.xpath(pod_id_xpath)[0]
+        pod_title = pod.xpath(pod_title_xpath)[0]
+        pod_is_result = pod.xpath(pod_primary_xpath)
+
+        subpods = pod.xpath(subpods_xpath)
+        if not subpods:
+            continue
+
+        # Appends either a text or an image, depending on which one is more suitable
+        for subpod in subpods:
+            content = subpod.xpath(plaintext_xpath)[0].text
+            image = subpod.xpath(image_xpath)
+
+            if content and pod_id not in image_pods:
+
+                if pod_is_result or not result_content:
+                    if pod_id != "Input":
+                        result_content = "%s: %s" % (pod_title, content)
+
+                # if no input pod was found, title is first plaintext pod
+                if not infobox_title:
+                    infobox_title = content
+
+                content = replace_pua_chars(content)
+                result_chunks.append({'label': pod_title, 'value': content})
+
+            elif image:
+                result_chunks.append({'label': pod_title,
+                                      'image': {'src': image[0].xpath(img_src_xpath)[0],
+                                                'alt': image[0].xpath(img_alt_xpath)[0]}})
+
+    if not result_chunks:
+        return []
+
+    title = "Wolfram|Alpha (%s)" % infobox_title
+
+    # append infobox
+    results.append({'infobox': infobox_title,
+                    'attributes': result_chunks,
+                    'urls': [{'title': 'Wolfram|Alpha', 'url': resp.request.headers['Referer']}]})
+
+    # append link to site
+    results.append({'url': resp.request.headers['Referer'],
+                    'title': title,
+                    'content': result_content})
+
+    return results

searx/engines/wolframalpha_noapi.py

@@ -0,0 +1,120 @@
+# Wolfram|Alpha (Science)
+#
+# @website     https://www.wolframalpha.com/
+# @provide-api yes (https://api.wolframalpha.com/v2/)
+#
+# @using-api   no
+# @results     JSON
+# @stable      no
+# @parse       url, infobox
+
+from json import loads
+from time import time
+
+from searx.poolrequests import get as http_get
+from searx.url_utils import urlencode
+
+# search-url
+url = 'https://www.wolframalpha.com/'
+
+search_url = url + 'input/json.jsp'\
+    '?async=false'\
+    '&banners=raw'\
+    '&debuggingdata=false'\
+    '&format=image,plaintext,imagemap,minput,moutput'\
+    '&formattimeout=2'\
+    '&{query}'\
+    '&output=JSON'\
+    '&parsetimeout=2'\
+    '&proxycode={token}'\
+    '&scantimeout=0.5'\
+    '&sponsorcategories=true'\
+    '&statemethod=deploybutton'
+
+referer_url = url + 'input/?{query}'
+
+token = {'value': '',
+         'last_updated': None}
+
+# pods to display as image in infobox
+# this pods do return a plaintext, but they look better and are more useful as images
+image_pods = {'VisualRepresentation',
+              'Illustration',
+              'Symbol'}
+
+
+# seems, wolframalpha resets its token in every hour
+def obtain_token():
+    update_time = time() - (time() % 3600)
+    try:
+        token_response = http_get('https://www.wolframalpha.com/input/api/v1/code?ts=9999999999999999999', timeout=2.0)
+        token['value'] = loads(token_response.text)['code']
+        token['last_updated'] = update_time
+    except:
+        pass
+    return token
+
+
+obtain_token()
+
+
+# do search-request
+def request(query, params):
+    # obtain token if last update was more than an hour
+    if time() - (token['last_updated'] or 0) > 3600:
+        obtain_token()
+    params['url'] = search_url.format(query=urlencode({'input': query}), token=token['value'])
+    params['headers']['Referer'] = referer_url.format(query=urlencode({'i': query}))
+
+    return params
+
+
+# get response from search-request
+def response(resp):
+    results = []
+
+    resp_json = loads(resp.text)
+
+    if not resp_json['queryresult']['success']:
+        return []
+
+    # TODO handle resp_json['queryresult']['assumptions']
+    result_chunks = []
+    infobox_title = ""
+    result_content = ""
+    for pod in resp_json['queryresult']['pods']:
+        pod_id = pod.get('id', '')
+        pod_title = pod.get('title', '')
+        pod_is_result = pod.get('primary', None)
+
+        if 'subpods' not in pod:
+            continue
+
+        if pod_id == 'Input' or not infobox_title:
+            infobox_title = pod['subpods'][0]['plaintext']
+
+        for subpod in pod['subpods']:
+            if subpod['plaintext'] != '' and pod_id not in image_pods:
+                # append unless it's not an actual answer
+                if subpod['plaintext'] != '(requires interactivity)':
+                    result_chunks.append({'label': pod_title, 'value': subpod['plaintext']})
+
+                if pod_is_result or not result_content:
+                    if pod_id != "Input":
+                        result_content = pod_title + ': ' + subpod['plaintext']
+
+            elif 'img' in subpod:
+                result_chunks.append({'label': pod_title, 'image': subpod['img']})
+
+    if not result_chunks:
+        return []
+
+    results.append({'infobox': infobox_title,
+                    'attributes': result_chunks,
+                    'urls': [{'title': 'Wolfram|Alpha', 'url': resp.request.headers['Referer']}]})
+
+    results.append({'url': resp.request.headers['Referer'],
+                    'title': 'Wolfram|Alpha (' + infobox_title + ')',
+                    'content': result_content})
+
+    return results

searx/engines/www1x.py

@@ -10,11 +10,9 @@
  @parse       url, title, thumbnail, img_src, content
 """
 
-from urllib import urlencode
-from urlparse import urljoin
 from lxml import html
-import string
 import re
+from searx.url_utils import urlencode, urljoin
 
 # engine dependent config
 categories = ['images']
@@ -22,7 +20,7 @@ paging = False
 
 # search-url
 base_url = 'https://1x.com'
-search_url = base_url+'/backend/search.php?{query}'
+search_url = base_url + '/backend/search.php?{query}'
 
 
 # do search-request
@@ -55,7 +53,7 @@ def response(resp):
         cur_element += result_part
 
         # fix xml-error
-        cur_element = string.replace(cur_element, '"></a>', '"/></a>')
+        cur_element = cur_element.replace('"></a>', '"/></a>')
 
         dom = html.fromstring(cur_element)
         link = dom.xpath('//a')[0]

searx/engines/www500px.py

@@ -12,12 +12,8 @@
  @todo        rewrite to api
 """
 
-
-from urllib import urlencode
-from urlparse import urljoin
-from lxml import html
-import re
-from searx.engines.xpath import extract_text
+from json import loads
+from searx.url_utils import urlencode, urljoin
 
 # engine dependent config
 categories = ['images']
@@ -25,13 +21,27 @@ paging = True
 
 # search-url
 base_url = 'https://500px.com'
-search_url = base_url + '/search?search?page={pageno}&type=photos&{query}'
+search_url = 'https://api.500px.com/v1/photos/search?type=photos'\
+    '&{query}'\
+    '&image_size%5B%5D=4'\
+    '&image_size%5B%5D=20'\
+    '&image_size%5B%5D=21'\
+    '&image_size%5B%5D=1080'\
+    '&image_size%5B%5D=1600'\
+    '&image_size%5B%5D=2048'\
+    '&include_states=true'\
+    '&formats=jpeg%2Clytro'\
+    '&include_tags=true'\
+    '&exclude_nude=true'\
+    '&page={pageno}'\
+    '&rpp=50'\
+    '&sdk_key=b68e60cff4c929bedea36ca978830c5caca790c3'
 
 
 # do search-request
 def request(query, params):
     params['url'] = search_url.format(pageno=params['pageno'],
-                                      query=urlencode({'q': query}))
+                                      query=urlencode({'term': query}))
 
     return params
 
@@ -40,19 +50,16 @@ def request(query, params):
 def response(resp):
     results = []
 
-    dom = html.fromstring(resp.text)
-    regex = re.compile('3\.jpg.*$')
+    response_json = loads(resp.text)
 
     # parse results
-    for result in dom.xpath('//div[@class="photo"]'):
-        link = result.xpath('.//a')[0]
-        url = urljoin(base_url, link.attrib.get('href'))
-        title = extract_text(result.xpath('.//div[@class="title"]'))
-        thumbnail_src = link.xpath('.//img')[0].attrib.get('src')
-        # To have a bigger thumbnail, uncomment the next line
-        # thumbnail_src = regex.sub('4.jpg', thumbnail_src)
-        content = extract_text(result.xpath('.//div[@class="info"]'))
-        img_src = regex.sub('2048.jpg', thumbnail_src)
+    for result in response_json['photos']:
+        url = urljoin(base_url, result['url'])
+        title = result['name']
+        # last index is the biggest resolution
+        img_src = result['image_url'][-1]
+        thumbnail_src = result['image_url'][0]
+        content = result['description'] or ''
 
         # append result
         results.append({'url': url,

searx/engines/xpath.py

@@ -1,16 +1,24 @@
 from lxml import html
-from urllib import urlencode, unquote
-from urlparse import urlparse, urljoin
 from lxml.etree import _ElementStringResult, _ElementUnicodeResult
 from searx.utils import html_to_text
+from searx.url_utils import unquote, urlencode, urljoin, urlparse
 
 search_url = None
 url_xpath = None
 content_xpath = None
 title_xpath = None
+paging = False
 suggestion_xpath = ''
 results_xpath = ''
 
+# parameters for engines with paging support
+#
+# number of results on each page
+# (only needed if the site requires not a page number, but an offset)
+page_size = 1
+# number of the first page (usually 0 or 1)
+first_page_num = 1
+
 
 '''
 if xpath_results is list, extract the text from each result and concat the list
@@ -23,8 +31,6 @@ if xpath_results is a string element, then it's already done
 def extract_text(xpath_results):
     if type(xpath_results) == list:
         # it's list of result : concat everything using recursive call
-        if not xpath_results:
-            raise Exception('Empty url resultset')
         result = ''
         for e in xpath_results:
             result = result + extract_text(e)
@@ -34,16 +40,20 @@ def extract_text(xpath_results):
         return ''.join(xpath_results)
     else:
         # it's a element
-        return html_to_text(xpath_results.text_content()).strip()
+        text = html.tostring(xpath_results, encoding='unicode', method='text', with_tail=False)
+        text = text.strip().replace('\n', ' ')
+        return ' '.join(text.split())
 
 
 def extract_url(xpath_results, search_url):
+    if xpath_results == []:
+        raise Exception('Empty url resultset')
     url = extract_text(xpath_results)
 
     if url.startswith('//'):
         # add http or https to this kind of url //example.com/
         parsed_search_url = urlparse(search_url)
-        url = parsed_search_url.scheme+url
+        url = u'{0}:{1}'.format(parsed_search_url.scheme, url)
     elif url.startswith('/'):
         # fix relative url to the search engine
         url = urljoin(search_url, url)
@@ -69,15 +79,21 @@ def normalize_url(url):
         p = parsed_url.path
         mark = p.find('/**')
         if mark != -1:
-            return unquote(p[mark+3:]).decode('utf-8')
+            return unquote(p[mark + 3:]).decode('utf-8')
 
     return url
 
 
 def request(query, params):
     query = urlencode({'q': query})[2:]
-    params['url'] = search_url.format(query=query)
+
+    fp = {'query': query}
+    if paging and search_url.find('{pageno}') >= 0:
+        fp['pageno'] = (params['pageno'] - 1) * page_size + first_page_num
+
+    params['url'] = search_url.format(**fp)
     params['query'] = query
+
     return params
 
 
@@ -87,8 +103,8 @@ def response(resp):
     if results_xpath:
         for result in dom.xpath(results_xpath):
             url = extract_url(result.xpath(url_xpath), search_url)
-            title = extract_text(result.xpath(title_xpath)[0])
-            content = extract_text(result.xpath(content_xpath)[0])
+            title = extract_text(result.xpath(title_xpath))
+            content = extract_text(result.xpath(content_xpath))
             results.append({'url': url, 'title': title, 'content': content})
     else:
         for url, title, content in zip(

searx/engines/yacy.py

@@ -13,8 +13,10 @@
 # @todo        parse video, audio and file results
 
 from json import loads
-from urllib import urlencode
 from dateutil import parser
+from searx.url_utils import urlencode
+
+from searx.utils import html_to_text
 
 # engine dependent config
 categories = ['general', 'images']  # TODO , 'music', 'videos', 'files'
@@ -51,7 +53,7 @@ def request(query, params):
 
     # add language tag if specified
     if params['language'] != 'all':
-        params['url'] += '&lr=lang_' + params['language'].split('_')[0]
+        params['url'] += '&lr=lang_' + params['language'].split('-')[0]
 
     return params
 
@@ -88,7 +90,7 @@ def response(resp):
             # append result
             results.append({'url': result['link'],
                             'title': result['title'],
-                            'content': result['description'],
+                            'content': html_to_text(result['description']),
                             'publishedDate': publishedDate})
 
         # TODO parse video, audio and file results

searx/engines/yahoo.py

@@ -11,19 +11,22 @@
  @parse       url, title, content, suggestion
 """
 
-from urllib import urlencode
-from urlparse import unquote
 from lxml import html
 from searx.engines.xpath import extract_text, extract_url
+from searx.url_utils import unquote, urlencode
 
 # engine dependent config
 categories = ['general']
 paging = True
 language_support = True
+time_range_support = True
 
 # search-url
 base_url = 'https://search.yahoo.com/'
 search_url = 'search?{query}&b={offset}&fl=1&vl=lang_{lang}'
+search_url_with_time = 'search?{query}&b={offset}&fl=1&vl=lang_{lang}&age={age}&btf={btf}&fr2=time'
+
+supported_languages_url = 'https://search.yahoo.com/web/advanced'
 
 # specific xpath variables
 results_xpath = "//div[contains(concat(' ', normalize-space(@class), ' '), ' Sr ')]"
@@ -32,6 +35,10 @@ title_xpath = './/h3/a'
 content_xpath = './/div[@class="compText aAbs"]'
 suggestion_xpath = "//div[contains(concat(' ', normalize-space(@class), ' '), ' AlsoTry ')]//a"
 
+time_range_dict = {'day': ['1d', 'd'],
+                   'week': ['1w', 'w'],
+                   'month': ['1m', 'm']}
+
 
 # remove yahoo-specific tracking-url
 def parse_url(url_string):
@@ -51,18 +58,39 @@ def parse_url(url_string):
         return unquote(url_string[start:end])
 
 
+def _get_url(query, offset, language, time_range):
+    if time_range in time_range_dict:
+        return base_url + search_url_with_time.format(offset=offset,
+                                                      query=urlencode({'p': query}),
+                                                      lang=language,
+                                                      age=time_range_dict[time_range][0],
+                                                      btf=time_range_dict[time_range][1])
+    return base_url + search_url.format(offset=offset,
+                                        query=urlencode({'p': query}),
+                                        lang=language)
+
+
+def _get_language(params):
+    if params['language'] == 'all':
+        return 'en'
+    elif params['language'][:2] == 'zh':
+        if params['language'] == 'zh' or params['language'] == 'zh-CH':
+            return 'szh'
+        else:
+            return 'tzh'
+    else:
+        return params['language'].split('-')[0]
+
+
 # do search-request
 def request(query, params):
+    if params['time_range'] and params['time_range'] not in time_range_dict:
+        return params
+
     offset = (params['pageno'] - 1) * 10 + 1
+    language = _get_language(params)
 
-    if params['language'] == 'all':
-        language = 'en'
-    else:
-        language = params['language'].split('_')[0]
-
-    params['url'] = base_url + search_url.format(offset=offset,
-                                                 query=urlencode({'p': query}),
-                                                 lang=language)
+    params['url'] = _get_url(query, offset, language, params['time_range'])
 
     # TODO required?
     params['cookies']['sB'] = 'fl=1&vl=lang_{lang}&sh=1&rw=new&v=1'\
@@ -77,6 +105,13 @@ def response(resp):
 
     dom = html.fromstring(resp.text)
 
+    try:
+        results_num = int(dom.xpath('//div[@class="compPagination"]/span[last()]/text()')[0]
+                          .split()[0].replace(',', ''))
+        results.append({'number_of_results': results_num})
+    except:
+        pass
+
     # parse results
     for result in dom.xpath(results_xpath):
         try:
@@ -104,3 +139,15 @@ def response(resp):
 
     # return results
     return results
+
+
+# get supported languages from their site
+def _fetch_supported_languages(resp):
+    supported_languages = []
+    dom = html.fromstring(resp.text)
+    options = dom.xpath('//div[@id="yschlang"]/span/label/input')
+    for option in options:
+        code = option.xpath('./@value')[0][5:].replace('_', '-')
+        supported_languages.append(code)
+
+    return supported_languages

searx/engines/yahoo_news.py

@@ -9,13 +9,13 @@
 # @stable      no (HTML can change)
 # @parse       url, title, content, publishedDate
 
-from urllib import urlencode
+import re
+from datetime import datetime, timedelta
 from lxml import html
 from searx.engines.xpath import extract_text, extract_url
-from searx.engines.yahoo import parse_url
-from datetime import datetime, timedelta
-import re
+from searx.engines.yahoo import parse_url, _fetch_supported_languages, supported_languages_url
 from dateutil import parser
+from searx.url_utils import urlencode
 
 # engine dependent config
 categories = ['news']
@@ -55,7 +55,7 @@ def request(query, params):
 
 def sanitize_url(url):
     if ".yahoo.com/" in url:
-        return re.sub(u"\;\_ylt\=.+$", "", url)
+        return re.sub(u"\\;\\_ylt\\=.+$", "", url)
     else:
         return url
 
@@ -80,16 +80,19 @@ def response(resp):
 
         # still useful ?
         if re.match("^[0-9]+ minute(s|) ago$", publishedDate):
-            publishedDate = datetime.now() - timedelta(minutes=int(re.match(r'\d+', publishedDate).group()))  # noqa
+            publishedDate = datetime.now() - timedelta(minutes=int(re.match(r'\d+', publishedDate).group()))
+        elif re.match("^[0-9]+ days? ago$", publishedDate):
+            publishedDate = datetime.now() - timedelta(days=int(re.match(r'\d+', publishedDate).group()))
+        elif re.match("^[0-9]+ hour(s|), [0-9]+ minute(s|) ago$", publishedDate):
+            timeNumbers = re.findall(r'\d+', publishedDate)
+            publishedDate = datetime.now()\
+                - timedelta(hours=int(timeNumbers[0]))\
+                - timedelta(minutes=int(timeNumbers[1]))
         else:
-            if re.match("^[0-9]+ hour(s|), [0-9]+ minute(s|) ago$",
-                        publishedDate):
-                timeNumbers = re.findall(r'\d+', publishedDate)
-                publishedDate = datetime.now()\
-                    - timedelta(hours=int(timeNumbers[0]))\
-                    - timedelta(minutes=int(timeNumbers[1]))
-            else:
+            try:
                 publishedDate = parser.parse(publishedDate)
+            except:
+                publishedDate = datetime.now()
 
         if publishedDate.year == 1900:
             publishedDate = publishedDate.replace(year=datetime.now().year)

searx/engines/yandex.py

@@ -9,9 +9,9 @@
  @parse       url, title, content
 """
 
-from urllib import urlencode
 from lxml import html
-from searx.search import logger
+from searx import logger
+from searx.url_utils import urlencode
 
 logger = logger.getChild('yandex engine')
 
@@ -22,23 +22,25 @@ language_support = True  # TODO
 
 default_tld = 'com'
 language_map = {'ru': 'ru',
-                'ua': 'uk',
+                'ua': 'ua',
+                'be': 'by',
+                'kk': 'kz',
                 'tr': 'com.tr'}
 
 # search-url
 base_url = 'https://yandex.{tld}/'
 search_url = 'search/?{query}&p={page}'
 
-results_xpath = '//div[@class="serp-item serp-item_plain_yes clearfix i-bem"]'
+results_xpath = '//li[@class="serp-item"]'
 url_xpath = './/h2/a/@href'
 title_xpath = './/h2/a//text()'
-content_xpath = './/div[@class="serp-item__text"]//text()'
+content_xpath = './/div[@class="text-container typo typo_text_m typo_line_m organic__text"]//text()'
 
 
 def request(query, params):
-    lang = params['language'].split('_')[0]
+    lang = params['language'].split('-')[0]
     host = base_url.format(tld=language_map.get(lang) or default_tld)
-    params['url'] = host + search_url.format(page=params['pageno']-1,
+    params['url'] = host + search_url.format(page=params['pageno'] - 1,
                                              query=urlencode({'text': query}))
     return params
 

searx/engines/youtube_api.py

@@ -9,8 +9,8 @@
 # @parse       url, title, content, publishedDate, thumbnail, embedded
 
 from json import loads
-from urllib import urlencode
 from dateutil import parser
+from searx.url_utils import urlencode
 
 # engine dependent config
 categories = ['videos', 'music']
@@ -36,7 +36,7 @@ def request(query, params):
 
     # add language tag if specified
     if params['language'] != 'all':
-        params['url'] += '&relevanceLanguage=' + params['language'].split('_')[0]
+        params['url'] += '&relevanceLanguage=' + params['language'].split('-')[0]
 
     return params
 

searx/engines/youtube_noapi.py

@@ -8,19 +8,25 @@
 # @stable      no
 # @parse       url, title, content, publishedDate, thumbnail, embedded
 
-from urllib import quote_plus
 from lxml import html
 from searx.engines.xpath import extract_text
 from searx.utils import list_get
+from searx.url_utils import quote_plus
 
 # engine dependent config
 categories = ['videos', 'music']
 paging = True
 language_support = False
+time_range_support = True
 
 # search-url
 base_url = 'https://www.youtube.com/results'
 search_url = base_url + '?search_query={query}&page={page}'
+time_range_url = '&sp=EgII{time_range}%253D%253D'
+time_range_dict = {'day': 'Ag',
+                   'week': 'Aw',
+                   'month': 'BA',
+                   'year': 'BQ'}
 
 embedded_url = '<iframe width="540" height="304" ' +\
     'data-src="//www.youtube-nocookie.com/embed/{videoid}" ' +\
@@ -47,6 +53,8 @@ def extract_text_from_dom(result, xpath):
 def request(query, params):
     params['url'] = search_url.format(query=quote_plus(query),
                                       page=params['pageno'])
+    if params['time_range'] in time_range_dict:
+        params['url'] += time_range_url.format(time_range=time_range_dict[params['time_range']])
 
     return params
 

searx/exceptions.py

@@ -0,0 +1,32 @@
+'''
+searx is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+searx is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with searx. If not, see < http://www.gnu.org/licenses/ >.
+
+(C) 2017- by Alexandre Flament, <alex@al-f.net>
+'''
+
+
+class SearxException(Exception):
+    pass
+
+
+class SearxParameterException(SearxException):
+
+    def __init__(self, name, value):
+        if value == '' or value is None:
+            message = 'Empty ' + name + ' parameter'
+        else:
+            message = 'Invalid value "' + value + '" for parameter ' + name
+        super(SearxParameterException, self).__init__(message)
+        self.parameter_name = name
+        self.parameter_value = value

searx/languages.py

@@ -1,77 +1,77 @@
-'''
-searx is free software: you can redistribute it and/or modify
-it under the terms of the GNU Affero General Public License as published by
-the Free Software Foundation, either version 3 of the License, or
-(at your option) any later version.
-
-searx is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU Affero General Public License for more details.
-
-You should have received a copy of the GNU Affero General Public License
-along with searx. If not, see < http://www.gnu.org/licenses/ >.
-
-(C) 2013- by Adam Tauber, <asciimoo@gmail.com>
-'''
-
+# -*- coding: utf-8 -*-
 # list of language codes
+# this file is generated automatically by utils/update_search_languages.py
+
 language_codes = (
-    ("ar_XA", "Arabic", "Arabia"),
-    ("bg_BG", "Bulgarian", "Bulgaria"),
-    ("cs_CZ", "Czech", "Czech Republic"),
-    ("de_DE", "German", "Germany"),
-    ("da_DK", "Danish", "Denmark"),
-    ("de_AT", "German", "Austria"),
-    ("de_CH", "German", "Switzerland"),
-    ("el_GR", "Greek", "Greece"),
-    ("en_AU", "English", "Australia"),
-    ("en_CA", "English", "Canada"),
-    ("en_GB", "English", "United Kingdom"),
-    ("en_ID", "English", "Indonesia"),
-    ("en_IE", "English", "Ireland"),
-    ("en_IN", "English", "India"),
-    ("en_MY", "English", "Malaysia"),
-    ("en_NZ", "English", "New Zealand"),
-    ("en_PH", "English", "Philippines"),
-    ("en_SG", "English", "Singapore"),
-    ("en_US", "English", "United States"),
-    ("en_XA", "English", "Arabia"),
-    ("en_ZA", "English", "South Africa"),
-    ("es_AR", "Spanish", "Argentina"),
-    ("es_CL", "Spanish", "Chile"),
-    ("es_ES", "Spanish", "Spain"),
-    ("es_MX", "Spanish", "Mexico"),
-    ("es_US", "Spanish", "United States"),
-    ("es_XL", "Spanish", "Latin America"),
-    ("et_EE", "Estonian", "Estonia"),
-    ("fi_FI", "Finnish", "Finland"),
-    ("fr_BE", "French", "Belgium"),
-    ("fr_CA", "French", "Canada"),
-    ("fr_CH", "French", "Switzerland"),
-    ("fr_FR", "French", "France"),
-    ("he_IL", "Hebrew", "Israel"),
-    ("hr_HR", "Croatian", "Croatia"),
-    ("hu_HU", "Hungarian", "Hungary"),
-    ("it_IT", "Italian", "Italy"),
-    ("ja_JP", "Japanese", "Japan"),
-    ("ko_KR", "Korean", "Korea"),
-    ("lt_LT", "Lithuanian", "Lithuania"),
-    ("lv_LV", "Latvian", "Latvia"),
-    ("nb_NO", "Norwegian", "Norway"),
-    ("nl_BE", "Dutch", "Belgium"),
-    ("nl_NL", "Dutch", "Netherlands"),
-    ("pl_PL", "Polish", "Poland"),
-    ("pt_BR", "Portuguese", "Brazil"),
-    ("pt_PT", "Portuguese", "Portugal"),
-    ("ro_RO", "Romanian", "Romania"),
-    ("ru_RU", "Russian", "Russia"),
-    ("sk_SK", "Slovak", "Slovak Republic"),
-    ("sl_SL", "Slovenian", "Slovenia"),
-    ("sv_SE", "Swedish", "Sweden"),
-    ("th_TH", "Thai", "Thailand"),
-    ("tr_TR", "Turkish", "Turkey"),
-    ("uk_UA", "Ukrainian", "Ukraine"),
-    ("zh_CN", "Chinese", "China"),
-    ("zh_HK", "Chinese", "Hong Kong SAR"),
-    ("zh_TW", "Chinese", "Taiwan"))
+    (u"ar-SA", u"العربية", u"", u"Arabic"),
+    (u"bg-BG", u"Български", u"", u"Bulgarian"),
+    (u"cs-CZ", u"Čeština", u"", u"Czech"),
+    (u"da-DK", u"Dansk", u"", u"Danish"),
+    (u"de", u"Deutsch", u"", u"German"),
+    (u"de-AT", u"Deutsch", u"Österreich", u"German"),
+    (u"de-CH", u"Deutsch", u"Schweiz", u"German"),
+    (u"de-DE", u"Deutsch", u"Deutschland", u"German"),
+    (u"el-GR", u"Ελληνικά", u"", u"Greek"),
+    (u"en", u"English", u"", u"English"),
+    (u"en-AU", u"English", u"Australia", u"English"),
+    (u"en-CA", u"English", u"Canada", u"English"),
+    (u"en-CY", u"English", u"Cyprus", u"English"),
+    (u"en-GB", u"English", u"United Kingdom", u"English"),
+    (u"en-GD", u"English", u"Grenada", u"English"),
+    (u"en-ID", u"English", u"Indonesia", u"English"),
+    (u"en-IE", u"English", u"Ireland", u"English"),
+    (u"en-IN", u"English", u"India", u"English"),
+    (u"en-MY", u"English", u"Malaysia", u"English"),
+    (u"en-NZ", u"English", u"New Zealand", u"English"),
+    (u"en-PH", u"English", u"Philippines", u"English"),
+    (u"en-SG", u"English", u"Singapore", u"English"),
+    (u"en-US", u"English", u"United States", u"English"),
+    (u"en-ZA", u"English", u"South Africa", u"English"),
+    (u"es", u"Español", u"", u"Spanish"),
+    (u"es-AR", u"Español", u"Argentina", u"Spanish"),
+    (u"es-CL", u"Español", u"Chile", u"Spanish"),
+    (u"es-CO", u"Español", u"Colombia", u"Spanish"),
+    (u"es-ES", u"Español", u"España", u"Spanish"),
+    (u"es-MX", u"Español", u"México", u"Spanish"),
+    (u"es-PE", u"Español", u"Perú", u"Spanish"),
+    (u"es-US", u"Español", u"Estados Unidos", u"Spanish"),
+    (u"et-EE", u"Eesti", u"", u"Estonian"),
+    (u"fi-FI", u"Suomi", u"", u"Finnish"),
+    (u"fr", u"Français", u"", u"French"),
+    (u"fr-BE", u"Français", u"Belgique", u"French"),
+    (u"fr-CA", u"Français", u"Canada", u"French"),
+    (u"fr-CH", u"Français", u"Suisse", u"French"),
+    (u"fr-FR", u"Français", u"France", u"French"),
+    (u"he-IL", u"עברית", u"", u"Hebrew"),
+    (u"hr-HR", u"Hrvatski", u"", u"Croatian"),
+    (u"hu-HU", u"Magyar", u"", u"Hungarian"),
+    (u"id-ID", u"Bahasa Indonesia", u"", u"Indonesian"),
+    (u"it", u"Italiano", u"", u"Italian"),
+    (u"it-CH", u"Italiano", u"Svizzera", u"Italian"),
+    (u"it-IT", u"Italiano", u"Italia", u"Italian"),
+    (u"ja-JP", u"日本語", u"", u"Japanese"),
+    (u"ko-KR", u"한국어", u"", u"Korean"),
+    (u"lt-LT", u"Lietuvių", u"", u"Lithuanian"),
+    (u"lv-LV", u"Latviešu", u"", u"Latvian"),
+    (u"ms-MY", u"Bahasa Melayu", u"", u"Malay"),
+    (u"nl", u"Nederlands", u"", u"Dutch"),
+    (u"nl-BE", u"Nederlands", u"België", u"Dutch"),
+    (u"nl-NL", u"Nederlands", u"Nederland", u"Dutch"),
+    (u"no-NO", u"Norsk", u"", u"Norwegian"),
+    (u"pl-PL", u"Polski", u"", u"Polish"),
+    (u"pt", u"Português", u"", u"Portuguese"),
+    (u"pt-BR", u"Português", u"Brasil", u"Portuguese"),
+    (u"pt-PT", u"Português", u"Portugal", u"Portuguese"),
+    (u"ro-RO", u"Română", u"", u"Romanian"),
+    (u"ru-RU", u"Русский", u"", u"Russian"),
+    (u"sk-SK", u"Slovenčina", u"", u"Slovak"),
+    (u"sl", u"Slovenščina", u"", u"Slovenian"),
+    (u"sv-SE", u"Svenska", u"", u"Swedish"),
+    (u"th-TH", u"ไทย", u"", u"Thai"),
+    (u"tr-TR", u"Türkçe", u"", u"Turkish"),
+    (u"vi-VN", u"Tiếng Việt", u"", u"Vietnamese"),
+    (u"zh", u"中文", u"", u"Chinese"),
+    (u"zh-CN", u"中文", u"中国", u"Chinese"),
+    (u"zh-HK", u"中文", u"香港", u"Chinese"),
+    (u"zh-TW", u"中文", u"台湾", u"Chinese")
+)

searx/plugins/__init__.py

@@ -14,18 +14,25 @@ along with searx. If not, see < http://www.gnu.org/licenses/ >.
 
 (C) 2015 by Adam Tauber, <asciimoo@gmail.com>
 '''
-from sys import exit
+from sys import exit, version_info
 from searx import logger
 
+if version_info[0] == 3:
+    unicode = str
+
 logger = logger.getChild('plugins')
 
-from searx.plugins import (https_rewrite,
+from searx.plugins import (doai_rewrite,
+                           https_rewrite,
+                           infinite_scroll,
+                           open_results_on_new_tab,
                            self_info,
                            search_on_category_select,
-                           tracker_url_remover)
+                           tracker_url_remover,
+                           vim_hotkeys)
 
-required_attrs = (('name', str),
-                  ('description', str),
+required_attrs = (('name', (str, unicode)),
+                  ('description', (str, unicode)),
                   ('default_on', bool))
 
 optional_attrs = (('js_dependencies', tuple),
@@ -59,9 +66,9 @@ class PluginStore():
             plugin.id = plugin.name.replace(' ', '_')
             self.plugins.append(plugin)
 
-    def call(self, plugin_type, request, *args, **kwargs):
+    def call(self, ordered_plugin_list, plugin_type, request, *args, **kwargs):
         ret = True
-        for plugin in request.user_plugins:
+        for plugin in ordered_plugin_list:
             if hasattr(plugin, plugin_type):
                 ret = getattr(plugin, plugin_type)(request, *args, **kwargs)
                 if not ret:
@@ -71,7 +78,11 @@ class PluginStore():
 
 
 plugins = PluginStore()
+plugins.register(doai_rewrite)
 plugins.register(https_rewrite)
+plugins.register(infinite_scroll)
+plugins.register(open_results_on_new_tab)
 plugins.register(self_info)
 plugins.register(search_on_category_select)
 plugins.register(tracker_url_remover)
+plugins.register(vim_hotkeys)

searx/plugins/doai_rewrite.py

@@ -0,0 +1,31 @@
+from flask_babel import gettext
+import re
+from searx.url_utils import urlparse, parse_qsl
+
+regex = re.compile(r'10\.\d{4,9}/[^\s]+')
+
+name = gettext('DOAI rewrite')
+description = gettext('Avoid paywalls by redirecting to open-access versions of publications when available')
+default_on = False
+
+
+def extract_doi(url):
+    match = regex.search(url.path)
+    if match:
+        return match.group(0)
+    for _, v in parse_qsl(url.query):
+        match = regex.search(v)
+        if match:
+            return match.group(0)
+    return None
+
+
+def on_result(request, search, result):
+    doi = extract_doi(result['parsed_url'])
+    if doi and len(doi) < 50:
+        for suffix in ('/', '.pdf', '/full', '/meta', '/abstract'):
+            if doi.endswith(suffix):
+                doi = doi[:-len(suffix)]
+        result['url'] = 'http://doai.io/' + doi
+        result['parsed_url'] = urlparse(result['url'])
+    return True

searx/plugins/https_rewrite.py

@@ -16,14 +16,17 @@ along with searx. If not, see < http://www.gnu.org/licenses/ >.
 '''
 
 import re
-from urlparse import urlparse
+import sys
 from lxml import etree
 from os import listdir, environ
 from os.path import isfile, isdir, join
 from searx.plugins import logger
-from flask.ext.babel import gettext
+from flask_babel import gettext
 from searx import searx_dir
+from searx.url_utils import urlparse
 
+if sys.version_info[0] == 3:
+    unicode = str
 
 name = "HTTPS rewrite"
 description = gettext('Rewrite HTTP links to HTTPS if possible')
@@ -87,7 +90,7 @@ def load_single_https_ruleset(rules_path):
 
             # convert host-rule to valid regex
             host = ruleset.attrib.get('host')\
-                .replace('.', '\.').replace('*', '.*')
+                .replace('.', r'\.').replace('*', '.*')
 
             # append to host list
             hosts.append(host)
@@ -103,10 +106,10 @@ def load_single_https_ruleset(rules_path):
             # into a valid python regex group
             rule_from = ruleset.attrib['from'].replace('$', '\\')
             if rule_from.endswith('\\'):
-                rule_from = rule_from[:-1]+'$'
+                rule_from = rule_from[:-1] + '$'
             rule_to = ruleset.attrib['to'].replace('$', '\\')
             if rule_to.endswith('\\'):
-                rule_to = rule_to[:-1]+'$'
+                rule_to = rule_to[:-1] + '$'
 
             # TODO, not working yet because of the hack above,
             # currently doing that in webapp.py
@@ -220,8 +223,7 @@ def https_url_rewrite(result):
     return result
 
 
-def on_result(request, ctx):
-    result = ctx['result']
+def on_result(request, search, result):
     if result['parsed_url'].scheme == 'http':
         https_url_rewrite(result)
     return True