Many things have been changed since last review of this engine. This patch fix
xpath selectors, implements suggestion and is a complete review / rewrite of the
engine.
Signed-off-by: Markus Heiser <markus@darmarit.de>
When initing engines a "SearxEngineResponseException" is logged very verbose,
including full traceback information:
ERROR:searx.engines:yggtorrent engine: Fail to initialize
Traceback (most recent call last):
File "share/searx/searx/engines/__init__.py", line 293, in engine_init
init_fn(get_engine_from_settings(engine_name))
File "share/searx/searx/engines/yggtorrent.py", line 42, in init
resp = http_get(url, allow_redirects=False)
File "share/searx/searx/poolrequests.py", line 197, in get
return request('get', url, **kwargs)
File "share/searx/searx/poolrequests.py", line 190, in request
raise_for_httperror(response)
File "share/searx/searx/raise_for_httperror.py", line 60, in raise_for_httperror
raise_for_captcha(resp)
File "share/searx/searx/raise_for_httperror.py", line 43, in raise_for_captcha
raise_for_cloudflare_captcha(resp)
File "share/searx/searx/raise_for_httperror.py", line 30, in raise_for_cloudflare_captcha
raise SearxEngineCaptchaException(message='Cloudflare CAPTCHA', suspended_time=3600 * 24 * 15)
searx.exceptions.SearxEngineCaptchaException: Cloudflare CAPTCHA, suspended_time=1296000
For SearxEngineResponseException this is not needed. Those types of exceptions
can be a normal use case. E.g. for CAPTCHA errors like shown in the example
above. It should be enough to log a warning for such issues:
WARNING:searx.engines:yggtorrent engine: Fail to initialize // Cloudflare CAPTCHA, suspended_time=1296000
closes: #2612
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Newer CentOS-7 images from https://images.linuxcontainers.org do no longer
include the which command.
Issue:
$ sudo -H ./utils/lxc.sh cmd searx-centos7 ./utils/filtron.sh install all
INFO: [searx-centos7] ./utils/filtron.sh install all
...
Install Go in user's HOME
-------------------------
download and install go binary ..
...
-bash: line 1: which: command not found
-->|ERROR - Go Installation not found in PATH!?!
-bash: line 2: which: command not found
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The wrong and unnecessary `pip install .` is executed in /usr/local/searx and is
responsible for the error message:
ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /usr/local/searx
The correct pip-install comes right after changing to `cd ${SEARX_SRC}`.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
LTS distros like Ubuntu 18.04 do not ship a up-to-date version of git.::
$ sudo -H ./utils/lxc.sh cmd searx-ubu1804 git --version
...
git version 2.17.1
The option `--show-current` was added in git v2.22, the alternative to this
option is::
git rev-parse --abbrev-ref HEAD
Issue when using option `--show-current`::
[searx-ubu1804] Clone searx sources
[searx-ubu1804] -------------------
[searx-ubu1804]
[searx-ubu1804] error: unknown option `show-current'
[searx-ubu1804] usage: git branch [<options>] [-r | -a] [--merged | --no-merged]
....
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
EOL of Ubuntu 16.04 (Xenial Xerus) is in April 2021 but we do not support 16.04
since a longer time. Issues are comming from dependencies (e.g. pip)::
[searx-ubu1604] |searx| SyntaxError: invalid syntax
[searx-ubu1604] |searx| Traceback (most recent call last):
[searx-ubu1604] |searx| File "/usr/local/searx/searx-pyenv/bin/pip", line 7, in <module>
[searx-ubu1604] |searx| from pip._internal.cli.main import main
[searx-ubu1604] |searx| File "/usr/local/searx/searx-pyenv/lib/python3.5/site-packages/pip/_internal/cli/main.py", line 60
[searx-ubu1604] |searx| sys.stderr.write(f"ERROR: {exc}")
[searx-ubu1604] |searx| ^
[searx-ubu1604] |searx| SyntaxError: invalid syntax
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The old xpath configuration for google scholar did not work and is replaced by a
python implementation.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- unittest2 is a backport of the new features added to the unittest testing
framework in Python 2.7
- unittest2 was only needed in py2 and can be dropped now
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The new version of MetaGer needs to reload the reults (into a iframe) with a
unique tag (see HTML response below).
Implementing a dedicated metager-engine for searx makes no sense to me. The
great days of MetaGer seems to be ended. I remember the good old days this
project started in the 90's of the last century. But in the last few years it
becomes more and more crap. As the name suggested, MetaGer was made for
germans in the first place. They have added a english and spain translation but
the i18n is very poor compared to what searx offers.
It's a pity, lets drop MetaGer.
This is the first response, the id (b82679980656899ba5a17ffd02a56846) is unique
for each query:
$ curl "https://metager.org/meta/meta.ger3?eingabe=foo&submit-query=&focus=web"
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<link rel="stylesheet" href="/index.css?id=b82679980656899ba5a17ffd02a56846">
<script src="/index.js?id=b82679980656899ba5a17ffd02a56846"></script>
<title>foo - MetaGer</title>
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />
</head>
<body>
<iframe id="mg-framed" src="https://metager.org/meta/meta.ger3?eingabe=foo&submit-query=&focus=web&mgv=b82679980656899ba5a17ffd02a56846" autofocus="true" onload="this.contentWindow.focus();"></iframe>
</body>
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Error:
Configuration error:
There is a programmable error in your configuration file:
...
NameError: name 'DOCS_URL' is not defined
make: *** [utils/makefile.sphinx:156: books/user.latex] Fehler 2
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
aka: ensure that 'make test' works as expected
The cache contains a copy './local' which is - under some circumstance -
corrupted. It is not possible to clear the cache [1] (see the top of the page).
Ensure that 'make test' works as expected [2] even if
- the python interpreter is missing
- the virtualenv exists but pyyaml is missing
To hardening when the workflow cache fails, this patch adds the new target
'travis.test' into the workflow. This target probes to import a python module
'yaml'. If this fails the virtualenv will be completely new build.
[1] https://github.com/actions/cache/issues/2#issuecomment-673493515
[2] https://github.com/searx/searx/pull/2517#discussion_r567240235
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Target pip-exe is a prerequisite of the targets:
- pyinstall
- pyuninstall
and was accidentally deleted in commit 9b48ae47.
HINT:
do not confuse pyinstall with penvinstall
pyinstall & pyuninstall
Installing into user's HOME using pip from OS,
therefore the message is needed.
pyenvinstall & pyenvuninstall
Installing into virtualenv (./local) using pip which is provided by
prerequisite 'pyenv' in the virtualenv.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1. This patch fixes error:
rm -rf gh-pages/
make V=1 gh-pages
make[1]: Leaving directory '/800GBPCIex4/share/searx'
[ -d "gh-pages/.git" ] || git clone gh-pages
fatal: repository 'gh-pages' does not exist
2. The gh-page build has been moved to ./build/gh-pages this also affects
'travis-gh-pages'
3. The gh-pages commit messages now includes a ref to the repository and commit
4. Since a gh-pages history has only the drawback that the reposetory grows
fast, this patch also flattens the history:
cd build/gh-pages/; git log --oneline
bash: cd: build/gh-pages/: Datei oder Verzeichnis nicht gefunden
026126be (HEAD -> gh-pages, origin/gh-pages) make gh-pages: from https://github.com/return42/searx.git@71d66979c2935312e0aed7fc7c3cf6199fbe88a2
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Avoid SearxEngineXPathException errors when parsing non valid results::
.//div[@class="yuRUbf"]//a/@href index 0 not found
Traceback (most recent call last):
File "./searx/engines/google.py", line 274, in response
url = eval_xpath_getindex(result, href_xpath, 0)
File "./searx/searx/utils.py", line 608, in eval_xpath_getindex
raise SearxEngineXPathException(xpath_spec, 'index ' + str(index) + ' not found')
searx.exceptions.SearxEngineXPathException: .//div[@class="yuRUbf"]//a/@href index 0 not found
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
BTW: fix indentation by 2 spaces
The additional tests has been commented out in the google engines to not release
any CAPTCHA issues.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
BTW: make the engines ready for search.checker:
- replace eval_xpath by eval_xpath_getindex and eval_xpath_list
- google_images: remove outer try/except block
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
To check all engines:
make search.checker
To check a engine 'google news' replace space by underline:
make search.checker.google_news
To see HTTP requests and more use SEARX_DEBUG:
make SEARX_DEBUG=1 search.checker.google_news
To filter out HTTP redirects:
make SEARX_DEBUG=1 search.checker.google_news | grep -A1 "HTTP/1.1\" 3[0-9][0-9]"
...
Engine google news Checking
https://news.google.com:443 "GET /search?q=life&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
https://news.google.com:443 "GET /search?q=life&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
--
https://news.google.com:443 "GET /search?q=computer&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
https://news.google.com:443 "GET /search?q=computer&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
--
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The 'video.html' template from the 'oscar' design supports replacement
for *author* and *length*. Google-videos does not have an author, alternatively
the publisher info from is used for the *author*.
Hint: these replacements are not supported by the 'simple' design.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This revise is based on the methods developed in the revise of the google engine
(see commit 410c2f9).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
$ make test.sh
In utils/lxc.sh line 42:
ubu2010_boilerplate="$ubu1904_boilerplate"
^-----------------^ SC2034: ubu2010_boilerplate appears unused. Verify use (or export if used externally).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This revise is based on the methods developed in the revise of the google engine
(see commit 410c2f9).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
pip install -e .
...
Obtaining file:///usr/local/searx/searx-src
ERROR: Command errored out with exit status 1:
command: /usr/local/searx/searx-pyenv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/usr/local/searx/searx-src/setup.py'"'"'; __file__='"'"'/usr/local/searx/searx-src/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'rn'"'"', '"'"'n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-vzer91m2
cwd: /usr/local/searx/searx-src/
Complete output (9 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/local/searx/searx-src/setup.py", line 10, in <module>
from searx.version import VERSION_STRING
File "/usr/local/searx/searx-src/searx/__init__.py", line 19, in <module>
import searx.settings_loader
File "/usr/local/searx/searx-src/searx/settings_loader.py", line 8, in <module>
import yaml
ModuleNotFoundError: No module named 'yaml'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Removes module searx/brand.py and creates a namespace at searx.brand.
This patch is a first 'proof of concept'. Later we can decide to remove the
brand namespace entirely or not.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Install boilerplate packages (pip, wheel, setuptools and requirements.txt) only
once, when virtualenv is created.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In commit c2a6f145 the sphinx-doc and sphinx-live targets has been removed,
since Sphinx is already installed by the pyenvinstall target.
This commit removes the targets from the remaining prerequisites of the 'BOOK'
targets
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Makefile targets 'pyenv-install' and 'pyenv-uninstall' are unused duplicates of
'pyenvinstall' and 'pyenvuninstall'.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This patch keeps an eye on:
1. When installing searx, the branch to install needs to be a local branch.
2. The branch to install can be different from the current branch.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In commit a70b9b9f the SEARX_SETTINGS_TEMPLATE environment was added to the
.config file, but was not use in utils/searx.sh.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Installation and use of the command 'virtualenv' was only needed in py2 and py2
is no longer suported by searx. In py3 the command is replaced by 'python -m
venv'.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The path to the makefile.lxc is not always static and
'/share/searx/utils/makefile.lxc' can be wrong.
When target 'lxc.activate' is called within the LXC container the path to
makefile.lxc depends on LXC_SHARE_FOLDER and the basename of the folder where
searx was cloned inital on the HOST system.
This patch determines the path of 'makefile.lxc' next to 'makefile.include'
itself.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Since #2291 is merged, it is recommend to use::
use_default_settings=True
1. Add a template file use_default_settings.yml::
SEARX_SETTINGS_TEMPLATE="${REPO_ROOT}/utils/templates/etc/searx/use_default_settings.yml"
2. In Chapter "Configuration" recommend to make use of
'use_default_settings=True' and describe it
3. Rewrite of docs/admin/settings.rst
- move chapter 'settings.yml location' to the top
- update and split chapter 'Global Settings'
4. Add environment SEARX_SETTINGS_TEMPLATE to .config.sh
5. Use environment $SEARX_SETTINGS_TEMPLATE in the utils/searx.sh script
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- strip html tags and superfluous quotation marks from content
- remove not needed cookie from request
- remove superfluous imports
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Error pattern::
Engines cannot retrieve results:
digg (unexpected crash time data '2020-10-16T14:09:55Z' does not match format '%Y-%m-%d %H:%M:%S')
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Devian's request and response forms has been changed.
- fixed title
- fixed time_range_dict to 'popular-*-***'
- use image from <noscript> if exists
- drop obsolete "http to https, remove domain sharding"
- use query URL https://www.deviantart.com/search/deviations?page=5&q=foo
- add searx/engines/deviantart.py to pylint check (test.pylint)
Error pattern::
There DEBUG:searx:result: invalid title: {'url': 'https://www.deviantart.com/ ...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Escape the all the quotation marks (" --> \") in docs/build-templates.
The scripts are *bash evaluated* (e.g)::
eval "echo \"$(< "${REPO_ROOT}/docs/build-templates/searx.rst")\""
Backslash, quotation marks and other need to be escaped.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Reloading webapp.py when a file is changed exit with error::
$ make run
....
INFO:werkzeug: * Detected change in 'searx/searx/engines/deviantart.py', reloading
INFO:werkzeug: * Restarting with stat
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
make: *** [Makefile:69: run] Error 1
Patch suggested by @dalf [1]
[1] https://github.com/searx/searx/issues/2278#issuecomment-719732579
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Since 1. October 2020 google has changed the 'class' attribute of the HTML
result page.
Fix the xpath expressions and ignore <div class="g" ../> sections which do not
match to title's xpath expression.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Since there is no centos-7 documentation 'searx.sh doc' produce an error
message::
ERROR: centos-: uWSGI not yet implemented
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
## What does this PR do?
When installing searx on a centos-7 image [1] an encoding exception is raised from setup.py::
command: /usr/local/searx/searx-pyenv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/usr/local/searx/searx-src/setup.py'"'"'; __file__='"'"'/usr/local/searx/searx-src/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'rn'"'"', '"'"'n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-pidntzkq
cwd: /usr/local/searx/searx-src/
Complete output (7 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/local/searx/searx-src/setup.py", line 16, in <module>
long_description = f.read()
File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc9 in position 482: ordinal not in range(128)
[1] https://uk.images.linuxcontainers.org/
## Why is this change important?
README.rst uses non ASCII characters _(pronunciation səːks.)_
## How to test this PR locally?
install searx on a centos-7 image from https://uk.images.linuxcontainers.org/
## What does this PR do?
To test centos-7 on non centos systems add centos-7 image to the LXC suite.
## Why is this change important?
PR #2112 adds centos-7 support to the install scripts
## How to test this PR locally?
`sudo ./utils/lxc.sh build searx-centos7`
Python's default encoding depends on the platform, set (python) default encoding
UTF-8 in uwsgi ini files:
LANG=C.UTF-8
LANGUAGE=C.UTF-8
LC_ALL=C.UTF-8
Error pattern:
Traceback (most recent call last):
File "/usr/local/searx/searx-src/searx/webapp.py", line 74, in <module>
from searx.search import SearchWithPlugins, get_search_query_from_webapp
File "/usr/local/searx/searx-src/searx/search.py", line 32, in <module>
from searx.external_bang import get_bang_url
File "/usr/local/searx/searx-src/searx/external_bang.py", line 13, in <module>
for bang in json.load(json_file)['bang']:
File "/usr/lib/python3.8/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/usr/lib/python3.8/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 31341: ordinal not in range(128)
close: https://github.com/asciimoo/searx/issues/2041
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Markus Heiser