api.openverse.engineering is a little picky and wants to have a trailing slash
in the path:
/v1/images? -->/ v1/images/?
otherwise it redirects, here is the debug log:
DEBUG searx.network.openverse : HTTP Request: GET https://api.openverse.engineering/v1/images?&page=1&page_size=20&format=json&q=foo "HTTP/2 301 Moved Permanently" (text/html; charset=utf-8)
DEBUG searx.network.openverse : HTTP Request: GET https://api.openverse.engineering/v1/images/?&page=1&page_size=20&format=json&q=foo "HTTP/2 200 OK" (application/json)
WARNING searx.engines.openverse : ErrorContext('searx/search/processors/online.py', 105, 'count_error(', None, '1 redirects, maximum: 0', ('200', 'OK', 'api.openverse.engineering')) True
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The implementation of the etools engine is poor. No date-range support, no
language support and it is broken by a CAPTCHA.
etools is a metasearch engine, the major search engines it supports (google,
bing, wikipedia, Yahoo) are already available in SeaarXNG.
While etools does support several engines we currently don't support directly,
support for them should be added directly to SearXNG if there is demand.
In practice: in SearXNG the worse etools results will be mixed with good results
from other engines we have (as long as there is no captcha).
At best case, what we win with etools is in e.g. results from de.ask.com in a
query from a german request .. in all other cases worse results are bubble up in
SearXNG's result list.
[1] https://github.com/searxng/searxng/issues/696#issuecomment-1005855499
Closes: https://github.com/searxng/searxng/issues/696
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Commit c7f27404 moves the `python -m searx.webapp` process to the background. A
background job can't open a simple python-debugger (pdb) console.
This patch moves all processes to the background except the searx.webapp.
Insert a break point somewhere in the webapp application::
import pdb
pdb.set_trace()
And start a debug session by::
make run
and test you break point.
To test that the entire 'make run' stops in the case of an error in the
themes.live [1] background process try:
make LIVE_THEME=typo-theme-name run
[1] https://github.com/searxng/searxng/pull/664#discussion_r776419585
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Add a redis connector, the default DB connector is a socket at::
unix:///usr/local/searxng-redis/run/redis.sock?db=0
To set up a redis instance simply use::
$ ./manage redis.build
$ sudo -H ./manage redis.install
A hint for developers:
To get access rights to this instance, your developer account needs to be added
to the *searxng-redis* group::
$ sudo -H ./manage redis.addgrp "${USER}"
# don't forget to logout & login to get member of group
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
A script to build & install a simple & isolated redis service, dedicated to
SearXNG and connected via Unix socket.
$ ./manage redis.help
redis.:
devpkg : install essential packages to compile redis
build : build redis binaries at /800GBPCIex4/share/SearXNG/dist/redis/6.2.6/amd64
install : create user (searxng-redis) and install systemd service (searxng-redis)
remove : delete user (searxng-redis) and remove service (searxng-redis)
shell : start bash interpreter from user searxng-redis
src : clone redis source code to <path> and checkput 6.2.6
useradd : create user (searxng-redis) at /usr/local/searxng-redis
userdel : delete user (searxng-redis)
addgrp : add <user> to group (searxng-redis)
rmgrp : remove <user> from group (searxng-redis)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
File searx/languages.py, created by update_languages.py has to pass quality
check from CI::
make format.python
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Environment variable to get live builds while modifying CSS & JS of a theme::
LIVE_THEME=simple make run
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
To have a live build while editing the HTML client of a theme. The first
argument of the command is the theme name:
./manage themes.live simple
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Add event listener to query selector::
'#urls img.image'
From the user point of view, I think it is better to hide the image:
img_load_error.svg is helplful in the image category because it still allows to
select the image. IMO, in the news category, the fact there is a missing image
won't help to choose the links. From a developer point of view, the place holder
is signal that may be the engine needs to be updated (at least give a look). The
browser console should show the same information too, but it requires some
additional steps. [1]
[1] https://github.com/searxng/searxng/pull/610#issuecomment-997640132
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Fix pylint issues from commit (3d96a983)
[format.python] initial formatting of the python code
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Disable the python code formatting from python-black, where the readability of
code suffers by formatting.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Follow up queries for the pages needed to be fixed.
- Split search-term in one for initial query and one for following queries.
- Set some headers in HTTP requests, bing needs for paging support.
- IMO //div[@class="sa_cc"] does no longer match in a bing response.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Fix remarks from pylint and improved code-style. In preparation for a bug-fix
of the Bing (Web) engine I add this engine to the pylint-list.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Add the 'SearXNG JavaScript Style Guide' as one quality gate in the build chain
of the simple theme::
make themes.simple
make themes.simple.test
In the github CI the `themes.all` target enforce a `themes.simple`.
BTW: Remove 'jshint' left overs from 0ee316f3d
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
automatically fix some of the problems reported by eslint rules::
$ ./manage nvm.bash
nvm-env$ npm --prefix searx/static/themes/simple run eslint-fix
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Markus Heiser