Pylint 2.10 added new default checks [1]:
use-list-literal
Emitted when list() is called with no arguments instead of using []
use-dict-literal
Emitted when dict() is called with no arguments instead of using {}
[1] https://pylint.pycqa.org/en/latest/whatsnew/2.10.html
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The settings template in utils/templates/etc/searx/settings.yml
does not include any replacements (shell environment variables).
BTW: utils/filtron.sh & utils/lib.sh
Add some more messages to installation procedures, to be more clear.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Functions implemented to run *Weblate* workflows should use prefix 'weblate.':
- babel.setup.translations.worktree --> weblate.translations.worktree
Create git worktree ${TRANSLATIONS_WORKTREE} and checkout branch
'translations' from Weblate's counterpart (weblate) of the SearXNG
(origin)::
remote weblate https://weblate.bubu1.eu/git/searxng/searxng/
- babel.weblate.to.translations --> weblate.to.translations
Update 'translations' branch of SearXNG (origin) with last additions from
Weblate.
- babel.translations.to.master --> weblate.translations.commit
Update 'translations' branch of SearXNG (origin) with last additions from
Weblate. Copy the changes to the master branch, compile translations and
create a commit in the local branch (master)
- babel.master.to.translations --> weblate.push.translations
Push *translation changes* from SearXNG (origin) to Weblate's
counterpart (weblate).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- Add ``# lint: pylint`` header to pylint this python file.
- Fix issues reported by pylint.
- Add source code documentation of modul searx.locales
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In ./manage implement babel.*:
- extract : extract messages from source files and generate POT file
- update : update existing message catalogs from POT file
- compile : compile translation catalogs into binary MO files
Replace searx_extra/update/update_translations.sh by command:
- ci.babel.update
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In commit 4d3f2f48d common used languages has been droped. By reducing the
number of `min_engines_per_lang` from 15 to 13 we get theses languages back.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In commit 4b43775c9 the brand.git_url & brand.git_branch has been removed, with
this patch these settings are removed from the documentation.
docs/admin/engines/settings.rst
- Remove brand.git_url & brand.git_branch
docs/dev/makefile.rst
- Remove brand.git_url & brand.git_branch
modified docs/dev/lxcdev.rst
- Remove brand.git_url & brand.git_branch
- fix searXNG to SearXNG
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The first import of names from the searx package implies loading the
settings.yml. Before this is done, the enviroment variables must be unset to
not overwrite the values from the settings.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Deny formats has been implemented in 6ed4616d.
To harden SearXNG instances by default, other formats than HTML should be
denied. Most of JSON, RSS and CSV requests are bots [1]::
Bots are the only users of this feature on a public instance, and they abuse
it too much that the engines rate limit pretty quickly the IP address of the
instance.
[1] https://github.com/searxng/searxng/issues/95
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In unit tests settings from
searx/settings.yml
and the user settings from:
unit/settings/test_settings.yml
are used. In the latter, settings can be activated that are needed in the unit
test but should not activated by default in production.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Do not merge this patch in master branch of SearXNG! This branch exists only
for testing the feature branch fix-searx.sh @return42.
This patch changes the buildenv to::
GIT_URL='https://github.com/return42/searxng'
GIT_BRANCH='fix-searx.sh'
SEARX_PORT='7777'
SEARX_BIND_ADDRESS='127.0.0.12'
To test installation procedure, clone feature branch (fix-searx.sh)::
$ cd ~/Downloads
$ git clone --branch fix-searx.sh https://github.com/return42/searxng searxng
$ cd searxng
$ ./utils/searx.sh install all
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The filtron target is the SearXNG installation and the default of FILTRON_TARGET
is taken from the YAML configuration ('server.port' & 'server.bind_address').
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Some defaults in the settings.yml are taken from the environment.
By example;
The manage scripts sources the ./utils/brand.env and sets SEARX_PORT
environment. This enviroment *wins over* any settings in a YAML file.
Whe we run a::
make test.robot
The searx/settings_robot.yml is used, in this file the server settings are::
server:
port: 11111
bind_address: 127.0.0.1
To get in use of the 'port: 11111' we have to unset the SEARX_PORT environment
which was sourced before.
The function buildenv.unset_env() can be called in all use cases where the
enviroment from ./utils/brand.env is not wanted. ATM it unset the enviroment
variables::
unset GIT_URL
unset GIT_BRANCH
unset SEARX_URL
unset SEARX_PORT
unset SEARX_BIND_ADDRESS
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Not all settings from the 'brand:' section of the YAML files are needed in the
shell scripts. This patch reduce the variables in ./utils/brand.env to what is
needed. The following ('brand:' settings) can be removed from this file:
- ISSUE_URL
- DOCS_URL
- PUBLIC_INSTANCES
- WIKI_URL
Tasks running outside of an *installed instance*, need the following settings
from the YAML configuration:
- GIT_URL <--> brand.git_url
- GIT_BRANCH <--> brand.git_branch
- SEARX_URL <--> server.base_url (aka PUBLIC_URL)
- SEARX_PORT <--> server.port
- SEARX_BIND_ADDRESS <--> server.bind_address
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Inspect YAML setting <key> from SearXNG instance (${SEARX_SRC})::
utils/searx.sh inspect settings server.base_url
utils/lib_install.sh
should not log on stdout which is used for the output of function
prompt_installation_setting(). Turned build_msg (stdout) into
info_msg (stderr).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This patch implements two new commands:
install dot-config :
- copy ./config.sh to ${SEARX_SRC}
- implemented in new function install_DOT_CONFIG
install init-src:
- sync files (SEARX_SRC_INIT_FILES) with ${SEARX_SRC}
- implemented in new function init_SEARX_SRC()
new functions: verify_continue_install() and prompt_installation_status()
These functions are used in installation procedures to get the status of the
installation procedure. The status is based on the existing function:
usage: install_searx_get_state
Prompts a string indicating the status of the installation procedure
missing-searx-clone:
There is no clone at ${SEARX_SRC}
missing-searx-pyenv:
There is no pyenv in ${SEARX_PYENV}
installer-modified:
There are files modified locally in the installer (clone),
see ${SEARX_SRC_INIT_FILES} description.
python-installed:
Scripts can be executed in instance's environment
- user: ${SERVICE_USER}
- pyenv: ${SEARX_PYENV}
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In commit 94851790 we have centralized all SearXNG setups in the settings.yml
file:
94851790 [mod] move brand options from Makefile to settings.yml
This step has not yet been completed for the installation procedures! Since all
SearXNG setups are done in the settings.yml these environment variables needs to
be removed from the ./conf.sh file. Scripts and other tasks running outside of
an instance got the needed values from the ./utils/brand.env file.
By example: ATM the environment variables of the ./config.sh file are in
conflict with them from settings.yml:
- PUBLIC_URL --> {server:base_url}
- SEARX_INTERNAL_HTTP --> {server:bind_address}.{server:port}
- GIT_BRANCH --> {brand:GIT_URL}
These environment variable of a SearXNG instance and additional
- SEARX_SETTINGS_TEMPLATE
has been remove from the '.config.sh' file. With this patch, the main focus of
./conf.sh resists on environment variables needed for the installation of morty,
filtron software.
modified .config.sh:
- removed no longer supported variables (see above)
- add comment about: SearXNG setup in settings.yml
modified utils/searx.sh:
- SEARX_INTERNAL_HTTP no longer take from .config.sh
- SEARX_SETTINGS_PATH /etc/searx/settings.yml
- SEARX_SETTINGS_TEMPLATE obsolete
modified utils/lib_install.sh:
Initialize environment variables SEARX_PYENV, SEARX_SETTINGS_PATH and
PUBLIC_URL.
modified: utils/morty.sh
Add missing hint about SEARX_SETTINGS_PATH and move PUBLIC_URL to
utils/lib_install.sh
modified: utils/morty.sh
Move PUBLIC_URL to utils/lib_install.sh
Renamed utils/templates/etc/searx/use_default_settings.yml -> settings.yml
- removed option which can't be modified after installation
- add some comments with examples
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
modified docs/admin/engines/settings.rst
- Fix documentation and add section 'brand'.
- Add remarks about **buildenv** variables.
- Add remarks about settings from environment variables $SEARX_DEBUG,
$SEARX_PORT, $SEARX_BIND_ADDRESS and $SEARX_SECRET
modified docs/admin/installation-searx.rst & docs/build-templates/searx.rst
Fix template location /templates/etc/searx/settings.yml
modified docs/dev/makefile.rst
Add description of the 'make buildenv' target and describe
- we have all SearXNG setups are centralized in the settings.yml file
- why some tasks need a utils/brand.env (aka instance's buildenv)
modified manage
Settings file from repository's working tree are used by default and
ask user if a /etc/searx/settings.yml file exists.
modified searx/settings.yml
Add comments about when it is needed to run 'make buildenv'
modified searx/settings_defaults.py
Default for server:port is taken from enviroment variable SEARX_PORT.
modified utils/build_env.py
- Some defaults in the settings.yml are taken from the environment,
e.g. SEARX_BIND_ADDRESS (searx.settings_defaults.SHEMA). When the
'brand.env' file is created these enviroment variables should be
unset first.
- The CONTACT_URL enviroment is not needed in the utils/brand.env
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
**new** utils/lib_install.sh:
Used to initialize installation procedures
- Modified source_dot_config function that
- loads .config.sh from an existing installation (at SEARX_SRC).
- initialize **SEARX_SRC_INIT_FILES**
- functions like:
- install_log_searx_instance()
- install_searx_get_state()
**modified** utils/searx.sh
- obsolete environment SEARX_INSTANCE_NAME has been replaced
by string 'SearXNG'.
**modified** utils/filtron.sh, utils/morty.sh, utils/searx.sh
- source utils/lib_install.sh
- normalize logging of environment variables using new function
install_log_searx_instance()
**modified** utils/lib.sh
- fix marginal typos
**Installation scripts**
The utils/lib_install.sh is sourced by the installations scripts:
- utils/searx.sh
- utils/morty.sh
- utils/filtron.sh
If '${SEARX_SRC}/.config.sh' exists, the modified source_dot_config() function
loads this configuration (instead of './.config.sh').
**SEARX_SRC_INIT_FILES**
Array of file names to sync into a installation at $SEARX_SRC. The file names
are relative to the $REPO_ROOT. Set by function init_SEARX_SRC_INIT_FILES().
Most often theses are files like:
- .config.sh
- searx/settings.yml
- utils/brand.env
- ...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Prompt the verbose messages ...
PYENV [check] import yaml --> OK
PYENV OK
only when environment $VERBOSE is active.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The environment variables ..
- SEARX_GIT_VERSION
- VERSION_GITCOMMIT
- SEARX_PYTHON_VERSION
- GITHUB_USER
- SEARX_IMAGE_NAME
- BUILD
are local to the docker.buildx() function, other variables like GIT_URL are
comming from the utils/brand.env file (rebuild by: make buildenv).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The usages of the searx.brand namespace has been removed, the searx.brand
namespace is now longer needed.
The searx.brand namespace was an interim solution which has been added in commit
9e53470b4, see commit message there ...
This patch is a first 'proof of concept'. Later we can decide to remove the
brand namespace entirely or not.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In the templates and the /config (JSON) the usage of the 'brand.*' name
space is replaced by 'searx.get_setting' function.
- new_issue_url --> get_setting('brand.new_issue_url')
- brand.GIT_URL --> get_setting('brand.git_url')
- brand.PUBLIC_INSTANCES --> get_setting('brand.public_instances')
- brand.DOCS_URL --> get_setting('brand.docs_url')
- brand.ISSUE_URL --> get_setting('brand.issue_url')
- brand.CONTACT_URL --> get_setting('general.contact_url', '')
The macro 'new_issue' from searx/templates/*/messages/no_results.html
is now imported with context::
{% from '__common__/new_issue.html' import new_issue with context %}
To get *public instances URL* from context's 'get_setting()' function::
get_setting('brand.public_instances','')
Macro's prototype does no longer need the 'new_issue_url' argument and has been
changed to::
macro new_issue(engine_name, engine_reliability)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Added function searx.get_setting(name, default=_unset):
Returns the value to which ``name`` point. If there is no such name in the
settings and the ``default`` is unset, a KeyError exception is raised.
In all the python processes ..
- make docs
- make buildenv
- make install (setup.py)
the usage of the 'brand.*' name space is replaced by 'searx.get_setting'
function.
- brand.SEARX_URL --> get_setting('server.base_url')
- brand.GIT_URL --> get_setting('brand.git_url')
- brand.GIT_BRANCH' --> get_setting('server.base_url')
- brand.ISSUE_URL --> get_setting('brand.issue_url')
- brand.DOCS_URL --> get_setting('brand.docs_url')
- brand.PUBLIC_INSTANCES --> get_setting('brand.public_instances')
- brand.CONTACT_URL --> get_setting('general.contact_url', '')
- brand.WIKI_URL --> get_setting('brand.wiki_url')
- brand.TWITTER_URL --> get_setting('brand.twitter_url', '')
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Qwant is a fast and reliable search engine and AFAIK there is no CAPTCHA. Let
us enable Qwant engines by default.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The implementation uses the Qwant API (https://api.qwant.com/v3). The API is
undocumented but can be reverse engineered by reading the network log of
https://www.qwant.com/ queries.
This implementation is used by different qwant engines in the settings.yml::
- name: qwant
categories: general
...
- name: qwant news
categories: news
...
- name: qwant images
categories: images
...
- name: qwant videos
categories: videos
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The engine was added in commit a4b07460 but now it shows new issues [1].
In the 90'th of the last century, dogpile had its own WEB index, but nowadays it
is a meta-search engine [2]
Powered by technology, Dogpile returns all the best results from leading
search engines including Google and Yahoo!
Using dogpile as an engine in SearXNG needs more investigation, a XPath solution
like we have is not enough. It is questionable whether it still makes sense to
investigate more into a meta-search engine with a ReCAPTCHA in front.
With this patch the dogpile engine is removed
[1] https://github.com/searxng/searxng/issues/202
[2] https://www.dogpile.com/support/aboutus
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
BTW add about section to the YAML configuration
It now shows descriptions with their correct URLs when there are videos in the
search results, pulling content_xpath from snippet-description instead of
snippet-content.
Suggested-by: @eagle-dogtooth https://github.com/searx/searx/issues/2857#issuecomment-869119968
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
From pacman's man page:
--noprogressbar
Do not show a progress bar when downloading files. This can be useful for
scripts that call pacman and capture the output.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In 2a12637 an ASCII escape code was added, such escape codes should not be
written to pipes (when the output is not a terminal).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Upgrade from pylint v2.8.3 to 2.9.3 raise some new issues::
searx/search/checker/__main__.py:37:26: R1732: Consider using 'with' for resource-allocating operations (consider-using-with)
searx/search/checker/__main__.py:38:26: R1732: Consider using 'with' for resource-allocating operations (consider-using-with)
searx/search/processors/__init__.py:20:0: R0402: Use 'from searx import engines' instead (consider-using-from-import)
searx/preferences.py:182:19: C0207: Use data.split('-', maxsplit=1)[0] instead (use-maxsplit-arg)
searx/preferences.py:506:15: R1733: Unnecessary dictionary index lookup, use 'user_setting' instead (unnecessary-dict-index-lookup)
searx/webapp.py:436:0: C0206: Consider iterating with .items() (consider-using-dict-items)
searx/webapp.py:950:4: C0206: Consider iterating with .items() (consider-using-dict-items)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Some piped commands hide the cursor, show cursory when the stream ends.
Most often this is a bug of the command which piped. The command should not
hide the cursor when it writes to a pipe. I have seen this bug with the package
manager (pacman) from ArchLinux.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
To compress saved preferences in the URL was introduced in 5f758b2d3 and
slightly fixed in 8f4401462. But the main fail was not fixed; The decompress
function returns a binary string and this binary should first be decoded to a
string before it is passed to urllib.parse_qs.
BTW: revert the hot-fix from 5973491
Related-to: https://github.com/searxng/searxng/issues/166
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The functions:
- static.build.commit
- static.build.commit.drop
- static.build.restore
are imported into the ./manage script. To avoid name collisions some variables
and fucntions has been renamed by adding the prefix *static_*.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Marginal simplification of the procedure to get only the last local commit which
is not in remotes.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Markus Heiser