Prompt the verbose messages ...
PYENV [check] import yaml --> OK
PYENV OK
only when environment $VERBOSE is active.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The environment variables ..
- SEARX_GIT_VERSION
- VERSION_GITCOMMIT
- SEARX_PYTHON_VERSION
- GITHUB_USER
- SEARX_IMAGE_NAME
- BUILD
are local to the docker.buildx() function, other variables like GIT_URL are
comming from the utils/brand.env file (rebuild by: make buildenv).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The usages of the searx.brand namespace has been removed, the searx.brand
namespace is now longer needed.
The searx.brand namespace was an interim solution which has been added in commit
9e53470b4, see commit message there ...
This patch is a first 'proof of concept'. Later we can decide to remove the
brand namespace entirely or not.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In the templates and the /config (JSON) the usage of the 'brand.*' name
space is replaced by 'searx.get_setting' function.
- new_issue_url --> get_setting('brand.new_issue_url')
- brand.GIT_URL --> get_setting('brand.git_url')
- brand.PUBLIC_INSTANCES --> get_setting('brand.public_instances')
- brand.DOCS_URL --> get_setting('brand.docs_url')
- brand.ISSUE_URL --> get_setting('brand.issue_url')
- brand.CONTACT_URL --> get_setting('general.contact_url', '')
The macro 'new_issue' from searx/templates/*/messages/no_results.html
is now imported with context::
{% from '__common__/new_issue.html' import new_issue with context %}
To get *public instances URL* from context's 'get_setting()' function::
get_setting('brand.public_instances','')
Macro's prototype does no longer need the 'new_issue_url' argument and has been
changed to::
macro new_issue(engine_name, engine_reliability)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Added function searx.get_setting(name, default=_unset):
Returns the value to which ``name`` point. If there is no such name in the
settings and the ``default`` is unset, a KeyError exception is raised.
In all the python processes ..
- make docs
- make buildenv
- make install (setup.py)
the usage of the 'brand.*' name space is replaced by 'searx.get_setting'
function.
- brand.SEARX_URL --> get_setting('server.base_url')
- brand.GIT_URL --> get_setting('brand.git_url')
- brand.GIT_BRANCH' --> get_setting('server.base_url')
- brand.ISSUE_URL --> get_setting('brand.issue_url')
- brand.DOCS_URL --> get_setting('brand.docs_url')
- brand.PUBLIC_INSTANCES --> get_setting('brand.public_instances')
- brand.CONTACT_URL --> get_setting('general.contact_url', '')
- brand.WIKI_URL --> get_setting('brand.wiki_url')
- brand.TWITTER_URL --> get_setting('brand.twitter_url', '')
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Qwant is a fast and reliable search engine and AFAIK there is no CAPTCHA. Let
us enable Qwant engines by default.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The implementation uses the Qwant API (https://api.qwant.com/v3). The API is
undocumented but can be reverse engineered by reading the network log of
https://www.qwant.com/ queries.
This implementation is used by different qwant engines in the settings.yml::
- name: qwant
categories: general
...
- name: qwant news
categories: news
...
- name: qwant images
categories: images
...
- name: qwant videos
categories: videos
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The engine was added in commit a4b07460 but now it shows new issues [1].
In the 90'th of the last century, dogpile had its own WEB index, but nowadays it
is a meta-search engine [2]
Powered by technology, Dogpile returns all the best results from leading
search engines including Google and Yahoo!
Using dogpile as an engine in SearXNG needs more investigation, a XPath solution
like we have is not enough. It is questionable whether it still makes sense to
investigate more into a meta-search engine with a ReCAPTCHA in front.
With this patch the dogpile engine is removed
[1] https://github.com/searxng/searxng/issues/202
[2] https://www.dogpile.com/support/aboutus
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
BTW add about section to the YAML configuration
It now shows descriptions with their correct URLs when there are videos in the
search results, pulling content_xpath from snippet-description instead of
snippet-content.
Suggested-by: @eagle-dogtooth https://github.com/searx/searx/issues/2857#issuecomment-869119968
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
From pacman's man page:
--noprogressbar
Do not show a progress bar when downloading files. This can be useful for
scripts that call pacman and capture the output.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In 2a12637 an ASCII escape code was added, such escape codes should not be
written to pipes (when the output is not a terminal).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Upgrade from pylint v2.8.3 to 2.9.3 raise some new issues::
searx/search/checker/__main__.py:37:26: R1732: Consider using 'with' for resource-allocating operations (consider-using-with)
searx/search/checker/__main__.py:38:26: R1732: Consider using 'with' for resource-allocating operations (consider-using-with)
searx/search/processors/__init__.py:20:0: R0402: Use 'from searx import engines' instead (consider-using-from-import)
searx/preferences.py:182:19: C0207: Use data.split('-', maxsplit=1)[0] instead (use-maxsplit-arg)
searx/preferences.py:506:15: R1733: Unnecessary dictionary index lookup, use 'user_setting' instead (unnecessary-dict-index-lookup)
searx/webapp.py:436:0: C0206: Consider iterating with .items() (consider-using-dict-items)
searx/webapp.py:950:4: C0206: Consider iterating with .items() (consider-using-dict-items)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Some piped commands hide the cursor, show cursory when the stream ends.
Most often this is a bug of the command which piped. The command should not
hide the cursor when it writes to a pipe. I have seen this bug with the package
manager (pacman) from ArchLinux.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
To compress saved preferences in the URL was introduced in 5f758b2d3 and
slightly fixed in 8f4401462. But the main fail was not fixed; The decompress
function returns a binary string and this binary should first be decoded to a
string before it is passed to urllib.parse_qs.
BTW: revert the hot-fix from 5973491
Related-to: https://github.com/searxng/searxng/issues/166
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The functions:
- static.build.commit
- static.build.commit.drop
- static.build.restore
are imported into the ./manage script. To avoid name collisions some variables
and fucntions has been renamed by adding the prefix *static_*.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Marginal simplification of the procedure to get only the last local commit which
is not in remotes.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- add script to shellcheck,
- fix error messages from shellcheck and
- moved global variables to local variables (lower case)
No functional change!
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- move jshint option from gruntfile to .jshintrc
- remove trailing-whitespace from gruntfile and
- add jshint esversion: 6
- .dir-locals.el add locals for js-mode to use JSHint from the simple theme
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This patch disables role 'no-descending-specificity'. IMO it is better to have
this rule active (see below [1]), but it is hard to rewrite the less files to
pass this rule, so for the first I chose to disable this rule.
---
Source order is important in CSS, and when two selectors have the same
specificity, the one that occurs last will take priority. However, the situation
is different when one of the selectors has a higher specificity. In that case,
source order does not matter: the selector with higher specificity will win out
even if it comes first.
The clashes of these two mechanisms for prioritization, source order and
specificity, can cause some confusion when reading stylesheets. If a selector
with higher specificity comes before the selector it overrides, we have to think
harder to understand it, because it violates the source order
expectation. Stylesheets are most legible when overriding selectors always come
after the selectors they override. That way both mechanisms, source order and
specificity, work together nicely.
This rule enforces that practice as best it can, reporting fewer errors than it
should. It cannot catch every actual overriding selector, but it can catch
certain common mistakes.
[1] https://stylelint.io/user-guide/rules/list/no-descending-specificity/
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This fix was autogenerated by::
npx stylelint -f unix --fix 'searx/static/themes/simple/src/less/**/*.less'
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Docker is blocking network of existing LXC containers / there is a conflict in
the iptables setup of Docker & LXC. With this patch:
- utils/lxc.sh checks internet connectivity (instead of silently hang)
- Chapter "Internet Connectivity & Docker" describes the problem and made a
suggestion for a solution a solution
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In 5a7b12ee we changed the signature of the YAML settings, this patch takes this
into account.
Related-to: 5a7b12ee [yamllint] searx/settings.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In the EU there exists a "General Data Protection Regulation" [1] aka GDPR (BTW:
very user friendly!) which requires consent to tracking. To get the consent
from the user, google-news requests are redirected to confirm and get a CONSENT
Cookie from https://consent.google.de/s?continue=...
This patch adds a CONSENT Cookie to the google-news request to avoid
redirection.
The behavior of the CONTENTS cookies over all google engines seems similar but
the pattern is not yet fully clear to me, here are some random samples from my
analysis ..
Using common google search from different domains::
google.com: CONSENT=YES+cb.{{date}}-14-p0.de+FX+816
google.de: CONSENT=YES+cb.{{date}}-14-p0.de+FX+333
google.fr: CONSENT=YES+srp.gws-{{date}}-0-RC2.fr+FX+826
When searching about videos (google-videos)::
google.es: CONSENT=YES+srp.gws-{{date}}-0-RC2.es+FX+076
google.de: CONSENT=YES+srp.gws-{{date}}-0-RC2.de+FX+171
Google news has only one domain for all languages::
news.google.com: CONSENT=YES+cb.{{date}}-14-p0.de+FX+816
Using google-scholar search from different domains::
scholar.google.de: CONSENT=YES+cb.{{date}}-14-p0.de+FX+333
scholar.google.fr: does not use such a cookie / did not ask the user
scholar.google.es: does not use such a cookie / did not ask the user
Interim summary:
Pattern is unclear and I won't apply the CONSENT cookie to all google engines.
More experience is need before we generalize the CONSENT cookies over all
google engines.
Related:
- e9a6ab401 [fix] youtube - send CONSENT Cookie to not be redirected
- https://github.com/benbusby/whoogle-search/issues/311
- https://github.com/benbusby/whoogle-search/issues/243
[1] https://en.wikipedia.org/wiki/General_Data_Protection_Regulation
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Since we added
- 1c67b6aec [enh] google engine: supports "default language"
there is a KeyError: 'hl in request,error pattern::
ERROR:searx.searx.search.processor.online:engine google news : exception : 'hl'
Traceback (most recent call last):
File "searx/search/processors/online.py", line 144, in search
search_results = self._search_basic(query, params)
File "searx/search/processors/online.py", line 118, in _search_basic
self.engine.request(query, params)
File "searx/engines/google_news.py", line 97, in request
if lang_info['hl'] == 'en':
KeyError: 'hl'
Closes: https://github.com/searxng/searxng/issues/154
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The node.env build environment require npm, ttfautohint and fontforge installed
in the OS. These tools can be installed by::
sudo -H ./utils/searx.sh install buildhost
If one of the tools is not installed, the script node.env stops with a
appropriate message.
BTW: We ignore CentOS-7 as developer & build platform
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Open YAML file and use:
M-x flycheck-mode
Since we use yamllint from local/py3/bin, creating a virtualenv is a
prerequisite:
make pyenv
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Markus Heiser