The use of img_src AND thumbnail in the default results makes no sense (only a
thumbnail is needed). In the current state this is rather confusing, because
img_src is displayed like a thumbnail (small) and thumbnail is displayed like an
image (large).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The names of the links are rather tags than real names, and they sometimes vary
greatly in their spelling:
- GitHub: github, Github
- Source code: Repository, SCM, Project Source Code
- Documentation: docs, Documentation
It was standardized to terms such as 'Source code' and 'Documentation', as
translations already exist for these terms.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- l10n support: parse and format decimal numbers by babel
- ability to add additional units
- improved unit detection (symbols are not unique)
- support for alias units (0,010C to F --> 32,018 °F)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Another trip into the hell of dependencies: docutils tends to put major changes
in minor patches: the executables have been renamed / e.g.
rst2html.py --> rts2html
so we have to use docutils at least from version 0.21.2, but this version of
docutils is only supported by myst-parser from version 3.0.1 on.
Additionally, docutils decided to drop python 3.8 in version 0.21 [1]
Further, linuxdoc needed an update to cope with docutils 0.21 [2]
[1] https://docutils.sourceforge.io/RELEASE-NOTES.html#release-0-21-2024-04-09
[2] https://github.com/return42/linuxdoc/pull/36
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Startpage has changed its HTML layout, classes like ``w-gl__result__main`` do no
longer exists and the result items have been slightly changed in their
structure.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In the "Engines" tab on searx.space [1] nearly all engines report a
TimeoutException: yep engine
As documented in issue #2444 [2], this problem can be fixed by increasing the
timeout. Note: on a local instance (`make run`) the timeout of 3sec was
sufficient / at least in my local test, but the balance of searx.space leads me
to believe that this tight timeout is usually not sufficient.
[1] https://searx.space/
[2] https://github.com/searxng/searxng/issues/2444
Closes https://github.com/searxng/searxng/issues/3421
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
CCC media serves several recording formats, to name a few:
- application/x-subrip
- video/mp4
- video/webm
- audio/mpeg
- audio/opus
- audio/mpeg
not all of them are suitable for a video frame. If available we should prefer
video/mp4 due to its minimal data rates.
Closes: https://github.com/searxng/searxng/issues/3431
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
To test this patch I used .. and checked the diff of the `messages.pot` file::
$ ./manage pyenv.cmd pybabel extract -F babel.cfg \
-o ./searx/translations/messages.pot searx/
$ git diff ./searx/translations/messages.pot
----
hint from @dalf: f-string are not supported [1] but there is no error [2].
[1] python-babel/babel#594
[2] python-babel/babel#715
Closes: https://github.com/searxng/searxng/issues/3412
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In lxml 5.1.1 the private name `_ElementStringResult` in module `lxml.etree`
does no longer exists.
This code was written nearly a decade ago, its no longer clear what the
intention `_ElementStringResult` and `_ElementUnicodeResult` had been. It can be
assumed that these classes will no longer occur.
Closes: https://github.com/searxng/searxng/issues/3368
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The settungs in the .dir-locals.el do not work well for every emacs setup:
Add .dir-locals.el to the git-ignore list and move content to a template file.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In the past, some files were tested with the standard profile, others with a
profile in which most of the messages were switched off ... some files were not
checked at all.
- ``PYLINT_SEARXNG_DISABLE_OPTION`` has been abolished
- the distinction ``# lint: pylint`` is no longer necessary
- the pylint tasks have been reduced from three to two
1. ./searx/engines -> lint engines with additional builtins
2. ./searx ./searxng_extra ./tests -> lint all other python files
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- pylint all scripts
- fix some errors reported by pyright
- from searx.data import data_dir (Path.open)
- fix import from pygments.formatters.html
NOTE: none functional changes!
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Add a leading tag (in square brackets) about the scope/type to commit messages
from automated tasks (commits from CI).
dependantbot::
[upd] pypi: Bump .. from .. to ..
[upd] npm: Bump .. from .. to .. in /searx/static/themes/simple
Weblate translation updates::
[l10n] update translations from Weblate
updates of ./data::
[data] update searx.data ...
build commit of gh-pages::
[doc] build from commit ...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The URL was accidentally deleted in a85907a98, but is still required in
base.html for auto-discovery / from base.html::
<link title="{{ instance_name }}"
type="application/opensearchdescription+xml"
rel="search" href="{{ opensearch_url }}"
/>
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
To avoid unnecessary changes to the file, the list should be sorted before it is
written to the file.
You can test it by calling multiple times::
make data.locales
and searx/data/locales.json should be unchanged.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Parse the result list from ask.com given in the variable named
window.MESON.initialState::
<script nonce="..">
window.MESON = window.MESON || {};
window.MESON.initialState = {"siteConfig": ...
...}};
window.MESON.loadedLang = "en";
</script>
The result list is in field::
json_resp['search']['webResults']['results']
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
timeout: 4.0
The timeout of presearch-WEB is left up from the default of 3sec to 4sec. The
engine has to send two HTTP requests, they often exceed the default timeout of
3sec. Since all other presearch categories (images, videos, news) also have a
timeout of 4 sec, the WEB search should also have the same timeout.
network: presearch
Place all HTTP requests in the same network, named ``presearch``.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In Presearch there are languages for the UI and regions for narrowing down the
search. With this change the SearXNG engine supports a search by region. The
details can be found in the documentation of the source code.
To test, you can search terms like::
!presearch bmw :zh-TW
!presearch bmw :en-CA
1. You should get results corresponding to the region (Taiwan, Canada)
2. and in the language (Chinese, Englisch).
3. The context in info box content is in the same language.
Exceptions:
1. Region or language is not supported by Presearch or
2. SearXNG user did not selected a region tag, example::
!presearch bmw :en
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This patch fixes issue reported by ``make test.unit``::
searx/search/checker/impl.py:39: SyntaxWarning: invalid escape sequence '\>'
rep = ['<' + tag + '[^\>]*>' for tag in HTML_TAGS]
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
All the environments defined in ./utils/brand.env are generated on the fly, so
there is no longer a need to define the brand environment in this file and all
the workflows to handle this file.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- the option server:public_instance lacks some documentation
- the processing of this option belongs in the limiter and not
in botdetection module
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This patch was inspired by the discussion around PR-2882 [2]. The goals of this
patch are:
1. Convert plugin searx.plugin.limiter to normal code [1]
2. isolation of botdetection from the limiter [2]
3. searx/{tools => botdetection}/config.py and drop searx.tools
4. in URL /config, 'limiter.enabled' is true only if the limiter is really
enabled (Redis is available).
This patch moves all the code that belongs to botdetection into namespace
searx.botdetection and code that belongs to limiter is placed in namespace
searx.limiter.
Tthe limiter used to be a plugin at some point botdetection was added, it was
not a plugin. The modularization of these two components was long overdue.
With the clear modularization, the documentation could then also be organized
according to the architecture.
[1] https://github.com/searxng/searxng/pull/2882
[2] https://github.com/searxng/searxng/pull/2882#issuecomment-1741716891
To test:
- check the app works without the limiter, check `/config`
- check the app works with the limiter and with the token, check `/config`
- make docs.live .. and read
- http://0.0.0.0:8000/admin/searx.limiter.html
- http://0.0.0.0:8000/src/searx.botdetection.html#botdetection
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
When ``searx/static/themes/simple/node_modules`` are installed the test from
black runs into this path / Error::
$ make test
TEST [yamllint] $YAMLLINT_FILES
TEST [black] $BLACK_TARGETS
--- searx/static/themes/simple/node_modules/flatted/python/test.py 2023-10-13 10:09:50.460026 +0000
+++ searx/static/themes/simple/node_modules/flatted/python/test.py 2023-10-13 10:10:04.974335 +0000
@@ -1,9 +1,11 @@
from flatted import stringify as _stringify, parse
...
black --exclude option is not a comma separated list, its a regular expression /
see:
$ black --help
...
--exclude TEXT A regular expression that matches files and
directories that should be excluded on recursive
searches ...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
DDG's bot detection is sensitive to the vqd value. For some search terms (such
as extremely long search terms that are often sent by bots), no vqd value can be
determined.
If SearXNG cannot determine a vqd value, then no request should go out to
DDG (WEB): a request with a wrong vqd value leads to DDG temporarily putting
SearXNG's IP on a block list.
Requests from IPs in this block list run into timeouts.
Not sure, but it seems the block list is a sliding window: to get my IP rid from
the bot list I had to cool down my IP for 1h (send no requests from that IP to
DDG).
Since such issues can't reproduce in a local instance I tested this patch 24h on
my public SearXNG instance: There are still errors (rare), but the reliability
is still 100%.
Related:
- https://github.com/searxng/searxng/pull/2922
- https://github.com/searxng/searxng/pull/2923
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Some search terms do not have results and therefore no vqd value
BTW: remove a leftover from 9197efa
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
We have had problems with this before, the bot protection from ddg-lite seems to
have included this referer in the rating [1][2].
From reverse engineering:
- The Referer ``https://google.com/`` was set in commt 257dc7d6c4 --> DDG lite
does not like this referer anymore!
- The 'Referer' header is only set on second and follow up pages but not on the
first page
- The vqd value is not needed on the first page, the ddg-lite client sets this
value only on follow up pages / this can help to reduce the vqd requests from
SearXNG.
Related to 'Referer' header & ddg requests:
[1] https://github.com/searxng/searxng/pull/2161
[2] https://github.com/searxng/searxng/pull/2081
Closes: https://github.com/searxng/searxng/issues/2796
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Anna’s Archive has cleaned up their languages, available file extensions and
changed the HTML form.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Caching files on the client side for more than a day can confuse the end user
when updating static files[1].
Depending on the way of providing a SearXNG instance via HTTP, there are several
ways to optimize the access to the /static files. However, since we don't know
what optimization an admin has provided for his static files, we should have
moderate settings in the defaults that run robustly in a wide variety of
installations.
In this sense, all caches on the client side should be cleared after one day at
the latest. So far the files were cached for one year on client side; as soon
as changes are made to the static files (with the option `static_use_hash:
true`) the old static files are kept for one year on the CLient side / which can
also be evaluated as unnecessary caching.
[1] https://github.com/searxng/searxng/discussions/2821
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Markus Heiser