Depending on the order the unit tests are executed, the searx.search module is
initalized or not, issue reported in [1]::
Traceback (most recent call last):
File "searxng/tests/unit/test_results.py", line 72, in test_result_merge_by_title
self.container.extend('stract', [fake_result(engine='stract', title='short title')])
File "searxng/searx/results.py", line 243, in extend
histogram_observe(standard_result_count, 'engine', engine_name, 'result', 'count')
File "searxng/searx/metrics/__init__.py", line 49, in histogram_observe
histogram_storage.get(*args).observe(duration)
^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get'
To ensure that the searx.search module is initialized, the
- searx.engines.load_engines is replace by
- searx.search.initialize
[1] https://github.com/searxng/searxng/pull/3932#discussion_r1822406569
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
A SearXNG maintainer on Matrix reported a traceback::
File "searxng-src/searx/engines/xpath.py", line 272, in response
dom = html.fromstring(resp.text)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "searx-pyenv/lib/python3.11/site-packages/lxml/html/__init__.py", line 850, in fromstring
doc = document_fromstring(html, parser=parser, base_url=base_url, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "searx-pyenv/lib/python3.11/site-packages/lxml/html/__init__.py", line 738, in document_fromstring
raise etree.ParserError(
lxml.etree.ParserError: Document is empty
I don't have an example to reproduce the issue, but the issue and this patch are
clearly recognizable even without an example.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In some results, Google returns a <script> tag that must be removed before
extracting the content.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The WEB page (PL only) has changed and there is now also a kind of CAPTCHA.
There is currently no possibility to restore the function of this engine.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The engines do not have / do not need a property `base_url`, lets remove it from
the settings.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The properties `item.service_medium` and `item.thumb_gallery` are not given for
every result item. It is more reliable to use the first (thumb) and
last (image) URL in the list of of URLs in `image_url`.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The engine has been revised; there is now the option ``adobe_content_types``
with which it is possible to configure engines for video and audio from the
adobe stock. BTW this patch adds documentation to the engine.
To test all three engines in one use a search term like::
!asi !asv !asa sound
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The quantities on the intro page were partly out of date / example; we already
have 210 engines and not just 70. To avoid having to change the quantities
manually in the future, they are now calculated from the jinja context
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In order to be able to implement error handling, it is necessary to know which
URL triggered the exception / the URL has not yet been logged.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The previous implementation could not distinguish a CAPTCHA response from an
ordinary result list. In the previous implementation a CAPTCHA was taken as a
result list where no items are in.
DDG does not block IPs. Instead, a CAPTCHA wall is placed in front of request
on a dubious request.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
All favicons implementations have been documented and moved to the Python
package:
searx.favicons
There is a configuration (based on Pydantic) for the favicons and all its
components:
searx.favicons.config
A solution for caching favicons has been implemented:
searx.favicon.cache
If the favicon is already in the cache, the returned URL is a data URL [1]
(something like `data:image/png;base64,...`). By generating a data url from
the FaviconCache, additional HTTP roundtripps via the favicon_proxy are saved:
favicons.proxy.favicon_url
The favicon proxy service now sets a HTTP header "Cache-Control: max-age=...":
favicons.proxy.favicon_proxy
The resolvers now also provide the mime type (data, mime):
searx.favicon.resolvers
[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The restriction of shell scripts to certain distributions is only required for
certain actions such as the installation of a SearXNG instance. The maintenance
scripts and build processes were previously also restricted to these specific
distributions. With this patch, the build processes (such as the build of
online documentation) can now also be executed on all Linux distributions.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Fault pattern: if there are no offers, then an exception has been thrown:
IndexError: list index out of range
This patch makes the addition of “best price” dependent on whether one exists.
Closes: #3685
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In my review of [1] I tried to reformat the template code of the language
menue whereby I have made two mistakes.
- default language was added twice
- in the 'Auto-detect' item a hard coded `[auto]` was implemented where the
`search_language` variable was needed.
[1] https://github.com/searxng/searxng/issues/3645
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The intention of this PR is to modernize the settings_loader implementations.
The concept is old (remember, this is partly from 2014), back then we only had
one config file, meanwhile we have had a folder with config files for a very
long time. Callers can now load a YAML configuration from this folder as
follows ::
settings_loader.get_yaml_cfg('my-config.yml')
- BTW this is a fix of #3557.
- Further the `existing_filename_or_none` construct dates back to times when
there was not yet a `pathlib.Path` in all Python versions we supported in the
past.
- Typehints have been added wherever appropriate
At the same time, this patch should also be downward compatible and not
introduce a new environment variable. The localization of the folder with the
configurations is further based on:
SEARXNG_SETTINGS_PATH (wich defaults to /etc/searxng/settings.yml)
Which means, the default config folder is `/etc/searxng/`.
ATTENTION: intended functional changes!
If SEARXNG_SETTINGS_PATH was set and pointed to a not existing file, the
previous implementation silently loaded the default configuration. This
behavior has been changed: if the file or folder does not exist, an
EnvironmentError exception will be thrown in future.
Closes: https://github.com/searxng/searxng/issues/3557
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Depending on the order in which the unit tests are executed, the python modules
of the engines are initialized (monkey patched) or not. As the order of the
tests is not static, random errors may occur.
To avaoid random `NameError: name 'logger' is not defined` in the unit tests of
the xpath engine, a logger is monkey patched into the xpath py-module.
```
make test.unit
TEST tests/unit
......EE...................
======================================================================
ERROR: test_response (tests.unit.engines.test_xpath.TestXpathEngine.test_response)
----------------------------------------------------------------------
Traceback (most recent call last):
File "./tests/unit/engines/test_xpath.py", line 60, in test_response
self.assertEqual(xpath.response(response), [])
^^^^^^^^^^^^^^^^^^^^^^^^
File "./searx/engines/xpath.py", line 309, in response
logger.debug("found %s results", len(results))
^^^^^^
NameError: name 'logger' is not defined
======================================================================
ERROR: test_response_results_xpath (tests.unit.engines.test_xpath.TestXpathEngine.test_response_results_xpath)
----------------------------------------------------------------------
Traceback (most recent call last):
File "./tests/unit/engines/test_xpath.py", line 102, in test_response_results_xpath
self.assertEqual(xpath.response(response), [])
^^^^^^^^^^^^^^^^^^^^^^^^
File "./searx/engines/xpath.py", line 309, in response
logger.debug("found %s results", len(results))
^^^^^^
NameError: name 'logger' is not defined
```
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The domains of zlibrary instances are known to be seized from time to time.
This leads to problems when, for example, the automated tasks try to update the
engine traits (aka fetch_traits). The search function should also generate a
suitable error message (currently either SSL errors or empty result lists are
returned). [1]
[1] https://github.com/searxng/searxng/issues/3610
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The use of img_src AND thumbnail in the default results makes no sense (only a
thumbnail is needed). In the current state this is rather confusing, because
img_src is displayed like a thumbnail (small) and thumbnail is displayed like an
image (large).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The names of the links are rather tags than real names, and they sometimes vary
greatly in their spelling:
- GitHub: github, Github
- Source code: Repository, SCM, Project Source Code
- Documentation: docs, Documentation
It was standardized to terms such as 'Source code' and 'Documentation', as
translations already exist for these terms.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- l10n support: parse and format decimal numbers by babel
- ability to add additional units
- improved unit detection (symbols are not unique)
- support for alias units (0,010C to F --> 32,018 °F)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Another trip into the hell of dependencies: docutils tends to put major changes
in minor patches: the executables have been renamed / e.g.
rst2html.py --> rts2html
so we have to use docutils at least from version 0.21.2, but this version of
docutils is only supported by myst-parser from version 3.0.1 on.
Additionally, docutils decided to drop python 3.8 in version 0.21 [1]
Further, linuxdoc needed an update to cope with docutils 0.21 [2]
[1] https://docutils.sourceforge.io/RELEASE-NOTES.html#release-0-21-2024-04-09
[2] https://github.com/return42/linuxdoc/pull/36
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Markus Heiser
