8,048 Commits 6 Branches 17 Tags 102 MiB
c151683a0b
Commit Graph

1695 Commits

Author SHA1 Message Date
Markus Heiser
f1a148f53e [fix] ddg engine: if no vqd value can be determined, don't save None 
Closes: https://github.com/searxng/searxng/issues/3370
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2024-04-08 10:56:11 +02:00
Austin-Olacsi
b6fe2cad71 [mod] Ask engine: remove tracking paramaters and set max page to 5 2024-04-08 07:50:14 +02:00
Bnyro
bf3845b093 [feat] engine: implementation of Open Meteo 2024-04-07 19:02:53 +02:00
Mikkel Denker
b21aaa8907 Add user-agent to stract requests 2024-03-22 13:40:46 +01:00
Markus Heiser
542f7d0d7b [mod] pylint all files with one profile / drop PYLINT_SEARXNG_DISABLE_OPTION 
In the past, some files were tested with the standard profile, others with a
profile in which most of the messages were switched off ... some files were not
checked at all.

- ``PYLINT_SEARXNG_DISABLE_OPTION`` has been abolished
- the distinction ``# lint: pylint`` is no longer necessary
- the pylint tasks have been reduced from three to two

  1. ./searx/engines -> lint engines with additional builtins
  2. ./searx ./searxng_extra ./tests -> lint all other python files

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2024-03-11 14:55:38 +01:00
Markus Heiser
8205f170ff [mod] pylint all engines without PYLINT_SEARXNG_DISABLE_OPTION 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2024-03-11 14:55:38 +01:00
Markus Heiser
707d6270c8 [doc] engine: mullvad leta 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2024-03-10 18:20:07 +01:00
Grant Lanham
9d6896a771 [feat] engine: implementation of mullvad leta 2024-03-10 18:20:07 +01:00
Markus Heiser
0ffec440b2 [doc] annas_archive: ctegs journal_article & book_any has been removed 
In commit 8af181533 in PR:

- https://github.com/searxng/searxng/pull/3321

the category `journal_article` has been removed, `book_any` has been removed
longer time ago.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2024-03-10 12:46:00 +01:00
Markus Heiser
e97e1f9110 [fix] duckduckgo.fetch_traist - URL of region definitions has changed 
- https://duckduckgo.com/dist/util/u.7669f071a13a7daa57cb.js

updated from u661.js to u.7669f071a13a7daa57cb / should be updated
automatically?  The last change was on March 23rd in dba8977b09 [1]

- [1] https://github.com/searxng/searxng/pull/2269

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2024-03-10 10:32:54 +01:00
Markus Heiser
33c099022b [drop] engine framalibre - sarch URL is no longer supported 
Closes: https://github.com/searxng/searxng/issues/3286
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2024-03-09 18:00:45 +01:00
Alexandre Flament
42515d98f7 engine npm use the packages.html template 2024-03-09 10:24:49 +01:00
Bnyro
6dd07b88a4 [feat] lib.rs: use packages.html template 2024-03-09 10:17:16 +01:00
Markus Heiser
a7b51f023e [black] upgrade black 22.12.0 --> 24.2.0 
The issue discussed in [1] has been solved since [2] has been merged into black
/ now we can upgrade without touching 69 files as it was needed with black
23.1.0 [3].

[1] https://github.com/searxng/searxng/pull/2159#issuecomment-1425723977
[2] https://github.com/psf/black/pull/4060
[3] https://github.com/searxng/searxng/pull/2159/files

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2024-03-09 08:15:50 +01:00
Bnyro
44decaa524 [feat] pkg.go.dev: use packages.html template 2024-03-08 17:18:54 +01:00
Markus Heiser
a5b81e2555 [fix] nyaa engine - paging support & filesize (GiB) 
BTW: pylint engine

Closes: https://github.com/searxng/searxng/issues/3290
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2024-03-07 07:38:31 +01:00
Markus Heiser
3529f0af50 [fix] engine searchcode.com - paging is broken in searchcode.com's API 
paging is broken in searchcode.com's API .. not sure it will ever been fixed /
this commit disables paging in the engine and BTW pylint `searchcode_code.py`.

Closes: https://github.com/searxng/searxng/issues/3287
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2024-03-06 12:33:29 +01:00
Markus Heiser
5602f0bf96 [fix] engine pinterest - ignore result type 'story' 
Close: https://github.com/searxng/searxng/issues/3289
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2024-03-06 07:33:43 +01:00
Markus Heiser
d97b84bea2 [fix] ddg engines (get_vqd) - the vqd value is no longer in the form 
Closes: https://github.com/searxng/searxng/issues/3276
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2024-03-05 16:27:04 +01:00
Yaksh Bariya
b1431e1670 [feat] engine: implementation of cppreference 2024-03-03 17:07:29 +01:00
Bnyro
f3b4bf86a7 [feat] engine: implementation of void linux packages 
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
 2024-02-29 13:12:40 +01:00
Alexandre Flament
d58760ef75 [mod] pypi engine: use packages.html 2024-02-29 07:48:44 +01:00
Austin-Olacsi
9330a072eb [feat] engine: implementation of pixiv 2024-02-25 16:29:57 +01:00
Bnyro
e76ab1a4b3 [refactor] images: add resolution, image format and filesize fields 
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
 2024-02-25 16:22:37 +01:00
Markus Heiser
b683aa63fb [feat] github: use packages template 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2024-02-25 14:56:57 +01:00
Bnyro
1d6f475fbc [feat] dockerhub: use packages template 2024-02-25 14:56:57 +01:00
Bnyro
cfbe59b6b3 [feat] engine: implementation of mozhi 2024-02-23 07:50:48 +01:00
Xinos
3c42252c78 [feat] engine: implementation of yandex music 
Access to music is limited to a few countries:

- https://yandex.com/support/music/access.html
 2024-02-21 17:17:12 +01:00
micsthepick
fde21c8f99 [feat] engine: Add LiveSpace livestreams 2024-02-20 14:45:07 +01:00
Markus Heiser
76845ea42c [mod] engine ask.com - parse JS result to JSON 
Parse the result list from ask.com given in the variable named
window.MESON.initialState::

    <script nonce="..">
        window.MESON = window.MESON || {};
        window.MESON.initialState = {"siteConfig": ...
          ...}};
        window.MESON.loadedLang = "en";
    </script>

The result list is in field::

    json_resp['search']['webResults']['results']

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2024-02-18 09:17:22 +01:00
Hackurei
3df53d6e50 [feat] engine: implementation of ask.com 2024-02-18 09:17:22 +01:00
Bnyro
11c0651ef4 [fix] lingva: redirect and parsing error 2024-02-11 11:28:47 +01:00
Hackurei
b0fe0924a3 [feat] engine: implementation of brave goggles 2024-02-10 08:09:50 +01:00
Markus Heiser
c14b117121 [fix] brave.search - update result selectors to fit to new layout 
Closes: https://github.com/searxng/searxng/issues/3167
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2024-01-31 16:21:56 +01:00
Bnyro
efbee96b1d [feat] brave: fix empty content and parse published dates 2024-01-30 07:41:45 +01:00
Bnyro
559b860618 [feat] engine: implementation of goodreads 2024-01-29 16:07:24 +01:00
Bnyro
8c73aa772b [fix] semantic scholar: bad request 2024-01-25 06:12:08 +01:00
Jinyuan Huang
9c5f9735cf [fix] engine: safesearch parameter in Google Videos engine (#2762) 
Closes: https://github.com/searxng/searxng/issues/2762
 2024-01-16 15:29:09 +01:00
Markus Heiser
e560d7e373 [mod] presearch: add language & region support 
In Presearch there are languages for the UI and regions for narrowing down the
search.  With this change the SearXNG engine supports a search by region.  The
details can be found in the documentation of the source code.

To test, you can search terms like::

   !presearch bmw :zh-TW
   !presearch bmw :en-CA

1. You should get results corresponding to the region (Taiwan, Canada)
2. and in the language (Chinese, Englisch).
3. The context in info box content is in the same language.

Exceptions:

1. Region or language is not supported by Presearch or
2. SearXNG user did not selected a region tag, example::

    !presearch bmw :en

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2024-01-15 19:23:26 +01:00
Markus Heiser
a2c269bbac [mod] presearch: hardening engine's response against KeyErrors 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2024-01-15 19:23:26 +01:00
Bnyro
0a78f59aba [fix] presearch: safesearch, time ranges, crash when no result found 2024-01-15 19:23:26 +01:00
Bnyro
621e1313af [feat] engine: implementation of stract (beta) 2024-01-07 11:18:16 +01:00
Markus Heiser
8dfc1dbc5c [fix] engine - offical URL moved to https://lingva.thedaviddelta.com/ 
The ML top-level domain has been removed from Freenom ... moved the official
instance to https://lingva.thedaviddelta.com  [1]

To test use translation syntax:

    !lv en-de Alternative front-end for Google Translate

[1] https://fosstodon.org/@thedaviddelta/111376540936289966

Closes: https://github.com/searxng/searxng/issues/2718
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-12-25 09:57:35 +01:00
Markus Heiser
6df922b9da [fix] fetch_traits of google and zlibrary 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-12-24 08:28:32 +01:00
Bnyro
8a4104b992 [feat] engine: implementation of rotten tomatoes 2023-12-11 18:59:36 +01:00
Bnyro
dda74fc9c9 [feat] engine: implementation of mdn 2023-12-11 18:57:40 +01:00
Bnyro
b5b81c9806 [feat] engine: implementation of fyyd 2023-12-11 18:56:32 +01:00
Bnyro
527e13ab46 [feat] engine: implementation of podcastindex.org 2023-12-11 18:55:16 +01:00
Bnyro
3bc85c511c [mod] tootfinder: implement python module 2023-12-11 18:54:24 +01:00
Markus Heiser
3829c253ff [mod] add option max_page to bing, brave, qwant, startpage & mojeek 
[1] https://github.com/searxng/searxng/issues/2982#issuecomment-1808975780

Reported-by: @Damaj301damaj-lol [1]
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-12-03 13:47:17 +01:00
Markus Heiser
2274d55d5a [mod] add option max_page 
Related: https://github.com/searxng/searxng/issues/2982
Closes: https://github.com/searxng/searxng/issues/2972

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-12-03 13:47:17 +01:00
Bnyro
bf75a8c2a0 [feat] engine: implementation of bpb 2023-11-27 16:46:41 +01:00
Bnyro
04cfce2eb8 [feat] yep: support for images and news (and safesearch) 2023-11-27 15:07:26 +01:00
Markus Heiser
44392bd436 [mod] improve implementation of presearch engine 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-11-27 14:16:42 +01:00
Bnyro
23582aac5c [feat] implementation of presearch engine 2023-11-27 14:16:42 +01:00
Bnyro
c3cc24be12 [feat] engine: implementation of destatis 2023-11-27 13:54:48 +01:00
Émilien (perso)
4280318fc5 fixing results parsing brave 2023-10-13 11:47:30 +02:00
Hackurei
efada7cba2 [fix] hackernews keyerror problem 2023-10-13 08:16:47 +02:00
Hackurei
af071121de [fix] imgur - incorrect wikidata id 2023-10-12 09:14:00 +02:00
Markus Heiser
14323d683f [fix] ddg-lite & ddg-extra: don't send empty vqd value 
DDG's bot detection is sensitive to the vqd value.  For some search terms (such
as extremely long search terms that are often sent by bots), no vqd value can be
determined.

If SearXNG cannot determine a vqd value, then no request should go out to
DDG (WEB): a request with a wrong vqd value leads to DDG temporarily putting
SearXNG's IP on a block list.

Requests from IPs in this block list run into timeouts.

Not sure, but it seems the block list is a sliding window: to get my IP rid from
the bot list I had to cool down my IP for 1h (send no requests from that IP to
DDG).

Since such issues can't reproduce in a local instance I tested this patch 24h on
my public SearXNG instance: There are still errors (rare), but the reliability
is still 100%.

Related:

- https://github.com/searxng/searxng/pull/2922
- https://github.com/searxng/searxng/pull/2923

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-10-12 08:52:28 +02:00
Markus Heiser
3388441917 [fix] ddg-lite vqd value: some search terms do not have a vqd value 
Some search terms do not have results and therefore no vqd value

BTW: remove a leftover from 9197efa

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-10-10 09:12:30 +02:00
Markus Heiser
9197efa2a7 [fix] duckduckgo lite engine: set HTTP header 'Referer' 
We have had problems with this before, the bot protection from ddg-lite seems to
have included this referer in the rating [1][2].

From reverse engineering:

- The Referer ``https://google.com/`` was set in commt 257dc7d6c4 --> DDG lite
  does not like this referer anymore!

- The 'Referer' header is only set on second and follow up pages but not on the
  first page

- The vqd value is not needed on the first page, the ddg-lite client sets this
  value only on follow up pages / this can help to reduce the vqd requests from
  SearXNG.

Related to 'Referer' header & ddg requests:

[1] https://github.com/searxng/searxng/pull/2161
[2] https://github.com/searxng/searxng/pull/2081

Closes: https://github.com/searxng/searxng/issues/2796
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-10-10 08:40:53 +02:00
Bnyro
fa5b2a7948 [mod] yacy: use official instance by default and fix crashes 2023-10-09 20:50:24 +02:00
Hackurei
ff78b1a902 [feat] implement hackernews engine - news.ycombinator.com 2023-10-09 14:00:04 +02:00
Aine
213cb74378 [fix] matrixrooms add proper MRS integration 
Related:

- https://github.com/searxng/searxng/issues/2918
 2023-10-09 13:25:13 +02:00
Bnyro
48cb58bd2e [feat] duckduckgo: support for videos and news 2023-10-09 06:53:43 +02:00
Bnyro
c3ab49cd90 [fix] kickass: crash when no results 2023-10-07 11:48:23 +02:00
Bnyro
f22daf8b47 [mod] piped: always show video length if available 2023-10-07 11:45:46 +02:00
Bnyro
ce270961e8 [feat] engine: implementation of mastodon 2023-10-06 10:58:23 +02:00
Markus Heiser
fd1422a670 [mod] engine - simplify region & lang handling, make filters configurable 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-10-05 10:55:08 +02:00
Bnyro
3e2ae756f0 [feat] engine: implementation of radio-browser.info 2023-10-05 10:55:08 +02:00
Jinyuan Huang
e509cb7c45 [typo] solved a typo in yahoo error message. 2023-10-01 08:29:06 +02:00
Jinyuan Huang
d4d9f2073e [fix] Bug: Yahoo results for simplified Chinese search sometimes have the first character cut off #2866 
Co-authored-by: Blair Noctis <n@sail.ng>
 2023-10-01 08:29:06 +02:00
Bnyro
fe9386b58d [fix] emojipedia: fix engine 2023-10-01 08:19:45 +02:00
Markus Heiser
32a4ea350e [fix] Revision of the Bing engines 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-10-01 08:01:38 +02:00
jazzzooo
079636c079 [fix] engine - bing fix search, pagination, remove safesearch 2023-10-01 08:01:38 +02:00
Bnyro
5ce1792432 [feat] engine: implementation of pinterest 2023-09-30 15:01:45 +02:00
Bnyro
6096457e4d [fix] matrixrooms.info: pagination not working properly 2023-09-30 14:51:07 +02:00
Markus Heiser
e1a8b8189f [fix] engine - moviepilot instead of thumbnail use img_src 
Instead of thumbnail use img_src in the result item, otherwise the "movies"
categories looks clunky.

Related:

- b4e0d2eedc (r128785388)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-09-30 11:29:19 +02:00
Bnyro
159629c588 [mod] tagesschau: add option to only use tagesschau urls 2023-09-30 11:00:11 +02:00
Bnyro
2ca60a19fc [feat] engine: implementation of matrixrooms.info 2023-09-30 09:09:23 +02:00
Bnyro
fc4a20f734 [mod] add movies category for tmdb, imdb and moviepilot 2023-09-29 22:37:51 +02:00
jazzzooo
e37d775fa2 [fix] engine - currency fix and simplify 2023-09-28 08:29:38 +02:00
Jinyuan Huang
ae28d429c9 [fix] bilibili new api used 2023-09-28 08:24:51 +02:00
jazzzooo
1a66d74673 [fix] engine - kickass update url, fix parsing, use multiple mirrors 2023-09-27 10:19:41 +02:00
Markus Heiser
b428ccc5a0 [fix] engine brave - fetch traits (modified settings menu) 
Brave has changed it settings menu fundamental.  Region codes are no longer in
the HTML page, we have to read the regional codes from a JS:

  https://cdn.search.brave.com/serp/v2/_app/immutable/chunks/parameters.734c106a.js

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-09-27 09:12:38 +02:00
Markus Heiser
3a456b1282 [fix] engine annas archive - fetch traits (modified xpath selectors) 
Anna’s Archive has cleaned up their languages, available file extensions and
changed the HTML form.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-09-27 09:12:38 +02:00
Émilien (perso)
1851f27154
[mod] remove twitter (#2843) 2023-09-24 08:32:53 +00:00
Bnyro
75c9de02d1 [feat] engine: implementation of imgur 2023-09-22 20:50:53 +02:00
Bnyro
fb72f71f0a [fix] internet archive scholar: crash when there's no title 2023-09-22 18:49:39 +02:00
Markus Heiser
71358e9c67 Revert "[fix] engine - duckduckgo vqd edge-case" 
This reverts commit 102502a4f0.
 2023-09-22 09:31:25 +02:00
Bnyro
51236ae47a [feat] engine: implementation of chefkoch.de 2023-09-21 17:23:59 +02:00
jazzzooo
8bcca0e620 [fix] engine - brave don't show ads 2023-09-21 16:55:39 +02:00
jazzzooo
b729542a66 [fix] engine - google images error when no results 2023-09-21 16:38:37 +02:00
Bnyro
cc2e0537a3 [feat] engine: implementation of google icons/material design icons 2023-09-21 15:16:49 +02:00
Bnyro
c999cfb422 [feat] engine: implementation of wallhaven 2023-09-21 14:25:43 +02:00
jazzzooo
102502a4f0 [fix] engine - duckduckgo vqd edge-case 2023-09-20 20:05:06 +02:00
Markus Heiser
043dcbf7c5 [fix] engine qwant (web-lite) - ignore advertising adds 
Closes: https://github.com/searxng/searxng/issues/2812
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-09-19 17:06:56 +02:00
Émilien (perso)
ad725ce7d7
wikipedia wikidata infobox + disable wikisource (#2806) 
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
 2023-09-19 10:31:02 +02:00
Bnyro
efd3a2d6d1 [feat] engine: implementation of internet archive scholar 2023-09-18 18:12:00 +02:00