Markus Heiser
3829c253ff
[mod] add option max_page to bing, brave, qwant, startpage & mojeek
...
[1] https://github.com/searxng/searxng/issues/2982#issuecomment-1808975780
Reported-by: @Damaj301damaj-lol [1]
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-12-03 13:47:17 +01:00
Markus Heiser
2274d55d5a
[mod] add option max_page
...
Related: https://github.com/searxng/searxng/issues/2982
Closes: https://github.com/searxng/searxng/issues/2972
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-12-03 13:47:17 +01:00
Bnyro
bf75a8c2a0
[feat] engine: implementation of bpb
2023-11-27 16:46:41 +01:00
Bnyro
04cfce2eb8
[feat] yep: support for images and news (and safesearch)
2023-11-27 15:07:26 +01:00
Markus Heiser
44392bd436
[mod] improve implementation of presearch engine
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-11-27 14:16:42 +01:00
Bnyro
23582aac5c
[feat] implementation of presearch engine
2023-11-27 14:16:42 +01:00
Bnyro
c3cc24be12
[feat] engine: implementation of destatis
2023-11-27 13:54:48 +01:00
Émilien (perso)
4280318fc5
fixing results parsing brave
2023-10-13 11:47:30 +02:00
Hackurei
efada7cba2
[fix] hackernews keyerror problem
2023-10-13 08:16:47 +02:00
Hackurei
af071121de
[fix] imgur - incorrect wikidata id
2023-10-12 09:14:00 +02:00
Markus Heiser
14323d683f
[fix] ddg-lite & ddg-extra: don't send empty vqd value
...
DDG's bot detection is sensitive to the vqd value. For some search terms (such
as extremely long search terms that are often sent by bots), no vqd value can be
determined.
If SearXNG cannot determine a vqd value, then no request should go out to
DDG (WEB): a request with a wrong vqd value leads to DDG temporarily putting
SearXNG's IP on a block list.
Requests from IPs in this block list run into timeouts.
Not sure, but it seems the block list is a sliding window: to get my IP rid from
the bot list I had to cool down my IP for 1h (send no requests from that IP to
DDG).
Since such issues can't reproduce in a local instance I tested this patch 24h on
my public SearXNG instance: There are still errors (rare), but the reliability
is still 100%.
Related:
- https://github.com/searxng/searxng/pull/2922
- https://github.com/searxng/searxng/pull/2923
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-12 08:52:28 +02:00
Markus Heiser
3388441917
[fix] ddg-lite vqd value: some search terms do not have a vqd value
...
Some search terms do not have results and therefore no vqd value
BTW: remove a leftover from 9197efa
2023-10-10 09:12:30 +02:00
Markus Heiser
9197efa2a7
[fix] duckduckgo lite engine: set HTTP header 'Referer'
...
We have had problems with this before, the bot protection from ddg-lite seems to
have included this referer in the rating [1][2].
From reverse engineering:
- The Referer ``https://google.com/ `` was set in commt 257dc7d6c4https://github.com/searxng/searxng/pull/2161
[2] https://github.com/searxng/searxng/pull/2081
Closes: https://github.com/searxng/searxng/issues/2796
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-10 08:40:53 +02:00
Bnyro
fa5b2a7948
[mod] yacy: use official instance by default and fix crashes
2023-10-09 20:50:24 +02:00
Hackurei
ff78b1a902
[feat] implement hackernews engine - news.ycombinator.com
2023-10-09 14:00:04 +02:00
Aine
213cb74378
[fix] matrixrooms add proper MRS integration
...
Related:
- https://github.com/searxng/searxng/issues/2918
2023-10-09 13:25:13 +02:00
Bnyro
48cb58bd2e
[feat] duckduckgo: support for videos and news
2023-10-09 06:53:43 +02:00
Bnyro
c3ab49cd90
[fix] kickass: crash when no results
2023-10-07 11:48:23 +02:00
Bnyro
f22daf8b47
[mod] piped: always show video length if available
2023-10-07 11:45:46 +02:00
Bnyro
ce270961e8
[feat] engine: implementation of mastodon
2023-10-06 10:58:23 +02:00
Markus Heiser
fd1422a670
[mod] engine - simplify region & lang handling, make filters configurable
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-05 10:55:08 +02:00
Bnyro
3e2ae756f0
[feat] engine: implementation of radio-browser.info
2023-10-05 10:55:08 +02:00
Jinyuan Huang
e509cb7c45
[typo] solved a typo in yahoo error message.
2023-10-01 08:29:06 +02:00
Jinyuan Huang
d4d9f2073e
[fix] Bug: Yahoo results for simplified Chinese search sometimes have the first character cut off #2866
...
Co-authored-by: Blair Noctis <n@sail.ng>
2023-10-01 08:29:06 +02:00
Bnyro
fe9386b58d
[fix] emojipedia: fix engine
2023-10-01 08:19:45 +02:00
Markus Heiser
32a4ea350e
[fix] Revision of the Bing engines
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-01 08:01:38 +02:00
jazzzooo
079636c079
[fix] engine - bing fix search, pagination, remove safesearch
2023-10-01 08:01:38 +02:00
Bnyro
5ce1792432
[feat] engine: implementation of pinterest
2023-09-30 15:01:45 +02:00
Bnyro
6096457e4d
[fix] matrixrooms.info: pagination not working properly
2023-09-30 14:51:07 +02:00
Markus Heiser
e1a8b8189f
[fix] engine - moviepilot instead of thumbnail use img_src
...
Instead of thumbnail use img_src in the result item, otherwise the "movies"
categories looks clunky.
Related:
- b4e0d2eedc (r128785388)
2023-09-30 11:29:19 +02:00
Bnyro
159629c588
[mod] tagesschau: add option to only use tagesschau urls
2023-09-30 11:00:11 +02:00
Bnyro
2ca60a19fc
[feat] engine: implementation of matrixrooms.info
2023-09-30 09:09:23 +02:00
Bnyro
fc4a20f734
[mod] add movies category for tmdb, imdb and moviepilot
2023-09-29 22:37:51 +02:00
jazzzooo
e37d775fa2
[fix] engine - currency fix and simplify
2023-09-28 08:29:38 +02:00
Jinyuan Huang
ae28d429c9
[fix] bilibili new api used
2023-09-28 08:24:51 +02:00
jazzzooo
1a66d74673
[fix] engine - kickass update url, fix parsing, use multiple mirrors
2023-09-27 10:19:41 +02:00
Markus Heiser
b428ccc5a0
[fix] engine brave - fetch traits (modified settings menu)
...
Brave has changed it settings menu fundamental. Region codes are no longer in
the HTML page, we have to read the regional codes from a JS:
https://cdn.search.brave.com/serp/v2/_app/immutable/chunks/parameters.734c106a.js
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-27 09:12:38 +02:00
Markus Heiser
3a456b1282
[fix] engine annas archive - fetch traits (modified xpath selectors)
...
Anna’s Archive has cleaned up their languages, available file extensions and
changed the HTML form.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-27 09:12:38 +02:00
Émilien (perso)
1851f27154
[mod] remove twitter ( #2843 )
2023-09-24 08:32:53 +00:00
Bnyro
75c9de02d1
[feat] engine: implementation of imgur
2023-09-22 20:50:53 +02:00
Bnyro
fb72f71f0a
[fix] internet archive scholar: crash when there's no title
2023-09-22 18:49:39 +02:00
Markus Heiser
71358e9c67
Revert "[fix] engine - duckduckgo vqd edge-case"
...
This reverts commit 102502a4f0
2023-09-22 09:31:25 +02:00
Bnyro
51236ae47a
[feat] engine: implementation of chefkoch.de
2023-09-21 17:23:59 +02:00
jazzzooo
8bcca0e620
[fix] engine - brave don't show ads
2023-09-21 16:55:39 +02:00
jazzzooo
b729542a66
[fix] engine - google images error when no results
2023-09-21 16:38:37 +02:00
Bnyro
cc2e0537a3
[feat] engine: implementation of google icons/material design icons
2023-09-21 15:16:49 +02:00
Bnyro
c999cfb422
[feat] engine: implementation of wallhaven
2023-09-21 14:25:43 +02:00
jazzzooo
102502a4f0
[fix] engine - duckduckgo vqd edge-case
2023-09-20 20:05:06 +02:00
Markus Heiser
043dcbf7c5
[fix] engine qwant (web-lite) - ignore advertising adds
...
Closes: https://github.com/searxng/searxng/issues/2812
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-19 17:06:56 +02:00
Émilien (perso)
ad725ce7d7
wikipedia wikidata infobox + disable wikisource ( #2806 )
...
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-19 10:31:02 +02:00
Bnyro
efd3a2d6d1
[feat] engine: implementation of internet archive scholar
2023-09-18 18:12:00 +02:00
jazzzooo
223b3487c3
[fix] spelling
2023-09-18 16:20:27 +02:00
Markus Heiser
a9b6963971
[fix] engine - qwant delivers only 5 pages maximum
...
all qwant engines (incl qwant-lite) delivers only 5 pages maximum
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-18 01:43:43 +02:00
jazzzooo
da1446c5ed
[fix] engine - qwant wrong error type
2023-09-18 01:43:43 +02:00
Markus Heiser
7398d525c8
[fix] qwant: subsequent fix of commit d9dbcedeb
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-17 17:52:56 +02:00
Markus Heiser
d9dbcedeb6
[feat] implementation of qwant lite for web search
...
Related: https://github.com/searxng/searxng/issues/2719
Replace: https://github.com/searxng/searxng/pull/2748
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-17 16:53:25 +02:00
Bnyro
b4e0d2eedc
[feat] engine: implemenation of moviepilot (de)
2023-09-17 14:30:56 +02:00
jazzzooo
7dfcc3386e
[fix] tagesschau videos
2023-09-16 18:40:26 +02:00
jazzzooo
ec540a967a
[fix] brave.videos
2023-09-15 22:00:09 +02:00
jazzzooo
27477f51fd
[fix] brave.news
2023-09-15 22:00:09 +02:00
Justas Zabulionis
41ef73ca3a
[fix] rumble redirect
2023-09-14 19:21:21 +02:00
Justas Zabulionis
be888810ba
[fix] pubmed content being None
2023-09-14 18:40:15 +02:00
Justas Zabulionis
92d39de410
[fix] solidtorrents redirects
2023-09-14 18:03:21 +02:00
Justas Zabulionis
cf8a6cf6db
[fix] solidtorrents pagination
2023-09-14 18:03:21 +02:00
Justas Zabulionis
8172f89075
[fix] solidtorrents
2023-09-14 18:03:21 +02:00
jazzzooo
74600c028d
[fix] engine - Crossref
...
Crossref was broken on result types journal-issue and component .. The old code
had lots of assumptions, and broke during parsing. Now the assumptions are more
explicit and checked them with the API.
2023-09-14 17:39:23 +02:00
Bnyro
3568a3cafb
[feat] odysee: implement fetch_traits for language support
2023-09-13 21:41:33 +02:00
Bnyro
09c61dabc9
[mod] odysee: time range support
2023-09-13 21:41:33 +02:00
jazzzooo
b98907e91f
[fix] engine - piped.music incorrect timestamps
2023-09-13 21:39:37 +02:00
jazzzooo
6039dbf211
[fix] engine - invidious thumbnails
2023-09-13 11:37:42 +02:00
jazzzooo
b2fd6304bf
[fix] engine - openstreetmap currency rendering
2023-09-13 10:56:52 +02:00
jazzzooo
54a3e03b45
[fix] engine - openstreetmap currency matching
2023-09-12 20:57:05 +02:00
Bnyro
64d9587ac8
[feat] new engine: svgrepo
2023-09-12 20:38:36 +02:00
jazzzooo
b189578b6b
[fix] engine - brave
2023-09-12 11:31:43 +02:00
Bnyro
f182abd6f8
[mod] library of congress: fix engine
2023-09-11 19:42:31 +02:00
Bnyro
e73a6f5d14
[fix] engine deviantart: review of the result-scrapper
...
The deviantart site changed and hence deviantart is currently unusable.
2023-09-11 13:22:36 +02:00
Alexandre Flament
d07c006aed
Replace chompjs with pure Python code
...
The new implementation is good enough for the current usage (brave)
2023-09-09 13:02:36 +02:00
Bnyro
9e83c0dedc
[feat] engine: implementation of Yummly
...
Co-authored-by: Markus Heiser <markus.heiser@damarit.de>
2023-09-08 11:47:13 +02:00
Bnyro
a3d7e9c285
[mod] utils.py: add markdown_to_text helper function
2023-09-08 11:47:13 +02:00
Hackurei
1f21ac7d62
[feat] engine: implementation of bilibili https://www.bilibili.com
2023-09-05 22:53:03 +02:00
Markus Heiser
696c35d2c3
[fix] engine - duckduckgo_images / determination of vqd value incorrect
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-05 22:24:51 +02:00
bonswouar
4fb26cd96d
[fix] engine duckduckgo weather api changes
2023-09-05 16:55:00 +02:00
Markus Heiser
01be9e0e20
[fix] engine: wikicommons - don't quoute ':|' in URL parameters
...
From [1]: It seems to be because of [2] For some reason it gets url encoded
twice, resulting in
- ``filetype%253Abitmap%257Cdrawing+birds`` instead of
- ``filetype:bitmap%7Cdrawing+birds``
[1] https://github.com/searxng/searxng/issues/2707
[2] https://github.com/searxng/searxng/blob/master/searx/engines/wikicommons.py#L43
Closes : #2707
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-03 18:45:39 +02:00
Markus Heiser
4f8895c6de
[fix] follow-up of 4da7003ae / add missing review from @Bnyro
...
[1] https://github.com/searxng/searxng/pull/2656#pullrequestreview-1607956209
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-02 09:26:45 +02:00
Hackurei
4da7003ae0
[feat] engine: implementation of odysee
2023-09-02 09:14:12 +02:00
Bnyro
9c4e9d3814
[feat] implementation of Wikimedia commons for images
2023-09-01 18:39:24 +02:00
Alexandre Flament
faa4280e1a
[mod] bing: resolve redirect without additional requests
...
Remove the usage of searx.network.multi_requests
The results from Bing contains the target URL encoded in base64
See the u parameter, remove the first two character "a1", and done.
Also add a comment the check of the result_len / pageno
( from https://github.com/searx/searx/pull/1387 )
2023-08-29 07:39:06 +02:00
Markus Heiser
b0d2cd5ca9
[doc] add documentation of Mwmbl engine & autocompleter
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-27 17:25:26 +02:00
Bnyro
19300a5659
[mod] engine mwmbl: add link to official api docs
2023-08-27 17:25:26 +02:00
Alexandre Flament
e16c007c22
[fix] openstreetmap engine
...
It seems there is an API change:
extratags can be either a dictionnary or None.
This commit avoid crash when extratags is None
Test query "!osm gare du nord"
2023-08-27 11:49:16 +02:00
Markus Heiser
0647f83a3e
[fix] google engine: don't overspecify the search query to Google
...
The method EngineTraits.get_region(..) returns engine's region string
that **best fits** to SearXNG's locale. This means it returns a
region (country) if only a language is set in the locale. By example the method
returns for a locale tag `es` a region `ES`.
Google's search parameter `cr` restricts search results to documents originating
in a particular country / in case of a locale tag (language) as described above,
this argument should be unset in the query send to Google.
Closes: https://github.com/searxng/searxng/issues/2672
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-26 07:47:07 +02:00
Markus Heiser
4b42644579
[fix] engine google_video: google has changed the layout of the rsponse
...
Closes: https://github.com/searxng/searxng/issues/2664
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-22 08:34:04 +02:00
Bnyro
c59ae91b76
[feat] engine: implementation of mwmbl
2023-08-19 18:23:42 +02:00
Markus Heiser
c741fc6f00
[mod] currency_convert: support for showing the answer source url
...
Show URL of the ddg-search page, not the URL of a (generic) Javascript. The
latter one is not usefull for the user.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-18 19:07:14 +02:00
Markus Heiser
e2744520f8
[mod] google: support for showing the answer source url
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-18 19:07:14 +02:00
Bnyro
5ec7df3480
[mod] engine duckduckgo definitions: support for answer source
2023-08-18 19:07:14 +02:00
Bnyro
64bc98b5fb
[mod] brave: support for showing the answer source url
2023-08-18 19:07:14 +02:00
Markus Heiser
9100a48541
[mod] improve seekr engines and add documentation
...
Tis patch adds some more fields to the result items and changed paging to the
``nextResultSet`` given in seekr's JSON response.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-15 16:17:42 +02:00
Bnyro
2bab658d39
[feat] engine: implementation of seekr for news, images and videos
2023-08-15 16:17:42 +02:00
Bnyro
e25d1c7288
[feat] engine: implementation of German news, Tagesschau
...
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-10 20:27:54 +02:00
