retoor/searxng
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
3,688 Commits 6 Branches 17 Tags
bef2f2efa8fe20bb40c9e79d91c2d9b42f33854a
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Alexandre Flament bef2f2efa8 [fix] wikidata: fix crash when the item has no description at all and at least one URL. 2020-12-04 17:17:20 +01:00
Alexandre Flament f56e78ee80 Merge pull request #2338 from unixfox/issue-template 
add new issue template
 2020-12-03 19:16:11 +01:00
Alexandre Flament cec73b5dcf Merge pull request #2350 from dalf/mod-engines-report-captcha 
[mod] stackoverflow & yandex: detect CAPTCHA response
 2020-12-03 13:50:12 +01:00
Alexandre Flament fa909c7c02 [mod] stackoverflow & yandex: detect CAPTCHA response 2020-12-03 13:23:19 +01:00
Alexandre Flament 7905d41487 Merge pull request #2349 from dalf/fix-settings-loader 
[fix] settings_loader: don't crash when a key exists only in the user settings
 2020-12-03 11:47:53 +01:00
Alexandre Flament d0d7a3e1c2 [fix] settings_loader: don't crash when a key exists only in the user settings 
typical use case: result_proxy can be defined in the user settings,
but are not defined the default settings.yml
 2020-12-03 11:35:12 +01:00
Alexandre Flament 89fbb85d45 Merge pull request #2332 from dalf/metrology-errors 
[enh] record exception details per engine
 2020-12-03 10:31:44 +01:00
Alexandre Flament 64cccae99e [mod] various engines: use eval_xpath* functions and searx.exceptions.* 
Engine list: ahmia, duckduckgo_images, elasticsearch, google, google_images, google_videos, youtube_api
 2020-12-03 10:22:48 +01:00
Alexandre Flament ad72803ed9 [mod] xpath, 1337x, acgsou, apkmirror, archlinux, arxiv: use eval_xpath_* functions 2020-12-03 10:22:48 +01:00
Alexandre Flament de887c6347 [mod] bing_news: use eval_xpath_getindex 
remove unused function searx.utils.list_get
 2020-12-03 10:22:48 +01:00
Alexandre Flament 1d0c368746 [enh] record details exception per engine 
add an new API /stats/errors
 2020-12-03 10:22:48 +01:00
Alexandre Flament 6b5a578822 Merge pull request #2285 from return42/fix-digg 
bugfix & refactor digg engine
 2020-12-03 10:20:40 +01:00
Alexandre Flament 219af243a2 Merge pull request #2318 from dalf/add-pylint 
Add pylint
 2020-12-01 15:31:10 +01:00
Alexandre Flament 8aa2a7556a [enh] activate pylint 
There are 3 invocations:
* one with the default .pylintrc file
* one for searx/engines with some disabled checks and some additional-builtins
* one for the all the code except the engines with some disabled checks
 2020-12-01 15:21:19 +01:00
Alexandre Flament b00d108673 [mod] pylint: numerous minor code fixes 2020-12-01 15:21:19 +01:00
Alexandre Flament 9ed3ee2beb [mod] wikidata: WDGeoAttribute class: doesn't change the method signature of get_str 2020-12-01 15:21:17 +01:00
Alexandre Flament 3cfef61123 [fix] /stats: report error percentage instead of error count 
This bug exists since the PR https://github.com/searx/searx/pull/751
 2020-12-01 15:07:09 +01:00
Alexandre Flament b7f1a8424a [fix] tests: fix duplicate method name 2020-12-01 15:07:09 +01:00
Alexandre Flament a302ac5629 [mod] remove dead code 2020-12-01 15:07:09 +01:00
Alexandre Flament a1e6bc4cee Merge pull request #2291 from dalf/settings2 
[enh] user settings can relied on the default settings
 2020-12-01 14:57:12 +01:00
Alexandre Flament 0e02fee930 Merge pull request #2345 from dalf/mod-oscar 
[mod] oscar theme: added option into gruntfile.js for generate sourceMap
 2020-12-01 14:11:13 +01:00
Alexandre Flament f1e016e9ea [mod] oscar theme: added option into gruntfile.js for generate sourceMap 
Credits go to @mrwormo  (see PR #2308 )
 2020-12-01 10:07:01 +01:00
Alexandre Flament 93c2603561 Merge pull request #2336 from efectn/patch-1 
Photon API Link Update
 2020-11-28 09:06:53 +01:00
Alexandre Flament b4b81a5e1a [enh] settings.yml: add use_default_settings option (2nd version) 2020-11-27 19:40:04 +01:00
Alexandre Flament 1cfe7f2a75 [enh] settings.yml: add use_default_settings option 
This change is backward compatible with the existing configurations.

If a settings.yml loaded from an user defined location (SEARX_SETTINGS_PATH or /etc/searx/settings.yml),
then this settings can relied on the default settings.yml with this option:
user_default_settings:True
 2020-11-26 18:27:27 +01:00
Alexandre Flament 6ada5bac60 Merge pull request #2327 from renyhp/master 
Add preference for displaying advanced settings
 2020-11-26 17:37:43 +01:00
Alexandre Flament aec6a2656c Merge pull request #2319 from dalf/multiple-proxies 
[enh] Add multiple outgoing proxies
 2020-11-23 08:43:53 +01:00
Alexandre Flament 3786920df9 [enh] Add multiple outgoing proxies 
credits go to @bauruine see https://github.com/searx/searx/pull/1958
 2020-11-20 15:29:21 +01:00
Alexandre Flament 2fc3b17c85 Merge pull request #2323 from kvch/fix-unresponsive-engine-set-modified 
Fix RuntimeError during search with translated errors
 2020-11-18 09:37:29 +01:00
Alexandre Flament 963b93b543 [fix] fix github workflow: deploy documentation 2020-11-17 15:54:14 +01:00
Alexandre Flament c9fb3ac34f [fix] fix github workflow 2020-11-17 15:44:50 +01:00
Alexandre Flament 197a013d2e [fix] fix github workflow 
bug fix of 9c25cd99c4
 2020-11-17 15:38:31 +01:00
Alexandre Flament 2fc7273141 Merge pull request #2322 from dalf/github-actions 
[mod] use github actions instead of travis
 2020-11-17 15:20:45 +01:00
Alexandre Flament 9c25cd99c4 [mod] use github actions instead of travis 
fix https://github.com/searx/searx/issues/2279
 2020-11-17 15:09:06 +01:00
Alexandre Flament a30d9f2655 Merge pull request #2321 from dalf/fix-standalone 
[fix] fix a test_standalone_searx test case
 2020-11-17 13:19:55 +01:00
Alexandre Flament 230a5ecd04 [fix] fix a test_standalone_searx test case 
If test_engines_init.py runs before test_standalone_searx.py, the engine list is not empty.
It makes test_get_search_query flaky.

This commit initializes the engline list in test_standalone_searx.py
 2020-11-17 10:39:44 +01:00
Alexandre Flament 17b48ff6e8 Merge pull request #2286 from return42/fix-deviantart 
[refactor] deviantart - improve results and clean up source code
 2020-11-14 17:31:39 +01:00
Alexandre Flament 4f18faebe1 Merge pull request #2288 from dalf/lint 
[mod] remove unused import
 2020-11-14 14:24:45 +01:00
Alexandre Flament 3038052c79 [mod] remove unused import 
use
from searx.engines.duckduckgo import _fetch_supported_languages, supported_languages_url  # NOQA
so it is possible to easily remove all unused import using autoflake:
autoflake --in-place --recursive --remove-all-unused-imports searx tests
 2020-11-14 14:11:02 +01:00
Alexandre Flament 6489a560ea Merge pull request #2299 from rachmadaniHaryono/feature/pep8-to-pycodestyle 
pep8 to pycodestyle
 2020-11-14 13:49:25 +01:00
Alexandre Flament e1bd617669 Merge pull request #2296 from rachmadaniHaryono/feature/suppress-output 
suppress test output
 2020-11-14 13:45:32 +01:00
Alexandre Flament c3d9b17c2a Merge pull request #2292 from kvch/elasticsearch-engine 
New engine: Elasticsearch
 2020-11-14 13:25:08 +01:00
Alexandre Flament 102c08838b Merge pull request #2289 from dalf/pylint 
[mod] pylint: add extension-pkg-whitelist=lxml.etree
 2020-11-14 13:24:31 +01:00
Alexandre Flament 46b454277f Merge pull request #2309 from dalf/mod_search_repr 
[mod] searx.search: EngineRef, SearchQuery: add __repr__ and __eq__ methods
 2020-11-14 13:23:44 +01:00
Alexandre Flament ebed1461bc Merge pull request #2300 from dalf/fix-webapp-index 
[fix] fix of / and /search
 2020-11-14 13:23:03 +01:00
Alexandre Flament 20c95712ea Merge pull request #2301 from pierrechtux/fix_typo_virualenv 
Fix typo virualenv
 2020-11-13 17:55:54 +01:00
Alexandre Flament 8fc74d0d7b [mod] searx.search: EngineRef, SearchQuery: add __repr__ and __eq__ methods 2020-11-10 10:45:40 +01:00
Alexandre Flament b3a3ccf2db [fix] fix of / and /search 
* URL / : the index page displayed the selected or the default category.
* URL / : when the q parameter is set using the URL, the redirect includes the URL query.
* URL /search : an empty query doesn't raise an exception.
 2020-11-06 12:11:52 +01:00
Alexandre Flament 58d72f2692 [mod] pylint: minor code change to allow pylint globally 
This commit is only a step, it doesn't fix all the issues reported by pylint
 2020-11-03 11:35:53 +01:00
Alexandre Flament e28b56e262 [fix] webadapter: fix locked categories 2020-11-03 10:55:49 +01:00
Alexandre Flament eed43783f9 [fix] comamnd engine: fix import 2020-11-03 10:55:08 +01:00
Alexandre Flament a08df82574 [fix] scanr_structure engine: fix import 2020-11-03 10:54:02 +01:00
Alexandre Flament d7f21484ee [mod] pylint: add extension-pkg-whitelist=lxml.etree 2020-11-02 15:55:19 +01:00
Alexandre Flament 54afe37e18 [mod] .travis.yml: build documentation automatically on the master branch 2020-10-28 22:48:16 +01:00
Alexandre Flament 95bd6033fa [mod] wikidata engine: use one SPARQL request instead of 2 HTTP requests. 2020-10-28 08:09:25 +01:00
Alexandre Flament ca593728af [mod] duckduckgo_definitions: display only user friendly attributes / URL 
various bug fixes
 2020-10-28 08:09:25 +01:00
Alexandre Flament 382fded665 [mod] result.py: merge infobox URL and attributes when the same label or the same entity 
entity are wikidata entity (like "Q42" for "Douglas Adams", see https://www.wikidata.org/wiki/Q42 )
 2020-10-28 08:09:25 +01:00
Alexandre Flament 23f4203dfb [fix] simple theme: infobox: remove useless entity information 2020-10-28 08:09:25 +01:00
Alexandre Flament ed6696e6bf [mod] add external_urls.json and wikidata_units.json 2020-10-28 08:09:25 +01:00
Alexandre Flament 5e7060053c [mod] ahmia_filter.py: minor changes 
- use result['parsed_url']
- load ahmia_blacklist.txt in searx.datae
 2020-10-27 20:00:04 +01:00
Alexandre Flament cfd21bc475 [fix] fix duckduckgo engine 
- remove paging support: a "vqd" parameter is required between each request. This parameter is uniq for each request
- update the URL (no redirect), use the POST method
- language support: works if there is no more than request per minute, otherwise it is ignored !
 2020-10-09 16:00:42 +02:00
Alexandre Flament 8b278cbfad Merge pull request #2246 from dalf/mod-searx-data 
[mod] Add searx.data module
 2020-10-07 10:38:13 +02:00
Alexandre Flament a9dc54bebc [mod] Add searx.data module 
Instead of loading the data/*.json in different location,
load these files in the new searx.data module.
 2020-10-07 10:29:34 +02:00
Alexandre Flament e30dc2f0ba Merge pull request #2247 from dalf/fix-opensearch 
[fix] opensearch.xml URL contains method and autocomplete parameters
 2020-10-07 10:14:57 +02:00
Alexandre Flament 474d56c77f Merge pull request #2248 from dalf/fix-webadapter 
[fix] various fixes of searx.webadapter
 2020-10-07 10:12:10 +02:00
Alexandre Flament 66bda23895 Merge pull request #2244 from dalf/fix-drop-python2 
Fix drop python2
 2020-10-07 10:11:53 +02:00
Alexandre Flament d5950079cf [fix] fix searx.webadapter 
* Fix "?q=test&engines=wikipedia": fix exception
* Fix "?q=test&engines=wikipedia&categories=images": now the engines from images category are included.
* Fix parse_timeout: make sure a value is always returned
* Various typing fixes (searx.webadapter, searx.search.SearchQuery)
 2020-10-06 15:23:19 +02:00
Alexandre Flament 8659212f5a [fix] drop Python 2: use collections.abc.Iterable instead of collections.Iterable 2020-10-06 09:43:24 +02:00
Alexandre Flament 15013e64d8 [fix] drop Python 2: use importlib instead of imp.load_source 
imp.load_source is not documented in Python 3
see documentation : https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly

partial fix of https://github.com/searx/searx/issues/1674
 2020-10-06 09:42:11 +02:00
Alexandre Flament bfdad7bc0f [fix] opensearch.xml URL contains method and autocomplete parameters 
When the user add searx as a search engine, the browser loads the /opensearch.xml URL without the cookies.
Without the query parameters, the user preferences are ignored (method and autocomplete).

In addition, opensearch.xml is modified to support automatic updates,
see https://developer.mozilla.org/en-US/docs/Web/OpenSearch
 2020-10-06 00:54:37 +02:00
Alexandre Flament 584760cf54 Merge pull request #2237 from dalf/mod-engines-init 
Mod engines init
 2020-10-05 11:20:46 +02:00
Alexandre Flament 6c39917c4d [mod] webapp.py: update engines initialization condition 
Always call initialize engines except on the first run of werkzeug with the reload feature.

the reload feature is activated when:
* searx_debug is True (SEARX_DEBUG environment variable or settings.yml)
* FLASK_APP=searx/webapp.py FLASK_ENV=development flask run (see https://flask.palletsprojects.com/en/1.1.x/cli/ )

Fix SEARX_DEBUG=0 make docs
docs/admin/engines.rst : engines are initialized
See https://github.com/searx/searx/issues/2204#issuecomment-701373438
 2020-10-05 11:13:32 +02:00
Alexandre Flament b728cb610b Merge pull request #2241 from dalf/move-extract-text-and-url 
Move the extract_text  and extract_url functions to searx.utils
 2020-10-04 09:06:20 +02:00
Alexandre Flament e2cd9b65bb Merge pull request #2239 from dalf/mod-preferences 
[mod] preferences.py: check language setting with a regex instead of match_language
 2020-10-04 09:05:14 +02:00
Alexandre Flament 8f914a28fa [mod] searx.utils.normalize_url: remove Yahoo hack 
* The hack for Yahoo URLs is not necessary anymore. (see searx.engines.yahoo.parse_url)
* move the URL normalization in extract_url to normalize_url
 2020-10-03 10:02:50 +02:00
Alexandre Flament c1d10bde02 [mod] searx/utils.py: add docstring 2020-10-02 18:17:01 +02:00
Alexandre Flament 2006eb4680 [mod] move extract_text, extract_url to searx.utils 2020-10-02 18:13:56 +02:00
Alexandre Flament 2cafc5462d [fix] revert PR #2232 and #2230 2020-10-01 16:38:54 +02:00
Alexandre Flament 507896c115 [mod] preferences.py: check language setting with a regex instead of match_language 2020-10-01 11:29:31 +02:00
Alexandre Flament ecb9f28869 Merge pull request #2236 from return42/fix-2234 
[fix] google engine - div classes has been renamed in HTML result
 2020-10-01 10:11:19 +02:00
Alexandre Flament fd5fe36984 Merge pull request #2232 from dalf/fix-sphinx-build2 
[fix] Sphinx 3.2.1 build: call searx.engines.initialize_engines
 2020-09-30 10:29:28 +02:00
Alexandre Flament 9a856083df [fix] Sphinx 3.2.1 build: call searx.engines.initialize_engines 
Fix admin/engines.html
Fix #2230
 2020-09-30 10:20:12 +02:00
Alexandre Flament 6c6985aaf8 Merge pull request #2230 from dalf/fix-sphinx-build 
[fix] Sphinx 3.2.1 build: add missing "engines" variable in the jinja context
 2020-09-30 09:29:09 +02:00
Alexandre Flament 9740618227 Merge pull request #2226 from dalf/fix-searx-migration 
[fix] migration from github.com/asciimoo/searx to github.com/searx/searx : fix URLs
 2020-09-29 12:35:11 +02:00
Alexandre Flament 6d56c754a7 [fix] Sphinx 3.2.1 build: add missing "engines" variable in the jinja context 
Fix the admin/engines.html page
 2020-09-29 12:30:10 +02:00
Alexandre Flament f204e4903d [fix] migration from github.com/asciimoo/searx to github.com/searx/searx : fix URLs 2020-09-28 16:44:14 +02:00
Alexandre Flament c2a6f14516 [fix] Update Sphinx package versions 
* Update call to sphinx-autobuild (remove --poll -B options, add --open-browser)
* Upgrade the Sphinx and sphinx-tabs versions.
* utils/makefile.sphinx: remove the SPHINXVERS variable.
* utils/makefile.sphinx: remove the sphinx-doc and sphinx-live targets, Sphinx is already installed by the pyenvinstall target.
 2020-09-28 13:27:50 +02:00
Alexandre Flament 93f7f7eee2 [mod] upgrade requests to version 2.24.0. use ssl instead of pyopenssl. 
requests 2.24.0 uses the ssl module except if it doesn't support SNI, in this case searx fallbacks to pyopenssl.
searx logs a critical message and exit if the ssl modules doesn't support SNI and pyOpenSSL is not installed.
searx logs a critical message and exit if the ssl version is older than 1.0.2.
in requirements.txt, pyopenssl is still required to install searx as a fallback.
 2020-09-26 19:30:27 +02:00
Alexandre Flament f2f3300bde [mod] more typing 2020-09-24 16:26:00 +02:00
Alexandre Flament 485a502b88 [mod] add typing and __slots__ 2020-09-22 19:05:05 +02:00
Alexandre Flament 678699beaf [mod] searx/webadapter.py: add get_selected_categories share common code with get_search_query_from_webapp 
Update searx/webapp.py to use get_selected_categories
Close #2142
 2020-09-22 18:59:51 +02:00
Alexandre Flament d5b5e48f04 [mod] searx/webadapter.py: each web parameter has a dedicated function (parse_pageno, page_lang,....). 2020-09-22 18:59:51 +02:00
Alexandre Flament 691d12726b [mod] check the engine tokens in searx/webadapter.py instead of searx/search.py 2020-09-22 18:59:51 +02:00
Alexandre Flament eecfff2689 [mod] check time_range parameter in searx/webadapter.py 2020-09-22 18:59:51 +02:00
Alexandre Flament 2929495112 [mod] add searx.search.EngineRef 
was previously a Dict with two or three keys: name, category, from_bang
make clear that this is a engine reference (see tests/unit/test_search.py for example)
all variables using this class are renamed accordingly.
 2020-09-22 18:59:51 +02:00
Alexandre Flament 2dbc0de0cd [mod] add searx/webadapter.py 
* move searx.search.get_search_query_from_webapp to searx.webadapter
* move searx.query.SearchQuery to searx.search
 2020-09-22 18:59:51 +02:00
Alexandre Flament e8df5d3f7f [mod] searx.search: change function declaration: online then offline then utility functions. 2020-09-22 12:36:26 +02:00
Alexandre Flament 50e717c452 [mod] searx.search: remove unused global variable number_of_searches 2020-09-22 12:36:26 +02:00
Alexandre Flament edd8dccd07 [mod] searx.query.RawTextQuery: getSearchQuery and changeSearchQuery rename to getQuery and changeQuery 
getSearchQuery is confusing, the method returns a str not a SearchQuery object
 2020-09-22 12:36:26 +02:00
Alexandre Flament c4067a9d02 [mod] searx.search.Search: split the search method into different methods 2020-09-22 12:14:46 +02:00