retoor/searxng
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
4,468 Commits 6 Branches 17 Tags
92c8a8829f2e68e7ceb3b4670ebea4c4e6541a7c
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Alexandre Flament 92c8a8829f [fix] strip spaces from searx user agent 
h11 (used by httpx) rejects HTTP request with a trailing space in HTTP headers
 2021-06-09 18:08:23 +02:00
Alexandre Flament 5f76238d5c Merge pull request #105 from return42/yaml 
[enh] add test.yamllint - lint yaml files
 2021-06-05 20:36:03 +02:00
Alexandre Flament cf2b549967 Update engines_languages.json 2021-06-04 16:17:22 +02:00
Alexandre Flament 8194db4e21 [fix] peertube fetch supported languages 
close #127
 2021-06-04 16:17:20 +02:00
Alexandre Flament dddf3e2be2 [mod] "make docs" doesn't initialize the engines 
avoid external HTTP requests with "make docs"
 2021-06-04 11:26:37 +02:00
Alexandre Flament cac6562a9e Merge pull request #129 from searxng/dependabot/pip/master/certifi-2021.5.30 
Bump certifi from 2020.12.05 to 2021.5.30
 2021-06-04 10:23:56 +02:00
Alexandre Flament a7d575c099 [enh] flask debug mode: reload the app when searx/settings.yml changes 2021-06-03 12:03:38 +02:00
Alexandre Flament 7457f3fe40 Merge pull request #124 from return42/searx-merge 
merge redis offline engine from searx
 2021-06-02 12:35:33 +02:00
Alexandre Flament 3ce1b40512 Merge pull request #113 from searxng/remove-overpass 
[mod] remove overpass API call
 2021-06-01 18:10:05 +02:00
Alexandre Flament a89b823f8d [mod] remove overpass API call 
prepare the code the PR #90
 2021-06-01 17:52:49 +02:00
Alexandre Flament ee83c99d2b Merge pull request #116 from searxng/minor-refactoring-searx-engines 
[mod] searx.engines.__init__: refactoring
 2021-06-01 16:40:38 +02:00
Alexandre Flament 8375974dff [fix] sys.exit(1) when there is duplicate engine name 2021-06-01 16:37:20 +02:00
Alexandre Flament 70a9208972 [mod] searx.engines.__init__: refactoring 2021-06-01 16:32:40 +02:00
Alexandre Flament e3f4a77311 Merge pull request #110 from searxng/mod-default-settings 
[mod] move all default settings into searx.settings_defaults
 2021-06-01 16:13:04 +02:00
Alexandre Flament e0af67f0af Merge pull request #123 from searxng/fix_update_languages 
[fix] searx_extra/update/update_languages.py
 2021-06-01 10:07:27 +02:00
Alexandre Flament 925c21ce42 Merge pull request #122 from searxng/update_data_update_ahmia_blacklist.py 
Update searx.data - update_ahmia_blacklist.py
 2021-06-01 10:03:50 +02:00
Alexandre Flament 3266738e7a Merge pull request #121 from searxng/update_data_update_currencies.py 
Update searx.data - update_currencies.py
 2021-06-01 10:03:35 +02:00
Alexandre Flament 54ae95b95b Merge pull request #120 from searxng/update_data_update_wikidata_units.py 
Update searx.data - update_wikidata_units.py
 2021-06-01 10:03:22 +02:00
Alexandre Flament 414ad5b733 Merge pull request #119 from searxng/update_data_update_firefox_version.py 
Update searx.data - update_firefox_version.py
 2021-06-01 10:03:09 +02:00
Alexandre Flament fa0ae252b0 [fix] searx_extra/update/update_languages.py 
* call searx.engines.load_engines (instead of initialize_engines)
* set timeout to 10 seconds
* remove some dead code
 2021-06-01 10:01:39 +02:00
Alexandre Flament 6fa114c9ba [mod] settings_default: remove searx.search.max_request_timeout global variable 2021-06-01 08:10:15 +02:00
Alexandre Flament 4b07df62e5 [mod] move all default settings into searx.settings_defaults 2021-06-01 08:10:15 +02:00
Alexandre Flament 856729226d Merge pull request #118 from searxng/fix-mankier 
[fix] mankier: fix paging
 2021-05-31 22:49:11 +02:00
Alexandre Flament 6ca74386d6 [fix] mankier: fix paging 
See https://github.com/searxng/searxng/pull/111#issuecomment-851641967
 2021-05-31 22:33:40 +02:00
Alexandre Flament 6793af7608 Merge pull request #117 from return42/add-docker-hub 
[enh] add engine - Docker Hub
 2021-05-31 17:47:08 +02:00
Alexandre Flament 737f5f05d2 Merge pull request #114 from searxng/only-files-category-for-bittorent 
[mod] the bittorent search engines are available only in the files category
 2021-05-30 07:01:49 +02:00
Alexandre Flament 1113f7e616 [mod] the bittorent search engines are available only in the files category 
related to #101
 2021-05-29 16:14:19 +02:00
Alexandre Flament e239369f1f Merge pull request #109 from return42/sql-engines 
Merge MySQL and PostgreSQL from searx
 2021-05-29 08:34:19 +02:00
Alexandre Flament c22c7cf35b Merge pull request #111 from return42/mankier 
[enh] add engine ManKier (JSON) / ManKier: Linux man pages
 2021-05-29 08:33:54 +02:00
Alexandre Flament a8b0705816 Merge pull request #112 from return42/pypi 
[enh] add engine PyPi (XPath)
 2021-05-29 08:33:41 +02:00
Alexandre Flament da111f7fec Merge pull request #108 from return42/dogpile 
[enh] add metasearch engin dogpile.com (XPath)
 2021-05-28 15:56:01 +02:00
Alexandre Flament 54e783f3b1 Merge pull request #107 from return42/wikimini 
[enh] add engine Wikimini (fr.wikimini.org)
 2021-05-28 14:47:38 +02:00
Alexandre Flament 83ccf7b04d Merge pull request #99 from return42/webapp-misc 
[enh] add settings option to enable/disable search formats
 2021-05-28 13:26:48 +02:00
Alexandre Flament b48b4c93d5 Merge pull request #100 from return42/webapp-pylint 
[pylint] webapp.py
 2021-05-27 17:04:32 +02:00
Alexandre Flament b1da97d092 Merge pull request #96 from return42/unsplash 
lint & fix unsplash engine
 2021-05-27 14:58:59 +02:00
Alexandre Flament 2ea34a3c36 [enh] add offline engine for sqlite database 
To test & demonstrate this implementation download:

  https://liste.mediathekview.de/filmliste-v2.db.bz2

and unpack into searx/data/filmliste-v2.db, in your settings.yml define a sqlite
engine named "demo"::

    - name : demo
      engine : sqlite
      shortcut: demo
      categories: general
      result_template: default.html
      database : searx/data/filmliste-v2.db
      query_str :  >-
        SELECT title || ' (' || time(duration, 'unixepoch') || ')' AS title,
               COALESCE( NULLIF(url_video_hd,''), NULLIF(url_video_sd,''), url_video) AS url,
               description AS content
          FROM film
         WHERE title LIKE :wildcard OR description LIKE :wildcard
         ORDER BY duration DESC
      disabled : False

Query to test: "!demo concert"

This is a rewrite of the implementation from commit [1]

[1] searx/searx@8e90a21

Suggested-by: @virtadpt searx/searx#2808
 2021-05-27 14:27:11 +02:00
Alexandre Flament 7c76cef746 Merge pull request #94 from return42/pylint-network 
Pylint searx.network
 2021-05-26 18:35:29 +02:00
Alexandre Flament 8283ce3d79 Merge pull request #92 from return42/xpath-misc 
Xpath enhencements
 2021-05-25 09:06:06 +02:00
Alexandre Flament 99aaf86b73 [fix] offline engines: fix templates /stats and /preferences 2021-05-22 15:17:18 +02:00
Alexandre Flament 3014463fed [fix] metrics: processing time = total time if there is no http time 
It was previsouly None

Fix /stats
 2021-05-22 15:17:18 +02:00
Alexandre Flament ec83493538 [fix] offline engine: don't crash on time recording 2021-05-22 15:17:18 +02:00
Alexandre Flament 2f76b570ab Merge pull request #86 from searxng/remove-call-to-gc 
[mod] remove gc.collect() after each user request
 2021-05-21 18:31:04 +02:00
Alexandre Flament 426fadccb3 [mod] remove gc.collect() after each user request 2021-05-21 17:23:18 +02:00
Alexandre Flament 0c1ab0227d Merge pull request #84 from searxng/fix-test-ignore-etc-settings 
[fix] unit test: don't load /etc/searx/settings.yml
 2021-05-20 13:31:53 +02:00
Alexandre Flament 94e6ed6110 Merge pull request #66 from return42/docker-buildx 
review of docker entry point & docker documentation
 2021-05-19 17:31:33 +02:00
Alexandre Flament d1cffc55cb [fix] unit test: don't load /etc/searx/settings.yml 
Add a new environment variable SEARX_DISABLE_ETC_SETTINGS
to disable loading of /etc/searx/settings.yml

unit tests:
* set SEARX_DISABLE_ETC_SETTINGS to 1
* remove SEARX_SETTINGS_PATH if it exists
 2021-05-18 17:23:21 +02:00
Alexandre Flament d255e5637b Merge pull request #81 from return42/csp-bar-graph 
[fix] make /stats more CSP compliant
 2021-05-18 08:55:56 +02:00
Alexandre Flament da2b0562d4 Merge pull request #83 from return42/fix-archive-is 
[fix] engine "archive is" - search_url has been changed
 2021-05-18 08:44:34 +02:00
Alexandre Flament 90fe232e08 [fix] webapp.py: pylint: disable=assigning-non-slot for request.* 
Note: actually, the code should use flask.g.* instead of request.*
 2021-05-14 11:15:35 +02:00
Alexandre Flament 9a383d1533 Merge pull request #67 from searxng/fix-issue-65 
[fix] fix usage of docker environment variable INSTANCE_NAME
 2021-05-10 08:33:32 +02:00
Alexandre Flament a40daec565 Merge pull request #68 from searxng/mod-docker-upgrade-alpine-version 
[mod] Dockerfile: upgrade to Alpine 3.13
 2021-05-10 08:32:31 +02:00
Alexandre Flament 8c5cb5039e [mod] Dockerfile: upgrade to Alpine 3.13 
See:
* https://alpinelinux.org/posts/Alpine-3.13.0-released.html
* musl>=1.2.1 have a new malloc implementation (mallocng) :
 2021-05-09 11:06:07 +02:00
Alexandre Flament b46a3ecfc7 [fix] fix usage of docker environment variable INSTANCE_NAME 
INSTANCE_NAME was ignored.

close #65
 2021-05-09 11:00:28 +02:00
Alexandre Flament 332e3a2a09 Merge pull request #64 from searxng/fix-test-sideeffect 
[fix] fix KeyError: 'ipv6'
 2021-05-07 11:05:09 +02:00
Alexandre Flament 605124679d [fix] fix KeyError: 'ipv6' 
tests/units/network/test_network.py requires a call to searx.network.network.initialize
Depending of the test order execution, this function was sometimes call in another test,
sometimes not.

This commit ensure there is a call to initialize()
 2021-05-07 10:57:55 +02:00
Alexandre Flament f92a8e432a Merge pull request #60 from searxng/mod-http-header-server-timing 
[mod] Server-Timing header: add render field
 2021-05-07 10:48:06 +02:00
Alexandre Flament d12033e166 [mod] Server-Timing header: add render field 
Add a "render" field to the Server-Timing header
This field is the time spent in the searx.webapp.render function (Jinja2 rendering)

See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Server-Timing
 2021-05-06 09:39:52 +02:00
Alexandre Flament 39eda78088 Merge pull request #59 from searxng/mod-sslcontext 
[mod] searx.network.client: the same configuration reuses the same ssl.SSLContext
 2021-05-05 20:42:04 +02:00
Alexandre Flament 0f4e995ab4 [mod] searx.network.client: the same configuration reuses the same ssl.SSLContext 
before there was one ssl.SSLContext per client.

see https://github.com/encode/httpx/issues/978
 2021-05-05 20:36:37 +02:00
Alexandre Flament 5b13786abb Merge pull request #58 from searxng/mod-multithreading 
[mod] multithreading only in searx.search.* packages
 2021-05-05 18:33:16 +02:00
Alexandre Flament 8c1a65d32f [mod] multithreading only in searx.search.* packages 
it prepares the new architecture change,
everything about multithreading in moved in the searx.search.* packages

previously the call to the "init" function of the engines was done in searx.engines:
* the network was not set (request not sent using the defined proxy)
* it requires to monkey patch the code to avoid HTTP requests during the tests
 2021-05-05 13:12:42 +02:00
Alexandre Flament e83862c22f [themes] /preferences, reliablity: warning icons & error logs link 
When there is at least one errors or one failed checker test:
* the warning icon is displayed in the reliability column
* the link "View error logs and submit a bug report" is displayed on engine name tooltip.

Before:
* the warning icon was displayed only when one or more checker test(s) failed.
* the link "View error logs and submit a bug report" was not shown when a checker test failed but there were no error.
 2021-05-04 10:50:03 +02:00
Alexandre Flament 2756e170ce Merge pull request #49 from searxng/update_data_update_ahmia_blacklist.py 
Update searx.data - update_ahmia_blacklist.py
 2021-05-01 10:28:32 +02:00
Alexandre Flament a3a7be90ff Merge pull request #50 from searxng/update_data_update_currencies.py 
Update searx.data - update_currencies.py
 2021-05-01 10:28:22 +02:00
Alexandre Flament 4558c0c027 Merge pull request #48 from searxng/update_data_update_wikidata_units.py 
Update searx.data - update_wikidata_units.py
 2021-05-01 10:28:11 +02:00
Alexandre Flament d16fdf662f Merge pull request #47 from searxng/update_data_update_firefox_version.py 
Update searx.data - update_firefox_version.py
 2021-05-01 10:26:51 +02:00
Alexandre Flament 96ae39c3b6 Update data-update.yml 
Call to peter-evans/create-pull-request@v3 : 
Use the default GITHUB_TOKEN
 2021-05-01 10:21:27 +02:00
Alexandre Flament 898d789e82 Update data-update.yml 
Disable fail fast: so even if a script fails, the other scripts create PR.
 2021-05-01 10:15:17 +02:00
Alexandre Flament 26b0de6843 Update data-update.yml 
- searxng brand
- fix call to ./manage
 2021-05-01 10:10:24 +02:00
Alexandre Flament 0ff14fd3ec Merge pull request #22 from searxng/docker-buildx 
[docker] multiarch build
 2021-04-30 16:20:33 +02:00
Alexandre Flament e5bcc3cbbe Merge pull request #40 from searxng/link-to-stats 
[mod] engine errors: link to the stats to create an github issue
 2021-04-29 14:37:12 +02:00
Alexandre Flament 34bced29ae [mod] engine errors: link to the stats to create an github issue 2021-04-29 11:10:38 +02:00
Alexandre Flament 3282cdca13 Merge pull request #38 from return42/filtron-rules 
[filtron.sh] make filtron rule file configurable
 2021-04-29 10:31:59 +02:00
Alexandre Flament 1e3148fea9 Merge pull request #41 from return42/fix-themes 
fix varius theme issues
 2021-04-29 10:28:53 +02:00
Alexandre Flament 20580bcbd4 [docker] multiarch support: linux/amd64,linux/arm64,linux/arm/v7 
make docker.buildx : build and push multiarch build.
(it can't be only build)

use buildx with the --cache-from and --cache-to options to cache the layers
(only the last built is cached)
 2021-04-28 14:40:00 +02:00
Alexandre Flament fe3831db1a Merge pull request #37 from return42/drop-req-idna 
[fix] drop 'idna' from requirements.txt
 2021-04-27 20:08:52 +02:00
Alexandre Flament b06a15376c Merge pull request #32 from searxng/fix-network 
[fix] searx.network: fix rare cases where LOOP is None
 2021-04-27 20:06:54 +02:00
Alexandre Flament 283ae7bfad [fix] searx.network: fix rare cases where LOOP is None 
* searx.network.client.LOOP is initialized in a thread
* searx.network.__init__ imports LOOP which may happen
  before the thread has initialized LOOP

This commit adds a new function "searx.network.client.get_loop()"
to fix this issue
 2021-04-27 17:47:36 +02:00
Alexandre Flament 5781d75ca2 [clean] drop deprecated files .codecov.yml, .landscape.yaml & tox.ini 
.codecov.yml :
  code coverage reports are attached on each build. See for example:
  https://github.com/searxng/searxng/actions/runs/788360104 (Artifacts section)

.landscape.yaml :
  landscape.io is offline

tox.ini :
  not in use
 2021-04-27 16:52:53 +02:00
Alexandre Flament 87e914e398 Merge pull request #28 from searxng/mod-processors-error-message 
[mod] processors: show identical error messages on /search and /stats
 2021-04-27 16:50:41 +02:00
Alexandre Flament f45729ae72 Merge pull request #35 from return42/fix-manage 
[fix] manage - fix miss usage of 'set -e'
 2021-04-27 14:39:43 +02:00
Alexandre Flament b1557b5443 [mod] processors: show identical error messages on /search and /stats 2021-04-27 14:20:07 +02:00
Alexandre Flament b8f5adf500 [mod] searx.metrics.error_recorder: store relative file name instead of the full absolute file name. 2021-04-27 10:42:00 +02:00
Alexandre Flament 9b482e8fcc Merge pull request #31 from return42/mod-pylint 
[pylint] get PYLINT_FILES from tag '# lint: pylint'
 2021-04-26 22:58:44 +02:00
Alexandre Flament f4ab1e1251 Merge pull request #30 from return42/brand-readme 
[brand] README.rst fix img.shields.io and searxng.github.io URLs
 2021-04-26 16:13:08 +02:00
Alexandre Flament 12bdf7146b Merge pull request #27 from return42/fix-debug-log 
[fix] debug log: UnicodeEncodeError: 'ascii' codec can't encode
 2021-04-26 13:50:29 +02:00
Alexandre Flament 51494849fc Merge pull request #23 from searxng/metrics-stats-engines 
Metrics stats engines
 2021-04-25 14:31:52 +02:00
Alexandre Flament d20f2822d0 Merge pull request #25 from return42/brand-001 
SearxNG brand
 2021-04-25 14:30:18 +02:00
Alexandre Flament df41b77121 [mod] /stats : detail per engine 
allow to submit a github issue including the technical details
(exceptions, errors, warning, checker result)
 2021-04-25 14:28:06 +02:00
Alexandre Flament eebac31ccd Merge pull request #26 from return42/brand-002 
[github] searxng does not use github's discussion
 2021-04-25 12:59:11 +02:00
Alexandre Flament 9937a908c0 Merge pull request #24 from return42/fix-doc 
[fix doc] manage.sh update_packages does not exists anymore
 2021-04-25 10:22:32 +02:00
Alexandre Flament 0603b043ce Merge pull request #16 from return42/add-core.ac 
Add a search engine for core.ac.uk
 2021-04-24 09:11:20 +02:00
Alexandre Flament 4863c1933e Rerevert c54bf42 2021-04-24 07:59:16 +02:00
Alexandre Flament 525bfe7830 Fix PEP8 2021-04-24 07:19:44 +02:00
Alexandre Flament a7b9eca98a Merge pull request #8 from return42/manage-script 
Replace Makefile boilerplate by shell scripts
 2021-04-24 07:14:35 +02:00
Alexandre Flament fe064a5c39 temporary partial revert of c54bf42cb9 2021-04-24 07:13:04 +02:00
Alexandre Flament 3cdd6a6a50 Merge pull request #12 from searxng/metrics-stats-page 
[mod] update /stats
 2021-04-24 07:02:52 +02:00
Alexandre Flament 7032f7521c [mod] /stats: simple theme implementation 2021-04-24 06:58:49 +02:00
Alexandre Flament 09e7ecdce2 [mod] /stats : add reliability column and sort by column links 2021-04-24 06:58:49 +02:00
Alexandre Flament 65c29081cc [mod] update /stats 2021-04-24 06:58:22 +02:00