<!DOCTYPE html>
< html lang = "en" data-content_root = "../../../" >
< head >
< meta charset = "utf-8" / >
< meta name = "viewport" content = "width=device-width, initial-scale=1.0" / >
< meta name = "viewport" content = "width=device-width, initial-scale=1" >
< title > searx.engines.wikipedia — SearXNG Documentation (2025.2.9+a1e2b2546)< / title >
< link rel = "stylesheet" type = "text/css" href = "../../../_static/pygments.css?v=6625fa76" / >
< link rel = "stylesheet" type = "text/css" href = "../../../_static/searxng.css?v=52e4ff28" / >
< script src = "../../../_static/documentation_options.js?v=09da0b18" > < / script >
< script src = "../../../_static/doctools.js?v=9a2dae69" > < / script >
< script src = "../../../_static/sphinx_highlight.js?v=dc90522c" > < / script >
< script data-project = "searxng" data-version = "2025.2.9+a1e2b2546" src = "../../../_static/describe_version.js?v=fa7f30d0" > < / script >
< link rel = "index" title = "Index" href = "../../../genindex.html" / >
< link rel = "search" title = "Search" href = "../../../search.html" / >
< / head > < body >
< div class = "related" role = "navigation" aria-label = "Related" >
< h3 > Navigation< / h3 >
< ul >
< li class = "right" style = "margin-right: 10px" >
< a href = "../../../genindex.html" title = "General Index"
accesskey = "I" > index< / a > < / li >
< li class = "right" >
< a href = "../../../py-modindex.html" title = "Python Module Index"
> modules< / a > |< / li >
< li class = "nav-item nav-item-0" > < a href = "../../../index.html" > SearXNG Documentation (2025.2.9+a1e2b2546)< / a > » < / li >
< li class = "nav-item nav-item-1" > < a href = "../../index.html" > Module code< / a > » < / li >
< li class = "nav-item nav-item-2" > < a href = "../engines.html" accesskey = "U" > searx.engines< / a > » < / li >
< li class = "nav-item nav-item-this" > < a href = "" > searx.engines.wikipedia< / a > < / li >
< / ul >
< / div >
< div class = "document" >
< div class = "documentwrapper" >
< div class = "bodywrapper" >
< div class = "body" role = "main" >
< h1 > Source code for searx.engines.wikipedia< / h1 > < div class = "highlight" > < pre >
< span > < / span > < span class = "c1" > # SPDX-License-Identifier: AGPL-3.0-or-later< / span >
< span class = "sd" > " " " This module implements the Wikipedia engine. Some of this implementations< / span >
< span class = "sd" > are shared by other engines:< / span >
< span class = "sd" > - :ref:`wikidata engine`< / span >
< span class = "sd" > The list of supported languages is :py:obj:`fetched < fetch_wikimedia_traits> ` from< / span >
< span class = "sd" > the article linked by :py:obj:`list_of_wikipedias`.< / span >
< span class = "sd" > Unlike traditional search engines, wikipedia does not support one Wikipedia for< / span >
< span class = "sd" > all languages, but there is one Wikipedia for each supported language. Some of< / span >
< span class = "sd" > these Wikipedias have a LanguageConverter_ enabled< / span >
< span class = "sd" > (:py:obj:`rest_v1_summary_url`).< / span >
< span class = "sd" > A LanguageConverter_ (LC) is a system based on language variants that< / span >
< span class = "sd" > automatically converts the content of a page into a different variant. A variant< / span >
< span class = "sd" > is mostly the same language in a different script.< / span >
< span class = "sd" > - `Wikipedias in multiple writing systems`_< / span >
< span class = "sd" > - `Automatic conversion between traditional and simplified Chinese characters`_< / span >
< span class = "sd" > PR-2554_:< / span >
< span class = "sd" > The Wikipedia link returned by the API is still the same in all cases< / span >
< span class = "sd" > (`https://zh.wikipedia.org/wiki/出租車`_) but if your browser' s< / span >
< span class = "sd" > ``Accept-Language`` is set to any of ``zh``, ``zh-CN``, ``zh-TW``, ``zh-HK``< / span >
< span class = "sd" > or .. Wikipedia' s LC automatically returns the desired script in their< / span >
< span class = "sd" > web-page.< / span >
< span class = "sd" > - You can test the API here: https://reqbin.com/gesg2kvx< / span >
< span class = "sd" > .. _https://zh.wikipedia.org/wiki/出租車:< / span >
< span class = "sd" > https://zh.wikipedia.org/wiki/%E5%87%BA%E7%A7%9F%E8%BB%8A< / span >
< span class = "sd" > To support Wikipedia' s LanguageConverter_, a SearXNG request to Wikipedia uses< / span >
< span class = "sd" > :py:obj:`get_wiki_params` and :py:obj:`wiki_lc_locale_variants' in the< / span >
< span class = "sd" > :py:obj:`fetch_wikimedia_traits` function.< / span >
< span class = "sd" > To test in SearXNG, query for ``!wp 出租車`` with each of the available Chinese< / span >
< span class = "sd" > options:< / span >
< span class = "sd" > - ``!wp 出租車 :zh`` should show 出租車< / span >
< span class = "sd" > - ``!wp 出租車 :zh-CN`` should show 出租车< / span >
< span class = "sd" > - ``!wp 出租車 :zh-TW`` should show 計程車< / span >
< span class = "sd" > - ``!wp 出租車 :zh-HK`` should show 的士< / span >
< span class = "sd" > - ``!wp 出租車 :zh-SG`` should show 德士< / span >
< span class = "sd" > .. _LanguageConverter:< / span >
< span class = "sd" > https://www.mediawiki.org/wiki/Writing_systems#LanguageConverter< / span >
< span class = "sd" > .. _Wikipedias in multiple writing systems:< / span >
< span class = "sd" > https://meta.wikimedia.org/wiki/Wikipedias_in_multiple_writing_systems< / span >
< span class = "sd" > .. _Automatic conversion between traditional and simplified Chinese characters:< / span >
< span class = "sd" > https://en.wikipedia.org/wiki/Chinese_Wikipedia#Automatic_conversion_between_traditional_and_simplified_Chinese_characters< / span >
< span class = "sd" > .. _PR-2554: https://github.com/searx/searx/pull/2554< / span >
< span class = "sd" > " " " < / span >
< span class = "kn" > import< / span > < span class = "w" > < / span > < span class = "nn" > urllib.parse< / span >
< span class = "kn" > import< / span > < span class = "w" > < / span > < span class = "nn" > babel< / span >
< span class = "kn" > from< / span > < span class = "w" > < / span > < span class = "nn" > lxml< / span > < span class = "w" > < / span > < span class = "kn" > import< / span > < span class = "n" > html< / span >
< span class = "kn" > from< / span > < span class = "w" > < / span > < span class = "nn" > searx< / span > < span class = "w" > < / span > < span class = "kn" > import< / span > < span class = "n" > utils< / span >
< span class = "kn" > from< / span > < span class = "w" > < / span > < span class = "nn" > searx< / span > < span class = "w" > < / span > < span class = "kn" > import< / span > < span class = "n" > network< / span > < span class = "k" > as< / span > < span class = "n" > _network< / span >
< span class = "kn" > from< / span > < span class = "w" > < / span > < span class = "nn" > searx< / span > < span class = "w" > < / span > < span class = "kn" > import< / span > < span class = "n" > locales< / span >
< span class = "kn" > from< / span > < span class = "w" > < / span > < span class = "nn" > searx.enginelib.traits< / span > < span class = "w" > < / span > < span class = "kn" > import< / span > < span class = "n" > EngineTraits< / span >
< span class = "n" > traits< / span > < span class = "p" > :< / span > < span class = "n" > EngineTraits< / span >
< span class = "c1" > # about< / span >
< span class = "n" > about< / span > < span class = "o" > =< / span > < span class = "p" > {< / span >
< span class = "s2" > " website" < / span > < span class = "p" > :< / span > < span class = "s1" > ' https://www.wikipedia.org/' < / span > < span class = "p" > ,< / span >
< span class = "s2" > " wikidata_id" < / span > < span class = "p" > :< / span > < span class = "s1" > ' Q52' < / span > < span class = "p" > ,< / span >
< span class = "s2" > " official_api_documentation" < / span > < span class = "p" > :< / span > < span class = "s1" > ' https://en.wikipedia.org/api/' < / span > < span class = "p" > ,< / span >
< span class = "s2" > " use_official_api" < / span > < span class = "p" > :< / span > < span class = "kc" > True< / span > < span class = "p" > ,< / span >
< span class = "s2" > " require_api_key" < / span > < span class = "p" > :< / span > < span class = "kc" > False< / span > < span class = "p" > ,< / span >
< span class = "s2" > " results" < / span > < span class = "p" > :< / span > < span class = "s1" > ' JSON' < / span > < span class = "p" > ,< / span >
< span class = "p" > }< / span >
< span class = "n" > display_type< / span > < span class = "o" > =< / span > < span class = "p" > [< / span > < span class = "s2" > " infobox" < / span > < span class = "p" > ]< / span >
< span class = "sd" > " " " A list of display types composed from ``infobox`` and ``list``. The latter< / span >
< span class = "sd" > one will add a hit to the result list. The first one will show a hit in the< / span >
< span class = "sd" > info box. Both values can be set, or one of the two can be set." " " < / span >
< span class = "n" > send_accept_language_header< / span > < span class = "o" > =< / span > < span class = "kc" > True< / span >
< span class = "sd" > " " " The HTTP ``Accept-Language`` header is needed for wikis where< / span >
< span class = "sd" > LanguageConverter_ is enabled." " " < / span >
< span class = "n" > list_of_wikipedias< / span > < span class = "o" > =< / span > < span class = "s1" > ' https://meta.wikimedia.org/wiki/List_of_Wikipedias' < / span >
< span class = "sd" > " " " `List of all wikipedias < https://meta.wikimedia.org/wiki/List_of_Wikipedias> `_< / span >
< span class = "sd" > " " " < / span >
< span class = "n" > wikipedia_article_depth< / span > < span class = "o" > =< / span > < span class = "s1" > ' https://meta.wikimedia.org/wiki/Wikipedia_article_depth' < / span >
< span class = "sd" > " " " The *editing depth* of Wikipedia is one of several possible rough indicators< / span >
< span class = "sd" > of the encyclopedia' s collaborative quality, showing how frequently its articles< / span >
< span class = "sd" > are updated. The measurement of depth was introduced after some limitations of< / span >
< span class = "sd" > the classic measurement of article count were realized.< / span >
< span class = "sd" > " " " < / span >
< span class = "n" > rest_v1_summary_url< / span > < span class = "o" > =< / span > < span class = "s1" > ' https://< / span > < span class = "si" > {wiki_netloc}< / span > < span class = "s1" > /api/rest_v1/page/summary/< / span > < span class = "si" > {title}< / span > < span class = "s1" > ' < / span >
< span class = "sd" > " " " < / span >
< span class = "sd" > `wikipedia rest_v1 summary API`_:< / span >
< span class = "sd" > The summary response includes an extract of the first paragraph of the page in< / span >
< span class = "sd" > plain text and HTML as well as the type of page. This is useful for page< / span >
< span class = "sd" > previews (fka. Hovercards, aka. Popups) on the web and link previews in the< / span >
< span class = "sd" > apps.< / span >
< span class = "sd" > HTTP ``Accept-Language`` header (:py:obj:`send_accept_language_header`):< / span >
< span class = "sd" > The desired language variant code for wikis where LanguageConverter_ is< / span >
< span class = "sd" > enabled.< / span >
< span class = "sd" > .. _wikipedia rest_v1 summary API:< / span >
< span class = "sd" > https://en.wikipedia.org/api/rest_v1/#/Page%20content/get_page_summary__title_< / span >
< span class = "sd" > " " " < / span >
< span class = "n" > wiki_lc_locale_variants< / span > < span class = "o" > =< / span > < span class = "p" > {< / span >
< span class = "s2" > " zh" < / span > < span class = "p" > :< / span > < span class = "p" > (< / span >
< span class = "s2" > " zh-CN" < / span > < span class = "p" > ,< / span >
< span class = "s2" > " zh-HK" < / span > < span class = "p" > ,< / span >
< span class = "s2" > " zh-MO" < / span > < span class = "p" > ,< / span >
< span class = "s2" > " zh-MY" < / span > < span class = "p" > ,< / span >
< span class = "s2" > " zh-SG" < / span > < span class = "p" > ,< / span >
< span class = "s2" > " zh-TW" < / span > < span class = "p" > ,< / span >
< span class = "p" > ),< / span >
< span class = "s2" > " zh-classical" < / span > < span class = "p" > :< / span > < span class = "p" > (< / span > < span class = "s2" > " zh-classical" < / span > < span class = "p" > ,),< / span >
< span class = "p" > }< / span >
< span class = "sd" > " " " Mapping rule of the LanguageConverter_ to map a language and its variants to< / span >
< span class = "sd" > a Locale (used in the HTTP ``Accept-Language`` header). For example see `LC< / span >
< span class = "sd" > Chinese`_.< / span >
< span class = "sd" > .. _LC Chinese:< / span >
< span class = "sd" > https://meta.wikimedia.org/wiki/Wikipedias_in_multiple_writing_systems#Chinese< / span >
< span class = "sd" > " " " < / span >
< span class = "n" > wikipedia_script_variants< / span > < span class = "o" > =< / span > < span class = "p" > {< / span >
< span class = "s2" > " zh" < / span > < span class = "p" > :< / span > < span class = "p" > (< / span >
< span class = "s2" > " zh_Hant" < / span > < span class = "p" > ,< / span >
< span class = "s2" > " zh_Hans" < / span > < span class = "p" > ,< / span >
< span class = "p" > )< / span >
< span class = "p" > }< / span >
< div class = "viewcode-block" id = "get_wiki_params" >
< a class = "viewcode-back" href = "../../../dev/engines/online/wikipedia.html#searx.engines.wikipedia.get_wiki_params" > [docs]< / a >
< span class = "k" > def< / span > < span class = "w" > < / span > < span class = "nf" > get_wiki_params< / span > < span class = "p" > (< / span > < span class = "n" > sxng_locale< / span > < span class = "p" > ,< / span > < span class = "n" > eng_traits< / span > < span class = "p" > ):< / span >
< span class = "w" > < / span > < span class = "sd" > " " " Returns the Wikipedia language tag and the netloc that fits to the< / span >
< span class = "sd" > ``sxng_locale``. To support LanguageConverter_ this function rates a locale< / span >
< span class = "sd" > (region) higher than a language (compare :py:obj:`wiki_lc_locale_variants`).< / span >
< span class = "sd" > " " " < / span >
< span class = "n" > eng_tag< / span > < span class = "o" > =< / span > < span class = "n" > eng_traits< / span > < span class = "o" > .< / span > < span class = "n" > get_region< / span > < span class = "p" > (< / span > < span class = "n" > sxng_locale< / span > < span class = "p" > ,< / span > < span class = "n" > eng_traits< / span > < span class = "o" > .< / span > < span class = "n" > get_language< / span > < span class = "p" > (< / span > < span class = "n" > sxng_locale< / span > < span class = "p" > ,< / span > < span class = "s1" > ' en' < / span > < span class = "p" > ))< / span >
< span class = "n" > wiki_netloc< / span > < span class = "o" > =< / span > < span class = "n" > eng_traits< / span > < span class = "o" > .< / span > < span class = "n" > custom< / span > < span class = "p" > [< / span > < span class = "s1" > ' wiki_netloc' < / span > < span class = "p" > ]< / span > < span class = "o" > .< / span > < span class = "n" > get< / span > < span class = "p" > (< / span > < span class = "n" > eng_tag< / span > < span class = "p" > ,< / span > < span class = "s1" > ' en.wikipedia.org' < / span > < span class = "p" > )< / span >
< span class = "k" > return< / span > < span class = "n" > eng_tag< / span > < span class = "p" > ,< / span > < span class = "n" > wiki_netloc< / span > < / div >
< div class = "viewcode-block" id = "request" >
< a class = "viewcode-back" href = "../../../dev/engines/online/wikipedia.html#searx.engines.wikipedia.request" > [docs]< / a >
< span class = "k" > def< / span > < span class = "w" > < / span > < span class = "nf" > request< / span > < span class = "p" > (< / span > < span class = "n" > query< / span > < span class = "p" > ,< / span > < span class = "n" > params< / span > < span class = "p" > ):< / span >
< span class = "w" > < / span > < span class = "sd" > " " " Assemble a request (`wikipedia rest_v1 summary API`_)." " " < / span >
< span class = "k" > if< / span > < span class = "n" > query< / span > < span class = "o" > .< / span > < span class = "n" > islower< / span > < span class = "p" > ():< / span >
< span class = "n" > query< / span > < span class = "o" > =< / span > < span class = "n" > query< / span > < span class = "o" > .< / span > < span class = "n" > title< / span > < span class = "p" > ()< / span >
< span class = "n" > _eng_tag< / span > < span class = "p" > ,< / span > < span class = "n" > wiki_netloc< / span > < span class = "o" > =< / span > < span class = "n" > get_wiki_params< / span > < span class = "p" > (< / span > < span class = "n" > params< / span > < span class = "p" > [< / span > < span class = "s1" > ' searxng_locale' < / span > < span class = "p" > ],< / span > < span class = "n" > traits< / span > < span class = "p" > )< / span >
< span class = "n" > title< / span > < span class = "o" > =< / span > < span class = "n" > urllib< / span > < span class = "o" > .< / span > < span class = "n" > parse< / span > < span class = "o" > .< / span > < span class = "n" > quote< / span > < span class = "p" > (< / span > < span class = "n" > query< / span > < span class = "p" > )< / span >
< span class = "n" > params< / span > < span class = "p" > [< / span > < span class = "s1" > ' url' < / span > < span class = "p" > ]< / span > < span class = "o" > =< / span > < span class = "n" > rest_v1_summary_url< / span > < span class = "o" > .< / span > < span class = "n" > format< / span > < span class = "p" > (< / span > < span class = "n" > wiki_netloc< / span > < span class = "o" > =< / span > < span class = "n" > wiki_netloc< / span > < span class = "p" > ,< / span > < span class = "n" > title< / span > < span class = "o" > =< / span > < span class = "n" > title< / span > < span class = "p" > )< / span >
< span class = "n" > params< / span > < span class = "p" > [< / span > < span class = "s1" > ' raise_for_httperror' < / span > < span class = "p" > ]< / span > < span class = "o" > =< / span > < span class = "kc" > False< / span >
< span class = "n" > params< / span > < span class = "p" > [< / span > < span class = "s1" > ' soft_max_redirects' < / span > < span class = "p" > ]< / span > < span class = "o" > =< / span > < span class = "mi" > 2< / span >
< span class = "k" > return< / span > < span class = "n" > params< / span > < / div >
< span class = "c1" > # get response from search-request< / span >
< span class = "k" > def< / span > < span class = "w" > < / span > < span class = "nf" > response< / span > < span class = "p" > (< / span > < span class = "n" > resp< / span > < span class = "p" > ):< / span >
< span class = "n" > results< / span > < span class = "o" > =< / span > < span class = "p" > []< / span >
< span class = "k" > if< / span > < span class = "n" > resp< / span > < span class = "o" > .< / span > < span class = "n" > status_code< / span > < span class = "o" > ==< / span > < span class = "mi" > 404< / span > < span class = "p" > :< / span >
< span class = "k" > return< / span > < span class = "p" > []< / span >
< span class = "k" > if< / span > < span class = "n" > resp< / span > < span class = "o" > .< / span > < span class = "n" > status_code< / span > < span class = "o" > ==< / span > < span class = "mi" > 400< / span > < span class = "p" > :< / span >
< span class = "k" > try< / span > < span class = "p" > :< / span >
< span class = "n" > api_result< / span > < span class = "o" > =< / span > < span class = "n" > resp< / span > < span class = "o" > .< / span > < span class = "n" > json< / span > < span class = "p" > ()< / span >
< span class = "k" > except< / span > < span class = "ne" > Exception< / span > < span class = "p" > :< / span > < span class = "c1" > # pylint: disable=broad-except< / span >
< span class = "k" > pass< / span >
< span class = "k" > else< / span > < span class = "p" > :< / span >
< span class = "k" > if< / span > < span class = "p" > (< / span >
< span class = "n" > api_result< / span > < span class = "p" > [< / span > < span class = "s1" > ' type' < / span > < span class = "p" > ]< / span > < span class = "o" > ==< / span > < span class = "s1" > ' https://mediawiki.org/wiki/HyperSwitch/errors/bad_request' < / span >
< span class = "ow" > and< / span > < span class = "n" > api_result< / span > < span class = "p" > [< / span > < span class = "s1" > ' detail' < / span > < span class = "p" > ]< / span > < span class = "o" > ==< / span > < span class = "s1" > ' title-invalid-characters' < / span >
< span class = "p" > ):< / span >
< span class = "k" > return< / span > < span class = "p" > []< / span >
< span class = "n" > _network< / span > < span class = "o" > .< / span > < span class = "n" > raise_for_httperror< / span > < span class = "p" > (< / span > < span class = "n" > resp< / span > < span class = "p" > )< / span >
< span class = "n" > api_result< / span > < span class = "o" > =< / span > < span class = "n" > resp< / span > < span class = "o" > .< / span > < span class = "n" > json< / span > < span class = "p" > ()< / span >
< span class = "n" > title< / span > < span class = "o" > =< / span > < span class = "n" > utils< / span > < span class = "o" > .< / span > < span class = "n" > html_to_text< / span > < span class = "p" > (< / span > < span class = "n" > api_result< / span > < span class = "o" > .< / span > < span class = "n" > get< / span > < span class = "p" > (< / span > < span class = "s1" > ' titles' < / span > < span class = "p" > ,< / span > < span class = "p" > {})< / span > < span class = "o" > .< / span > < span class = "n" > get< / span > < span class = "p" > (< / span > < span class = "s1" > ' display' < / span > < span class = "p" > )< / span > < span class = "ow" > or< / span > < span class = "n" > api_result< / span > < span class = "o" > .< / span > < span class = "n" > get< / span > < span class = "p" > (< / span > < span class = "s1" > ' title' < / span > < span class = "p" > ))< / span >
< span class = "n" > wikipedia_link< / span > < span class = "o" > =< / span > < span class = "n" > api_result< / span > < span class = "p" > [< / span > < span class = "s1" > ' content_urls' < / span > < span class = "p" > ][< / span > < span class = "s1" > ' desktop' < / span > < span class = "p" > ][< / span > < span class = "s1" > ' page' < / span > < span class = "p" > ]< / span >
< span class = "k" > if< / span > < span class = "s2" > " list" < / span > < span class = "ow" > in< / span > < span class = "n" > display_type< / span > < span class = "ow" > or< / span > < span class = "n" > api_result< / span > < span class = "o" > .< / span > < span class = "n" > get< / span > < span class = "p" > (< / span > < span class = "s1" > ' type' < / span > < span class = "p" > )< / span > < span class = "o" > !=< / span > < span class = "s1" > ' standard' < / span > < span class = "p" > :< / span >
< span class = "c1" > # show item in the result list if ' list' is in the display options or it< / span >
< span class = "c1" > # is a item that can' t be displayed in a infobox.< / span >
< span class = "n" > results< / span > < span class = "o" > .< / span > < span class = "n" > append< / span > < span class = "p" > ({< / span > < span class = "s1" > ' url' < / span > < span class = "p" > :< / span > < span class = "n" > wikipedia_link< / span > < span class = "p" > ,< / span > < span class = "s1" > ' title' < / span > < span class = "p" > :< / span > < span class = "n" > title< / span > < span class = "p" > ,< / span > < span class = "s1" > ' content' < / span > < span class = "p" > :< / span > < span class = "n" > api_result< / span > < span class = "o" > .< / span > < span class = "n" > get< / span > < span class = "p" > (< / span > < span class = "s1" > ' description' < / span > < span class = "p" > ,< / span > < span class = "s1" > ' ' < / span > < span class = "p" > )})< / span >
< span class = "k" > if< / span > < span class = "s2" > " infobox" < / span > < span class = "ow" > in< / span > < span class = "n" > display_type< / span > < span class = "p" > :< / span >
< span class = "k" > if< / span > < span class = "n" > api_result< / span > < span class = "o" > .< / span > < span class = "n" > get< / span > < span class = "p" > (< / span > < span class = "s1" > ' type' < / span > < span class = "p" > )< / span > < span class = "o" > ==< / span > < span class = "s1" > ' standard' < / span > < span class = "p" > :< / span >
< span class = "n" > results< / span > < span class = "o" > .< / span > < span class = "n" > append< / span > < span class = "p" > (< / span >
< span class = "p" > {< / span >
< span class = "s1" > ' infobox' < / span > < span class = "p" > :< / span > < span class = "n" > title< / span > < span class = "p" > ,< / span >
< span class = "s1" > ' id' < / span > < span class = "p" > :< / span > < span class = "n" > wikipedia_link< / span > < span class = "p" > ,< / span >
< span class = "s1" > ' content' < / span > < span class = "p" > :< / span > < span class = "n" > api_result< / span > < span class = "o" > .< / span > < span class = "n" > get< / span > < span class = "p" > (< / span > < span class = "s1" > ' extract' < / span > < span class = "p" > ,< / span > < span class = "s1" > ' ' < / span > < span class = "p" > ),< / span >
< span class = "s1" > ' img_src' < / span > < span class = "p" > :< / span > < span class = "n" > api_result< / span > < span class = "o" > .< / span > < span class = "n" > get< / span > < span class = "p" > (< / span > < span class = "s1" > ' thumbnail' < / span > < span class = "p" > ,< / span > < span class = "p" > {})< / span > < span class = "o" > .< / span > < span class = "n" > get< / span > < span class = "p" > (< / span > < span class = "s1" > ' source' < / span > < span class = "p" > ),< / span >
< span class = "s1" > ' urls' < / span > < span class = "p" > :< / span > < span class = "p" > [{< / span > < span class = "s1" > ' title' < / span > < span class = "p" > :< / span > < span class = "s1" > ' Wikipedia' < / span > < span class = "p" > ,< / span > < span class = "s1" > ' url' < / span > < span class = "p" > :< / span > < span class = "n" > wikipedia_link< / span > < span class = "p" > }],< / span >
< span class = "p" > }< / span >
< span class = "p" > )< / span >
< span class = "k" > return< / span > < span class = "n" > results< / span >
< span class = "c1" > # Nonstandard language codes< / span >
< span class = "c1" > #< / span >
< span class = "c1" > # These Wikipedias use language codes that do not conform to the ISO 639< / span >
< span class = "c1" > # standard (which is how wiki subdomains are chosen nowadays).< / span >
< span class = "n" > lang_map< / span > < span class = "o" > =< / span > < span class = "n" > locales< / span > < span class = "o" > .< / span > < span class = "n" > LOCALE_BEST_MATCH< / span > < span class = "o" > .< / span > < span class = "n" > copy< / span > < span class = "p" > ()< / span >
< span class = "n" > lang_map< / span > < span class = "o" > .< / span > < span class = "n" > update< / span > < span class = "p" > (< / span >
< span class = "p" > {< / span >
< span class = "s1" > ' be-tarask' < / span > < span class = "p" > :< / span > < span class = "s1" > ' bel' < / span > < span class = "p" > ,< / span >
< span class = "s1" > ' ak' < / span > < span class = "p" > :< / span > < span class = "s1" > ' aka' < / span > < span class = "p" > ,< / span >
< span class = "s1" > ' als' < / span > < span class = "p" > :< / span > < span class = "s1" > ' gsw' < / span > < span class = "p" > ,< / span >
< span class = "s1" > ' bat-smg' < / span > < span class = "p" > :< / span > < span class = "s1" > ' sgs' < / span > < span class = "p" > ,< / span >
< span class = "s1" > ' cbk-zam' < / span > < span class = "p" > :< / span > < span class = "s1" > ' cbk' < / span > < span class = "p" > ,< / span >
< span class = "s1" > ' fiu-vro' < / span > < span class = "p" > :< / span > < span class = "s1" > ' vro' < / span > < span class = "p" > ,< / span >
< span class = "s1" > ' map-bms' < / span > < span class = "p" > :< / span > < span class = "s1" > ' map' < / span > < span class = "p" > ,< / span >
< span class = "s1" > ' no' < / span > < span class = "p" > :< / span > < span class = "s1" > ' nb-NO' < / span > < span class = "p" > ,< / span >
< span class = "s1" > ' nrm' < / span > < span class = "p" > :< / span > < span class = "s1" > ' nrf' < / span > < span class = "p" > ,< / span >
< span class = "s1" > ' roa-rup' < / span > < span class = "p" > :< / span > < span class = "s1" > ' rup' < / span > < span class = "p" > ,< / span >
< span class = "s1" > ' nds-nl' < / span > < span class = "p" > :< / span > < span class = "s1" > ' nds' < / span > < span class = "p" > ,< / span >
< span class = "c1" > #' simple: – < / span >
< span class = "s1" > ' zh-min-nan' < / span > < span class = "p" > :< / span > < span class = "s1" > ' nan' < / span > < span class = "p" > ,< / span >
< span class = "s1" > ' zh-yue' < / span > < span class = "p" > :< / span > < span class = "s1" > ' yue' < / span > < span class = "p" > ,< / span >
< span class = "s1" > ' an' < / span > < span class = "p" > :< / span > < span class = "s1" > ' arg' < / span > < span class = "p" > ,< / span >
< span class = "p" > }< / span >
< span class = "p" > )< / span >
< span class = "k" > def< / span > < span class = "w" > < / span > < span class = "nf" > fetch_traits< / span > < span class = "p" > (< / span > < span class = "n" > engine_traits< / span > < span class = "p" > :< / span > < span class = "n" > EngineTraits< / span > < span class = "p" > ):< / span >
< span class = "n" > fetch_wikimedia_traits< / span > < span class = "p" > (< / span > < span class = "n" > engine_traits< / span > < span class = "p" > )< / span >
< span class = "nb" > print< / span > < span class = "p" > (< / span > < span class = "s2" > " WIKIPEDIA_LANGUAGES: < / span > < span class = "si" > %s< / span > < span class = "s2" > " < / span > < span class = "o" > %< / span > < span class = "nb" > len< / span > < span class = "p" > (< / span > < span class = "n" > engine_traits< / span > < span class = "o" > .< / span > < span class = "n" > custom< / span > < span class = "p" > [< / span > < span class = "s1" > ' WIKIPEDIA_LANGUAGES' < / span > < span class = "p" > ]))< / span >
< div class = "viewcode-block" id = "fetch_wikimedia_traits" >
< a class = "viewcode-back" href = "../../../dev/engines/online/wikipedia.html#searx.engines.wikipedia.fetch_wikimedia_traits" > [docs]< / a >
< span class = "k" > def< / span > < span class = "w" > < / span > < span class = "nf" > fetch_wikimedia_traits< / span > < span class = "p" > (< / span > < span class = "n" > engine_traits< / span > < span class = "p" > :< / span > < span class = "n" > EngineTraits< / span > < span class = "p" > ):< / span >
< span class = "w" > < / span > < span class = "sd" > " " " Fetch languages from Wikipedia. Not all languages from the< / span >
< span class = "sd" > :py:obj:`list_of_wikipedias` are supported by SearXNG locales, only those< / span >
< span class = "sd" > known from :py:obj:`searx.locales.LOCALE_NAMES` or those with a minimal< / span >
< span class = "sd" > :py:obj:`editing depth < wikipedia_article_depth> `.< / span >
< span class = "sd" > The location of the Wikipedia address of a language is mapped in a< / span >
< span class = "sd" > :py:obj:`custom field < searx.enginelib.traits.EngineTraits.custom> `< / span >
< span class = "sd" > (``wiki_netloc``). Here is a reduced example:< / span >
< span class = "sd" > .. code:: python< / span >
< span class = "sd" > traits.custom[' wiki_netloc' ] = {< / span >
< span class = "sd" > " en" : " en.wikipedia.org" ,< / span >
< span class = "sd" > ..< / span >
< span class = "sd" > " gsw" : " als.wikipedia.org" ,< / span >
< span class = "sd" > ..< / span >
< span class = "sd" > " zh" : " zh.wikipedia.org" ,< / span >
< span class = "sd" > " zh-classical" : " zh-classical.wikipedia.org" < / span >
< span class = "sd" > }< / span >
< span class = "sd" > " " " < / span >
< span class = "c1" > # pylint: disable=too-many-branches< / span >
< span class = "n" > engine_traits< / span > < span class = "o" > .< / span > < span class = "n" > custom< / span > < span class = "p" > [< / span > < span class = "s1" > ' wiki_netloc' < / span > < span class = "p" > ]< / span > < span class = "o" > =< / span > < span class = "p" > {}< / span >
< span class = "n" > engine_traits< / span > < span class = "o" > .< / span > < span class = "n" > custom< / span > < span class = "p" > [< / span > < span class = "s1" > ' WIKIPEDIA_LANGUAGES' < / span > < span class = "p" > ]< / span > < span class = "o" > =< / span > < span class = "p" > []< / span >
< span class = "c1" > # insert alias to map from a script or region to a wikipedia variant< / span >
< span class = "k" > for< / span > < span class = "n" > eng_tag< / span > < span class = "p" > ,< / span > < span class = "n" > sxng_tag_list< / span > < span class = "ow" > in< / span > < span class = "n" > wikipedia_script_variants< / span > < span class = "o" > .< / span > < span class = "n" > items< / span > < span class = "p" > ():< / span >
< span class = "k" > for< / span > < span class = "n" > sxng_tag< / span > < span class = "ow" > in< / span > < span class = "n" > sxng_tag_list< / span > < span class = "p" > :< / span >
< span class = "n" > engine_traits< / span > < span class = "o" > .< / span > < span class = "n" > languages< / span > < span class = "p" > [< / span > < span class = "n" > sxng_tag< / span > < span class = "p" > ]< / span > < span class = "o" > =< / span > < span class = "n" > eng_tag< / span >
< span class = "k" > for< / span > < span class = "n" > eng_tag< / span > < span class = "p" > ,< / span > < span class = "n" > sxng_tag_list< / span > < span class = "ow" > in< / span > < span class = "n" > wiki_lc_locale_variants< / span > < span class = "o" > .< / span > < span class = "n" > items< / span > < span class = "p" > ():< / span >
< span class = "k" > for< / span > < span class = "n" > sxng_tag< / span > < span class = "ow" > in< / span > < span class = "n" > sxng_tag_list< / span > < span class = "p" > :< / span >
< span class = "n" > engine_traits< / span > < span class = "o" > .< / span > < span class = "n" > regions< / span > < span class = "p" > [< / span > < span class = "n" > sxng_tag< / span > < span class = "p" > ]< / span > < span class = "o" > =< / span > < span class = "n" > eng_tag< / span >
< span class = "n" > resp< / span > < span class = "o" > =< / span > < span class = "n" > _network< / span > < span class = "o" > .< / span > < span class = "n" > get< / span > < span class = "p" > (< / span > < span class = "n" > list_of_wikipedias< / span > < span class = "p" > )< / span >
< span class = "k" > if< / span > < span class = "ow" > not< / span > < span class = "n" > resp< / span > < span class = "o" > .< / span > < span class = "n" > ok< / span > < span class = "p" > :< / span >
< span class = "nb" > print< / span > < span class = "p" > (< / span > < span class = "s2" > " ERROR: response from Wikipedia is not OK." < / span > < span class = "p" > )< / span >
< span class = "n" > dom< / span > < span class = "o" > =< / span > < span class = "n" > html< / span > < span class = "o" > .< / span > < span class = "n" > fromstring< / span > < span class = "p" > (< / span > < span class = "n" > resp< / span > < span class = "o" > .< / span > < span class = "n" > text< / span > < span class = "p" > )< / span >
< span class = "k" > for< / span > < span class = "n" > row< / span > < span class = "ow" > in< / span > < span class = "n" > dom< / span > < span class = "o" > .< / span > < span class = "n" > xpath< / span > < span class = "p" > (< / span > < span class = "s1" > ' //table[contains(@class," sortable" )]//tbody/tr' < / span > < span class = "p" > ):< / span >
< span class = "n" > cols< / span > < span class = "o" > =< / span > < span class = "n" > row< / span > < span class = "o" > .< / span > < span class = "n" > xpath< / span > < span class = "p" > (< / span > < span class = "s1" > ' ./td' < / span > < span class = "p" > )< / span >
< span class = "k" > if< / span > < span class = "ow" > not< / span > < span class = "n" > cols< / span > < span class = "p" > :< / span >
< span class = "k" > continue< / span >
< span class = "n" > cols< / span > < span class = "o" > =< / span > < span class = "p" > [< / span > < span class = "n" > c< / span > < span class = "o" > .< / span > < span class = "n" > text_content< / span > < span class = "p" > ()< / span > < span class = "o" > .< / span > < span class = "n" > strip< / span > < span class = "p" > ()< / span > < span class = "k" > for< / span > < span class = "n" > c< / span > < span class = "ow" > in< / span > < span class = "n" > cols< / span > < span class = "p" > ]< / span >
< span class = "n" > depth< / span > < span class = "o" > =< / span > < span class = "nb" > float< / span > < span class = "p" > (< / span > < span class = "n" > cols< / span > < span class = "p" > [< / span > < span class = "mi" > 11< / span > < span class = "p" > ]< / span > < span class = "o" > .< / span > < span class = "n" > replace< / span > < span class = "p" > (< / span > < span class = "s1" > ' -' < / span > < span class = "p" > ,< / span > < span class = "s1" > ' 0' < / span > < span class = "p" > )< / span > < span class = "o" > .< / span > < span class = "n" > replace< / span > < span class = "p" > (< / span > < span class = "s1" > ' ,' < / span > < span class = "p" > ,< / span > < span class = "s1" > ' ' < / span > < span class = "p" > ))< / span >
< span class = "n" > articles< / span > < span class = "o" > =< / span > < span class = "nb" > int< / span > < span class = "p" > (< / span > < span class = "n" > cols< / span > < span class = "p" > [< / span > < span class = "mi" > 4< / span > < span class = "p" > ]< / span > < span class = "o" > .< / span > < span class = "n" > replace< / span > < span class = "p" > (< / span > < span class = "s1" > ' ,' < / span > < span class = "p" > ,< / span > < span class = "s1" > ' ' < / span > < span class = "p" > )< / span > < span class = "o" > .< / span > < span class = "n" > replace< / span > < span class = "p" > (< / span > < span class = "s1" > ' ,' < / span > < span class = "p" > ,< / span > < span class = "s1" > ' ' < / span > < span class = "p" > ))< / span >
< span class = "n" > eng_tag< / span > < span class = "o" > =< / span > < span class = "n" > cols< / span > < span class = "p" > [< / span > < span class = "mi" > 3< / span > < span class = "p" > ]< / span >
< span class = "n" > wiki_url< / span > < span class = "o" > =< / span > < span class = "n" > row< / span > < span class = "o" > .< / span > < span class = "n" > xpath< / span > < span class = "p" > (< / span > < span class = "s1" > ' ./td[4]/a/@href' < / span > < span class = "p" > )[< / span > < span class = "mi" > 0< / span > < span class = "p" > ]< / span >
< span class = "n" > wiki_url< / span > < span class = "o" > =< / span > < span class = "n" > urllib< / span > < span class = "o" > .< / span > < span class = "n" > parse< / span > < span class = "o" > .< / span > < span class = "n" > urlparse< / span > < span class = "p" > (< / span > < span class = "n" > wiki_url< / span > < span class = "p" > )< / span >
< span class = "k" > try< / span > < span class = "p" > :< / span >
< span class = "n" > sxng_tag< / span > < span class = "o" > =< / span > < span class = "n" > locales< / span > < span class = "o" > .< / span > < span class = "n" > language_tag< / span > < span class = "p" > (< / span > < span class = "n" > babel< / span > < span class = "o" > .< / span > < span class = "n" > Locale< / span > < span class = "o" > .< / span > < span class = "n" > parse< / span > < span class = "p" > (< / span > < span class = "n" > lang_map< / span > < span class = "o" > .< / span > < span class = "n" > get< / span > < span class = "p" > (< / span > < span class = "n" > eng_tag< / span > < span class = "p" > ,< / span > < span class = "n" > eng_tag< / span > < span class = "p" > ),< / span > < span class = "n" > sep< / span > < span class = "o" > =< / span > < span class = "s1" > ' -' < / span > < span class = "p" > ))< / span >
< span class = "k" > except< / span > < span class = "n" > babel< / span > < span class = "o" > .< / span > < span class = "n" > UnknownLocaleError< / span > < span class = "p" > :< / span >
< span class = "c1" > # print(" ERROR: %s [%s] is unknown by babel" % (cols[0], eng_tag))< / span >
< span class = "k" > continue< / span >
< span class = "k" > finally< / span > < span class = "p" > :< / span >
< span class = "n" > engine_traits< / span > < span class = "o" > .< / span > < span class = "n" > custom< / span > < span class = "p" > [< / span > < span class = "s1" > ' WIKIPEDIA_LANGUAGES' < / span > < span class = "p" > ]< / span > < span class = "o" > .< / span > < span class = "n" > append< / span > < span class = "p" > (< / span > < span class = "n" > eng_tag< / span > < span class = "p" > )< / span >
< span class = "k" > if< / span > < span class = "n" > sxng_tag< / span > < span class = "ow" > not< / span > < span class = "ow" > in< / span > < span class = "n" > locales< / span > < span class = "o" > .< / span > < span class = "n" > LOCALE_NAMES< / span > < span class = "p" > :< / span >
< span class = "k" > if< / span > < span class = "n" > articles< / span > < span class = "o" > < < / span > < span class = "mi" > 10000< / span > < span class = "p" > :< / span >
< span class = "c1" > # exclude languages with too few articles< / span >
< span class = "k" > continue< / span >
< span class = "k" > if< / span > < span class = "nb" > int< / span > < span class = "p" > (< / span > < span class = "n" > depth< / span > < span class = "p" > )< / span > < span class = "o" > < < / span > < span class = "mi" > 20< / span > < span class = "p" > :< / span >
< span class = "c1" > # Rough indicator of a Wikipedia’ < / span >
< span class = "c1" > # frequently its articles are updated.< / span >
< span class = "k" > continue< / span >
< span class = "n" > conflict< / span > < span class = "o" > =< / span > < span class = "n" > engine_traits< / span > < span class = "o" > .< / span > < span class = "n" > languages< / span > < span class = "o" > .< / span > < span class = "n" > get< / span > < span class = "p" > (< / span > < span class = "n" > sxng_tag< / span > < span class = "p" > )< / span >
< span class = "k" > if< / span > < span class = "n" > conflict< / span > < span class = "p" > :< / span >
< span class = "k" > if< / span > < span class = "n" > conflict< / span > < span class = "o" > !=< / span > < span class = "n" > eng_tag< / span > < span class = "p" > :< / span >
< span class = "nb" > print< / span > < span class = "p" > (< / span > < span class = "s2" > " CONFLICT: babel < / span > < span class = "si" > %s< / span > < span class = "s2" > --> < / span > < span class = "si" > %s< / span > < span class = "s2" > , < / span > < span class = "si" > %s< / span > < span class = "s2" > " < / span > < span class = "o" > %< / span > < span class = "p" > (< / span > < span class = "n" > sxng_tag< / span > < span class = "p" > ,< / span > < span class = "n" > conflict< / span > < span class = "p" > ,< / span > < span class = "n" > eng_tag< / span > < span class = "p" > ))< / span >
< span class = "k" > continue< / span >
< span class = "n" > engine_traits< / span > < span class = "o" > .< / span > < span class = "n" > languages< / span > < span class = "p" > [< / span > < span class = "n" > sxng_tag< / span > < span class = "p" > ]< / span > < span class = "o" > =< / span > < span class = "n" > eng_tag< / span >
< span class = "n" > engine_traits< / span > < span class = "o" > .< / span > < span class = "n" > custom< / span > < span class = "p" > [< / span > < span class = "s1" > ' wiki_netloc' < / span > < span class = "p" > ][< / span > < span class = "n" > eng_tag< / span > < span class = "p" > ]< / span > < span class = "o" > =< / span > < span class = "n" > wiki_url< / span > < span class = "o" > .< / span > < span class = "n" > netloc< / span >
< span class = "n" > engine_traits< / span > < span class = "o" > .< / span > < span class = "n" > custom< / span > < span class = "p" > [< / span > < span class = "s1" > ' WIKIPEDIA_LANGUAGES' < / span > < span class = "p" > ]< / span > < span class = "o" > .< / span > < span class = "n" > sort< / span > < span class = "p" > ()< / span > < / div >
< / pre > < / div >
< div class = "clearer" > < / div >
< / div >
< / div >
< / div >
< span id = "sidebar-top" > < / span >
< div class = "sphinxsidebar" role = "navigation" aria-label = "Main" >
< div class = "sphinxsidebarwrapper" >
< p class = "logo" > < a href = "../../../index.html" >
< img class = "logo" src = "../../../_static/searxng-wordmark.svg" alt = "Logo of SearXNG" / >
< / a > < / p >
< h3 > < a href = "../../../index.html" > Table of Contents< / a > < / h3 >
< ul >
< li class = "toctree-l1" > < a class = "reference internal" href = "../../../user/index.html" > User information< / a > < / li >
< li class = "toctree-l1" > < a class = "reference internal" href = "../../../own-instance.html" > Why use a private instance?< / a > < / li >
< li class = "toctree-l1" > < a class = "reference internal" href = "../../../admin/index.html" > Administrator documentation< / a > < / li >
< li class = "toctree-l1" > < a class = "reference internal" href = "../../../dev/index.html" > Developer documentation< / a > < / li >
< li class = "toctree-l1" > < a class = "reference internal" href = "../../../utils/index.html" > DevOps tooling box< / a > < / li >
< li class = "toctree-l1" > < a class = "reference internal" href = "../../../src/index.html" > Source-Code< / a > < / li >
< / ul >
< h3 > Project Links< / h3 >
< ul >
< li > < a href = "https://github.com/searxng/searxng/tree/master" > Source< / a >
< li > < a href = "https://github.com/searxng/searxng/wiki" > Wiki< / a >
< li > < a href = "https://searx.space" > Public instances< / a >
< li > < a href = "https://github.com/searxng/searxng/issues" > Issue Tracker< / a >
< / ul > < h3 > Navigation< / h3 >
< ul >
< li > < a href = "../../../index.html" > Overview< / a >
< ul >
< li > < a href = "../../index.html" > Module code< / a >
< ul >
< li > < a href = "../engines.html" > searx.engines< / a >
< / ul >
< / li > < / ul >
< / li >
< / ul >
< / li >
< / ul >
< search id = "searchbox" style = "display: none" role = "search" >
< h3 id = "searchlabel" > Quick search< / h3 >
< div class = "searchformwrapper" >
< form class = "search" action = "../../../search.html" method = "get" >
< input type = "text" name = "q" aria-labelledby = "searchlabel" autocomplete = "off" autocorrect = "off" autocapitalize = "off" spellcheck = "false" / >
< input type = "submit" value = "Go" / >
< / form >
< / div >
< / search >
< script > document . getElementById ( 'searchbox' ) . style . display = "block" < / script >
< / div >
< / div >
< div class = "clearer" > < / div >
< / div >
< div class = "footer" role = "contentinfo" >
© Copyright SearXNG team.
< / div >
< / body >
< / html >