This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<pid="module-searx.limiter">Bot protection / IP rate limitation. The intention of rate limitation is to
limit suspicious requests from an IP. The motivation behind this is the fact
that SearXNG passes through requests from bots and is thus classified as a bot
itself. As a result, the SearXNG engine then receives a CAPTCHA or is blocked
by the search engine (the origin) in some other way.</p>
<p>To avoid blocking, the requests from bots to SearXNG must also be blocked, this
is the task of the limiter. To perform this task, the limiter uses the methods
from the <aclass="reference internal"href="../src/searx.botdetection.html#botdetection"><spanclass="std std-ref">Bot Detection</span></a>:</p>
<ulclass="simple">
<li><p>Analysis of the HTTP header in the request / <aclass="reference internal"href="../src/searx.botdetection.html#botdetection-probe-headers"><spanclass="std std-ref">Probe HTTP headers</span></a>
can be easily bypassed.</p></li>
<li><p>Block and pass lists in which IPs are listed / <aclass="reference internal"href="../src/searx.botdetection.html#botdetection-ip-lists"><spanclass="std std-ref">IP lists</span></a>
are hard to maintain, since the IPs of bots are not all known and change over
the time.</p></li>
<li><p>Detection & dynamically <aclass="reference internal"href="../src/searx.botdetection.html#botdetection-rate-limit"><spanclass="std std-ref">Rate limit</span></a> of bots based on the
behavior of the requests. For dynamically changeable IP lists a Redis
database is needed.</p></li>
</ul>
<p>The prerequisite for IP based methods is the correct determination of the IP of
the client. The IP of the client is determined via the <aclass="reference external"href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For">X-Forwarded-For</a> HTTP
header.</p>
<divclass="admonition attention">
<pclass="admonition-title">Attention</p>
<p>A correct setup of the HTTP request headers <codeclass="docutils literal notranslate"><spanclass="pre">X-Forwarded-For</span></code> and
<codeclass="docutils literal notranslate"><spanclass="pre">X-Real-IP</span></code> is essential to be able to assign a request to an IP correctly:</p>
<h2><aclass="toc-backref"href="#id3"role="doc-backlink">Enable Limiter</a><aclass="headerlink"href="#enable-limiter"title="Link to this heading">¶</a></h2>
<spanclass="w"></span><spanclass="l l-Scalar l-Scalar-Plain">limiter</span><spanclass="p p-Indicator">:</span><spanclass="w"></span><spanclass="l l-Scalar l-Scalar-Plain">true</span><spanclass="w"></span><spanclass="c1"># rate limit the number of request on the instance, block some bots</span>
</pre></div>
</div>
<p>and set the redis-url connection. Check the value, it depends on your redis DB
(see <aclass="reference internal"href="settings/settings_redis.html#settings-redis"><spanclass="std std-ref">redis:</span></a>), by example:</p>
<h2><aclass="toc-backref"href="#id4"role="doc-backlink">Configure Limiter</a><aclass="headerlink"href="#configure-limiter"title="Link to this heading">¶</a></h2>
<p>The methods of <aclass="reference internal"href="../src/searx.botdetection.html#botdetection"><spanclass="std std-ref">Bot Detection</span></a> the limiter uses are configured in a local
file <codeclass="docutils literal notranslate"><spanclass="pre">/etc/searxng/limiter.toml</span></code>. The defaults are shown in <aclass="reference internal"href="#limiter-toml">limiter.toml</a> /
Don’t copy all values to your local configuration, just enable what you need by
overwriting the defaults. For instance to activate the <codeclass="docutils literal notranslate"><spanclass="pre">link_token</span></code> method in
the <aclass="reference internal"href="../src/searx.botdetection.html#botdetection-ip-limit"><spanclass="std std-ref">Method ip_limit</span></a> you only need to set this option to <codeclass="docutils literal notranslate"><spanclass="pre">true</span></code>:</p>
<spanid="id2"></span><h2><aclass="toc-backref"href="#id5"role="doc-backlink"><codeclass="docutils literal notranslate"><spanclass="pre">limiter.toml</span></code></a><aclass="headerlink"href="#limiter-toml"title="Link to this heading">¶</a></h2>
<p>In this file the limiter finds the configuration of the <aclass="reference internal"href="../src/searx.botdetection.html#botdetection"><spanclass="std std-ref">Bot Detection</span></a>:</p>
<h2><aclass="toc-backref"href="#id6"role="doc-backlink">Implementation</a><aclass="headerlink"href="#implementation"title="Link to this heading">¶</a></h2>
<spanclass="sig-prename descclassname"><spanclass="pre">searx.limiter.</span></span><spanclass="sig-name descname"><spanclass="pre">LIMITER_CFG_SCHEMA</span></span><emclass="property"><spanclass="w"></span><spanclass="p"><spanclass="pre">=</span></span><spanclass="w"></span><spanclass="pre">PosixPath('/home/runner/work/searxng/searxng/searx/limiter.toml')</span></em><aclass="headerlink"href="#searx.limiter.LIMITER_CFG_SCHEMA"title="Link to this definition">¶</a></dt>
<dd><p>Base configuration (schema) of the botdetection.</p>
<spanclass="sig-prename descclassname"><spanclass="pre">searx.limiter.</span></span><spanclass="sig-name descname"><spanclass="pre">pre_request</span></span><spanclass="sig-paren">(</span><spanclass="sig-paren">)</span><aclass="reference internal"href="../_modules/searx/limiter.html#pre_request"><spanclass="viewcode-link"><spanclass="pre">[source]</span></span></a><aclass="headerlink"href="#searx.limiter.pre_request"title="Link to this definition">¶</a></dt>
<spanclass="sig-prename descclassname"><spanclass="pre">searx.limiter.</span></span><spanclass="sig-name descname"><spanclass="pre">is_installed</span></span><spanclass="sig-paren">(</span><spanclass="sig-paren">)</span><aclass="reference internal"href="../_modules/searx/limiter.html#is_installed"><spanclass="viewcode-link"><spanclass="pre">[source]</span></span></a><aclass="headerlink"href="#searx.limiter.is_installed"title="Link to this definition">¶</a></dt>
<dd><p>Returns <codeclass="docutils literal notranslate"><spanclass="pre">True</span></code> if limiter is active and a redis DB is available.</p>