|
# Downie
|
|
|
|
**Downie** is a full **site downloader**. I tried several downloaders and they all were not capable to download big sites like **molodetz.nl** (containing **300.000+** links / resources). See **features** to see what makes this site downloader **better than the competition**.
|
|
|
|
If you encounter a site that can't be downloaded using this downloader, contact me @ *retoor@molodetz.nl*. Also, creating a bug ticket is possible after logging in. **I respond to both methods within 24h**.
|
|
|
|
## Features
|
|
- Fast as light **async** fetching of your pages allowing concurrency.
|
|
- Limit fetching only to given domain. **Remote URLs are not followed**. Also redirects to remote URLs (yes, that can be an issue) are not followed.
|
|
- Discriminates between file and folder for urls by adding `.html` to pages not having an extension. So `/pony` becomes /pony.html so that `/pony/photos.html` is possible.
|
|
- **Absolute URLs will be converted to relative URLs** so that the site becomes **portable** and **will work isolated** on your webserver.
|
|
- **Progress is saved.** You do not have to start over completely after exiting the application. It will just continue.
|
|
- Caching of already visited urls.
|
|
|
|
## Using Downie
|
|
After following the **installation instructions**, use `downie https://target-site.com` if **globally installed**. If not globally installed, execute `./downie https://target-site.com` from the current directory.
|
|
|
|
## Installation
|
|
|
|
### Debian (for development)
|
|
Only **requirement** is `make`. Install by executing `sudo apt install make -y` in the terminal.
|
|
Installation of all required Python dependencies are done by executing `make install`.
|
|
This will **install system packages** using `apt`:
|
|
- `python`
|
|
- `python3-venv`
|
|
- `python3-pip`
|
|
*There is a big chance you already have them.*
|
|
After that:
|
|
1. A Python environment will be created named `.venv`. *You can ignore this folder.*
|
|
2. Required Python libraries will be installed. *(aiohttp, aiofiles, beautifullsoup4, app (molodetz)).*
|
|
3. Downie will be installed. *(For the devs: in edit mode with -e!).*
|
|
4. A executable file named `downie` will be placed in the current directory. This is the main application.
|
|
**Tip:** make downie globally accessable by copying `downie` to `/usr/local/bin`.
|
|
|
|
### Other operating systems; Windows / Mac; not for development.
|
|
You have to figure out how to install python3, python3-venv and python3-pip on your OS. Make sure you have these on your system. Chance is big that you already got those.
|
|
Run the following commands:
|
|
```bash
|
|
python3 -m venv .venv
|
|
source .venv/bin/activate
|
|
pip install git+https://retoor.molodetz.nl/retoor/downie.git
|
|
cp .venv/bin/downie .
|
|
```
|
|
**Tip:** make downie globally accessable by copying `downie` to `/usr/local/bin`.
|
|
|
|
## Default configuration / internals
|
|
Not defined by CLI parameters, but configurable as Downie class constructor parameters:
|
|
- `concurrent_request_count` - Amount of concurrent requests. This defaults to 500.
|
|
- `concurrent_write_count` - Amount of concurrent downloads of files. This defaults to 10.
|
|
These values are based on my usage. There is a possibility that there are more optimial parameters.
|
|
|
|
## Running a downloaded site:
|
|
|
|
### Open site statically
|
|
Just open a page (.html) from the site using your browser.
|
|
|
|
### Serve the site
|
|
Simple, run:
|
|
```bash
|
|
python3 -m http.server [port]
|
|
```
|
|
If you don't know what port to use, use `7331`. Your website is accessable on `https://localhost:7331` now.
|
|
|
|
## Caution
|
|
URLs of downloaded content can be modified to make the site portable. So it is possible that your downloaded site is literally the content online. It is optimized. A literal copy of the site could be unusable.
|