retoor
969ef8b270
The CI test workflow previously attempted to run an invalid shell command "Check if starts correcly. Syntax check." which would fail. This commit replaces it with a safe echo statement that prints the same message, allowing the pipeline to proceed to subsequent steps.
PDF2Text
I've converted 8gb of PDF's to text in one afternoon on a decade old x270 using this script. Performant enough imho. Try to get 8Gb in your LLM and getting it to actually use it. That's the challenge.
Convert all PDF's to text
This is an script for converting a batch of PDF's to text for machine learning. It only has two dependencies:
python3pdf.miner(python requirement, specified in requirements.txt file)
Installation
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
Usage:
Activate your virtual environment.
source .venv/bin/activate
./pdf2text [source/destination dir]
You read that correctly, the source directory is also the destination directory.
Todo:
Make decent python package so it's installable on system without having to load environment first. Not sure if worth it, it's not something you daily use.