feat: add dataset export command and contributions views with user stats aggregation

Add dr.dataset console script entry point and implement dump function in new dataset module. Create SQL views for contributions, contributions_extended, rants_of_user, and posts_of_user in db.py to aggregate user activity metrics. Update Makefile with sync, export_stats, and export_dataset targets. Add timeout parameter to HTTP requests in devrant.py and remove demo code. Extend README with usage instructions for environment setup and available dr.* commands.
This commit is contained in:
2024-11-23 18:56:52 +00:00
parent 507a8dcaeb
commit d1594ed798
16 changed files with 157 additions and 59 deletions
+14 -4
View File
@@ -1,6 +1,16 @@
all: build
all: build sync export_stats export_dataset
build:
pip install build
python -m build .
pip install -e .
time pip install build
time python -m build .
time pip install -e .
sync:
@echo "Synchronizing with devrant.com."
time dr.sync
export_stats:
@echo "Exporting statisticts."
time dr.stats_all
export_dataset:
@echo "Exporting dataset to be used for LLM embedding."
time dr.dataset > export/dataset.txt