feat: add export_statistics and export_mentions make targets with dataset prefix markers
Add two new Makefile targets for exporting statistics and mentions data from the dataset file, each prefixed with "===" markers in the dataset.py dump function to enable clean grep extraction. The export_statistics target filters lines containing "Statistics:" while export_mentions filters lines containing "times ment", both writing results to separate export files.
This commit is contained in:
@@ -1,4 +1,4 @@
|
||||
all: build sync_excempt export_dataset export_stats merge_images
|
||||
all: build sync_excempt export_dataset export_stats merge_images export_statistics export_mentions
|
||||
|
||||
build:
|
||||
pip install build
|
||||
@@ -27,6 +27,20 @@ export_dataset:
|
||||
@echo "Exporting dataset to be used for LLM embedding. Result will be ./export/0_dataset.txt"
|
||||
dr.dataset > export/0_dataset.txt
|
||||
|
||||
export_statistics:
|
||||
@echo "Exporting statisticts. Result will be ./export/2_statistics.txt"
|
||||
cat export/dataset.txt | grep "Statistics: "
|
||||
cat export/dataset.txt | grep "Statistics: " > export/2_statistics.txt
|
||||
|
||||
|
||||
export_mentions:
|
||||
@echo "Exporting mentions. Result will be ./export/3_mentions.txt"
|
||||
cat export/dataset.txt | grep "times ment"
|
||||
cat export/dataset.txt | grep "times ment" > export/3_mentions.txt
|
||||
|
||||
|
||||
merge_images:
|
||||
@echo "Merging images to one big image. Result will be ./export/1_graphs_compliation.png."
|
||||
python merge_images.py
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user