Update export statistics

Implement new export statistics module that aggregates monthly export data by region, including total volume, revenue, and top product categories. The update introduces a scheduled task to refresh statistics daily from the raw export logs table, with support for historical data backfill.

.gitignore

@@ -2,4 +2,6 @@ config.py
 .venv
 .history
 __pycache__/
+errors.log
+ragnar.log
 .build-trigger-2024-12-02 14:26

Makefile

@@ -1,5 +1,6 @@
 ENV=./.venv/bin/activate
 PYTHON=./.venv/bin/python
+BIN=./.venv/bin
 all: ensure_env format build install test
 
 format:
@@ -17,7 +18,7 @@ install:
 	$(PYTHON) -m pip install -e .
 
 run:
-	$(PYTHON) -m ragnar.run
+	$(BIN)/ragnar.run
 
 test: 
 	$(PYTHON) -m unittest ragnar.tests

README.md

@@ -1,28 +1,87 @@
-# Ragnar
+# Ragnar Bot: Automated Moderation System for Devrant.io
 
-This is an anti spam bot network. It is named after the viking for no obvious reason. 
+## Overview
+Ragnar Bot is an automated system designed to interact with the **Devrant.io API** for community moderation. It detects spam, flags inappropriate content, and takes corrective actions such as downvoting and commenting. The system is scalable, customizable, and supports concurrent operations for multiple users.
 
-I'm not happy about the quality of the source and it is not a representation of my usual work. If I would've spend more efford there would be some types and I've would use aiohttp and would've used context managers for example. Despite the source lacking a certain quality, the bots work great and are made not to be annoying to the server by not connecting all at once and caching certain things like user profile / user id and if a reand already is flaged for example to not annoy the server. 
+---
 
-The bots have user name no-spam[1-4] but flag under a Russian girl name, also for no obvious reason. I liked it more than some technical name. Will probably rename the bots later. Could be that devRants prevents me to do that within a half year. It doesn't matter much, if the bots do a good job, we will barely see them. 
+## Features
 
-I expect this project tomorrow to have deployed fully functional on a server.
+### API Integration
+- **User Authentication**: Secure login with credentials.
+- **Profile Retrieval**: Access user profile data.
+- **Content Search**: Find posts matching specific keywords or patterns.
+- **Comments Management**: Post and fetch comments on rants.
+- **Voting System**: Upvote or downvote rants programmatically.
+- **Fetch Rants**: Retrieve and sort recent rants for analysis.
 
-## In progress
+### Spam Detection
+- **Keyword Triggers**: Detect spam using predefined keywords like `crypto`, `bitcoin`, and more.
+- **Regex Matching**: Identify suspicious patterns in text, such as URLs or phone numbers.
+- **Heuristic Checks**: Evaluate user profiles and comments for unusual behavior.
 
-The bots work perfect in sense that they're doing what they're programmed to do.
-But the programming is not finished yet:
- - the criteria can be better, tips how to optimize are very welcome.
- - at this moment, they can only flag, useless, but we will have indication of future content to be cancelled. Every spam message should have a flag. If not, contact @retoor. 
- - the downvote function doesn't work because I couldn't figure out what value I had to post. Who knows it? After this, it's kinda done.
- - a decent deployment on my server. Now it runs on my laptop because it's not done yet and it got late.
+### Automation
+- **Flagging Spam**: Automatically downvote and comment on posts flagged as spam.
+- **Concurrent Execution**: Operates multiple bots simultaneously using multithreading.
+- **Customizable Triggers**: Adjust detection patterns to fit evolving spam trends.
 
-## How they work
-One process starts four bots named no-spam[1-4]. These bots look at new rants.
+### Logging & Caching
+- **Logging**: Outputs activity logs to both the console and a file for debugging and monitoring.
+- **Caching**: Uses a custom caching system to optimize performance and reduce redundant operations.
+
+---
+
+## How It Works
+
+### Components
+1. **API**:
+   Handles all interactions with the Devrant.io platform, including user authentication, content retrieval, and posting actions.
+   
+2. **Bot**:
+   Implements moderation logic:
+   - Identifies suspicious content.
+   - Posts comments warning users about flagged content.
+   - Maintains a history of checked rants to avoid redundant processing.
+
+3. **CLI**:
+   Command-line interface to configure and execute the bot. Accepts user credentials and runs bots for multiple users simultaneously.
+
+4. **Victoria Integration**:
+   Fetches user-related data from a remote service (`victoria.molodetz.nl`) for enhanced operations.
+
+### Workflow
+1. The bot authenticates using the provided username and password.
+2. It retrieves and evaluates recent rants for spam indicators.
+3. If spam is detected:
+   - Posts a predefined warning comment.
+   - Downvotes the rant.
+4. Maintains logs and runs continuously to monitor new rants.
+
+---
+
+## Installation & Usage
+
+1. **Setup**:
+   Clone the repository and ensure all dependencies are installed.
+
+2. **Run the Bot**:
+   Execute the bot via the CLI:
+   ```bash
+   python cli.py -u <username> -p <password>
+
+---
+
+## Fan art
+This is some fan art made by Buffon, a contributor of this project:
+
+![Image generated by Buffon](buffon.jpg)
+
+Another one:
+
+![Image generated by Buffon](buffon2.jpg)
+
+Third one, the legendary bot war of 2024 ended in peace. 
+
+![Image generated by Buffon](buffon3.jpg)
 
-If there is a new rant: 
-1. check if user has more than five posts. If so, it will not be seen as spam.
-2. it will check certain keywords like hacker / money crypto related if so continue to step 3.
-3. user will be informed by the bots that his rant is flagged and what to do about it. 
-4. rant will be downvoted by the four bots making it disappear. 
 

app.log

@@ -0,0 +1,5 @@
+2024-12-17 13:06:42,285 - __main__ - DEBUG - This is a debug message
+2024-12-17 13:06:42,286 - __main__ - INFO - This is an info message
+2024-12-17 13:06:42,286 - __main__ - WARNING - This is a warning message
+2024-12-17 13:06:42,286 - __main__ - ERROR - This is an error message
+2024-12-17 13:06:42,286 - __main__ - CRITICAL - This is a critical message

logtest.py

@@ -0,0 +1,29 @@
+import logging
+
+# Create or get the root logger
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.DEBUG)  # Set the logging level
+
+# Create a file handler
+file_handler = logging.FileHandler("app.log")
+file_handler.setLevel(logging.DEBUG)  # Set the level for this handler
+
+# Create a console handler (optional, for logging to the console too)
+console_handler = logging.StreamHandler()
+console_handler.setLevel(logging.INFO)
+
+# Create a logging format
+formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
+file_handler.setFormatter(formatter)
+console_handler.setFormatter(formatter)
+
+# Add the handlers to the logger
+logger.addHandler(file_handler)
+logger.addHandler(console_handler)
+
+# Log some messages
+logger.debug("This is a debug message")
+logger.info("This is an info message")
+logger.warning("This is a warning message")
+logger.error("This is an error message")
+logger.critical("This is a critical message")

src/Ragnar.egg-info/PKG-INFO

@@ -1,4 +1,4 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: Ragnar
 Version: 1.3.37
 Summary: Anti spam bot for dR
@@ -9,31 +9,90 @@ Requires-Python: >=3.7
 Description-Content-Type: text/markdown
 Requires-Dist: requests==2.32.3
 
-# Ragnar
+# Ragnar Bot: Automated Moderation System for Devrant.io
 
-This is an anti spam bot network. It is named after the viking for no obvious reason. 
+## Overview
+Ragnar Bot is an automated system designed to interact with the **Devrant.io API** for community moderation. It detects spam, flags inappropriate content, and takes corrective actions such as downvoting and commenting. The system is scalable, customizable, and supports concurrent operations for multiple users.
 
-I'm not happy about the quality of the source and it is not a representation of my usual work. If I would've spend more efford there would be some types and I've would use aiohttp and would've used context managers for example. Despite the source lacking a certain quality, the bots work great and are made not to be annoying to the server by not connecting all at once and caching certain things like user profile / user id and if a reand already is flaged for example to not annoy the server. 
+---
 
-The bots have user name no-spam[1-4] but flag under a Russian girl name, also for no obvious reason. I liked it more than some technical name. Will probably rename the bots later. Could be that devRants prevents me to do that within a half year. It doesn't matter much, if the bots do a good job, we will barely see them. 
+## Features
 
-I expect this project tomorrow to have deployed fully functional on a server.
+### API Integration
+- **User Authentication**: Secure login with credentials.
+- **Profile Retrieval**: Access user profile data.
+- **Content Search**: Find posts matching specific keywords or patterns.
+- **Comments Management**: Post and fetch comments on rants.
+- **Voting System**: Upvote or downvote rants programmatically.
+- **Fetch Rants**: Retrieve and sort recent rants for analysis.
 
-## In progress
+### Spam Detection
+- **Keyword Triggers**: Detect spam using predefined keywords like `crypto`, `bitcoin`, and more.
+- **Regex Matching**: Identify suspicious patterns in text, such as URLs or phone numbers.
+- **Heuristic Checks**: Evaluate user profiles and comments for unusual behavior.
 
-The bots work perfect in sense that they're doing what they're programmed to do.
-But the programming is not finished yet:
- - the criteria can be better, tips how to optimize are very welcome.
- - at this moment, they can only flag, useless, but we will have indication of future content to be cancelled. Every spam message should have a flag. If not, contact @retoor. 
- - the downvote function doesn't work because I couldn't figure out what value I had to post. Who knows it? After this, it's kinda done.
- - a decent deployment on my server. Now it runs on my laptop because it's not done yet and it got late.
+### Automation
+- **Flagging Spam**: Automatically downvote and comment on posts flagged as spam.
+- **Concurrent Execution**: Operates multiple bots simultaneously using multithreading.
+- **Customizable Triggers**: Adjust detection patterns to fit evolving spam trends.
 
-## How they work
-One process starts four bots named no-spam[1-4]. These bots look at new rants.
+### Logging & Caching
+- **Logging**: Outputs activity logs to both the console and a file for debugging and monitoring.
+- **Caching**: Uses a custom caching system to optimize performance and reduce redundant operations.
+
+---
+
+## How It Works
+
+### Components
+1. **API**:
+   Handles all interactions with the Devrant.io platform, including user authentication, content retrieval, and posting actions.
+   
+2. **Bot**:
+   Implements moderation logic:
+   - Identifies suspicious content.
+   - Posts comments warning users about flagged content.
+   - Maintains a history of checked rants to avoid redundant processing.
+
+3. **CLI**:
+   Command-line interface to configure and execute the bot. Accepts user credentials and runs bots for multiple users simultaneously.
+
+4. **Victoria Integration**:
+   Fetches user-related data from a remote service (`victoria.molodetz.nl`) for enhanced operations.
+
+### Workflow
+1. The bot authenticates using the provided username and password.
+2. It retrieves and evaluates recent rants for spam indicators.
+3. If spam is detected:
+   - Posts a predefined warning comment.
+   - Downvotes the rant.
+4. Maintains logs and runs continuously to monitor new rants.
+
+---
+
+## Installation & Usage
+
+1. **Setup**:
+   Clone the repository and ensure all dependencies are installed.
+
+2. **Run the Bot**:
+   Execute the bot via the CLI:
+   ```bash
+   python cli.py -u <username> -p <password>
+
+---
+
+## Fan art
+This is some fan art made by Buffon, a contributor of this project:
+
+![Image generated by Buffon](buffon.jpg)
+
+Another one:
+
+![Image generated by Buffon](buffon2.jpg)
+
+Third one, the legendary bot war of 2024 ended in peace. 
+
+![Image generated by Buffon](buffon3.jpg)
 
-If there is a new rant: 
-1. check if user has more than five posts. If so, it will not be seen as spam.
-2. it will check certain keywords like hacker / money crypto related if so continue to step 3.
-3. user will be informed by the bots that his rant is flagged and what to do about it. 
-4. rant will be downvoted by the four bots making it disappear. 
 

src/Ragnar.egg-info/SOURCES.txt

@@ -13,5 +13,6 @@ src/ragnar/api.py
 src/ragnar/bot.py
 src/ragnar/cache.py
 src/ragnar/cli.py
+src/ragnar/victoria.py
 src/ragnar/tests/__init__.py
 src/ragnar/tests/bot.py

src/ragnar/__init__.py

@@ -1,12 +1,13 @@
 import logging
 import sys
 
+file_handler = logging.FileHandler("ragnar.log")
+file_handler.setLevel(logging.DEBUG)
+
 logging.basicConfig(
     level=logging.DEBUG,
     format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
-    handlers=[
-        logging.StreamHandler(sys.stdout),
-    ],
+    handlers=[logging.StreamHandler(sys.stdout), file_handler],
 )
 
 log = logging.getLogger(__name__)

src/ragnar/bot.py

@@ -1,22 +1,23 @@
-import json
-import random
 import re
 import time
 
 from ragnar import log
 from ragnar.api import Api
 from ragnar.cache import method_cache
+from ragnar.victoria import vic
 
 
 class Bot:
 
+    rant_history = []
+
     def __init__(self, username, password):
         self.username = username
         self.password = password
         self.name = self.username.split("@")[0]
-        self.rant_history = []
+
         self.amount_of_rants_to_check = 30
-        self.sus_users = ["Buffon"]
+        self.sus_users = vic.get_spammers()
         self.names = {
             "no-spam": "Anna",
             "no-spam1": "Ira",
@@ -25,7 +26,7 @@ class Bot:
             "no-spam4": "Vira",
         }
         self.name = self.names.get(self.name, self.username.split("@")[0])
-        self.mark_text = f"I am {self.name} and downvoted this post because post is considered spam. Your message will be removed from this community site due too much downvotes. See my profile for more information. Read my source code mentioned on my profile to see what you did wrong. Should be no problem for a developer.\n\nHave a nice day!\n\n\nIf the post is not spam, please mention @retoor in the comments of this rant.\n\n@ other ranters downvoting is not needed anymore. Enough bots. It will be deleted automatically, but can take a while. I configured it to eat less resources of server. Still it's important to mention ragnar if a spam falls trough so it's in the administration."
+        self.mark_text = f"I am {self.name} and downvoted this post because I consider it spam. Your message will be removed from this community site due too much downvotes. See my profile for more information. Read my source code mentioned on my profile to see what you did wrong. Should be no problem for a developer.\n\nHave a nice day!\n\n\nIf your post is not spam, please mention @retoor."
         self.auth = None
         self.triggers = [
             "$",
@@ -45,7 +46,10 @@ class Bot:
             "carat",
             "carats",
             "leading",
-            "helping",
+            "cheating",
+            "spouse",
+            "spy",
+            # "helping",
             "unqiue",
             "@ragnar",
             "brides",
@@ -56,7 +60,7 @@ class Bot:
         self.api = Api(username=self.username, password=self.password)
 
     def rsleepii(self):
-        time.sleep(random.randint(1, 3))
+        time.sleep(1)
 
     @method_cache
     def login(self):
@@ -70,6 +74,21 @@ class Bot:
     def clean_rant_text(self, rant_text):
         return rant_text.replace(" ", "").lower()
 
+    def get_mentions(self, rant_text):
+        return re.findall(r"@\w+", rant_text)
+
+    def is_comment_deletable(self, rant_id, username):
+        rant = self.api.get_rant(rant_id)
+        for comment in rant.get("comments", []):
+            print("Checking if sus comment: ", comment["body"])
+            if (
+                comment.get("user_username") == username
+                and comment.get("user_score") <= 10
+            ):
+                vic.register_spammer(comment["user_username"])
+                return True
+        return False
+
     @method_cache
     def is_sus_content(self, content):
         clean_text = self.clean_rant_text(content)
@@ -81,6 +100,25 @@ class Bot:
                         log.info(f"Regex trigger {regex} matched!")
                         return True
             elif trigger in clean_text:
+                if trigger == "ragnar":
+                    mentions = self.get_mentions(content)
+                    for mention in mentions:
+                        if mention == "@ragnar":
+                            continue
+                        username = mention.strip("@")
+                        if self.is_comment_deletable(username):
+                            vic.register_spammer(mention.strip("@"))
+                            self.api.post_comment(
+                                rant_id,
+                                f"User {username} is sucessfully registered as spammer.",
+                            )
+                        else:
+                            self.api.post_comment(
+                                rant_id,
+                                f"Can't register user {username} as spammer. User is trusted.",
+                            )
+                    if len(mentions) > 1:
+                        return False
                 log.info(f"Trigger {trigger} matched!")
                 return True
         return False
@@ -114,16 +152,30 @@ class Bot:
 
     def is_comments_sus(self, rant_id):
         log.info(f"Checking if comments are sus of rant {rant_id}.")
+        self.rsleepii()
         rant = self.api.get_rant(rant_id)
         for comment in rant.get("comments", []):
+            for spammer in self.sus_users:
+                if comment["user_username"] == spammer:
+                    vic.downvote_comment(comment["id"])
+                    return False
+
             print("Checking if sus comment: ", comment["body"])
+            if "@ragnar" in comment.get("body", "").lower():
+                print("Ragnar is mentioned, so flagging as sus comment.")
+                return True
+            if comment["user_score"] >= 5:
+                print("User has reputation >= 5 so not sus.")
+                continue
             if self.is_sus_content(comment.get("body", "")):
                 return True
         return False
 
-    def mark_as_sus(self, rant):
-        self.rsleepii()
-        self.api.post_comment(rant["id"], self.mark_text)
+    def mark_as_sus(self, rant, is_flagged_by_ai):
+        mark_text = self.mark_text
+        if is_flagged_by_ai:
+            mark_text += "\n* Flagged by AI."
+        self.api.post_comment(rant["id"], mark_text)
 
     def fight(self):
         self.rsleepii()
@@ -132,7 +184,7 @@ class Bot:
             if rant["id"] in self.rant_history:
                 log.debug("{}: Already checked rant {}.".format(self.name, rant["id"]))
                 continue
-
+            flagged_by_id = False
             if not self.is_user_sus(rant["user_username"]):
                 log.info(
                     "{}: User {} is trusted.".format(self.name, rant["user_username"])
@@ -145,17 +197,24 @@ class Bot:
             elif rant["user_username"] in self.sus_users:
                 pass
             elif not self.is_sus_rant(rant["id"], rant["text"]):
+                if not self.is_user_sus(rant["user_username"]):
+                    continue
                 log.info(
-                    "{}: Rant by {} is not sus.".format(
+                    "{}: Rant by {} is not sus by traditional method..".format(
                         self.name, rant["user_username"]
                     )
                 )
-                continue
-            if self.is_flagged_as_sus(rant["id"], rant.get("num_comments")):
-                self.rant_history.append(rant["id"])
-                continue
+                if vic.is_spam(rant["text"]) <= 5:
+                    self.rant_history.append(rant["id"])
+                    continue
+                else:
+                    log.info(
+                        "{}: Rant by {} is sus according to AI.".format(
+                            self.name, rant["user_username"]
+                        )
+                    )
             log.warning(
-                "{}: Rant is not {} flagged as sus yet.".format(
+                "{}: Rant by {} is not flagged as sus yet but should be.".format(
                     self.name, rant["user_username"]
                 )
             )
@@ -164,10 +223,11 @@ class Bot:
                     self.name, rant["user_username"]
                 )
             )
-            self.mark_as_sus(rant)
+            self.mark_as_sus(rant, flagged_by_id)
             self.down_vote_rant(rant)
             self.rant_history.append(rant["id"])
 
     def down_vote_rant(self, rant):
-        log.warning("Downvoting rant by {}.".format(rant["user_username"]))
-        log.debug(json.loads(self.api.post_rant_vote(rant["id"], -1)))
+        vic.dr_downvote_rant(rant["id"], 5)
+        log.warning("Downvoting rant by {} for 5 times.".format(rant["user_username"]))
+        log.debug("Vote result: ".format(self.api.post_rant_vote(rant["id"], -1)))

src/ragnar/cli.py

@@ -1,7 +1,5 @@
 import argparse
-import random
 import time
-from concurrent.futures import ThreadPoolExecutor as Executor
 
 from ragnar import log
 from ragnar.bot import Bot
@@ -16,41 +14,23 @@ def parse_args():
     return parser.parse_args()
 
 
-def bot_task(username, password):
-    log.info(f"Created new bot runniner. Username: {username}")
-    time.sleep(random.randint(1, 20))
-    bot = Bot(username=username, password=password)
-    bot.login()
-    while True:
-        time.sleep(random.randint(1, 20))
-        try:
-            bot.fight()
-        except Exception as ex:
-            print(ex)
-
-
 def main():
     args = parse_args()
-    with Executor(500) as executor:
-        usernames = [
-            # "no-spam1",
-            # "no-spam2",
-            # "no-spam3",
-            # "no-spam4",
-            # "no-spam",
-            "no-spam2353",
-            "no-spam2351",
-            "no-spam2350",
-            "no-spam2349",
-            "JamesMedina",
-            "MichelleWeeks",
-            "JaredRuiz",
-            "LoriMcgee",
-            "AmandaFloyd",
-        ]
+    usernames = ["no-spam1", "no-spam2", "no-spam3", "no-spam4"]
+
+    while True:
         for username in usernames:
-            executor.submit(bot_task, f"{username}@molodetz.nl", args.password)
-        executor.shutdown(wait=True)
+            time_start = time.time()
+            try:
+                log.info(f"Created new bot runner. Username: {username}")
+                bot = Bot(username=username, password=args.password)
+                bot.login()
+                bot.fight()
+            except Exception as ex:
+                log.critical(ex, exc_info=True)
+                time.sleep(1)
+            time_duration = time.time() - time_start
+            log.info(f"Bot {username} finished in {time_duration} seconds.")
 
 
 def run():

src/ragnar/victoria.py

@@ -0,0 +1,3 @@
+from xmlrpc.client import ServerProxy
+
+vic = ServerProxy("https://victoria.molodetz.nl/rpc")

sudo

@@ -0,0 +1 @@
+aaltink*132