Merged master into autodetecttelegram

* added a bit of documentation on how to run tests

* formatting

* formatting

* formatting

* formatting

* formatting

* formatting

.github/workflows/tests.yml

@@ -21,15 +21,15 @@ jobs:
 
     name: Python ${{ matrix.python-version }}
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Setup Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
 
       - name: Cached PIP dependencies
-        uses: actions/cache@v3
+        uses: actions/cache@v4
         with:
           path: |
             ~/.cache/pip
@@ -44,4 +44,4 @@ jobs:
         run: tox
 
       - name: Code coverage upload
-        uses: codecov/codecov-action@v3
+        uses: codecov/codecov-action@v4

CHANGELOG.rst

@@ -1,30 +1,39 @@
 Change Log
 ----------
 
-**1.4.0** (2024-03-12)
-- Q3D add CURRENT_ELECTRICITY_DELIVERY
- (`PR #149 <https://github.com/ndokter/dsmr_parser/pull/149>`_ by `Aeroid <https://github.com/Aeroid>`_)
-- Copy head_parsers list on construct.
- (`PR #150 <https://github.com/ndokter/dsmr_parser/pull/150>`_ by `dupondje <https://github.com/dupondje>`_)
+**1.5.0** (2024-08-25)
+- Allow the telegram specification to optionally be autodetected (`PR #87 <https://github.com/ndokter/dsmr_parser/pull/87>`_ by `clonerswords <https://github.com/clonerswords>`_)
 
+**1.4.2** (2024-07-14)
+
+- Bump Github Actions to latest versions in favor of Node deprecations (`PR #159 <https://github.com/ndokter/dsmr_parser/pull/159>`_ by `dennissiemensma <https://github.com/dennissiemensma>`_)
+- Swap pyserial-asyncio for pyserial-asyncio-fast (`PR #158 <https://github.com/ndokter/dsmr_parser/pull/158>`_ by `bdraco <https://github.com/bdraco>`_)
+
+**1.4.1** (2024-06-04)
+
+- Avoid loading timezone at runtime (`PR #157 <https://github.com/ndokter/dsmr_parser/pull/157>`_ by `elupus <https://github.com/elupus>`_)
+
+**1.4.0** (2024-03-12)
+
+- Mbus alt (`PR #142 <https://github.com/ndokter/dsmr_parser/pull/142>`_ by `dupondje <https://github.com/dupondje>`_)
+- Q3D add CURRENT_ELECTRICITY_DELIVERY (`PR #149 <https://github.com/ndokter/dsmr_parser/pull/149>`_ by `Aeroid <https://github.com/Aeroid>`_)
+- Copy head_parsers list on construct. (`PR #150 <https://github.com/ndokter/dsmr_parser/pull/150>`_ by `dupondje <https://github.com/dupondje>`_)
 
 **1.3.2** (2024-01-29)
-- Fix unit test for pyton 3.12
- (`PR #148 <https://github.com/ndokter/dsmr_parser/pull/148>`_ by `ndokter <https://github.com/ndokter>`_)
+
+- Fix unit test for pyton 3.12 (`PR #148 <https://github.com/ndokter/dsmr_parser/pull/148>`_ by `ndokter <https://github.com/ndokter>`_)
 
 **1.3.1** (2023-11-06)
-- Fix parsing peak usage with invalid timestamps
- (`PR #143 <https://github.com/ndokter/dsmr_parser/pull/143>`_ by `dupondje <https://github.com/dupondje>`_)
+
+- Fix parsing peak usage with invalid timestamps (`PR #143 <https://github.com/ndokter/dsmr_parser/pull/143>`_ by `dupondje <https://github.com/dupondje>`_)
 
 **1.3.0** (2023-08-01)
 
-- added E.ON Hungary; refactored DSMR specifications to fix obis reference conflicts
- (`PR #137 <https://github.com/ndokter/dsmr_parser/pull/137>`_ by `balazs92117 <https://github.com/balazs92117>`_)
+- added E.ON Hungary; refactored DSMR specifications to fix obis reference conflicts (`PR #137 <https://github.com/ndokter/dsmr_parser/pull/137>`_ by `balazs92117 <https://github.com/balazs92117>`_)
 
 **1.2.4** (2023-07-11)
 
-- EQUIPMENT IDENTIFIER is wrong for Fluvius meters when other mbus devices are present
- (`PR #133 <https://github.com/ndokter/dsmr_parser/pull/133>`_ by `ejpalacios <https://github.com/ejpalacios>`_)
+- EQUIPMENT IDENTIFIER is wrong for Fluvius meters when other mbus devices are present (`PR #133 <https://github.com/ndokter/dsmr_parser/pull/133>`_ by `ejpalacios <https://github.com/ejpalacios>`_)
 
 
 **1.2.3** (2023-04-18)
@@ -161,7 +170,7 @@ Remove deprecated asyncio coroutine decorator (`PR #76 <https://github.com/ndokt
 **0.12** (2018-09-23)
 
 - Add serial settings for DSMR v5.0 (`PR #31 <https://github.com/ndokter/dsmr_parser/pull/31>`_).
-- Lux-creos-obis-1.8.0 (`PR #32 <https://github.com/ndokter/dsmr_parser/pull/32>`_). 
+- Lux-creos-obis-1.8.0 (`PR #32 <https://github.com/ndokter/dsmr_parser/pull/32>`_).
 
 **0.11** (2017-09-18)
 

README.rst

@@ -293,6 +293,36 @@ To install DSMR Parser:
 
     $ pip install dsmr-parser
 
+Development
+-----------
+
+Create a virtualenv and activate it followed by the installation of the dsmr-parser:
+
+.. code-block:: bash
+
+    python3 -m venv venv
+    source venv/bin/activate
+    pip install -e .
+
+Install tox and run it:
+
+.. code-block:: bash
+
+    pip install tox
+    tox
+
+You should see that the tests have succeeded:
+
+.. code-block:: text
+
+    ======================================================================================================== 59 passed in 0.91s ========================================================================================================
+    py: commands[1]> pylama dsmr_parser test
+      py: OK (11.55=setup[9.73]+cmd[1.29,0.53] seconds)
+      congratulations :) (11.69 seconds)
+
+
+Now you can make changes by editing the code and rerunning tox to verify your changes.
+
 Known issues
 ------------
 

dsmr_parser/clients/filereader.py

@@ -73,7 +73,7 @@ class FileReader(object):
                     try:
                         yield self.telegram_parser.parse(telegram)
                     except InvalidChecksumError as e:
-                        logger.warning(str(e))
+                        logger.info(str(e))
                     except ParseError as e:
                         logger.error('Failed to parse telegram: %s', e)
 

dsmr_parser/clients/protocol.py

@@ -4,7 +4,7 @@ from functools import partial
 import asyncio
 import logging
 
-from serial_asyncio import create_serial_connection
+from serial_asyncio_fast import create_serial_connection
 
 from dsmr_parser import telegram_specifications
 from dsmr_parser.clients.telegram_buffer import TelegramBuffer
@@ -158,7 +158,7 @@ class DSMRProtocol(asyncio.Protocol):
         try:
             parsed_telegram = self.telegram_parser.parse(telegram)
         except InvalidChecksumError as e:
-            self.log.warning(str(e))
+            self.log.info(str(e))
         except ParseError:
             self.log.exception("failed to parse telegram")
         else:

dsmr_parser/clients/rfxtrx_protocol.py

@@ -2,7 +2,7 @@
 
 import asyncio
 
-from serial_asyncio import create_serial_connection
+from serial_asyncio_fast import create_serial_connection
 from .protocol import DSMRProtocol, _create_dsmr_protocol
 
 

dsmr_parser/clients/serial_.py

@@ -1,6 +1,6 @@
 import logging
 import serial
-import serial_asyncio
+import serial_asyncio_fast
 
 from dsmr_parser.clients.telegram_buffer import TelegramBuffer
 from dsmr_parser.exceptions import ParseError, InvalidChecksumError
@@ -37,7 +37,7 @@ class SerialReader(object):
                     try:
                         yield self.telegram_parser.parse(telegram)
                     except InvalidChecksumError as e:
-                        logger.warning(str(e))
+                        logger.info(str(e))
                     except ParseError as e:
                         logger.error('Failed to parse telegram: %s', e)
 
@@ -77,7 +77,7 @@ class AsyncSerialReader(SerialReader):
         :rtype: None
         """
         # create Serial StreamReader
-        conn = serial_asyncio.open_serial_connection(**self.serial_settings)
+        conn = serial_asyncio_fast.open_serial_connection(**self.serial_settings)
         reader, _ = await conn
 
         while True:
@@ -107,7 +107,7 @@ class AsyncSerialReader(SerialReader):
         """
 
         # create Serial StreamReader
-        conn = serial_asyncio.open_serial_connection(**self.serial_settings)
+        conn = serial_asyncio_fast.open_serial_connection(**self.serial_settings)
         reader, _ = await conn
 
         while True:

dsmr_parser/clients/socket_.py

@@ -31,11 +31,15 @@ class SocketReader(object):
         buffer = b""
 
         with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as socket_handle:
-
+            socket_handle.settimeout(60)
             socket_handle.connect((self.host, self.port))
 
             while True:
-                buffer += socket_handle.recv(self.BUFFER_SIZE)
+                try:
+                    buffer += socket_handle.recv(self.BUFFER_SIZE)
+                except socket.timeout:
+                    logger.error("Socket timeout occurred, exiting")
+                    break
 
                 lines = buffer.splitlines(keepends=True)
 
@@ -54,7 +58,7 @@ class SocketReader(object):
                     try:
                         yield self.telegram_parser.parse(telegram)
                     except InvalidChecksumError as e:
-                        logger.warning(str(e))
+                        logger.info(str(e))
                     except ParseError as e:
                         logger.error('Failed to parse telegram: %s', e)
 

dsmr_parser/exceptions.py

@@ -4,3 +4,7 @@ class ParseError(Exception):
 
 class InvalidChecksumError(ParseError):
     pass
+
+
+class TelegramSpecificationMatchError(ParseError):
+    pass

dsmr_parser/parsers.py

@@ -9,7 +9,7 @@ from dlms_cosem.connection import XDlmsApduFactory
 from dlms_cosem.protocol.xdlms import GeneralGlobalCipher
 
 from dsmr_parser.objects import MBusObject, MBusObjectPeak, CosemObject, ProfileGenericObject, Telegram
-from dsmr_parser.exceptions import ParseError, InvalidChecksumError
+from dsmr_parser.exceptions import ParseError, InvalidChecksumError, TelegramSpecificationMatchError
 from dsmr_parser.value_types import timestamp
 
 logger = logging.getLogger(__name__)
@@ -18,20 +18,17 @@ logger = logging.getLogger(__name__)
 class TelegramParser(object):
     crc16_tab = []
 
-    def __init__(self, telegram_specification, apply_checksum_validation=True):
+    def __init__(self, telegram_specification=None, apply_checksum_validation=True):
         """
-        :param telegram_specification: determines how the telegram is parsed
+        :param telegram_specification: determines how the telegram is parsed.
+            Will attempt to autodetect if omitted.
         :param apply_checksum_validation: validate checksum if applicable for
             telegram DSMR version (v4 and up).
         :type telegram_specification: dict
         """
         self.apply_checksum_validation = apply_checksum_validation
         self.telegram_specification = telegram_specification
-        # Regexes are compiled once to improve performance
-        self.telegram_specification_regexes = {
-            object["obis_reference"]: re.compile(object["obis_reference"], re.DOTALL | re.MULTILINE)
-            for object in self.telegram_specification['objects']
-        }
+        self._telegram_specification_regex = None
 
     def parse(self, telegram_data, encryption_key="", authentication_key="", throw_ex=False):  # noqa: C901
         """
@@ -46,6 +43,15 @@ class TelegramParser(object):
         :raises ParseError:
         :raises InvalidChecksumError:
         """
+        if not self.telegram_specification:
+            self.telegram_specification = \
+                match_telegram_specification(telegram_data)
+        if not self._telegram_specification_regex:
+            # Regexes are compiled once to improve performance
+            self._telegram_specification_regexes = {
+                object["obis_reference"]: re.compile(object["obis_reference"], re.DOTALL | re.MULTILINE)
+                for object in self.telegram_specification['objects']
+            }
 
         if "general_global_cipher" in self.telegram_specification:
             if self.telegram_specification["general_global_cipher"]:
@@ -85,7 +91,7 @@ class TelegramParser(object):
         telegram = Telegram()
 
         for object in self.telegram_specification['objects']:
-            pattern = self.telegram_specification_regexes[object["obis_reference"]]
+            pattern = self._telegram_specification_regexes[object["obis_reference"]]
             matches = pattern.findall(telegram_data)
 
             # Some signatures are optional and may not be present,
@@ -173,6 +179,37 @@ class TelegramParser(object):
         return crcValue
 
 
+def match_telegram_specification(telegram_data):
+    """
+    Find telegram specification that matches the telegram data by trying all
+    specifications.
+
+    Could be further optimized to check the actual 0.2.8 OBIS reference which
+    is available for DSMR version 4 and up.
+
+    :param str telegram_data: full telegram from start ('/') to checksum
+            ('!ABCD') including line endings in between the telegram's lines
+    :return: telegram specification
+    :rtype: dict
+    """
+    # Prevent circular import
+    from dsmr_parser import telegram_specifications
+
+    for specification in telegram_specifications.ALL:
+        try:
+            TelegramParser(specification).parse(telegram_data)
+        except ParseError:
+            pass
+        else:
+            return specification
+
+    raise TelegramSpecificationMatchError(
+        'Could automatically match telegram specification. Make sure the data'
+        'is not corrupt. Alternatively manually specify one.'
+    )
+
+
+
 class DSMRObjectParser(object):
     """
     Parses an object (can also be see as a 'line') from a telegram.

dsmr_parser/telegram_specifications.py

@@ -486,8 +486,6 @@ V5 = {
     ]
 }
 
-ALL = (V2_2, V3, V4, V5)
-
 BELGIUM_FLUVIUS = {
     'checksum_support': True,
     'objects': [
@@ -1368,3 +1366,17 @@ EON_HUNGARY = {
         }
     ]
 }
+
+ALL = (
+    V2_2,
+    V3,
+    V4,
+    V5,
+    BELGIUM_FLUVIUS,
+    LUXEMBOURG_SMARTY,
+    SWEDEN,
+    SAGEMCOM_T210_D_R,
+    AUSTRIA_ENERGIENETZE_STEIERMARK,
+    ISKRA_IE,
+    EON_HUNGARY
+)

dsmr_parser/value_types.py

@@ -2,6 +2,10 @@ import datetime
 
 import pytz
 
+# TODO : Use system timezone
+# Preload timezone to avoid loading in event loop later
+local_tz = pytz.timezone('Europe/Amsterdam')
+
 
 def timestamp(value):
     try:
@@ -20,8 +24,6 @@ def timestamp(value):
     else:
         is_dst = False
 
-    # TODO : Use system timezone
-    local_tz = pytz.timezone('Europe/Amsterdam')
     localized_datetime = local_tz.localize(naive_datetime, is_dst=is_dst)
 
     return localized_datetime.astimezone(pytz.utc)

setup.py

@@ -7,11 +7,11 @@ setup(
     author_email='mail@nldr.net',
     license='MIT',
     url='https://github.com/ndokter/dsmr_parser',
-    version='1.4.0',
+    version='1.5.0',
     packages=find_packages(exclude=('test', 'test.*')),
     install_requires=[
         'pyserial>=3,<4',
-        'pyserial-asyncio<1',
+        'pyserial-asyncio-fast>=0.11',
         'pytz',
         'Tailer==0.4.1',
         'dlms_cosem==21.3.2'

test/test_match_telegram_specification.py

@@ -0,0 +1,29 @@
+import unittest
+
+from dsmr_parser.exceptions import TelegramSpecificationMatchError
+from dsmr_parser.parsers import match_telegram_specification
+from dsmr_parser import telegram_specifications
+from test import example_telegrams
+
+
+class MatchTelegramSpecificationTest(unittest.TestCase):
+
+    def test_v2_2(self):
+        assert match_telegram_specification(example_telegrams.TELEGRAM_V2_2) \
+           == telegram_specifications.V2_2
+
+    def test_v3(self):
+        assert match_telegram_specification(example_telegrams.TELEGRAM_V3) \
+           == telegram_specifications.V3
+
+    def test_v4_2(self):
+        assert match_telegram_specification(example_telegrams.TELEGRAM_V4_2) \
+           == telegram_specifications.V4
+
+    def test_v5(self):
+        assert match_telegram_specification(example_telegrams.TELEGRAM_V5) \
+           == telegram_specifications.V5
+
+    def test_malformed_telegram(self):
+        with self.assertRaises(TelegramSpecificationMatchError):
+            match_telegram_specification(example_telegrams.TELEGRAM_V5[:-4])

test/test_parse_v2_2.py

@@ -12,6 +12,15 @@ from test.example_telegrams import TELEGRAM_V2_2
 class TelegramParserV2_2Test(unittest.TestCase):
     """ Test parsing of a DSMR v2.2 telegram. """
 
+    def test_telegram_specification_matching(self):
+        parser = TelegramParser()
+        parser.parse(TELEGRAM_V2_2)
+
+        self.assertEqual(
+            parser.telegram_specification,
+            telegram_specifications.V2_2
+        )
+
     def test_parse(self):
         parser = TelegramParser(telegram_specifications.V2_2)
         try:

test/test_parse_v3.py

@@ -12,6 +12,15 @@ from test.example_telegrams import TELEGRAM_V3
 class TelegramParserV3Test(unittest.TestCase):
     """ Test parsing of a DSMR v3 telegram. """
 
+    def test_telegram_specification_matching(self):
+        parser = TelegramParser()
+        parser.parse(TELEGRAM_V3)
+
+        self.assertEqual(
+            parser.telegram_specification,
+            telegram_specifications.V3
+        )
+
     def test_parse(self):
         parser = TelegramParser(telegram_specifications.V3)
         try:

test/test_parse_v4_2.py

@@ -15,6 +15,15 @@ from test.example_telegrams import TELEGRAM_V4_2
 class TelegramParserV4_2Test(unittest.TestCase):
     """ Test parsing of a DSMR v4.2 telegram. """
 
+    def test_telegram_specification_matching(self):
+        parser = TelegramParser()
+        parser.parse(TELEGRAM_V4_2)
+
+        self.assertEqual(
+            parser.telegram_specification,
+            telegram_specifications.V4
+        )
+
     def test_parse(self):
         parser = TelegramParser(telegram_specifications.V4)
         try:

test/test_parse_v5.py

@@ -15,6 +15,15 @@ from test.example_telegrams import TELEGRAM_V5
 class TelegramParserV5Test(unittest.TestCase):
     """ Test parsing of a DSMR v5.x telegram. """
 
+    def test_telegram_specification_matching(self):
+        parser = TelegramParser()
+        parser.parse(TELEGRAM_V5)
+
+        self.assertEqual(
+            parser.telegram_specification,
+            telegram_specifications.V5
+        )
+
     def test_parse(self):
         parser = TelegramParser(telegram_specifications.V5)
         try:

tox.ini

@@ -14,6 +14,9 @@ commands=
 [pylama:dsmr_parser/clients/__init__.py]
 ignore = W0611
 
+[pylama:dsmr_parser/clients/socket_.py]
+ignore = C901
+
 [pylama:dsmr_parser/parsers.py]
 ignore = W605
 
@@ -24,4 +27,4 @@ ignore = E501
 max_line_length = 120
 
 [pylama:pycodestyle]
-max_line_length = 120
+max_line_length = 120