moved decryption logic and simplified it a bit

* added a bit of documentation on how to run tests

* formatting

* formatting

* formatting

* formatting

* formatting

* formatting

.github/workflows/tests.yml

@@ -21,15 +21,15 @@ jobs:
 
     name: Python ${{ matrix.python-version }}
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Setup Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
 
       - name: Cached PIP dependencies
-        uses: actions/cache@v3
+        uses: actions/cache@v4
         with:
           path: |
             ~/.cache/pip
@@ -44,4 +44,4 @@ jobs:
         run: tox
 
       - name: Code coverage upload
-        uses: codecov/codecov-action@v3
+        uses: codecov/codecov-action@v4

.gitignore

@@ -3,6 +3,7 @@
 .tox
 .cache
 .venv
+.history
 *.egg-info
 /.project
 /.pydevproject

CHANGELOG.rst

@@ -1,30 +1,36 @@
 Change Log
 ----------
 
-**1.4.0** (2024-03-12)
-- Q3D add CURRENT_ELECTRICITY_DELIVERY
- (`PR #149 <https://github.com/ndokter/dsmr_parser/pull/149>`_ by `Aeroid <https://github.com/Aeroid>`_)
-- Copy head_parsers list on construct.
- (`PR #150 <https://github.com/ndokter/dsmr_parser/pull/150>`_ by `dupondje <https://github.com/dupondje>`_)
+**1.4.2** (2024-07-14)
 
+- Bump Github Actions to latest versions in favor of Node deprecations (`PR #159 <https://github.com/ndokter/dsmr_parser/pull/159>`_ by `dennissiemensma <https://github.com/dennissiemensma>`_)
+- Swap pyserial-asyncio for pyserial-asyncio-fast (`PR #158 <https://github.com/ndokter/dsmr_parser/pull/158>`_ by `bdraco <https://github.com/bdraco>`_)
+
+**1.4.1** (2024-06-04)
+
+- Avoid loading timezone at runtime (`PR #157 <https://github.com/ndokter/dsmr_parser/pull/157>`_ by `elupus <https://github.com/elupus>`_)
+
+**1.4.0** (2024-03-12)
+
+- Mbus alt (`PR #142 <https://github.com/ndokter/dsmr_parser/pull/142>`_ by `dupondje <https://github.com/dupondje>`_)
+- Q3D add CURRENT_ELECTRICITY_DELIVERY (`PR #149 <https://github.com/ndokter/dsmr_parser/pull/149>`_ by `Aeroid <https://github.com/Aeroid>`_)
+- Copy head_parsers list on construct. (`PR #150 <https://github.com/ndokter/dsmr_parser/pull/150>`_ by `dupondje <https://github.com/dupondje>`_)
 
 **1.3.2** (2024-01-29)
-- Fix unit test for pyton 3.12
- (`PR #148 <https://github.com/ndokter/dsmr_parser/pull/148>`_ by `ndokter <https://github.com/ndokter>`_)
+
+- Fix unit test for pyton 3.12 (`PR #148 <https://github.com/ndokter/dsmr_parser/pull/148>`_ by `ndokter <https://github.com/ndokter>`_)
 
 **1.3.1** (2023-11-06)
-- Fix parsing peak usage with invalid timestamps
- (`PR #143 <https://github.com/ndokter/dsmr_parser/pull/143>`_ by `dupondje <https://github.com/dupondje>`_)
+
+- Fix parsing peak usage with invalid timestamps (`PR #143 <https://github.com/ndokter/dsmr_parser/pull/143>`_ by `dupondje <https://github.com/dupondje>`_)
 
 **1.3.0** (2023-08-01)
 
-- added E.ON Hungary; refactored DSMR specifications to fix obis reference conflicts
- (`PR #137 <https://github.com/ndokter/dsmr_parser/pull/137>`_ by `balazs92117 <https://github.com/balazs92117>`_)
+- added E.ON Hungary; refactored DSMR specifications to fix obis reference conflicts (`PR #137 <https://github.com/ndokter/dsmr_parser/pull/137>`_ by `balazs92117 <https://github.com/balazs92117>`_)
 
 **1.2.4** (2023-07-11)
 
-- EQUIPMENT IDENTIFIER is wrong for Fluvius meters when other mbus devices are present
- (`PR #133 <https://github.com/ndokter/dsmr_parser/pull/133>`_ by `ejpalacios <https://github.com/ejpalacios>`_)
+- EQUIPMENT IDENTIFIER is wrong for Fluvius meters when other mbus devices are present (`PR #133 <https://github.com/ndokter/dsmr_parser/pull/133>`_ by `ejpalacios <https://github.com/ejpalacios>`_)
 
 
 **1.2.3** (2023-04-18)

README.rst

@@ -293,6 +293,36 @@ To install DSMR Parser:
 
     $ pip install dsmr-parser
 
+Development
+-----------
+
+Create a virtualenv and activate it followed by the installation of the dsmr-parser:
+
+.. code-block:: bash
+
+    python3 -m venv venv
+    source venv/bin/activate
+    pip install -e .
+
+Install tox and run it:
+
+.. code-block:: bash
+
+    pip install tox
+    tox
+
+You should see that the tests have succeeded:
+
+.. code-block:: text
+
+    ======================================================================================================== 59 passed in 0.91s ========================================================================================================
+    py: commands[1]> pylama dsmr_parser test
+      py: OK (11.55=setup[9.73]+cmd[1.29,0.53] seconds)
+      congratulations :) (11.69 seconds)
+
+
+Now you can make changes by editing the code and rerunning tox to verify your changes.
+
 Known issues
 ------------
 

dsmr_parser/clients/filereader.py

@@ -73,7 +73,7 @@ class FileReader(object):
                     try:
                         yield self.telegram_parser.parse(telegram)
                     except InvalidChecksumError as e:
-                        logger.warning(str(e))
+                        logger.info(str(e))
                     except ParseError as e:
                         logger.error('Failed to parse telegram: %s', e)
 

dsmr_parser/clients/protocol.py

@@ -4,7 +4,7 @@ from functools import partial
 import asyncio
 import logging
 
-from serial_asyncio import create_serial_connection
+from serial_asyncio_fast import create_serial_connection
 
 from dsmr_parser import telegram_specifications
 from dsmr_parser.clients.telegram_buffer import TelegramBuffer
@@ -158,7 +158,7 @@ class DSMRProtocol(asyncio.Protocol):
         try:
             parsed_telegram = self.telegram_parser.parse(telegram)
         except InvalidChecksumError as e:
-            self.log.warning(str(e))
+            self.log.info(str(e))
         except ParseError:
             self.log.exception("failed to parse telegram")
         else:

dsmr_parser/clients/rfxtrx_protocol.py

@@ -2,7 +2,7 @@
 
 import asyncio
 
-from serial_asyncio import create_serial_connection
+from serial_asyncio_fast import create_serial_connection
 from .protocol import DSMRProtocol, _create_dsmr_protocol
 
 

dsmr_parser/clients/serial_.py

@@ -1,6 +1,6 @@
 import logging
 import serial
-import serial_asyncio
+import serial_asyncio_fast
 
 from dsmr_parser.clients.telegram_buffer import TelegramBuffer
 from dsmr_parser.exceptions import ParseError, InvalidChecksumError
@@ -37,7 +37,7 @@ class SerialReader(object):
                     try:
                         yield self.telegram_parser.parse(telegram)
                     except InvalidChecksumError as e:
-                        logger.warning(str(e))
+                        logger.info(str(e))
                     except ParseError as e:
                         logger.error('Failed to parse telegram: %s', e)
 
@@ -77,7 +77,7 @@ class AsyncSerialReader(SerialReader):
         :rtype: None
         """
         # create Serial StreamReader
-        conn = serial_asyncio.open_serial_connection(**self.serial_settings)
+        conn = serial_asyncio_fast.open_serial_connection(**self.serial_settings)
         reader, _ = await conn
 
         while True:
@@ -107,7 +107,7 @@ class AsyncSerialReader(SerialReader):
         """
 
         # create Serial StreamReader
-        conn = serial_asyncio.open_serial_connection(**self.serial_settings)
+        conn = serial_asyncio_fast.open_serial_connection(**self.serial_settings)
         reader, _ = await conn
 
         while True:

dsmr_parser/clients/socket_.py

@@ -31,11 +31,15 @@ class SocketReader(object):
         buffer = b""
 
         with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as socket_handle:
-
+            socket_handle.settimeout(60)
             socket_handle.connect((self.host, self.port))
 
             while True:
-                buffer += socket_handle.recv(self.BUFFER_SIZE)
+                try:
+                    buffer += socket_handle.recv(self.BUFFER_SIZE)
+                except socket.timeout:
+                    logger.error("Socket timeout occurred, exiting")
+                    break
 
                 lines = buffer.splitlines(keepends=True)
 
@@ -54,7 +58,7 @@ class SocketReader(object):
                     try:
                         yield self.telegram_parser.parse(telegram)
                     except InvalidChecksumError as e:
-                        logger.warning(str(e))
+                        logger.info(str(e))
                     except ParseError as e:
                         logger.error('Failed to parse telegram: %s', e)
 

dsmr_parser/parsers.py

@@ -32,6 +32,7 @@ class TelegramParser(object):
             object["obis_reference"]: re.compile(object["obis_reference"], re.DOTALL | re.MULTILINE)
             for object in self.telegram_specification['objects']
         }
+        self._telegram_encryption_active = None
 
     def parse(self, telegram_data, encryption_key="", authentication_key="", throw_ex=False):  # noqa: C901
         """
@@ -46,38 +47,11 @@ class TelegramParser(object):
         :raises ParseError:
         :raises InvalidChecksumError:
         """
-
-        if "general_global_cipher" in self.telegram_specification:
-            if self.telegram_specification["general_global_cipher"]:
-                enc_key = unhexlify(encryption_key)
-                auth_key = unhexlify(authentication_key)
-                telegram_data = unhexlify(telegram_data)
-                apdu = XDlmsApduFactory.apdu_from_bytes(apdu_bytes=telegram_data)
-                if apdu.security_control.security_suite != 0:
-                    logger.warning("Untested security suite")
-                if apdu.security_control.authenticated and not apdu.security_control.encrypted:
-                    logger.warning("Untested authentication only")
-                if not apdu.security_control.authenticated and not apdu.security_control.encrypted:
-                    logger.warning("Untested not encrypted or authenticated")
-                if apdu.security_control.compressed:
-                    logger.warning("Untested compression")
-                if apdu.security_control.broadcast_key:
-                    logger.warning("Untested broadcast key")
-                telegram_data = apdu.to_plain_apdu(enc_key, auth_key).decode("ascii")
-            else:
-                try:
-                    if unhexlify(telegram_data[0:2])[0] == GeneralGlobalCipher.TAG:
-                        raise RuntimeError("Looks like a general_global_cipher frame "
-                                           "but telegram specification is not matching!")
-                except Exception:
-                    pass
-        else:
-            try:
-                if unhexlify(telegram_data[0:2])[0] == GeneralGlobalCipher.TAG:
-                    raise RuntimeError(
-                        "Looks like a general_global_cipher frame but telegram specification is not matching!")
-            except Exception:
-                pass
+        telegram_data = self.decrypt_telegram_data(
+            telegram_data=telegram_data,
+            encryption_key=encryption_key,
+            authentication_key=authentication_key
+        )
 
         if self.apply_checksum_validation and self.telegram_specification['checksum_support']:
             self.validate_checksum(telegram_data)
@@ -112,6 +86,42 @@ class TelegramParser(object):
 
         return telegram
 
+    def decrypt_telegram_data(self, encryption_key, authentication_key, telegram_data):
+        """
+        Check if telegram data is encrypted and decrypt if applicable.
+        """
+        # if self._telegram_encryption_active is False:
+        #     # If encryption is not working, stop trying and logging warnings.
+        #     return telegram_data
+
+        if self.telegram_specification.get("general_global_cipher"):
+            enc_key = unhexlify(encryption_key)
+            auth_key = unhexlify(authentication_key)
+            telegram_data = unhexlify(telegram_data)
+            apdu = XDlmsApduFactory.apdu_from_bytes(apdu_bytes=telegram_data)
+            if apdu.security_control.security_suite != 0:
+                logger.warning("Untested security suite")
+            if apdu.security_control.authenticated and not apdu.security_control.encrypted:
+                logger.warning("Untested authentication only")
+            if not apdu.security_control.authenticated and not apdu.security_control.encrypted:
+                logger.warning("Untested not encrypted or authenticated")
+            if apdu.security_control.compressed:
+                logger.warning("Untested compression")
+            if apdu.security_control.broadcast_key:
+                logger.warning("Untested broadcast key")
+            telegram_data = apdu.to_plain_apdu(enc_key, auth_key).decode("ascii")
+            self._telegram_encryption_active = True
+        else:
+            try:
+                if unhexlify(telegram_data[0:2])[0] == GeneralGlobalCipher.TAG:
+                    logger.warning("Looks like a general_global_cipher frame "
+                                   "but telegram specification is not matching!")
+            except Exception:
+                pass
+            self._telegram_encryption_active = False
+
+        return telegram_data
+
     @staticmethod
     def validate_checksum(telegram):
         """

dsmr_parser/value_types.py

@@ -2,6 +2,10 @@ import datetime
 
 import pytz
 
+# TODO : Use system timezone
+# Preload timezone to avoid loading in event loop later
+local_tz = pytz.timezone('Europe/Amsterdam')
+
 
 def timestamp(value):
     try:
@@ -20,8 +24,6 @@ def timestamp(value):
     else:
         is_dst = False
 
-    # TODO : Use system timezone
-    local_tz = pytz.timezone('Europe/Amsterdam')
     localized_datetime = local_tz.localize(naive_datetime, is_dst=is_dst)
 
     return localized_datetime.astimezone(pytz.utc)

setup.py

@@ -7,11 +7,11 @@ setup(
     author_email='mail@nldr.net',
     license='MIT',
     url='https://github.com/ndokter/dsmr_parser',
-    version='1.4.0',
+    version='1.4.2',
     packages=find_packages(exclude=('test', 'test.*')),
     install_requires=[
         'pyserial>=3,<4',
-        'pyserial-asyncio<1',
+        'pyserial-asyncio-fast>=0.11',
         'pytz',
         'Tailer==0.4.1',
         'dlms_cosem==21.3.2'

test/clients/test_filereader.py

@@ -0,0 +1,21 @@
+import unittest
+import tempfile
+
+from dsmr_parser.clients.filereader import FileReader
+from dsmr_parser.telegram_specifications import V5
+from test.example_telegrams import TELEGRAM_V5
+
+
+class FileReaderTest(unittest.TestCase):
+    def test_read_as_object(self):
+        with tempfile.NamedTemporaryFile() as file:
+            with open(file.name, "w") as f:
+                f.write(TELEGRAM_V5)
+
+            telegrams = []
+            reader = FileReader(file=file.name, telegram_specification=V5)
+            # Call
+            for telegram in reader.read_as_object():
+                telegrams.append(telegram)
+
+            self.assertEqual(len(telegrams), 1)

test/clients/test_rfxtrx_protocol.py

@@ -0,0 +1,77 @@
+from unittest.mock import Mock
+
+import unittest
+
+from dsmr_parser import obis_references as obis
+from dsmr_parser.clients.rfxtrx_protocol import create_rfxtrx_dsmr_protocol, PACKETTYPE_DSMR, SUBTYPE_P1
+from dsmr_parser.objects import Telegram
+
+TELEGRAM_V2_2 = (
+    '/ISk5\2MT382-1004\r\n'
+    '\r\n'
+    '0-0:96.1.1(00000000000000)\r\n'
+    '1-0:1.8.1(00001.001*kWh)\r\n'
+    '1-0:1.8.2(00001.001*kWh)\r\n'
+    '1-0:2.8.1(00001.001*kWh)\r\n'
+    '1-0:2.8.2(00001.001*kWh)\r\n'
+    '0-0:96.14.0(0001)\r\n'
+    '1-0:1.7.0(0001.01*kW)\r\n'
+    '1-0:2.7.0(0000.00*kW)\r\n'
+    '0-0:17.0.0(0999.00*kW)\r\n'
+    '0-0:96.3.10(1)\r\n'
+    '0-0:96.13.1()\r\n'
+    '0-0:96.13.0()\r\n'
+    '0-1:24.1.0(3)\r\n'
+    '0-1:96.1.0(000000000000)\r\n'
+    '0-1:24.3.0(161107190000)(00)(60)(1)(0-1:24.2.1)(m3)\r\n'
+    '(00001.001)\r\n'
+    '0-1:24.4.0(1)\r\n'
+    '!\r\n'
+)
+
+OTHER_RF_PACKET = b'\x03\x01\x02\x03'
+
+
+def encode_telegram_as_RF_packets(telegram):
+    data = b''
+
+    for line in telegram.split('\n'):
+        packet_data = (line + '\n').encode('ascii')
+        packet_header = bytes(bytearray([
+            len(packet_data) + 3,  # excluding length byte
+            PACKETTYPE_DSMR,
+            SUBTYPE_P1,
+            0  # seq num (ignored)
+        ]))
+
+        data += packet_header + packet_data
+        # other RF packets can pass by on the line
+        data += OTHER_RF_PACKET
+
+    return data
+
+
+class RFXtrxProtocolTest(unittest.TestCase):
+
+    def setUp(self):
+        new_protocol, _ = create_rfxtrx_dsmr_protocol('2.2',
+                                                      telegram_callback=Mock(),
+                                                      keep_alive_interval=1)
+        self.protocol = new_protocol()
+
+    def test_complete_packet(self):
+        """Protocol should assemble incoming lines into complete packet."""
+
+        data = encode_telegram_as_RF_packets(TELEGRAM_V2_2)
+        # send data broken up in two parts
+        self.protocol.data_received(data[0:200])
+        self.protocol.data_received(data[200:])
+
+        telegram = self.protocol.telegram_callback.call_args_list[0][0][0]
+        assert isinstance(telegram, Telegram)
+
+        assert float(telegram[obis.CURRENT_ELECTRICITY_USAGE].value) == 1.01
+        assert telegram[obis.CURRENT_ELECTRICITY_USAGE].unit == 'kW'
+
+        assert float(telegram[obis.GAS_METER_READING].value) == 1.001
+        assert telegram[obis.GAS_METER_READING].unit == 'm3'

test/test_parse_v5.py

@@ -21,7 +21,7 @@ class TelegramParserV5Test(unittest.TestCase):
             telegram = parser.parse(TELEGRAM_V5, throw_ex=True)
         except Exception as ex:
             assert False, f"parse trigged an exception {ex}"
-        print('test: ', type(telegram.P1_MESSAGE_HEADER), telegram.P1_MESSAGE_HEADER.__dict__)
+
         # P1_MESSAGE_HEADER (1-3:0.2.8)
         assert isinstance(telegram.P1_MESSAGE_HEADER, CosemObject)
         assert telegram.P1_MESSAGE_HEADER.unit is None

tox.ini

@@ -14,6 +14,9 @@ commands=
 [pylama:dsmr_parser/clients/__init__.py]
 ignore = W0611
 
+[pylama:dsmr_parser/clients/socket_.py]
+ignore = C901
+
 [pylama:dsmr_parser/parsers.py]
 ignore = W605
 
@@ -24,4 +27,4 @@ ignore = E501
 max_line_length = 120
 
 [pylama:pycodestyle]
-max_line_length = 120
+max_line_length = 120