Merge pull request #175 from TUB-DVG/154-add-db-diff-functionality-fo…

…r-another-app

Rollback feature for tools_over app

.github/workflows/test.yml

@@ -33,8 +33,11 @@ jobs:
       - name: Run dev-environment
         run: ./run up_initial dev postgres/webcentral_db_20241023_fixed_translation_error_in_usage.sql
 
-      - name: Execute first test
+      - name: Execute unittest for project_listing app
         run: docker exec -w /webcentral/src webcentral python manage.py test project_listing
+
+      - name: Execute unittest for tools_over app
+        run: docker exec -w /webcentral/src webcentral python manage.py test tools_over
 
       - name: Setup selenium dev-environment
         run: |

webcentral/doc/06_sphinx/source/HowTo/rollback_tool.md

@@ -0,0 +1,36 @@
+# Updating and rollback tools
+In this guide it is shown how a bulk update and import of tools and digital applications can be done using csv/excel-files. First a up-to-date excel file is needed, which holds the german and english data of all tools of the database. This file can be exported from the database using the django custom management command `data_export`:
+
+1. Open a terminal in the root of the project repository. Ensure that the app is running (in this guide the development mode is used). Switch into the shell of the `webcentral` docker container: 
+```
+    ./run webcentral_shell
+```
+And move into the `src/` directory:
+```
+    cd src/
+```
+2. Export a file `tools_latest_dump.xlsx`:
+```
+    python manage.py data_export tools_latest_dump.xlsx
+```
+When using the development-mode the file will be available in `webcentral/src/`-folder. 
+```{note}
+    If the production-mode was used to export the excel-file, it has to be copied out of the docker container using the docker cp-command from a terminal within the host-system
+
+```
+3. Update the content of the excel-file.
+4. Import the excel file using the `data_import`-command:
+```
+   python manage.py data_import tools_over tools_latest_dump.xlsx
+```
+When updating a tool or application creates a new state of the tool in the database a `History` object is created. 
+In that object the old state of the tool is saved as a stringified JSON object. When a rollback needs to be done, the following steps can be taken:
+5. Login into the admin panel. The credentials can be fund in the `.env`-file.
+6. Find the `History`-tab on the left side of the admin panel inside the `tools_over`-menu.
+ ![Image of the tools_over tab](../img/tools_over_tab.png)
+Click on the history-element to be redirected to the history object listing page. There all history objects are shown, whereby each object is named after the tool or digtal application its state it is saving and the timestamp at which the update for the tool was done. When clicking on one of the history object a details page should open, which should look like the following:
+![Image of the tools_over tab](../img/details_tools_comparison.png)
+On the left side the current state of the tool `BESMod` is shown. On the right side the state is show to which the tool potentially could be rollbacked. The lines with the yellow background show, where attributes differ between the current state and the rollback state. To rollback the tool-state the user needs to go back to the history listing page and select the tool states, which should be rolled back:
+![Image of the tools_over tab](../img/select_rollback.png)
+After selecting the tools, which should rollback, select from the actions dropdown menu `Rollback selected change` and press apply.
+The history object should then disappear and the state of the selected tools should be rollbacked.

webcentral/doc/06_sphinx/source/dev/custom_tags_and_template_tags.md

@@ -0,0 +1,2 @@
+# Custom tags and template tags
+Build-in tags and template tags are decsribed in the [django documentation](https://docs.djangoproject.com/en/5.1/ref/templates/builtins/)

webcentral/doc/06_sphinx/source/general/data_export.md

@@ -0,0 +1,20 @@
+# Export data
+In the following sections it is described how to export structured data for a specific `django` app from the database into spreadsheet files. Thereby the data from the database is exported in files with the tabular structure present in `webcentral/doc/01_data/`. Each of the subsequent folders corespond to a `django` app whereby the different `django` apps are present in the folder `webcentral/src/`.
+This approach can be helpful, when data was imported into the database in several ways (via admin-panel, via spreadsheet files etc.) and a central data source is needed for future imports.
+
+## Structure of the source code 
+The `data_export` function can be called from the `django` `manage.py` management script. The general structure of the custom management command is written out below:
+```bash
+python manage.py data_export app-name spreadsheet-filename.xlsx
+```
+This command starts the `data_export` for one of the `django` apps inside the `webcentral/src/` folder with the name `app-name` and exports a spreadsheet file `spreadsheet-filename.xlsx` into the  `webcentral/src/` folder. 
+```{note}
+The spreadsheet file will only be visible on the host system when using the development mode of the `EWB Wissensplattform`. If you are using the production mode of the application you need to copy the created .xlsx file manually to the host filesystem.
+```
+```{note}
+It can happen, that it is not possible to open the spreadsheet on the host-system because of insuficient rights. If on a linux-system you can use the `chown` utility to change the file owner to the current OS-user. 
+```
+## Code structure
+Similar to the the `data_import`-structure, a script `data_export.py` inside the folder `webcentral/src/common/management/commands/` is needed to introduce a custom management command `data_export`. It defines a class `CustomCommand`, which inherits from the `BaseCommand` `django` class. In the `CustomCommand` two methods need to be implemented: The `add_arguments()`- and the `handle()`-method, whereby the `add_arguments()` is used to add argument-structure to the command and the `handle()`-method is used to add functionality.
+The `handle()`-method then searches the installed `django`-app for the given app in the first argument and loads the `data_export.py`-module inside that app-directory. Inside the app specific `data_export.py`-module a class `DataExport` is definied, which is implemented differently for each app based on the structure of the apps models and the corresponding spreadsheet file.
+

webcentral/doc/06_sphinx/source/general/data_import.md

@@ -10,21 +10,25 @@ In general, data can be imported using the following ways:
 In the following sections these methods are described in detail.
 
 ## Data-import using the django admin panel
-This approach can be considered straight forward. If only small chunks of data need to be imported this method is apropriate. For that method to work, the django superuser-credentials are needed, which can be found in the `.env`-file in the root project folder.
+This approach can be considered straight forward. If only small chunks of data need to be imported, this method is apropriate. For that method to work, the django superuser-credentials are needed, which can be found in the `.env`-file in the root project folder.
+```{note}
+    When importing the SQL-dump file from the `postgres/`-folder, a django superuser is automatically created. This user can login to the django admin panel at (http://127.0.0.1:8000/admin)[http://127.0.0.1:8000/admin] and create and edit data. The login-credentials for the superuser can be found in the `.env`-file.
+```
 The admin panel can be entered by either opening your locally hosted version or the server hosted production version. For your local version enter the following link in your browser of choice:
 ```
 http://127.0.0.1:8000/admin
 ```
-On the opened site, enter username and password from the `.env`-file. You should then be able to create, modify or delete 
+On the opened site, enter username and password from the `.env`-file. You should then be able to create, modify or delete data from the database.
+The data is organized 
 
 ## Data-import using the data_import command
-To import greater numbers of structured data of a specific type, python-scripts has been written. These are accessible through a django custom-command `data_import`. The command can be started using the django `manage.py`:
+To import greater numbers of structured data of a specific type, python-scripts have been written. These are accessible through a django custom-command `data_import`. The command can be started using the django `manage.py`:
 ```
-    python manage.py data_import <app_label> <path_to_xlsx_or_csv> <path_to_diff_file>
+    python manage.py data_import <app_label> <path_to_xlsx_or_csv>
 ```
-The command gets 3 arguments. `app_label` specifies the app-label, which holds the model into which the data should be imported. the app-label is the name of the folder in which the corrsponding model lies. Please notethat the data import is only working, if a `data_import.py` is present in the specified app folder. Please note further, that the structured data needs to have the right structure for a successfull data import. That means, that the columns in the excel need to have the name, which is used in the app-specific data-import-script. Please consult the data-folder to inspect the needed structure of the execl file.
-`path_to_xlsx_or_csv` specfies the path to a .csv- or .xlsx-file, which holds the structured data. 
-`path_to_diff_file` is the path, where a diff-file is saved. It is only created on collisions and will be explained in detail here (link to execute_db_changes).
+The command gets 2 arguments. `<app_label>` specifies the app-label, which holds the model into which the data should be imported. the app-label is the name of the folder in which the corresponding model lies. Please note, that the data import is only working, if a `data_import.py` is present in the specified app folder. Please note further, that the structured data needs to have the right structure for a successfull data import. That means, that the columns in the excel need to have the name, which is used in the app-specific data-import-script. Please consult the data-folder to inspect the needed structure of the execl file.
+`<path_to_xlsx_or_csv>` specfies the path to a .csv- or .xlsx-file, which holds the structured data. 
+
 The structure of the implemented python scripts is as follows: In the app `common`, which holds code used across apps. It is placed under `common/data_import.py` and holds a class `DataImport`. This class handles general functionality, which is used by the app-specfic data-import classes like e.g. reading a file. The app-specific data-import classes are located in each app in the file `data_import.py`. Each of theses files holds a class `DataImportApp`, which inherits from the general `DataImport`. 
 ```{mermaid}
 classDiagram
@@ -49,11 +53,31 @@ classDiagram
 
 ```
 ### Tools import
-To import digital tools and digital applications into the database, a excel-file can be used as shown in `/02_work_doc/01_daten/02_toolUebersicht/2024_05_EWB_newToolsImportWithTranslation.xlsx`. The file holds, besides others, the sheets `German` and `English`. The import script scans for these 2 specific sheets and imports the content of the sheet `German` into the fields with the suffices `_de`, while it imports the content of the sheet `English` to the fields with the suffice `_en`. If the two sheets `German` and `English` are not present, it will import the first sheet (the sheet most left, when the file is opened in Excel) into the german fields of the `Tools`. It will not import any present english translations. 
-
+To import digital tools and digital applications into the database, a excel-file can be used as shown in `webcentral/doc/01_daten/02_toolUebersicht/2024_05_EWB_tools_with_english_translation.xlsx`. Since the plattform is bilingual, text-data should be imported with its german and english representation. Thats why the given excel file holds the sheets `German` and `English`. 
+```{note}
+If only one sheet is present, the import algorithm will import the given table into the german fields and will skip the english fields. 
+If you would like to import both languages, please make sure that 2 worksheets are present with the names `German` and `English`
+```
+The import script scans for these 2 specific sheets and imports the content of the sheet `German` into the fields with the suffices `_de`, while it imports the content of the sheet `English` to the fields with the suffice `_en`. If the two sheets `German` and `English` are not present, it will import the first sheet (the sheet most left, when the file is opened in Excel) into the german fields of the `Tools`. It will not import any present english translations. 
+```{warning}
+Please make sure to keep the names of the headers as they are in the presented excel-file. Otherwise the import will fail.
+```
 To map the english translation from the sheet `English` onto the model fields the same header names are used as in the sheet `German`. When the data is imported, the 2 sheets ge merged into one list datastructure. To differantiate between german and english fields, the header names of the english fields get the suffice `__en`.
 inside the `data_import.py` in the `tools_over`-app a dictionary `MAPPING_EXCEL_DB_EN` is defined as a class-attribute. That datastructure holds the name of the imported english header as key and the corresponding name of the ORM-model-field as value. For `Tools` that feels redundant at the moment since each key-value-field differs only in one `_`, but it can be used in other model-import-scripts if the headername differs from the ORM field name.
 
+#### Tools update
+When running the `data_import`-command the state within the `Tools`-database table is updated to the state of the excel-file. If a `Tools`-row is updated, the prior state is saved in the `History` table as a serialized JSON-string. That history-object can be used to rollback the state of the `Tools`-object to the state prior to the update. The general flow can be seen in the following flow chart:
+```{mermaid}
+flowchart TD
+    A[Example Tool Excel file] -->|data_import|C{Example Tool in database}
+    C -->|Yes| D[Serialize old state of Example Tool]
+    D --> E[Create History Object and save serialized state of Example Tool]
+    E --> G[Update Example Tool from Excel file]
+    C -->|No| F[Create Example Tool]
+```
+#### Rollback a tool
+A tool can be rolled back to its previous state using the admin-panel. 
+
 ### Enargus data import
 The data from the enargus database can be imported via the `data_import` custom django management command. Since the data is given as a XML-file but the `data_import` command only allows tabular input as CSV or excel-file, a preprocessing step has to be done. This step can be started using the `run`-script:
 ```

webcentral/doc/06_sphinx/source/index.rst

@@ -60,6 +60,8 @@ HowTo Guides
    :caption: HowTo:
 
    HowTo/create_dump_for_repo
+   HowTo/rollback_tool
+
 
 Old to be refactored
 ====================

webcentral/src/common/data_import.py

@@ -349,17 +349,19 @@ def _importEnglishManyToManyRel(
         germanManyToManyStr = self._processListInput(
             row[header.index(headerExcel)], ";;"
         )
+
         englishManyToManyStr = self._processListInput(
             row[header.index(f"{headerExcel}__en")], ";;"
         )
-
         elementsForAttr = getattr(ormObj, dbAttr).all()
         for ormRelObj in elementsForAttr:
             for indexInGerList, germanyManyToManyElement in enumerate(
                 germanManyToManyStr
             ):
                 if germanyManyToManyElement in str(ormRelObj):
                     if getattr(ormRelObj, f"{dbAttr}_en") is None:
+                        if englishManyToManyStr[indexInGerList] is None:
+                            englishManyToManyStr[indexInGerList] = ""
                         setattr(
                             ormRelObj,
                             f"{dbAttr}_en",
@@ -371,6 +373,8 @@ def _importEnglishManyToManyRel(
     def _importEnglishAttr(self, ormObj, header, row, headerExcel, dbAttr):
         """ """
         englishTranslation = row[header.index(f"{headerExcel}__en")]
+        if englishTranslation is None:
+            englishTranslation = ""
         setattr(ormObj, f"{dbAttr}_en", englishTranslation)
 
         return ormObj
@@ -410,8 +414,12 @@ def _compareDjangoOrmObj(self, modelType, oldObj, newObj):
                 newValue = getattr(newObj, field.name)
                 if oldValue != newValue:
                     if isinstance(oldValue, str):
+
                         oldValueWithoutNewLine = oldValue.replace("\n", "<br>")
-                        newValueWithoutNewLine = newValue.replace("\n", "<br>")
+                        if isinstance(newValue, str):
+                            newValueWithoutNewLine = newValue.replace(
+                                "\n", "<br>"
+                            )
                     else:
                         oldValueWithoutNewLine = oldValue
                         newValueWithoutNewLine = newValue
@@ -438,6 +446,20 @@ def _compareDjangoOrmObj(self, modelType, oldObj, newObj):
                     diffStr += f"""   {field.name}: {oldValueWithoutNewLine} ->
                     {newValueWithoutNewLine}\n"""
 
+            elif isinstance(field, ManyToManyField):
+                fieldName = field.name
+                oldValueDe = oldObj.getManyToManyAttrAsStr(fieldName, "_de")
+                oldValueEn = oldObj.getManyToManyAttrAsStr(fieldName, "_en")
+
+                newValueDe = newObj.getManyToManyAttrAsStr(fieldName, "_de")
+                newValueEn = newObj.getManyToManyAttrAsStr(fieldName, "_en")
+
+                oldStr = f"German: {oldValueDe}, English: {oldValueEn}"
+                newStr = f"German: {newValueDe}, English: {newValueEn}"
+
+                if oldStr != newStr:
+                    diffStr += f"""   {field.name}: {oldStr} -> {newStr}\n"""
+
         if diffStr != "":
             diffStr = diffStrModelName + diffStr + ";;"
         self.diffStrDict[self.dictIdentifier] += diffStr

webcentral/src/common/management/commands/data_export.py

@@ -0,0 +1,40 @@
+import importlib
+
+from django.core.management.base import BaseCommand, CommandError
+from django.conf import settings
+
+
+class Command(BaseCommand):
+    help = "Exports Data as a Excel sheet from specified App"
+
+    def add_arguments(self, parser):
+        parser.add_argument("type_of_data", nargs="+", type=str)
+        parser.add_argument("filename", nargs="+", type=str)
+
+    def handle(self, *args, **options):
+        """ """
+
+        filename = options["filename"][0]
+
+        type_of_data = options["type_of_data"][0]
+        data_import_module = self._checkIfInInstalledApps(type_of_data)
+        appDataExportObj = data_import_module.DataExport(filename)
+        appDataExportObj.exportToXlsx()
+
+    def _checkIfInInstalledApps(self, type_of_data):
+        """Check if user given argument `type_of_data` matches
+        one of the installed apps. If it does, check if a `data_export`-module
+        is present in that app.
+        """
+        installed_django_apps = settings.INSTALLED_APPS
+        app_names = [app.split(".")[0] for app in installed_django_apps]
+        if type_of_data in app_names:
+            if (
+                importlib.util.find_spec(type_of_data + ".data_export")
+                is not None
+            ):
+                return importlib.import_module(type_of_data + ".data_export")
+        raise CommandError(
+            """specified type_of_data has no corresponding app or has no 
+            data_export.py in the app."""
+        )