Skip to content

Latest commit

 

History

History
118 lines (71 loc) · 5.08 KB

README.md

File metadata and controls

118 lines (71 loc) · 5.08 KB
Raw

DCNiOS

DCNiOS is an open-source command-line tool to easily manage the creation of event-driven data processing flows. DCNiOS, Data Connector through Apache NiFi for OSCAR, facilitates the creation of event-driven processes connecting a Storage System like dCache or Amazon S3 to a scalable OSCAR cluster by employing predefined dataflows that are processed by Apache NiFi.

Apache NiFi is a reliable system to process and distribute data through powerful and scalable directed graphs of data routing, transformation, and system mediation logic.

OSCAR is an open-source platform for serverless event-driven data processing of containerized applications across the computing continuum.

The DCNiOS command-line interface allows you to create a NiFi dataflow though a YAML file (see figure).

After creating a dataflow using DCNiOS, when an event is captured (e.g., Server-sent event by dCache), Apache NiFi stores it in an internal queue and then sends it for processing to an OSCAR cluster at a customized rate. In the OSCAR cluster, a user-defined service creates a job to process the stored data.

DCNiOS Workflow

Therefore, DCNiOS has been made to interact with NiFi and deploy a complete dataflow. It uses HTTP calls to communicate with a Nifi cluster.

Apache NiFi can be automatically deployed using the Infrastructure Manager (IM) on a dynamically provisioned Kubernetes Cluster. Note that a NiFi registry is not required.

Achievements

Silver Badge

This software has received a silver badge according to the Software Quality Baseline criteria defined by the EOSC-Synergy project.

Getting Started

Prerequisites

  • OSCAR cluster with services deployed.
  • NiFi Cluster deployed.
  • A Python distribution such as Anaconda.

Installation

  1. Download the repository in a local folder.

    git clone git@github.com:interTwin-eu/dcnios.git
cd dcnios

  2. Set up a virtual environment (optional but recommended):

    You can choose between Conda or Venv for managing Python dependencies.

    Using Conda

    Create a new Conda environment and activate it:

    conda create --name dcnios python=3.7.6
conda activate dcnios

    Using Venv

    Create a virtual environment with Venv and activate it:

    python -m venv dcniosenv
source dcniosenv/bin/activate

  3. Install Dependencies

    Install all the required dependencies listed in requirements.txt:

    pip install -r requirements.txt

    Alternatively, you can install the minimum required dependencies for DCNiOS:

    pip install pyyaml==6.0 requests==2.28.2 oscar_python==1.0.3

  4. Verify Installation

    Once you've installed the dependencies. You can verify this by running a simple Python script. Try the following Python command to check if the packages are installed correctly:

    python -c "import yaml, requests, oscar_python; print('All dependencies are correctly installed.')"

Authors

Instituto de Instrumentación para Imagen Molecular (I3M), Centro Mixto CSIC — Universitat Politècnica de València, Camino de Vera s/n, 46022 Valencia, España.

Versions and Maintenance

There is only one version in maintenance:

  • The main branch in the source code repository maintains a working state version of the software component.
  • Documentation is updated with the new software versions involving any substantial or minimal change in the application's behavior. Any issue can be reported in the Issues section of the GitHub project.
  • Documentation is updated whenever reported as inaccurate or unclear.

Licensing

DCNiOS is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

Acknowledgements

DCNiOS Workflow

This work was supported by the project “An interdisciplinary Digital Twin Engine for science’’ (interTwin), which has received funding from the European Union’s Horizon Europe Programme under Grant 101058386.

More information

You can find more information in the OSCAR's blog.