Quantitative Probing in Python

Can we validate the quality of a causal analysis (correctly recovered causal graph and target causal effect) by checking how many previously known causal effects could be correctly recovered by the model (hit rate)? Yes, we can!

This is a repository for quantitative probing, which is a method of validating graphical causal models using quantitative domain knowledge. For more info about the method, check out this presentation from CDSM22.

The repo contains two main components:

• The qprobing package provides methods for a statistical evaluation of the effectiveness of different quantitative probing variants.
• The Juypter notebooks analysis.ipynb and connected_analysis.ipynb, together with the pkl files in this repo, can be used to recreate the results of a related research article. They should also be used as a guide for performing your own analyses.

Installation of the qprobing package

1. Install Python 3.8 or 3.9. More recent versions should work, too, but the build and test pipeline ensures a working state only for these versions. We recommend using a virtual environment for the installation.
2. Install the cause2e package for causal end-to-end analysis by following these instructions.
3. Install qprobing from source by running

   pip install https://github.com/MLResearchAtOSRAM/qprobing/archive/main.tar.gz
   

If you want to clone the repository into a folder for development on your local machine, please navigate to the folder and run:

git clone https://github.com/MLResearchAtOSRAM/qprobing

Citation

If you use the qprobing package in your work, please cite

Daniel Grünbaum (2022). qprobing: A Python package for evaluating the
effectiveness of quantitative probing for causal model validation.
https://github.com/MLResearchAtOSRAM/qprobing

and the related research article.