features_selection_limma.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# BIO-SELECT - Marigliano\n",
    "## Features selection using Limma and R"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of this script is to use Limma algorithm and add the selected features to the ones we already have selected in the features_selection.ipynb notebook.\n",
    "\n",
    "To use this notebook, you need to have Docker installed.\n",
    "\n",
    "The steps are the following:\n",
    "1. Build docker image to setup a ready-to-use R environment\n",
    "2. Run the two docker containers, one for MILE and one for Golub\n",
    "    1. Run the container\n",
    "    1. Read the limma CSV file\n",
    "    1. Sort this file\n",
    "    1. Convert limma features indices to dataset indices\n",
    "    1. Append the feature list to the CSV files generated in the features_selection.ipynb notebook\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build Docker image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# execute this every time you change the R scripts.\n",
    "!cd docker-R && \\\n",
    "docker build -t rdocker ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run Limma for Golub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "####\n",
    "#### change the cell type to \"Code\" to be able to run it\n",
    "####\n",
    "\n",
    "# $HOST_WD is a environment variable which contains the host current folder since Docker in Docker containers use the host context\n",
    "!cd docker-R && \\\n",
    "docker run -it -v $HOST_WD/docker-R/dataset:/dataset --rm rdocker Rscript --no-save --no-restore --verbose limma-golub.R"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!ls -al docker-R/dataset/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!head -n4 docker-R/dataset/limma-golub.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parse Limma CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from datasets.Golub99.GolubDataset import GolubDataset\n",
    "from algorithms.Algorithm import Algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "GROUP_NAME_GOLUB = \"golub_11052017\" #TODO: change it to match the group name of the previously generated lists (CSV)\n",
    "N_FEATURES = 1000\n",
    "ALG_NAME = \"Limma\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ds_golub = GolubDataset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "filename = r\"docker-R/dataset/limma-golub.csv\"\n",
    "\n",
    "df = pd.read_csv(filename, sep=\"\\t\", usecols=[\"ID\", \"B\"])\n",
    "df = df.dropna()  # ignore NaN values\n",
    "\n",
    "df = df.sort_values(['B'], ascending=[0])\n",
    "\n",
    "# convert pandas dataframe to array of tuples\n",
    "features_by_score = [tuple(x) for x in df.to_records(index=False)]\n",
    "\n",
    "# convert features name to features indices\n",
    "f_names, f_scores = zip(*features_by_score)\n",
    "f_names = ds_golub.get_features_indices(f_names)\n",
    "features_by_score = zip(f_names, f_scores)\n",
    "print(features_by_score[:3])\n",
    "\n",
    "# normalize the score\n",
    "features_by_score_normed = Algorithm.normalize_scores(features_by_score)[:N_FEATURES]\n",
    "\n",
    "# transform the rank tuples in the format: (index, rank)\n",
    "# reverse the rank to have the best features with a higher score appear first\n",
    "r = [f[0] for f in features_by_score_normed]\n",
    "features_by_rank = [(v, 1.0/(1.0+k)) for k, v in enumerate(r)]\n",
    "\n",
    "# assign the same weight for all features\n",
    "features = [(f[0], 1) for f in features_by_score_normed]\n",
    "\n",
    "# prepare the subsets dict to export in CSV\n",
    "subsets = {}\n",
    "subsets[ALG_NAME] = {\"features\": [], \"features_by_rank\": [], \"features_by_score\": []}\n",
    "subsets[ALG_NAME][\"features\"] = features\n",
    "subsets[ALG_NAME][\"features_by_rank\"] = features_by_rank\n",
    "subsets[ALG_NAME][\"features_by_score\"] = features_by_score_normed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save the features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from utils.CSVFeaturesExporter import CSVFeaturesExporter\n",
    "\n",
    "group_name = GROUP_NAME_GOLUB + \"_limma\"\n",
    "features_exporter = CSVFeaturesExporter(subsets, group_name)\n",
    "features_exporter.export()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ[\"GROUP_NAME\"] = GROUP_NAME_GOLUB"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# TODO: change the cell type of this cell to \"Code\" to concatenate \n",
    "# the CSVs (features_selection.ipynb and features_selection_limma.ipynb)\n",
    "\n",
    "!cat outputs/$GROUP_NAME\\_limma_features.csv >> outputs/$GROUP_NAME\\_features.csv\n",
    "!cat outputs/$GROUP_NAME\\_limma_features_by_rank.csv >> outputs/$GROUP_NAME\\_features_by_rank.csv\n",
    "!cat outputs/$GROUP_NAME\\_limma_features_by_score.csv >> outputs/$GROUP_NAME\\_features_by_score.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run Limma for MILE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "####\n",
    "#### change the cell type to \"Code\" to be able to run it\n",
    "####\n",
    "\n",
    "# $HOST_WD is a environment variable which contains the host current folder since Docker in Docker containers use the host context\n",
    "!cd docker-R && \\\n",
    "docker run -it -v $HOST_WD/docker-R/dataset:/dataset --rm rdocker Rscript --no-save --no-restore --verbose limma-mile.R"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!ls -al docker-R/dataset/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!head -n3 docker-R/dataset/limma-mile.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parse Limma CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from datasets.MILE.MileDataset import MileDataset\n",
    "from algorithms.Algorithm import Algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "GROUP_NAME_MILE = \"MILE_11052017\" #TODO: change it to match the group name of the previously generated lists (CSV)\n",
    "N_FEATURES = 1000\n",
    "ALG_NAME = \"Limma\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# load only 20 samples is enough because we only want to convert the names of the features in indices\n",
    "ds_mile = MileDataset(samples_limit=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "filename = r\"docker-R/dataset/limma-mile.csv\"\n",
    "\n",
    "df = pd.read_csv(filename, sep=\"\\t\", usecols=[\"Genes.ID\", \"F\"])\n",
    "df = df.dropna()  # ignore NaN values\n",
    "\n",
    "df = df[[\"Genes.ID\", \"F\"]] # order the columns\n",
    "\n",
    "df = df.sort_values(['F'], ascending=[0])\n",
    "\n",
    "# convert pandas dataframe to array of tuples\n",
    "features_by_score = [tuple(x) for x in df.to_records(index=False)]\n",
    "\n",
    "# convert features name to features indices\n",
    "f_names, f_scores = zip(*features_by_score)\n",
    "f_names = ds_mile.get_features_indices(f_names)\n",
    "features_by_score = zip(f_names, f_scores)\n",
    "\n",
    "# normalize the score\n",
    "features_by_score_normed = Algorithm.normalize_scores(features_by_score)[:N_FEATURES]\n",
    "print(features_by_score_normed[:10])\n",
    "\n",
    "# transform the rank tuples in the format: (index, rank)\n",
    "# reverse the rank to have the best features with a higher score appear first\n",
    "r = [f[0] for f in features_by_score_normed]\n",
    "features_by_rank = [(v, 1.0/(1.0+k)) for k, v in enumerate(r)]\n",
    "\n",
    "# assign the same weight for all features\n",
    "features = [(f[0], 1) for f in features_by_score_normed]\n",
    "\n",
    "# prepare the subsets dict to export in CSV\n",
    "subsets = {}\n",
    "subsets[ALG_NAME] = {\"features\": [], \"features_by_rank\": [], \"features_by_score\": []}\n",
    "subsets[ALG_NAME][\"features\"] = features\n",
    "subsets[ALG_NAME][\"features_by_rank\"] = features_by_rank\n",
    "subsets[ALG_NAME][\"features_by_score\"] = features_by_score_normed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save the features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from utils.CSVFeaturesExporter import CSVFeaturesExporter\n",
    "\n",
    "group_name = GROUP_NAME_MILE + \"_limma\"\n",
    "features_exporter = CSVFeaturesExporter(subsets, group_name)\n",
    "features_exporter.export()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ[\"GROUP_NAME\"] = GROUP_NAME_MILE"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# TODO: change the cell type of this cell to \"Code\" to concatenate \n",
    "# the CSVs (features_selection.ipynb and features_selection_limma.ipynb)\n",
    "\n",
    "!cat outputs/$GROUP_NAME\\_limma_features.csv >> outputs/$GROUP_NAME\\_features.csv\n",
    "!cat outputs/$GROUP_NAME\\_limma_features_by_rank.csv >> outputs/$GROUP_NAME\\_features_by_rank.csv\n",
    "!cat outputs/$GROUP_NAME\\_limma_features_by_score.csv >> outputs/$GROUP_NAME\\_features_by_score.csv"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}