TextClassifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TAMU Datathon - Bloomberg Challenge (post-competition work)\n",
    "\n",
    "I wanted to revisit this project, and see it to completion.\n",
    "\n",
    "Part 1 was guessing what 5 embeddings' original news articles were about (see other notebook).\n",
    "\n",
    "Part 2 was building a general classifier for Bloomberg's embedding system.\n",
    "\n",
    "This will have two phases:\n",
    "\n",
    "A: A genre classifier for bodies of text, trained on a large set of articles. This will be used to generate genre labels for the set of ~1,000 embeddings we were originally given at the start of the contest.\n",
    "\n",
    "B: A genre classifier for Bloomberg's embeddings. This will only be trained using the ~1,000 embeddings we were given as features, and the labels generated in Part A as targets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PART 2: Classifier\n",
    "\n",
    "## Phase A: Genre Classifier, Label Generation for Embedding Set\n",
    "\n",
    "First, import the training sets (texts with genre labels) and embedding data (embeddings with texts).\n",
    "\n",
    "Second, preprocess the data using TF-IDF vectors.\n",
    "\n",
    "Third, train and cross-validate a supervised learning model for label generation.\n",
    "\n",
    "Fourth, once the model is satisfactory, apply to the embedding data.\n",
    "\n",
    "Fifth, export the embedding-label pairings (so I don't have to repeat this cell)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training the label classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    00       000  0001  000bn  000m  000s  000th  001  001and  001st  ...  \\\n",
      "0  0.0  0.020868   0.0    0.0   0.0   0.0    0.0  0.0     0.0    0.0  ...   \n",
      "1  0.0  0.000000   0.0    0.0   0.0   0.0    0.0  0.0     0.0    0.0  ...   \n",
      "2  0.0  0.000000   0.0    0.0   0.0   0.0    0.0  0.0     0.0    0.0  ...   \n",
      "3  0.0  0.018989   0.0    0.0   0.0   0.0    0.0  0.0     0.0    0.0  ...   \n",
      "4  0.0  0.000000   0.0    0.0   0.0   0.0    0.0  0.0     0.0    0.0  ...   \n",
      "\n",
      "   zooms  zooropa  zornotza  zorro  zubair  zuluaga  zurich  zutons  \\\n",
      "0    0.0      0.0       0.0    0.0     0.0      0.0     0.0     0.0   \n",
      "1    0.0      0.0       0.0    0.0     0.0      0.0     0.0     0.0   \n",
      "2    0.0      0.0       0.0    0.0     0.0      0.0     0.0     0.0   \n",
      "3    0.0      0.0       0.0    0.0     0.0      0.0     0.0     0.0   \n",
      "4    0.0      0.0       0.0    0.0     0.0      0.0     0.0     0.0   \n",
      "\n",
      "   zvonareva  zvyagintsev  \n",
      "0        0.0          0.0  \n",
      "1        0.0          0.0  \n",
      "2        0.0          0.0  \n",
      "3        0.0          0.0  \n",
      "4        0.0          0.0  \n",
      "\n",
      "[5 rows x 29421 columns]\n",
      "[0.81123596 0.90786517 0.86067416 0.92134831 0.88764045]\n",
      "['Entertainment' 'Politics' 'Entertainment' 'Sports' 'Politics' 'Politics'\n",
      " 'Politics' 'Business' 'Business' 'Sports' 'Entertainment' 'Entertainment'\n",
      " 'Entertainment' 'Technology' 'Entertainment' 'Technology' 'Technology'\n",
      " 'Politics' 'Entertainment' 'Business' 'Sports' 'Politics' 'Business'\n",
      " 'Politics' 'Technology' 'Business' 'Technology' 'Entertainment'\n",
      " 'Entertainment' 'Technology' 'Business' 'Sports' 'Business' 'Sports'\n",
      " 'Business' 'Business' 'Business' 'Business' 'Entertainment' 'Technology'\n",
      " 'Entertainment' 'Business' 'Sports' 'Technology' 'Sports' 'Sports'\n",
      " 'Sports' 'Politics' 'Entertainment' 'Business' 'Entertainment' 'Business'\n",
      " 'Technology' 'Sports' 'Sports' 'Business' 'Business' 'Sports' 'Business'\n",
      " 'Entertainment' 'Business' 'Sports' 'Technology' 'Sports' 'Business'\n",
      " 'Politics' 'Technology' 'Sports' 'Technology' 'Business' 'Technology'\n",
      " 'Technology' 'Technology' 'Business' 'Entertainment' 'Business'\n",
      " 'Business' 'Entertainment' 'Technology' 'Business' 'Business' 'Business'\n",
      " 'Sports' 'Politics' 'Sports' 'Business' 'Entertainment' 'Business'\n",
      " 'Politics' 'Entertainment' 'Technology' 'Technology' 'Politics'\n",
      " 'Business' 'Technology' 'Technology' 'Sports' 'Technology' 'Technology'\n",
      " 'Technology' 'Politics' 'Sports' 'Entertainment' 'Entertainment'\n",
      " 'Business' 'Sports' 'Entertainment' 'Business' 'Sports' 'Entertainment'\n",
      " 'Entertainment' 'Business' 'Entertainment' 'Business' 'Technology'\n",
      " 'Business' 'Politics' 'Business' 'Business' 'Sports' 'Business'\n",
      " 'Technology' 'Technology' 'Sports' 'Politics' 'Technology' 'Business'\n",
      " 'Sports' 'Technology' 'Sports' 'Business' 'Entertainment' 'Politics'\n",
      " 'Technology' 'Technology' 'Sports' 'Business' 'Sports' 'Sports'\n",
      " 'Politics' 'Sports' 'Business' 'Business' 'Business' 'Politics' 'Sports'\n",
      " 'Sports' 'Technology' 'Business' 'Sports' 'Entertainment' 'Business'\n",
      " 'Entertainment' 'Entertainment' 'Entertainment' 'Sports' 'Politics'\n",
      " 'Business' 'Politics' 'Technology' 'Entertainment' 'Business'\n",
      " 'Entertainment' 'Entertainment' 'Politics' 'Business' 'Entertainment'\n",
      " 'Business' 'Sports' 'Sports' 'Business' 'Business' 'Technology' 'Sports'\n",
      " 'Business' 'Politics' 'Politics' 'Business' 'Technology' 'Politics'\n",
      " 'Business' 'Technology' 'Sports' 'Business' 'Technology' 'Entertainment'\n",
      " 'Business' 'Technology' 'Business' 'Sports' 'Business' 'Entertainment'\n",
      " 'Business' 'Entertainment' 'Sports' 'Sports' 'Politics' 'Entertainment'\n",
      " 'Politics' 'Business' 'Sports' 'Technology' 'Entertainment' 'Business'\n",
      " 'Technology' 'Business' 'Technology' 'Entertainment' 'Technology'\n",
      " 'Politics' 'Technology' 'Politics' 'Technology' 'Sports' 'Entertainment'\n",
      " 'Business' 'Technology' 'Business' 'Technology' 'Politics' 'Politics'\n",
      " 'Entertainment' 'Business' 'Entertainment' 'Business' 'Business' 'Sports'\n",
      " 'Sports' 'Business' 'Technology' 'Technology' 'Business' 'Sports'\n",
      " 'Entertainment' 'Entertainment' 'Technology' 'Technology' 'Technology'\n",
      " 'Sports' 'Technology' 'Politics' 'Technology' 'Business' 'Politics'\n",
      " 'Politics' 'Entertainment' 'Politics' 'Business' 'Entertainment'\n",
      " 'Entertainment' 'Sports' 'Politics' 'Entertainment' 'Business' 'Business'\n",
      " 'Politics' 'Technology' 'Technology' 'Technology' 'Entertainment'\n",
      " 'Sports' 'Entertainment' 'Technology' 'Business' 'Entertainment'\n",
      " 'Politics' 'Business' 'Sports' 'Sports' 'Politics' 'Sports' 'Technology'\n",
      " 'Business' 'Politics' 'Business' 'Business' 'Entertainment' 'Politics'\n",
      " 'Technology' 'Politics' 'Politics' 'Technology' 'Business' 'Technology'\n",
      " 'Business' 'Sports' 'Business' 'Technology' 'Entertainment'\n",
      " 'Entertainment' 'Business' 'Business' 'Entertainment' 'Business' 'Sports'\n",
      " 'Technology' 'Technology' 'Technology' 'Sports' 'Sports' 'Entertainment'\n",
      " 'Business' 'Entertainment' 'Entertainment' 'Entertainment' 'Technology'\n",
      " 'Business' 'Politics' 'Business' 'Entertainment' 'Entertainment'\n",
      " 'Business' 'Entertainment' 'Entertainment' 'Business' 'Technology'\n",
      " 'Sports' 'Sports' 'Business' 'Business' 'Business' 'Politics'\n",
      " 'Technology' 'Technology' 'Business' 'Technology' 'Business' 'Business'\n",
      " 'Sports' 'Business' 'Technology' 'Business' 'Sports' 'Sports' 'Sports'\n",
      " 'Entertainment' 'Business' 'Entertainment' 'Sports' 'Entertainment'\n",
      " 'Technology' 'Politics' 'Politics' 'Politics' 'Sports' 'Sports'\n",
      " 'Entertainment' 'Technology' 'Business' 'Entertainment' 'Business'\n",
      " 'Sports' 'Business' 'Business' 'Business' 'Politics' 'Entertainment'\n",
      " 'Entertainment' 'Technology' 'Business' 'Business' 'Sports' 'Sports'\n",
      " 'Business' 'Business' 'Sports' 'Politics' 'Technology' 'Entertainment'\n",
      " 'Politics' 'Business' 'Business' 'Technology' 'Entertainment' 'Politics'\n",
      " 'Technology' 'Technology' 'Business' 'Politics' 'Business' 'Politics'\n",
      " 'Politics' 'Politics' 'Politics' 'Business' 'Sports' 'Politics'\n",
      " 'Politics' 'Business' 'Technology' 'Business' 'Entertainment' 'Politics'\n",
      " 'Business' 'Politics' 'Business' 'Business' 'Entertainment' 'Business'\n",
      " 'Sports' 'Sports' 'Business' 'Sports' 'Politics' 'Entertainment'\n",
      " 'Technology' 'Sports' 'Technology' 'Business' 'Sports' 'Business'\n",
      " 'Technology' 'Business' 'Entertainment' 'Business' 'Politics' 'Politics'\n",
      " 'Business' 'Entertainment' 'Politics' 'Politics' 'Technology'\n",
      " 'Technology' 'Business' 'Business' 'Politics' 'Entertainment' 'Business'\n",
      " 'Technology' 'Sports' 'Politics' 'Sports' 'Business' 'Business'\n",
      " 'Business' 'Sports' 'Business' 'Sports' 'Politics' 'Technology'\n",
      " 'Entertainment' 'Entertainment' 'Politics' 'Technology' 'Technology'\n",
      " 'Sports' 'Business' 'Sports' 'Sports' 'Business' 'Technology' 'Politics'\n",
      " 'Business' 'Business' 'Entertainment' 'Sports' 'Business' 'Business'\n",
      " 'Sports' 'Business' 'Entertainment' 'Business' 'Entertainment' 'Business'\n",
      " 'Business' 'Technology' 'Technology' 'Politics' 'Sports' 'Sports'\n",
      " 'Sports' 'Technology' 'Business' 'Sports' 'Politics' 'Politics'\n",
      " 'Politics' 'Politics' 'Business' 'Entertainment' 'Sports' 'Business'\n",
      " 'Business' 'Politics' 'Business' 'Entertainment' 'Sports' 'Politics'\n",
      " 'Entertainment' 'Technology' 'Business' 'Business' 'Politics'\n",
      " 'Technology' 'Business' 'Business' 'Business' 'Technology' 'Sports'\n",
      " 'Business' 'Sports' 'Business' 'Entertainment' 'Business' 'Business'\n",
      " 'Sports' 'Entertainment' 'Sports' 'Politics' 'Sports' 'Entertainment'\n",
      " 'Technology' 'Sports' 'Sports' 'Business' 'Business' 'Sports'\n",
      " 'Technology' 'Technology' 'Technology' 'Technology' 'Business'\n",
      " 'Technology' 'Sports' 'Sports' 'Technology' 'Business' 'Sports'\n",
      " 'Business' 'Politics' 'Business' 'Sports' 'Business' 'Sports' 'Business'\n",
      " 'Sports' 'Technology' 'Business' 'Politics' 'Business' 'Entertainment'\n",
      " 'Sports' 'Politics' 'Entertainment' 'Entertainment' 'Business'\n",
      " 'Technology' 'Politics' 'Technology' 'Business' 'Technology'\n",
      " 'Entertainment' 'Sports' 'Politics' 'Entertainment' 'Sports' 'Politics'\n",
      " 'Politics' 'Technology' 'Entertainment' 'Politics' 'Sports' 'Sports'\n",
      " 'Politics' 'Business' 'Business' 'Technology' 'Technology'\n",
      " 'Entertainment' 'Technology' 'Entertainment' 'Sports' 'Politics' 'Sports'\n",
      " 'Sports' 'Business' 'Business' 'Politics' 'Business' 'Sports'\n",
      " 'Technology' 'Politics' 'Sports' 'Entertainment' 'Business' 'Business'\n",
      " 'Business' 'Technology' 'Entertainment' 'Business' 'Entertainment'\n",
      " 'Business' 'Sports' 'Technology' 'Business' 'Business' 'Business'\n",
      " 'Technology' 'Technology' 'Sports' 'Sports' 'Business' 'Politics'\n",
      " 'Entertainment' 'Technology' 'Business' 'Entertainment' 'Business'\n",
      " 'Technology' 'Business' 'Technology' 'Business' 'Sports' 'Politics'\n",
      " 'Sports' 'Business' 'Business' 'Technology' 'Technology' 'Technology'\n",
      " 'Politics' 'Technology' 'Business' 'Business' 'Politics' 'Sports'\n",
      " 'Politics' 'Politics' 'Sports' 'Technology' 'Business' 'Entertainment'\n",
      " 'Entertainment' 'Technology' 'Business' 'Technology' 'Entertainment'\n",
      " 'Politics' 'Sports' 'Business' 'Sports' 'Technology' 'Business'\n",
      " 'Politics' 'Politics' 'Politics' 'Politics' 'Business' 'Entertainment'\n",
      " 'Politics' 'Business' 'Technology' 'Business' 'Sports' 'Politics'\n",
      " 'Business' 'Business' 'Business' 'Technology' 'Technology' 'Politics'\n",
      " 'Politics' 'Business' 'Business' 'Business' 'Entertainment'\n",
      " 'Entertainment' 'Sports' 'Technology' 'Business' 'Business' 'Sports'\n",
      " 'Business' 'Business' 'Business' 'Entertainment' 'Sports' 'Technology'\n",
      " 'Entertainment' 'Sports' 'Technology' 'Technology' 'Technology' 'Sports'\n",
      " 'Politics' 'Business' 'Business' 'Business' 'Sports' 'Technology'\n",
      " 'Sports' 'Business' 'Entertainment' 'Technology' 'Sports' 'Technology'\n",
      " 'Business' 'Business' 'Entertainment' 'Entertainment' 'Business'\n",
      " 'Technology' 'Entertainment' 'Business' 'Business' 'Business' 'Business'\n",
      " 'Entertainment' 'Entertainment' 'Entertainment' 'Business' 'Sports'\n",
      " 'Politics' 'Business' 'Entertainment' 'Politics' 'Business' 'Politics'\n",
      " 'Technology' 'Entertainment' 'Sports' 'Business' 'Politics' 'Technology'\n",
      " 'Technology' 'Business' 'Business' 'Entertainment' 'Entertainment'\n",
      " 'Business' 'Sports' 'Politics' 'Entertainment' 'Technology'\n",
      " 'Entertainment' 'Politics' 'Sports' 'Sports' 'Sports' 'Technology'\n",
      " 'Politics' 'Politics' 'Entertainment' 'Entertainment' 'Technology'\n",
      " 'Business' 'Entertainment' 'Technology' 'Technology' 'Technology'\n",
      " 'Sports' 'Sports' 'Entertainment' 'Sports' 'Sports' 'Technology'\n",
      " 'Business' 'Sports' 'Business' 'Politics' 'Technology' 'Entertainment'\n",
      " 'Entertainment' 'Entertainment' 'Entertainment' 'Technology' 'Politics'\n",
      " 'Politics' 'Technology' 'Politics' 'Sports' 'Politics' 'Politics'\n",
      " 'Politics' 'Business' 'Technology' 'Entertainment' 'Politics' 'Politics'\n",
      " 'Politics' 'Business' 'Sports' 'Entertainment' 'Entertainment' 'Business'\n",
      " 'Technology' 'Politics' 'Politics' 'Entertainment' 'Business' 'Politics'\n",
      " 'Business' 'Politics' 'Entertainment' 'Sports' 'Technology' 'Politics'\n",
      " 'Business' 'Entertainment' 'Business' 'Entertainment' 'Sports' 'Business'\n",
      " 'Entertainment' 'Entertainment' 'Technology' 'Entertainment'\n",
      " 'Entertainment' 'Sports' 'Entertainment' 'Politics' 'Business' 'Sports'\n",
      " 'Entertainment' 'Technology' 'Technology' 'Politics' 'Sports'\n",
      " 'Entertainment' 'Technology' 'Technology' 'Entertainment' 'Technology'\n",
      " 'Technology' 'Business' 'Business' 'Politics' 'Business' 'Sports'\n",
      " 'Technology' 'Entertainment' 'Sports' 'Politics' 'Entertainment'\n",
      " 'Technology' 'Sports' 'Politics' 'Entertainment' 'Entertainment' 'Sports'\n",
      " 'Technology' 'Technology' 'Business' 'Entertainment' 'Business' 'Sports'\n",
      " 'Business' 'Technology' 'Sports' 'Sports' 'Technology' 'Politics'\n",
      " 'Technology' 'Business' 'Entertainment' 'Business' 'Entertainment'\n",
      " 'Sports']\n",
      "<class 'numpy.ndarray'>\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.14285714 0.42857143 0.14285714 0.         0.28571429]\n",
      " [0.         0.         0.57142857 0.28571429 0.14285714]\n",
      " [0.28571429 0.42857143 0.         0.         0.28571429]\n",
      " ...\n",
      " [0.28571429 0.         0.28571429 0.14285714 0.28571429]\n",
      " [0.         0.42857143 0.14285714 0.14285714 0.28571429]\n",
      " [0.         0.         0.         0.71428571 0.28571429]]\n"
     ]
    },
    {
     "ename": "AttributeError",
     "evalue": "'numpy.ndarray' object has no attribute 'to_csv'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-13-459d7bc8779e>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m    175\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    176\u001b[0m \u001b[0mcnngov_samples\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mto_csv\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'000cnngov_samples.csv'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 177\u001b[1;33m \u001b[0membeddingLabels\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mto_csv\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'000embeddingLabels.csv'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m: 'numpy.ndarray' object has no attribute 'to_csv'"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns; sns.set()  # for plot styling\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import os\n",
    "import json\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "os.chdir('./BBC_Training_data/')\n",
    "os.chdir('./business/')\n",
    "\n",
    "# next, make a NP array of every txt file in the directory\n",
    "SampleList = np.array(os.listdir())\n",
    "SampleList = SampleList[ np.char.endswith(SampleList, '.txt') ] #ignore everything that's not a txt file.\n",
    "\n",
    "#make a fresh blank series.\n",
    "Business = pd.Series(dtype='str')\n",
    "\n",
    "for SampleName in SampleList:\n",
    "    with open(SampleName) as f:\n",
    "        contents = np.array(f.readlines(), dtype = 'str')\n",
    "    contents = [line.rstrip('\\n') for line in contents]\n",
    "    #contents.str.rstrip('\\n')\n",
    "    Business[SampleName] = ' '.join(contents)\n",
    "\n",
    "#print(Business.head())\n",
    "\n",
    "\n",
    "os.chdir('../')\n",
    "os.chdir('./entertainment/')\n",
    "# next, make a NP array of every txt file in the directory\n",
    "SampleList = np.array(os.listdir())\n",
    "SampleList = SampleList[ np.char.endswith(SampleList, '.txt') ] #ignore everything that's not a txt file.\n",
    "#make a fresh blank series.\n",
    "Entertainment = pd.Series(dtype='str')\n",
    "for SampleName in SampleList:\n",
    "    with open(SampleName) as f:\n",
    "        contents = np.array(f.readlines(), dtype = 'str')\n",
    "    contents = [line.rstrip('\\n') for line in contents]\n",
    "    #contents.str.rstrip('\\n')\n",
    "    Entertainment[SampleName] = ' '.join(contents)\n",
    "\n",
    "#print(Entertainment.head())\n",
    "\n",
    "\n",
    "os.chdir('../')\n",
    "os.chdir('./politics/')\n",
    "# next, make a NP array of every txt file in the directory\n",
    "SampleList = np.array(os.listdir())\n",
    "SampleList = SampleList[ np.char.endswith(SampleList, '.txt') ] #ignore everything that's not a txt file.\n",
    "#make a fresh blank series.\n",
    "Politics = pd.Series(dtype='str')\n",
    "for SampleName in SampleList:\n",
    "    with open(SampleName) as f:\n",
    "        contents = np.array(f.readlines(), dtype = 'str')\n",
    "    contents = [line.rstrip('\\n') for line in contents]\n",
    "    #contents.str.rstrip('\\n')\n",
    "    Politics[SampleName] = ' '.join(contents)\n",
    "\n",
    "#print(Politics.head())\n",
    "\n",
    "\n",
    "os.chdir('../')\n",
    "os.chdir('./sport/')\n",
    "# next, make a NP array of every txt file in the directory\n",
    "SampleList = np.array(os.listdir())\n",
    "SampleList = SampleList[ np.char.endswith(SampleList, '.txt') ] #ignore everything that's not a txt file.\n",
    "#make a fresh blank series.\n",
    "Sports = pd.Series(dtype='str')\n",
    "for SampleName in SampleList:\n",
    "    with open(SampleName) as f:\n",
    "        contents = np.array(f.readlines(), dtype = 'str')\n",
    "    contents = [line.rstrip('\\n') for line in contents]\n",
    "    #contents.str.rstrip('\\n')\n",
    "    Sports[SampleName] = ' '.join(contents)\n",
    "\n",
    "#print(Sports.head())\n",
    "\n",
    "\n",
    "os.chdir('../')\n",
    "os.chdir('./tech/')\n",
    "# next, make a NP array of every txt file in the directory\n",
    "SampleList = np.array(os.listdir())\n",
    "SampleList = SampleList[ np.char.endswith(SampleList, '.txt') ] #ignore everything that's not a txt file.\n",
    "#make a fresh blank series.\n",
    "Technology = pd.Series(dtype='str')\n",
    "for SampleName in SampleList:\n",
    "    with open(SampleName) as f:\n",
    "        contents = np.array(f.readlines(), dtype = 'str')\n",
    "    contents = [line.rstrip('\\n') for line in contents]\n",
    "    #contents.str.rstrip('\\n')\n",
    "    Technology[SampleName] = ' '.join(contents)\n",
    "\n",
    "#print(Technology.head())\n",
    "\n",
    "\n",
    "vec = TfidfVectorizer()\n",
    "X = vec.fit_transform(pd.concat([Business, Entertainment, Politics, Sports, Technology]))\n",
    "Features = pd.DataFrame(X.toarray(), columns=vec.get_feature_names())\n",
    "\n",
    "BusLab = np.full((int(len(Business)), 1), 'Business')\n",
    "EntLab = np.full((int(len(Entertainment)), 1), 'Entertainment')\n",
    "PolLab = np.full((int(len(Politics)), 1), 'Politics')\n",
    "SpoLab = np.full((int(len(Sports)), 1), 'Sports')\n",
    "TecLab = np.full((int(len(Technology)), 1), 'Technology')\n",
    "Targets = np.concatenate([BusLab, EntLab, PolLab, SpoLab, TecLab])\n",
    "\n",
    "print(Features.head())\n",
    "\n",
    "\n",
    "# Next, get the embedding data.\n",
    "\n",
    "os.chdir('../')\n",
    "os.chdir('../')\n",
    "cnn_samples0 = pd.read_csv('cnn_samples-54b19b96f3c0775b116bad527df8c7b5.csv')\n",
    "\n",
    "# Wrangling the data from strings to NP arrays.\n",
    "cnn_samples1 = np.fromstring((cnn_samples0.values[0,3]).replace('[','').replace(']',''), sep=',').reshape(1,512)\n",
    "\n",
    "# Rebuilding the DataFrame after this, with headline as index.\n",
    "for i in np.arange(1,np.shape(cnn_samples0)[0]):\n",
    "    temp = np.fromstring((cnn_samples0.values[i,3]).replace('[','').replace(']',''), sep=',').reshape(1,512)\n",
    "    cnn_samples1 = np.vstack([cnn_samples1, temp])\n",
    "cnn_samples = pd.DataFrame(cnn_samples1, index = cnn_samples0['text'])\n",
    "\n",
    "#Repeating the process for the challenge data.\n",
    "gov_samples0 = pd.read_csv('federal_samples-a586d0681e005629453435bea5b173eb.csv')\n",
    "gov_samples1 = np.fromstring((gov_samples0.values[0,3]).replace('[','').replace(']',''), sep=',').reshape(1,512)\n",
    "for i in np.arange(1,np.shape(gov_samples0)[0]):\n",
    "    temp = np.fromstring((gov_samples0.values[i,3]).replace('[','').replace(']',''), sep=',').reshape(1,512)\n",
    "    gov_samples1 = np.vstack([gov_samples1, temp])\n",
    "gov_samples = pd.DataFrame(gov_samples1, index = gov_samples0['text'])\n",
    "\n",
    "#also need to merge the two DataFrames\n",
    "cnngov_samples = pd.concat([cnn_samples, gov_samples], axis = 'rows')\n",
    "\n",
    "\n",
    "# TF-IDF word frequency counter (for articles)\n",
    "vec2 = TfidfVectorizer()\n",
    "embeddingText = vec2.fit_transform(cnngov_samples.index.values)\n",
    "embeddingX = pd.DataFrame(embeddingText.toarray(), columns=vec2.get_feature_names())\n",
    "\n",
    "\n",
    "# Need to make a common set of word columns for the Features and embeddingX TF-IDF dataframes (i.e. union).\n",
    "\n",
    "rowOfZeros1 = pd.DataFrame(0, index = ['0'], columns = embeddingX.columns.values)\n",
    "rowOfZeros2 = pd.DataFrame(0, index = ['0'], columns = Features.columns.values)\n",
    "\n",
    "Features = pd.concat([Features, rowOfZeros1]).fillna(0)\n",
    "embeddingX = pd.concat([embeddingX, rowOfZeros2]).fillna(0)\n",
    "\n",
    "Features = Features.drop(index = '0')\n",
    "embeddingX = embeddingX.drop(index = '0')\n",
    "\n",
    "#Let's try k-NN classification.\n",
    "\n",
    "TextClassifier = KNeighborsClassifier(n_neighbors=7)\n",
    "TextClassifier.fit(Features, np.ravel(Targets))\n",
    "\n",
    "#Need to cross-validate this model.\n",
    "\n",
    "print(cross_val_score(TextClassifier, Features, np.ravel(Targets), cv=5))\n",
    "\n",
    "# finally, use the KNN model previously constructed to assign (predict) genre labels.\n",
    "\n",
    "embeddingLabels = TextClassifier.predict(embeddingX)\n",
    "\n",
    "print(embeddingLabels)\n",
    "print(type(embeddingLabels))\n",
    "\n",
    "print(TextClassifier.predict_proba(embeddingX))\n",
    "\n",
    "cnngov_samples.to_csv('000cnngov_samples.csv')\n",
    "pd.DataFrame(embeddingLabels, columns = ['Targets']).to_csv('000embeddingLabels.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Phase B: Genre Classifier for the Embeddings\n",
    "\n",
    "First, import the CSV file generated by the previous cell.\n",
    "\n",
    "Second, train and cross-validate a supervised learning model for label generation.\n",
    "\n",
    "Third, once the model is satisfactory, predict the genres of the five challenge articles to test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "__init__() got an unexpected keyword argument 'n'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-18-92ae8dd6d88e>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     46\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     47\u001b[0m \u001b[0mModel\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mCustomBayesClassifier\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 48\u001b[1;33m \u001b[0mModel\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mFeat\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mTarg\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     49\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     50\u001b[0m \u001b[1;31m#Will cross-validate once I work out all the bugs.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-18-92ae8dd6d88e>\u001b[0m in \u001b[0;36mfit\u001b[1;34m(self, X, y)\u001b[0m\n\u001b[0;32m     26\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mclasses_\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msort\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0munique\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0my\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     27\u001b[0m         \u001b[0mtraining_sets\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0my\u001b[0m \u001b[1;33m==\u001b[0m \u001b[0myi\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0myi\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mclasses_\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 28\u001b[1;33m         self.models_ = [GaussianMixture(n=self.n, covariance_type=self.covariance_type, random_state=self.random_state).fit(Xi)\n\u001b[0m\u001b[0;32m     29\u001b[0m                         for Xi in training_sets]\n\u001b[0;32m     30\u001b[0m         self.logpriors_ = [np.log(Xi.shape[0] / X.shape[0])\n",
      "\u001b[1;32m<ipython-input-18-92ae8dd6d88e>\u001b[0m in \u001b[0;36m<listcomp>\u001b[1;34m(.0)\u001b[0m\n\u001b[0;32m     26\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mclasses_\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msort\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0munique\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0my\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     27\u001b[0m         \u001b[0mtraining_sets\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0my\u001b[0m \u001b[1;33m==\u001b[0m \u001b[0myi\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0myi\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mclasses_\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 28\u001b[1;33m         self.models_ = [GaussianMixture(n=self.n, covariance_type=self.covariance_type, random_state=self.random_state).fit(Xi)\n\u001b[0m\u001b[0;32m     29\u001b[0m                         for Xi in training_sets]\n\u001b[0;32m     30\u001b[0m         self.logpriors_ = [np.log(Xi.shape[0] / X.shape[0])\n",
      "\u001b[1;32m~\\anaconda3\\lib\\site-packages\\sklearn\\utils\\validation.py\u001b[0m in \u001b[0;36minner_f\u001b[1;34m(*args, **kwargs)\u001b[0m\n\u001b[0;32m     71\u001b[0m                           FutureWarning)\n\u001b[0;32m     72\u001b[0m         \u001b[0mkwargs\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m{\u001b[0m\u001b[0mk\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0marg\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mk\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0marg\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mzip\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msig\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mparameters\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0margs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m}\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 73\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     74\u001b[0m     \u001b[1;32mreturn\u001b[0m \u001b[0minner_f\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     75\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mTypeError\u001b[0m: __init__() got an unexpected keyword argument 'n'"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.base import BaseEstimator, ClassifierMixin\n",
    "from sklearn.mixture import GaussianMixture\n",
    "# Going to try a custom estimator based on Gaussian mixtures, based on promising results\n",
    "# from a scrapped idea at the end of the notebook I submitted at the competition.\n",
    "\n",
    "# Crucial note: Gaussian Mixtures is unsupervised, while Bayes is supervised.\n",
    "\n",
    "class CustomBayesClassifier(BaseEstimator, ClassifierMixin):\n",
    "    \"\"\"Parameters\n",
    "    ----------\n",
    "    n : int\n",
    "        Number of clusters total (a single label can have several clusters).\n",
    "    covariance_type : str\n",
    "        Controls the degrees of freedom in the shape of each cluster.\n",
    "        Three common options: 'full' (default), 'diag', or 'spherical'.\n",
    "    \"\"\"\n",
    "    def __init__(self, n, covariance_type = 'full', random_state=0):\n",
    "        self.n = n\n",
    "        self.covariance_type = covariance_type\n",
    "        self.random_state = random_state # for reproducibility.\n",
    "        \n",
    "    def fit(self, X, y):\n",
    "        self.classes_ = np.sort(np.unique(y))\n",
    "        training_sets = [X[y == yi] for yi in self.classes_]\n",
    "        self.models_ = [GaussianMixture(n=self.n, covariance_type=self.covariance_type, random_state=self.random_state).fit(Xi)\n",
    "                        for Xi in training_sets]\n",
    "        self.logpriors_ = [np.log(Xi.shape[0] / X.shape[0])\n",
    "                           for Xi in training_sets]\n",
    "        return self\n",
    "        \n",
    "    def predict_proba(self, X):\n",
    "        logprobs = np.array([model.score_samples(X)\n",
    "                             for model in self.models_]).T\n",
    "        result = np.exp(logprobs + self.logpriors_)\n",
    "        return result / result.sum(1, keepdims=True)\n",
    "        \n",
    "    def predict(self, X):\n",
    "        return self.classes_[np.argmax(self.predict_proba(X), 1)]\n",
    "\n",
    "\n",
    "Feat = pd.read_csv('000cnngov_samples.csv', header = 0, index_col = 0)\n",
    "Targ = pd.read_csv('000embeddingLabels.csv', header = 0, index_col = 0)\n",
    "\n",
    "Model = CustomBayesClassifier(n=10)\n",
    "Model.fit(Feat, Targ)\n",
    "\n",
    "#Will cross-validate once I work out all the bugs.\n",
    "\n",
    "#The bug is somewhere in the fit function.\n",
    "\n",
    "#print(cross_val_score(Model, Feat, Targ, cv=5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predicting the genre of the 5 challenge embedding and the bonus embedding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Reading the challenge data.\n",
    "challenge0 = pd.read_csv('challenge-ddec63cf66ea88f128e3c21e457f393a.csv')\n",
    "challenge1 = np.fromstring((challenge0.values[0,1]).replace('[','').replace(']',''), sep=',').reshape(1,512)\n",
    "for i in np.arange(1,np.shape(challenge0)[0]):\n",
    "    temp = np.fromstring((challenge0.values[i,1]).replace('[','').replace(']',''), sep=',').reshape(1,512)\n",
    "    challenge1 = np.vstack([challenge1, temp])\n",
    "challenge = pd.DataFrame(challenge1, index = challenge0['id'])\n",
    "\n",
    "\n",
    "#Finally, getting the bonus 6th embedding.\n",
    "with open('mystery.json') as file:\n",
    "    mystery0 = json.load(file)['embedding']\n",
    "\n",
    "mystery = pd.DataFrame(np.array(mystery0).reshape(1,512), index = ['mystery'], columns = np.arange(0,512)) #it's a dict\n",
    "\n",
    "#Need to merge these two DataFrames.\n",
    "challenge = pd.concat([challenge, mystery], axis = 'rows')\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>00</th>\n",
       "      <th>000</th>\n",
       "      <th>001</th>\n",
       "      <th>00145</th>\n",
       "      <th>00232</th>\n",
       "      <th>005</th>\n",
       "      <th>00692</th>\n",
       "      <th>007</th>\n",
       "      <th>010</th>\n",
       "      <th>01081</th>\n",
       "      <th>...</th>\n",
       "      <th>zurich</th>\n",
       "      <th>zvjezdan</th>\n",
       "      <th>zvonareva</th>\n",
       "      <th>zwally</th>\n",
       "      <th>zweig</th>\n",
       "      <th>zynga</th>\n",
       "      <th>zzzz</th>\n",
       "      <th>zzzzz</th>\n",
       "      <th>zzzzzz</th>\n",
       "      <th>état</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 27298 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   00  000  001  00145  00232  005  00692  007  010  01081  ...  zurich  \\\n",
       "0   0    0    0      0      0    0      0    0    0      0  ...       0   \n",
       "\n",
       "   zvjezdan  zvonareva  zwally  zweig  zynga  zzzz  zzzzz  zzzzzz  état  \n",
       "0         0          0       0      0      0     0      0       0     0  \n",
       "\n",
       "[1 rows x 27298 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}