data-collection-by-survey.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stack Overflow Developer Survey 2019 - Data Collection through a survey\n",
    "\n",
    "The dataset collected through a survey conducted on the Stack Overflow website in 2019 and available in .csv format was subjected to a Data Wrangling process. This involved various tasks such as identifying and removing duplicate rows, finding and imputing missing values, removing outliers and normalizing the data. The dataset also includes comparisons of two years."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Wrangling\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/LargeData/m1_survey_data.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> Finding duplicates </b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "154"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.duplicated(keep = 'first').sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> Removing duplicates </b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.drop_duplicates(ignore_index = True, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# verifying whether the duplicates were actually dropped\n",
    "df.duplicated(keep = 'first').sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(11398, 85)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<bound method IndexOpsMixin.nunique of 0            4\n",
       "1            9\n",
       "2           13\n",
       "3           16\n",
       "4           17\n",
       "         ...  \n",
       "11393    25136\n",
       "11394    25137\n",
       "11395    25138\n",
       "11396    25141\n",
       "11397    25142\n",
       "Name: Respondent, Length: 11398, dtype: int64>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"Respondent\"].nunique"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> Finding Missing Values</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Respondent        0\n",
       "MainBranch        0\n",
       "Hobbyist          0\n",
       "OpenSourcer       0\n",
       "OpenSource       81\n",
       "               ... \n",
       "Sexuality       542\n",
       "Ethnicity       675\n",
       "Dependents      140\n",
       "SurveyLength     19\n",
       "SurveyEase       14\n",
       "Length: 85, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Finding missing values for all columns\n",
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Finding how many rows are missing in the column 'WorkLoc'\n",
    "df['WorkLoc'].isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Imputing Missing Values</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Office                                            6806\n",
       "Home                                              3589\n",
       "Other place, such as a coworking space or cafe     971\n",
       "Name: WorkLoc, dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Finding values counts for the column WorkLoc\n",
    "df[\"WorkLoc\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Identifying the value that is most frequent (majority) in the WorkLoc column.\n",
    "# --> Office\n",
    "\n",
    "## Imputing (Replacing) all empty rows in the column WorkLoc with the value that\n",
    "## held the most majorit ('Office').\n",
    "\n",
    "df[\"WorkLoc\"].fillna(value = \"Office\", inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Verifying the imputation operation's success\n",
    "df['WorkLoc'].isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> Normalizing data </b>\n",
    "\n",
    "There are two columns in the dataset that talks about compensation. <br>\n",
    "\n",
    "1. \"CompFreq\", which shows how often a developer is paid (Yearly, Monthly or Weekly).\n",
    "2. \"CompTotal\", which shows how much a developer is paid per Year, Month or Week - depending on their \"CompFreq\"\n",
    "\n",
    "=> this will make it difficutl to compare the total compensation of the developers, thus: <br>\n",
    "\n",
    "A new column will be created \"NormalizedAnnualCompensation' which will contain 'Annual Compensation' irrespective of the 'CompFreq'.\n",
    "\n",
    "Creating this column, will make it easier to compare salaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Yearly', 'Monthly', 'Weekly', nan], dtype=object)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Listing various categories in the column 'CompFreq'\n",
    "df[\"CompFreq\"].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0     6073\n",
       "12.0    4788\n",
       "52.0     331\n",
       "Name: CompFreq, dtype: int64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Creating a new column named \"NormalizedAnnualCompensation\"\n",
    "df[\"CompFreq\"].replace(to_replace = \"Yearly\", value = 1, inplace = True)\n",
    "df[\"CompFreq\"].replace(to_replace = \"Monthly\", value = 12, inplace = True)\n",
    "df[\"CompFreq\"].replace(to_replace = \"Weekly\", value = 52, inplace = True)\n",
    "df[\"CompFreq\"].unique()\n",
    "df[\"CompFreq\"].value_counts()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0         61000.0\n",
       "1        138000.0\n",
       "2         90000.0\n",
       "3        348000.0\n",
       "4         90000.0\n",
       "           ...   \n",
       "11393    130000.0\n",
       "11394     74400.0\n",
       "11395    105000.0\n",
       "11396     80000.0\n",
       "11397         NaN\n",
       "Name: NormalizedAnnualCompensation, Length: 11398, dtype: float64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['NormalizedAnnualCompensation'] = df[\"CompTotal\"] * df[\"CompFreq\"]\n",
    "df['NormalizedAnnualCompensation']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "81794d4967e6c3204c66dcd87b604927b115b27c00565d3d43f05ba2f3a2cb0d"
  },
  "kernelspec": {
   "display_name": "Python 3.9.7 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}