{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Guided exercise: classifiers in Machine Learning\n",
        "\n",
        "This notebook was designed to accompany an introductory lecture on **supervised classification**.\n",
        "\n",
        "## Objectives\n",
        "By the end of this exercise, you should be able to:\n",
        "\n",
        "- understand the difference between **features**, **labels**, and the classifier **score**;\n",
        "- train simple classifiers on a binary problem;\n",
        "- visualize **decision boundaries**;\n",
        "- interpret the effect of the **threshold**;\n",
        "- compare **underfitting** and **overfitting**;\n",
        "- use metrics such as **accuracy**, **ROC curve**, and **AUC**.\n",
        "\n",
        "## Structure\n",
        "1. Generate a simple 2D dataset;\n",
        "2. Train a linear classifier;\n",
        "3. Inspect scores and thresholds;\n",
        "4. Compare with a decision tree;\n",
        "5. Study model complexity;\n",
        "6. Evaluate using ROC/AUC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "from sklearn.datasets import make_classification, make_moons\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.tree import DecisionTreeClassifier\n",
        "from sklearn.metrics import (\n",
        "    accuracy_score,\n",
        "    confusion_matrix,\n",
        "    roc_curve,\n",
        "    roc_auc_score,\n",
        ")\n",
        "\n",
        "np.random.seed(42)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 1. Building a classification problem\n",
        "\n",
        "Let’s start with a synthetic dataset in 2 dimensions. This is useful because we can **visualize** the problem.\n",
        "\n",
        "Each event will have:\n",
        "- two **features**: `x1` and `x2`;\n",
        "- one binary **label**: `y = 0` or `y = 1`.\n",
        "\n",
        "### Exercise 1\n",
        "Before running the cell below, try to answer:\n",
        "\n",
        "1. What will the classifier learn from this information?\n",
        "2. What would be a good decision boundary in this space?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "X, y = make_classification(\n",
        "    n_samples=800,\n",
        "    n_features=2,\n",
        "    n_redundant=0,\n",
        "    n_informative=2,\n",
        "    n_clusters_per_class=1,\n",
        "    class_sep=1.4,\n",
        "    flip_y=0.05,\n",
        "    random_state=7,\n",
        ")\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(\n",
        "    X, y, test_size=0.30, random_state=42, stratify=y\n",
        ")\n",
        "\n",
        "print(\"Train shape:\", X_train.shape)\n",
        "print(\"Test shape: \", X_test.shape)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "plt.figure(figsize=(6, 5))\n",
        "plt.scatter(X_train[:, 0], X_train[:, 1], c=y_train, s=18)\n",
        "plt.xlabel(\"x1\")\n",
        "plt.ylabel(\"x2\")\n",
        "plt.title(\"Training sample\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Discussion\n",
        "\n",
        "Notice that the two classes occupy different regions of the plane, but there is some overlap.\n",
        "\n",
        "This already illustrates an important idea:\n",
        "a classifier usually does not produce only a rigid answer, but rather a **score** that reflects how compatible an event is with each class."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 2. First model: logistic regression\n",
        "\n",
        "Despite its name, **logistic regression** is a classifier.\n",
        "\n",
        "It learns a linear combination of the features and produces a number between 0 and 1, which can be interpreted as a score/probability for the positive class.\n",
        "\n",
        "**Accuracy** is defined as the fraction of correctly classified events:\n",
        "$$\n",
        "\\mathrm{accuracy} = \\frac{\\mathrm{number\\ of\\ correct\\ predictions}}{\\mathrm{total\\ number\\ of\\ predictions}}\n",
        "$$\n",
        "\n",
        "### Exercise 2\n",
        "- Train the model;\n",
        "- obtain predictions on the test set;\n",
        "- compute the accuracy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "log_reg = LogisticRegression()\n",
        "log_reg.fit(X_train, y_train)\n",
        "\n",
        "y_pred = log_reg.predict(X_test)\n",
        "y_score = log_reg.predict_proba(X_test)[:, 1]\n",
        "\n",
        "acc = accuracy_score(y_test, y_pred)\n",
        "print(f\"Accuracy = {acc:.3f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 3. Classifier score and threshold\n",
        "\n",
        "The classifier does not only output “class 0” or “class 1”.\n",
        "It produces a **score**.\n",
        "\n",
        "Typically:\n",
        "- if `score > 0.5`, we classify the event as class 1;\n",
        "- if `score <= 0.5`, we classify it as class 0.\n",
        "\n",
        "However, this threshold can be adjusted."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "print(\"First 10 scores in the test set:\")\n",
        "print(np.round(y_score[:10], 3))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Exercise 3\n",
        "Change the threshold and observe what happens to the confusion matrix.\n",
        "\n",
        "Questions for discussion:\n",
        "1. What happens when we increase the threshold?\n",
        "2. What happens when we decrease the threshold?\n",
        "3. Is accuracy always the best metric?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "def predict_with_threshold(scores, threshold=0.5):\n",
        "    return (scores >= threshold).astype(int)\n",
        "\n",
        "for thr in [0.3, 0.5, 0.7]:\n",
        "    pred_thr = predict_with_threshold(y_score, threshold=thr)\n",
        "    acc_thr = accuracy_score(y_test, pred_thr)\n",
        "    cm = confusion_matrix(y_test, pred_thr)\n",
        "    print(f\"\\nThreshold = {thr:.1f}\")\n",
        "    print(f\"Accuracy  = {acc_thr:.3f}\")\n",
        "    print(\"Confusion matrix:\")\n",
        "    print(cm)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 4. Visualizing the decision boundary\n",
        "\n",
        "A major advantage of 2D examples is that we can visualize the region of the space classified as class 0 or class 1."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "def plot_decision_boundary(model, X, y, title, ax=None):\n",
        "    if ax is None:\n",
        "        fig, ax = plt.subplots(figsize=(6, 5))\n",
        "\n",
        "    x_min, x_max = X[:, 0].min() - 1.0, X[:, 0].max() + 1.0\n",
        "    y_min, y_max = X[:, 1].min() - 1.0, X[:, 1].max() + 1.0\n",
        "\n",
        "    xx, yy = np.meshgrid(\n",
        "        np.linspace(x_min, x_max, 400),\n",
        "        np.linspace(y_min, y_max, 400)\n",
        "    )\n",
        "\n",
        "    grid = np.c_[xx.ravel(), yy.ravel()]\n",
        "\n",
        "    if hasattr(model, \"predict_proba\"):\n",
        "        zz = model.predict_proba(grid)[:, 1]\n",
        "        zz = zz.reshape(xx.shape)\n",
        "        contour = ax.contourf(xx, yy, zz, levels=25, alpha=0.35)\n",
        "        plt.colorbar(contour, ax=ax, label=\"score(class 1)\")\n",
        "    else:\n",
        "        zz = model.predict(grid).reshape(xx.shape)\n",
        "        ax.contourf(xx, yy, zz, levels=25, alpha=0.35)\n",
        "\n",
        "    ax.scatter(X[:, 0], X[:, 1], c=y, s=18, edgecolor=\"k\", linewidth=0.2)\n",
        "    ax.set_xlabel(\"x1\")\n",
        "    ax.set_ylabel(\"x2\")\n",
        "    ax.set_title(title)\n",
        "    return ax"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "plot_decision_boundary(\n",
        "    log_reg,\n",
        "    X_test,\n",
        "    y_test,\n",
        "    title=\"Logistic regression: decision boundary on test sample\"\n",
        ")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Discussion\n",
        "\n",
        "Logistic regression produces an approximately **linear** boundary.\n",
        "\n",
        "This works well when the separation between the classes can be approximated by a line/hyperplane, but not when the structure is more complex."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 5. Comparing with a decision tree\n",
        "\n",
        "Decision trees can learn non-linear boundaries.\n",
        "\n",
        "Let us compare two trees:\n",
        "- a **shallow** one (`max_depth=1`), which may suffer from **underfitting**;\n",
        "- a **deep** one (`max_depth=8`), which may capture more complex structures."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "tree_shallow = DecisionTreeClassifier(max_depth=1, random_state=42)\n",
        "tree_deep = DecisionTreeClassifier(max_depth=8, random_state=42)\n",
        "\n",
        "tree_shallow.fit(X_train, y_train)\n",
        "tree_deep.fit(X_train, y_train)\n",
        "\n",
        "print(\"Shallow tree test accuracy:\", round(accuracy_score(y_test, tree_shallow.predict(X_test)), 3))\n",
        "print(\"Deep tree test accuracy:   \", round(accuracy_score(y_test, tree_deep.predict(X_test)), 3))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "fig, axes = plt.subplots(1, 2, figsize=(12, 5))\n",
        "plot_decision_boundary(tree_shallow, X_test, y_test, \"Decision tree (max_depth=1)\", ax=axes[0])\n",
        "plot_decision_boundary(tree_deep, X_test, y_test, \"Decision tree (max_depth=8)\", ax=axes[1])\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Exercise 4\n",
        "Compare the two plots and answer:\n",
        "\n",
        "1. Which model looks **too simple**?\n",
        "2. Which model looks much more flexible?\n",
        "3. Does greater flexibility always imply better generalization?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 6. Underfitting vs overfitting\n",
        "\n",
        "Now let us vary the tree depth and compare the performance on the **training** and **test** sets.\n",
        "\n",
        "This is a classic diagnostic:\n",
        "- if both training and test performance are poor: possible **underfitting**;\n",
        "- if training becomes very good but test performance worsens: possible **overfitting**."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "depths = range(1, 16)\n",
        "train_acc = []\n",
        "test_acc = []\n",
        "\n",
        "for depth in depths:\n",
        "    clf = DecisionTreeClassifier(max_depth=depth, random_state=42)\n",
        "    clf.fit(X_train, y_train)\n",
        "    train_acc.append(accuracy_score(y_train, clf.predict(X_train)))\n",
        "    test_acc.append(accuracy_score(y_test, clf.predict(X_test)))\n",
        "\n",
        "plt.figure(figsize=(7, 5))\n",
        "plt.plot(list(depths), train_acc, marker=\"o\", label=\"train\")\n",
        "plt.plot(list(depths), test_acc, marker=\"o\", label=\"test\")\n",
        "plt.xlabel(\"max_depth\")\n",
        "plt.ylabel(\"accuracy\")\n",
        "plt.title(\"Model complexity scan\")\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Exercise 5\n",
        "Look at the plot above and identify:\n",
        "\n",
        "1. In which region does the tree appear to suffer from underfitting?\n",
        "2. In which region does overfitting start to appear?\n",
        "3. Which depth seems like a good compromise?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 7. ROC curve and AUC\n",
        "\n",
        "A more complete way to evaluate a classifier is to vary the threshold continuously.\n",
        "\n",
        "The ROC curve shows:\n",
        "- **TPR** (true positive rate) on the y-axis;\n",
        "- **FPR** (false positive rate) on the x-axis.\n",
        "\n",
        "The **AUC** summarizes this curve into a single number:\n",
        "- `AUC = 0.5`: random performance;\n",
        "- `AUC = 1.0`: perfect separation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "fpr_log, tpr_log, _ = roc_curve(y_test, y_score)\n",
        "auc_log = roc_auc_score(y_test, y_score)\n",
        "\n",
        "tree_score = tree_deep.predict_proba(X_test)[:, 1]\n",
        "fpr_tree, tpr_tree, _ = roc_curve(y_test, tree_score)\n",
        "auc_tree = roc_auc_score(y_test, tree_score)\n",
        "\n",
        "plt.figure(figsize=(6, 6))\n",
        "plt.plot(fpr_log, tpr_log, label=f\"Logistic regression (AUC = {auc_log:.3f})\")\n",
        "plt.plot(fpr_tree, tpr_tree, label=f\"Decision tree depth=8 (AUC = {auc_tree:.3f})\")\n",
        "plt.plot([0, 1], [0, 1], \"--\", label=\"random classifier\")\n",
        "plt.xlabel(\"False Positive Rate\")\n",
        "plt.ylabel(\"True Positive Rate\")\n",
        "plt.title(\"ROC curve\")\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Exercise 6\n",
        "Answer:\n",
        "\n",
        "1. Why is the ROC curve more informative than looking at a single threshold?\n",
        "2. In which situations is AUC particularly useful?\n",
        "3. Does a model with better accuracy necessarily have a better AUC?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 8. Non-linear example: two moons\n",
        "\n",
        "Now we will use a dataset where a linear boundary is **not** sufficient."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "X_moon, y_moon = make_moons(n_samples=800, noise=0.22, random_state=12)\n",
        "\n",
        "X_train_m, X_test_m, y_train_m, y_test_m = train_test_split(\n",
        "    X_moon, y_moon, test_size=0.30, random_state=42, stratify=y_moon\n",
        ")\n",
        "\n",
        "log_reg_m = LogisticRegression()\n",
        "tree_m = DecisionTreeClassifier(max_depth=5, random_state=42)\n",
        "\n",
        "log_reg_m.fit(X_train_m, y_train_m)\n",
        "tree_m.fit(X_train_m, y_train_m)\n",
        "\n",
        "print(\"Logistic regression accuracy:\", round(accuracy_score(y_test_m, log_reg_m.predict(X_test_m)), 3))\n",
        "print(\"Decision tree accuracy:      \", round(accuracy_score(y_test_m, tree_m.predict(X_test_m)), 3))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "fig, axes = plt.subplots(1, 2, figsize=(12, 5))\n",
        "plot_decision_boundary(log_reg_m, X_test_m, y_test_m, \"Logistic regression on moons\", ax=axes[0])\n",
        "plot_decision_boundary(tree_m, X_test_m, y_test_m, \"Decision tree on moons\", ax=axes[1])\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "bf18bd83",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Get scores\n",
        "log_score_m = log_reg_m.predict_proba(X_test_m)[:, 1]\n",
        "tree_score_m = tree_m.predict_proba(X_test_m)[:, 1]\n",
        "\n",
        "# Plot distributions\n",
        "plt.figure(figsize=(8, 5))\n",
        "\n",
        "plt.hist(log_score_m[y_test_m == 0], bins=30, alpha=0.5, label=\"LogReg - class 0\", density=True)\n",
        "plt.hist(log_score_m[y_test_m == 1], bins=30, alpha=0.5, label=\"LogReg - class 1\", density=True)\n",
        "\n",
        "plt.hist(tree_score_m[y_test_m == 0], bins=30, alpha=0.5, linestyle=\"dashed\", label=\"Tree - class 0\", density=True)\n",
        "plt.hist(tree_score_m[y_test_m == 1], bins=30, alpha=0.5, linestyle=\"dashed\", label=\"Tree - class 1\", density=True)\n",
        "\n",
        "plt.xlabel(\"Classifier score\")\n",
        "plt.ylabel(\"Density\")\n",
        "plt.title(\"Score distributions (two moons)\")\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7996d4f7",
      "metadata": {},
      "outputs": [],
      "source": [
        "from sklearn.metrics import roc_curve, roc_auc_score\n",
        "\n",
        "fpr_log_m, tpr_log_m, _ = roc_curve(y_test_m, log_score_m)\n",
        "auc_log_m = roc_auc_score(y_test_m, log_score_m)\n",
        "\n",
        "fpr_tree_m, tpr_tree_m, _ = roc_curve(y_test_m, tree_score_m)\n",
        "auc_tree_m = roc_auc_score(y_test_m, tree_score_m)\n",
        "\n",
        "plt.figure(figsize=(6, 6))\n",
        "plt.plot(fpr_log_m, tpr_log_m, label=f\"Logistic regression (AUC = {auc_log_m:.3f})\")\n",
        "plt.plot(fpr_tree_m, tpr_tree_m, label=f\"Decision tree (AUC = {auc_tree_m:.3f})\")\n",
        "plt.plot([0, 1], [0, 1], \"--\", label=\"random classifier\")\n",
        "\n",
        "plt.xlabel(\"False Positive Rate\")\n",
        "plt.ylabel(\"True Positive Rate\")\n",
        "plt.title(\"ROC curve (two moons)\")\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4ee27a54",
      "metadata": {},
      "source": [
        "### Exercise 7\n",
        "Answer:\n",
        "\n",
        "1. By looking at the score distributions, which model achieves better separation between the two classes? How can you tell?\n",
        "\n",
        "2. Compare the ROC curves of the two models. Which one performs better, and why?\n",
        "\n",
        "3. Why does logistic regression perform worse in this example, even though it worked well in the previous (linear) case?\n",
        "\n",
        "4. How does the geometry of the data (linear vs non-linear) influence the choice of classifier?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Final discussion of this section\n",
        "\n",
        "This example makes it clear that:\n",
        "- the **same dataset** may favor different architectures/models;\n",
        "- the choice of classifier depends on the geometry of the problem;\n",
        "- a simpler model may be preferable when it already describes the data well."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 9. Final challenges\n",
        "\n",
        "### Challenge A\n",
        "Replace the decision tree with others using `max_depth=2`, `4`, `10`.\n",
        "What changes in the decision boundaries and in the metrics?\n",
        "\n",
        "### Challenge B\n",
        "Change the `class_sep` parameter in `make_classification`.\n",
        "How does this affect the separation between the classes?\n",
        "\n",
        "### Challenge C\n",
        "Add more noise (`flip_y` or `noise`) and observe how the performance degrades.\n",
        "\n",
        "### Challenge D\n",
        "Test an additional classifier, for example:\n",
        "- `KNeighborsClassifier`\n",
        "- `RandomForestClassifier`\n",
        "- `MLPClassifier`\n",
        "\n",
        "and compare it with the models already studied."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 10. Conceptual summary\n",
        "\n",
        "In this notebook, we saw that:\n",
        "\n",
        "- supervised classification uses **features** to predict **labels**;\n",
        "- the classifier produces a **score**, not just a discrete class;\n",
        "- the **threshold** controls the trade-off between different types of errors;\n",
        "- simple models may suffer from **underfitting**;\n",
        "- highly flexible models may suffer from **overfitting**;\n",
        "- **ROC/AUC** help evaluate the classifier in a more complete way;\n",
        "- the geometry of the problem influences which model works best.\n",
        "\n",
        "---\n",
        "\n",
        "## Takeaway \n",
        "\n",
        "A good question to end with is:\n",
        "\n",
        "> “When I see a high accuracy, is that alone enough to trust the classifier?”\n",
        "\n",
        "In general, the answer is **no**. We always need to consider the context:\n",
        "- class distribution;\n",
        "- score;\n",
        "- threshold;\n",
        "- generalization;\n",
        "- and complementary metrics."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1e47f41c",
      "metadata": {},
      "source": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "rare",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12.12"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}