{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "block_hidden": true
   },
   "outputs": [],
   "source": [
    "%load_ext rpy2.ipython\n",
    "%matplotlib inline\n",
    "\n",
    "from prophet import Prophet\n",
    "import pandas as pd\n",
    "import logging\n",
    "import warnings\n",
    "\n",
    "logging.getLogger('prophet').setLevel(logging.ERROR)\n",
    "logging.getLogger('numexpr').setLevel(logging.ERROR)\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "block_hidden": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "R[write to console]: Loading required package: Rcpp\n",
      "\n",
      "R[write to console]: Loading required package: rlang\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%R\n",
    "library(prophet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two main ways that outliers can affect Prophet forecasts. Here we make a forecast on the logged Wikipedia visits to the R page from before, but with a block of bad data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "output_hidden": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "R[write to console]: Disabling daily seasonality. Run prophet with daily.seasonality=TRUE to override this.\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 32788 characters]\n"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%R -w 10 -h 6 -u in\n",
    "df <- read.csv('https://raw.githubusercontent.com/facebook/prophet/main/examples/example_wp_log_R_outliers1.csv')\n",
    "m <- prophet(df)\n",
    "future <- make_future_dataframe(m, periods = 1096)\n",
    "forecast <- predict(m, future)\n",
    "plot(m, forecast)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 50488 characters]\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df = pd.read_csv('https://raw.githubusercontent.com/facebook/prophet/main/examples/example_wp_log_R_outliers1.csv')\n",
    "m = Prophet()\n",
    "m.fit(df)\n",
    "future = m.make_future_dataframe(periods=1096)\n",
    "forecast = m.predict(future)\n",
    "fig = m.plot(forecast)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The trend forecast seems reasonable, but the uncertainty intervals seem way too wide. Prophet is able to handle the outliers in the history, but only by fitting them with trend changes. The uncertainty model then expects future trend changes of similar magnitude.\n",
    "\n",
    "The best way to handle outliers is to remove them - Prophet has no problem with missing data. If you set their values to `NA` in the history but leave the dates in `future`, then Prophet will give you a prediction for their values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "output_hidden": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "R[write to console]: Disabling daily seasonality. Run prophet with daily.seasonality=TRUE to override this.\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 70288 characters]\n"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%R -w 10 -h 6 -u in\n",
    "outliers <- (as.Date(df$ds) > as.Date('2010-01-01')\n",
    "             & as.Date(df$ds) < as.Date('2011-01-01'))\n",
    "df$y[outliers] = NA\n",
    "m <- prophet(df)\n",
    "forecast <- predict(m, future)\n",
    "plot(m, forecast)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 102636 characters]\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.loc[(df['ds'] > '2010-01-01') & (df['ds'] < '2011-01-01'), 'y'] = None\n",
    "model = Prophet().fit(df)\n",
    "fig = model.plot(model.predict(future))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above example the outliers messed up the uncertainty estimation but did not impact the main forecast `yhat`. This isn't always the case, as in this example with added outliers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "output_hidden": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "R[write to console]: Disabling daily seasonality. Run prophet with daily.seasonality=TRUE to override this.\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 69484 characters]\n"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%R -w 10 -h 6 -u in\n",
    "df <- read.csv('https://raw.githubusercontent.com/facebook/prophet/main/examples/example_wp_log_R_outliers2.csv')\n",
    "m <- prophet(df)\n",
    "future <- make_future_dataframe(m, periods = 1096)\n",
    "forecast <- predict(m, future)\n",
    "plot(m, forecast)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 129092 characters]\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df = pd.read_csv('https://raw.githubusercontent.com/facebook/prophet/main/examples/example_wp_log_R_outliers2.csv')\n",
    "m = Prophet()\n",
    "m.fit(df)\n",
    "future = m.make_future_dataframe(periods=1096)\n",
    "forecast = m.predict(future)\n",
    "fig = m.plot(forecast)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here a group of extreme outliers in June 2015 mess up the seasonality estimate, so their effect reverberates into the future forever. Again the right approach is to remove them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "output_hidden": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "R[write to console]: Disabling daily seasonality. Run prophet with daily.seasonality=TRUE to override this.\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 64880 characters]\n"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%R -w 10 -h 6 -u in\n",
    "outliers <- (as.Date(df$ds) > as.Date('2015-06-01')\n",
    "             & as.Date(df$ds) < as.Date('2015-06-30'))\n",
    "df$y[outliers] = NA\n",
    "m <- prophet(df)\n",
    "forecast <- predict(m, future)\n",
    "plot(m, forecast)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 100536 characters]\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.loc[(df['ds'] > '2015-06-01') & (df['ds'] < '2015-06-30'), 'y'] = None\n",
    "m = Prophet().fit(df)\n",
    "fig = m.plot(m.predict(future))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
