prophet/notebooks/diagnostics.ipynb

505 lines
270 KiB
Text
Raw Normal View History

2017-09-02 17:53:38 +00:00
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"block_hidden": true,
"collapsed": true
2017-09-02 17:53:38 +00:00
},
"outputs": [],
"source": [
"%load_ext rpy2.ipython\n",
"%matplotlib inline\n",
"from fbprophet import Prophet\n",
"import pandas as pd\n",
"from matplotlib import pyplot as plt\n",
"import logging\n",
"logging.getLogger('fbprophet').setLevel(logging.ERROR)\n",
"import warnings\n",
"warnings.filterwarnings(\"ignore\")\n",
"df = pd.read_csv('../examples/example_wp_log_peyton_manning.csv')\n",
2017-09-02 17:53:38 +00:00
"m = Prophet()\n",
"m.fit(df)\n",
"future = m.make_future_dataframe(periods=366)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"block_hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"Initial log joint probability = -19.4685\n",
2017-09-02 17:53:38 +00:00
"Optimization terminated normally: \n",
" Convergence detected: relative gradient magnitude is below tolerance\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"%%R\n",
"library(prophet)\n",
"df <- read.csv('../examples/example_wp_log_peyton_manning.csv')\n",
2017-09-02 17:53:38 +00:00
"m <- prophet(df)\n",
"future <- make_future_dataframe(m, periods=366)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Prophet includes functionality for time series cross validation to measure forecast error using historical data. This is done by selecting cutoff points in the history, and for each of them fitting the model using data only up to that cutoff point. We can then compare the forecasted values to the actual values. This figure illustrates a simulated historical forecast on the Peyton Manning dataset, where the model was fit to a initial history of 5 years, and a forecast was made on a one year horizon."
]
},
{
"cell_type": "code",
"execution_count": 3,
2017-09-02 20:07:49 +00:00
"metadata": {
"input_hidden": true
},
2017-09-02 17:53:38 +00:00
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAAF3CAYAAADtkpxQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzsnXt8FNXd/z+zs5uAAiKRCEggtSgg\nBBMIyLYVV1FQFMUnrdX2aVDQKIpKbWulPjxPqJb4w0dNi1QIAg+rrZcaKqAiymURzWoIFwHv2sag\n3AMRELKXmfP7Y3MmM7Ozt2Qvs7vf9+vlS7KXmTk7Z875nO/3e75fgTHGQBAEQRAEQSQVS6ovgCAI\ngiAIIhshEUYQBEEQBJECSIQRBEEQBEGkABJhBEEQBEEQKYBEGEEQBEEQRAogEUYQBEEQBJECSIQR\nBEEQBEGkABJhBEEQBEEQKYBEGEEQBEEQRAogEUYQBEEQBJECrKm+gGg455xzUFhYmOrLIAiCyChO\nnDih+bt79+4pupLkk81tJxJPY2Mjjhw5EvFzaSHCCgsL0dDQkOrLIAiCyChcLpfmb4fDkZLrSAXZ\n3HYi8ZSWlkb1OXJHEgRBEARBpAASYQRBEARBECmARBhBEARBEEQKSJgImzZtGvLz8zF8+HDltTlz\n5mDEiBEoLi7GhAkTsG/fvkSdniAIgiAIwtQkTITdeuutePPNNzWv/e53v8OuXbuwc+dOXHfddfjj\nH/+YqNMTBEEQBEGYmoSJsHHjxqFXr16a13r06KH8+/vvv4cgCIk6PUEQBEEQhKlJeoqKhx9+GE6n\nE2eddRY2bdoU8nM1NTWoqakBABw+fDhZl0cQBEFkAZSSgjADSQ/M/9Of/oS9e/fil7/8JZ5++umQ\nn6uoqEBDQwMaGhrQu3fvJF4hQRAEQRBE4knZ7shf/vKXqK2tTdXpCYIgCIIgUkpSRdgXX3yh/HvV\nqlUYMmRIMk9PEARBEARhGhIWE3bLLbfA5XLhyJEj6N+/P+bOnYs33ngDn332GSwWCwYOHIhFixYl\n6vQEQRAEERVutxsulwsOhwN2uz3Vl0NkEQkTYS+88ELQa9OnT0/U6QiCIAgiZtxuN8aPHw+v14uc\nnBxs2LCBhBiRNChjPkEQBJF1fPbZZ/jss8+wceNGDBw4EJIkwev1BhX2JohEQiKMIAiCyDr279+P\n/fv3Y8CAAejfvz9EUUROTg6lriCSStLzhBEEQRCEWSgoKEB5eTmuuOIKigkjkg6JMIIgCCKrKSgo\nwK9+9atUXwaRhZA7kiAIgiAIIgWQCCMIglDhdrtRVVUFt9ud6kshCCLDIXckQRBEG5SugCCIZEKW\nMIIgiDZcLhe8Xi+lKyAIIimQCCMIgmjD4XAgJyeH0hUQBJEUyB1JEATRht1ux4YNG6iEDUEQSYFE\nGEEQhAq73U7iiyCIpEDuSIIgCIIgiBRAIowgCIIgCCIFkAgjCIIgCIJIASTCCIIgCIIgUgAF5hME\nQRBZx6hRo1J9CQRBIowgCILIPrp3757qSyAIckcSBEEQBEGkAhJhBEEQBEEQKYBEGEEQBEEQRAog\nEUYQBEEQBJECKDCfIAiCyDr27dun+btfv34puhIimyERRhA6ampqUFNTAwCoqKhARUVFTN/ft28f\nrr/+egDAddddh8rKSgDA559/DpfLBQBwOBy48MILNd+bPHky9u/fj759+2LNmjUxX3dlZSVee+01\nAMDq1atpUskAmpqa8OKLL6K+vh4HDx6EIAjo3bs3SkpKcMMNN6CoqCim4+3bt0/pI/FI0fD3v/8d\nr7zyCg4cOACv14tu3bopfTzce2bg888/1/xNzwuRCkiEEUSS+OyzzxRx17dv3yARRhBqVq9ejcce\newxer1fz+tdff42vv/4ax44dwxNPPBHTMffv369ZYHSmD9bV1eHJJ5+M+T2CINohEUYQcaZfv35o\naGiI+XsdsX4RmcnWrVvx6KOPQpZlCIKAadOmoaysDGeffTb279+PDRs2oKmpKaXX+Omnnyr/rqys\nxLXXXgtBECK+RxBEOyTCCCIK1K6+ZcuW4R//+AfeffddCIKA0tJS/P73v0deXh4AY3dkRUUFtm/f\nrhxv7ty5mDt3LgDgf/7nfzB58mRDd+Tnn3+OJUuW4IsvvsDRo0fh8Xhw1lln4eKLL8Ztt92Giy66\nKJk/A5Eknn76aciyDAC4+eabMWPGDOW9AQMG4LbbboMkSQCA0tJSAMDIkSMVK5fR6+o+DATc7idO\nnAAQ6KeTJ08GALzzzjt48cUX8cknn+D06dPIy8vDJZdcgttvv11x2fG+yqmsrERlZSVGjhyJ/fv3\nh3xPfX0EQZAII4iYuf/++5XJCwA2btyIkydP4q9//Wvcz9XY2IhNmzZpXjt69Cg2bdoEt9uN5557\nDj/4wQ/ifl4idRw9ehQfffSR8vevfvUrw8+Johj3cy9fvhwLFy7UvHbw4EGsXr0aLpcLzz77LM4/\n//y4n5cg4oHb7YbL5YLD4YDdbk/15UQFiTCCiJF+/fph/vz5kCQJt99+O44ePYr6+nocOXIE55xz\njuF3ampqsGbNmiDrVySGDBmCp59+GhdccAF69OgBn8+HtWvXoqqqCq2trVi5ciV+85vfxLV9RGpR\nW5HOPPNM5Ofnx+W4lZWVmDx5Mu68804AwTFhzc3NWLRoEYBASZ8nnngCgwcPhtPpxNKlS3H8+HE8\n8cQTWLhwIdasWaPZwLJ48WJNoH+49wgiEbjdbowfPx5erxc5OTnYsGFDWggxyhMWA263G1VVVXC7\n3am+FCKF3HXXXTjvvPMwYMAAFBcXK6+rJ894kZeXh/r6etx1111wOBwYN24cqqqqlPe//vrruJ+T\nyE4++ugjxcV57bXXYuTIkTjzzDNx5513omfPngCAhoaGoI0CBGEGXC4XvF4vJEmC1+s11U7ccJAl\nLErSVWUT8WfgwIHKv7t27ar8OxGT00MPPRRW9Le2tsb9nERq6du3r/Lv77//HocPH0bv3r1jOgYX\nU7Fw8uRJ5d99+vRR/m2xWJCfn4+WlhZIkoTvvvsu5ushiETjcDiQk5OjzNEOhyPVlxQVCbOETZs2\nDfn5+Rg+fLjy2u9+9zsMGTIEI0aMwI033oiWlpZEnT7upKvKJuKP1dq+dollx1esu8OOHz+uCLBe\nvXrh5ZdfRn19PV588cWYjkOkF7169cKwYcOUv5977jnDz3GhZbPZAGgXAd9++63hd8L1we7duyv/\nPnDggPJvWZZx6NAhAIE4tLPOOitSEwgi6djtdmzYsAGPPPJIWhlJEibCbr31Vrz55pua16666irs\n2bMHu3btwoUXXqhxq5gdrrJFUUwrlU2YB/Xk9dVXX0W0VlitVmXStFqt6NatG1paWvDMM88k9DqJ\n1HPPPffAYgkMzy+++CJqampw+PBh+P1+NDU1YdmyZXj00UcBtFvOvvzyS+zfvx9+vz9kH1H3wX//\n+9/w+XzK38OGDVOC/d944w3s3LkT33//PZYsWaIsmEePHo2cnJz4N5gg4oDdbsfs2bPTRoABCXRH\njhs3Do2NjZrXJkyYoPx77NixeOWVVxJ1+rjDVXa67bwgzMPgwYNhs9ng8/nw/PPP4/nnnwcQOrv9\nGWecgdGjR6O+vh6HDh3CpEmTAARSFBCZzZgxY/CHP/wBjz32GPx+vybQnXPZZZcBAK6++mrU1NSg\ntbUVU6ZM0Yh3PQUFBejZsydaWlrw9ttvY+XKlQCABx54AIMHD8Zdd92FhQsX4vjx47j99ts13+3R\nowceeOCBBLQ2dezduxeNjY0oLCxM9aUQWUrKAvOXLVuGa665JlWn7xDpqLIJ85Cfn4+5c+fi/PPP\nj9qa8Oijj2LChAno0aMHunXrhkmTJqWVBZnoOFOmTMGLL76In/3sZxgwYAByc3PRtWtXDBw4EDfc\ncANuvfVWAAGvwy9+8Qv
2017-09-02 17:53:38 +00:00
"text/plain": [
"<Figure size 720x432 with 1 Axes>"
2017-09-02 17:53:38 +00:00
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from fbprophet.diagnostics import cross_validation\n",
"df_cv = cross_validation(\n",
" m, '365 days', initial='1825 days', period='365 days')\n",
"cutoff = df_cv['cutoff'].unique()[0]\n",
"df_cv = df_cv[df_cv['cutoff'].values == cutoff]\n",
2017-09-02 17:53:38 +00:00
"\n",
"fig = plt.figure(facecolor='w', figsize=(10, 6))\n",
"ax = fig.add_subplot(111)\n",
"ax.plot(m.history['ds'].values, m.history['y'], 'k.')\n",
"ax.plot(df_cv['ds'].values, df_cv['yhat'], ls='-', c='#0072B2')\n",
"ax.fill_between(df_cv['ds'].values, df_cv['yhat_lower'],\n",
" df_cv['yhat_upper'], color='#0072B2',\n",
" alpha=0.2)\n",
"ax.axvline(x=pd.to_datetime(cutoff), c='gray', lw=4, alpha=0.5)\n",
2017-09-02 17:53:38 +00:00
"ax.set_ylabel('y')\n",
"ax.set_xlabel('ds')\n",
"ax.text(x=pd.to_datetime('2010-01-01'),y=12, s='Initial', color='black',\n",
" fontsize=16, fontweight='bold', alpha=0.8)\n",
"ax.text(x=pd.to_datetime('2012-08-01'),y=12, s='Cutoff', color='black',\n",
" fontsize=16, fontweight='bold', alpha=0.8)\n",
"ax.axvline(x=pd.to_datetime(cutoff) + pd.Timedelta('365 days'), c='gray', lw=4,\n",
2017-09-02 17:53:38 +00:00
" alpha=0.5, ls='--')\n",
"ax.text(x=pd.to_datetime('2013-01-01'),y=6, s='Horizon', color='black',\n",
" fontsize=16, fontweight='bold', alpha=0.8);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[The Prophet paper](https://peerj.com/preprints/3190.pdf) gives further description of simulated historical forecasts.\n",
"\n",
2017-09-02 20:07:49 +00:00
"This cross validation procedure can be done automatically for a range of historical cutoffs using the `cross_validation` function. We specify the forecast horizon (`horizon`), and then optionally the size of the initial training period (`initial`) and the spacing between cutoff dates (`period`). By default, the initial training period is set to three times the horizon, and cutoffs are made every half a horizon.\n",
2017-09-02 17:53:38 +00:00
"\n",
"The output of `cross_validation` is a dataframe with the true values `y` and the out-of-sample forecast values `yhat`, at each simulated forecast date and for each cutoff date. In particular, a forecast is made for every observed point between `cutoff` and `cutoff + horizon`. This dataframe can then be used to compute error measures of `yhat` vs. `y`.\n",
"\n",
"Here we do cross-validation to assess prediction performance on a horizon of 365 days, starting with 730 days of training data in the first cutoff and then making predictions every 180 days. On this 8 year time series, this corresponds to 11 total forecasts."
2017-09-02 17:53:38 +00:00
]
},
{
"cell_type": "code",
"execution_count": 4,
2017-09-02 20:07:49 +00:00
"metadata": {
"output_hidden": true
},
2017-09-02 17:53:38 +00:00
"outputs": [
{
"data": {
"text/plain": [
"Initial log joint probability = -6.79277\n",
"Optimization terminated normally: \n",
" Convergence detected: relative gradient magnitude is below tolerance\n",
"Initial log joint probability = -6.89856\n",
2017-09-02 17:53:38 +00:00
"Optimization terminated normally: \n",
" Convergence detected: relative gradient magnitude is below tolerance\n",
"Initial log joint probability = -8.79007\n",
"Optimization terminated normally: \n",
" Convergence detected: relative gradient magnitude is below tolerance\n",
"Initial log joint probability = -9.06272\n",
"Optimization terminated normally: \n",
" Convergence detected: relative gradient magnitude is below tolerance\n",
"Initial log joint probability = -30.435\n",
"Optimization terminated normally: \n",
" Convergence detected: relative gradient magnitude is below tolerance\n",
"Initial log joint probability = -10.2217\n",
"Optimization terminated normally: \n",
" Convergence detected: relative gradient magnitude is below tolerance\n",
"Initial log joint probability = -11.8515\n",
"Optimization terminated normally: \n",
" Convergence detected: relative gradient magnitude is below tolerance\n",
"Initial log joint probability = -12.6813\n",
"Optimization terminated normally: \n",
" Convergence detected: relative gradient magnitude is below tolerance\n",
"Initial log joint probability = -18.0636\n",
"Optimization terminated normally: \n",
" Convergence detected: relative gradient magnitude is below tolerance\n",
"Initial log joint probability = -12.4143\n",
"Optimization terminated normally: \n",
" Convergence detected: relative gradient magnitude is below tolerance\n",
"Initial log joint probability = -17.0767\n",
"Optimization terminated normally: \n",
" Convergence detected: relative gradient magnitude is below tolerance\n",
" ds y yhat yhat_lower yhat_upper cutoff\n",
"1 2010-02-16 8.242493 8.955689 8.439668 9.457640 2010-02-15\n",
"2 2010-02-17 8.008033 8.722100 8.208056 9.249314 2010-02-15\n",
"3 2010-02-18 8.045268 8.605801 8.088639 9.106629 2010-02-15\n",
"4 2010-02-19 7.928766 8.527643 8.023303 9.023126 2010-02-15\n",
"5 2010-02-20 7.745003 8.269555 7.803144 8.763514 2010-02-15\n",
"6 2010-02-21 7.866339 8.600783 8.134993 9.119948 2010-02-15\n"
2017-09-02 17:53:38 +00:00
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"%%R\n",
"df.cv <- cross_validation(m, initial = 730, period = 180, horizon = 365, units = 'days')\n",
2017-09-02 17:53:38 +00:00
"head(df.cv)"
]
},
{
"cell_type": "code",
"execution_count": 5,
2017-09-02 17:53:38 +00:00
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
2017-09-02 17:53:38 +00:00
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>ds</th>\n",
" <th>yhat</th>\n",
" <th>yhat_lower</th>\n",
" <th>yhat_upper</th>\n",
" <th>y</th>\n",
" <th>cutoff</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>2010-02-16</td>\n",
" <td>8.957184</td>\n",
" <td>8.438130</td>\n",
" <td>9.431683</td>\n",
" <td>8.242493</td>\n",
" <td>2010-02-15</td>\n",
2017-09-02 17:53:38 +00:00
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>2010-02-17</td>\n",
" <td>8.723619</td>\n",
" <td>8.228941</td>\n",
" <td>9.225985</td>\n",
" <td>8.008033</td>\n",
" <td>2010-02-15</td>\n",
2017-09-02 17:53:38 +00:00
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>2010-02-18</td>\n",
" <td>8.607378</td>\n",
" <td>8.086717</td>\n",
" <td>9.125563</td>\n",
" <td>8.045268</td>\n",
" <td>2010-02-15</td>\n",
2017-09-02 17:53:38 +00:00
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>2010-02-19</td>\n",
" <td>8.529250</td>\n",
" <td>8.053584</td>\n",
" <td>9.056437</td>\n",
" <td>7.928766</td>\n",
" <td>2010-02-15</td>\n",
2017-09-02 17:53:38 +00:00
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>2010-02-20</td>\n",
" <td>8.271228</td>\n",
" <td>7.748368</td>\n",
" <td>8.756539</td>\n",
" <td>7.745003</td>\n",
" <td>2010-02-15</td>\n",
2017-09-02 17:53:38 +00:00
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" ds yhat yhat_lower yhat_upper y cutoff\n",
"0 2010-02-16 8.957184 8.438130 9.431683 8.242493 2010-02-15\n",
"1 2010-02-17 8.723619 8.228941 9.225985 8.008033 2010-02-15\n",
"2 2010-02-18 8.607378 8.086717 9.125563 8.045268 2010-02-15\n",
"3 2010-02-19 8.529250 8.053584 9.056437 7.928766 2010-02-15\n",
"4 2010-02-20 8.271228 7.748368 8.756539 7.745003 2010-02-15"
2017-09-02 17:53:38 +00:00
]
},
"execution_count": 5,
2017-09-02 17:53:38 +00:00
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from fbprophet.diagnostics import cross_validation\n",
"df_cv = cross_validation(m, initial='730 days', period='180 days', horizon = '365 days')\n",
2017-09-02 17:53:38 +00:00
"df_cv.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The `performance_metrics` utility can be used to compute some useful statistics of the prediction performance (`yhat`, `yhat_lower`, and `yhat_upper` compared to `y`), as a function of the distance from the cutoff (how far into the future the prediction was). The statistics computed are mean squared error (MSE), root mean squared error (RMSE), mean absolute error (MAE), mean absolute percent error (MAPE), and coverage of the `yhat_lower` and `yhat_upper` estimates. These are computed on a rolling window of the predictions in `df_cv` after sorting by horizon (`ds` minus `cutoff`). By default 10% of the predictions will be included in each window, but this can be changed with the `rolling_window` argument."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"output_hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
" horizon mse rmse mae mape coverage\n",
"1844 37 days 0.5045114 0.7102897 0.5090638 0.05883992 0.6809045\n",
"2208 37 days 0.5034789 0.7095625 0.5080573 0.05872376 0.6834171\n",
"2571 37 days 0.5027744 0.7090658 0.5070098 0.05859898 0.6834171\n",
"2935 37 days 0.5024847 0.7088615 0.5067267 0.05856947 0.6859296\n",
"3298 37 days 0.5024450 0.7088335 0.5065010 0.05854188 0.6859296\n",
"3661 37 days 0.4806071 0.6932583 0.4991170 0.05796052 0.6884422\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"%%R\n",
"df.p <- performance_metrics(df.cv)\n",
"head(df.p)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>horizon</th>\n",
" <th>mse</th>\n",
" <th>rmse</th>\n",
" <th>mae</th>\n",
" <th>mape</th>\n",
" <th>coverage</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>3297</th>\n",
" <td>37 days</td>\n",
" <td>0.481970</td>\n",
" <td>0.694241</td>\n",
" <td>0.502930</td>\n",
" <td>0.058371</td>\n",
" <td>0.673367</td>\n",
" </tr>\n",
" <tr>\n",
" <th>35</th>\n",
" <td>37 days</td>\n",
" <td>0.480991</td>\n",
" <td>0.693535</td>\n",
" <td>0.502007</td>\n",
" <td>0.058262</td>\n",
" <td>0.675879</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2207</th>\n",
" <td>37 days</td>\n",
" <td>0.480936</td>\n",
" <td>0.693496</td>\n",
" <td>0.501928</td>\n",
" <td>0.058257</td>\n",
" <td>0.675879</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2934</th>\n",
" <td>37 days</td>\n",
" <td>0.481455</td>\n",
" <td>0.693870</td>\n",
" <td>0.502999</td>\n",
" <td>0.058393</td>\n",
" <td>0.675879</td>\n",
" </tr>\n",
" <tr>\n",
" <th>393</th>\n",
" <td>37 days</td>\n",
" <td>0.483990</td>\n",
" <td>0.695694</td>\n",
" <td>0.503418</td>\n",
" <td>0.058494</td>\n",
" <td>0.675879</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" horizon mse rmse mae mape coverage\n",
"3297 37 days 0.481970 0.694241 0.502930 0.058371 0.673367\n",
"35 37 days 0.480991 0.693535 0.502007 0.058262 0.675879\n",
"2207 37 days 0.480936 0.693496 0.501928 0.058257 0.675879\n",
"2934 37 days 0.481455 0.693870 0.502999 0.058393 0.675879\n",
"393 37 days 0.483990 0.695694 0.503418 0.058494 0.675879"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from fbprophet.diagnostics import performance_metrics\n",
"df_p = performance_metrics(df_cv)\n",
"df_p.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Cross validation performance metrics can be visualized with `plot_cross_validation_metric`, here shown for MAPE. Dots show the absolute percent error for each prediction in `df_cv`. The blue line shows the MAPE, where the mean is taken over a rolling window of the dots. We see for this forecast that errors around 5% are typical for predictions one month into the future, and that errors increase up to around 11% for predictions that are a year out."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"output_hidden": true
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAtAAAAGwCAIAAAAPKcUMAAAACXBIWXMAAAsSAAALEgHS3X78AAAg\nAElEQVR4nOydaYhs6V3/z1L7vld19Xr7Ts/MnXsnmJBoEsk7MaCIJAr6clAQxAgJmCCSRIJIkOCb\nmCAafSEqEQwaEdQEopIXJpIEk8ySmbkzd+nb3bWv59R21v+LH3n+T04tXdVdp6r61vfz6nR11ann\nnDrnOd/nt4q2bQsAAAAAAG4irXsAAAAAAHj6geAAAAAAgOtAcAAAAADAdSA4AAAAAOA6nnUPYAK9\nXm+5O5QkSZIkwzCWu9sNR5Ik27a3LSjY5/Ppur5tRy3Lsmma6x7FSpFlWRCELTzqbTtkURS9Xq+m\naeseyEoRRVEURcuy1j2QWYTD4UU/somCYzAYLHeHgUBAkqSl73bDCQaDo9Fowy/ZpRMOhxVF2cKj\n3rbLOxwO27a9hUe9bYcsSVIwGOx0OuseyEqRZdnr9Q6Hw3UPZBZXEBxwqQAAAADAdSA4AAAAAOA6\nEBwAAAAAcB0IDgAAAAC4DgQHAAAAAFwHggMAAAAArgPBAQAAAADXgeAAAAAAgOtAcAAAAADAdSA4\nAAAAAOA6EBwAAAAAcB0IDgAAAAC4DgQHAAAAAFwHggMAAAAArrOJ7ekB2DQsyzo/P2+329FotFgs\n+ny+dY8IAABuGLBwAHA5tVqt3W4LgqAoSrlcXvdwAADg5gHBAcDlaJrGtjudzhpHAgAANxQIDgAu\nJxqNsu1MJrPGkQAAwA0FMRwAXE4ikRBFsdfr+f3+VCq17uEAAMDNA4IDgLmIx+PxeHzdowAAgJsK\nXCoAAAAAcB0IDgAAAAC4DgQHAAAAAFwHggMAAAAArgPBAQAAAADXgeAAAAAAgOtAcAAAAADAdSA4\nAAAAAOA6EBwAAAAAcB0IDgAAAAC4DgQHAAAAAFwHggMAAAAArgPBAQAAAADXgeAAAAAAgOtAcAAA\nAADAdSA4AAAAAOA6EBwAAAAAcB0IDgAAAAC4DgQHAAAAAFwHggMAAAAArgPBAQAAAADXgeAAAAAA\ngOtAcAAAAADAdSA4AAAAAOA6Hre/wDCMz3/+86qqHhwcvPTSS/Sibdtf+tKXqtVqLBb73d/9XVEU\n3R4GAAAAANaI6xaOb3/728Vi8dOf/nSpVDo7O6MXv/vd74bD4U9+8pPvfOc7K5WK22MAAAAAwHpx\n3cJx//79u3fvCoJwfHx8//79vb09QRBee+01URQ///nPP//884VCgd55enra6/UEQSgUCsu1eciy\nLEmSx+P6wW4UdMiWZa17IKuGfu51j2KlbOflbdv2Fh71Fh6yIAhbeNRP5W/t+vH0+/1MJiMIQjqd\nVlWVXlRVtdfrvfTSS3/xF3+RzWbf+c53CoLw13/919///vcFQfjKV76y3AeGKIqiKEaj0SXuc/OR\nJMnv99u2ve6BrJpIJLLuIayap3Jumg1NEV6vd90DWSlb+EMLgrCFszctuf1+/7oHsmRcv3ZDoVCj\n0Tg+Pm40Grlcjl4Mh8Pve9/7crncBz7wgbfffpsEx2c+8xn6b71eX+4YAoGA1+tVFGW5u91wgsHg\naDTaNgtHJpPpdDrbdtThcJisg9tDOBy2bbvf7697ICtlC39oSZKSyWSr1Vr3QFaKLMter3c4HK57\nILMgU8JCuG55Pjk5efTokSAIp6enzzzzDL34zDPPvPXWW4IgPHz4MJ/Puz0GAAAAAKwX1wXHe9/7\n3vPz88997nO5XG5/f//NN9/8whe+8N73vvfhw4d/8Ad/UK/X3//+97s9BgAAAACsF3EDffxwqSyF\nrXWpNJvNpR817XBjY1G30NIOl8qWQC6VRqOx7oGslKfVpbJ18UcALEq5XK7VaoIg5PN5FocEAABg\nITZ0xQbAhjAcDkltCIJQqVQ0TVvveAAA4IYCwQHALAzDmPEnAACAOYHgAGAWoVCIFfaIRqOBQGC9\n4wEAgBsKYjgAmIUkSQcHB+12W5KkeDy+sXGjAACw4UBwAHAJsiyn0+l1jwIAAG42WK4BAAAAwHUg\nOAAAAADgOhAcAAAAAHAdCA4AAAAAuA4EBwAAAABcB4IDAAAAAK4DwQEAAAAA14HgAAAAAIDrQHAA\nAAAAwHUgOAAAAADgOhAcAAAAAHAdCA4AAAAAuA4EBwAAAABcB4IDAAAAAK4DwQEAAAAA14HgAAAA\nAIDrQHAAAAAAwHUgOAAAAADgOhAcAAAAAHAdCA4AAAAAuA4EBwAAAABcB4IDAAAAAK4DwQEAAAAA\n14HgAAAAAIDrQHAAAAAAwHUgOAAAAADgOhAcAAAAAHAdCA4AAAAAuA4EBwAAAABcB4IDAAAAAK4D\nwQEAAAAA14HgAAAAAIDrQHAAAAAAwHUgOAAAAADgOhAcAAAAAHAdCA4AAAAAuA4EBwAAAABcB4ID\nAAAAAK4DwQEAAAAA14HgAAAAAIDrQHAAAAAAwHUgOAAAAADgOhAcAAAAAHAdz7oHMIFwOLzcHXo8\nHkmSlr7bDcfj8ciybNv2ugeyakKh0LYdtdfr3bbL2+v1CoIgiuK6B7JStvCHFkVRFMUtPGpZlmVZ\nXvdAlswmCo5er7fcHQYCAa/Xu/TdbjjBYHA0GlmWte6BrJRgMNjv97ftqMPh8LZd3uFw2Lbtfr+/\n7oGslC38oSVJ8vv923bUsix7vd7hcLjugcwiGAwu+hG4VAAAAADgOhAcAAAAAHAdCA4AAAAAuA4E\nBwAAAABcB4IDAAAAAK4DwQEAAAAA19nEtFgANgTbthuNhqZp0Wg0Go2uezgAAHCDgeAAYCoXFxfN\nZlMQhEajcXR0BM0BAABXBi4VAKZCaoNQFGWNIwEAgJsOBAcAU+FNGj6fb40jAQCAmw4EBwBTKRQK\npDkSiUQ6nV73cAAA4AaDGA4AphIIBI6OjtY9CgAAeBqAhQMAAAAArgPBAQAAAADXgeAAAAAAgOtA\ncAAAAADAdSA4AAAAAOA6EBwAAAAAcB0IDgAAAAC4DgQHAAAAAFwHggMAAAAArgPBAQAAAADXgeAA\nAAAAgOtAcAAAAADAdSA4AAAAAOA6EBwAAAAAcB0IDgAAAAC4DgQHAAAAAFwHggMAAAAArgPBAQAA\nAADXgeAAAAAAgOtAcAAAAADAdSA4AAAAAOA6EBwAAAAAcB0IDgAAAAC4DgQHAAAAAFwHggMAAAAA\nrgPBAQAAAADXgeAAAAAAgOtAcAAAAADAdSA4AAAAAOA6EBwAAAAAcB0IDgAAAAC4DgQHAAAAAFwH\nggMAAAAArgPBAQAAAADXgeAAAAAAgOtAcAAAAADAdSA4AAAAAOA6Hre/wDCMz3/+86qqHhwcvPTS\nS/y/fvCDH/znf/7nxz72MbfHAAAAAID14rqF49vf/naxWPz0pz9dKpXOzs7Y6/1+/8tf/rJt224P\nAAAAAABrx3XBcf/+/ePjY0EQjo+P79+/z17/m7/5m1/91V91+9sBAAAAsAm47lLp9/uZTEYQhHQ6\nraoqvfid73wnn88fHBzw7/yXf/mX09NTQRB+4zd+Q5KWqYQ8Ho8kSeFweIn73Hw8Ho8sy1toQwqF\nQtt21F6vd9sub6/XKwiCKIrrHshK2cIfWhRFURS38KglSZJled0DWTKuWzhCoVCj0RAEodFoRCIR\nevGrX/3qq6+++sUvfvGVV1752te+5vYYAAAAALBeXLdwnJycPHr06D3vec/p6en73/9+evGP//iP\nBUGoVqt/93d/98EPfpBe/OVf/mXaqNfryx1DIBDwer29Xm+5u91wgsHgaDSyLGvdA1kpwWCw3+9v\n21GHw+Ftu7zD4bBt2/1+f90DWSlb+ENLkuT3+7ftqGVZ9nq9w+Fw3QOZRTAYXPQjrls43vve956f\nn3/uc5/L5XL7+/tvvvnmF77wBbe/FAAAAAAbhbiB3m6XLByKoix3txvOdlo4MplMs9nctqPewoUv\nLBxbgiRJyWSS/PLbw42
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"%%R -w 10 -h 6 -u in\n",
"plot_cross_validation_metric(df.cv, metric = 'mape')"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAmQAAAF3CAYAAAALu1cUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzsnXtsHNd59p+9kcvb7pIiRYtLm4xM\nUhblWjYjJYoTSHIDw4iMqlHRpm6CIkjgqhcDCRqkaXozGqNFAgQIkEBFW9Vt0sZN3CJBagOGFSiK\nRbWGZUWl7NSiJVKyRGl3xZvI3SX3wr1+f+g747OzM7Oz5A53SD2/f2yKw5kzZ87lOe/7nvc4isVi\nEYQQQgghpG44610AQgghhJC7HQoyQgghhJA6Q0FGCCGEEFJnKMgIIYQQQuoMBRkhhBBCSJ2hICOE\nEEIIqTMUZIQQQgghdYaCjBBCCCGkzlCQEUIIIYTUGQoyQgghhJA64653Aaqls7MT/f39lt0/kUig\npaXFsvtvVFgv5bBOtGG9aMN6KYd1og3rRZuNWi/Xr1/H/Px8xes2nCDr7+/H+fPnLbv/6dOncfDg\nQcvuv1FhvZTDOtGG9aIN66Uc1ok2rBdtNmq97Nmzx9R1dFkSQgghhNQZCjJCCCGEkDpDQUYIIYQQ\nUmcoyAghhBBC6gwFGSGEEEJInaEgI4QQQgipMxRkhBBCCCF1hoKMEEIIIaTOUJARQgghhNQZCjJC\nCCGEkDpDQUaIzYnFYpiamkIsFqt3UQghhFjEhjvLkpC7iVgshtHRURQKBTidThw4cAB+v7/exSKE\nEFJjaCEjxMZEo1EUCgUEAgEUCgVEo9F6F4kQQogFUJARYmMCgQCcTiei0SicTicCgUC9i0QIIcQC\n6LIkxMb4/X4cOHAA0WgUgUCA7kpCCNmkUJARYnP8fj+FGCGEbHLosiSEEEIIqTMUZIQQQgghdYaC\njBBCCCGkzlCQEUIIIYTUGQoyQgghhJA6Q0FGCCGEEFJnKMgIIYQQQuoMBRkhhBBCSJ2hICOEEEII\nqTMUZIQQQgghdYaCjBBCCCGkzlCQEUIIIYTUGQoyQgghhJA6Q0FGCCGEEFJnKMgIIYQQQuoMBRkh\nhBBCSJ2hICOEEEIIqTMUZIQQQgghdYaCjBBCCCGkzlgqyE6cOIEdO3ZgYGAA3/jGNzSv+c///E8M\nDw9j165d+PSnP21lcQghhBBCbInbqhvn83k888wzOHnyJHp7e7F3714cPnwYw8PDyjWTk5P4+te/\njtdffx3t7e2YnZ21qjiEEEIIIbbFMgvZuXPnMDAwgO3bt6OhoQFPPfUUXnrppZJr/umf/gnPPPMM\n2tvbAQBbt261qjiEEEIIIbbFMkEWDodx7733Kj/39vYiHA6XXDMxMYGJiQl89KMfxb59+3DixAmr\nikMIIYQQYlssc1maIZfLYXJyEqdPn0YoFML+/fvxf//3fwgEAiXXHT9+HMePHwcAhEIhnD592rIy\nLS8vW3r/jQrrpRzWiTasF21YL+WwTrRhvWiz2evFMkEWDAZx8+ZN5edQKIRgMFhyTW9vLz784Q/D\n4/HgAx/4AIaGhjA5OYm9e/eWXHf06FEcPXoUALBnzx4cPHjQqmLj9OnTlt5/o8J6KYd1og3rRRvW\nSzmsE21YL9ps9nqxzGW5d+9eTE5O4tq1a8hkMnjxxRdx+PDhkms++clPKmp3fn4eExMT2L59u1VF\nIoQQQgixJZYJMrfbjWPHjuGJJ57Azp078alPfQq7du3Cs88+i5dffhkA8MQTT2DLli0YHh7GY489\nhm9+85vYsmWLVUUiZEMRi8UwNTWFWCxW76IQQgixGEtjyA4dOoRDhw6V/Ntzzz2n/L/D4cC3vvUt\nfOtb37KyGIRsOGKxGEZHR1EoFOB0OnHgwAH4/f56F4sQQohFMFM/ITYkGo2iUCggEAigUCggGo3W\nu0iEEEIshIKMEBsSCATgdDoRjUbhdDrLdh4TQgjZXNQ17QUhRBu/348DBw4gGo0iEAjQXUkIIZsc\nCjJCbIrf76cQI4SQuwS6LAkhhBBC6gwFGSGEEEJInaEgI4QQQgipMxRkhBBCCCF1hoKMEEIIIaTO\nUJARQgghhNQZCjJCCCGEkDpDQUYIIYQQUmcoyAghhBBC6gwFGSGEEEJInaEgI4QQQgipMxRkhBBC\nCCF1hoKMEEIIIaTOUJARQgghhNQZCjJCCCGEkDpDQUYIIYQQUmcoyAghhBBC6gwFGSGEEEJInaEg\nI4QQQgipMxRkhBBCCCF1hoKMEEIIIaTOUJARQgghhNQZCjJCCCGEkDpDQUYIIYQQUmcoyAghhBBC\n6gwFGSGEEEJInaEgI4QQQgipMxRkhBBCCCF1hoKMEEIIIaTOUJARQgghhNQZCjJCCCGEkDpDQUYI\nIYQQUmcsFWQnTpzAjh07MDAwgG984xtlv//e976Hrq4uPPzww3j44Yfx/PPPW1kcQgghhBBb4rbq\nxvl8Hs888wxOnjyJ3t5e7N27F4cPH8bw8HDJdb/927+NY8eOWVUMQgghhBDbY5mF7Ny5cxgYGMD2\n7dvR0NCAp556Ci+99JJVjyOEEEII2bBYJsjC4TDuvfde5efe3l6Ew+Gy63784x/joYcewm/+5m/i\n5s2bVhWHEEIIIcS2WOayNMOv/dqv4Xd+53fQ2NiIf/zHf8RnP/tZ/PznPy+77vjx4zh+/DgAIBQK\n4fTp05aVaXl52dL7b1RYL+WwTrRhvWjDeimHdaIN60WbzV4vlgmyYDBYYvEKhUIIBoMl12zZskX5\n/6effhpf+cpXNO919OhRHD16FACwZ88eHDx4sPYF/v+cPn3a0vtvVFgv5bBOtGG9aMN6KYd1og3r\nRZvNXi+WuSz37t2LyclJXLt2DZlMBi+++CIOHz5ccs2tW7eU/3/55Zexc+dOq4pDCCGEEGJbLLOQ\nud1uHDt2DE888QTy+Tw+//nPY9euXXj22WexZ88eHD58GN/5znfw8ssvw+12o6OjA9/73vesKg4h\nhBBCiG2xNIbs0KFDOHToUMm/Pffcc8r/f/3rX8fXv/51K4tACCGEEGJ7mKmfEEIIIaTOUJARYkNi\nsRimpqYQi8XqXRRCCCHrQF3TXhBCyonFYhgdHUWhUIDT6cSBAwfg9/vrXSxCCCEWQgsZITYjGo2i\nUCggEAigUCggGo3Wu0iEEEIshoKMEJsRCATgdDoRjUbhdDoRCATqXSRCCCEWQ5clITbD7/fjwIED\niEajCAQCdFcSQshdAAUZITbE7/dTiBFCyF0EXZaEEEIIIXWGgowQQgghpM5QkBFCCCGE1BkKMkII\nIYSQOkNBRgghhBBSZyjICCGEEELqDAUZIYQQQkidoSAjhBBCCKkzFGSEEEIIIXWGgowQQgghpM5Q\nkBFCCCGE1BkKMkIIIYSQOkNBRgghhBBSZyjICCGEEELqDAUZIYQQQkidoSAjhBBCCKkzFGSEEEII\nIXWGgowQQgghpM5QkBFCCCGE1BkKMkIIIYSQOkNBRgghhBBSZyjICCGEEELqDAUZIYQQQkidoSAj\nhBBCCKkzFGSEEEIIIXWGgowQQgghpM5QkBFCCCGE1BkKMkIIIYSQOkNBRgghhBBSZywVZCdOnMCO\nHTswMDCAb3zjG7rX/fjHP4bD4cD58+etLA4hhBBCiC2xTJDl83k888wzePXVVzE+Po4f/vCHGB8f\nL7tuaWkJ3/72t/HhD3/YqqIQQgghhNgaywTZuXPnMDAwgO3bt6OhoQFPPfUUXnrppbLr/uqv/gp/\n+qd/Cq/Xa1VRCCGEEEJsjWWCLBwO495771V+7u3tRTgcLrlmbGwMN2/exJNPPmlVMQghhBBCbI+7\nXg8uFAr40pe+hO9973sVrz1+/DiOHz8OAAiFQjh9+rRl5VpeXrb0/hsV1ks5rBNtWC/asF7KYZ1o\nw3rRZrPXi2WCLBgM4ub
"text/plain": [
"<Figure size 720x432 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from fbprophet.plot import plot_cross_validation_metric\n",
"fig = plot_cross_validation_metric(df_cv, metric='mape')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The size of the rolling window in the figure can be changed with the optional argument `rolling_window`, which specifies the proportion of forecasts to use in each rolling window. The default is 0.1, corresponding to 10% of rows from `df_cv` included in each window; increasing this will lead to a smoother average curve in the figure.\n",
"\n",
"When using cross validation on a model with extra regressors, the cross validation will exit with an error if the extra regressor is constant in the simulated history. The `initial` period should be long enough for the extra regressor to take on multiple values. Similarly, the initial period should be long enough to capture any seasonalities that are included in the model: at least a year for yearly seasonality, at least a week for weekly seasonality, etc."
]
2017-09-02 17:53:38 +00:00
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
2017-09-02 17:53:38 +00:00
"language": "python",
"name": "python2"
2017-09-02 17:53:38 +00:00
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
2017-09-02 17:53:38 +00:00
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.14+"
2017-09-02 17:53:38 +00:00
}
},
"nbformat": 4,
"nbformat_minor": 1
}