Time Series Analysis and Forecasting

Sometimes the data we're working with has a special dependence on time as its primary predictive feature, and we want to predict how a variable evolves with time. These situations occur all of the time, from predicting stock prices to tomorrow's weather. In these cases, the data is called a time series, and we can apply a special set of statistical methods for analyzing the data.

Time Series Exploration With Pandas

Pandas has great support for datetime objects and general time series analysis operations. We'll be working with an example of predicting the number of airline passengers (in thousands) by month adapted from this tutorial.

First, download this dataset and load it into a Pandas Dataframe by specifying the 'Month' column as the datetime index.

Note that Pandas is using the 'Month' column as the Dataframe index.

We can index into the Dataframe in two ways - either by using a string representation for the index or by constructing a datetime object.

We can also use the Pandas datetime index support to retrieve entire years

Finally, let's plot the time series to get an intial visualization of how the series grows.

Stationarity

Most of the important results for time series forecasting (including the ARIMA model, which we focus on today) assume that the series is stationary - that is, its statistical properties like mean and variance are constant. However, the graph above certainly isn't stationary, given the obvious growth. Thus, we want to manipulate the time series to make it stationary. This process of reducing a time series to a stationary series is a hallmark of time series analysis and forecasting, as most real world time series' aren't initially stationary.

To solve this nonstationarity issue, we can break a time series up into its trend and seasonality. These are the two factors that make a series nonstationary, so the main idea is to remove these factors, operate on the resulting stationary series, then add these factors back in.

First, we will take the log of the series to reduce the positive trend. This gives a seemingly linear trend, making it easier to estimate.

A simple moving average is the most basic way to predict the trend of a series, taking advantage of the generally continuous nature of trends. For example, if I told you to predict the number of wins of a basketball team this season, without giving you any information about the team apart from its past record, you would take the average of the team's wins over the last few seasons as a reasonable predictor.

The simple moving average operates on this exact principle. Choosing an $n$ element window to average over, the prediction at each point is obtained by taking the average of the last $n$ values. Notice that the moving average is undefined for first 12 values because we're looking at a 12 value window.

You might be unhappy with having to choose a window size. How do we know what window size we want if we don't know much about the data? One solution is to average over all past data, discounting earlier values because they have less predictive power than more recent values. This method is known as smoothing.

Now we can subtract the trend from the original data (eliminating the null values in the case of the simple moving average) to create a new series that is hopefully more stationary. The blue graph represents the smoothing difference, while the red graph represents the simple moving average difference

Now there is no longer an upward trend, suggesting a stationarity. There does seem to be a strong seasonality effect, as the number of passengers is low at the beginning and middle of the year but spikes at the first and third quarters.

Dealing with Seasonality

One baseline way of dealing with both trend and seasonality at once is differencing, taking a single step lag (subtracting the last value of the series from the current value at each step) to represent how the time series grows. Of course, this method can be refined by factoring in more complex lags.

Another method of dealing with trend and seasonality is separating the two effects, then removing both from the time series to obtain the stationary series. We'll be using the statsmodels module, which you can get via pip by running the following command in the terminal.

python -mpip install statsmodels

We will use the seasonal decompose tool to separate seasonality from trend.

Forecasting

Using the seasonal decomposition, we were able to separate the trend and seasonality effects, which is great for time series analysis. However, another goal of working with time series is forecasting the future - how do we do that given the tools that we've been using and the stationary series we've obtained?

The ARIMA (Autoregressive Integrated Moving Average) model, which operates on stationary series', is one of the most commonly used models for time series forecasting. ARIMA, with parameters $p$, $d$, and $q$, combines an Autoregressive Model with a Moving Average model. Let's take a look at what this means.

Autoregressive model: output variable depends linearly on previous values. The $p$ parameter determines the number of lag terms used in the regression. Formally, $X_t = c + \sum_{i = 1}^p \varphi_iX_{t - i} + \epsilon_t$.

Moving average model: generalizes the same concept of moving average we saw earlier - the $q$ parameter determines the order of the model. Formally, $X_t = \mu + \sum_{i = 1}^q \theta_i\epsilon_{t - i}$.

Integrated model: the $d$ parameter represents the number of times past values have been subtracted, extending on the differencing method described earlier. This integrates the differencing for stationality into the ARIMA model, allowing it to be fit on non-stationalized data.

We don't have time to cover the math behind these models in depth, but Wikipedia provides some more detailed descriptions of the AR, MA, ARMA, and ARIMA models.

Comparing our model's results (red) to the actual differenced data (blue).

Now that we have a model for the stationary series that we can use to predict future values in the stationary series, and we want to get back to the original series. Note that we won't have a value for the first element because we are working with a one step lag. The following procedure takes care of that.

Now, we can plot the prediction (green) against the actual data. Note that the prediction model captures the seasonality and trend of the original series. It's not perfect, and additional steps can be made to tune the model. The important takeaway from this workshop is the general time series procedure of separating the time series into the trend and seasonality effects, and understanding how to work with time series' in Pandas.

Challenge: ARIMA Tuning

This is an open ended challenge. There aren't any right or wrong answers, we'd just like to see how you would approach tuning the ARIMA model.

As you can see above, the ARIMA predictions could certainly use some tuning. Try manually tuning $p$, $d$, and $q$ and see how that changes the ARIMA predictions. How would you use the AR, MA, and ARMA models individually using the ARIMA model? Do these results match what you would expect from these individual models? Can you automate this process to find the parameters that minimize RMSE? Do you see any issues with tuning $p$, $d$ and $q$ this way?