Time Series Prediction

Objectives

  1. Build a linear, DNN and CNN model in keras to predict stock market behavior.
  2. Build a simple RNN model and a multi-layer RNN model in keras.
  3. Combine RNN and CNN architecture to create a keras model to predict stock market behavior.

In this lab we will build a custom Keras model to predict stock market behavior using the stock market dataset we created in the previous labs. We'll start with a linear, DNN and CNN model

Since the features of our model are sequential in nature, we'll next look at how to build various RNN models in keras. We'll start with a simple RNN model and then see how to create a multi-layer RNN in keras. We'll also see how to combine features of 1-dimensional CNNs with a typical RNN architecture.

We will be exploring a lot of different model types in this notebook. To keep track of your results, record the accuracy on the validation set in the table here. In machine learning there are rarely any "one-size-fits-all" so feel free to test out different hyperparameters (e.g. train steps, regularization, learning rates, optimizers, batch size) for each of the models. Keep track of your model performance in the chart below.

Model Validation Accuracy
Baseline 0.295
Linear --
DNN --
1-d CNN --
simple RNN --
multi-layer RNN --
RNN using CNN features --
CNN using RNN features --

Load necessary libraries and set up environment variables


In [61]:
import os

import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import tensorflow as tf

from google.cloud import bigquery
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import (Dense, DenseFeatures,
                                     Conv1D, MaxPool1D,
                                     Reshape, RNN,
                                     LSTM, GRU, Bidirectional)
from tensorflow.keras.callbacks import TensorBoard, ModelCheckpoint
from tensorflow.keras.optimizers import Adam

# To plot pretty figures
%matplotlib inline
mpl.rc('axes', labelsize=14)
mpl.rc('xtick', labelsize=12)
mpl.rc('ytick', labelsize=12)

# For reproducible results.
from numpy.random import seed
seed(1)
tf.random.set_seed(2)

In [12]:
PROJECT = "your-gcp-project-here" # REPLACE WITH YOUR PROJECT NAME
BUCKET = "your-gcp-bucket-here" # REPLACE WITH YOUR BUCKET
REGION = "us-central1" # REPLACE WITH YOUR BUCKET REGION e.g. us-central1

In [19]:
%env 
PROJECT = PROJECT
BUCKET = BUCKET
REGION = REGION

Explore time series data

We'll start by pulling a small sample of the time series data from Big Query and write some helper functions to clean up the data for modeling. We'll use the data from the percent_change_sp500 table in BigQuery. The close_values_prior_260 column contains the close values for any given stock for the previous 260 days.


In [29]:
%%time
bq = bigquery.Client(project=PROJECT)

bq_query = '''
#standardSQL
SELECT
  symbol,
  Date,
  direction,
  close_values_prior_260
FROM
  `stock_market.eps_percent_change_sp500`
LIMIT
  100
'''

df_stock_raw = bq.query(bq_query).to_dataframe()


CPU times: user 76 ms, sys: 20 ms, total: 96 ms
Wall time: 1.59 s

In [30]:
df_stock_raw.head()


Out[30]:
symbol Date direction close_values_prior_260
0 A 2008-07-01 UP [38.74, 39.14, 38.65, 38.48, 38.4, 38.49, 38.4...
1 A 2005-11-14 UP [24.84, 24.96, 25.21, 25.77, 25.56, 25.05, 25....
2 A 2009-11-13 UP [23.0, 24.64, 23.26, 21.43, 21.84, 21.71, 20.9...
3 A 2003-05-19 UP [27.81, 30.04, 28.74, 27.95, 28.46, 31.05, 29....
4 A 2008-07-01 UP [38.74, 39.14, 38.65, 38.48, 38.4, 38.49, 38.4...

The function clean_data below does three things:

  1. First, we'll remove any inf or NA values
  2. Next, we parse the Date field to read it as a string.
  3. Lastly, we convert the label direction into a numeric quantity, mapping 'DOWN' to 0, 'STAY' to 1 and 'UP' to 2.

In [31]:
def clean_data(input_df):
    """Cleans data to prepare for training.

    Args:
        input_df: Pandas dataframe.
    Returns:
        Pandas dataframe.
    """
    df = input_df.copy()

    # Remove inf/na values.
    real_valued_rows = ~(df == np.inf).max(axis=1)
    df = df[real_valued_rows].dropna()

    # TF doesn't accept datetimes in DataFrame.
    df['Date'] = pd.to_datetime(df['Date'], errors='coerce')
    df['Date'] = df['Date'].dt.strftime('%Y-%m-%d')

    # TF requires numeric label.
    df['direction_numeric'] = df['direction'].apply(lambda x: {'DOWN': 0,
                                                               'STAY': 1,
                                                               'UP': 2}[x])
    return df

In [32]:
df_stock = clean_data(df_stock_raw)

In [33]:
df_stock.head()


Out[33]:
symbol Date direction close_values_prior_260 direction_numeric
0 A 2008-07-01 UP [38.74, 39.14, 38.65, 38.48, 38.4, 38.49, 38.4... 2
1 A 2005-11-14 UP [24.84, 24.96, 25.21, 25.77, 25.56, 25.05, 25.... 2
2 A 2009-11-13 UP [23.0, 24.64, 23.26, 21.43, 21.84, 21.71, 20.9... 2
3 A 2003-05-19 UP [27.81, 30.04, 28.74, 27.95, 28.46, 31.05, 29.... 2
4 A 2008-07-01 UP [38.74, 39.14, 38.65, 38.48, 38.4, 38.49, 38.4... 2

Read data and preprocessing

Before we begin modeling, we'll preprocess our features by scaling to the z-score. This will ensure that the range of the feature values being fed to the model are comparable and should help with convergence during gradient descent.


In [34]:
STOCK_HISTORY_COLUMN = 'close_values_prior_260'
COL_NAMES = ['day_' + str(day) for day in range(0, 260)]
LABEL = 'direction_numeric'

In [35]:
def _scale_features(df):
    """z-scale feature columns of Pandas dataframe.

    Args:
        features: Pandas dataframe.
    Returns:
        Pandas dataframe with each column standardized according to the
        values in that column.
    """
    avg = df.mean()
    std = df.std()
    return (df - avg) / std


def create_features(df, label_name):
    """Create modeling features and label from Pandas dataframe.

    Args:
        df: Pandas dataframe.
        label_name: str, the column name of the label.
    Returns:
        Pandas dataframe
    """
    # Expand 1 column containing a list of close prices to 260 columns.
    time_series_features = df[STOCK_HISTORY_COLUMN].apply(pd.Series)

    # Rename columns.
    time_series_features.columns = COL_NAMES
    time_series_features = _scale_features(time_series_features)

    # Concat time series features with static features and label.
    label_column = df[LABEL]

    return pd.concat([time_series_features,
                      label_column], axis=1)

In [36]:
df_features = create_features(df_stock, LABEL)

In [37]:
df_features.head()


Out[37]:
day_0 day_1 day_2 day_3 day_4 day_5 day_6 day_7 day_8 day_9 ... day_251 day_252 day_253 day_254 day_255 day_256 day_257 day_258 day_259 direction_numeric
0 0.305363 0.325672 0.305088 0.294639 0.283100 0.296432 0.292771 0.285282 0.334862 0.359355 ... 0.223886 0.242028 0.228025 0.208775 0.182055 0.201991 0.152421 0.148440 0.146139 2
1 -0.377435 -0.371494 -0.355443 -0.324910 -0.340068 -0.362491 -0.328215 -0.476867 -0.463193 -0.459966 ... -0.004858 0.015732 0.017295 0.023240 0.020810 0.029342 0.036185 0.037244 0.032023 2
2 -0.467820 -0.387227 -0.451279 -0.536463 -0.520612 -0.526242 -0.558174 -0.631122 -0.551704 -0.567707 ... -0.312465 -0.315907 -0.313728 -0.265901 -0.270121 -0.232024 -0.226324 -0.206079 -0.208453 2
3 -0.231542 -0.121734 -0.181956 -0.218646 -0.199321 -0.068329 -0.130761 -0.139522 -0.112068 -0.144089 ... -0.704598 -0.707806 -0.704465 -0.690725 -0.696792 -0.698219 -0.692573 -0.674848 -0.685907 2
4 0.305363 0.325672 0.305088 0.294639 0.283100 0.296432 0.292771 0.285282 0.334862 0.359355 ... 0.223886 0.242028 0.228025 0.208775 0.182055 0.201991 0.152421 0.148440 0.146139 2

5 rows × 261 columns

Let's plot a few examples and see that the preprocessing steps were implemented correctly.


In [38]:
ix_to_plot = [0, 1, 9, 5]
fig, ax = plt.subplots(1, 1, figsize=(15, 8))
for ix in ix_to_plot:
    label = df_features['direction_numeric'].iloc[ix]
    example = df_features[COL_NAMES].iloc[ix]
    ax = example.plot(label=label, ax=ax)
    ax.set_ylabel('scaled price')
    ax.set_xlabel('prior days')
    ax.legend()


Make train-eval-test split

Next, we'll make repeatable splits for our train/validation/test datasets and save these datasets to local csv files. The query below will take a subsample of the entire dataset and then create a 70-15-15 split for the train/validation/test sets.


In [251]:
def _create_split(phase):
    """Create string to produce train/valid/test splits for a SQL query.

    Args:
        phase: str, either TRAIN, VALID, or TEST.
    Returns:
        String.
    """
    floor, ceiling = '2002-11-01', '2010-07-01'
    if phase == 'VALID':
        floor, ceiling = '2010-07-01', '2011-09-01'
    elif phase == 'TEST':
        floor, ceiling = '2011-09-01', '2012-11-30'
    return '''
    WHERE Date >= '{0}'
    AND Date < '{1}'
    '''.format(floor, ceiling)


def create_query(phase):
    """Create SQL query to create train/valid/test splits on subsample.

    Args:
        phase: str, either TRAIN, VALID, or TEST.
        sample_size: str, amount of data to take for subsample.
    Returns:
        String.
    """
    basequery = """
    #standardSQL
    SELECT
      symbol,
      Date,
      direction,
      close_values_prior_260
    FROM
      `stock_market.eps_percent_change_sp500`
    """
    
    return basequery + _create_split(phase)

In [249]:
bq = bigquery.Client(project=PROJECT)

for phase in ['TRAIN', 'VALID', 'TEST']:
    # 1. Create query string
    query_string = create_query(phase)
    # 2. Load results into DataFrame
    df = bq.query(query_string).to_dataframe()

    # 3. Clean, preprocess dataframe
    df = clean_data(df)
    df = create_features(df, label_name='direction_numeric')

    # 3. Write DataFrame to CSV
    if not os.path.exists('../data'):
        os.mkdir('../data')
    df.to_csv('../data/stock-{}.csv'.format(phase.lower()),
              index_label=False, index=False)
    print("Wrote {} lines to {}".format(
        len(df),
        '../data/stock-{}.csv'.format(phase.lower())))


Wrote 11164 lines to ./data/stock-train.csv
Wrote 2394 lines to ./data/stock-valid.csv
Wrote 2455 lines to ./data/stock-test.csv

In [2]:
ls -la ../data


total 13012
drwxr-xr-x 3 jupyter jupyter    4096 Nov  6 14:08 ./
drwxr-xr-x 7 jupyter jupyter    4096 Nov  6 14:13 ../
drwxr-xr-x 2 jupyter jupyter    4096 Nov  6 14:08 .ipynb_checkpoints/
-rw-r--r-- 1 jupyter jupyter 2917139 Nov  5 16:05 stock-test.csv
-rw-r--r-- 1 jupyter jupyter 7340032 Nov  5 16:05 stock-train.csv
-rw-r--r-- 1 jupyter jupyter 3047685 Nov  5 16:05 stock-valid.csv

Modeling

For experimentation purposes, we'll train various models using data we can fit in memory using the .csv files we created above.


In [253]:
N_TIME_STEPS = 260
N_LABELS = 3

Xtrain = pd.read_csv('../data/stock-train.csv')
Xvalid = pd.read_csv('../data/stock-valid.csv')

ytrain = Xtrain.pop(LABEL)
yvalid  = Xvalid.pop(LABEL)

ytrain_categorical = to_categorical(ytrain.values)
yvalid_categorical = to_categorical(yvalid.values)

To monitor training progress and compare evaluation metrics for different models, we'll use the function below to plot metrics captured from the training job such as training and validation loss or accuracy.


In [254]:
def plot_curves(train_data, val_data, label='Accuracy'):
    """Plot training and validation metrics on single axis.

    Args:
        train_data: list, metrics obtrained from training data.
        val_data: list, metrics obtained from validation data.
        label: str, title and label for plot.
    Returns:
        Matplotlib plot.
    """
    plt.plot(np.arange(len(train_data)) + 0.5,
             train_data,
             "b.-", label="Training " + label)
    plt.plot(np.arange(len(val_data)) + 1,
             val_data, "r.-",
             label="Validation " + label)
    plt.gca().xaxis.set_major_locator(mpl.ticker.MaxNLocator(integer=True))
    plt.legend(fontsize=14)
    plt.xlabel("Epochs")
    plt.ylabel(label)
    plt.grid(True)

Baseline

Before we begin modeling in keras, let's create a benchmark using a simple heuristic. Let's see what kind of accuracy we would get on the validation set if we predict the majority class of the training set.


In [255]:
sum(yvalid == ytrain.value_counts().idxmax()) / yvalid.shape[0]


Out[255]:
0.29490392648287383

Ok. So just naively guessing the most common outcome UP will give about 29.5% accuracy on the validation set.

Linear model

We'll start with a simple linear model, mapping our sequential input to a single fully dense layer.


In [241]:
# TODO 1a
model = Sequential()
model.add(Dense(units=N_LABELS,
                activation='softmax',
                kernel_regularizer=tf.keras.regularizers.l1(l=0.1)))

model.compile(optimizer=Adam(lr=0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

history = model.fit(x=Xtrain.values,
                    y=ytrain_categorical,
                    batch_size=Xtrain.shape[0],
                    validation_data=(Xvalid.values, yvalid_categorical),
                    epochs=30,
                    verbose=0)

In [242]:
plot_curves(history.history['loss'],
            history.history['val_loss'],
            label='Loss')



In [243]:
plot_curves(history.history['accuracy'],
            history.history['val_accuracy'],
            label='Accuracy')


The accuracy seems to level out pretty quickly. To report the accuracy, we'll average the accuracy on the validation set across the last few epochs of training.


In [244]:
np.mean(history.history['val_accuracy'][-5:])


Out[244]:
0.3241512

Deep Neural Network

The linear model is an improvement on our naive benchmark. Perhaps we can do better with a more complicated model. Next, we'll create a deep neural network with keras. We'll experiment with a two layer DNN here but feel free to try a more complex model or add any other additional techniques to try an improve your performance.


In [191]:
# TODO 1b
dnn_hidden_units = [16, 8]

model = Sequential()
for layer in dnn_hidden_units:
    model.add(Dense(units=layer,
                    activation="relu"))

model.add(Dense(units=N_LABELS,
                activation="softmax",
                kernel_regularizer=tf.keras.regularizers.l1(l=0.1)))

model.compile(optimizer=Adam(lr=0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

history = model.fit(x=Xtrain.values,
                    y=ytrain_categorical,
                    batch_size=Xtrain.shape[0],
                    validation_data=(Xvalid.values, yvalid_categorical),
                    epochs=10,
                    verbose=0)

In [192]:
plot_curves(history.history['loss'],
            history.history['val_loss'],
            label='Loss')



In [193]:
plot_curves(history.history['accuracy'],
            history.history['val_accuracy'],
            label='Accuracy')



In [195]:
np.mean(history.history['val_accuracy'][-5:])


Out[195]:
0.3669173

Convolutional Neural Network

The DNN does slightly better. Let's see how a convolutional neural network performs.

A 1-dimensional convolutional can be useful for extracting features from sequential data or deriving features from shorter, fixed-length segments of the data set. Check out the documentation for how to implement a Conv1d in Tensorflow. Max pooling is a downsampling strategy commonly used in conjunction with convolutional neural networks. Next, we'll build a CNN model in keras using the Conv1D to create convolution layers and MaxPool1D to perform max pooling before passing to a fully connected dense layer.


In [200]:
# TODO 1c
model = Sequential()

# Convolutional layer
model.add(Reshape(target_shape=[N_TIME_STEPS, 1]))
model.add(Conv1D(filters=5,
                 kernel_size=5,
                 strides=2,
                 padding="valid",
                 input_shape=[None, 1]))
model.add(MaxPool1D(pool_size=2,
                    strides=None,
                    padding='valid'))


# Flatten the result and pass through DNN.
model.add(tf.keras.layers.Flatten())
model.add(Dense(units=N_TIME_STEPS//4,
                activation="relu"))

model.add(Dense(units=N_LABELS, 
                activation="softmax",
                kernel_regularizer=tf.keras.regularizers.l1(l=0.1)))

model.compile(optimizer=Adam(lr=0.01),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

history = model.fit(x=Xtrain.values,
                    y=ytrain_categorical,
                    batch_size=Xtrain.shape[0],
                    validation_data=(Xvalid.values, yvalid_categorical),
                    epochs=10,
                    verbose=0)

In [201]:
plot_curves(history.history['loss'],
            history.history['val_loss'],
            label='Loss')



In [202]:
plot_curves(history.history['accuracy'],
            history.history['val_accuracy'],
            label='Accuracy')



In [203]:
np.mean(history.history['val_accuracy'][-5:])


Out[203]:
0.34569758

Recurrent Neural Network

RNNs are particularly well-suited for learning sequential data. They retain state information from one iteration to the next by feeding the output from one cell as input for the next step. In the cell below, we'll build a RNN model in keras. The final state of the RNN is captured and then passed through a fully connected layer to produce a prediction.


In [211]:
# TODO 2a
model = Sequential()

# Reshape inputs to pass through RNN layer.
model.add(Reshape(target_shape=[N_TIME_STEPS, 1]))
model.add(LSTM(N_TIME_STEPS // 8,
               activation='relu',
               return_sequences=False))

model.add(Dense(units=N_LABELS,
                activation='softmax',
                kernel_regularizer=tf.keras.regularizers.l1(l=0.1)))

# Create the model.
model.compile(optimizer=Adam(lr=0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

history = model.fit(x=Xtrain.values,
                    y=ytrain_categorical,
                    batch_size=Xtrain.shape[0],
                    validation_data=(Xvalid.values, yvalid_categorical),
                    epochs=40,
                    verbose=0)

In [212]:
plot_curves(history.history['loss'],
            history.history['val_loss'],
            label='Loss')



In [213]:
plot_curves(history.history['accuracy'],
            history.history['val_accuracy'],
            label='Accuracy')



In [214]:
np.mean(history.history['val_accuracy'][-5:])


Out[214]:
0.35045946

Multi-layer RNN

Next, we'll build multi-layer RNN. Just as multiple layers of a deep neural network allow for more complicated features to be learned during training, additional RNN layers can potentially learn complex features in sequential data. For a multi-layer RNN the output of the first RNN layer is fed as the input into the next RNN layer.


In [218]:
# TODO 2b
rnn_hidden_units = [N_TIME_STEPS // 16,
                    N_TIME_STEPS // 32]

model = Sequential()

# Reshape inputs to pass through RNN layer.
model.add(Reshape(target_shape=[N_TIME_STEPS, 1]))

for layer in rnn_hidden_units[:-1]:
    model.add(GRU(units=layer,
                  activation='relu',
                  return_sequences=True))

model.add(GRU(units=rnn_hidden_units[-1],
              return_sequences=False))
model.add(Dense(units=N_LABELS,
                activation="softmax",
                kernel_regularizer=tf.keras.regularizers.l1(l=0.1)))

model.compile(optimizer=Adam(lr=0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

history = model.fit(x=Xtrain.values,
                    y=ytrain_categorical,
                    batch_size=Xtrain.shape[0],
                    validation_data=(Xvalid.values, yvalid_categorical),
                    epochs=50,
                    verbose=0)

In [219]:
plot_curves(history.history['loss'],
            history.history['val_loss'],
            label='Loss')



In [220]:
plot_curves(history.history['accuracy'],
            history.history['val_accuracy'],
            label='Accuracy')



In [221]:
np.mean(history.history['val_accuracy'][-5:])


Out[221]:
0.36363825

Combining CNN and RNN architecture

Finally, we'll look at some model architectures which combine aspects of both convolutional and recurrant networks. For example, we can use a 1-dimensional convolution layer to process our sequences and create features which are then passed to a RNN model before prediction.


In [222]:
# TODO 3a
model = Sequential()

# Reshape inputs for convolutional layer
model.add(Reshape(target_shape=[N_TIME_STEPS, 1]))

model.add(Conv1D(filters=20,
                 kernel_size=4,
                 strides=2,
                 padding="valid",
                 input_shape=[None, 1]))
model.add(MaxPool1D(pool_size=2,
                    strides=None,
                    padding='valid'))

model.add(LSTM(units=N_TIME_STEPS//2,
               return_sequences=False,
               kernel_regularizer=tf.keras.regularizers.l1(l=0.1)))
model.add(Dense(units=N_LABELS, activation="softmax"))

model.compile(optimizer=Adam(lr=0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

history = model.fit(x=Xtrain.values,
                    y=ytrain_categorical,
                    batch_size=Xtrain.shape[0],
                    validation_data=(Xvalid.values, yvalid_categorical),
                    epochs=30,
                    verbose=0)

In [223]:
plot_curves(history.history['loss'],
            history.history['val_loss'],
            label='Loss')



In [224]:
plot_curves(history.history['accuracy'],
            history.history['val_accuracy'],
            label='Accuracy')



In [226]:
np.mean(history.history['val_accuracy'][-5:])


Out[226]:
0.39465332

We can also try building a hybrid model which uses a 1-dimensional CNN to create features from the outputs of an RNN.


In [227]:
# TODO 3b
rnn_hidden_units = [N_TIME_STEPS // 32,
                    N_TIME_STEPS // 64]

model = Sequential()

# Reshape inputs and pass through RNN layer.
model.add(Reshape(target_shape=[N_TIME_STEPS, 1]))
for layer in rnn_hidden_units:
    model.add(LSTM(layer, return_sequences=True))

# Apply 1d convolution to RNN outputs.
model.add(Conv1D(filters=5,
                 kernel_size=3,
                 strides=2,
                 padding="valid"))
model.add(MaxPool1D(pool_size=4,
                    strides=None,
                    padding='valid'))

# Flatten the convolution output and pass through DNN.
model.add(tf.keras.layers.Flatten())
model.add(Dense(units=N_TIME_STEPS // 32,
                activation="relu",
                kernel_regularizer=tf.keras.regularizers.l1(l=0.1)))
model.add(Dense(units=N_LABELS,
                activation="softmax",
                kernel_regularizer=tf.keras.regularizers.l1(l=0.1)))

model.compile(optimizer=Adam(lr=0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

history = model.fit(x=Xtrain.values,
                    y=ytrain_categorical,
                    batch_size=Xtrain.shape[0],
                    validation_data=(Xvalid.values, yvalid_categorical),
                    epochs=80,
                    verbose=0)

In [228]:
plot_curves(history.history['loss'],
            history.history['val_loss'],
            label='Loss')



In [229]:
plot_curves(history.history['accuracy'],
            history.history['val_accuracy'],
            label='Accuracy')



In [231]:
np.mean(history.history['val_accuracy'][-5:])


Out[231]:
0.3202172

Extra Credit

  1. The eps_percent_change_sp500 table also has static features for each example. Namely, the engineered features we created in the previous labs with aggregated information capturing the MAX, MIN, AVG and STD across the last 5 days, 20 days and 260 days (e.g. close_MIN_prior_5_days, close_MIN_prior_20_days, close_MIN_prior_260_days, etc.). Try building a model which incorporates these features in addition to the sequence features we used above. Does this improve performance?

  2. The eps_percent_change_sp500 table also contains a surprise feature which captures information about the earnings per share. Try building a model which uses the surprise feature in addition to the sequence features we used above. Does this improve performance?

Copyright 2019 Google Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License