Deep Q-learning

In this notebook, we'll build a neural network that can learn to play games through reinforcement learning. More specifically, we'll use Q-learning to train an agent to play a game called Cart-Pole. In this game, a freely swinging pole is attached to a cart. The cart can move to the left and right, and the goal is to keep the pole upright as long as possible.

We can simulate this game using OpenAI Gym. First, let's check out how OpenAI Gym works. Then, we'll get into training an agent to play the Cart-Pole game.


In [1]:
import gym
import tensorflow as tf
import numpy as np

Note: Make sure you have OpenAI Gym cloned into the same directory with this notebook. I've included gym as a submodule, so you can run git submodule --init --recursive to pull the contents into the gym repo.


In [25]:
# Create the Cart-Pole game environment
env = gym.make('CartPole-v0')


[2017-08-28 12:43:16,232] Making new env: CartPole-v0

We interact with the simulation through env. To show the simulation running, you can use env.render() to render one frame. Passing in an action as an integer to env.step will generate the next step in the simulation. You can see how many actions are possible from env.action_space and to get a random action you can use env.action_space.sample(). This is general to all Gym games. In the Cart-Pole game, there are two possible actions, moving the cart left or right. So there are two actions we can take, encoded as 0 and 1.

Run the code below to watch the simulation run.


In [26]:
env.reset()
rewards = []
for _ in range(100):
    env.render()
    state, reward, done, info = env.step(env.action_space.sample()) # take a random action
    rewards.append(reward)
    if done:
        rewards = []
        env.reset()

To shut the window showing the simulation, use env.close().


In [27]:
env.close()

If you ran the simulation above, we can look at the rewards:


In [12]:
print(rewards[-20:])


[1.0, 1.0, 1.0, 1.0, 1.0, 1.0]

The game resets after the pole has fallen past a certain angle. For each frame while the simulation is running, it returns a reward of 1.0. The longer the game runs, the more reward we get. Then, our network's goal is to maximize the reward by keeping the pole vertical. It will do this by moving the cart to the left and the right.

Q-Network

We train our Q-learning agent using the Bellman Equation:

$$ Q(s, a) = r + \gamma \max{Q(s', a')} $$

where $s$ is a state, $a$ is an action, and $s'$ is the next state from state $s$ and action $a$.

Before we used this equation to learn values for a Q-table. However, for this game there are a huge number of states available. The state has four values: the position and velocity of the cart, and the position and velocity of the pole. These are all real-valued numbers, so ignoring floating point precisions, you practically have infinite states. Instead of using a table then, we'll replace it with a neural network that will approximate the Q-table lookup function.

Now, our Q value, $Q(s, a)$ is calculated by passing in a state to the network. The output will be Q-values for each available action, with fully connected hidden layers.

As I showed before, we can define our targets for training as $\hat{Q}(s,a) = r + \gamma \max{Q(s', a')}$. Then we update the weights by minimizing $(\hat{Q}(s,a) - Q(s,a))^2$.

For this Cart-Pole game, we have four inputs, one for each value in the state, and two outputs, one for each action. To get $\hat{Q}$, we'll first choose an action, then simulate the game using that action. This will get us the next state, $s'$, and the reward. With that, we can calculate $\hat{Q}$ then pass it back into the $Q$ network to run the optimizer and update the weights.

Below is my implementation of the Q-network. I used two fully connected layers with ReLU activations. Two seems to be good enough, three might be better. Feel free to try it out.


In [13]:
class QNetwork:
    def __init__(self, learning_rate=0.01, state_size=4, 
                 action_size=2, hidden_size=10, 
                 name='QNetwork'):
        # state inputs to the Q-network
        with tf.variable_scope(name):
            self.inputs_ = tf.placeholder(tf.float32, [None, state_size], name='inputs')
            
            # One hot encode the actions to later choose the Q-value for the action
            self.actions_ = tf.placeholder(tf.int32, [None], name='actions')
            one_hot_actions = tf.one_hot(self.actions_, action_size)
            
            # Target Q values for training
            self.targetQs_ = tf.placeholder(tf.float32, [None], name='target')
            
            # ReLU hidden layers
            self.fc1 = tf.contrib.layers.fully_connected(self.inputs_, hidden_size)
            self.fc2 = tf.contrib.layers.fully_connected(self.fc1, hidden_size)

            # Linear output layer
            self.output = tf.contrib.layers.fully_connected(self.fc2, action_size, 
                                                            activation_fn=None)
            
            ### Train with loss (targetQ - Q)^2
            # output has length 2, for two actions. This next line chooses
            # one value from output (per row) according to the one-hot encoded actions.
            self.Q = tf.reduce_sum(tf.multiply(self.output, one_hot_actions), axis=1)
            
            self.loss = tf.reduce_mean(tf.square(self.targetQs_ - self.Q))
            self.opt = tf.train.AdamOptimizer(learning_rate).minimize(self.loss)

Experience replay

Reinforcement learning algorithms can have stability issues due to correlations between states. To reduce correlations when training, we can store the agent's experiences and later draw a random mini-batch of those experiences to train on.

Here, we'll create a Memory object that will store our experiences, our transitions $<s, a, r, s'>$. This memory will have a maxmium capacity, so we can keep newer experiences in memory while getting rid of older experiences. Then, we'll sample a random mini-batch of transitions $<s, a, r, s'>$ and train on those.

Below, I've implemented a Memory object. If you're unfamiliar with deque, this is a double-ended queue. You can think of it like a tube open on both sides. You can put objects in either side of the tube. But if it's full, adding anything more will push an object out the other side. This is a great data structure to use for the memory buffer.


In [14]:
from collections import deque
class Memory():
    def __init__(self, max_size = 1000):
        self.buffer = deque(maxlen=max_size)
    
    def add(self, experience):
        self.buffer.append(experience)
            
    def sample(self, batch_size):
        idx = np.random.choice(np.arange(len(self.buffer)), 
                               size=batch_size, 
                               replace=False)
        return [self.buffer[ii] for ii in idx]

Exploration - Exploitation

To learn about the environment and rules of the game, the agent needs to explore by taking random actions. We'll do this by choosing a random action with some probability $\epsilon$ (epsilon). That is, with some probability $\epsilon$ the agent will make a random action and with probability $1 - \epsilon$, the agent will choose an action from $Q(s,a)$. This is called an $\epsilon$-greedy policy.

At first, the agent needs to do a lot of exploring. Later when it has learned more, the agent can favor choosing actions based on what it has learned. This is called exploitation. We'll set it up so the agent is more likely to explore early in training, then more likely to exploit later in training.

Q-Learning training algorithm

Putting all this together, we can list out the algorithm we'll use to train the network. We'll train the network in episodes. One episode is one simulation of the game. For this game, the goal is to keep the pole upright for 195 frames. So we can start a new episode once meeting that goal. The game ends if the pole tilts over too far, or if the cart moves too far the left or right. When a game ends, we'll start a new episode. Now, to train the agent:

  • Initialize the memory $D$
  • Initialize the action-value network $Q$ with random weights
  • For episode = 1, $M$ do
    • For $t$, $T$ do
      • With probability $\epsilon$ select a random action $a_t$, otherwise select $a_t = \mathrm{argmax}_a Q(s,a)$
      • Execute action $a_t$ in simulator and observe reward $r_{t+1}$ and new state $s_{t+1}$
      • Store transition $<s_t, a_t, r_{t+1}, s_{t+1}>$ in memory $D$
      • Sample random mini-batch from $D$: $<s_j, a_j, r_j, s'_j>$
      • Set $\hat{Q}_j = r_j$ if the episode ends at $j+1$, otherwise set $\hat{Q}_j = r_j + \gamma \max_{a'}{Q(s'_j, a')}$
      • Make a gradient descent step with loss $(\hat{Q}_j - Q(s_j, a_j))^2$
    • endfor
  • endfor

Hyperparameters

One of the more difficult aspects of reinforcememt learning are the large number of hyperparameters. Not only are we tuning the network, but we're tuning the simulation.


In [15]:
train_episodes = 1000          # max number of episodes to learn from
max_steps = 200                # max steps in an episode
gamma = 0.99                   # future reward discount

# Exploration parameters
explore_start = 1.0            # exploration probability at start
explore_stop = 0.01            # minimum exploration probability 
decay_rate = 0.0001            # exponential decay rate for exploration prob

# Network parameters
hidden_size = 64               # number of units in each Q-network hidden layer
learning_rate = 0.0001         # Q-network learning rate

# Memory parameters
memory_size = 10000            # memory capacity
batch_size = 20                # experience mini-batch size
pretrain_length = batch_size   # number experiences to pretrain the memory

In [16]:
tf.reset_default_graph()
mainQN = QNetwork(name='main', hidden_size=hidden_size, learning_rate=learning_rate)

Populate the experience memory

Here I'm re-initializing the simulation and pre-populating the memory. The agent is taking random actions and storing the transitions in memory. This will help the agent with exploring the game.


In [17]:
# Initialize the simulation
env.reset()
# Take one random step to get the pole and cart moving
state, reward, done, _ = env.step(env.action_space.sample())

memory = Memory(max_size=memory_size)

# Make a bunch of random actions and store the experiences
for ii in range(pretrain_length):
    # Uncomment the line below to watch the simulation
    # env.render()

    # Make a random action
    action = env.action_space.sample()
    next_state, reward, done, _ = env.step(action)

    if done:
        # The simulation fails so no next state
        next_state = np.zeros(state.shape)
        # Add experience to memory
        memory.add((state, action, reward, next_state))
        
        # Start new episode
        env.reset()
        # Take one random step to get the pole and cart moving
        state, reward, done, _ = env.step(env.action_space.sample())
    else:
        # Add experience to memory
        memory.add((state, action, reward, next_state))
        state = next_state

Training

Below we'll train our agent. If you want to watch it train, uncomment the env.render() line. This is slow because it's rendering the frames slower than the network can train. But, it's cool to watch the agent get better at the game.


In [18]:
# Now train with experiences
saver = tf.train.Saver()
rewards_list = []
with tf.Session() as sess:
    # Initialize variables
    sess.run(tf.global_variables_initializer())
    
    step = 0
    for ep in range(1, train_episodes):
        total_reward = 0
        t = 0
        while t < max_steps:
            step += 1
            # Uncomment this next line to watch the training
            # env.render() 
            
            # Explore or Exploit
            explore_p = explore_stop + (explore_start - explore_stop)*np.exp(-decay_rate*step) 
            if explore_p > np.random.rand():
                # Make a random action
                action = env.action_space.sample()
            else:
                # Get action from Q-network
                feed = {mainQN.inputs_: state.reshape((1, *state.shape))}
                Qs = sess.run(mainQN.output, feed_dict=feed)
                action = np.argmax(Qs)
            
            # Take action, get new state and reward
            next_state, reward, done, _ = env.step(action)
    
            total_reward += reward
            
            if done:
                # the episode ends so no next state
                next_state = np.zeros(state.shape)
                t = max_steps
                
                print('Episode: {}'.format(ep),
                      'Total reward: {}'.format(total_reward),
                      'Training loss: {:.4f}'.format(loss),
                      'Explore P: {:.4f}'.format(explore_p))
                rewards_list.append((ep, total_reward))
                
                # Add experience to memory
                memory.add((state, action, reward, next_state))
                
                # Start new episode
                env.reset()
                # Take one random step to get the pole and cart moving
                state, reward, done, _ = env.step(env.action_space.sample())

            else:
                # Add experience to memory
                memory.add((state, action, reward, next_state))
                state = next_state
                t += 1
            
            # Sample mini-batch from memory
            batch = memory.sample(batch_size)
            states = np.array([each[0] for each in batch])
            actions = np.array([each[1] for each in batch])
            rewards = np.array([each[2] for each in batch])
            next_states = np.array([each[3] for each in batch])
            
            # Train network
            target_Qs = sess.run(mainQN.output, feed_dict={mainQN.inputs_: next_states})
            
            # Set target_Qs to 0 for states where episode ends
            episode_ends = (next_states == np.zeros(states[0].shape)).all(axis=1)
            target_Qs[episode_ends] = (0, 0)
            
            targets = rewards + gamma * np.max(target_Qs, axis=1)

            loss, _ = sess.run([mainQN.loss, mainQN.opt],
                                feed_dict={mainQN.inputs_: states,
                                           mainQN.targetQs_: targets,
                                           mainQN.actions_: actions})
        
    saver.save(sess, "checkpoints/cartpole.ckpt")


Episode: 1 Total reward: 5.0 Training loss: 1.0464 Explore P: 0.9995
Episode: 2 Total reward: 10.0 Training loss: 1.0639 Explore P: 0.9985
Episode: 3 Total reward: 14.0 Training loss: 1.0336 Explore P: 0.9971
Episode: 4 Total reward: 32.0 Training loss: 1.0677 Explore P: 0.9940
Episode: 5 Total reward: 22.0 Training loss: 1.0657 Explore P: 0.9918
Episode: 6 Total reward: 17.0 Training loss: 1.0275 Explore P: 0.9901
Episode: 7 Total reward: 16.0 Training loss: 1.0167 Explore P: 0.9886
Episode: 8 Total reward: 26.0 Training loss: 1.0180 Explore P: 0.9860
Episode: 9 Total reward: 10.0 Training loss: 1.0972 Explore P: 0.9851
Episode: 10 Total reward: 41.0 Training loss: 1.0839 Explore P: 0.9811
Episode: 11 Total reward: 16.0 Training loss: 1.1496 Explore P: 0.9795
Episode: 12 Total reward: 18.0 Training loss: 1.1368 Explore P: 0.9778
Episode: 13 Total reward: 12.0 Training loss: 0.9586 Explore P: 0.9766
Episode: 14 Total reward: 24.0 Training loss: 1.0593 Explore P: 0.9743
Episode: 15 Total reward: 24.0 Training loss: 1.2311 Explore P: 0.9720
Episode: 16 Total reward: 18.0 Training loss: 1.1104 Explore P: 0.9703
Episode: 17 Total reward: 39.0 Training loss: 1.2714 Explore P: 0.9665
Episode: 18 Total reward: 19.0 Training loss: 1.2194 Explore P: 0.9647
Episode: 19 Total reward: 18.0 Training loss: 1.7778 Explore P: 0.9630
Episode: 20 Total reward: 14.0 Training loss: 1.0043 Explore P: 0.9617
Episode: 21 Total reward: 23.0 Training loss: 1.1952 Explore P: 0.9595
Episode: 22 Total reward: 15.0 Training loss: 1.5018 Explore P: 0.9580
Episode: 23 Total reward: 23.0 Training loss: 2.1811 Explore P: 0.9559
Episode: 24 Total reward: 25.0 Training loss: 1.8826 Explore P: 0.9535
Episode: 25 Total reward: 11.0 Training loss: 1.9758 Explore P: 0.9525
Episode: 26 Total reward: 12.0 Training loss: 1.8218 Explore P: 0.9513
Episode: 27 Total reward: 13.0 Training loss: 3.3685 Explore P: 0.9501
Episode: 28 Total reward: 14.0 Training loss: 2.3023 Explore P: 0.9488
Episode: 29 Total reward: 38.0 Training loss: 2.3396 Explore P: 0.9452
Episode: 30 Total reward: 21.0 Training loss: 6.6428 Explore P: 0.9433
Episode: 31 Total reward: 10.0 Training loss: 3.1706 Explore P: 0.9423
Episode: 32 Total reward: 19.0 Training loss: 2.8359 Explore P: 0.9406
Episode: 33 Total reward: 11.0 Training loss: 2.6445 Explore P: 0.9396
Episode: 34 Total reward: 10.0 Training loss: 2.8249 Explore P: 0.9386
Episode: 35 Total reward: 11.0 Training loss: 2.2643 Explore P: 0.9376
Episode: 36 Total reward: 14.0 Training loss: 1.5762 Explore P: 0.9363
Episode: 37 Total reward: 17.0 Training loss: 5.3928 Explore P: 0.9347
Episode: 38 Total reward: 15.0 Training loss: 3.0656 Explore P: 0.9333
Episode: 39 Total reward: 34.0 Training loss: 2.6108 Explore P: 0.9302
Episode: 40 Total reward: 28.0 Training loss: 3.6353 Explore P: 0.9276
Episode: 41 Total reward: 31.0 Training loss: 6.4496 Explore P: 0.9248
Episode: 42 Total reward: 32.0 Training loss: 21.5352 Explore P: 0.9219
Episode: 43 Total reward: 14.0 Training loss: 12.2846 Explore P: 0.9206
Episode: 44 Total reward: 32.0 Training loss: 3.7079 Explore P: 0.9177
Episode: 45 Total reward: 21.0 Training loss: 3.4844 Explore P: 0.9158
Episode: 46 Total reward: 15.0 Training loss: 5.1182 Explore P: 0.9144
Episode: 47 Total reward: 13.0 Training loss: 16.5061 Explore P: 0.9133
Episode: 48 Total reward: 15.0 Training loss: 16.9885 Explore P: 0.9119
Episode: 49 Total reward: 9.0 Training loss: 16.4966 Explore P: 0.9111
Episode: 50 Total reward: 15.0 Training loss: 2.9900 Explore P: 0.9097
Episode: 51 Total reward: 13.0 Training loss: 7.9527 Explore P: 0.9086
Episode: 52 Total reward: 12.0 Training loss: 4.3669 Explore P: 0.9075
Episode: 53 Total reward: 15.0 Training loss: 43.5110 Explore P: 0.9061
Episode: 54 Total reward: 25.0 Training loss: 6.1112 Explore P: 0.9039
Episode: 55 Total reward: 9.0 Training loss: 5.1637 Explore P: 0.9031
Episode: 56 Total reward: 18.0 Training loss: 6.7333 Explore P: 0.9015
Episode: 57 Total reward: 16.0 Training loss: 5.1244 Explore P: 0.9001
Episode: 58 Total reward: 20.0 Training loss: 12.3028 Explore P: 0.8983
Episode: 59 Total reward: 11.0 Training loss: 22.1520 Explore P: 0.8973
Episode: 60 Total reward: 10.0 Training loss: 6.9606 Explore P: 0.8964
Episode: 61 Total reward: 14.0 Training loss: 6.1844 Explore P: 0.8952
Episode: 62 Total reward: 24.0 Training loss: 33.0733 Explore P: 0.8931
Episode: 63 Total reward: 11.0 Training loss: 6.1406 Explore P: 0.8921
Episode: 64 Total reward: 11.0 Training loss: 14.5872 Explore P: 0.8911
Episode: 65 Total reward: 11.0 Training loss: 6.8158 Explore P: 0.8902
Episode: 66 Total reward: 22.0 Training loss: 12.2534 Explore P: 0.8882
Episode: 67 Total reward: 13.0 Training loss: 71.5944 Explore P: 0.8871
Episode: 68 Total reward: 15.0 Training loss: 6.7516 Explore P: 0.8858
Episode: 69 Total reward: 11.0 Training loss: 33.3906 Explore P: 0.8848
Episode: 70 Total reward: 10.0 Training loss: 15.6211 Explore P: 0.8839
Episode: 71 Total reward: 20.0 Training loss: 23.1066 Explore P: 0.8822
Episode: 72 Total reward: 12.0 Training loss: 36.4077 Explore P: 0.8811
Episode: 73 Total reward: 27.0 Training loss: 8.9286 Explore P: 0.8788
Episode: 74 Total reward: 23.0 Training loss: 36.2811 Explore P: 0.8768
Episode: 75 Total reward: 19.0 Training loss: 8.1093 Explore P: 0.8752
Episode: 76 Total reward: 13.0 Training loss: 124.1991 Explore P: 0.8740
Episode: 77 Total reward: 21.0 Training loss: 45.5268 Explore P: 0.8722
Episode: 78 Total reward: 29.0 Training loss: 82.3647 Explore P: 0.8697
Episode: 79 Total reward: 13.0 Training loss: 120.3920 Explore P: 0.8686
Episode: 80 Total reward: 22.0 Training loss: 23.9437 Explore P: 0.8667
Episode: 81 Total reward: 32.0 Training loss: 44.1111 Explore P: 0.8640
Episode: 82 Total reward: 11.0 Training loss: 9.6996 Explore P: 0.8630
Episode: 83 Total reward: 19.0 Training loss: 24.7708 Explore P: 0.8614
Episode: 84 Total reward: 10.0 Training loss: 10.5381 Explore P: 0.8606
Episode: 85 Total reward: 12.0 Training loss: 54.1062 Explore P: 0.8595
Episode: 86 Total reward: 27.0 Training loss: 8.6097 Explore P: 0.8573
Episode: 87 Total reward: 9.0 Training loss: 9.6745 Explore P: 0.8565
Episode: 88 Total reward: 8.0 Training loss: 123.6923 Explore P: 0.8558
Episode: 89 Total reward: 11.0 Training loss: 72.9574 Explore P: 0.8549
Episode: 90 Total reward: 12.0 Training loss: 139.5458 Explore P: 0.8539
Episode: 91 Total reward: 24.0 Training loss: 13.1264 Explore P: 0.8519
Episode: 92 Total reward: 17.0 Training loss: 31.7458 Explore P: 0.8504
Episode: 93 Total reward: 11.0 Training loss: 38.6591 Explore P: 0.8495
Episode: 94 Total reward: 15.0 Training loss: 75.0431 Explore P: 0.8482
Episode: 95 Total reward: 27.0 Training loss: 40.8726 Explore P: 0.8460
Episode: 96 Total reward: 24.0 Training loss: 39.0887 Explore P: 0.8440
Episode: 97 Total reward: 29.0 Training loss: 40.8689 Explore P: 0.8416
Episode: 98 Total reward: 14.0 Training loss: 11.3458 Explore P: 0.8404
Episode: 99 Total reward: 11.0 Training loss: 29.9740 Explore P: 0.8395
Episode: 100 Total reward: 24.0 Training loss: 9.6544 Explore P: 0.8375
Episode: 101 Total reward: 16.0 Training loss: 15.3328 Explore P: 0.8362
Episode: 102 Total reward: 14.0 Training loss: 76.3183 Explore P: 0.8350
Episode: 103 Total reward: 28.0 Training loss: 104.7171 Explore P: 0.8327
Episode: 104 Total reward: 23.0 Training loss: 74.5275 Explore P: 0.8308
Episode: 105 Total reward: 19.0 Training loss: 14.7383 Explore P: 0.8293
Episode: 106 Total reward: 14.0 Training loss: 18.9300 Explore P: 0.8281
Episode: 107 Total reward: 13.0 Training loss: 13.2264 Explore P: 0.8271
Episode: 108 Total reward: 8.0 Training loss: 16.5618 Explore P: 0.8264
Episode: 109 Total reward: 23.0 Training loss: 18.0484 Explore P: 0.8245
Episode: 110 Total reward: 9.0 Training loss: 24.7454 Explore P: 0.8238
Episode: 111 Total reward: 14.0 Training loss: 40.6250 Explore P: 0.8227
Episode: 112 Total reward: 24.0 Training loss: 108.0030 Explore P: 0.8207
Episode: 113 Total reward: 22.0 Training loss: 129.4294 Explore P: 0.8189
Episode: 114 Total reward: 15.0 Training loss: 168.2079 Explore P: 0.8177
Episode: 115 Total reward: 11.0 Training loss: 46.5109 Explore P: 0.8168
Episode: 116 Total reward: 26.0 Training loss: 16.0488 Explore P: 0.8147
Episode: 117 Total reward: 29.0 Training loss: 50.0126 Explore P: 0.8124
Episode: 118 Total reward: 21.0 Training loss: 16.5956 Explore P: 0.8107
Episode: 119 Total reward: 34.0 Training loss: 218.3181 Explore P: 0.8080
Episode: 120 Total reward: 18.0 Training loss: 62.3463 Explore P: 0.8066
Episode: 121 Total reward: 25.0 Training loss: 131.0283 Explore P: 0.8046
Episode: 122 Total reward: 8.0 Training loss: 14.3761 Explore P: 0.8039
Episode: 123 Total reward: 11.0 Training loss: 176.8931 Explore P: 0.8031
Episode: 124 Total reward: 15.0 Training loss: 293.7336 Explore P: 0.8019
Episode: 125 Total reward: 28.0 Training loss: 109.4541 Explore P: 0.7997
Episode: 126 Total reward: 21.0 Training loss: 66.1601 Explore P: 0.7980
Episode: 127 Total reward: 24.0 Training loss: 207.9736 Explore P: 0.7961
Episode: 128 Total reward: 16.0 Training loss: 156.6235 Explore P: 0.7949
Episode: 129 Total reward: 15.0 Training loss: 14.8070 Explore P: 0.7937
Episode: 130 Total reward: 15.0 Training loss: 152.1814 Explore P: 0.7925
Episode: 131 Total reward: 7.0 Training loss: 89.1122 Explore P: 0.7920
Episode: 132 Total reward: 23.0 Training loss: 202.7337 Explore P: 0.7902
Episode: 133 Total reward: 35.0 Training loss: 20.7784 Explore P: 0.7874
Episode: 134 Total reward: 42.0 Training loss: 17.6947 Explore P: 0.7842
Episode: 135 Total reward: 18.0 Training loss: 185.2218 Explore P: 0.7828
Episode: 136 Total reward: 33.0 Training loss: 190.1159 Explore P: 0.7802
Episode: 137 Total reward: 20.0 Training loss: 15.2150 Explore P: 0.7787
Episode: 138 Total reward: 20.0 Training loss: 129.5927 Explore P: 0.7772
Episode: 139 Total reward: 14.0 Training loss: 15.3673 Explore P: 0.7761
Episode: 140 Total reward: 17.0 Training loss: 24.2611 Explore P: 0.7748
Episode: 141 Total reward: 13.0 Training loss: 208.3850 Explore P: 0.7738
Episode: 142 Total reward: 28.0 Training loss: 17.8488 Explore P: 0.7717
Episode: 143 Total reward: 13.0 Training loss: 315.0073 Explore P: 0.7707
Episode: 144 Total reward: 24.0 Training loss: 272.5026 Explore P: 0.7689
Episode: 145 Total reward: 15.0 Training loss: 76.4669 Explore P: 0.7677
Episode: 146 Total reward: 22.0 Training loss: 81.3098 Explore P: 0.7660
Episode: 147 Total reward: 22.0 Training loss: 246.3788 Explore P: 0.7644
Episode: 148 Total reward: 11.0 Training loss: 13.3806 Explore P: 0.7636
Episode: 149 Total reward: 8.0 Training loss: 246.6667 Explore P: 0.7630
Episode: 150 Total reward: 21.0 Training loss: 14.3410 Explore P: 0.7614
Episode: 151 Total reward: 17.0 Training loss: 167.9116 Explore P: 0.7601
Episode: 152 Total reward: 11.0 Training loss: 148.6973 Explore P: 0.7593
Episode: 153 Total reward: 44.0 Training loss: 220.4499 Explore P: 0.7560
Episode: 154 Total reward: 41.0 Training loss: 246.3356 Explore P: 0.7529
Episode: 155 Total reward: 29.0 Training loss: 86.5874 Explore P: 0.7508
Episode: 156 Total reward: 18.0 Training loss: 277.1852 Explore P: 0.7494
Episode: 157 Total reward: 17.0 Training loss: 86.0560 Explore P: 0.7482
Episode: 158 Total reward: 10.0 Training loss: 169.6241 Explore P: 0.7475
Episode: 159 Total reward: 55.0 Training loss: 490.1582 Explore P: 0.7434
Episode: 160 Total reward: 17.0 Training loss: 14.6569 Explore P: 0.7422
Episode: 161 Total reward: 23.0 Training loss: 89.3946 Explore P: 0.7405
Episode: 162 Total reward: 37.0 Training loss: 10.2175 Explore P: 0.7378
Episode: 163 Total reward: 38.0 Training loss: 13.9813 Explore P: 0.7350
Episode: 164 Total reward: 16.0 Training loss: 89.6436 Explore P: 0.7339
Episode: 165 Total reward: 13.0 Training loss: 10.7168 Explore P: 0.7329
Episode: 166 Total reward: 12.0 Training loss: 96.2570 Explore P: 0.7321
Episode: 167 Total reward: 15.0 Training loss: 199.2585 Explore P: 0.7310
Episode: 168 Total reward: 8.0 Training loss: 166.4361 Explore P: 0.7304
Episode: 169 Total reward: 28.0 Training loss: 157.5051 Explore P: 0.7284
Episode: 170 Total reward: 15.0 Training loss: 11.5789 Explore P: 0.7273
Episode: 171 Total reward: 28.0 Training loss: 471.0313 Explore P: 0.7253
Episode: 172 Total reward: 18.0 Training loss: 8.9261 Explore P: 0.7240
Episode: 173 Total reward: 18.0 Training loss: 399.1953 Explore P: 0.7227
Episode: 174 Total reward: 27.0 Training loss: 11.1106 Explore P: 0.7208
Episode: 175 Total reward: 11.0 Training loss: 94.6046 Explore P: 0.7200
Episode: 176 Total reward: 9.0 Training loss: 155.4553 Explore P: 0.7194
Episode: 177 Total reward: 17.0 Training loss: 171.6220 Explore P: 0.7182
Episode: 178 Total reward: 13.0 Training loss: 263.4909 Explore P: 0.7173
Episode: 179 Total reward: 10.0 Training loss: 9.4648 Explore P: 0.7166
Episode: 180 Total reward: 24.0 Training loss: 307.8528 Explore P: 0.7149
Episode: 181 Total reward: 13.0 Training loss: 8.7169 Explore P: 0.7139
Episode: 182 Total reward: 24.0 Training loss: 189.2672 Explore P: 0.7123
Episode: 183 Total reward: 23.0 Training loss: 186.2729 Explore P: 0.7106
Episode: 184 Total reward: 27.0 Training loss: 103.3072 Explore P: 0.7088
Episode: 185 Total reward: 12.0 Training loss: 117.3636 Explore P: 0.7079
Episode: 186 Total reward: 9.0 Training loss: 92.8217 Explore P: 0.7073
Episode: 187 Total reward: 13.0 Training loss: 154.7633 Explore P: 0.7064
Episode: 188 Total reward: 12.0 Training loss: 8.8698 Explore P: 0.7056
Episode: 189 Total reward: 8.0 Training loss: 150.4532 Explore P: 0.7050
Episode: 190 Total reward: 12.0 Training loss: 113.1910 Explore P: 0.7042
Episode: 191 Total reward: 14.0 Training loss: 118.1429 Explore P: 0.7032
Episode: 192 Total reward: 32.0 Training loss: 9.4825 Explore P: 0.7010
Episode: 193 Total reward: 37.0 Training loss: 4.9164 Explore P: 0.6984
Episode: 194 Total reward: 10.0 Training loss: 244.5711 Explore P: 0.6977
Episode: 195 Total reward: 20.0 Training loss: 92.5701 Explore P: 0.6964
Episode: 196 Total reward: 28.0 Training loss: 99.9900 Explore P: 0.6944
Episode: 197 Total reward: 25.0 Training loss: 103.5031 Explore P: 0.6927
Episode: 198 Total reward: 12.0 Training loss: 129.4658 Explore P: 0.6919
Episode: 199 Total reward: 15.0 Training loss: 96.1397 Explore P: 0.6909
Episode: 200 Total reward: 13.0 Training loss: 4.7050 Explore P: 0.6900
Episode: 201 Total reward: 40.0 Training loss: 3.1471 Explore P: 0.6873
Episode: 202 Total reward: 11.0 Training loss: 199.7003 Explore P: 0.6865
Episode: 203 Total reward: 42.0 Training loss: 2.6086 Explore P: 0.6837
Episode: 204 Total reward: 12.0 Training loss: 103.7226 Explore P: 0.6829
Episode: 205 Total reward: 15.0 Training loss: 3.1267 Explore P: 0.6819
Episode: 206 Total reward: 17.0 Training loss: 3.1889 Explore P: 0.6808
Episode: 207 Total reward: 8.0 Training loss: 207.1312 Explore P: 0.6802
Episode: 208 Total reward: 14.0 Training loss: 95.5794 Explore P: 0.6793
Episode: 209 Total reward: 16.0 Training loss: 185.8737 Explore P: 0.6782
Episode: 210 Total reward: 28.0 Training loss: 92.8630 Explore P: 0.6763
Episode: 211 Total reward: 30.0 Training loss: 88.2243 Explore P: 0.6743
Episode: 212 Total reward: 16.0 Training loss: 174.8557 Explore P: 0.6733
Episode: 213 Total reward: 13.0 Training loss: 147.5846 Explore P: 0.6724
Episode: 214 Total reward: 17.0 Training loss: 116.9047 Explore P: 0.6713
Episode: 215 Total reward: 13.0 Training loss: 75.0063 Explore P: 0.6704
Episode: 216 Total reward: 15.0 Training loss: 143.7906 Explore P: 0.6694
Episode: 217 Total reward: 14.0 Training loss: 86.4775 Explore P: 0.6685
Episode: 218 Total reward: 10.0 Training loss: 147.6130 Explore P: 0.6679
Episode: 219 Total reward: 11.0 Training loss: 75.2203 Explore P: 0.6671
Episode: 220 Total reward: 10.0 Training loss: 75.0154 Explore P: 0.6665
Episode: 221 Total reward: 15.0 Training loss: 1.3757 Explore P: 0.6655
Episode: 222 Total reward: 16.0 Training loss: 66.4977 Explore P: 0.6645
Episode: 223 Total reward: 17.0 Training loss: 127.5542 Explore P: 0.6633
Episode: 224 Total reward: 10.0 Training loss: 83.3320 Explore P: 0.6627
Episode: 225 Total reward: 20.0 Training loss: 79.2254 Explore P: 0.6614
Episode: 226 Total reward: 20.0 Training loss: 1.7829 Explore P: 0.6601
Episode: 227 Total reward: 11.0 Training loss: 2.6374 Explore P: 0.6594
Episode: 228 Total reward: 15.0 Training loss: 225.8111 Explore P: 0.6584
Episode: 229 Total reward: 11.0 Training loss: 64.2073 Explore P: 0.6577
Episode: 230 Total reward: 34.0 Training loss: 84.6612 Explore P: 0.6555
Episode: 231 Total reward: 18.0 Training loss: 2.4220 Explore P: 0.6543
Episode: 232 Total reward: 12.0 Training loss: 158.0316 Explore P: 0.6536
Episode: 233 Total reward: 30.0 Training loss: 52.0541 Explore P: 0.6516
Episode: 234 Total reward: 17.0 Training loss: 125.5875 Explore P: 0.6505
Episode: 235 Total reward: 12.0 Training loss: 120.8080 Explore P: 0.6498
Episode: 236 Total reward: 13.0 Training loss: 150.3370 Explore P: 0.6489
Episode: 237 Total reward: 11.0 Training loss: 108.1471 Explore P: 0.6482
Episode: 238 Total reward: 8.0 Training loss: 45.4989 Explore P: 0.6477
Episode: 239 Total reward: 23.0 Training loss: 45.1390 Explore P: 0.6463
Episode: 240 Total reward: 9.0 Training loss: 41.4332 Explore P: 0.6457
Episode: 241 Total reward: 11.0 Training loss: 0.7771 Explore P: 0.6450
Episode: 242 Total reward: 9.0 Training loss: 1.8831 Explore P: 0.6444
Episode: 243 Total reward: 8.0 Training loss: 2.5384 Explore P: 0.6439
Episode: 244 Total reward: 40.0 Training loss: 144.1435 Explore P: 0.6414
Episode: 245 Total reward: 19.0 Training loss: 40.3015 Explore P: 0.6402
Episode: 246 Total reward: 8.0 Training loss: 2.4265 Explore P: 0.6397
Episode: 247 Total reward: 11.0 Training loss: 59.3343 Explore P: 0.6390
Episode: 248 Total reward: 13.0 Training loss: 38.3142 Explore P: 0.6382
Episode: 249 Total reward: 16.0 Training loss: 71.7615 Explore P: 0.6372
Episode: 250 Total reward: 17.0 Training loss: 2.9574 Explore P: 0.6361
Episode: 251 Total reward: 14.0 Training loss: 34.2255 Explore P: 0.6352
Episode: 252 Total reward: 14.0 Training loss: 72.8868 Explore P: 0.6343
Episode: 253 Total reward: 13.0 Training loss: 130.1698 Explore P: 0.6335
Episode: 254 Total reward: 17.0 Training loss: 1.9424 Explore P: 0.6325
Episode: 255 Total reward: 11.0 Training loss: 74.2656 Explore P: 0.6318
Episode: 256 Total reward: 15.0 Training loss: 65.6902 Explore P: 0.6309
Episode: 257 Total reward: 10.0 Training loss: 63.5881 Explore P: 0.6302
Episode: 258 Total reward: 12.0 Training loss: 62.8264 Explore P: 0.6295
Episode: 259 Total reward: 10.0 Training loss: 93.9584 Explore P: 0.6289
Episode: 260 Total reward: 10.0 Training loss: 2.3116 Explore P: 0.6283
Episode: 261 Total reward: 12.0 Training loss: 59.6284 Explore P: 0.6275
Episode: 262 Total reward: 19.0 Training loss: 78.7909 Explore P: 0.6263
Episode: 263 Total reward: 10.0 Training loss: 2.1801 Explore P: 0.6257
Episode: 264 Total reward: 9.0 Training loss: 63.9586 Explore P: 0.6252
Episode: 265 Total reward: 13.0 Training loss: 59.9067 Explore P: 0.6244
Episode: 266 Total reward: 15.0 Training loss: 122.1445 Explore P: 0.6235
Episode: 267 Total reward: 19.0 Training loss: 30.2481 Explore P: 0.6223
Episode: 268 Total reward: 16.0 Training loss: 42.4709 Explore P: 0.6213
Episode: 269 Total reward: 9.0 Training loss: 128.1557 Explore P: 0.6208
Episode: 270 Total reward: 10.0 Training loss: 32.2689 Explore P: 0.6202
Episode: 271 Total reward: 9.0 Training loss: 88.4312 Explore P: 0.6196
Episode: 272 Total reward: 15.0 Training loss: 2.3669 Explore P: 0.6187
Episode: 273 Total reward: 15.0 Training loss: 3.0233 Explore P: 0.6178
Episode: 274 Total reward: 17.0 Training loss: 30.0205 Explore P: 0.6167
Episode: 275 Total reward: 13.0 Training loss: 2.0129 Explore P: 0.6160
Episode: 276 Total reward: 12.0 Training loss: 2.3461 Explore P: 0.6152
Episode: 277 Total reward: 18.0 Training loss: 100.2479 Explore P: 0.6141
Episode: 278 Total reward: 42.0 Training loss: 70.9912 Explore P: 0.6116
Episode: 279 Total reward: 18.0 Training loss: 94.2060 Explore P: 0.6105
Episode: 280 Total reward: 12.0 Training loss: 1.4879 Explore P: 0.6098
Episode: 281 Total reward: 22.0 Training loss: 1.6038 Explore P: 0.6085
Episode: 282 Total reward: 22.0 Training loss: 2.6660 Explore P: 0.6072
Episode: 283 Total reward: 16.0 Training loss: 3.3126 Explore P: 0.6062
Episode: 284 Total reward: 12.0 Training loss: 36.4877 Explore P: 0.6055
Episode: 285 Total reward: 10.0 Training loss: 68.1339 Explore P: 0.6049
Episode: 286 Total reward: 16.0 Training loss: 3.0494 Explore P: 0.6040
Episode: 287 Total reward: 16.0 Training loss: 62.1408 Explore P: 0.6030
Episode: 288 Total reward: 9.0 Training loss: 24.9772 Explore P: 0.6025
Episode: 289 Total reward: 18.0 Training loss: 121.7850 Explore P: 0.6014
Episode: 290 Total reward: 40.0 Training loss: 57.0085 Explore P: 0.5990
Episode: 291 Total reward: 14.0 Training loss: 64.3713 Explore P: 0.5982
Episode: 292 Total reward: 16.0 Training loss: 56.7680 Explore P: 0.5973
Episode: 293 Total reward: 14.0 Training loss: 2.4255 Explore P: 0.5965
Episode: 294 Total reward: 19.0 Training loss: 134.6109 Explore P: 0.5953
Episode: 295 Total reward: 15.0 Training loss: 62.0619 Explore P: 0.5945
Episode: 296 Total reward: 18.0 Training loss: 87.1042 Explore P: 0.5934
Episode: 297 Total reward: 11.0 Training loss: 187.9176 Explore P: 0.5928
Episode: 298 Total reward: 11.0 Training loss: 2.7753 Explore P: 0.5921
Episode: 299 Total reward: 12.0 Training loss: 47.8788 Explore P: 0.5914
Episode: 300 Total reward: 17.0 Training loss: 43.5747 Explore P: 0.5905
Episode: 301 Total reward: 9.0 Training loss: 49.8861 Explore P: 0.5899
Episode: 302 Total reward: 11.0 Training loss: 26.4768 Explore P: 0.5893
Episode: 303 Total reward: 17.0 Training loss: 2.3388 Explore P: 0.5883
Episode: 304 Total reward: 13.0 Training loss: 2.0098 Explore P: 0.5876
Episode: 305 Total reward: 26.0 Training loss: 68.8007 Explore P: 0.5861
Episode: 306 Total reward: 18.0 Training loss: 83.4249 Explore P: 0.5850
Episode: 307 Total reward: 17.0 Training loss: 23.7231 Explore P: 0.5840
Episode: 308 Total reward: 10.0 Training loss: 25.5227 Explore P: 0.5835
Episode: 309 Total reward: 9.0 Training loss: 108.8411 Explore P: 0.5830
Episode: 310 Total reward: 11.0 Training loss: 129.1377 Explore P: 0.5823
Episode: 311 Total reward: 21.0 Training loss: 4.8049 Explore P: 0.5811
Episode: 312 Total reward: 8.0 Training loss: 62.2261 Explore P: 0.5807
Episode: 313 Total reward: 22.0 Training loss: 76.2316 Explore P: 0.5794
Episode: 314 Total reward: 14.0 Training loss: 107.0479 Explore P: 0.5786
Episode: 315 Total reward: 14.0 Training loss: 2.9901 Explore P: 0.5778
Episode: 316 Total reward: 11.0 Training loss: 3.8593 Explore P: 0.5772
Episode: 317 Total reward: 12.0 Training loss: 61.8549 Explore P: 0.5765
Episode: 318 Total reward: 17.0 Training loss: 3.2752 Explore P: 0.5756
Episode: 319 Total reward: 18.0 Training loss: 53.2155 Explore P: 0.5745
Episode: 320 Total reward: 24.0 Training loss: 23.2920 Explore P: 0.5732
Episode: 321 Total reward: 9.0 Training loss: 93.0383 Explore P: 0.5727
Episode: 322 Total reward: 9.0 Training loss: 38.8469 Explore P: 0.5722
Episode: 323 Total reward: 27.0 Training loss: 22.1743 Explore P: 0.5707
Episode: 324 Total reward: 18.0 Training loss: 1.1076 Explore P: 0.5696
Episode: 325 Total reward: 11.0 Training loss: 1.7854 Explore P: 0.5690
Episode: 326 Total reward: 12.0 Training loss: 132.3235 Explore P: 0.5684
Episode: 327 Total reward: 12.0 Training loss: 17.3080 Explore P: 0.5677
Episode: 328 Total reward: 11.0 Training loss: 3.4079 Explore P: 0.5671
Episode: 329 Total reward: 27.0 Training loss: 20.2195 Explore P: 0.5656
Episode: 330 Total reward: 15.0 Training loss: 2.6052 Explore P: 0.5647
Episode: 331 Total reward: 10.0 Training loss: 4.9651 Explore P: 0.5642
Episode: 332 Total reward: 18.0 Training loss: 46.3014 Explore P: 0.5632
Episode: 333 Total reward: 13.0 Training loss: 92.7882 Explore P: 0.5625
Episode: 334 Total reward: 19.0 Training loss: 101.2969 Explore P: 0.5614
Episode: 335 Total reward: 19.0 Training loss: 32.0976 Explore P: 0.5604
Episode: 336 Total reward: 10.0 Training loss: 22.4202 Explore P: 0.5598
Episode: 337 Total reward: 18.0 Training loss: 20.9047 Explore P: 0.5588
Episode: 338 Total reward: 10.0 Training loss: 1.0599 Explore P: 0.5583
Episode: 339 Total reward: 9.0 Training loss: 52.4607 Explore P: 0.5578
Episode: 340 Total reward: 23.0 Training loss: 79.5002 Explore P: 0.5565
Episode: 341 Total reward: 15.0 Training loss: 33.0649 Explore P: 0.5557
Episode: 342 Total reward: 12.0 Training loss: 25.6800 Explore P: 0.5551
Episode: 343 Total reward: 13.0 Training loss: 56.6804 Explore P: 0.5544
Episode: 344 Total reward: 13.0 Training loss: 15.9646 Explore P: 0.5536
Episode: 345 Total reward: 8.0 Training loss: 66.9290 Explore P: 0.5532
Episode: 346 Total reward: 22.0 Training loss: 2.2529 Explore P: 0.5520
Episode: 347 Total reward: 14.0 Training loss: 22.0043 Explore P: 0.5513
Episode: 348 Total reward: 17.0 Training loss: 32.0159 Explore P: 0.5503
Episode: 349 Total reward: 22.0 Training loss: 65.4223 Explore P: 0.5492
Episode: 350 Total reward: 9.0 Training loss: 1.7515 Explore P: 0.5487
Episode: 351 Total reward: 23.0 Training loss: 51.1738 Explore P: 0.5474
Episode: 352 Total reward: 16.0 Training loss: 46.7353 Explore P: 0.5466
Episode: 353 Total reward: 28.0 Training loss: 40.0789 Explore P: 0.5451
Episode: 354 Total reward: 20.0 Training loss: 48.2301 Explore P: 0.5440
Episode: 355 Total reward: 25.0 Training loss: 2.2006 Explore P: 0.5427
Episode: 356 Total reward: 44.0 Training loss: 23.3608 Explore P: 0.5403
Episode: 357 Total reward: 13.0 Training loss: 30.7291 Explore P: 0.5396
Episode: 358 Total reward: 41.0 Training loss: 1.1910 Explore P: 0.5375
Episode: 359 Total reward: 15.0 Training loss: 20.1046 Explore P: 0.5367
Episode: 360 Total reward: 45.0 Training loss: 51.0633 Explore P: 0.5343
Episode: 361 Total reward: 69.0 Training loss: 71.1091 Explore P: 0.5307
Episode: 362 Total reward: 101.0 Training loss: 12.6453 Explore P: 0.5255
Episode: 363 Total reward: 89.0 Training loss: 47.8872 Explore P: 0.5209
Episode: 364 Total reward: 20.0 Training loss: 16.7935 Explore P: 0.5199
Episode: 365 Total reward: 53.0 Training loss: 30.8576 Explore P: 0.5172
Episode: 366 Total reward: 34.0 Training loss: 0.8174 Explore P: 0.5155
Episode: 367 Total reward: 104.0 Training loss: 2.5924 Explore P: 0.5102
Episode: 368 Total reward: 13.0 Training loss: 17.1838 Explore P: 0.5096
Episode: 369 Total reward: 16.0 Training loss: 25.1496 Explore P: 0.5088
Episode: 370 Total reward: 48.0 Training loss: 1.7215 Explore P: 0.5064
Episode: 371 Total reward: 27.0 Training loss: 1.3265 Explore P: 0.5051
Episode: 372 Total reward: 46.0 Training loss: 2.0504 Explore P: 0.5028
Episode: 373 Total reward: 37.0 Training loss: 21.9372 Explore P: 0.5010
Episode: 374 Total reward: 48.0 Training loss: 36.1576 Explore P: 0.4986
Episode: 375 Total reward: 36.0 Training loss: 0.8755 Explore P: 0.4969
Episode: 376 Total reward: 33.0 Training loss: 9.5190 Explore P: 0.4953
Episode: 377 Total reward: 59.0 Training loss: 11.7436 Explore P: 0.4924
Episode: 378 Total reward: 52.0 Training loss: 1.5317 Explore P: 0.4899
Episode: 379 Total reward: 39.0 Training loss: 8.4456 Explore P: 0.4880
Episode: 380 Total reward: 42.0 Training loss: 10.5401 Explore P: 0.4860
Episode: 381 Total reward: 28.0 Training loss: 23.0169 Explore P: 0.4847
Episode: 382 Total reward: 49.0 Training loss: 1.9918 Explore P: 0.4824
Episode: 383 Total reward: 86.0 Training loss: 1.2075 Explore P: 0.4783
Episode: 384 Total reward: 25.0 Training loss: 25.2553 Explore P: 0.4772
Episode: 385 Total reward: 68.0 Training loss: 35.0445 Explore P: 0.4740
Episode: 386 Total reward: 59.0 Training loss: 14.8200 Explore P: 0.4713
Episode: 387 Total reward: 52.0 Training loss: 1.0970 Explore P: 0.4689
Episode: 388 Total reward: 57.0 Training loss: 12.8536 Explore P: 0.4663
Episode: 389 Total reward: 32.0 Training loss: 8.8374 Explore P: 0.4648
Episode: 390 Total reward: 51.0 Training loss: 1.9343 Explore P: 0.4625
Episode: 391 Total reward: 33.0 Training loss: 29.6006 Explore P: 0.4610
Episode: 392 Total reward: 44.0 Training loss: 28.8844 Explore P: 0.4590
Episode: 393 Total reward: 81.0 Training loss: 2.0337 Explore P: 0.4554
Episode: 394 Total reward: 95.0 Training loss: 1.7556 Explore P: 0.4512
Episode: 395 Total reward: 30.0 Training loss: 1.7401 Explore P: 0.4499
Episode: 396 Total reward: 35.0 Training loss: 22.5443 Explore P: 0.4483
Episode: 397 Total reward: 29.0 Training loss: 25.5804 Explore P: 0.4471
Episode: 398 Total reward: 34.0 Training loss: 86.8032 Explore P: 0.4456
Episode: 399 Total reward: 56.0 Training loss: 13.6207 Explore P: 0.4432
Episode: 400 Total reward: 29.0 Training loss: 4.3599 Explore P: 0.4419
Episode: 401 Total reward: 61.0 Training loss: 31.6881 Explore P: 0.4393
Episode: 402 Total reward: 21.0 Training loss: 1.9395 Explore P: 0.4384
Episode: 403 Total reward: 64.0 Training loss: 42.7700 Explore P: 0.4356
Episode: 404 Total reward: 47.0 Training loss: 17.2258 Explore P: 0.4336
Episode: 405 Total reward: 84.0 Training loss: 43.5858 Explore P: 0.4301
Episode: 406 Total reward: 52.0 Training loss: 2.2286 Explore P: 0.4279
Episode: 407 Total reward: 89.0 Training loss: 29.1889 Explore P: 0.4242
Episode: 408 Total reward: 48.0 Training loss: 8.8012 Explore P: 0.4222
Episode: 409 Total reward: 144.0 Training loss: 2.5474 Explore P: 0.4163
Episode: 410 Total reward: 147.0 Training loss: 13.8711 Explore P: 0.4104
Episode: 411 Total reward: 71.0 Training loss: 2.1834 Explore P: 0.4076
Episode: 412 Total reward: 96.0 Training loss: 39.4321 Explore P: 0.4038
Episode: 413 Total reward: 81.0 Training loss: 2.3310 Explore P: 0.4006
Episode: 414 Total reward: 154.0 Training loss: 16.8555 Explore P: 0.3946
Episode: 415 Total reward: 31.0 Training loss: 28.9758 Explore P: 0.3934
Episode: 416 Total reward: 62.0 Training loss: 30.9740 Explore P: 0.3911
Episode: 417 Total reward: 83.0 Training loss: 2.1106 Explore P: 0.3879
Episode: 418 Total reward: 70.0 Training loss: 110.1025 Explore P: 0.3853
Episode: 419 Total reward: 22.0 Training loss: 1.8111 Explore P: 0.3845
Episode: 420 Total reward: 108.0 Training loss: 49.9563 Explore P: 0.3804
Episode: 421 Total reward: 77.0 Training loss: 14.0999 Explore P: 0.3776
Episode: 422 Total reward: 45.0 Training loss: 1.3868 Explore P: 0.3760
Episode: 423 Total reward: 101.0 Training loss: 31.9498 Explore P: 0.3723
Episode: 424 Total reward: 119.0 Training loss: 21.2076 Explore P: 0.3680
Episode: 425 Total reward: 73.0 Training loss: 2.7455 Explore P: 0.3654
Episode: 426 Total reward: 84.0 Training loss: 26.1352 Explore P: 0.3624
Episode: 427 Total reward: 66.0 Training loss: 3.2966 Explore P: 0.3601
Episode: 428 Total reward: 49.0 Training loss: 45.9868 Explore P: 0.3584
Episode: 429 Total reward: 122.0 Training loss: 2.5628 Explore P: 0.3542
Episode: 430 Total reward: 40.0 Training loss: 3.9643 Explore P: 0.3528
Episode: 431 Total reward: 63.0 Training loss: 39.2243 Explore P: 0.3506
Episode: 432 Total reward: 66.0 Training loss: 46.1203 Explore P: 0.3484
Episode: 433 Total reward: 79.0 Training loss: 93.8128 Explore P: 0.3457
Episode: 434 Total reward: 102.0 Training loss: 25.5086 Explore P: 0.3423
Episode: 435 Total reward: 108.0 Training loss: 3.5880 Explore P: 0.3388
Episode: 436 Total reward: 89.0 Training loss: 2.8485 Explore P: 0.3358
Episode: 437 Total reward: 80.0 Training loss: 1.4840 Explore P: 0.3332
Episode: 438 Total reward: 77.0 Training loss: 2.2307 Explore P: 0.3308
Episode: 439 Total reward: 74.0 Training loss: 14.0202 Explore P: 0.3284
Episode: 440 Total reward: 56.0 Training loss: 2.7155 Explore P: 0.3266
Episode: 441 Total reward: 66.0 Training loss: 3.7821 Explore P: 0.3245
Episode: 442 Total reward: 130.0 Training loss: 32.1869 Explore P: 0.3205
Episode: 443 Total reward: 63.0 Training loss: 63.0777 Explore P: 0.3185
Episode: 444 Total reward: 79.0 Training loss: 1.9555 Explore P: 0.3161
Episode: 445 Total reward: 85.0 Training loss: 59.2950 Explore P: 0.3135
Episode: 446 Total reward: 130.0 Training loss: 72.0068 Explore P: 0.3096
Episode: 447 Total reward: 80.0 Training loss: 2.7540 Explore P: 0.3072
Episode: 448 Total reward: 62.0 Training loss: 2.3563 Explore P: 0.3054
Episode: 449 Total reward: 106.0 Training loss: 1.2784 Explore P: 0.3022
Episode: 450 Total reward: 102.0 Training loss: 134.5789 Explore P: 0.2993
Episode: 451 Total reward: 86.0 Training loss: 68.2934 Explore P: 0.2968
Episode: 452 Total reward: 66.0 Training loss: 1.1912 Explore P: 0.2949
Episode: 453 Total reward: 109.0 Training loss: 2.9589 Explore P: 0.2918
Episode: 454 Total reward: 66.0 Training loss: 1.6711 Explore P: 0.2900
Episode: 455 Total reward: 59.0 Training loss: 56.3568 Explore P: 0.2883
Episode: 456 Total reward: 145.0 Training loss: 65.3034 Explore P: 0.2843
Episode: 457 Total reward: 113.0 Training loss: 2.2755 Explore P: 0.2812
Episode: 458 Total reward: 65.0 Training loss: 21.7818 Explore P: 0.2795
Episode: 459 Total reward: 151.0 Training loss: 30.0308 Explore P: 0.2754
Episode: 460 Total reward: 55.0 Training loss: 75.5429 Explore P: 0.2740
Episode: 461 Total reward: 61.0 Training loss: 2.3660 Explore P: 0.2724
Episode: 462 Total reward: 73.0 Training loss: 1.8722 Explore P: 0.2705
Episode: 463 Total reward: 74.0 Training loss: 55.4448 Explore P: 0.2686
Episode: 464 Total reward: 135.0 Training loss: 1.3780 Explore P: 0.2651
Episode: 465 Total reward: 127.0 Training loss: 91.1052 Explore P: 0.2619
Episode: 466 Total reward: 96.0 Training loss: 4.2004 Explore P: 0.2595
Episode: 467 Total reward: 139.0 Training loss: 77.5971 Explore P: 0.2560
Episode: 468 Total reward: 64.0 Training loss: 1.4318 Explore P: 0.2544
Episode: 469 Total reward: 130.0 Training loss: 1.5963 Explore P: 0.2513
Episode: 470 Total reward: 81.0 Training loss: 75.5699 Explore P: 0.2493
Episode: 471 Total reward: 71.0 Training loss: 3.1246 Explore P: 0.2477
Episode: 472 Total reward: 70.0 Training loss: 102.5213 Explore P: 0.2460
Episode: 473 Total reward: 184.0 Training loss: 29.3300 Explore P: 0.2417
Episode: 474 Total reward: 96.0 Training loss: 3.3314 Explore P: 0.2395
Episode: 475 Total reward: 79.0 Training loss: 1.9354 Explore P: 0.2377
Episode: 476 Total reward: 120.0 Training loss: 3.0959 Explore P: 0.2350
Episode: 477 Total reward: 84.0 Training loss: 1.2917 Explore P: 0.2331
Episode: 478 Total reward: 52.0 Training loss: 2.5311 Explore P: 0.2319
Episode: 479 Total reward: 87.0 Training loss: 2.4883 Explore P: 0.2300
Episode: 480 Total reward: 199.0 Training loss: 1.6380 Explore P: 0.2257
Episode: 481 Total reward: 117.0 Training loss: 2.0938 Explore P: 0.2232
Episode: 482 Total reward: 56.0 Training loss: 1.1408 Explore P: 0.2220
Episode: 483 Total reward: 144.0 Training loss: 178.4479 Explore P: 0.2189
Episode: 484 Total reward: 199.0 Training loss: 2.0189 Explore P: 0.2148
Episode: 485 Total reward: 95.0 Training loss: 78.1326 Explore P: 0.2129
Episode: 486 Total reward: 90.0 Training loss: 1.3762 Explore P: 0.2111
Episode: 487 Total reward: 82.0 Training loss: 2.4521 Explore P: 0.2094
Episode: 488 Total reward: 67.0 Training loss: 1.6046 Explore P: 0.2081
Episode: 489 Total reward: 103.0 Training loss: 1.4769 Explore P: 0.2061
Episode: 490 Total reward: 65.0 Training loss: 1.8774 Explore P: 0.2048
Episode: 491 Total reward: 57.0 Training loss: 1.2176 Explore P: 0.2037
Episode: 492 Total reward: 98.0 Training loss: 1.4260 Explore P: 0.2018
Episode: 493 Total reward: 90.0 Training loss: 1.6935 Explore P: 0.2001
Episode: 494 Total reward: 65.0 Training loss: 1.8675 Explore P: 0.1988
Episode: 495 Total reward: 100.0 Training loss: 2.6606 Explore P: 0.1970
Episode: 496 Total reward: 104.0 Training loss: 1.3314 Explore P: 0.1950
Episode: 497 Total reward: 76.0 Training loss: 117.1040 Explore P: 0.1936
Episode: 498 Total reward: 117.0 Training loss: 1.8401 Explore P: 0.1915
Episode: 499 Total reward: 71.0 Training loss: 0.8920 Explore P: 0.1902
Episode: 500 Total reward: 53.0 Training loss: 1.3839 Explore P: 0.1893
Episode: 501 Total reward: 67.0 Training loss: 2.1544 Explore P: 0.1881
Episode: 502 Total reward: 61.0 Training loss: 114.0612 Explore P: 0.1870
Episode: 503 Total reward: 82.0 Training loss: 139.2569 Explore P: 0.1855
Episode: 504 Total reward: 96.0 Training loss: 1.0755 Explore P: 0.1839
Episode: 505 Total reward: 86.0 Training loss: 240.8243 Explore P: 0.1824
Episode: 506 Total reward: 178.0 Training loss: 1.2189 Explore P: 0.1793
Episode: 507 Total reward: 63.0 Training loss: 1.3765 Explore P: 0.1783
Episode: 508 Total reward: 93.0 Training loss: 151.5921 Explore P: 0.1767
Episode: 509 Total reward: 140.0 Training loss: 1.8600 Explore P: 0.1744
Episode: 510 Total reward: 74.0 Training loss: 1.0246 Explore P: 0.1732
Episode: 511 Total reward: 123.0 Training loss: 116.0839 Explore P: 0.1712
Episode: 512 Total reward: 87.0 Training loss: 1.6434 Explore P: 0.1698
Episode: 513 Total reward: 72.0 Training loss: 1.9410 Explore P: 0.1686
Episode: 514 Total reward: 83.0 Training loss: 1.4664 Explore P: 0.1673
Episode: 515 Total reward: 97.0 Training loss: 0.7835 Explore P: 0.1658
Episode: 516 Total reward: 138.0 Training loss: 0.9587 Explore P: 0.1637
Episode: 517 Total reward: 83.0 Training loss: 0.5636 Explore P: 0.1624
Episode: 518 Total reward: 186.0 Training loss: 82.5996 Explore P: 0.1596
Episode: 519 Total reward: 78.0 Training loss: 1.2124 Explore P: 0.1584
Episode: 520 Total reward: 119.0 Training loss: 0.4105 Explore P: 0.1567
Episode: 521 Total reward: 91.0 Training loss: 1.0593 Explore P: 0.1553
Episode: 522 Total reward: 118.0 Training loss: 0.8144 Explore P: 0.1536
Episode: 523 Total reward: 153.0 Training loss: 0.2282 Explore P: 0.1515
Episode: 524 Total reward: 199.0 Training loss: 0.6304 Explore P: 0.1487
Episode: 525 Total reward: 105.0 Training loss: 0.8452 Explore P: 0.1472
Episode: 526 Total reward: 98.0 Training loss: 75.5467 Explore P: 0.1459
Episode: 527 Total reward: 100.0 Training loss: 0.2589 Explore P: 0.1445
Episode: 528 Total reward: 123.0 Training loss: 1.4266 Explore P: 0.1429
Episode: 529 Total reward: 141.0 Training loss: 0.5679 Explore P: 0.1410
Episode: 530 Total reward: 112.0 Training loss: 1.0655 Explore P: 0.1396
Episode: 531 Total reward: 90.0 Training loss: 0.7389 Explore P: 0.1384
Episode: 532 Total reward: 148.0 Training loss: 43.9076 Explore P: 0.1365
Episode: 533 Total reward: 67.0 Training loss: 1.0437 Explore P: 0.1357
Episode: 534 Total reward: 81.0 Training loss: 46.4445 Explore P: 0.1347
Episode: 535 Total reward: 86.0 Training loss: 0.8551 Explore P: 0.1336
Episode: 536 Total reward: 114.0 Training loss: 0.5396 Explore P: 0.1322
Episode: 537 Total reward: 71.0 Training loss: 0.4915 Explore P: 0.1313
Episode: 538 Total reward: 93.0 Training loss: 0.8941 Explore P: 0.1302
Episode: 539 Total reward: 74.0 Training loss: 45.2519 Explore P: 0.1293
Episode: 540 Total reward: 84.0 Training loss: 83.5334 Explore P: 0.1283
Episode: 541 Total reward: 64.0 Training loss: 0.7515 Explore P: 0.1276
Episode: 542 Total reward: 82.0 Training loss: 158.2172 Explore P: 0.1266
Episode: 543 Total reward: 108.0 Training loss: 0.5980 Explore P: 0.1254
Episode: 544 Total reward: 78.0 Training loss: 0.7430 Explore P: 0.1245
Episode: 545 Total reward: 88.0 Training loss: 1.1839 Explore P: 0.1235
Episode: 546 Total reward: 105.0 Training loss: 1.1002 Explore P: 0.1223
Episode: 547 Total reward: 111.0 Training loss: 244.2250 Explore P: 0.1210
Episode: 548 Total reward: 133.0 Training loss: 18.7763 Explore P: 0.1196
Episode: 549 Total reward: 78.0 Training loss: 0.6984 Explore P: 0.1187
Episode: 550 Total reward: 149.0 Training loss: 0.9587 Explore P: 0.1171
Episode: 551 Total reward: 78.0 Training loss: 0.6970 Explore P: 0.1163
Episode: 552 Total reward: 86.0 Training loss: 0.6062 Explore P: 0.1154
Episode: 553 Total reward: 85.0 Training loss: 10.7892 Explore P: 0.1145
Episode: 554 Total reward: 70.0 Training loss: 0.2297 Explore P: 0.1137
Episode: 555 Total reward: 95.0 Training loss: 0.2179 Explore P: 0.1128
Episode: 556 Total reward: 101.0 Training loss: 0.5487 Explore P: 0.1117
Episode: 557 Total reward: 73.0 Training loss: 0.5987 Explore P: 0.1110
Episode: 558 Total reward: 144.0 Training loss: 0.2508 Explore P: 0.1095
Episode: 559 Total reward: 106.0 Training loss: 2.0358 Explore P: 0.1085
Episode: 560 Total reward: 90.0 Training loss: 0.8427 Explore P: 0.1076
Episode: 561 Total reward: 80.0 Training loss: 0.8772 Explore P: 0.1068
Episode: 562 Total reward: 71.0 Training loss: 0.1904 Explore P: 0.1061
Episode: 563 Total reward: 99.0 Training loss: 0.3917 Explore P: 0.1052
Episode: 564 Total reward: 141.0 Training loss: 0.2831 Explore P: 0.1039
Episode: 565 Total reward: 88.0 Training loss: 0.7999 Explore P: 0.1030
Episode: 566 Total reward: 70.0 Training loss: 0.1302 Explore P: 0.1024
Episode: 567 Total reward: 65.0 Training loss: 7.1835 Explore P: 0.1018
Episode: 568 Total reward: 72.0 Training loss: 0.4140 Explore P: 0.1011
Episode: 569 Total reward: 102.0 Training loss: 0.7623 Explore P: 0.1002
Episode: 570 Total reward: 78.0 Training loss: 0.3448 Explore P: 0.0995
Episode: 571 Total reward: 83.0 Training loss: 224.2064 Explore P: 0.0988
Episode: 572 Total reward: 85.0 Training loss: 5.6413 Explore P: 0.0980
Episode: 573 Total reward: 103.0 Training loss: 0.3635 Explore P: 0.0971
Episode: 574 Total reward: 101.0 Training loss: 0.5738 Explore P: 0.0962
Episode: 575 Total reward: 142.0 Training loss: 0.2670 Explore P: 0.0950
Episode: 576 Total reward: 89.0 Training loss: 0.1497 Explore P: 0.0943
Episode: 577 Total reward: 66.0 Training loss: 0.2078 Explore P: 0.0937
Episode: 578 Total reward: 97.0 Training loss: 0.6515 Explore P: 0.0929
Episode: 579 Total reward: 121.0 Training loss: 3.6289 Explore P: 0.0919
Episode: 580 Total reward: 109.0 Training loss: 3.2554 Explore P: 0.0910
Episode: 581 Total reward: 137.0 Training loss: 0.3160 Explore P: 0.0899
Episode: 582 Total reward: 129.0 Training loss: 0.3108 Explore P: 0.0889
Episode: 583 Total reward: 108.0 Training loss: 0.1379 Explore P: 0.0881
Episode: 584 Total reward: 67.0 Training loss: 0.5634 Explore P: 0.0875
Episode: 585 Total reward: 102.0 Training loss: 0.5034 Explore P: 0.0867
Episode: 586 Total reward: 63.0 Training loss: 0.5144 Explore P: 0.0863
Episode: 587 Total reward: 67.0 Training loss: 0.1774 Explore P: 0.0858
Episode: 588 Total reward: 69.0 Training loss: 0.3250 Explore P: 0.0852
Episode: 589 Total reward: 62.0 Training loss: 0.5540 Explore P: 0.0848
Episode: 590 Total reward: 91.0 Training loss: 0.2327 Explore P: 0.0841
Episode: 591 Total reward: 90.0 Training loss: 0.1510 Explore P: 0.0834
Episode: 592 Total reward: 69.0 Training loss: 1.2428 Explore P: 0.0829
Episode: 593 Total reward: 94.0 Training loss: 0.6605 Explore P: 0.0822
Episode: 594 Total reward: 119.0 Training loss: 0.2665 Explore P: 0.0814
Episode: 595 Total reward: 73.0 Training loss: 0.1829 Explore P: 0.0809
Episode: 596 Total reward: 95.0 Training loss: 0.5121 Explore P: 0.0802
Episode: 597 Total reward: 95.0 Training loss: 0.2926 Explore P: 0.0795
Episode: 598 Total reward: 81.0 Training loss: 0.2295 Explore P: 0.0790
Episode: 599 Total reward: 75.0 Training loss: 0.3446 Explore P: 0.0785
Episode: 600 Total reward: 71.0 Training loss: 0.1489 Explore P: 0.0780
Episode: 601 Total reward: 77.0 Training loss: 0.3127 Explore P: 0.0775
Episode: 602 Total reward: 83.0 Training loss: 0.2346 Explore P: 0.0769
Episode: 603 Total reward: 97.0 Training loss: 0.4467 Explore P: 0.0762
Episode: 604 Total reward: 142.0 Training loss: 0.1426 Explore P: 0.0753
Episode: 605 Total reward: 76.0 Training loss: 0.2484 Explore P: 0.0748
Episode: 606 Total reward: 118.0 Training loss: 0.2294 Explore P: 0.0741
Episode: 607 Total reward: 119.0 Training loss: 2.8665 Explore P: 0.0733
Episode: 608 Total reward: 156.0 Training loss: 0.2853 Explore P: 0.0723
Episode: 609 Total reward: 137.0 Training loss: 0.3217 Explore P: 0.0715
Episode: 610 Total reward: 157.0 Training loss: 0.2449 Explore P: 0.0705
Episode: 611 Total reward: 184.0 Training loss: 2.9511 Explore P: 0.0694
Episode: 612 Total reward: 196.0 Training loss: 0.3582 Explore P: 0.0683
Episode: 613 Total reward: 199.0 Training loss: 5.9570 Explore P: 0.0671
Episode: 614 Total reward: 199.0 Training loss: 0.4304 Explore P: 0.0660
Episode: 615 Total reward: 199.0 Training loss: 0.5034 Explore P: 0.0649
Episode: 616 Total reward: 199.0 Training loss: 24.9514 Explore P: 0.0638
Episode: 617 Total reward: 199.0 Training loss: 0.2556 Explore P: 0.0627
Episode: 618 Total reward: 199.0 Training loss: 0.4214 Explore P: 0.0617
Episode: 619 Total reward: 137.0 Training loss: 0.5628 Explore P: 0.0610
Episode: 620 Total reward: 199.0 Training loss: 0.5681 Explore P: 0.0600
Episode: 621 Total reward: 199.0 Training loss: 0.6185 Explore P: 0.0590
Episode: 622 Total reward: 199.0 Training loss: 0.2247 Explore P: 0.0580
Episode: 623 Total reward: 100.0 Training loss: 0.9046 Explore P: 0.0576
Episode: 624 Total reward: 113.0 Training loss: 1.0858 Explore P: 0.0570
Episode: 625 Total reward: 70.0 Training loss: 0.7711 Explore P: 0.0567
Episode: 626 Total reward: 74.0 Training loss: 422.8281 Explore P: 0.0564
Episode: 627 Total reward: 42.0 Training loss: 0.5130 Explore P: 0.0562
Episode: 628 Total reward: 80.0 Training loss: 0.4686 Explore P: 0.0558
Episode: 629 Total reward: 72.0 Training loss: 0.4373 Explore P: 0.0555
Episode: 630 Total reward: 83.0 Training loss: 0.4546 Explore P: 0.0551
Episode: 631 Total reward: 89.0 Training loss: 0.9309 Explore P: 0.0547
Episode: 632 Total reward: 106.0 Training loss: 0.4549 Explore P: 0.0542
Episode: 633 Total reward: 84.0 Training loss: 0.4977 Explore P: 0.0539
Episode: 634 Total reward: 104.0 Training loss: 0.9289 Explore P: 0.0534
Episode: 635 Total reward: 76.0 Training loss: 83.0158 Explore P: 0.0531
Episode: 636 Total reward: 78.0 Training loss: 0.4362 Explore P: 0.0527
Episode: 637 Total reward: 99.0 Training loss: 0.6442 Explore P: 0.0523
Episode: 638 Total reward: 91.0 Training loss: 58.1092 Explore P: 0.0519
Episode: 639 Total reward: 73.0 Training loss: 0.9958 Explore P: 0.0516
Episode: 640 Total reward: 77.0 Training loss: 0.8829 Explore P: 0.0513
Episode: 641 Total reward: 76.0 Training loss: 56.0177 Explore P: 0.0510
Episode: 642 Total reward: 89.0 Training loss: 1.2268 Explore P: 0.0506
Episode: 643 Total reward: 100.0 Training loss: 0.7640 Explore P: 0.0502
Episode: 644 Total reward: 100.0 Training loss: 347.0724 Explore P: 0.0498
Episode: 645 Total reward: 95.0 Training loss: 0.5744 Explore P: 0.0494
Episode: 646 Total reward: 101.0 Training loss: 0.8791 Explore P: 0.0491
Episode: 647 Total reward: 62.0 Training loss: 1.4007 Explore P: 0.0488
Episode: 648 Total reward: 79.0 Training loss: 0.7291 Explore P: 0.0485
Episode: 649 Total reward: 83.0 Training loss: 0.8389 Explore P: 0.0482
Episode: 650 Total reward: 96.0 Training loss: 1.2544 Explore P: 0.0478
Episode: 651 Total reward: 102.0 Training loss: 0.8183 Explore P: 0.0474
Episode: 652 Total reward: 87.0 Training loss: 0.8193 Explore P: 0.0471
Episode: 653 Total reward: 75.0 Training loss: 0.4992 Explore P: 0.0468
Episode: 654 Total reward: 111.0 Training loss: 1.2904 Explore P: 0.0464
Episode: 655 Total reward: 97.0 Training loss: 1.0048 Explore P: 0.0461
Episode: 656 Total reward: 84.0 Training loss: 1.2299 Explore P: 0.0458
Episode: 657 Total reward: 127.0 Training loss: 1.2875 Explore P: 0.0453
Episode: 658 Total reward: 82.0 Training loss: 0.8504 Explore P: 0.0450
Episode: 659 Total reward: 76.0 Training loss: 1.1740 Explore P: 0.0448
Episode: 660 Total reward: 91.0 Training loss: 1.2466 Explore P: 0.0445
Episode: 661 Total reward: 89.0 Training loss: 1.2232 Explore P: 0.0442
Episode: 662 Total reward: 97.0 Training loss: 269.3861 Explore P: 0.0438
Episode: 663 Total reward: 94.0 Training loss: 1.2381 Explore P: 0.0435
Episode: 664 Total reward: 120.0 Training loss: 1.0860 Explore P: 0.0431
Episode: 665 Total reward: 74.0 Training loss: 1.3064 Explore P: 0.0429
Episode: 666 Total reward: 85.0 Training loss: 1.2399 Explore P: 0.0426
Episode: 667 Total reward: 93.0 Training loss: 83.2018 Explore P: 0.0423
Episode: 668 Total reward: 93.0 Training loss: 1.6725 Explore P: 0.0420
Episode: 669 Total reward: 108.0 Training loss: 1.5807 Explore P: 0.0416
Episode: 670 Total reward: 194.0 Training loss: 0.7581 Explore P: 0.0410
Episode: 671 Total reward: 87.0 Training loss: 1.9543 Explore P: 0.0408
Episode: 672 Total reward: 180.0 Training loss: 0.7686 Explore P: 0.0402
Episode: 673 Total reward: 199.0 Training loss: 1.1259 Explore P: 0.0396
Episode: 674 Total reward: 123.0 Training loss: 36.0647 Explore P: 0.0393
Episode: 675 Total reward: 188.0 Training loss: 100.4362 Explore P: 0.0387
Episode: 676 Total reward: 136.0 Training loss: 1.0600 Explore P: 0.0383
Episode: 677 Total reward: 119.0 Training loss: 1.4425 Explore P: 0.0380
Episode: 678 Total reward: 118.0 Training loss: 1.2462 Explore P: 0.0377
Episode: 679 Total reward: 129.0 Training loss: 0.8661 Explore P: 0.0373
Episode: 680 Total reward: 199.0 Training loss: 0.3368 Explore P: 0.0368
Episode: 681 Total reward: 199.0 Training loss: 0.8732 Explore P: 0.0362
Episode: 682 Total reward: 199.0 Training loss: 0.9899 Explore P: 0.0357
Episode: 683 Total reward: 199.0 Training loss: 3.9173 Explore P: 0.0352
Episode: 684 Total reward: 199.0 Training loss: 1.5790 Explore P: 0.0347
Episode: 685 Total reward: 199.0 Training loss: 0.8764 Explore P: 0.0342
Episode: 686 Total reward: 199.0 Training loss: 0.8754 Explore P: 0.0338
Episode: 687 Total reward: 199.0 Training loss: 1.0181 Explore P: 0.0333
Episode: 688 Total reward: 199.0 Training loss: 0.4257 Explore P: 0.0328
Episode: 689 Total reward: 199.0 Training loss: 0.6489 Explore P: 0.0324
Episode: 690 Total reward: 199.0 Training loss: 0.8078 Explore P: 0.0319
Episode: 691 Total reward: 199.0 Training loss: 0.4728 Explore P: 0.0315
Episode: 692 Total reward: 199.0 Training loss: 0.4989 Explore P: 0.0311
Episode: 693 Total reward: 199.0 Training loss: 1.7207 Explore P: 0.0307
Episode: 694 Total reward: 103.0 Training loss: 0.9278 Explore P: 0.0305
Episode: 695 Total reward: 199.0 Training loss: 0.4483 Explore P: 0.0301
Episode: 696 Total reward: 199.0 Training loss: 0.3839 Explore P: 0.0297
Episode: 697 Total reward: 199.0 Training loss: 0.3591 Explore P: 0.0293
Episode: 698 Total reward: 89.0 Training loss: 178.9215 Explore P: 0.0291
Episode: 699 Total reward: 199.0 Training loss: 0.5049 Explore P: 0.0287
Episode: 700 Total reward: 107.0 Training loss: 0.2903 Explore P: 0.0285
Episode: 701 Total reward: 199.0 Training loss: 0.5026 Explore P: 0.0282
Episode: 702 Total reward: 199.0 Training loss: 0.6328 Explore P: 0.0278
Episode: 703 Total reward: 199.0 Training loss: 0.5231 Explore P: 0.0274
Episode: 704 Total reward: 131.0 Training loss: 0.5173 Explore P: 0.0272
Episode: 705 Total reward: 199.0 Training loss: 0.3179 Explore P: 0.0269
Episode: 706 Total reward: 199.0 Training loss: 0.3487 Explore P: 0.0266
Episode: 707 Total reward: 199.0 Training loss: 0.2103 Explore P: 0.0262
Episode: 708 Total reward: 199.0 Training loss: 0.4512 Explore P: 0.0259
Episode: 709 Total reward: 199.0 Training loss: 0.4056 Explore P: 0.0256
Episode: 710 Total reward: 199.0 Training loss: 0.2917 Explore P: 0.0253
Episode: 711 Total reward: 199.0 Training loss: 0.3734 Explore P: 0.0250
Episode: 712 Total reward: 199.0 Training loss: 0.5876 Explore P: 0.0247
Episode: 713 Total reward: 199.0 Training loss: 0.4593 Explore P: 0.0244
Episode: 714 Total reward: 144.0 Training loss: 0.5623 Explore P: 0.0242
Episode: 715 Total reward: 199.0 Training loss: 0.4393 Explore P: 0.0239
Episode: 716 Total reward: 199.0 Training loss: 0.3014 Explore P: 0.0236
Episode: 717 Total reward: 199.0 Training loss: 0.2958 Explore P: 0.0234
Episode: 718 Total reward: 199.0 Training loss: 0.2294 Explore P: 0.0231
Episode: 719 Total reward: 199.0 Training loss: 0.2823 Explore P: 0.0228
Episode: 720 Total reward: 199.0 Training loss: 0.3925 Explore P: 0.0226
Episode: 721 Total reward: 199.0 Training loss: 0.3798 Explore P: 0.0223
Episode: 722 Total reward: 199.0 Training loss: 0.6536 Explore P: 0.0221
Episode: 723 Total reward: 199.0 Training loss: 4.5508 Explore P: 0.0219
Episode: 724 Total reward: 199.0 Training loss: 0.1815 Explore P: 0.0216
Episode: 725 Total reward: 199.0 Training loss: 0.4316 Explore P: 0.0214
Episode: 726 Total reward: 199.0 Training loss: 0.7156 Explore P: 0.0212
Episode: 727 Total reward: 199.0 Training loss: 0.4694 Explore P: 0.0210
Episode: 728 Total reward: 199.0 Training loss: 0.7696 Explore P: 0.0207
Episode: 729 Total reward: 199.0 Training loss: 0.3264 Explore P: 0.0205
Episode: 730 Total reward: 199.0 Training loss: 0.4307 Explore P: 0.0203
Episode: 731 Total reward: 199.0 Training loss: 287.9337 Explore P: 0.0201
Episode: 732 Total reward: 199.0 Training loss: 0.2789 Explore P: 0.0199
Episode: 733 Total reward: 199.0 Training loss: 0.3288 Explore P: 0.0197
Episode: 734 Total reward: 199.0 Training loss: 0.3415 Explore P: 0.0195
Episode: 735 Total reward: 188.0 Training loss: 0.6200 Explore P: 0.0194
Episode: 736 Total reward: 151.0 Training loss: 0.5011 Explore P: 0.0192
Episode: 737 Total reward: 128.0 Training loss: 0.7147 Explore P: 0.0191
Episode: 738 Total reward: 139.0 Training loss: 0.4609 Explore P: 0.0190
Episode: 739 Total reward: 83.0 Training loss: 0.6710 Explore P: 0.0189
Episode: 740 Total reward: 147.0 Training loss: 0.3320 Explore P: 0.0188
Episode: 741 Total reward: 90.0 Training loss: 0.3135 Explore P: 0.0187
Episode: 742 Total reward: 115.0 Training loss: 242.9771 Explore P: 0.0186
Episode: 743 Total reward: 76.0 Training loss: 0.4372 Explore P: 0.0185
Episode: 744 Total reward: 136.0 Training loss: 105.4232 Explore P: 0.0184
Episode: 745 Total reward: 103.0 Training loss: 318.7264 Explore P: 0.0183
Episode: 746 Total reward: 86.0 Training loss: 0.4156 Explore P: 0.0183
Episode: 747 Total reward: 62.0 Training loss: 0.4297 Explore P: 0.0182
Episode: 748 Total reward: 89.0 Training loss: 0.7780 Explore P: 0.0181
Episode: 749 Total reward: 80.0 Training loss: 0.5270 Explore P: 0.0181
Episode: 750 Total reward: 133.0 Training loss: 0.2474 Explore P: 0.0180
Episode: 751 Total reward: 111.0 Training loss: 0.4241 Explore P: 0.0179
Episode: 752 Total reward: 107.0 Training loss: 0.2361 Explore P: 0.0178
Episode: 753 Total reward: 132.0 Training loss: 0.3251 Explore P: 0.0177
Episode: 754 Total reward: 85.0 Training loss: 0.4182 Explore P: 0.0176
Episode: 755 Total reward: 147.0 Training loss: 0.2429 Explore P: 0.0175
Episode: 756 Total reward: 199.0 Training loss: 0.4886 Explore P: 0.0174
Episode: 757 Total reward: 160.0 Training loss: 0.5213 Explore P: 0.0172
Episode: 758 Total reward: 166.0 Training loss: 0.3211 Explore P: 0.0171
Episode: 759 Total reward: 170.0 Training loss: 0.2819 Explore P: 0.0170
Episode: 760 Total reward: 190.0 Training loss: 0.2683 Explore P: 0.0169
Episode: 761 Total reward: 199.0 Training loss: 37.4399 Explore P: 0.0167
Episode: 762 Total reward: 199.0 Training loss: 0.2745 Explore P: 0.0166
Episode: 763 Total reward: 199.0 Training loss: 0.3535 Explore P: 0.0165
Episode: 764 Total reward: 199.0 Training loss: 0.2349 Explore P: 0.0163
Episode: 765 Total reward: 199.0 Training loss: 0.4036 Explore P: 0.0162
Episode: 766 Total reward: 199.0 Training loss: 0.5008 Explore P: 0.0161
Episode: 767 Total reward: 199.0 Training loss: 1.8527 Explore P: 0.0160
Episode: 768 Total reward: 75.0 Training loss: 2.8535 Explore P: 0.0159
Episode: 769 Total reward: 35.0 Training loss: 148.0324 Explore P: 0.0159
Episode: 770 Total reward: 31.0 Training loss: 154.4424 Explore P: 0.0159
Episode: 771 Total reward: 26.0 Training loss: 5.4478 Explore P: 0.0159
Episode: 772 Total reward: 18.0 Training loss: 5.1143 Explore P: 0.0159
Episode: 773 Total reward: 14.0 Training loss: 5.6651 Explore P: 0.0159
Episode: 774 Total reward: 11.0 Training loss: 591.9304 Explore P: 0.0159
Episode: 775 Total reward: 13.0 Training loss: 8.0600 Explore P: 0.0158
Episode: 776 Total reward: 14.0 Training loss: 9.6898 Explore P: 0.0158
Episode: 777 Total reward: 13.0 Training loss: 7.4176 Explore P: 0.0158
Episode: 778 Total reward: 12.0 Training loss: 5.4555 Explore P: 0.0158
Episode: 779 Total reward: 12.0 Training loss: 373.7031 Explore P: 0.0158
Episode: 780 Total reward: 11.0 Training loss: 960.1841 Explore P: 0.0158
Episode: 781 Total reward: 11.0 Training loss: 8.9984 Explore P: 0.0158
Episode: 782 Total reward: 11.0 Training loss: 595.7084 Explore P: 0.0158
Episode: 783 Total reward: 11.0 Training loss: 12.0014 Explore P: 0.0158
Episode: 784 Total reward: 11.0 Training loss: 6.4664 Explore P: 0.0158
Episode: 785 Total reward: 11.0 Training loss: 8.9147 Explore P: 0.0158
Episode: 786 Total reward: 11.0 Training loss: 9.3465 Explore P: 0.0158
Episode: 787 Total reward: 10.0 Training loss: 17.3452 Explore P: 0.0158
Episode: 788 Total reward: 12.0 Training loss: 16.8506 Explore P: 0.0158
Episode: 789 Total reward: 8.0 Training loss: 16.2138 Explore P: 0.0158
Episode: 790 Total reward: 9.0 Training loss: 20.7453 Explore P: 0.0157
Episode: 791 Total reward: 11.0 Training loss: 13.8549 Explore P: 0.0157
Episode: 792 Total reward: 11.0 Training loss: 1975.3300 Explore P: 0.0157
Episode: 793 Total reward: 12.0 Training loss: 980.0167 Explore P: 0.0157
Episode: 794 Total reward: 9.0 Training loss: 19.4801 Explore P: 0.0157
Episode: 795 Total reward: 7.0 Training loss: 22.1316 Explore P: 0.0157
Episode: 796 Total reward: 10.0 Training loss: 22.9151 Explore P: 0.0157
Episode: 797 Total reward: 8.0 Training loss: 1258.3632 Explore P: 0.0157
Episode: 798 Total reward: 8.0 Training loss: 21.3545 Explore P: 0.0157
Episode: 799 Total reward: 7.0 Training loss: 26.9820 Explore P: 0.0157
Episode: 800 Total reward: 10.0 Training loss: 26.4484 Explore P: 0.0157
Episode: 801 Total reward: 12.0 Training loss: 1199.9598 Explore P: 0.0157
Episode: 802 Total reward: 10.0 Training loss: 23.6516 Explore P: 0.0157
Episode: 803 Total reward: 11.0 Training loss: 23.4732 Explore P: 0.0157
Episode: 804 Total reward: 9.0 Training loss: 33.4697 Explore P: 0.0157
Episode: 805 Total reward: 11.0 Training loss: 26.1122 Explore P: 0.0157
Episode: 806 Total reward: 10.0 Training loss: 36.5750 Explore P: 0.0157
Episode: 807 Total reward: 11.0 Training loss: 30.6061 Explore P: 0.0157
Episode: 808 Total reward: 9.0 Training loss: 27.3712 Explore P: 0.0156
Episode: 809 Total reward: 8.0 Training loss: 27.5319 Explore P: 0.0156
Episode: 810 Total reward: 7.0 Training loss: 1494.5630 Explore P: 0.0156
Episode: 811 Total reward: 9.0 Training loss: 24.8773 Explore P: 0.0156
Episode: 812 Total reward: 8.0 Training loss: 22.4522 Explore P: 0.0156
Episode: 813 Total reward: 9.0 Training loss: 17.9110 Explore P: 0.0156
Episode: 814 Total reward: 9.0 Training loss: 16.5346 Explore P: 0.0156
Episode: 815 Total reward: 12.0 Training loss: 1411.2380 Explore P: 0.0156
Episode: 816 Total reward: 8.0 Training loss: 12.6134 Explore P: 0.0156
Episode: 817 Total reward: 11.0 Training loss: 16.7752 Explore P: 0.0156
Episode: 818 Total reward: 8.0 Training loss: 1758.9554 Explore P: 0.0156
Episode: 819 Total reward: 11.0 Training loss: 15.2404 Explore P: 0.0156
Episode: 820 Total reward: 11.0 Training loss: 10.9318 Explore P: 0.0156
Episode: 821 Total reward: 11.0 Training loss: 12.3753 Explore P: 0.0156
Episode: 822 Total reward: 9.0 Training loss: 9.6393 Explore P: 0.0156
Episode: 823 Total reward: 11.0 Training loss: 9.4524 Explore P: 0.0156
Episode: 824 Total reward: 11.0 Training loss: 1685.9849 Explore P: 0.0156
Episode: 825 Total reward: 10.0 Training loss: 8.9818 Explore P: 0.0156
Episode: 826 Total reward: 12.0 Training loss: 14.4835 Explore P: 0.0156
Episode: 827 Total reward: 9.0 Training loss: 1437.7875 Explore P: 0.0155
Episode: 828 Total reward: 11.0 Training loss: 9.6167 Explore P: 0.0155
Episode: 829 Total reward: 9.0 Training loss: 1397.3116 Explore P: 0.0155
Episode: 830 Total reward: 12.0 Training loss: 8.0831 Explore P: 0.0155
Episode: 831 Total reward: 12.0 Training loss: 10.3349 Explore P: 0.0155
Episode: 832 Total reward: 15.0 Training loss: 9.1107 Explore P: 0.0155
Episode: 833 Total reward: 14.0 Training loss: 1704.7867 Explore P: 0.0155
Episode: 834 Total reward: 14.0 Training loss: 12.7733 Explore P: 0.0155
Episode: 835 Total reward: 12.0 Training loss: 10.3284 Explore P: 0.0155
Episode: 836 Total reward: 17.0 Training loss: 7.4346 Explore P: 0.0155
Episode: 837 Total reward: 22.0 Training loss: 8.3414 Explore P: 0.0155
Episode: 838 Total reward: 29.0 Training loss: 2962.3210 Explore P: 0.0155
Episode: 839 Total reward: 35.0 Training loss: 14.3920 Explore P: 0.0154
Episode: 840 Total reward: 41.0 Training loss: 6.4658 Explore P: 0.0154
Episode: 841 Total reward: 61.0 Training loss: 6.8598 Explore P: 0.0154
Episode: 842 Total reward: 84.0 Training loss: 4.2329 Explore P: 0.0153
Episode: 843 Total reward: 199.0 Training loss: 4.9140 Explore P: 0.0152
Episode: 844 Total reward: 199.0 Training loss: 2.3713 Explore P: 0.0151
Episode: 845 Total reward: 199.0 Training loss: 1232.5986 Explore P: 0.0150
Episode: 846 Total reward: 199.0 Training loss: 5.4302 Explore P: 0.0149
Episode: 847 Total reward: 199.0 Training loss: 93.3932 Explore P: 0.0148
Episode: 848 Total reward: 199.0 Training loss: 289.0457 Explore P: 0.0147
Episode: 849 Total reward: 199.0 Training loss: 75.5415 Explore P: 0.0146
Episode: 850 Total reward: 199.0 Training loss: 3.0538 Explore P: 0.0145
Episode: 851 Total reward: 199.0 Training loss: 330.7434 Explore P: 0.0145
Episode: 852 Total reward: 199.0 Training loss: 61.6567 Explore P: 0.0144
Episode: 853 Total reward: 199.0 Training loss: 1.0513 Explore P: 0.0143
Episode: 854 Total reward: 199.0 Training loss: 4.3808 Explore P: 0.0142
Episode: 855 Total reward: 199.0 Training loss: 2.3188 Explore P: 0.0141
Episode: 856 Total reward: 199.0 Training loss: 3.2048 Explore P: 0.0140
Episode: 857 Total reward: 199.0 Training loss: 0.8991 Explore P: 0.0140
Episode: 858 Total reward: 199.0 Training loss: 146.5759 Explore P: 0.0139
Episode: 859 Total reward: 199.0 Training loss: 2.4078 Explore P: 0.0138
Episode: 860 Total reward: 199.0 Training loss: 2.1784 Explore P: 0.0137
Episode: 861 Total reward: 199.0 Training loss: 0.9660 Explore P: 0.0137
Episode: 862 Total reward: 199.0 Training loss: 0.9386 Explore P: 0.0136
Episode: 863 Total reward: 199.0 Training loss: 6.0319 Explore P: 0.0135
Episode: 864 Total reward: 199.0 Training loss: 8.5772 Explore P: 0.0134
Episode: 865 Total reward: 199.0 Training loss: 1.2786 Explore P: 0.0134
Episode: 866 Total reward: 199.0 Training loss: 100.4507 Explore P: 0.0133
Episode: 867 Total reward: 199.0 Training loss: 3.5499 Explore P: 0.0132
Episode: 868 Total reward: 199.0 Training loss: 4.0679 Explore P: 0.0132
Episode: 869 Total reward: 187.0 Training loss: 25.1083 Explore P: 0.0131
Episode: 870 Total reward: 199.0 Training loss: 1.1960 Explore P: 0.0131
Episode: 871 Total reward: 23.0 Training loss: 1.7546 Explore P: 0.0131
Episode: 872 Total reward: 199.0 Training loss: 0.4881 Explore P: 0.0130
Episode: 873 Total reward: 199.0 Training loss: 112.8861 Explore P: 0.0129
Episode: 874 Total reward: 199.0 Training loss: 10.6608 Explore P: 0.0129
Episode: 875 Total reward: 199.0 Training loss: 0.9530 Explore P: 0.0128
Episode: 876 Total reward: 199.0 Training loss: 5.4341 Explore P: 0.0128
Episode: 877 Total reward: 199.0 Training loss: 10.6834 Explore P: 0.0127
Episode: 878 Total reward: 199.0 Training loss: 15.1466 Explore P: 0.0127
Episode: 879 Total reward: 199.0 Training loss: 2.8011 Explore P: 0.0126
Episode: 880 Total reward: 199.0 Training loss: 1.5876 Explore P: 0.0126
Episode: 881 Total reward: 199.0 Training loss: 8.9239 Explore P: 0.0125
Episode: 882 Total reward: 199.0 Training loss: 3.3218 Explore P: 0.0125
Episode: 883 Total reward: 199.0 Training loss: 59.6555 Explore P: 0.0124
Episode: 884 Total reward: 199.0 Training loss: 9.8491 Explore P: 0.0124
Episode: 885 Total reward: 199.0 Training loss: 12.2038 Explore P: 0.0123
Episode: 886 Total reward: 199.0 Training loss: 203.5361 Explore P: 0.0123
Episode: 887 Total reward: 199.0 Training loss: 9.3231 Explore P: 0.0122
Episode: 888 Total reward: 199.0 Training loss: 6.9743 Explore P: 0.0122
Episode: 889 Total reward: 199.0 Training loss: 4.2461 Explore P: 0.0121
Episode: 890 Total reward: 199.0 Training loss: 1.4453 Explore P: 0.0121
Episode: 891 Total reward: 199.0 Training loss: 2.4222 Explore P: 0.0121
Episode: 892 Total reward: 199.0 Training loss: 0.4468 Explore P: 0.0120
Episode: 893 Total reward: 199.0 Training loss: 0.4094 Explore P: 0.0120
Episode: 894 Total reward: 199.0 Training loss: 99.8225 Explore P: 0.0119
Episode: 895 Total reward: 199.0 Training loss: 0.3762 Explore P: 0.0119
Episode: 896 Total reward: 199.0 Training loss: 0.9595 Explore P: 0.0119
Episode: 897 Total reward: 199.0 Training loss: 0.6896 Explore P: 0.0118
Episode: 898 Total reward: 199.0 Training loss: 0.3152 Explore P: 0.0118
Episode: 899 Total reward: 199.0 Training loss: 245.7176 Explore P: 0.0117
Episode: 900 Total reward: 199.0 Training loss: 0.4408 Explore P: 0.0117
Episode: 901 Total reward: 199.0 Training loss: 0.1884 Explore P: 0.0117
Episode: 902 Total reward: 199.0 Training loss: 0.1825 Explore P: 0.0116
Episode: 903 Total reward: 199.0 Training loss: 0.4235 Explore P: 0.0116
Episode: 904 Total reward: 199.0 Training loss: 305.1464 Explore P: 0.0116
Episode: 905 Total reward: 199.0 Training loss: 0.7576 Explore P: 0.0116
Episode: 906 Total reward: 199.0 Training loss: 0.2161 Explore P: 0.0115
Episode: 907 Total reward: 199.0 Training loss: 0.2421 Explore P: 0.0115
Episode: 908 Total reward: 199.0 Training loss: 0.2278 Explore P: 0.0115
Episode: 909 Total reward: 199.0 Training loss: 0.1054 Explore P: 0.0114
Episode: 910 Total reward: 199.0 Training loss: 0.5683 Explore P: 0.0114
Episode: 911 Total reward: 199.0 Training loss: 0.3151 Explore P: 0.0114
Episode: 912 Total reward: 199.0 Training loss: 0.2195 Explore P: 0.0114
Episode: 913 Total reward: 199.0 Training loss: 0.2123 Explore P: 0.0113
Episode: 914 Total reward: 199.0 Training loss: 0.5331 Explore P: 0.0113
Episode: 915 Total reward: 199.0 Training loss: 0.4552 Explore P: 0.0113
Episode: 916 Total reward: 199.0 Training loss: 0.4098 Explore P: 0.0112
Episode: 917 Total reward: 199.0 Training loss: 0.4852 Explore P: 0.0112
Episode: 918 Total reward: 199.0 Training loss: 0.4903 Explore P: 0.0112
Episode: 919 Total reward: 199.0 Training loss: 0.6835 Explore P: 0.0112
Episode: 920 Total reward: 199.0 Training loss: 0.7577 Explore P: 0.0112
Episode: 921 Total reward: 199.0 Training loss: 21.4523 Explore P: 0.0111
Episode: 922 Total reward: 199.0 Training loss: 0.7297 Explore P: 0.0111
Episode: 923 Total reward: 199.0 Training loss: 0.5634 Explore P: 0.0111
Episode: 924 Total reward: 199.0 Training loss: 0.4518 Explore P: 0.0111
Episode: 925 Total reward: 199.0 Training loss: 0.4575 Explore P: 0.0110
Episode: 926 Total reward: 199.0 Training loss: 0.3465 Explore P: 0.0110
Episode: 927 Total reward: 199.0 Training loss: 0.5357 Explore P: 0.0110
Episode: 928 Total reward: 199.0 Training loss: 0.4640 Explore P: 0.0110
Episode: 929 Total reward: 199.0 Training loss: 0.3394 Explore P: 0.0110
Episode: 930 Total reward: 199.0 Training loss: 0.6234 Explore P: 0.0109
Episode: 931 Total reward: 199.0 Training loss: 0.6385 Explore P: 0.0109
Episode: 932 Total reward: 199.0 Training loss: 0.3004 Explore P: 0.0109
Episode: 933 Total reward: 199.0 Training loss: 0.5557 Explore P: 0.0109
Episode: 934 Total reward: 199.0 Training loss: 0.3823 Explore P: 0.0109
Episode: 935 Total reward: 199.0 Training loss: 0.2407 Explore P: 0.0109
Episode: 936 Total reward: 199.0 Training loss: 0.2773 Explore P: 0.0108
Episode: 937 Total reward: 199.0 Training loss: 69.6006 Explore P: 0.0108
Episode: 938 Total reward: 199.0 Training loss: 0.2382 Explore P: 0.0108
Episode: 939 Total reward: 199.0 Training loss: 0.3433 Explore P: 0.0108
Episode: 940 Total reward: 199.0 Training loss: 0.2395 Explore P: 0.0108
Episode: 941 Total reward: 199.0 Training loss: 0.3894 Explore P: 0.0108
Episode: 942 Total reward: 199.0 Training loss: 0.2475 Explore P: 0.0107
Episode: 943 Total reward: 199.0 Training loss: 0.2117 Explore P: 0.0107
Episode: 944 Total reward: 199.0 Training loss: 0.2771 Explore P: 0.0107
Episode: 945 Total reward: 199.0 Training loss: 0.2079 Explore P: 0.0107
Episode: 946 Total reward: 199.0 Training loss: 0.2999 Explore P: 0.0107
Episode: 947 Total reward: 199.0 Training loss: 0.2151 Explore P: 0.0107
Episode: 948 Total reward: 199.0 Training loss: 0.4279 Explore P: 0.0107
Episode: 949 Total reward: 199.0 Training loss: 0.1842 Explore P: 0.0106
Episode: 950 Total reward: 199.0 Training loss: 0.3059 Explore P: 0.0106
Episode: 951 Total reward: 199.0 Training loss: 0.2066 Explore P: 0.0106
Episode: 952 Total reward: 199.0 Training loss: 0.2269 Explore P: 0.0106
Episode: 953 Total reward: 199.0 Training loss: 0.1992 Explore P: 0.0106
Episode: 954 Total reward: 199.0 Training loss: 0.1579 Explore P: 0.0106
Episode: 955 Total reward: 199.0 Training loss: 0.1446 Explore P: 0.0106
Episode: 956 Total reward: 199.0 Training loss: 0.2887 Explore P: 0.0106
Episode: 957 Total reward: 199.0 Training loss: 0.1113 Explore P: 0.0106
Episode: 958 Total reward: 199.0 Training loss: 0.1901 Explore P: 0.0105
Episode: 959 Total reward: 199.0 Training loss: 0.0884 Explore P: 0.0105
Episode: 960 Total reward: 199.0 Training loss: 0.3229 Explore P: 0.0105
Episode: 961 Total reward: 199.0 Training loss: 0.1960 Explore P: 0.0105
Episode: 962 Total reward: 199.0 Training loss: 0.1165 Explore P: 0.0105
Episode: 963 Total reward: 199.0 Training loss: 0.2153 Explore P: 0.0105
Episode: 964 Total reward: 199.0 Training loss: 0.1342 Explore P: 0.0105
Episode: 965 Total reward: 199.0 Training loss: 0.2474 Explore P: 0.0105
Episode: 966 Total reward: 199.0 Training loss: 0.1029 Explore P: 0.0105
Episode: 967 Total reward: 199.0 Training loss: 0.1852 Explore P: 0.0105
Episode: 968 Total reward: 199.0 Training loss: 0.1313 Explore P: 0.0104
Episode: 969 Total reward: 199.0 Training loss: 0.1610 Explore P: 0.0104
Episode: 970 Total reward: 199.0 Training loss: 0.2061 Explore P: 0.0104
Episode: 971 Total reward: 199.0 Training loss: 0.1041 Explore P: 0.0104
Episode: 972 Total reward: 199.0 Training loss: 0.1653 Explore P: 0.0104
Episode: 973 Total reward: 199.0 Training loss: 0.1682 Explore P: 0.0104
Episode: 974 Total reward: 199.0 Training loss: 0.0519 Explore P: 0.0104
Episode: 975 Total reward: 199.0 Training loss: 0.1639 Explore P: 0.0104
Episode: 976 Total reward: 199.0 Training loss: 0.0846 Explore P: 0.0104
Episode: 977 Total reward: 199.0 Training loss: 0.0593 Explore P: 0.0104
Episode: 978 Total reward: 199.0 Training loss: 0.0522 Explore P: 0.0104
Episode: 979 Total reward: 199.0 Training loss: 196.5552 Explore P: 0.0104
Episode: 980 Total reward: 199.0 Training loss: 112.7854 Explore P: 0.0103
Episode: 981 Total reward: 199.0 Training loss: 0.1098 Explore P: 0.0103
Episode: 982 Total reward: 199.0 Training loss: 119.3835 Explore P: 0.0103
Episode: 983 Total reward: 199.0 Training loss: 0.3272 Explore P: 0.0103
Episode: 984 Total reward: 199.0 Training loss: 0.2872 Explore P: 0.0103
Episode: 985 Total reward: 199.0 Training loss: 0.1907 Explore P: 0.0103
Episode: 986 Total reward: 199.0 Training loss: 0.1777 Explore P: 0.0103
Episode: 987 Total reward: 199.0 Training loss: 93.1428 Explore P: 0.0103
Episode: 988 Total reward: 199.0 Training loss: 0.1818 Explore P: 0.0103
Episode: 989 Total reward: 199.0 Training loss: 0.1872 Explore P: 0.0103
Episode: 990 Total reward: 199.0 Training loss: 0.0902 Explore P: 0.0103
Episode: 991 Total reward: 199.0 Training loss: 0.2229 Explore P: 0.0103
Episode: 992 Total reward: 199.0 Training loss: 0.1591 Explore P: 0.0103
Episode: 993 Total reward: 199.0 Training loss: 0.2108 Explore P: 0.0103
Episode: 994 Total reward: 199.0 Training loss: 0.1574 Explore P: 0.0103
Episode: 995 Total reward: 199.0 Training loss: 0.1063 Explore P: 0.0103
Episode: 996 Total reward: 199.0 Training loss: 0.2037 Explore P: 0.0103
Episode: 997 Total reward: 199.0 Training loss: 65.6803 Explore P: 0.0102
Episode: 998 Total reward: 199.0 Training loss: 0.6028 Explore P: 0.0102
Episode: 999 Total reward: 199.0 Training loss: 0.1940 Explore P: 0.0102

Visualizing training

Below I'll plot the total rewards for each episode. I'm plotting the rolling average too, in blue.


In [19]:
%matplotlib inline
import matplotlib.pyplot as plt

def running_mean(x, N):
    cumsum = np.cumsum(np.insert(x, 0, 0)) 
    return (cumsum[N:] - cumsum[:-N]) / N

In [20]:
eps, rews = np.array(rewards_list).T
smoothed_rews = running_mean(rews, 10)
plt.plot(eps[-len(smoothed_rews):], smoothed_rews)
plt.plot(eps, rews, color='grey', alpha=0.3)
plt.xlabel('Episode')
plt.ylabel('Total Reward')


Out[20]:
<matplotlib.text.Text at 0x7f9816991ba8>

Testing

Let's checkout how our trained agent plays the game.


In [21]:
test_episodes = 10
test_max_steps = 400
env.reset()
with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('checkpoints'))
    
    for ep in range(1, test_episodes):
        t = 0
        while t < test_max_steps:
            env.render() 
            
            # Get action from Q-network
            feed = {mainQN.inputs_: state.reshape((1, *state.shape))}
            Qs = sess.run(mainQN.output, feed_dict=feed)
            action = np.argmax(Qs)
            
            # Take action, get new state and reward
            next_state, reward, done, _ = env.step(action)
            
            if done:
                t = test_max_steps
                env.reset()
                # Take one random step to get the pole and cart moving
                state, reward, done, _ = env.step(env.action_space.sample())

            else:
                state = next_state
                t += 1


INFO:tensorflow:Restoring parameters from checkpoints/cartpole.ckpt
[2017-08-28 12:42:04,409] Restoring parameters from checkpoints/cartpole.ckpt

In [22]:
env.close()

Extending this

So, Cart-Pole is a pretty simple game. However, the same model can be used to train an agent to play something much more complicated like Pong or Space Invaders. Instead of a state like we're using here though, you'd want to use convolutional layers to get the state from the screen images.

I'll leave it as a challenge for you to use deep Q-learning to train an agent to play Atari games. Here's the original paper which will get you started: http://www.davidqiu.com:8888/research/nature14236.pdf.