In [7]:
import sys
sys.path.append('../deeprl')
import gym
import tensorflow as tf
import tflearn as nn
import numpy as np
#create the environment
env = gym.make('CartPole-v1')
[2017-02-26 19:20:48,884] Making new env: CartPole-v1
First we create a policy network. This network takes an observation as input and returns action porbabilities.
In [2]:
def create_mlp(inputs, n_out):
net = nn.input_data(placeholder=inputs)
net = nn.fully_connected(net, 25, activation='relu', weight_decay=0.0001)
net = nn.dropout(net, 0.3)
net = nn.fully_connected(net, 25, activation='relu', weight_decay=0.0001)
net = nn.dropout(net, 0.3)
#softmax to normalize to probabilities
net = nn.fully_connected(net, n_out, activation='softmax')
return net
We can now apply this network to the environment. On every step the network produces action probabilities for the current observation that can be used to select actions
In [12]:
# create a policy network
nn_input = tf.placeholder("float",(None,)+env.observation_space.shape)
n_acts = env.action_space.n
policy = create_mlp(nn_input, env.action_space.n)
# run the network for a single episode
obs = env.reset()
done = False
session = tf.Session()
session.run(tf.global_variables_initializer())
steps = 0
while not done:
# get action probs for current observation
act_probs = policy.eval(session=session,feed_dict={nn_input:obs.reshape(1,-1)}).flatten()
# select action according to probs
act = np.random.choice(n_acts, p = act_probs)
#step environment
obs,rew,done,_ = env.step(act)
print('step % d reward %f'%(steps,rew))
steps += 1
step 0 reward 1.000000
step 1 reward 1.000000
step 2 reward 1.000000
step 3 reward 1.000000
step 4 reward 1.000000
step 5 reward 1.000000
step 6 reward 1.000000
step 7 reward 1.000000
step 8 reward 1.000000
step 9 reward 1.000000
step 10 reward 1.000000
step 11 reward 1.000000
step 12 reward 1.000000
step 13 reward 1.000000
step 14 reward 1.000000
As the network uses randomely initialized weights, we cannot expect the resulting policy to perform well. To goal of learning now becomes to adapt the network weights such that the performance of the policy improves. In order to do this, we first define a cost function that we want to optimize:
$$ J(\theta) = E_{\pi_\theta}[R] = \sum_\tau p(\tau \mid \pi_\theta)R(\tau) $$This cost function indicates we want to optimize the expected return (sum of discounted rewards) for the policy. Here $\pi_\theta$ is the policy represented by the network with weights $\theta$, $p(\tau \mid \pi_\theta)$ is the probability of observing a trajectory (episode) $\tau= s_0,a_0,s_1,a_t,...,s_T$ when applying this policy and $R(\tau)$ is the return for this trajectory, ie $R = \sum_{t=0}^T \gamma^t r_{t+1}$. Changing our policy changes the probability $p$ of achieving a trajectory, so essentially this cost says that we want to change the policy in such a way that the probability of trajectories which have a high return increases.
In order to optimize this cost function, we calculate the gradient w.r.t. the network weights:
$$\nabla_\theta J(\theta) = \nabla_\theta \sum_\tau p(\tau \mid \pi_\theta)R(\tau) = \sum_\tau \nabla_\theta p(\tau \mid \pi_\theta)R(\tau)$$Using $\nabla_\theta p(\tau \mid \pi_\theta) = \frac{p(\tau \mid \pi_\theta)}{p(\tau \mid \pi_\theta)} \nabla_\theta p(\tau \mid \pi_\theta)$ and $\nabla_\theta log p(\tau \mid \pi_\theta) = \frac{1}{p(\tau \mid \pi_\theta)}\nabla_\theta p(\tau \mid \pi_\theta)$ we can rewrite this as:
$$ \sum_\tau p(\tau \mid \pi_\theta) \nabla_\theta log p(\tau \mid \pi_\theta) R(\tau)$$$$ = E_{\pi_\theta}[ \nabla_\theta log p(\tau \mid \pi_\theta) R(\tau) ] $$Where $E_{\pi_\theta}$ is an expectation over trajectories generated by the policy. This means we can sample this expectation by running the policy and averaging over the generated trajectories.
Moreover, we do not actually need to compute the the probability $p(\tau \mid \pi_\theta)$ of a trajectory. This probability of $\tau= s_0,a_0,s_1,a_t,...,s_T$ is $\prod _{t=0}^T T(s_t,a_t,s_{t+1}) \pi_\theta(s_t,a_t)$, where $T(s_0,a_0,s_1)$ are the transition probabilities of the MDP. Since $T$ does not depend on the network weights, we only have to look at $\pi$ to compute the gradient $ \nabla_\theta log p(\tau \mid \pi_\theta)$:
$$\nabla_\theta log p(\tau \mid \pi_\theta)= \nabla_\theta log \prod _{t=0}^T \pi_\theta(s_t,a_t)T(s_t,a_t,s_{t+1}) \\ = \sum_{t=0}^T \nabla_\theta log \pi_\theta(s_t,a_t) $$This results in the final formula $\nabla_\theta J(\theta) = E_{\pi_\theta}[ \sum_{t=0}^T \nabla_\theta log \pi_\theta(s_t,a_t) R(\tau) ]$, i.e. to compute the policy gradient from a trajectory, we sum over the log of the action probabilities output by the network and multiply by the return.
This results in the Monte Carlo Policy gradient (REINFORCE) algorithm:
In tensorflow this can be implemented by defining the following loss:
loss_op = -tf.reduce_mean(tf.log(probs) * R)
where probs is the output of the network for the selected actions for each time step and R is the return (here: sum of discounted rewards) starting from that time step. Here we take a mean over time steps rather than a sum. This helps with variations in magnitude because trajectories can have very different lengths.
You can run the full policy gradient algorithm using the DPGAgent:
In [14]:
from agents import DPGAgent
from experiment import Experiment
#erase previously created tf graphs
tf.reset_default_graph()
#create a DQNAgent
agent = DPGAgent(create_mlp, env.action_space.n, env.observation_space.shape, alpha=0.001, gamma=0.99)
# create experiment on cartpole
exp = Experiment(agent, env)
#run agent for 150K steps and return episode rewards
rew = exp.run_epoch(150000)
WARNING:tensorflow:From ../deeprl/agents.py:224: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use `tf.global_variables_initializer` instead.
[2017-02-26 20:18:37,257] From ../deeprl/agents.py:224: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use `tf.global_variables_initializer` instead.
episode reward: 15.000000
episode reward: 20.000000
episode reward: 18.000000
episode reward: 29.000000
episode reward: 14.000000
episode reward: 35.000000
episode reward: 11.000000
episode reward: 9.000000
episode reward: 12.000000
episode reward: 21.000000
episode reward: 20.000000
episode reward: 20.000000
episode reward: 33.000000
episode reward: 17.000000
episode reward: 32.000000
episode reward: 26.000000
episode reward: 20.000000
episode reward: 22.000000
episode reward: 38.000000
episode reward: 25.000000
episode reward: 11.000000
episode reward: 14.000000
episode reward: 10.000000
episode reward: 16.000000
episode reward: 15.000000
episode reward: 63.000000
episode reward: 44.000000
episode reward: 17.000000
episode reward: 20.000000
episode reward: 12.000000
episode reward: 19.000000
episode reward: 15.000000
episode reward: 13.000000
episode reward: 12.000000
episode reward: 17.000000
episode reward: 25.000000
episode reward: 45.000000
episode reward: 11.000000
episode reward: 16.000000
episode reward: 11.000000
episode reward: 21.000000
episode reward: 21.000000
episode reward: 13.000000
episode reward: 12.000000
episode reward: 28.000000
episode reward: 14.000000
episode reward: 10.000000
episode reward: 22.000000
episode reward: 55.000000
episode reward: 88.000000
episode reward: 10.000000
episode reward: 14.000000
episode reward: 34.000000
episode reward: 26.000000
episode reward: 13.000000
episode reward: 41.000000
episode reward: 25.000000
episode reward: 13.000000
episode reward: 16.000000
episode reward: 12.000000
episode reward: 41.000000
episode reward: 30.000000
episode reward: 14.000000
episode reward: 15.000000
episode reward: 32.000000
episode reward: 52.000000
episode reward: 13.000000
episode reward: 33.000000
episode reward: 10.000000
episode reward: 13.000000
episode reward: 12.000000
episode reward: 48.000000
episode reward: 19.000000
episode reward: 51.000000
episode reward: 43.000000
episode reward: 31.000000
episode reward: 15.000000
episode reward: 23.000000
episode reward: 20.000000
episode reward: 51.000000
episode reward: 30.000000
episode reward: 27.000000
episode reward: 17.000000
episode reward: 13.000000
episode reward: 49.000000
episode reward: 28.000000
episode reward: 11.000000
episode reward: 34.000000
episode reward: 18.000000
episode reward: 27.000000
episode reward: 28.000000
episode reward: 16.000000
episode reward: 18.000000
episode reward: 22.000000
episode reward: 10.000000
episode reward: 12.000000
episode reward: 16.000000
episode reward: 59.000000
episode reward: 21.000000
episode reward: 13.000000
episode reward: 14.000000
episode reward: 21.000000
episode reward: 29.000000
episode reward: 26.000000
episode reward: 23.000000
episode reward: 10.000000
episode reward: 16.000000
episode reward: 11.000000
episode reward: 23.000000
episode reward: 19.000000
episode reward: 12.000000
episode reward: 15.000000
episode reward: 37.000000
episode reward: 21.000000
episode reward: 27.000000
episode reward: 11.000000
episode reward: 23.000000
episode reward: 16.000000
episode reward: 11.000000
episode reward: 15.000000
episode reward: 12.000000
episode reward: 13.000000
episode reward: 36.000000
episode reward: 18.000000
episode reward: 40.000000
episode reward: 13.000000
episode reward: 12.000000
episode reward: 19.000000
episode reward: 59.000000
episode reward: 14.000000
episode reward: 13.000000
episode reward: 9.000000
episode reward: 18.000000
episode reward: 20.000000
episode reward: 22.000000
episode reward: 15.000000
episode reward: 37.000000
episode reward: 27.000000
episode reward: 33.000000
episode reward: 27.000000
episode reward: 12.000000
episode reward: 9.000000
episode reward: 21.000000
episode reward: 9.000000
episode reward: 16.000000
episode reward: 19.000000
episode reward: 13.000000
episode reward: 10.000000
episode reward: 11.000000
episode reward: 10.000000
episode reward: 64.000000
episode reward: 23.000000
episode reward: 75.000000
episode reward: 13.000000
episode reward: 24.000000
episode reward: 29.000000
episode reward: 21.000000
episode reward: 25.000000
episode reward: 20.000000
episode reward: 10.000000
episode reward: 19.000000
episode reward: 16.000000
episode reward: 24.000000
episode reward: 14.000000
episode reward: 18.000000
episode reward: 14.000000
episode reward: 10.000000
episode reward: 30.000000
episode reward: 16.000000
episode reward: 22.000000
episode reward: 9.000000
episode reward: 10.000000
episode reward: 34.000000
episode reward: 17.000000
episode reward: 21.000000
episode reward: 12.000000
episode reward: 76.000000
episode reward: 14.000000
episode reward: 24.000000
episode reward: 16.000000
episode reward: 9.000000
episode reward: 13.000000
episode reward: 33.000000
episode reward: 14.000000
episode reward: 12.000000
episode reward: 12.000000
episode reward: 13.000000
episode reward: 22.000000
episode reward: 26.000000
episode reward: 17.000000
episode reward: 13.000000
episode reward: 30.000000
episode reward: 27.000000
episode reward: 51.000000
episode reward: 33.000000
episode reward: 21.000000
episode reward: 20.000000
episode reward: 14.000000
episode reward: 26.000000
episode reward: 10.000000
episode reward: 39.000000
episode reward: 11.000000
episode reward: 21.000000
episode reward: 15.000000
episode reward: 13.000000
episode reward: 18.000000
episode reward: 12.000000
episode reward: 21.000000
episode reward: 21.000000
episode reward: 16.000000
episode reward: 17.000000
episode reward: 15.000000
episode reward: 18.000000
episode reward: 34.000000
episode reward: 12.000000
episode reward: 27.000000
episode reward: 15.000000
episode reward: 23.000000
episode reward: 23.000000
episode reward: 31.000000
episode reward: 23.000000
episode reward: 22.000000
episode reward: 21.000000
episode reward: 10.000000
episode reward: 34.000000
episode reward: 13.000000
episode reward: 10.000000
episode reward: 46.000000
episode reward: 16.000000
episode reward: 11.000000
episode reward: 31.000000
episode reward: 11.000000
episode reward: 34.000000
episode reward: 23.000000
episode reward: 30.000000
episode reward: 24.000000
episode reward: 15.000000
episode reward: 18.000000
episode reward: 14.000000
episode reward: 36.000000
episode reward: 16.000000
episode reward: 13.000000
episode reward: 23.000000
episode reward: 21.000000
episode reward: 15.000000
episode reward: 14.000000
episode reward: 13.000000
episode reward: 27.000000
episode reward: 33.000000
episode reward: 39.000000
episode reward: 21.000000
episode reward: 20.000000
episode reward: 15.000000
episode reward: 29.000000
episode reward: 13.000000
episode reward: 17.000000
episode reward: 16.000000
episode reward: 20.000000
episode reward: 15.000000
episode reward: 12.000000
episode reward: 19.000000
episode reward: 19.000000
episode reward: 24.000000
episode reward: 17.000000
episode reward: 11.000000
episode reward: 22.000000
episode reward: 21.000000
episode reward: 15.000000
episode reward: 12.000000
episode reward: 21.000000
episode reward: 11.000000
episode reward: 47.000000
episode reward: 20.000000
episode reward: 11.000000
episode reward: 29.000000
episode reward: 35.000000
episode reward: 50.000000
episode reward: 35.000000
episode reward: 14.000000
episode reward: 17.000000
episode reward: 21.000000
episode reward: 36.000000
episode reward: 21.000000
episode reward: 29.000000
episode reward: 16.000000
episode reward: 16.000000
episode reward: 37.000000
episode reward: 55.000000
episode reward: 18.000000
episode reward: 27.000000
episode reward: 16.000000
episode reward: 15.000000
episode reward: 19.000000
episode reward: 14.000000
episode reward: 24.000000
episode reward: 10.000000
episode reward: 37.000000
episode reward: 11.000000
episode reward: 19.000000
episode reward: 12.000000
episode reward: 68.000000
episode reward: 15.000000
episode reward: 25.000000
episode reward: 11.000000
episode reward: 18.000000
episode reward: 20.000000
episode reward: 41.000000
episode reward: 23.000000
episode reward: 38.000000
episode reward: 32.000000
episode reward: 17.000000
episode reward: 22.000000
episode reward: 15.000000
episode reward: 16.000000
episode reward: 21.000000
episode reward: 20.000000
episode reward: 17.000000
episode reward: 37.000000
episode reward: 17.000000
episode reward: 19.000000
episode reward: 20.000000
episode reward: 23.000000
episode reward: 43.000000
episode reward: 21.000000
episode reward: 9.000000
episode reward: 36.000000
episode reward: 32.000000
episode reward: 17.000000
episode reward: 60.000000
episode reward: 24.000000
episode reward: 27.000000
episode reward: 14.000000
episode reward: 30.000000
episode reward: 13.000000
episode reward: 12.000000
episode reward: 16.000000
episode reward: 15.000000
episode reward: 49.000000
episode reward: 15.000000
episode reward: 26.000000
episode reward: 11.000000
episode reward: 43.000000
episode reward: 23.000000
episode reward: 45.000000
episode reward: 31.000000
episode reward: 15.000000
episode reward: 20.000000
episode reward: 28.000000
episode reward: 29.000000
episode reward: 39.000000
episode reward: 13.000000
episode reward: 15.000000
episode reward: 17.000000
episode reward: 12.000000
episode reward: 64.000000
episode reward: 43.000000
episode reward: 44.000000
episode reward: 14.000000
episode reward: 9.000000
episode reward: 14.000000
episode reward: 12.000000
episode reward: 19.000000
episode reward: 24.000000
episode reward: 39.000000
episode reward: 79.000000
episode reward: 15.000000
episode reward: 11.000000
episode reward: 32.000000
episode reward: 12.000000
episode reward: 32.000000
episode reward: 17.000000
episode reward: 11.000000
episode reward: 17.000000
episode reward: 26.000000
episode reward: 16.000000
episode reward: 14.000000
episode reward: 52.000000
episode reward: 20.000000
episode reward: 66.000000
episode reward: 13.000000
episode reward: 15.000000
episode reward: 57.000000
episode reward: 42.000000
episode reward: 33.000000
episode reward: 18.000000
episode reward: 24.000000
episode reward: 13.000000
episode reward: 31.000000
episode reward: 37.000000
episode reward: 36.000000
episode reward: 27.000000
episode reward: 30.000000
episode reward: 30.000000
episode reward: 22.000000
episode reward: 35.000000
episode reward: 61.000000
episode reward: 44.000000
episode reward: 10.000000
episode reward: 23.000000
episode reward: 16.000000
episode reward: 23.000000
episode reward: 11.000000
episode reward: 14.000000
episode reward: 13.000000
episode reward: 21.000000
episode reward: 12.000000
episode reward: 45.000000
episode reward: 21.000000
episode reward: 23.000000
episode reward: 21.000000
episode reward: 21.000000
episode reward: 32.000000
episode reward: 79.000000
episode reward: 50.000000
episode reward: 30.000000
episode reward: 17.000000
episode reward: 17.000000
episode reward: 41.000000
episode reward: 41.000000
episode reward: 31.000000
episode reward: 11.000000
episode reward: 16.000000
episode reward: 14.000000
episode reward: 22.000000
episode reward: 16.000000
episode reward: 68.000000
episode reward: 20.000000
episode reward: 33.000000
episode reward: 15.000000
episode reward: 30.000000
episode reward: 15.000000
episode reward: 13.000000
episode reward: 27.000000
episode reward: 44.000000
episode reward: 16.000000
episode reward: 72.000000
episode reward: 12.000000
episode reward: 28.000000
episode reward: 23.000000
episode reward: 88.000000
episode reward: 10.000000
episode reward: 24.000000
episode reward: 19.000000
episode reward: 22.000000
episode reward: 28.000000
episode reward: 12.000000
episode reward: 15.000000
episode reward: 33.000000
episode reward: 17.000000
episode reward: 19.000000
episode reward: 47.000000
episode reward: 14.000000
episode reward: 19.000000
episode reward: 16.000000
episode reward: 16.000000
episode reward: 29.000000
episode reward: 35.000000
episode reward: 15.000000
episode reward: 13.000000
episode reward: 13.000000
episode reward: 17.000000
episode reward: 13.000000
episode reward: 18.000000
episode reward: 23.000000
episode reward: 29.000000
episode reward: 12.000000
episode reward: 29.000000
episode reward: 13.000000
episode reward: 21.000000
episode reward: 26.000000
episode reward: 21.000000
episode reward: 19.000000
episode reward: 32.000000
episode reward: 18.000000
episode reward: 16.000000
episode reward: 17.000000
episode reward: 13.000000
episode reward: 23.000000
episode reward: 17.000000
episode reward: 24.000000
episode reward: 14.000000
episode reward: 33.000000
episode reward: 29.000000
episode reward: 22.000000
episode reward: 15.000000
episode reward: 11.000000
episode reward: 38.000000
episode reward: 19.000000
episode reward: 12.000000
episode reward: 27.000000
episode reward: 16.000000
episode reward: 39.000000
episode reward: 13.000000
episode reward: 15.000000
episode reward: 14.000000
episode reward: 18.000000
episode reward: 28.000000
episode reward: 14.000000
episode reward: 17.000000
episode reward: 12.000000
episode reward: 13.000000
episode reward: 11.000000
episode reward: 25.000000
episode reward: 16.000000
episode reward: 16.000000
episode reward: 47.000000
episode reward: 31.000000
episode reward: 17.000000
episode reward: 23.000000
episode reward: 31.000000
episode reward: 13.000000
episode reward: 21.000000
episode reward: 27.000000
episode reward: 11.000000
episode reward: 29.000000
episode reward: 12.000000
episode reward: 10.000000
episode reward: 13.000000
episode reward: 25.000000
episode reward: 17.000000
episode reward: 32.000000
episode reward: 21.000000
episode reward: 28.000000
episode reward: 17.000000
episode reward: 18.000000
episode reward: 21.000000
episode reward: 35.000000
episode reward: 9.000000
episode reward: 20.000000
episode reward: 25.000000
episode reward: 9.000000
episode reward: 20.000000
episode reward: 13.000000
episode reward: 11.000000
episode reward: 26.000000
episode reward: 35.000000
episode reward: 26.000000
episode reward: 21.000000
episode reward: 30.000000
episode reward: 17.000000
episode reward: 25.000000
episode reward: 17.000000
episode reward: 13.000000
episode reward: 23.000000
episode reward: 13.000000
episode reward: 17.000000
episode reward: 17.000000
episode reward: 31.000000
episode reward: 20.000000
episode reward: 14.000000
episode reward: 20.000000
episode reward: 31.000000
episode reward: 20.000000
episode reward: 15.000000
episode reward: 39.000000
episode reward: 9.000000
episode reward: 42.000000
episode reward: 14.000000
episode reward: 19.000000
episode reward: 18.000000
episode reward: 28.000000
episode reward: 17.000000
episode reward: 21.000000
episode reward: 16.000000
episode reward: 22.000000
episode reward: 31.000000
episode reward: 22.000000
episode reward: 27.000000
episode reward: 18.000000
episode reward: 18.000000
episode reward: 16.000000
episode reward: 36.000000
episode reward: 13.000000
episode reward: 27.000000
episode reward: 14.000000
episode reward: 13.000000
episode reward: 36.000000
episode reward: 24.000000
episode reward: 28.000000
episode reward: 34.000000
episode reward: 18.000000
episode reward: 16.000000
episode reward: 14.000000
episode reward: 40.000000
episode reward: 13.000000
episode reward: 22.000000
episode reward: 16.000000
episode reward: 27.000000
episode reward: 17.000000
episode reward: 23.000000
episode reward: 12.000000
episode reward: 26.000000
episode reward: 31.000000
episode reward: 24.000000
episode reward: 80.000000
episode reward: 17.000000
episode reward: 40.000000
episode reward: 28.000000
episode reward: 19.000000
episode reward: 16.000000
episode reward: 23.000000
episode reward: 10.000000
episode reward: 24.000000
episode reward: 15.000000
episode reward: 17.000000
episode reward: 13.000000
episode reward: 15.000000
episode reward: 10.000000
episode reward: 29.000000
episode reward: 22.000000
episode reward: 24.000000
episode reward: 12.000000
episode reward: 27.000000
episode reward: 24.000000
episode reward: 28.000000
episode reward: 65.000000
episode reward: 13.000000
episode reward: 14.000000
episode reward: 16.000000
episode reward: 17.000000
episode reward: 23.000000
episode reward: 14.000000
episode reward: 20.000000
episode reward: 25.000000
episode reward: 21.000000
episode reward: 13.000000
episode reward: 12.000000
episode reward: 14.000000
episode reward: 24.000000
episode reward: 23.000000
episode reward: 27.000000
episode reward: 23.000000
episode reward: 18.000000
episode reward: 14.000000
episode reward: 23.000000
episode reward: 18.000000
episode reward: 16.000000
episode reward: 25.000000
episode reward: 32.000000
episode reward: 12.000000
episode reward: 29.000000
episode reward: 19.000000
episode reward: 94.000000
episode reward: 19.000000
episode reward: 13.000000
episode reward: 28.000000
episode reward: 18.000000
episode reward: 16.000000
episode reward: 36.000000
episode reward: 17.000000
episode reward: 18.000000
episode reward: 16.000000
episode reward: 30.000000
episode reward: 23.000000
episode reward: 11.000000
episode reward: 16.000000
episode reward: 12.000000
episode reward: 29.000000
episode reward: 23.000000
episode reward: 30.000000
episode reward: 34.000000
episode reward: 19.000000
episode reward: 14.000000
episode reward: 22.000000
episode reward: 13.000000
episode reward: 15.000000
episode reward: 82.000000
episode reward: 14.000000
episode reward: 19.000000
episode reward: 50.000000
episode reward: 22.000000
episode reward: 13.000000
episode reward: 18.000000
episode reward: 10.000000
episode reward: 19.000000
episode reward: 20.000000
episode reward: 23.000000
episode reward: 19.000000
episode reward: 13.000000
episode reward: 16.000000
episode reward: 33.000000
episode reward: 33.000000
episode reward: 27.000000
episode reward: 13.000000
episode reward: 29.000000
episode reward: 30.000000
episode reward: 13.000000
episode reward: 16.000000
episode reward: 13.000000
episode reward: 13.000000
episode reward: 16.000000
episode reward: 32.000000
episode reward: 25.000000
episode reward: 24.000000
episode reward: 14.000000
episode reward: 19.000000
episode reward: 11.000000
episode reward: 13.000000
episode reward: 19.000000
episode reward: 22.000000
episode reward: 11.000000
episode reward: 39.000000
episode reward: 16.000000
episode reward: 16.000000
episode reward: 21.000000
episode reward: 33.000000
episode reward: 10.000000
episode reward: 30.000000
episode reward: 17.000000
episode reward: 17.000000
episode reward: 24.000000
episode reward: 18.000000
episode reward: 19.000000
episode reward: 18.000000
episode reward: 11.000000
episode reward: 19.000000
episode reward: 19.000000
episode reward: 21.000000
episode reward: 25.000000
episode reward: 59.000000
episode reward: 14.000000
episode reward: 20.000000
episode reward: 18.000000
episode reward: 12.000000
episode reward: 11.000000
episode reward: 12.000000
episode reward: 32.000000
episode reward: 23.000000
episode reward: 33.000000
episode reward: 21.000000
episode reward: 11.000000
episode reward: 13.000000
episode reward: 17.000000
episode reward: 19.000000
episode reward: 15.000000
episode reward: 17.000000
episode reward: 18.000000
episode reward: 23.000000
episode reward: 55.000000
episode reward: 19.000000
episode reward: 14.000000
episode reward: 25.000000
episode reward: 21.000000
episode reward: 18.000000
episode reward: 22.000000
episode reward: 19.000000
episode reward: 49.000000
episode reward: 11.000000
episode reward: 27.000000
episode reward: 23.000000
episode reward: 12.000000
episode reward: 22.000000
episode reward: 14.000000
episode reward: 36.000000
episode reward: 35.000000
episode reward: 14.000000
episode reward: 22.000000
episode reward: 15.000000
episode reward: 20.000000
episode reward: 18.000000
episode reward: 27.000000
episode reward: 24.000000
episode reward: 44.000000
episode reward: 17.000000
episode reward: 11.000000
episode reward: 42.000000
episode reward: 21.000000
episode reward: 31.000000
episode reward: 20.000000
episode reward: 15.000000
episode reward: 33.000000
episode reward: 26.000000
episode reward: 22.000000
episode reward: 12.000000
episode reward: 12.000000
episode reward: 43.000000
episode reward: 46.000000
episode reward: 11.000000
episode reward: 16.000000
episode reward: 16.000000
episode reward: 15.000000
episode reward: 73.000000
episode reward: 28.000000
episode reward: 18.000000
episode reward: 29.000000
episode reward: 22.000000
episode reward: 11.000000
episode reward: 40.000000
episode reward: 15.000000
episode reward: 37.000000
episode reward: 31.000000
episode reward: 15.000000
episode reward: 13.000000
episode reward: 22.000000
episode reward: 23.000000
episode reward: 35.000000
episode reward: 24.000000
episode reward: 72.000000
episode reward: 16.000000
episode reward: 33.000000
episode reward: 16.000000
episode reward: 59.000000
episode reward: 22.000000
episode reward: 14.000000
episode reward: 28.000000
episode reward: 28.000000
episode reward: 13.000000
episode reward: 27.000000
episode reward: 44.000000
episode reward: 14.000000
episode reward: 36.000000
episode reward: 39.000000
episode reward: 28.000000
episode reward: 39.000000
episode reward: 23.000000
episode reward: 20.000000
episode reward: 18.000000
episode reward: 17.000000
episode reward: 18.000000
episode reward: 14.000000
episode reward: 23.000000
episode reward: 48.000000
episode reward: 10.000000
episode reward: 26.000000
episode reward: 14.000000
episode reward: 54.000000
episode reward: 38.000000
episode reward: 22.000000
episode reward: 34.000000
episode reward: 19.000000
episode reward: 15.000000
episode reward: 30.000000
episode reward: 35.000000
episode reward: 40.000000
episode reward: 55.000000
episode reward: 30.000000
episode reward: 31.000000
episode reward: 21.000000
episode reward: 30.000000
episode reward: 15.000000
episode reward: 14.000000
episode reward: 20.000000
episode reward: 15.000000
episode reward: 19.000000
episode reward: 44.000000
episode reward: 65.000000
episode reward: 14.000000
episode reward: 23.000000
episode reward: 20.000000
episode reward: 37.000000
episode reward: 25.000000
episode reward: 28.000000
episode reward: 24.000000
episode reward: 16.000000
episode reward: 31.000000
episode reward: 23.000000
episode reward: 11.000000
episode reward: 52.000000
episode reward: 15.000000
episode reward: 11.000000
episode reward: 32.000000
episode reward: 12.000000
episode reward: 13.000000
episode reward: 40.000000
episode reward: 15.000000
episode reward: 49.000000
episode reward: 36.000000
episode reward: 37.000000
episode reward: 68.000000
episode reward: 24.000000
episode reward: 16.000000
episode reward: 26.000000
episode reward: 12.000000
episode reward: 27.000000
episode reward: 21.000000
episode reward: 37.000000
episode reward: 11.000000
episode reward: 21.000000
episode reward: 30.000000
episode reward: 46.000000
episode reward: 22.000000
episode reward: 22.000000
episode reward: 25.000000
episode reward: 20.000000
episode reward: 12.000000
episode reward: 47.000000
episode reward: 57.000000
episode reward: 24.000000
episode reward: 51.000000
episode reward: 30.000000
episode reward: 61.000000
episode reward: 12.000000
episode reward: 35.000000
episode reward: 67.000000
episode reward: 34.000000
episode reward: 16.000000
episode reward: 45.000000
episode reward: 19.000000
episode reward: 9.000000
episode reward: 25.000000
episode reward: 23.000000
episode reward: 13.000000
episode reward: 17.000000
episode reward: 24.000000
episode reward: 85.000000
episode reward: 18.000000
episode reward: 41.000000
episode reward: 55.000000
episode reward: 24.000000
episode reward: 44.000000
episode reward: 12.000000
episode reward: 24.000000
episode reward: 26.000000
episode reward: 14.000000
episode reward: 11.000000
episode reward: 13.000000
episode reward: 20.000000
episode reward: 19.000000
episode reward: 19.000000
episode reward: 14.000000
episode reward: 81.000000
episode reward: 53.000000
episode reward: 29.000000
episode reward: 12.000000
episode reward: 17.000000
episode reward: 28.000000
episode reward: 71.000000
episode reward: 12.000000
episode reward: 45.000000
episode reward: 22.000000
episode reward: 32.000000
episode reward: 18.000000
episode reward: 23.000000
episode reward: 64.000000
episode reward: 17.000000
episode reward: 12.000000
episode reward: 26.000000
episode reward: 10.000000
episode reward: 19.000000
episode reward: 21.000000
episode reward: 19.000000
episode reward: 25.000000
episode reward: 20.000000
episode reward: 14.000000
episode reward: 15.000000
episode reward: 16.000000
episode reward: 14.000000
episode reward: 13.000000
episode reward: 14.000000
episode reward: 39.000000
episode reward: 15.000000
episode reward: 13.000000
episode reward: 34.000000
episode reward: 18.000000
episode reward: 17.000000
episode reward: 13.000000
episode reward: 15.000000
episode reward: 22.000000
episode reward: 12.000000
episode reward: 20.000000
episode reward: 44.000000
episode reward: 15.000000
episode reward: 58.000000
episode reward: 26.000000
episode reward: 77.000000
episode reward: 34.000000
episode reward: 41.000000
episode reward: 16.000000
episode reward: 27.000000
episode reward: 17.000000
episode reward: 22.000000
episode reward: 38.000000
episode reward: 28.000000
episode reward: 35.000000
episode reward: 21.000000
episode reward: 38.000000
episode reward: 11.000000
episode reward: 19.000000
episode reward: 17.000000
episode reward: 14.000000
episode reward: 32.000000
episode reward: 13.000000
episode reward: 72.000000
episode reward: 20.000000
episode reward: 18.000000
episode reward: 36.000000
episode reward: 24.000000
episode reward: 13.000000
episode reward: 9.000000
episode reward: 35.000000
episode reward: 25.000000
episode reward: 28.000000
episode reward: 56.000000
episode reward: 18.000000
episode reward: 27.000000
episode reward: 21.000000
episode reward: 69.000000
episode reward: 18.000000
episode reward: 24.000000
episode reward: 20.000000
episode reward: 12.000000
episode reward: 12.000000
episode reward: 11.000000
episode reward: 22.000000
episode reward: 21.000000
episode reward: 46.000000
episode reward: 22.000000
episode reward: 22.000000
episode reward: 49.000000
episode reward: 36.000000
episode reward: 82.000000
episode reward: 33.000000
episode reward: 14.000000
episode reward: 16.000000
episode reward: 31.000000
episode reward: 39.000000
episode reward: 76.000000
episode reward: 12.000000
episode reward: 46.000000
episode reward: 27.000000
episode reward: 19.000000
episode reward: 22.000000
episode reward: 23.000000
episode reward: 32.000000
episode reward: 11.000000
episode reward: 33.000000
episode reward: 17.000000
episode reward: 19.000000
episode reward: 32.000000
episode reward: 40.000000
episode reward: 41.000000
episode reward: 40.000000
episode reward: 13.000000
episode reward: 30.000000
episode reward: 42.000000
episode reward: 15.000000
episode reward: 81.000000
episode reward: 40.000000
episode reward: 24.000000
episode reward: 31.000000
episode reward: 23.000000
episode reward: 37.000000
episode reward: 22.000000
episode reward: 31.000000
episode reward: 25.000000
episode reward: 13.000000
episode reward: 64.000000
episode reward: 11.000000
episode reward: 41.000000
episode reward: 19.000000
episode reward: 10.000000
episode reward: 11.000000
episode reward: 26.000000
episode reward: 29.000000
episode reward: 15.000000
episode reward: 44.000000
episode reward: 16.000000
episode reward: 10.000000
episode reward: 37.000000
episode reward: 13.000000
episode reward: 39.000000
episode reward: 33.000000
episode reward: 31.000000
episode reward: 23.000000
episode reward: 65.000000
episode reward: 52.000000
episode reward: 11.000000
episode reward: 24.000000
episode reward: 17.000000
episode reward: 26.000000
episode reward: 14.000000
episode reward: 17.000000
episode reward: 40.000000
episode reward: 23.000000
episode reward: 9.000000
episode reward: 15.000000
episode reward: 22.000000
episode reward: 36.000000
episode reward: 20.000000
episode reward: 36.000000
episode reward: 9.000000
episode reward: 9.000000
episode reward: 29.000000
episode reward: 62.000000
episode reward: 15.000000
episode reward: 21.000000
episode reward: 11.000000
episode reward: 17.000000
episode reward: 26.000000
episode reward: 18.000000
episode reward: 15.000000
episode reward: 58.000000
episode reward: 14.000000
episode reward: 14.000000
episode reward: 33.000000
episode reward: 19.000000
episode reward: 22.000000
episode reward: 17.000000
episode reward: 15.000000
episode reward: 30.000000
episode reward: 25.000000
episode reward: 11.000000
episode reward: 42.000000
episode reward: 46.000000
episode reward: 36.000000
episode reward: 19.000000
episode reward: 59.000000
episode reward: 29.000000
episode reward: 18.000000
episode reward: 37.000000
episode reward: 16.000000
episode reward: 57.000000
episode reward: 65.000000
episode reward: 27.000000
episode reward: 22.000000
episode reward: 13.000000
episode reward: 29.000000
episode reward: 20.000000
episode reward: 12.000000
episode reward: 16.000000
episode reward: 12.000000
episode reward: 23.000000
episode reward: 28.000000
episode reward: 30.000000
episode reward: 18.000000
episode reward: 31.000000
episode reward: 29.000000
episode reward: 24.000000
episode reward: 11.000000
episode reward: 29.000000
episode reward: 17.000000
episode reward: 42.000000
episode reward: 19.000000
episode reward: 11.000000
episode reward: 20.000000
episode reward: 39.000000
episode reward: 20.000000
episode reward: 19.000000
episode reward: 39.000000
episode reward: 24.000000
episode reward: 53.000000
episode reward: 28.000000
episode reward: 34.000000
episode reward: 25.000000
episode reward: 13.000000
episode reward: 22.000000
episode reward: 29.000000
episode reward: 17.000000
episode reward: 55.000000
episode reward: 16.000000
episode reward: 59.000000
episode reward: 18.000000
episode reward: 19.000000
episode reward: 21.000000
episode reward: 31.000000
episode reward: 84.000000
episode reward: 17.000000
episode reward: 15.000000
episode reward: 18.000000
episode reward: 21.000000
episode reward: 12.000000
episode reward: 15.000000
episode reward: 36.000000
episode reward: 33.000000
episode reward: 22.000000
episode reward: 18.000000
episode reward: 33.000000
episode reward: 66.000000
episode reward: 44.000000
episode reward: 25.000000
episode reward: 70.000000
episode reward: 59.000000
episode reward: 17.000000
episode reward: 23.000000
episode reward: 24.000000
episode reward: 20.000000
episode reward: 32.000000
episode reward: 21.000000
episode reward: 28.000000
episode reward: 57.000000
episode reward: 19.000000
episode reward: 21.000000
episode reward: 13.000000
episode reward: 35.000000
episode reward: 17.000000
episode reward: 117.000000
episode reward: 21.000000
episode reward: 45.000000
episode reward: 34.000000
episode reward: 100.000000
episode reward: 85.000000
episode reward: 31.000000
episode reward: 49.000000
episode reward: 28.000000
episode reward: 19.000000
episode reward: 41.000000
episode reward: 67.000000
episode reward: 22.000000
episode reward: 33.000000
episode reward: 27.000000
episode reward: 60.000000
episode reward: 23.000000
episode reward: 26.000000
episode reward: 35.000000
episode reward: 14.000000
episode reward: 85.000000
episode reward: 15.000000
episode reward: 41.000000
episode reward: 40.000000
episode reward: 22.000000
episode reward: 16.000000
episode reward: 44.000000
episode reward: 17.000000
episode reward: 59.000000
episode reward: 65.000000
episode reward: 18.000000
episode reward: 26.000000
episode reward: 19.000000
episode reward: 15.000000
episode reward: 13.000000
episode reward: 53.000000
episode reward: 33.000000
episode reward: 66.000000
episode reward: 13.000000
episode reward: 16.000000
episode reward: 17.000000
episode reward: 38.000000
episode reward: 10.000000
episode reward: 25.000000
episode reward: 55.000000
episode reward: 26.000000
episode reward: 17.000000
episode reward: 30.000000
episode reward: 18.000000
episode reward: 28.000000
episode reward: 23.000000
episode reward: 13.000000
episode reward: 30.000000
episode reward: 40.000000
episode reward: 11.000000
episode reward: 56.000000
episode reward: 34.000000
episode reward: 36.000000
episode reward: 51.000000
episode reward: 16.000000
episode reward: 23.000000
episode reward: 22.000000
episode reward: 23.000000
episode reward: 14.000000
episode reward: 33.000000
episode reward: 10.000000
episode reward: 26.000000
episode reward: 18.000000
episode reward: 16.000000
episode reward: 113.000000
episode reward: 81.000000
episode reward: 101.000000
episode reward: 25.000000
episode reward: 27.000000
episode reward: 28.000000
episode reward: 74.000000
episode reward: 34.000000
episode reward: 38.000000
episode reward: 54.000000
episode reward: 14.000000
episode reward: 19.000000
episode reward: 32.000000
episode reward: 34.000000
episode reward: 43.000000
episode reward: 17.000000
episode reward: 35.000000
episode reward: 32.000000
episode reward: 24.000000
episode reward: 12.000000
episode reward: 75.000000
episode reward: 38.000000
episode reward: 46.000000
episode reward: 17.000000
episode reward: 14.000000
episode reward: 14.000000
episode reward: 38.000000
episode reward: 27.000000
episode reward: 15.000000
episode reward: 83.000000
episode reward: 17.000000
episode reward: 37.000000
episode reward: 32.000000
episode reward: 15.000000
episode reward: 20.000000
episode reward: 25.000000
episode reward: 40.000000
episode reward: 37.000000
episode reward: 20.000000
episode reward: 94.000000
episode reward: 19.000000
episode reward: 13.000000
episode reward: 30.000000
episode reward: 21.000000
episode reward: 14.000000
episode reward: 25.000000
episode reward: 31.000000
episode reward: 11.000000
episode reward: 23.000000
episode reward: 44.000000
episode reward: 22.000000
episode reward: 25.000000
episode reward: 15.000000
episode reward: 27.000000
episode reward: 14.000000
episode reward: 44.000000
episode reward: 51.000000
episode reward: 20.000000
episode reward: 23.000000
episode reward: 87.000000
episode reward: 67.000000
episode reward: 28.000000
episode reward: 25.000000
episode reward: 18.000000
episode reward: 20.000000
episode reward: 44.000000
episode reward: 12.000000
episode reward: 16.000000
episode reward: 35.000000
episode reward: 13.000000
episode reward: 35.000000
episode reward: 58.000000
episode reward: 15.000000
episode reward: 17.000000
episode reward: 42.000000
episode reward: 30.000000
episode reward: 48.000000
episode reward: 13.000000
episode reward: 39.000000
episode reward: 87.000000
episode reward: 16.000000
episode reward: 62.000000
episode reward: 25.000000
episode reward: 39.000000
episode reward: 25.000000
episode reward: 25.000000
episode reward: 27.000000
episode reward: 23.000000
episode reward: 41.000000
episode reward: 34.000000
episode reward: 55.000000
episode reward: 21.000000
episode reward: 61.000000
episode reward: 17.000000
episode reward: 33.000000
episode reward: 29.000000
episode reward: 65.000000
episode reward: 54.000000
episode reward: 24.000000
episode reward: 37.000000
episode reward: 33.000000
episode reward: 70.000000
episode reward: 51.000000
episode reward: 32.000000
episode reward: 36.000000
episode reward: 49.000000
episode reward: 54.000000
episode reward: 11.000000
episode reward: 22.000000
episode reward: 29.000000
episode reward: 38.000000
episode reward: 107.000000
episode reward: 55.000000
episode reward: 34.000000
episode reward: 32.000000
episode reward: 29.000000
episode reward: 50.000000
episode reward: 17.000000
episode reward: 47.000000
episode reward: 30.000000
episode reward: 14.000000
episode reward: 26.000000
episode reward: 37.000000
episode reward: 21.000000
episode reward: 29.000000
episode reward: 42.000000
episode reward: 43.000000
episode reward: 46.000000
episode reward: 30.000000
episode reward: 19.000000
episode reward: 34.000000
episode reward: 19.000000
episode reward: 63.000000
episode reward: 28.000000
episode reward: 59.000000
episode reward: 77.000000
episode reward: 46.000000
episode reward: 13.000000
episode reward: 15.000000
episode reward: 100.000000
episode reward: 24.000000
episode reward: 34.000000
episode reward: 24.000000
episode reward: 16.000000
episode reward: 77.000000
episode reward: 36.000000
episode reward: 19.000000
episode reward: 21.000000
episode reward: 12.000000
episode reward: 22.000000
episode reward: 116.000000
episode reward: 14.000000
episode reward: 18.000000
episode reward: 51.000000
episode reward: 62.000000
episode reward: 25.000000
episode reward: 35.000000
episode reward: 16.000000
episode reward: 61.000000
episode reward: 43.000000
episode reward: 51.000000
episode reward: 10.000000
episode reward: 22.000000
episode reward: 18.000000
episode reward: 48.000000
episode reward: 22.000000
episode reward: 48.000000
episode reward: 62.000000
episode reward: 39.000000
episode reward: 62.000000
episode reward: 16.000000
episode reward: 70.000000
episode reward: 52.000000
episode reward: 12.000000
episode reward: 48.000000
episode reward: 43.000000
episode reward: 25.000000
episode reward: 53.000000
episode reward: 53.000000
episode reward: 50.000000
episode reward: 60.000000
episode reward: 30.000000
episode reward: 55.000000
episode reward: 43.000000
episode reward: 46.000000
episode reward: 15.000000
episode reward: 21.000000
episode reward: 15.000000
episode reward: 19.000000
episode reward: 60.000000
episode reward: 9.000000
episode reward: 70.000000
episode reward: 140.000000
episode reward: 78.000000
episode reward: 35.000000
episode reward: 31.000000
episode reward: 28.000000
episode reward: 53.000000
episode reward: 44.000000
episode reward: 15.000000
episode reward: 30.000000
episode reward: 39.000000
episode reward: 24.000000
episode reward: 44.000000
episode reward: 24.000000
episode reward: 24.000000
episode reward: 37.000000
episode reward: 11.000000
episode reward: 29.000000
episode reward: 27.000000
episode reward: 123.000000
episode reward: 23.000000
episode reward: 33.000000
episode reward: 31.000000
episode reward: 20.000000
episode reward: 40.000000
episode reward: 58.000000
episode reward: 44.000000
episode reward: 46.000000
episode reward: 20.000000
episode reward: 23.000000
episode reward: 40.000000
episode reward: 83.000000
episode reward: 47.000000
episode reward: 31.000000
episode reward: 34.000000
episode reward: 27.000000
episode reward: 42.000000
episode reward: 37.000000
episode reward: 51.000000
episode reward: 18.000000
episode reward: 64.000000
episode reward: 35.000000
episode reward: 28.000000
episode reward: 73.000000
episode reward: 30.000000
episode reward: 62.000000
episode reward: 42.000000
episode reward: 74.000000
episode reward: 28.000000
episode reward: 55.000000
episode reward: 26.000000
episode reward: 31.000000
episode reward: 12.000000
episode reward: 72.000000
episode reward: 27.000000
episode reward: 33.000000
episode reward: 18.000000
episode reward: 75.000000
episode reward: 41.000000
episode reward: 38.000000
episode reward: 23.000000
episode reward: 84.000000
episode reward: 56.000000
episode reward: 90.000000
episode reward: 26.000000
episode reward: 54.000000
episode reward: 23.000000
episode reward: 49.000000
episode reward: 12.000000
episode reward: 37.000000
episode reward: 54.000000
episode reward: 57.000000
episode reward: 113.000000
episode reward: 50.000000
episode reward: 20.000000
episode reward: 86.000000
episode reward: 38.000000
episode reward: 61.000000
episode reward: 57.000000
episode reward: 91.000000
episode reward: 46.000000
episode reward: 27.000000
episode reward: 44.000000
episode reward: 64.000000
episode reward: 103.000000
episode reward: 28.000000
episode reward: 13.000000
episode reward: 38.000000
episode reward: 36.000000
episode reward: 84.000000
episode reward: 23.000000
episode reward: 82.000000
episode reward: 48.000000
episode reward: 48.000000
episode reward: 15.000000
episode reward: 111.000000
episode reward: 15.000000
episode reward: 26.000000
episode reward: 48.000000
episode reward: 47.000000
episode reward: 21.000000
episode reward: 107.000000
episode reward: 51.000000
episode reward: 34.000000
episode reward: 31.000000
episode reward: 9.000000
episode reward: 100.000000
episode reward: 74.000000
episode reward: 62.000000
episode reward: 11.000000
episode reward: 36.000000
episode reward: 45.000000
episode reward: 32.000000
episode reward: 12.000000
episode reward: 59.000000
episode reward: 51.000000
episode reward: 26.000000
episode reward: 23.000000
episode reward: 73.000000
episode reward: 58.000000
episode reward: 29.000000
episode reward: 104.000000
episode reward: 15.000000
episode reward: 15.000000
episode reward: 32.000000
episode reward: 20.000000
episode reward: 82.000000
episode reward: 51.000000
episode reward: 28.000000
episode reward: 13.000000
episode reward: 28.000000
episode reward: 57.000000
episode reward: 28.000000
episode reward: 37.000000
episode reward: 57.000000
episode reward: 51.000000
episode reward: 26.000000
episode reward: 55.000000
episode reward: 39.000000
episode reward: 13.000000
episode reward: 99.000000
episode reward: 19.000000
episode reward: 41.000000
episode reward: 33.000000
episode reward: 36.000000
episode reward: 37.000000
episode reward: 40.000000
episode reward: 26.000000
episode reward: 40.000000
episode reward: 28.000000
episode reward: 17.000000
episode reward: 52.000000
episode reward: 39.000000
episode reward: 18.000000
episode reward: 66.000000
episode reward: 38.000000
episode reward: 81.000000
episode reward: 78.000000
episode reward: 59.000000
episode reward: 69.000000
episode reward: 23.000000
episode reward: 54.000000
episode reward: 22.000000
episode reward: 139.000000
episode reward: 28.000000
episode reward: 44.000000
episode reward: 86.000000
episode reward: 30.000000
episode reward: 34.000000
episode reward: 31.000000
episode reward: 48.000000
episode reward: 89.000000
episode reward: 42.000000
episode reward: 12.000000
episode reward: 46.000000
episode reward: 32.000000
episode reward: 63.000000
episode reward: 24.000000
episode reward: 23.000000
episode reward: 19.000000
episode reward: 29.000000
episode reward: 103.000000
episode reward: 36.000000
episode reward: 48.000000
episode reward: 36.000000
episode reward: 56.000000
episode reward: 47.000000
episode reward: 56.000000
episode reward: 115.000000
episode reward: 49.000000
episode reward: 62.000000
episode reward: 71.000000
episode reward: 38.000000
episode reward: 16.000000
episode reward: 18.000000
episode reward: 53.000000
episode reward: 16.000000
episode reward: 20.000000
episode reward: 13.000000
episode reward: 73.000000
episode reward: 30.000000
episode reward: 25.000000
episode reward: 22.000000
episode reward: 25.000000
episode reward: 33.000000
episode reward: 40.000000
episode reward: 71.000000
episode reward: 63.000000
episode reward: 53.000000
episode reward: 34.000000
episode reward: 60.000000
episode reward: 57.000000
episode reward: 50.000000
episode reward: 69.000000
episode reward: 11.000000
episode reward: 19.000000
episode reward: 28.000000
episode reward: 14.000000
episode reward: 111.000000
episode reward: 55.000000
episode reward: 88.000000
episode reward: 17.000000
episode reward: 23.000000
episode reward: 88.000000
episode reward: 101.000000
episode reward: 44.000000
episode reward: 137.000000
episode reward: 82.000000
episode reward: 25.000000
episode reward: 101.000000
episode reward: 48.000000
episode reward: 24.000000
episode reward: 14.000000
episode reward: 33.000000
episode reward: 120.000000
episode reward: 16.000000
episode reward: 36.000000
episode reward: 42.000000
episode reward: 68.000000
episode reward: 58.000000
episode reward: 85.000000
episode reward: 51.000000
episode reward: 89.000000
episode reward: 41.000000
episode reward: 18.000000
episode reward: 37.000000
episode reward: 22.000000
episode reward: 32.000000
episode reward: 61.000000
episode reward: 56.000000
episode reward: 45.000000
episode reward: 54.000000
episode reward: 40.000000
episode reward: 111.000000
episode reward: 153.000000
episode reward: 20.000000
episode reward: 40.000000
episode reward: 41.000000
episode reward: 83.000000
episode reward: 51.000000
episode reward: 85.000000
episode reward: 37.000000
episode reward: 12.000000
episode reward: 32.000000
episode reward: 138.000000
episode reward: 51.000000
episode reward: 29.000000
episode reward: 18.000000
episode reward: 85.000000
episode reward: 74.000000
episode reward: 109.000000
episode reward: 95.000000
episode reward: 33.000000
episode reward: 113.000000
episode reward: 33.000000
episode reward: 44.000000
episode reward: 17.000000
episode reward: 78.000000
episode reward: 162.000000
episode reward: 173.000000
episode reward: 14.000000
episode reward: 148.000000
episode reward: 11.000000
episode reward: 54.000000
episode reward: 108.000000
episode reward: 129.000000
episode reward: 52.000000
episode reward: 18.000000
episode reward: 89.000000
episode reward: 21.000000
episode reward: 88.000000
episode reward: 78.000000
episode reward: 93.000000
episode reward: 67.000000
episode reward: 158.000000
episode reward: 89.000000
episode reward: 53.000000
episode reward: 15.000000
episode reward: 13.000000
episode reward: 93.000000
episode reward: 122.000000
episode reward: 33.000000
episode reward: 77.000000
episode reward: 29.000000
episode reward: 13.000000
episode reward: 102.000000
episode reward: 213.000000
episode reward: 31.000000
episode reward: 128.000000
episode reward: 60.000000
episode reward: 27.000000
episode reward: 15.000000
episode reward: 26.000000
episode reward: 49.000000
episode reward: 52.000000
episode reward: 33.000000
episode reward: 13.000000
episode reward: 60.000000
episode reward: 59.000000
episode reward: 125.000000
episode reward: 20.000000
episode reward: 103.000000
episode reward: 95.000000
episode reward: 65.000000
episode reward: 114.000000
episode reward: 17.000000
episode reward: 39.000000
episode reward: 198.000000
episode reward: 16.000000
episode reward: 163.000000
episode reward: 167.000000
episode reward: 85.000000
episode reward: 22.000000
episode reward: 39.000000
episode reward: 83.000000
episode reward: 49.000000
episode reward: 138.000000
episode reward: 111.000000
episode reward: 131.000000
episode reward: 89.000000
episode reward: 29.000000
episode reward: 18.000000
episode reward: 69.000000
episode reward: 102.000000
episode reward: 107.000000
episode reward: 31.000000
episode reward: 47.000000
episode reward: 102.000000
episode reward: 101.000000
episode reward: 42.000000
episode reward: 155.000000
episode reward: 19.000000
episode reward: 131.000000
episode reward: 44.000000
episode reward: 18.000000
episode reward: 66.000000
episode reward: 16.000000
episode reward: 107.000000
episode reward: 120.000000
episode reward: 139.000000
episode reward: 116.000000
episode reward: 22.000000
episode reward: 138.000000
episode reward: 90.000000
episode reward: 66.000000
episode reward: 47.000000
episode reward: 42.000000
episode reward: 138.000000
episode reward: 72.000000
episode reward: 48.000000
episode reward: 69.000000
episode reward: 32.000000
episode reward: 27.000000
episode reward: 47.000000
episode reward: 45.000000
episode reward: 76.000000
episode reward: 125.000000
episode reward: 30.000000
episode reward: 91.000000
episode reward: 126.000000
episode reward: 179.000000
episode reward: 117.000000
episode reward: 50.000000
episode reward: 178.000000
episode reward: 71.000000
episode reward: 145.000000
episode reward: 52.000000
episode reward: 113.000000
episode reward: 83.000000
episode reward: 128.000000
episode reward: 116.000000
episode reward: 15.000000
episode reward: 61.000000
episode reward: 76.000000
episode reward: 27.000000
episode reward: 74.000000
episode reward: 33.000000
episode reward: 63.000000
episode reward: 13.000000
episode reward: 258.000000
episode reward: 123.000000
episode reward: 25.000000
episode reward: 118.000000
episode reward: 160.000000
episode reward: 22.000000
episode reward: 18.000000
episode reward: 76.000000
episode reward: 80.000000
episode reward: 80.000000
episode reward: 158.000000
episode reward: 72.000000
episode reward: 112.000000
episode reward: 103.000000
episode reward: 125.000000
episode reward: 57.000000
episode reward: 131.000000
episode reward: 125.000000
episode reward: 146.000000
episode reward: 136.000000
episode reward: 93.000000
episode reward: 26.000000
episode reward: 68.000000
episode reward: 142.000000
episode reward: 121.000000
episode reward: 101.000000
episode reward: 27.000000
episode reward: 75.000000
episode reward: 135.000000
episode reward: 47.000000
episode reward: 61.000000
episode reward: 41.000000
episode reward: 131.000000
episode reward: 104.000000
episode reward: 152.000000
episode reward: 140.000000
episode reward: 117.000000
episode reward: 138.000000
episode reward: 80.000000
episode reward: 75.000000
episode reward: 61.000000
episode reward: 15.000000
episode reward: 214.000000
episode reward: 170.000000
episode reward: 224.000000
episode reward: 81.000000
episode reward: 67.000000
episode reward: 155.000000
episode reward: 87.000000
episode reward: 19.000000
episode reward: 12.000000
episode reward: 42.000000
episode reward: 158.000000
episode reward: 280.000000
episode reward: 189.000000
episode reward: 177.000000
episode reward: 75.000000
episode reward: 59.000000
episode reward: 26.000000
episode reward: 248.000000
episode reward: 163.000000
episode reward: 214.000000
episode reward: 61.000000
episode reward: 172.000000
episode reward: 13.000000
episode reward: 188.000000
episode reward: 119.000000
episode reward: 197.000000
episode reward: 164.000000
episode reward: 187.000000
episode reward: 276.000000
episode reward: 126.000000
episode reward: 157.000000
episode reward: 156.000000
episode reward: 129.000000
episode reward: 71.000000
episode reward: 144.000000
episode reward: 55.000000
episode reward: 114.000000
episode reward: 55.000000
episode reward: 111.000000
episode reward: 192.000000
episode reward: 77.000000
episode reward: 16.000000
episode reward: 163.000000
episode reward: 87.000000
episode reward: 45.000000
episode reward: 169.000000
episode reward: 146.000000
episode reward: 28.000000
episode reward: 192.000000
episode reward: 166.000000
episode reward: 23.000000
episode reward: 159.000000
episode reward: 129.000000
episode reward: 141.000000
episode reward: 272.000000
episode reward: 164.000000
episode reward: 229.000000
episode reward: 138.000000
episode reward: 232.000000
episode reward: 122.000000
episode reward: 126.000000
episode reward: 307.000000
episode reward: 124.000000
episode reward: 36.000000
episode reward: 204.000000
episode reward: 56.000000
episode reward: 113.000000
episode reward: 131.000000
episode reward: 116.000000
episode reward: 101.000000
episode reward: 187.000000
episode reward: 56.000000
episode reward: 295.000000
episode reward: 327.000000
episode reward: 57.000000
episode reward: 198.000000
episode reward: 55.000000
episode reward: 90.000000
episode reward: 102.000000
episode reward: 31.000000
episode reward: 102.000000
episode reward: 107.000000
episode reward: 130.000000
episode reward: 367.000000
episode reward: 60.000000
episode reward: 47.000000
episode reward: 212.000000
episode reward: 261.000000
episode reward: 293.000000
episode reward: 63.000000
episode reward: 107.000000
episode reward: 85.000000
episode reward: 226.000000
episode reward: 190.000000
episode reward: 176.000000
episode reward: 220.000000
episode reward: 186.000000
episode reward: 90.000000
episode reward: 206.000000
episode reward: 147.000000
episode reward: 182.000000
episode reward: 177.000000
episode reward: 181.000000
episode reward: 156.000000
episode reward: 61.000000
episode reward: 137.000000
episode reward: 159.000000
episode reward: 114.000000
episode reward: 190.000000
episode reward: 89.000000
episode reward: 21.000000
episode reward: 46.000000
episode reward: 138.000000
episode reward: 166.000000
episode reward: 122.000000
episode reward: 161.000000
episode reward: 108.000000
episode reward: 145.000000
episode reward: 125.000000
episode reward: 38.000000
episode reward: 125.000000
episode reward: 130.000000
episode reward: 128.000000
episode reward: 165.000000
episode reward: 29.000000
episode reward: 18.000000
episode reward: 56.000000
episode reward: 129.000000
episode reward: 134.000000
episode reward: 126.000000
episode reward: 137.000000
episode reward: 135.000000
episode reward: 144.000000
episode reward: 219.000000
episode reward: 70.000000
episode reward: 115.000000
episode reward: 130.000000
episode reward: 213.000000
episode reward: 148.000000
episode reward: 115.000000
episode reward: 129.000000
episode reward: 148.000000
episode reward: 137.000000
episode reward: 187.000000
episode reward: 128.000000
episode reward: 140.000000
episode reward: 139.000000
episode reward: 164.000000
episode reward: 150.000000
episode reward: 153.000000
episode reward: 119.000000
episode reward: 32.000000
episode reward: 159.000000
episode reward: 163.000000
episode reward: 167.000000
episode reward: 85.000000
episode reward: 49.000000
episode reward: 149.000000
episode reward: 180.000000
episode reward: 151.000000
episode reward: 147.000000
episode reward: 141.000000
episode reward: 198.000000
episode reward: 134.000000
episode reward: 265.000000
episode reward: 147.000000
episode reward: 176.000000
episode reward: 126.000000
episode reward: 165.000000
episode reward: 163.000000
episode reward: 236.000000
episode reward: 175.000000
episode reward: 139.000000
episode reward: 119.000000
episode reward: 225.000000
episode reward: 222.000000
episode reward: 203.000000
episode reward: 155.000000
episode reward: 134.000000
episode reward: 93.000000
episode reward: 181.000000
episode reward: 147.000000
episode reward: 139.000000
episode reward: 158.000000
episode reward: 131.000000
episode reward: 137.000000
episode reward: 167.000000
episode reward: 140.000000
episode reward: 122.000000
episode reward: 130.000000
episode reward: 142.000000
episode reward: 161.000000
episode reward: 122.000000
episode reward: 180.000000
episode reward: 101.000000
episode reward: 105.000000
episode reward: 146.000000
episode reward: 129.000000
episode reward: 144.000000
episode reward: 107.000000
episode reward: 126.000000
episode reward: 107.000000
episode reward: 138.000000
episode reward: 121.000000
episode reward: 43.000000
episode reward: 125.000000
episode reward: 108.000000
episode reward: 33.000000
episode reward: 114.000000
episode reward: 126.000000
episode reward: 119.000000
episode reward: 103.000000
episode reward: 111.000000
episode reward: 142.000000
episode reward: 105.000000
episode reward: 131.000000
episode reward: 111.000000
episode reward: 20.000000
episode reward: 116.000000
episode reward: 118.000000
episode reward: 101.000000
episode reward: 103.000000
episode reward: 97.000000
episode reward: 15.000000
episode reward: 94.000000
episode reward: 63.000000
episode reward: 107.000000
episode reward: 51.000000
episode reward: 19.000000
episode reward: 106.000000
episode reward: 51.000000
episode reward: 21.000000
episode reward: 57.000000
episode reward: 39.000000
episode reward: 108.000000
episode reward: 132.000000
episode reward: 25.000000
episode reward: 26.000000
episode reward: 76.000000
episode reward: 59.000000
episode reward: 46.000000
episode reward: 47.000000
episode reward: 39.000000
episode reward: 32.000000
episode reward: 100.000000
episode reward: 62.000000
episode reward: 109.000000
episode reward: 117.000000
episode reward: 61.000000
episode reward: 122.000000
episode reward: 111.000000
episode reward: 106.000000
episode reward: 105.000000
episode reward: 129.000000
episode reward: 26.000000
episode reward: 78.000000
episode reward: 127.000000
episode reward: 105.000000
episode reward: 117.000000
episode reward: 116.000000
episode reward: 103.000000
episode reward: 151.000000
episode reward: 107.000000
episode reward: 139.000000
episode reward: 132.000000
episode reward: 122.000000
episode reward: 113.000000
episode reward: 128.000000
episode reward: 85.000000
episode reward: 121.000000
episode reward: 172.000000
episode reward: 91.000000
episode reward: 123.000000
episode reward: 116.000000
episode reward: 127.000000
episode reward: 143.000000
episode reward: 116.000000
episode reward: 164.000000
episode reward: 116.000000
episode reward: 131.000000
episode reward: 125.000000
episode reward: 115.000000
episode reward: 169.000000
episode reward: 123.000000
episode reward: 128.000000
episode reward: 178.000000
episode reward: 153.000000
episode reward: 78.000000
episode reward: 134.000000
episode reward: 165.000000
episode reward: 124.000000
episode reward: 79.000000
episode reward: 132.000000
episode reward: 128.000000
episode reward: 153.000000
episode reward: 108.000000
episode reward: 126.000000
episode reward: 159.000000
episode reward: 117.000000
episode reward: 131.000000
episode reward: 19.000000
episode reward: 119.000000
episode reward: 145.000000
episode reward: 135.000000
episode reward: 132.000000
episode reward: 112.000000
episode reward: 141.000000
episode reward: 27.000000
episode reward: 124.000000
episode reward: 125.000000
episode reward: 144.000000
episode reward: 109.000000
episode reward: 127.000000
episode reward: 29.000000
episode reward: 131.000000
episode reward: 34.000000
episode reward: 136.000000
episode reward: 127.000000
episode reward: 141.000000
episode reward: 119.000000
episode reward: 139.000000
episode reward: 117.000000
episode reward: 121.000000
episode reward: 52.000000
episode reward: 137.000000
episode reward: 140.000000
episode reward: 153.000000
episode reward: 141.000000
episode reward: 162.000000
episode reward: 141.000000
episode reward: 134.000000
episode reward: 61.000000
episode reward: 127.000000
episode reward: 23.000000
episode reward: 176.000000
episode reward: 174.000000
episode reward: 182.000000
episode reward: 166.000000
episode reward: 133.000000
episode reward: 174.000000
episode reward: 184.000000
episode reward: 192.000000
episode reward: 165.000000
episode reward: 202.000000
episode reward: 68.000000
episode reward: 99.000000
episode reward: 229.000000
episode reward: 236.000000
episode reward: 244.000000
episode reward: 239.000000
episode reward: 195.000000
episode reward: 224.000000
episode reward: 316.000000
episode reward: 58.000000
episode reward: 274.000000
episode reward: 314.000000
episode reward: 228.000000
episode reward: 300.000000
episode reward: 247.000000
episode reward: 395.000000
episode reward: 101.000000
episode reward: 220.000000
episode reward: 355.000000
episode reward: 422.000000
episode reward: 282.000000
episode reward: 241.000000
episode reward: 257.000000
episode reward: 198.000000
episode reward: 397.000000
episode reward: 220.000000
episode reward: 188.000000
episode reward: 252.000000
episode reward: 96.000000
episode reward: 303.000000
episode reward: 278.000000
episode reward: 227.000000
episode reward: 98.000000
episode reward: 187.000000
episode reward: 249.000000
episode reward: 199.000000
episode reward: 208.000000
episode reward: 218.000000
episode reward: 209.000000
episode reward: 111.000000
episode reward: 203.000000
episode reward: 184.000000
episode reward: 81.000000
episode reward: 199.000000
episode reward: 253.000000
episode reward: 218.000000
episode reward: 230.000000
episode reward: 147.000000
episode reward: 193.000000
episode reward: 184.000000
episode reward: 191.000000
episode reward: 206.000000
episode reward: 71.000000
episode reward: 175.000000
episode reward: 190.000000
episode reward: 216.000000
episode reward: 139.000000
episode reward: 188.000000
episode reward: 157.000000
episode reward: 183.000000
episode reward: 66.000000
episode reward: 153.000000
episode reward: 159.000000
episode reward: 143.000000
episode reward: 195.000000
episode reward: 137.000000
episode reward: 156.000000
episode reward: 104.000000
episode reward: 130.000000
episode reward: 132.000000
episode reward: 154.000000
episode reward: 150.000000
episode reward: 170.000000
episode reward: 47.000000
episode reward: 152.000000
episode reward: 66.000000
episode reward: 140.000000
episode reward: 139.000000
episode reward: 192.000000
episode reward: 166.000000
episode reward: 142.000000
episode reward: 106.000000
episode reward: 142.000000
episode reward: 132.000000
episode reward: 153.000000
episode reward: 89.000000
episode reward: 190.000000
episode reward: 145.000000
episode reward: 25.000000
episode reward: 138.000000
episode reward: 29.000000
episode reward: 140.000000
episode reward: 135.000000
episode reward: 149.000000
episode reward: 200.000000
episode reward: 169.000000
episode reward: 164.000000
episode reward: 144.000000
episode reward: 145.000000
episode reward: 169.000000
episode reward: 213.000000
episode reward: 172.000000
episode reward: 180.000000
episode reward: 187.000000
episode reward: 196.000000
episode reward: 214.000000
episode reward: 177.000000
episode reward: 176.000000
episode reward: 180.000000
episode reward: 186.000000
episode reward: 207.000000
episode reward: 221.000000
episode reward: 253.000000
episode reward: 153.000000
episode reward: 196.000000
episode reward: 48.000000
episode reward: 218.000000
episode reward: 194.000000
episode reward: 269.000000
episode reward: 269.000000
episode reward: 276.000000
episode reward: 194.000000
episode reward: 271.000000
episode reward: 128.000000
episode reward: 126.000000
episode reward: 239.000000
episode reward: 273.000000
episode reward: 274.000000
episode reward: 236.000000
episode reward: 236.000000
episode reward: 303.000000
episode reward: 32.000000
episode reward: 241.000000
episode reward: 260.000000
episode reward: 251.000000
episode reward: 338.000000
episode reward: 292.000000
episode reward: 244.000000
episode reward: 392.000000
episode reward: 211.000000
episode reward: 329.000000
episode reward: 282.000000
episode reward: 333.000000
episode reward: 11.000000
episode reward: 302.000000
episode reward: 344.000000
episode reward: 161.000000
episode reward: 358.000000
episode reward: 464.000000
episode reward: 343.000000
episode reward: 525.000000
episode reward: 351.000000
episode reward: 602.000000
episode reward: 230.000000
episode reward: 71.000000
episode reward: 450.000000
episode reward: 193.000000
episode reward: 90.000000
episode reward: 787.000000
episode reward: 668.000000
episode reward: 1249.000000
episode reward: 32.000000
episode reward: 360.000000
episode reward: 640.000000
episode reward: 756.000000
episode reward: 1204.000000
episode reward: 396.000000
episode reward: 124.000000
episode reward: 1434.000000
episode reward: 521.000000
episode reward: 185.000000
episode reward: 532.000000
ran 2402 episodes, avg reward: 62.447960
In [28]:
# plot the episode rewards (balancing steps)
%matplotlib inline
import matplotlib.pyplot as plt
plt.xlabel('episode')
plt.ylabel('balancing steps')
plt.bar(np.arange(len(rew)),rew)
Out[28]:
<Container object of 2402 artists>
In [ ]:
Content source: vub-ai-lab/MAL-seminar
Similar notebooks: