Removing previous artifacts...
Experiment started at 10:02:21
.......................................
WARNING:tensorflow:RunConfig.uid (from tensorflow.contrib.learn.python.learn.estimators.run_config) is experimental and may change or be removed at any time, and without warning.
INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x12228d940>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_tf_config': gpu_options {
per_process_gpu_memory_fraction: 1
}
, '_tf_random_seed': 19830610, '_save_summary_steps': 100, '_save_checkpoints_secs': None, '_log_step_count_steps': 100, '_session_config': None, '_save_checkpoints_steps': 2400, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'trained_models/reg-model-03'}
Estimator Type: <class 'tensorflow.python.estimator.canned.dnn.DNNRegressor'>
WARNING:tensorflow:RunConfig.uid (from tensorflow.contrib.learn.python.learn.estimators.run_config) is experimental and may change or be removed at any time, and without warning.
WARNING:tensorflow:From /Users/khalidsalama/anaconda/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/monitors.py:267: BaseMonitor.__init__ (from tensorflow.contrib.learn.python.learn.monitors) is deprecated and will be removed after 2016-12-05.
Instructions for updating:
Monitors are deprecated. Please use tf.train.SessionRunHook.
* data input_fn:
================
Input file(s): data/train-*.csv
Batch size: 500
Epoch Count: 1000
Mode: train
Shuffle: True
================
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Saving checkpoints for 1 into trained_models/reg-model-03/model.ckpt.
* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================
INFO:tensorflow:Starting evaluation at 2017-11-16-10:02:26
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-1
INFO:tensorflow:Finished evaluation at 2017-11-16-10:02:26
INFO:tensorflow:Saving dict for global step 1: average_loss = 328.465, global_step = 1, loss = 164232.0
INFO:tensorflow:Validation (step 1): average_loss = 328.465, loss = 164232.0, global_step = 1
INFO:tensorflow:loss = 155660.0, step = 1
INFO:tensorflow:global_step/sec: 40.1198
INFO:tensorflow:loss = 185020.0, step = 101 (0.776 sec)
INFO:tensorflow:global_step/sec: 129.882
INFO:tensorflow:loss = 137635.0, step = 201 (0.771 sec)
INFO:tensorflow:global_step/sec: 125.641
INFO:tensorflow:loss = 171515.0, step = 301 (0.797 sec)
INFO:tensorflow:global_step/sec: 118.015
INFO:tensorflow:loss = 140163.0, step = 401 (0.844 sec)
INFO:tensorflow:global_step/sec: 126.49
INFO:tensorflow:loss = 141613.0, step = 501 (0.795 sec)
INFO:tensorflow:global_step/sec: 116.976
INFO:tensorflow:loss = 136004.0, step = 601 (0.855 sec)
INFO:tensorflow:global_step/sec: 113.562
INFO:tensorflow:loss = 130641.0, step = 701 (0.879 sec)
INFO:tensorflow:global_step/sec: 126.747
INFO:tensorflow:loss = 111425.0, step = 801 (0.788 sec)
INFO:tensorflow:global_step/sec: 131.066
INFO:tensorflow:loss = 113015.0, step = 901 (0.763 sec)
INFO:tensorflow:global_step/sec: 108.453
INFO:tensorflow:loss = 93464.3, step = 1001 (0.922 sec)
INFO:tensorflow:global_step/sec: 139.519
INFO:tensorflow:loss = 87641.5, step = 1101 (0.717 sec)
INFO:tensorflow:global_step/sec: 134.695
INFO:tensorflow:loss = 88079.6, step = 1201 (0.746 sec)
INFO:tensorflow:global_step/sec: 121.82
INFO:tensorflow:loss = 85597.5, step = 1301 (0.817 sec)
INFO:tensorflow:global_step/sec: 121.88
INFO:tensorflow:loss = 63857.3, step = 1401 (0.826 sec)
INFO:tensorflow:global_step/sec: 119.062
INFO:tensorflow:loss = 70258.2, step = 1501 (0.834 sec)
INFO:tensorflow:global_step/sec: 97.4661
INFO:tensorflow:loss = 55564.2, step = 1601 (1.027 sec)
INFO:tensorflow:global_step/sec: 118.683
INFO:tensorflow:loss = 63768.5, step = 1701 (0.842 sec)
INFO:tensorflow:global_step/sec: 104.212
INFO:tensorflow:loss = 54877.1, step = 1801 (0.962 sec)
INFO:tensorflow:global_step/sec: 114.517
INFO:tensorflow:loss = 60213.6, step = 1901 (0.874 sec)
INFO:tensorflow:global_step/sec: 107.671
INFO:tensorflow:loss = 54891.5, step = 2001 (0.925 sec)
INFO:tensorflow:global_step/sec: 136.533
INFO:tensorflow:loss = 47515.1, step = 2101 (0.733 sec)
INFO:tensorflow:global_step/sec: 118.671
INFO:tensorflow:loss = 51576.9, step = 2201 (0.841 sec)
INFO:tensorflow:global_step/sec: 129.906
INFO:tensorflow:loss = 50460.4, step = 2301 (0.770 sec)
INFO:tensorflow:Saving checkpoints for 2401 into trained_models/reg-model-03/model.ckpt.
INFO:tensorflow:global_step/sec: 75.2384
* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================
INFO:tensorflow:Starting evaluation at 2017-11-16-10:02:48
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-2401
INFO:tensorflow:Finished evaluation at 2017-11-16-10:02:48
INFO:tensorflow:Saving dict for global step 2401: average_loss = 104.677, global_step = 2401, loss = 52338.7
INFO:tensorflow:Validation (step 2401): average_loss = 104.677, loss = 52338.7, global_step = 2401
INFO:tensorflow:loss = 54241.1, step = 2401 (2.193 sec)
INFO:tensorflow:global_step/sec: 63.4717
INFO:tensorflow:loss = 42089.1, step = 2501 (0.712 sec)
INFO:tensorflow:global_step/sec: 142.283
INFO:tensorflow:loss = 43037.2, step = 2601 (0.702 sec)
INFO:tensorflow:global_step/sec: 137.143
INFO:tensorflow:loss = 46601.3, step = 2701 (0.731 sec)
INFO:tensorflow:global_step/sec: 139.558
INFO:tensorflow:loss = 42919.7, step = 2801 (0.715 sec)
INFO:tensorflow:global_step/sec: 141.816
INFO:tensorflow:loss = 44527.5, step = 2901 (0.705 sec)
INFO:tensorflow:global_step/sec: 140.665
INFO:tensorflow:loss = 47868.0, step = 3001 (0.711 sec)
INFO:tensorflow:global_step/sec: 141.832
INFO:tensorflow:loss = 47298.0, step = 3101 (0.706 sec)
INFO:tensorflow:global_step/sec: 121.252
INFO:tensorflow:loss = 49777.9, step = 3201 (0.825 sec)
INFO:tensorflow:global_step/sec: 90.0606
INFO:tensorflow:loss = 44413.0, step = 3301 (1.122 sec)
INFO:tensorflow:global_step/sec: 96.3119
INFO:tensorflow:loss = 47032.9, step = 3401 (1.026 sec)
INFO:tensorflow:global_step/sec: 120.489
INFO:tensorflow:loss = 36708.7, step = 3501 (0.832 sec)
INFO:tensorflow:global_step/sec: 130.31
INFO:tensorflow:loss = 48602.3, step = 3601 (0.765 sec)
INFO:tensorflow:global_step/sec: 121.533
INFO:tensorflow:loss = 46881.4, step = 3701 (0.822 sec)
INFO:tensorflow:global_step/sec: 134.723
INFO:tensorflow:loss = 44374.4, step = 3801 (0.745 sec)
INFO:tensorflow:global_step/sec: 118.789
INFO:tensorflow:loss = 45259.9, step = 3901 (0.840 sec)
INFO:tensorflow:global_step/sec: 133.68
INFO:tensorflow:loss = 50385.1, step = 4001 (0.752 sec)
INFO:tensorflow:global_step/sec: 123.912
INFO:tensorflow:loss = 46569.6, step = 4101 (0.804 sec)
INFO:tensorflow:global_step/sec: 136.72
INFO:tensorflow:loss = 47248.7, step = 4201 (0.734 sec)
INFO:tensorflow:global_step/sec: 95.0957
INFO:tensorflow:loss = 40043.4, step = 4301 (1.051 sec)
INFO:tensorflow:global_step/sec: 96.5031
INFO:tensorflow:loss = 45197.2, step = 4401 (1.036 sec)
INFO:tensorflow:global_step/sec: 111.985
INFO:tensorflow:loss = 47951.4, step = 4501 (0.892 sec)
INFO:tensorflow:global_step/sec: 136.538
INFO:tensorflow:loss = 44457.1, step = 4601 (0.733 sec)
INFO:tensorflow:global_step/sec: 116.219
INFO:tensorflow:loss = 37337.1, step = 4701 (0.862 sec)
INFO:tensorflow:Saving checkpoints for 4801 into trained_models/reg-model-03/model.ckpt.
INFO:tensorflow:global_step/sec: 49.1258
* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================
INFO:tensorflow:Starting evaluation at 2017-11-16-10:03:10
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-4801
INFO:tensorflow:Finished evaluation at 2017-11-16-10:03:10
INFO:tensorflow:Saving dict for global step 4801: average_loss = 96.7373, global_step = 4801, loss = 48368.7
INFO:tensorflow:Validation (step 4801): average_loss = 96.7373, loss = 48368.7, global_step = 4801
INFO:tensorflow:loss = 46016.4, step = 4801 (3.300 sec)
INFO:tensorflow:global_step/sec: 46.0907
INFO:tensorflow:loss = 46275.1, step = 4901 (0.905 sec)
INFO:tensorflow:global_step/sec: 94.0795
INFO:tensorflow:loss = 33856.8, step = 5001 (1.060 sec)
INFO:tensorflow:global_step/sec: 109.878
INFO:tensorflow:loss = 58067.3, step = 5101 (0.911 sec)
INFO:tensorflow:global_step/sec: 124.923
INFO:tensorflow:loss = 36591.6, step = 5201 (0.800 sec)
INFO:tensorflow:global_step/sec: 139.443
INFO:tensorflow:loss = 49899.2, step = 5301 (0.717 sec)
INFO:tensorflow:global_step/sec: 139.537
INFO:tensorflow:loss = 59058.9, step = 5401 (0.716 sec)
INFO:tensorflow:global_step/sec: 133.873
INFO:tensorflow:loss = 39515.4, step = 5501 (0.747 sec)
INFO:tensorflow:global_step/sec: 134.871
INFO:tensorflow:loss = 37766.9, step = 5601 (0.741 sec)
INFO:tensorflow:global_step/sec: 121.034
INFO:tensorflow:loss = 45288.4, step = 5701 (0.826 sec)
INFO:tensorflow:global_step/sec: 116.242
INFO:tensorflow:loss = 41809.8, step = 5801 (0.861 sec)
INFO:tensorflow:global_step/sec: 131.099
INFO:tensorflow:loss = 42418.6, step = 5901 (0.763 sec)
INFO:tensorflow:global_step/sec: 104.885
INFO:tensorflow:loss = 49231.9, step = 6001 (0.955 sec)
INFO:tensorflow:global_step/sec: 133.241
INFO:tensorflow:loss = 37564.2, step = 6101 (0.749 sec)
INFO:tensorflow:global_step/sec: 119.417
INFO:tensorflow:loss = 42841.4, step = 6201 (0.838 sec)
INFO:tensorflow:global_step/sec: 111.944
INFO:tensorflow:loss = 47508.5, step = 6301 (0.892 sec)
INFO:tensorflow:global_step/sec: 124.069
INFO:tensorflow:loss = 44208.4, step = 6401 (0.807 sec)
INFO:tensorflow:global_step/sec: 132.681
INFO:tensorflow:loss = 40910.1, step = 6501 (0.754 sec)
INFO:tensorflow:global_step/sec: 141.678
INFO:tensorflow:loss = 50287.4, step = 6601 (0.706 sec)
INFO:tensorflow:global_step/sec: 130.994
INFO:tensorflow:loss = 39495.4, step = 6701 (0.766 sec)
INFO:tensorflow:global_step/sec: 122.829
INFO:tensorflow:loss = 43742.4, step = 6801 (0.811 sec)
INFO:tensorflow:global_step/sec: 137.6
INFO:tensorflow:loss = 44626.8, step = 6901 (0.728 sec)
INFO:tensorflow:global_step/sec: 135.7
INFO:tensorflow:loss = 40497.6, step = 7001 (0.735 sec)
INFO:tensorflow:global_step/sec: 141.821
INFO:tensorflow:loss = 37368.0, step = 7101 (0.707 sec)
INFO:tensorflow:Saving checkpoints for 7201 into trained_models/reg-model-03/model.ckpt.
INFO:tensorflow:global_step/sec: 74.2526
* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================
INFO:tensorflow:Starting evaluation at 2017-11-16-10:03:31
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-7201
INFO:tensorflow:Finished evaluation at 2017-11-16-10:03:31
INFO:tensorflow:Saving dict for global step 7201: average_loss = 95.7069, global_step = 7201, loss = 47853.4
INFO:tensorflow:Validation (step 7201): average_loss = 95.7069, loss = 47853.4, global_step = 7201
INFO:tensorflow:loss = 51782.6, step = 7201 (2.507 sec)
INFO:tensorflow:global_step/sec: 48.8914
INFO:tensorflow:loss = 42918.2, step = 7301 (0.885 sec)
INFO:tensorflow:global_step/sec: 102.411
INFO:tensorflow:loss = 40595.4, step = 7401 (0.976 sec)
INFO:tensorflow:global_step/sec: 130.36
INFO:tensorflow:loss = 29395.8, step = 7501 (0.767 sec)
INFO:tensorflow:global_step/sec: 94.1363
INFO:tensorflow:loss = 51844.5, step = 7601 (1.066 sec)
INFO:tensorflow:global_step/sec: 79.7989
INFO:tensorflow:loss = 43390.5, step = 7701 (1.253 sec)
INFO:tensorflow:global_step/sec: 117.166
INFO:tensorflow:loss = 47932.0, step = 7801 (0.854 sec)
INFO:tensorflow:global_step/sec: 121.538
INFO:tensorflow:loss = 50953.5, step = 7901 (0.819 sec)
INFO:tensorflow:global_step/sec: 123.403
INFO:tensorflow:loss = 45364.8, step = 8001 (0.811 sec)
INFO:tensorflow:global_step/sec: 134.762
INFO:tensorflow:loss = 37158.0, step = 8101 (0.742 sec)
INFO:tensorflow:global_step/sec: 139.909
INFO:tensorflow:loss = 51231.4, step = 8201 (0.716 sec)
INFO:tensorflow:global_step/sec: 134.113
INFO:tensorflow:loss = 45554.2, step = 8301 (0.744 sec)
INFO:tensorflow:global_step/sec: 139.928
INFO:tensorflow:loss = 38198.0, step = 8401 (0.715 sec)
INFO:tensorflow:global_step/sec: 130.906
INFO:tensorflow:loss = 37683.4, step = 8501 (0.764 sec)
INFO:tensorflow:global_step/sec: 133.234
INFO:tensorflow:loss = 48373.8, step = 8601 (0.751 sec)
INFO:tensorflow:global_step/sec: 132.825
INFO:tensorflow:loss = 39778.3, step = 8701 (0.753 sec)
INFO:tensorflow:global_step/sec: 132.1
INFO:tensorflow:loss = 46058.1, step = 8801 (0.759 sec)
INFO:tensorflow:global_step/sec: 137.113
INFO:tensorflow:loss = 46736.0, step = 8901 (0.728 sec)
INFO:tensorflow:global_step/sec: 139.206
INFO:tensorflow:loss = 40734.2, step = 9001 (0.717 sec)
INFO:tensorflow:global_step/sec: 134.553
INFO:tensorflow:loss = 43216.0, step = 9101 (0.744 sec)
INFO:tensorflow:global_step/sec: 133.409
INFO:tensorflow:loss = 51448.4, step = 9201 (0.750 sec)
INFO:tensorflow:global_step/sec: 133.238
INFO:tensorflow:loss = 46285.5, step = 9301 (0.753 sec)
INFO:tensorflow:global_step/sec: 117.503
INFO:tensorflow:loss = 45954.6, step = 9401 (0.849 sec)
INFO:tensorflow:global_step/sec: 111.311
INFO:tensorflow:loss = 48068.8, step = 9501 (0.901 sec)
INFO:tensorflow:Saving checkpoints for 9601 into trained_models/reg-model-03/model.ckpt.
INFO:tensorflow:global_step/sec: 60.0009
* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================
INFO:tensorflow:Starting evaluation at 2017-11-16-10:03:52
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-9601
INFO:tensorflow:Finished evaluation at 2017-11-16-10:03:53
INFO:tensorflow:Saving dict for global step 9601: average_loss = 95.0352, global_step = 9601, loss = 47517.6
INFO:tensorflow:Validation (step 9601): average_loss = 95.0352, loss = 47517.6, global_step = 9601
INFO:tensorflow:loss = 53438.2, step = 9601 (2.797 sec)
INFO:tensorflow:global_step/sec: 50.6965
INFO:tensorflow:loss = 53259.5, step = 9701 (0.840 sec)
INFO:tensorflow:global_step/sec: 118.401
INFO:tensorflow:loss = 34251.2, step = 9801 (0.844 sec)
INFO:tensorflow:global_step/sec: 123.185
INFO:tensorflow:loss = 42877.2, step = 9901 (0.810 sec)
INFO:tensorflow:global_step/sec: 123.078
INFO:tensorflow:loss = 49685.0, step = 10001 (0.814 sec)
INFO:tensorflow:global_step/sec: 110.534
INFO:tensorflow:loss = 47262.9, step = 10101 (0.905 sec)
INFO:tensorflow:global_step/sec: 106.524
INFO:tensorflow:loss = 49317.5, step = 10201 (0.938 sec)
INFO:tensorflow:global_step/sec: 111.962
INFO:tensorflow:loss = 41488.7, step = 10301 (0.893 sec)
INFO:tensorflow:global_step/sec: 134.916
INFO:tensorflow:loss = 40965.7, step = 10401 (0.740 sec)
INFO:tensorflow:global_step/sec: 137.907
INFO:tensorflow:loss = 34246.2, step = 10501 (0.726 sec)
INFO:tensorflow:global_step/sec: 138.882
INFO:tensorflow:loss = 41920.2, step = 10601 (0.719 sec)
INFO:tensorflow:global_step/sec: 138.611
INFO:tensorflow:loss = 40829.2, step = 10701 (0.721 sec)
INFO:tensorflow:global_step/sec: 140.418
INFO:tensorflow:loss = 53730.8, step = 10801 (0.712 sec)
INFO:tensorflow:global_step/sec: 136.933
INFO:tensorflow:loss = 39976.0, step = 10901 (0.731 sec)
INFO:tensorflow:global_step/sec: 134.049
INFO:tensorflow:loss = 42867.9, step = 11001 (0.746 sec)
INFO:tensorflow:global_step/sec: 130.021
INFO:tensorflow:loss = 44854.7, step = 11101 (0.769 sec)
INFO:tensorflow:global_step/sec: 108.442
INFO:tensorflow:loss = 43357.3, step = 11201 (0.928 sec)
INFO:tensorflow:global_step/sec: 109.895
INFO:tensorflow:loss = 44547.3, step = 11301 (0.905 sec)
INFO:tensorflow:global_step/sec: 117.914
INFO:tensorflow:loss = 41676.9, step = 11401 (0.848 sec)
INFO:tensorflow:global_step/sec: 147.302
INFO:tensorflow:loss = 30990.5, step = 11501 (0.677 sec)
INFO:tensorflow:global_step/sec: 145.749
INFO:tensorflow:loss = 50680.9, step = 11601 (0.688 sec)
INFO:tensorflow:global_step/sec: 132.172
INFO:tensorflow:loss = 41594.9, step = 11701 (0.758 sec)
INFO:tensorflow:global_step/sec: 136.047
INFO:tensorflow:loss = 42287.2, step = 11801 (0.733 sec)
INFO:tensorflow:global_step/sec: 147.073
INFO:tensorflow:loss = 49489.5, step = 11901 (0.679 sec)
INFO:tensorflow:Saving checkpoints for 12001 into trained_models/reg-model-03/model.ckpt.
INFO:tensorflow:global_step/sec: 80.9467
* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================
INFO:tensorflow:Starting evaluation at 2017-11-16-10:04:13
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-12001
INFO:tensorflow:Finished evaluation at 2017-11-16-10:04:13
INFO:tensorflow:Saving dict for global step 12001: average_loss = 94.4759, global_step = 12001, loss = 47237.9
INFO:tensorflow:Validation (step 12001): average_loss = 94.4759, loss = 47237.9, global_step = 12001
INFO:tensorflow:loss = 48240.1, step = 12001 (2.193 sec)
INFO:tensorflow:global_step/sec: 59.5909
INFO:tensorflow:loss = 33612.9, step = 12101 (0.720 sec)
INFO:tensorflow:global_step/sec: 137.643
INFO:tensorflow:loss = 38861.4, step = 12201 (0.728 sec)
INFO:tensorflow:global_step/sec: 137.798
INFO:tensorflow:loss = 42655.6, step = 12301 (0.725 sec)
INFO:tensorflow:global_step/sec: 136.279
INFO:tensorflow:loss = 35384.6, step = 12401 (0.733 sec)
INFO:tensorflow:global_step/sec: 134.919
INFO:tensorflow:loss = 43430.5, step = 12501 (0.742 sec)
INFO:tensorflow:global_step/sec: 140.924
INFO:tensorflow:loss = 48275.4, step = 12601 (0.709 sec)
INFO:tensorflow:global_step/sec: 145.792
INFO:tensorflow:loss = 37730.8, step = 12701 (0.685 sec)
INFO:tensorflow:global_step/sec: 145.99
INFO:tensorflow:loss = 48645.6, step = 12801 (0.685 sec)
INFO:tensorflow:global_step/sec: 143.333
INFO:tensorflow:loss = 30703.5, step = 12901 (0.698 sec)
INFO:tensorflow:global_step/sec: 142.704
INFO:tensorflow:loss = 43611.7, step = 13001 (0.700 sec)
INFO:tensorflow:global_step/sec: 142.848
INFO:tensorflow:loss = 38099.9, step = 13101 (0.701 sec)
INFO:tensorflow:global_step/sec: 135.326
INFO:tensorflow:loss = 45194.5, step = 13201 (0.739 sec)
INFO:tensorflow:global_step/sec: 135.168
INFO:tensorflow:loss = 38871.8, step = 13301 (0.740 sec)
INFO:tensorflow:global_step/sec: 129.326
INFO:tensorflow:loss = 36956.7, step = 13401 (0.773 sec)
INFO:tensorflow:global_step/sec: 125.692
INFO:tensorflow:loss = 39388.8, step = 13501 (0.796 sec)
INFO:tensorflow:global_step/sec: 126.122
INFO:tensorflow:loss = 46446.1, step = 13601 (0.791 sec)
INFO:tensorflow:global_step/sec: 136.738
INFO:tensorflow:loss = 44792.6, step = 13701 (0.730 sec)
INFO:tensorflow:global_step/sec: 144.558
INFO:tensorflow:loss = 55869.1, step = 13801 (0.693 sec)
INFO:tensorflow:global_step/sec: 122.661
INFO:tensorflow:loss = 42912.5, step = 13901 (0.818 sec)
INFO:tensorflow:global_step/sec: 124.063
INFO:tensorflow:loss = 44276.9, step = 14001 (0.803 sec)
INFO:tensorflow:global_step/sec: 140.367
INFO:tensorflow:loss = 32231.6, step = 14101 (0.713 sec)
INFO:tensorflow:global_step/sec: 128.227
INFO:tensorflow:loss = 33367.5, step = 14201 (0.779 sec)
INFO:tensorflow:global_step/sec: 121.112
INFO:tensorflow:loss = 43804.7, step = 14301 (0.827 sec)
INFO:tensorflow:Saving checkpoints for 14401 into trained_models/reg-model-03/model.ckpt.
INFO:tensorflow:global_step/sec: 53.1746
* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================
INFO:tensorflow:Starting evaluation at 2017-11-16-10:04:33
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-14401
INFO:tensorflow:Finished evaluation at 2017-11-16-10:04:33
INFO:tensorflow:Saving dict for global step 14401: average_loss = 93.9907, global_step = 14401, loss = 46995.4
INFO:tensorflow:Validation (step 14401): average_loss = 93.9907, loss = 46995.4, global_step = 14401
INFO:tensorflow:loss = 57803.0, step = 14401 (3.205 sec)
INFO:tensorflow:global_step/sec: 46.7932
INFO:tensorflow:loss = 46377.5, step = 14501 (0.813 sec)
INFO:tensorflow:global_step/sec: 134.385
INFO:tensorflow:loss = 34797.7, step = 14601 (0.744 sec)
INFO:tensorflow:global_step/sec: 112.674
INFO:tensorflow:loss = 44715.1, step = 14701 (0.888 sec)
INFO:tensorflow:global_step/sec: 130.88
INFO:tensorflow:loss = 39205.6, step = 14801 (0.764 sec)
INFO:tensorflow:global_step/sec: 135.123
INFO:tensorflow:loss = 40149.7, step = 14901 (0.740 sec)
INFO:tensorflow:global_step/sec: 133.989
INFO:tensorflow:loss = 43184.4, step = 15001 (0.747 sec)
INFO:tensorflow:global_step/sec: 134.434
INFO:tensorflow:loss = 38481.5, step = 15101 (0.744 sec)
INFO:tensorflow:global_step/sec: 104.198
INFO:tensorflow:loss = 40600.9, step = 15201 (0.959 sec)
INFO:tensorflow:global_step/sec: 137.205
INFO:tensorflow:loss = 49492.6, step = 15301 (0.729 sec)
INFO:tensorflow:global_step/sec: 134.984
INFO:tensorflow:loss = 29930.4, step = 15401 (0.741 sec)
INFO:tensorflow:global_step/sec: 137.089
INFO:tensorflow:loss = 46129.1, step = 15501 (0.730 sec)
INFO:tensorflow:global_step/sec: 132.123
INFO:tensorflow:loss = 47175.0, step = 15601 (0.758 sec)
INFO:tensorflow:global_step/sec: 103.908
INFO:tensorflow:loss = 39116.7, step = 15701 (0.963 sec)
INFO:tensorflow:global_step/sec: 95.1968
INFO:tensorflow:loss = 35493.4, step = 15801 (1.057 sec)
INFO:tensorflow:global_step/sec: 124.185
INFO:tensorflow:loss = 41123.8, step = 15901 (0.797 sec)
INFO:tensorflow:global_step/sec: 119.053
INFO:tensorflow:loss = 28964.1, step = 16001 (0.843 sec)
INFO:tensorflow:global_step/sec: 130.346
INFO:tensorflow:loss = 42644.8, step = 16101 (0.766 sec)
INFO:tensorflow:global_step/sec: 126.435
INFO:tensorflow:loss = 41032.1, step = 16201 (0.791 sec)
INFO:tensorflow:global_step/sec: 134.572
INFO:tensorflow:loss = 38422.3, step = 16301 (0.742 sec)
INFO:tensorflow:global_step/sec: 139.46
INFO:tensorflow:loss = 40377.9, step = 16401 (0.718 sec)
INFO:tensorflow:global_step/sec: 138.869
INFO:tensorflow:loss = 45031.1, step = 16501 (0.720 sec)
INFO:tensorflow:global_step/sec: 140.669
INFO:tensorflow:loss = 44225.1, step = 16601 (0.711 sec)
INFO:tensorflow:global_step/sec: 137.776
INFO:tensorflow:loss = 44445.1, step = 16701 (0.727 sec)
INFO:tensorflow:Saving checkpoints for 16801 into trained_models/reg-model-03/model.ckpt.
INFO:tensorflow:global_step/sec: 71.631
* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================
INFO:tensorflow:Starting evaluation at 2017-11-16-10:04:54
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-16801
INFO:tensorflow:Finished evaluation at 2017-11-16-10:04:54
INFO:tensorflow:Saving dict for global step 16801: average_loss = 93.6571, global_step = 16801, loss = 46828.6
INFO:tensorflow:Validation (step 16801): average_loss = 93.6571, loss = 46828.6, global_step = 16801
INFO:tensorflow:loss = 43306.4, step = 16801 (2.605 sec)
INFO:tensorflow:global_step/sec: 46.2648
INFO:tensorflow:loss = 54241.1, step = 16901 (0.952 sec)
INFO:tensorflow:global_step/sec: 140.135
INFO:tensorflow:loss = 34022.9, step = 17001 (0.714 sec)
INFO:tensorflow:global_step/sec: 130.628
INFO:tensorflow:loss = 45510.8, step = 17101 (0.766 sec)
INFO:tensorflow:global_step/sec: 140.74
INFO:tensorflow:loss = 43210.0, step = 17201 (0.710 sec)
INFO:tensorflow:global_step/sec: 138.249
INFO:tensorflow:loss = 50877.1, step = 17301 (0.723 sec)
INFO:tensorflow:global_step/sec: 135.93
INFO:tensorflow:loss = 38702.2, step = 17401 (0.735 sec)
INFO:tensorflow:global_step/sec: 137.277
INFO:tensorflow:loss = 34215.1, step = 17501 (0.729 sec)
INFO:tensorflow:global_step/sec: 134.199
INFO:tensorflow:loss = 39389.1, step = 17601 (0.746 sec)
INFO:tensorflow:global_step/sec: 130.333
INFO:tensorflow:loss = 43825.1, step = 17701 (0.767 sec)
INFO:tensorflow:global_step/sec: 131.614
INFO:tensorflow:loss = 43972.8, step = 17801 (0.760 sec)
INFO:tensorflow:global_step/sec: 134.144
INFO:tensorflow:loss = 48032.7, step = 17901 (0.745 sec)
INFO:tensorflow:global_step/sec: 139.993
INFO:tensorflow:loss = 43332.0, step = 18001 (0.715 sec)
INFO:tensorflow:global_step/sec: 142.514
INFO:tensorflow:loss = 39407.1, step = 18101 (0.702 sec)
INFO:tensorflow:global_step/sec: 138.503
INFO:tensorflow:loss = 46066.7, step = 18201 (0.722 sec)
INFO:tensorflow:global_step/sec: 126.932
INFO:tensorflow:loss = 43000.3, step = 18301 (0.788 sec)
INFO:tensorflow:global_step/sec: 136.917
INFO:tensorflow:loss = 43819.3, step = 18401 (0.731 sec)
INFO:tensorflow:global_step/sec: 117.591
INFO:tensorflow:loss = 43357.4, step = 18501 (0.850 sec)
INFO:tensorflow:global_step/sec: 107.758
INFO:tensorflow:loss = 50256.3, step = 18601 (0.930 sec)
INFO:tensorflow:global_step/sec: 104.861
INFO:tensorflow:loss = 42203.0, step = 18701 (0.956 sec)
INFO:tensorflow:global_step/sec: 111.49
INFO:tensorflow:loss = 39352.0, step = 18801 (0.894 sec)
INFO:tensorflow:global_step/sec: 114.315
INFO:tensorflow:loss = 44110.5, step = 18901 (0.874 sec)
INFO:tensorflow:global_step/sec: 119.923
INFO:tensorflow:loss = 52128.4, step = 19001 (0.834 sec)
INFO:tensorflow:global_step/sec: 124.208
INFO:tensorflow:loss = 44822.1, step = 19101 (0.804 sec)
INFO:tensorflow:Saving checkpoints for 19201 into trained_models/reg-model-03/model.ckpt.
INFO:tensorflow:global_step/sec: 49.8886
* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================
INFO:tensorflow:Starting evaluation at 2017-11-16-10:05:15
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-19201
INFO:tensorflow:Finished evaluation at 2017-11-16-10:05:16
INFO:tensorflow:Saving dict for global step 19201: average_loss = 93.4035, global_step = 19201, loss = 46701.7
INFO:tensorflow:Validation (step 19201): average_loss = 93.4035, loss = 46701.7, global_step = 19201
INFO:tensorflow:loss = 41447.1, step = 19201 (3.591 sec)
INFO:tensorflow:global_step/sec: 38.6598
INFO:tensorflow:loss = 39285.7, step = 19301 (1.002 sec)
INFO:tensorflow:global_step/sec: 99.165
INFO:tensorflow:loss = 47136.1, step = 19401 (1.010 sec)
INFO:tensorflow:global_step/sec: 93.02
INFO:tensorflow:loss = 48500.1, step = 19501 (1.072 sec)
INFO:tensorflow:global_step/sec: 104.564
INFO:tensorflow:loss = 41969.5, step = 19601 (0.955 sec)
INFO:tensorflow:global_step/sec: 138.815
INFO:tensorflow:loss = 45881.1, step = 19701 (0.722 sec)
INFO:tensorflow:global_step/sec: 104.89
INFO:tensorflow:loss = 38483.6, step = 19801 (0.957 sec)
INFO:tensorflow:global_step/sec: 97.0502
INFO:tensorflow:loss = 38991.4, step = 19901 (1.026 sec)
INFO:tensorflow:global_step/sec: 117.119
INFO:tensorflow:loss = 33769.5, step = 20001 (0.852 sec)
INFO:tensorflow:global_step/sec: 114.517
INFO:tensorflow:loss = 37037.8, step = 20101 (0.874 sec)
INFO:tensorflow:global_step/sec: 101.101
INFO:tensorflow:loss = 44271.2, step = 20201 (0.996 sec)
INFO:tensorflow:global_step/sec: 121.698
INFO:tensorflow:loss = 47297.9, step = 20301 (0.814 sec)
INFO:tensorflow:global_step/sec: 131.795
INFO:tensorflow:loss = 49067.2, step = 20401 (0.759 sec)
INFO:tensorflow:global_step/sec: 130.245
INFO:tensorflow:loss = 29920.0, step = 20501 (0.768 sec)
INFO:tensorflow:global_step/sec: 117.683
INFO:tensorflow:loss = 41268.2, step = 20601 (0.853 sec)
INFO:tensorflow:global_step/sec: 117.073
INFO:tensorflow:loss = 48263.5, step = 20701 (0.851 sec)
INFO:tensorflow:global_step/sec: 115.17
INFO:tensorflow:loss = 43600.9, step = 20801 (0.867 sec)
INFO:tensorflow:global_step/sec: 132.257
INFO:tensorflow:loss = 53666.0, step = 20901 (0.758 sec)
INFO:tensorflow:global_step/sec: 115.403
INFO:tensorflow:loss = 43610.4, step = 21001 (0.867 sec)
INFO:tensorflow:global_step/sec: 113.231
INFO:tensorflow:loss = 39353.9, step = 21101 (0.885 sec)
INFO:tensorflow:global_step/sec: 117.822
INFO:tensorflow:loss = 41081.1, step = 21201 (0.846 sec)
INFO:tensorflow:global_step/sec: 108.78
INFO:tensorflow:loss = 43726.5, step = 21301 (0.920 sec)
INFO:tensorflow:global_step/sec: 101.509
INFO:tensorflow:loss = 46147.2, step = 21401 (0.988 sec)
INFO:tensorflow:global_step/sec: 106.962
INFO:tensorflow:loss = 42668.0, step = 21501 (0.933 sec)
INFO:tensorflow:Saving checkpoints for 21601 into trained_models/reg-model-03/model.ckpt.
INFO:tensorflow:global_step/sec: 68.8721
* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================
INFO:tensorflow:Starting evaluation at 2017-11-16-10:05:38
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-21601
INFO:tensorflow:Finished evaluation at 2017-11-16-10:05:39
INFO:tensorflow:Saving dict for global step 21601: average_loss = 93.2296, global_step = 21601, loss = 46614.8
INFO:tensorflow:Validation (step 21601): average_loss = 93.2296, loss = 46614.8, global_step = 21601
INFO:tensorflow:loss = 39012.0, step = 21601 (2.402 sec)
INFO:tensorflow:global_step/sec: 55.8269
INFO:tensorflow:loss = 42405.3, step = 21701 (0.839 sec)
INFO:tensorflow:global_step/sec: 119.44
INFO:tensorflow:loss = 47157.8, step = 21801 (0.837 sec)
INFO:tensorflow:global_step/sec: 110.084
INFO:tensorflow:loss = 40285.0, step = 21901 (0.910 sec)
INFO:tensorflow:global_step/sec: 114.772
INFO:tensorflow:loss = 51357.3, step = 22001 (0.873 sec)
INFO:tensorflow:global_step/sec: 129.632
INFO:tensorflow:loss = 43420.6, step = 22101 (0.770 sec)
INFO:tensorflow:global_step/sec: 137.865
INFO:tensorflow:loss = 50706.4, step = 22201 (0.725 sec)
INFO:tensorflow:global_step/sec: 116.058
INFO:tensorflow:loss = 47902.4, step = 22301 (0.863 sec)
INFO:tensorflow:global_step/sec: 120.49
INFO:tensorflow:loss = 34552.7, step = 22401 (0.829 sec)
INFO:tensorflow:global_step/sec: 116.685
INFO:tensorflow:loss = 40107.3, step = 22501 (0.856 sec)
INFO:tensorflow:global_step/sec: 127.137
INFO:tensorflow:loss = 43026.8, step = 22601 (0.786 sec)
INFO:tensorflow:global_step/sec: 133.658
INFO:tensorflow:loss = 38352.9, step = 22701 (0.751 sec)
INFO:tensorflow:global_step/sec: 143.827
INFO:tensorflow:loss = 47005.3, step = 22801 (0.693 sec)
INFO:tensorflow:global_step/sec: 126.76
INFO:tensorflow:loss = 34692.8, step = 22901 (0.788 sec)
INFO:tensorflow:global_step/sec: 123.352
INFO:tensorflow:loss = 45891.2, step = 23001 (0.811 sec)
INFO:tensorflow:global_step/sec: 123.05
INFO:tensorflow:loss = 44916.1, step = 23101 (0.813 sec)
INFO:tensorflow:global_step/sec: 131.7
INFO:tensorflow:loss = 42158.5, step = 23201 (0.758 sec)
INFO:tensorflow:global_step/sec: 149.588
INFO:tensorflow:loss = 34986.8, step = 23301 (0.668 sec)
INFO:tensorflow:global_step/sec: 115.691
INFO:tensorflow:loss = 49937.9, step = 23401 (0.865 sec)
INFO:tensorflow:global_step/sec: 145.735
INFO:tensorflow:loss = 51601.8, step = 23501 (0.685 sec)
INFO:tensorflow:global_step/sec: 96.7756
INFO:tensorflow:loss = 44515.1, step = 23601 (1.039 sec)
INFO:tensorflow:global_step/sec: 121.551
INFO:tensorflow:loss = 43017.3, step = 23701 (0.817 sec)
INFO:tensorflow:global_step/sec: 114.851
INFO:tensorflow:loss = 34417.2, step = 23801 (0.876 sec)
INFO:tensorflow:global_step/sec: 109.275
INFO:tensorflow:loss = 45028.1, step = 23901 (0.911 sec)
INFO:tensorflow:Saving checkpoints for 24000 into trained_models/reg-model-03/model.ckpt.
INFO:tensorflow:Loss for final step: 41759.8.
* data input_fn:
================
Input file(s): data/valid-*.csv
Batch size: 500
Epoch Count: 1
Mode: eval
Shuffle: False
================
INFO:tensorflow:Starting evaluation at 2017-11-16-10:05:59
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-24000
INFO:tensorflow:Finished evaluation at 2017-11-16-10:06:00
INFO:tensorflow:Saving dict for global step 24000: average_loss = 93.1534, global_step = 24000, loss = 46576.7
INFO:tensorflow:Restoring parameters from trained_models/reg-model-03/model.ckpt-24000
INFO:tensorflow:Assets added to graph.
INFO:tensorflow:No assets to write.
INFO:tensorflow:SavedModel written to: b"trained_models/reg-model-03/export/Servo/temp-b'1510826760'/saved_model.pbtxt"
.......................................
Experiment finished at 10:06:01
Experiment elapsed time: 219.736155 seconds