notebook.community

Edit and run

Licensed under the Apache License, Version 2.0 (the "License");





In [0]:



    


# Copyright 2018 The TensorFlow Hub Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================

BERT End to End (Fine-tuning + Predicting) in 5 minutes with Cloud TPU

Overview

BERT, or Bidirectional Embedding Representations from Transformers, is a new method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks. The academic paper can be found here: https://arxiv.org/abs/1810.04805.

This Colab demonstates using a free Colab Cloud TPU to fine-tune sentence and sentence-pair classification tasks built on top of pretrained BERT models and run predictions on tuned model. The colab demonsrates loading pretrained BERT models from both TF Hub and checkpoints.

Note: You will need a GCP (Google Compute Engine) account and a GCS (Google Cloud Storage) bucket for this Colab to run.

Please follow the Google Cloud TPU quickstart for how to create GCP account and GCS bucket. You have $300 free credit to get started with any GCP product. You can learn more about Cloud TPU at https://cloud.google.com/tpu/docs.

This notebook is hosted on GitHub. To view it in its original repository, after opening the notebook, select File > View on GitHub.

Learning objectives

In this notebook, you will learn how to train and evaluate a BERT model using TPU.

Instructions

  Train on TPU

  1. Create a Cloud Storage bucket for your TensorBoard logs at http://console.cloud.google.com/storage and fill in the BUCKET parameter in the "Parameters" section below.

  2. On the main menu, click Runtime and select Change runtime type. Set "TPU" as the hardware accelerator.

  3. Click Runtime again and select Runtime > Run All (Watch out: the "Colab-only auth for this notebook and the TPU" cell requires user input). You can also run the cells manually with Shift-ENTER.

Set up your TPU environment

In this section, you perform the following tasks:

  • Set up a Colab TPU running environment
  • Verify that you are connected to a TPU device
  • Upload your credentials to TPU to access your GCS bucket.




In [1]:



    


import datetime
import json
import os
import pprint
import random
import string
import sys
import tensorflow as tf

assert 'COLAB_TPU_ADDR' in os.environ, 'ERROR: Not connected to a TPU runtime; please see the first cell in this notebook for instructions!'
TPU_ADDRESS = 'grpc://' + os.environ['COLAB_TPU_ADDR']
print('TPU address is', TPU_ADDRESS)

from google.colab import auth
auth.authenticate_user()
with tf.Session(TPU_ADDRESS) as session:
  print('TPU devices:')
  pprint.pprint(session.list_devices())

  # Upload credentials to TPU.
  with open('/content/adc.json', 'r') as f:
    auth_info = json.load(f)
  tf.contrib.cloud.configure_gcs(session, credentials=auth_info)
  # Now credentials are set for all future sessions on this TPU.



    


















    






TPU address is grpc://10.114.88.226:8470

WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.

TPU devices:
[_DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:CPU:0, CPU, -1, 3020137555896628722),
 _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 17179869184, 15223542790104685402),
 _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:0, TPU, 17179869184, 1843375914324221839),
 _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:1, TPU, 17179869184, 9894822734449577370),
 _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:2, TPU, 17179869184, 14380569740864114478),
 _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:3, TPU, 17179869184, 17084474974895924727),
 _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:4, TPU, 17179869184, 7110320513217349863),
 _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:5, TPU, 17179869184, 11587941208496677740),
 _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:6, TPU, 17179869184, 10553633026221058370),
 _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU:7, TPU, 17179869184, 9279467740370141958),
 _DeviceAttributes(/job:tpu_worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 17179869184, 2228303855717980436)]

Prepare and import BERT modules

​ With your environment configured, you can now prepare and import the BERT modules. The following step clones the source code from GitHub and import the modules from the source. Alternatively, you can install BERT using pip (!pip install bert-tensorflow).





In [2]:



    


import sys

!test -d bert_repo || git clone https://github.com/google-research/bert bert_repo
if not 'bert_repo' in sys.path:
  sys.path += ['bert_repo']

# import python modules defined by BERT
import modeling
import optimization
import run_classifier
import run_classifier_with_tfhub
import tokenization

# import tfhub 
import tensorflow_hub as hub



    


















    






Cloning into 'bert_repo'...
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 325 (delta 0), reused 1 (delta 0), pack-reused 320
Receiving objects: 100% (325/325), 267.86 KiB | 3.62 MiB/s, done.
Resolving deltas: 100% (180/180), done.










    






WARNING: Logging before flag parsing goes to stderr.
W0328 17:50:06.286101 140342252849024 __init__.py:56] Some hub symbols are not available because TensorFlow version is less than 1.14

Prepare for training

This next section of code performs the following tasks:

  • Specify task and download training data.
  • Specify BERT pretrained model
  • Specify GS bucket, create output directory for model checkpoints and eval results.




In [3]:



    


TASK = 'MRPC' #@param {type:"string"}
assert TASK in ('MRPC', 'CoLA'), 'Only (MRPC, CoLA) are demonstrated here.'

# Download glue data.
! test -d download_glue_repo || git clone https://gist.github.com/60c2bdb54d156a41194446737ce03e2e.git download_glue_repo
!python download_glue_repo/download_glue_data.py --data_dir='glue_data' --tasks=$TASK

TASK_DATA_DIR = 'glue_data/' + TASK
print('***** Task data directory: {} *****'.format(TASK_DATA_DIR))
!ls $TASK_DATA_DIR

BUCKET = 'YOUR_BUCKET' #@param {type:"string"}
assert BUCKET, 'Must specify an existing GCS bucket name'
OUTPUT_DIR = 'gs://{}/bert-tfhub/models/{}'.format(BUCKET, TASK)
tf.gfile.MakeDirs(OUTPUT_DIR)
print('***** Model output directory: {} *****'.format(OUTPUT_DIR))

# Available pretrained model checkpoints:
#   uncased_L-12_H-768_A-12: uncased BERT base model
#   uncased_L-24_H-1024_A-16: uncased BERT large model
#   cased_L-12_H-768_A-12: cased BERT large model
BERT_MODEL = 'uncased_L-12_H-768_A-12' #@param {type:"string"}
BERT_MODEL_HUB = 'https://tfhub.dev/google/bert_' + BERT_MODEL + '/1'



    


















    






Cloning into 'download_glue_repo'...
remote: Enumerating objects: 21, done.
remote: Total 21 (delta 0), reused 0 (delta 0), pack-reused 21
Unpacking objects: 100% (21/21), done.
Processing MRPC...
Local MRPC data not specified, downloading data from https://dl.fbaipublicfiles.com/senteval/senteval_data/msr_paraphrase_train.txt
	Completed!
***** Task data directory: glue_data/MRPC *****
dev_ids.tsv  msr_paraphrase_test.txt   test.tsv
dev.tsv      msr_paraphrase_train.txt  train.tsv
***** Model output directory: gs://YOUR_BUCKET/bert-tfhub/models/MRPC *****

Now let's load tokenizer module from TF Hub and play with it.





In [4]:



    


tokenizer = run_classifier_with_tfhub.create_tokenizer_from_hub_module(BERT_MODEL_HUB)
tokenizer.tokenize("This here's an example of using the BERT tokenizer")



    


















    






WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/control_flow_ops.py:3632: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.










    






W0328 17:56:25.475618 140342252849024 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/control_flow_ops.py:3632: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.










    






INFO:tensorflow:Saver not created because there are no variables in the graph to restore










    






I0328 17:56:28.058523 140342252849024 saver.py:1483] Saver not created because there are no variables in the graph to restore










    
Out[4]:








['this',
 'here',
 "'",
 's',
 'an',
 'example',
 'of',
 'using',
 'the',
 'bert',
 'token',
 '##izer']

Also we initilize our hyperprams, prepare the training data and initialize TPU config.





In [0]:



    


TRAIN_BATCH_SIZE = 32
EVAL_BATCH_SIZE = 8
PREDICT_BATCH_SIZE = 8
LEARNING_RATE = 2e-5
NUM_TRAIN_EPOCHS = 3.0
MAX_SEQ_LENGTH = 128
# Warmup is a period of time where hte learning rate 
# is small and gradually increases--usually helps training.
WARMUP_PROPORTION = 0.1
# Model configs
SAVE_CHECKPOINTS_STEPS = 1000
SAVE_SUMMARY_STEPS = 500

processors = {
  "cola": run_classifier.ColaProcessor,
  "mnli": run_classifier.MnliProcessor,
  "mrpc": run_classifier.MrpcProcessor,
}
processor = processors[TASK.lower()]()
label_list = processor.get_labels()

# Compute number of train and warmup steps from batch size
train_examples = processor.get_train_examples(TASK_DATA_DIR)
num_train_steps = int(len(train_examples) / TRAIN_BATCH_SIZE * NUM_TRAIN_EPOCHS)
num_warmup_steps = int(num_train_steps * WARMUP_PROPORTION)

# Setup TPU related config
tpu_cluster_resolver = tf.contrib.cluster_resolver.TPUClusterResolver(TPU_ADDRESS)
NUM_TPU_CORES = 8
ITERATIONS_PER_LOOP = 1000

def get_run_config(output_dir):
  return tf.contrib.tpu.RunConfig(
    cluster=tpu_cluster_resolver,
    model_dir=output_dir,
    save_checkpoints_steps=SAVE_CHECKPOINTS_STEPS,
    tpu_config=tf.contrib.tpu.TPUConfig(
        iterations_per_loop=ITERATIONS_PER_LOOP,
        num_shards=NUM_TPU_CORES,
        per_host_input_for_training=tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2))

Fine-tune and Run Predictions on a pretrained BERT Model from TF Hub

This section demonstrates fine-tuning from a pre-trained BERT TF Hub module and running predictions.





In [6]:



    


# Force TF Hub writes to the GS bucket we provide.
os.environ['TFHUB_CACHE_DIR'] = OUTPUT_DIR

model_fn = run_classifier_with_tfhub.model_fn_builder(
  num_labels=len(label_list),
  learning_rate=LEARNING_RATE,
  num_train_steps=num_train_steps,
  num_warmup_steps=num_warmup_steps,
  use_tpu=True,
  bert_hub_module_handle=BERT_MODEL_HUB
)

estimator_from_tfhub = tf.contrib.tpu.TPUEstimator(
  use_tpu=True,
  model_fn=model_fn,
  config=get_run_config(OUTPUT_DIR),
  train_batch_size=TRAIN_BATCH_SIZE,
  eval_batch_size=EVAL_BATCH_SIZE,
  predict_batch_size=PREDICT_BATCH_SIZE,
)



    


















    






WARNING:tensorflow:Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7fa3b44f2b70>) includes params argument, but params are not passed to Estimator.










    






W0328 17:58:05.653864 140342252849024 estimator.py:1924] Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7fa3b44f2b70>) includes params argument, but params are not passed to Estimator.










    






INFO:tensorflow:Using config: {'_model_dir': 'gs://YOUR_BUCKET/bert-tfhub/models/MRPC', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 500, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
cluster_def {
  job {
    name: "worker"
    tasks {
      key: 0
      value: "10.114.88.226:8470"
    }
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fa3b7fe4588>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': 'grpc://10.114.88.226:8470', '_evaluation_master': 'grpc://10.114.88.226:8470', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None), '_cluster': <tensorflow.python.distribute.cluster_resolver.tpu_cluster_resolver.TPUClusterResolver object at 0x7fa3de0b4908>}










    






I0328 17:58:05.659372 140342252849024 estimator.py:201] Using config: {'_model_dir': 'gs://YOUR_BUCKET/bert-tfhub/models/MRPC', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 500, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
cluster_def {
  job {
    name: "worker"
    tasks {
      key: 0
      value: "10.114.88.226:8470"
    }
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fa3b7fe4588>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': 'grpc://10.114.88.226:8470', '_evaluation_master': 'grpc://10.114.88.226:8470', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None), '_cluster': <tensorflow.python.distribute.cluster_resolver.tpu_cluster_resolver.TPUClusterResolver object at 0x7fa3de0b4908>}










    






INFO:tensorflow:_TPUContext: eval_on_tpu True










    






I0328 17:58:05.663435 140342252849024 tpu_context.py:202] _TPUContext: eval_on_tpu True

At this point, you can now fine-tune the model, evaluate it, and run predictions on it.





In [0]:



    


# Train the model
def model_train(estimator):
  print('MRPC/CoLA on BERT base model normally takes about 2-3 minutes. Please wait...')
  # We'll set sequences to be at most 128 tokens long.
  train_features = run_classifier.convert_examples_to_features(
      train_examples, label_list, MAX_SEQ_LENGTH, tokenizer)
  print('***** Started training at {} *****'.format(datetime.datetime.now()))
  print('  Num examples = {}'.format(len(train_examples)))
  print('  Batch size = {}'.format(TRAIN_BATCH_SIZE))
  tf.logging.info("  Num steps = %d", num_train_steps)
  train_input_fn = run_classifier.input_fn_builder(
      features=train_features,
      seq_length=MAX_SEQ_LENGTH,
      is_training=True,
      drop_remainder=True)
  estimator.train(input_fn=train_input_fn, max_steps=num_train_steps)
  print('***** Finished training at {} *****'.format(datetime.datetime.now()))



    











In [8]:



    


model_train(estimator_from_tfhub)



    


















    






MRPC/CoLA on BERT base model normally takes about 2-3 minutes. Please wait...
INFO:tensorflow:Writing example 0 of 3668










    






I0328 17:58:18.675322 140342252849024 run_classifier.py:774] Writing example 0 of 3668










    






INFO:tensorflow:*** Example ***










    






I0328 17:58:18.687642 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: train-1










    






I0328 17:58:18.694066 140342252849024 run_classifier.py:462] guid: train-1










    






INFO:tensorflow:tokens: [CLS] am ##ro ##zi accused his brother , whom he called " the witness " , of deliberately di ##stor ##ting his evidence . [SEP] referring to him as only " the witness " , am ##ro ##zi accused his brother of deliberately di ##stor ##ting his evidence . [SEP]










    






I0328 17:58:18.699178 140342252849024 run_classifier.py:464] tokens: [CLS] am ##ro ##zi accused his brother , whom he called " the witness " , of deliberately di ##stor ##ting his evidence . [SEP] referring to him as only " the witness " , am ##ro ##zi accused his brother of deliberately di ##stor ##ting his evidence . [SEP]










    






INFO:tensorflow:input_ids: 101 2572 3217 5831 5496 2010 2567 1010 3183 2002 2170 1000 1996 7409 1000 1010 1997 9969 4487 23809 3436 2010 3350 1012 102 7727 2000 2032 2004 2069 1000 1996 7409 1000 1010 2572 3217 5831 5496 2010 2567 1997 9969 4487 23809 3436 2010 3350 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 17:58:18.704080 140342252849024 run_classifier.py:465] input_ids: 101 2572 3217 5831 5496 2010 2567 1010 3183 2002 2170 1000 1996 7409 1000 1010 1997 9969 4487 23809 3436 2010 3350 1012 102 7727 2000 2032 2004 2069 1000 1996 7409 1000 1010 2572 3217 5831 5496 2010 2567 1997 9969 4487 23809 3436 2010 3350 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 17:58:18.706292 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 17:58:18.707699 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 1 (id = 1)










    






I0328 17:58:18.709444 140342252849024 run_classifier.py:468] label: 1 (id = 1)










    






INFO:tensorflow:*** Example ***










    






I0328 17:58:18.711830 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: train-2










    






I0328 17:58:18.714613 140342252849024 run_classifier.py:462] guid: train-2










    






INFO:tensorflow:tokens: [CLS] yu ##ca ##ip ##a owned dominic ##k ' s before selling the chain to safe ##way in 1998 for $ 2 . 5 billion . [SEP] yu ##ca ##ip ##a bought dominic ##k ' s in 1995 for $ 69 ##3 million and sold it to safe ##way for $ 1 . 8 billion in 1998 . [SEP]










    






I0328 17:58:18.716314 140342252849024 run_classifier.py:464] tokens: [CLS] yu ##ca ##ip ##a owned dominic ##k ' s before selling the chain to safe ##way in 1998 for $ 2 . 5 billion . [SEP] yu ##ca ##ip ##a bought dominic ##k ' s in 1995 for $ 69 ##3 million and sold it to safe ##way for $ 1 . 8 billion in 1998 . [SEP]










    






INFO:tensorflow:input_ids: 101 9805 3540 11514 2050 3079 11282 2243 1005 1055 2077 4855 1996 4677 2000 3647 4576 1999 2687 2005 1002 1016 1012 1019 4551 1012 102 9805 3540 11514 2050 4149 11282 2243 1005 1055 1999 2786 2005 1002 6353 2509 2454 1998 2853 2009 2000 3647 4576 2005 1002 1015 1012 1022 4551 1999 2687 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 17:58:18.718473 140342252849024 run_classifier.py:465] input_ids: 101 9805 3540 11514 2050 3079 11282 2243 1005 1055 2077 4855 1996 4677 2000 3647 4576 1999 2687 2005 1002 1016 1012 1019 4551 1012 102 9805 3540 11514 2050 4149 11282 2243 1005 1055 1999 2786 2005 1002 6353 2509 2454 1998 2853 2009 2000 3647 4576 2005 1002 1015 1012 1022 4551 1999 2687 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 17:58:18.720058 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 17:58:18.722793 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 0 (id = 0)










    






I0328 17:58:18.725336 140342252849024 run_classifier.py:468] label: 0 (id = 0)










    






INFO:tensorflow:*** Example ***










    






I0328 17:58:18.729561 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: train-3










    






I0328 17:58:18.733903 140342252849024 run_classifier.py:462] guid: train-3










    






INFO:tensorflow:tokens: [CLS] they had published an advertisement on the internet on june 10 , offering the cargo for sale , he added . [SEP] on june 10 , the ship ' s owners had published an advertisement on the internet , offering the explosives for sale . [SEP]










    






I0328 17:58:18.735062 140342252849024 run_classifier.py:464] tokens: [CLS] they had published an advertisement on the internet on june 10 , offering the cargo for sale , he added . [SEP] on june 10 , the ship ' s owners had published an advertisement on the internet , offering the explosives for sale . [SEP]










    






INFO:tensorflow:input_ids: 101 2027 2018 2405 2019 15147 2006 1996 4274 2006 2238 2184 1010 5378 1996 6636 2005 5096 1010 2002 2794 1012 102 2006 2238 2184 1010 1996 2911 1005 1055 5608 2018 2405 2019 15147 2006 1996 4274 1010 5378 1996 14792 2005 5096 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 17:58:18.736261 140342252849024 run_classifier.py:465] input_ids: 101 2027 2018 2405 2019 15147 2006 1996 4274 2006 2238 2184 1010 5378 1996 6636 2005 5096 1010 2002 2794 1012 102 2006 2238 2184 1010 1996 2911 1005 1055 5608 2018 2405 2019 15147 2006 1996 4274 1010 5378 1996 14792 2005 5096 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 17:58:18.737497 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 17:58:18.738644 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 1 (id = 1)










    






I0328 17:58:18.739753 140342252849024 run_classifier.py:468] label: 1 (id = 1)










    






INFO:tensorflow:*** Example ***










    






I0328 17:58:18.748307 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: train-4










    






I0328 17:58:18.755753 140342252849024 run_classifier.py:462] guid: train-4










    






INFO:tensorflow:tokens: [CLS] around 03 ##35 gm ##t , tab shares were up 19 cents , or 4 . 4 % , at a $ 4 . 56 , having earlier set a record high of a $ 4 . 57 . [SEP] tab shares jumped 20 cents , or 4 . 6 % , to set a record closing high at a $ 4 . 57 . [SEP]










    






I0328 17:58:18.758370 140342252849024 run_classifier.py:464] tokens: [CLS] around 03 ##35 gm ##t , tab shares were up 19 cents , or 4 . 4 % , at a $ 4 . 56 , having earlier set a record high of a $ 4 . 57 . [SEP] tab shares jumped 20 cents , or 4 . 6 % , to set a record closing high at a $ 4 . 57 . [SEP]










    






INFO:tensorflow:input_ids: 101 2105 6021 19481 13938 2102 1010 21628 6661 2020 2039 2539 16653 1010 2030 1018 1012 1018 1003 1010 2012 1037 1002 1018 1012 5179 1010 2383 3041 2275 1037 2501 2152 1997 1037 1002 1018 1012 5401 1012 102 21628 6661 5598 2322 16653 1010 2030 1018 1012 1020 1003 1010 2000 2275 1037 2501 5494 2152 2012 1037 1002 1018 1012 5401 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 17:58:18.761564 140342252849024 run_classifier.py:465] input_ids: 101 2105 6021 19481 13938 2102 1010 21628 6661 2020 2039 2539 16653 1010 2030 1018 1012 1018 1003 1010 2012 1037 1002 1018 1012 5179 1010 2383 3041 2275 1037 2501 2152 1997 1037 1002 1018 1012 5401 1012 102 21628 6661 5598 2322 16653 1010 2030 1018 1012 1020 1003 1010 2000 2275 1037 2501 5494 2152 2012 1037 1002 1018 1012 5401 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 17:58:18.764492 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 17:58:18.765792 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 0 (id = 0)










    






I0328 17:58:18.768823 140342252849024 run_classifier.py:468] label: 0 (id = 0)










    






INFO:tensorflow:*** Example ***










    






I0328 17:58:18.773404 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: train-5










    






I0328 17:58:18.776632 140342252849024 run_classifier.py:462] guid: train-5










    






INFO:tensorflow:tokens: [CLS] the stock rose $ 2 . 11 , or about 11 percent , to close friday at $ 21 . 51 on the new york stock exchange . [SEP] pg & e corp . shares jumped $ 1 . 63 or 8 percent to $ 21 . 03 on the new york stock exchange on friday . [SEP]










    






I0328 17:58:18.779201 140342252849024 run_classifier.py:464] tokens: [CLS] the stock rose $ 2 . 11 , or about 11 percent , to close friday at $ 21 . 51 on the new york stock exchange . [SEP] pg & e corp . shares jumped $ 1 . 63 or 8 percent to $ 21 . 03 on the new york stock exchange on friday . [SEP]










    






INFO:tensorflow:input_ids: 101 1996 4518 3123 1002 1016 1012 2340 1010 2030 2055 2340 3867 1010 2000 2485 5958 2012 1002 2538 1012 4868 2006 1996 2047 2259 4518 3863 1012 102 18720 1004 1041 13058 1012 6661 5598 1002 1015 1012 6191 2030 1022 3867 2000 1002 2538 1012 6021 2006 1996 2047 2259 4518 3863 2006 5958 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 17:58:18.781990 140342252849024 run_classifier.py:465] input_ids: 101 1996 4518 3123 1002 1016 1012 2340 1010 2030 2055 2340 3867 1010 2000 2485 5958 2012 1002 2538 1012 4868 2006 1996 2047 2259 4518 3863 1012 102 18720 1004 1041 13058 1012 6661 5598 1002 1015 1012 6191 2030 1022 3867 2000 1002 2538 1012 6021 2006 1996 2047 2259 4518 3863 2006 5958 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 17:58:18.784460 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 17:58:18.788443 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 1 (id = 1)










    






I0328 17:58:18.789558 140342252849024 run_classifier.py:468] label: 1 (id = 1)










    






***** Started training at 2019-03-28 17:58:22.644191 *****
  Num examples = 3668
  Batch size = 32
INFO:tensorflow:  Num steps = 343










    






I0328 17:58:22.645097 140342252849024 <ipython-input-7-108f6f632247>:9]   Num steps = 343










    






INFO:tensorflow:Querying Tensorflow master (grpc://10.114.88.226:8470) for TPU system metadata.










    






I0328 17:58:22.832043 140342252849024 tpu_system_metadata.py:59] Querying Tensorflow master (grpc://10.114.88.226:8470) for TPU system metadata.










    






INFO:tensorflow:Found TPU system:










    






I0328 17:58:22.856567 140342252849024 tpu_system_metadata.py:120] Found TPU system:










    






INFO:tensorflow:*** Num TPU Cores: 8










    






I0328 17:58:22.859071 140342252849024 tpu_system_metadata.py:121] *** Num TPU Cores: 8










    






INFO:tensorflow:*** Num TPU Workers: 1










    






I0328 17:58:22.864658 140342252849024 tpu_system_metadata.py:122] *** Num TPU Workers: 1










    






INFO:tensorflow:*** Num TPU Cores Per Worker: 8










    






I0328 17:58:22.870570 140342252849024 tpu_system_metadata.py:124] *** Num TPU Cores Per Worker: 8










    






INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, -1, 3020137555896628722)










    






I0328 17:58:22.873450 140342252849024 tpu_system_metadata.py:126] *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, -1, 3020137555896628722)










    






INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 17179869184, 15223542790104685402)










    






I0328 17:58:22.877541 140342252849024 tpu_system_metadata.py:126] *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 17179869184, 15223542790104685402)










    






INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 17179869184, 1843375914324221839)










    






I0328 17:58:22.880562 140342252849024 tpu_system_metadata.py:126] *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 17179869184, 1843375914324221839)










    






INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 17179869184, 9894822734449577370)










    






I0328 17:58:22.884891 140342252849024 tpu_system_metadata.py:126] *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 17179869184, 9894822734449577370)










    






INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 17179869184, 14380569740864114478)










    






I0328 17:58:22.889165 140342252849024 tpu_system_metadata.py:126] *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 17179869184, 14380569740864114478)










    






INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 17179869184, 17084474974895924727)










    






I0328 17:58:22.893801 140342252849024 tpu_system_metadata.py:126] *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 17179869184, 17084474974895924727)










    






INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 17179869184, 7110320513217349863)










    






I0328 17:58:22.897368 140342252849024 tpu_system_metadata.py:126] *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 17179869184, 7110320513217349863)










    






INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 17179869184, 11587941208496677740)










    






I0328 17:58:22.901369 140342252849024 tpu_system_metadata.py:126] *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 17179869184, 11587941208496677740)










    






INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 17179869184, 10553633026221058370)










    






I0328 17:58:22.906192 140342252849024 tpu_system_metadata.py:126] *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 17179869184, 10553633026221058370)










    






INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 17179869184, 9279467740370141958)










    






I0328 17:58:22.910486 140342252849024 tpu_system_metadata.py:126] *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 17179869184, 9279467740370141958)










    






INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 17179869184, 2228303855717980436)










    






I0328 17:58:22.916454 140342252849024 tpu_system_metadata.py:126] *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 17179869184, 2228303855717980436)










    






INFO:tensorflow:Calling model_fn.










    






I0328 17:58:22.957275 140342252849024 estimator.py:1111] Calling model_fn.










    






INFO:tensorflow:*** Features ***










    






I0328 17:58:25.292212 140342252849024 run_classifier_with_tfhub.py:94] *** Features ***










    






INFO:tensorflow:  name = input_ids, shape = (4, 128)










    






I0328 17:58:25.299383 140342252849024 run_classifier_with_tfhub.py:96]   name = input_ids, shape = (4, 128)










    






INFO:tensorflow:  name = input_mask, shape = (4, 128)










    






I0328 17:58:25.301375 140342252849024 run_classifier_with_tfhub.py:96]   name = input_mask, shape = (4, 128)










    






INFO:tensorflow:  name = label_ids, shape = (4,)










    






I0328 17:58:25.306400 140342252849024 run_classifier_with_tfhub.py:96]   name = label_ids, shape = (4,)










    






INFO:tensorflow:  name = segment_ids, shape = (4, 128)










    






I0328 17:58:25.309485 140342252849024 run_classifier_with_tfhub.py:96]   name = segment_ids, shape = (4, 128)










    






ERROR:tensorflow:Operation of type Placeholder (module_apply_tokens/input_ids) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






E0328 17:58:50.165885 140342252849024 tpu.py:330] Operation of type Placeholder (module_apply_tokens/input_ids) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






ERROR:tensorflow:Operation of type Placeholder (module_apply_tokens/input_mask) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






E0328 17:58:50.168838 140342252849024 tpu.py:330] Operation of type Placeholder (module_apply_tokens/input_mask) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






ERROR:tensorflow:Operation of type Placeholder (module_apply_tokens/segment_ids) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






E0328 17:58:50.174988 140342252849024 tpu.py:330] Operation of type Placeholder (module_apply_tokens/segment_ids) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






ERROR:tensorflow:Operation of type Placeholder (module_apply_tokens/mlm_positions) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






E0328 17:58:50.179913 140342252849024 tpu.py:330] Operation of type Placeholder (module_apply_tokens/mlm_positions) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






INFO:tensorflow:Saver not created because there are no variables in the graph to restore










    






I0328 17:58:50.933571 140342252849024 saver.py:1483] Saver not created because there are no variables in the graph to restore










    






WARNING:tensorflow:From bert_repo/run_classifier_with_tfhub.py:72: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.










    






W0328 17:58:51.109370 140342252849024 deprecation.py:506] From bert_repo/run_classifier_with_tfhub.py:72: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.










    






WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/learning_rate_decay_v2.py:321: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.










    






W0328 17:58:51.178931 140342252849024 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/learning_rate_decay_v2.py:321: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.










    






WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.










    






W0328 17:58:51.287854 140342252849024 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_impl.py:110: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "










    






INFO:tensorflow:Create CheckpointSaverHook.










    






I0328 17:59:04.854490 140342252849024 basic_session_run_hooks.py:527] Create CheckpointSaverHook.










    






INFO:tensorflow:Done calling model_fn.










    






I0328 17:59:05.293682 140342252849024 estimator.py:1113] Done calling model_fn.










    






INFO:tensorflow:TPU job name worker










    






I0328 17:59:08.950665 140342252849024 tpu_estimator.py:447] TPU job name worker










    






INFO:tensorflow:Graph was finalized.










    






I0328 17:59:10.589298 140342252849024 monitored_session.py:222] Graph was finalized.










    






INFO:tensorflow:Running local_init_op.










    






I0328 17:59:20.763206 140342252849024 session_manager.py:491] Running local_init_op.










    






INFO:tensorflow:Done running local_init_op.










    






I0328 17:59:21.676876 140342252849024 session_manager.py:493] Done running local_init_op.










    






INFO:tensorflow:Saving checkpoints for 0 into gs://YOUR_BUCKET/bert-tfhub/models/MRPC/model.ckpt.










    






I0328 17:59:36.707026 140342252849024 basic_session_run_hooks.py:594] Saving checkpoints for 0 into gs://YOUR_BUCKET/bert-tfhub/models/MRPC/model.ckpt.










    






INFO:tensorflow:Initialized dataset iterators in 0 seconds










    






I0328 18:00:03.074676 140342252849024 util.py:51] Initialized dataset iterators in 0 seconds










    






INFO:tensorflow:Installing graceful shutdown hook.










    






I0328 18:00:03.078616 140342252849024 session_support.py:345] Installing graceful shutdown hook.










    






INFO:tensorflow:Creating heartbeat manager for ['/job:worker/replica:0/task:0/device:CPU:0']










    






I0328 18:00:03.099291 140342252849024 session_support.py:102] Creating heartbeat manager for ['/job:worker/replica:0/task:0/device:CPU:0']










    






INFO:tensorflow:Configuring worker heartbeat: shutdown_mode: WAIT_FOR_COORDINATOR











    






I0328 18:00:03.123145 140342252849024 session_support.py:130] Configuring worker heartbeat: shutdown_mode: WAIT_FOR_COORDINATOR











    






INFO:tensorflow:Init TPU system










    






I0328 18:00:03.140023 140342252849024 tpu_estimator.py:504] Init TPU system










    






INFO:tensorflow:Initialized TPU in 7 seconds










    






I0328 18:00:10.538449 140342252849024 tpu_estimator.py:510] Initialized TPU in 7 seconds










    






INFO:tensorflow:Starting infeed thread controller.










    






I0328 18:00:10.542649 140340707829504 tpu_estimator.py:463] Starting infeed thread controller.










    






INFO:tensorflow:Starting outfeed thread controller.










    






I0328 18:00:10.543798 140340699436800 tpu_estimator.py:482] Starting outfeed thread controller.










    






INFO:tensorflow:Enqueue next (343) batch(es) of data to infeed.










    






I0328 18:00:11.287537 140342252849024 tpu_estimator.py:536] Enqueue next (343) batch(es) of data to infeed.










    






INFO:tensorflow:Dequeue next (343) batch(es) of data from outfeed.










    






I0328 18:00:11.294393 140342252849024 tpu_estimator.py:540] Dequeue next (343) batch(es) of data from outfeed.










    






INFO:tensorflow:loss = 0.16517235, step = 343










    






I0328 18:01:13.881572 140342252849024 basic_session_run_hooks.py:249] loss = 0.16517235, step = 343










    






INFO:tensorflow:Saving checkpoints for 343 into gs://YOUR_BUCKET/bert-tfhub/models/MRPC/model.ckpt.










    






I0328 18:01:13.892630 140342252849024 basic_session_run_hooks.py:594] Saving checkpoints for 343 into gs://YOUR_BUCKET/bert-tfhub/models/MRPC/model.ckpt.










    






INFO:tensorflow:Stop infeed thread controller










    






I0328 18:01:36.015296 140342252849024 tpu_estimator.py:545] Stop infeed thread controller










    






INFO:tensorflow:Shutting down InfeedController thread.










    






I0328 18:01:36.022627 140342252849024 tpu_estimator.py:392] Shutting down InfeedController thread.










    






INFO:tensorflow:InfeedController received shutdown signal, stopping.










    






I0328 18:01:36.025938 140340707829504 tpu_estimator.py:387] InfeedController received shutdown signal, stopping.










    






INFO:tensorflow:Infeed thread finished, shutting down.










    






I0328 18:01:36.028549 140340707829504 tpu_estimator.py:479] Infeed thread finished, shutting down.










    






INFO:tensorflow:infeed marked as finished










    






I0328 18:01:36.032959 140342252849024 error_handling.py:93] infeed marked as finished










    






INFO:tensorflow:Stop output thread controller










    






I0328 18:01:36.035238 140342252849024 tpu_estimator.py:549] Stop output thread controller










    






INFO:tensorflow:Shutting down OutfeedController thread.










    






I0328 18:01:36.036949 140342252849024 tpu_estimator.py:392] Shutting down OutfeedController thread.










    






INFO:tensorflow:OutfeedController received shutdown signal, stopping.










    






I0328 18:01:36.040416 140340699436800 tpu_estimator.py:387] OutfeedController received shutdown signal, stopping.










    






INFO:tensorflow:Outfeed thread finished, shutting down.










    






I0328 18:01:36.042259 140340699436800 tpu_estimator.py:488] Outfeed thread finished, shutting down.










    






INFO:tensorflow:outfeed marked as finished










    






I0328 18:01:36.051629 140342252849024 error_handling.py:93] outfeed marked as finished










    






INFO:tensorflow:Shutdown TPU system.










    






I0328 18:01:36.054212 140342252849024 tpu_estimator.py:553] Shutdown TPU system.










    






INFO:tensorflow:Loss for final step: 0.16517235.










    






I0328 18:01:37.386233 140342252849024 estimator.py:359] Loss for final step: 0.16517235.










    






INFO:tensorflow:training_loop marked as finished










    






I0328 18:01:37.393169 140342252849024 error_handling.py:93] training_loop marked as finished










    






***** Finished training at 2019-03-28 18:01:37.396652 *****

















In [0]:



    


def model_eval(estimator):
  # Eval the model.
  eval_examples = processor.get_dev_examples(TASK_DATA_DIR)
  eval_features = run_classifier.convert_examples_to_features(
      eval_examples, label_list, MAX_SEQ_LENGTH, tokenizer)
  print('***** Started evaluation at {} *****'.format(datetime.datetime.now()))
  print('  Num examples = {}'.format(len(eval_examples)))
  print('  Batch size = {}'.format(EVAL_BATCH_SIZE))

  # Eval will be slightly WRONG on the TPU because it will truncate
  # the last batch.
  eval_steps = int(len(eval_examples) / EVAL_BATCH_SIZE)
  eval_input_fn = run_classifier.input_fn_builder(
      features=eval_features,
      seq_length=MAX_SEQ_LENGTH,
      is_training=False,
      drop_remainder=True)
  result = estimator.evaluate(input_fn=eval_input_fn, steps=eval_steps)
  print('***** Finished evaluation at {} *****'.format(datetime.datetime.now()))
  output_eval_file = os.path.join(OUTPUT_DIR, "eval_results.txt")
  with tf.gfile.GFile(output_eval_file, "w") as writer:
    print("***** Eval results *****")
    for key in sorted(result.keys()):
      print('  {} = {}'.format(key, str(result[key])))
      writer.write("%s = %s\n" % (key, str(result[key])))



    











In [10]:



    


model_eval(estimator_from_tfhub)



    


















    






INFO:tensorflow:Writing example 0 of 408










    






I0328 18:02:12.905318 140342252849024 run_classifier.py:774] Writing example 0 of 408










    






INFO:tensorflow:*** Example ***










    






I0328 18:02:12.910581 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: dev-1










    






I0328 18:02:12.913884 140342252849024 run_classifier.py:462] guid: dev-1










    






INFO:tensorflow:tokens: [CLS] he said the foods ##er ##vic ##e pie business doesn ' t fit the company ' s long - term growth strategy . [SEP] " the foods ##er ##vic ##e pie business does not fit our long - term growth strategy . [SEP]










    






I0328 18:02:12.920161 140342252849024 run_classifier.py:464] tokens: [CLS] he said the foods ##er ##vic ##e pie business doesn ' t fit the company ' s long - term growth strategy . [SEP] " the foods ##er ##vic ##e pie business does not fit our long - term growth strategy . [SEP]










    






INFO:tensorflow:input_ids: 101 2002 2056 1996 9440 2121 7903 2063 11345 2449 2987 1005 1056 4906 1996 2194 1005 1055 2146 1011 2744 3930 5656 1012 102 1000 1996 9440 2121 7903 2063 11345 2449 2515 2025 4906 2256 2146 1011 2744 3930 5656 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:02:12.922273 140342252849024 run_classifier.py:465] input_ids: 101 2002 2056 1996 9440 2121 7903 2063 11345 2449 2987 1005 1056 4906 1996 2194 1005 1055 2146 1011 2744 3930 5656 1012 102 1000 1996 9440 2121 7903 2063 11345 2449 2515 2025 4906 2256 2146 1011 2744 3930 5656 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:02:12.927431 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:02:12.930144 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 1 (id = 1)










    






I0328 18:02:12.935292 140342252849024 run_classifier.py:468] label: 1 (id = 1)










    






INFO:tensorflow:*** Example ***










    






I0328 18:02:12.942759 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: dev-2










    






I0328 18:02:12.945500 140342252849024 run_classifier.py:462] guid: dev-2










    






INFO:tensorflow:tokens: [CLS] magna ##relli said ra ##cic ##ot hated the iraqi regime and looked forward to using his long years of training in the war . [SEP] his wife said he was " 100 percent behind george bush " and looked forward to using his years of training in the war . [SEP]










    






I0328 18:02:12.948870 140342252849024 run_classifier.py:464] tokens: [CLS] magna ##relli said ra ##cic ##ot hated the iraqi regime and looked forward to using his long years of training in the war . [SEP] his wife said he was " 100 percent behind george bush " and looked forward to using his years of training in the war . [SEP]










    






INFO:tensorflow:input_ids: 101 20201 22948 2056 10958 19053 4140 6283 1996 8956 6939 1998 2246 2830 2000 2478 2010 2146 2086 1997 2731 1999 1996 2162 1012 102 2010 2564 2056 2002 2001 1000 2531 3867 2369 2577 5747 1000 1998 2246 2830 2000 2478 2010 2086 1997 2731 1999 1996 2162 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:02:12.951623 140342252849024 run_classifier.py:465] input_ids: 101 20201 22948 2056 10958 19053 4140 6283 1996 8956 6939 1998 2246 2830 2000 2478 2010 2146 2086 1997 2731 1999 1996 2162 1012 102 2010 2564 2056 2002 2001 1000 2531 3867 2369 2577 5747 1000 1998 2246 2830 2000 2478 2010 2086 1997 2731 1999 1996 2162 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:02:12.953913 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:02:12.957401 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 0 (id = 0)










    






I0328 18:02:12.962664 140342252849024 run_classifier.py:468] label: 0 (id = 0)










    






INFO:tensorflow:*** Example ***










    






I0328 18:02:12.968867 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: dev-3










    






I0328 18:02:12.974296 140342252849024 run_classifier.py:462] guid: dev-3










    






INFO:tensorflow:tokens: [CLS] the dollar was at 116 . 92 yen against the yen , flat on the session , and at 1 . 289 ##1 against the swiss fran ##c , also flat . [SEP] the dollar was at 116 . 78 yen jp ##y = , virtually flat on the session , and at 1 . 287 ##1 against the swiss fran ##c ch ##f = , down 0 . 1 percent . [SEP]










    






I0328 18:02:12.976569 140342252849024 run_classifier.py:464] tokens: [CLS] the dollar was at 116 . 92 yen against the yen , flat on the session , and at 1 . 289 ##1 against the swiss fran ##c , also flat . [SEP] the dollar was at 116 . 78 yen jp ##y = , virtually flat on the session , and at 1 . 287 ##1 against the swiss fran ##c ch ##f = , down 0 . 1 percent . [SEP]










    






INFO:tensorflow:input_ids: 101 1996 7922 2001 2012 12904 1012 6227 18371 2114 1996 18371 1010 4257 2006 1996 5219 1010 1998 2012 1015 1012 27054 2487 2114 1996 5364 23151 2278 1010 2036 4257 1012 102 1996 7922 2001 2012 12904 1012 6275 18371 16545 2100 1027 1010 8990 4257 2006 1996 5219 1010 1998 2012 1015 1012 23090 2487 2114 1996 5364 23151 2278 10381 2546 1027 1010 2091 1014 1012 1015 3867 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:02:12.979778 140342252849024 run_classifier.py:465] input_ids: 101 1996 7922 2001 2012 12904 1012 6227 18371 2114 1996 18371 1010 4257 2006 1996 5219 1010 1998 2012 1015 1012 27054 2487 2114 1996 5364 23151 2278 1010 2036 4257 1012 102 1996 7922 2001 2012 12904 1012 6275 18371 16545 2100 1027 1010 8990 4257 2006 1996 5219 1010 1998 2012 1015 1012 23090 2487 2114 1996 5364 23151 2278 10381 2546 1027 1010 2091 1014 1012 1015 3867 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:02:12.982804 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:02:12.984977 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 0 (id = 0)










    






I0328 18:02:12.988495 140342252849024 run_classifier.py:468] label: 0 (id = 0)










    






INFO:tensorflow:*** Example ***










    






I0328 18:02:12.994262 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: dev-4










    






I0328 18:02:12.998415 140342252849024 run_classifier.py:462] guid: dev-4










    






INFO:tensorflow:tokens: [CLS] the afl - ci ##o is waiting until october to decide if it will end ##ors ##e a candidate . [SEP] the afl - ci ##o announced wednesday that it will decide in october whether to end ##ors ##e a candidate before the primaries . [SEP]










    






I0328 18:02:13.002157 140342252849024 run_classifier.py:464] tokens: [CLS] the afl - ci ##o is waiting until october to decide if it will end ##ors ##e a candidate . [SEP] the afl - ci ##o announced wednesday that it will decide in october whether to end ##ors ##e a candidate before the primaries . [SEP]










    






INFO:tensorflow:input_ids: 101 1996 10028 1011 25022 2080 2003 3403 2127 2255 2000 5630 2065 2009 2097 2203 5668 2063 1037 4018 1012 102 1996 10028 1011 25022 2080 2623 9317 2008 2009 2097 5630 1999 2255 3251 2000 2203 5668 2063 1037 4018 2077 1996 27419 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:02:13.005905 140342252849024 run_classifier.py:465] input_ids: 101 1996 10028 1011 25022 2080 2003 3403 2127 2255 2000 5630 2065 2009 2097 2203 5668 2063 1037 4018 1012 102 1996 10028 1011 25022 2080 2623 9317 2008 2009 2097 5630 1999 2255 3251 2000 2203 5668 2063 1037 4018 2077 1996 27419 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:02:13.009269 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:02:13.011682 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 1 (id = 1)










    






I0328 18:02:13.014828 140342252849024 run_classifier.py:468] label: 1 (id = 1)










    






INFO:tensorflow:*** Example ***










    






I0328 18:02:13.019813 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: dev-5










    






I0328 18:02:13.022576 140342252849024 run_classifier.py:462] guid: dev-5










    






INFO:tensorflow:tokens: [CLS] no dates have been set for the civil or the criminal trial . [SEP] no dates have been set for the criminal or civil cases , but shan ##ley has pleaded not guilty . [SEP]










    






I0328 18:02:13.024523 140342252849024 run_classifier.py:464] tokens: [CLS] no dates have been set for the civil or the criminal trial . [SEP] no dates have been set for the criminal or civil cases , but shan ##ley has pleaded not guilty . [SEP]










    






INFO:tensorflow:input_ids: 101 2053 5246 2031 2042 2275 2005 1996 2942 2030 1996 4735 3979 1012 102 2053 5246 2031 2042 2275 2005 1996 4735 2030 2942 3572 1010 2021 17137 3051 2038 12254 2025 5905 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:02:13.026606 140342252849024 run_classifier.py:465] input_ids: 101 2053 5246 2031 2042 2275 2005 1996 2942 2030 1996 4735 3979 1012 102 2053 5246 2031 2042 2275 2005 1996 4735 2030 2942 3572 1010 2021 17137 3051 2038 12254 2025 5905 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:02:13.028499 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:02:13.030654 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 0 (id = 0)










    






I0328 18:02:13.032691 140342252849024 run_classifier.py:468] label: 0 (id = 0)










    






***** Started evaluation at 2019-03-28 18:02:13.489809 *****
  Num examples = 408
  Batch size = 8
INFO:tensorflow:Calling model_fn.










    






I0328 18:02:13.817483 140342252849024 estimator.py:1111] Calling model_fn.










    






INFO:tensorflow:*** Features ***










    






I0328 18:02:14.140701 140342252849024 run_classifier_with_tfhub.py:94] *** Features ***










    






INFO:tensorflow:  name = input_ids, shape = (1, 128)










    






I0328 18:02:14.145235 140342252849024 run_classifier_with_tfhub.py:96]   name = input_ids, shape = (1, 128)










    






INFO:tensorflow:  name = input_mask, shape = (1, 128)










    






I0328 18:02:14.149084 140342252849024 run_classifier_with_tfhub.py:96]   name = input_mask, shape = (1, 128)










    






INFO:tensorflow:  name = label_ids, shape = (1,)










    






I0328 18:02:14.151639 140342252849024 run_classifier_with_tfhub.py:96]   name = label_ids, shape = (1,)










    






INFO:tensorflow:  name = segment_ids, shape = (1, 128)










    






I0328 18:02:14.155844 140342252849024 run_classifier_with_tfhub.py:96]   name = segment_ids, shape = (1, 128)










    






ERROR:tensorflow:Operation of type Placeholder (module_apply_tokens/input_ids) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






E0328 18:02:20.953701 140342252849024 tpu.py:330] Operation of type Placeholder (module_apply_tokens/input_ids) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






ERROR:tensorflow:Operation of type Placeholder (module_apply_tokens/input_mask) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






E0328 18:02:20.956325 140342252849024 tpu.py:330] Operation of type Placeholder (module_apply_tokens/input_mask) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






ERROR:tensorflow:Operation of type Placeholder (module_apply_tokens/segment_ids) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






E0328 18:02:20.965242 140342252849024 tpu.py:330] Operation of type Placeholder (module_apply_tokens/segment_ids) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






ERROR:tensorflow:Operation of type Placeholder (module_apply_tokens/mlm_positions) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






E0328 18:02:20.970981 140342252849024 tpu.py:330] Operation of type Placeholder (module_apply_tokens/mlm_positions) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






INFO:tensorflow:Saver not created because there are no variables in the graph to restore










    






I0328 18:02:21.605878 140342252849024 saver.py:1483] Saver not created because there are no variables in the graph to restore










    






WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/metrics_impl.py:455: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.










    






W0328 18:02:22.251052 140342252849024 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/metrics_impl.py:455: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.










    






INFO:tensorflow:Done calling model_fn.










    






I0328 18:02:22.294389 140342252849024 estimator.py:1113] Done calling model_fn.










    






INFO:tensorflow:Starting evaluation at 2019-03-28T18:02:22Z










    






I0328 18:02:22.319931 140342252849024 evaluation.py:257] Starting evaluation at 2019-03-28T18:02:22Z










    






INFO:tensorflow:TPU job name worker










    






I0328 18:02:22.322047 140342252849024 tpu_estimator.py:447] TPU job name worker










    






INFO:tensorflow:Graph was finalized.










    






I0328 18:02:22.942793 140342252849024 monitored_session.py:222] Graph was finalized.










    






WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.










    






W0328 18:02:22.945611 140342252849024 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.










    






INFO:tensorflow:Restoring parameters from gs://YOUR_BUCKET/bert-tfhub/models/MRPC/model.ckpt-343










    






I0328 18:02:23.021796 140342252849024 saver.py:1270] Restoring parameters from gs://YOUR_BUCKET/bert-tfhub/models/MRPC/model.ckpt-343










    






INFO:tensorflow:Running local_init_op.










    






I0328 18:02:31.270408 140342252849024 session_manager.py:491] Running local_init_op.










    






INFO:tensorflow:Done running local_init_op.










    






I0328 18:02:31.560442 140342252849024 session_manager.py:493] Done running local_init_op.










    






INFO:tensorflow:Init TPU system










    






I0328 18:02:32.181740 140342252849024 tpu_estimator.py:504] Init TPU system










    






INFO:tensorflow:Initialized TPU in 7 seconds










    






I0328 18:02:39.660038 140342252849024 tpu_estimator.py:510] Initialized TPU in 7 seconds










    






INFO:tensorflow:Starting infeed thread controller.










    






I0328 18:02:39.666680 140340985972480 tpu_estimator.py:463] Starting infeed thread controller.










    






INFO:tensorflow:Starting outfeed thread controller.










    






I0328 18:02:39.674470 140340977579776 tpu_estimator.py:482] Starting outfeed thread controller.










    






INFO:tensorflow:Initialized dataset iterators in 0 seconds










    






I0328 18:02:39.964791 140342252849024 util.py:51] Initialized dataset iterators in 0 seconds










    






INFO:tensorflow:Enqueue next (51) batch(es) of data to infeed.










    






I0328 18:02:40.239301 140342252849024 tpu_estimator.py:536] Enqueue next (51) batch(es) of data to infeed.










    






INFO:tensorflow:Dequeue next (51) batch(es) of data from outfeed.










    






I0328 18:02:40.241971 140342252849024 tpu_estimator.py:540] Dequeue next (51) batch(es) of data from outfeed.










    






INFO:tensorflow:Evaluation [51/51]










    






I0328 18:02:44.945057 140342252849024 evaluation.py:169] Evaluation [51/51]










    






INFO:tensorflow:Stop infeed thread controller










    






I0328 18:02:44.949223 140342252849024 tpu_estimator.py:545] Stop infeed thread controller










    






INFO:tensorflow:Shutting down InfeedController thread.










    






I0328 18:02:44.955980 140342252849024 tpu_estimator.py:392] Shutting down InfeedController thread.










    






INFO:tensorflow:InfeedController received shutdown signal, stopping.










    






I0328 18:02:44.960723 140340985972480 tpu_estimator.py:387] InfeedController received shutdown signal, stopping.










    






INFO:tensorflow:Infeed thread finished, shutting down.










    






I0328 18:02:44.964605 140340985972480 tpu_estimator.py:479] Infeed thread finished, shutting down.










    






INFO:tensorflow:infeed marked as finished










    






I0328 18:02:44.970584 140342252849024 error_handling.py:93] infeed marked as finished










    






INFO:tensorflow:Stop output thread controller










    






I0328 18:02:44.984945 140342252849024 tpu_estimator.py:549] Stop output thread controller










    






INFO:tensorflow:Shutting down OutfeedController thread.










    






I0328 18:02:44.987492 140342252849024 tpu_estimator.py:392] Shutting down OutfeedController thread.










    






INFO:tensorflow:OutfeedController received shutdown signal, stopping.










    






I0328 18:02:45.001528 140340977579776 tpu_estimator.py:387] OutfeedController received shutdown signal, stopping.










    






INFO:tensorflow:Outfeed thread finished, shutting down.










    






I0328 18:02:45.004570 140340977579776 tpu_estimator.py:488] Outfeed thread finished, shutting down.










    






INFO:tensorflow:outfeed marked as finished










    






I0328 18:02:45.007973 140342252849024 error_handling.py:93] outfeed marked as finished










    






INFO:tensorflow:Shutdown TPU system.










    






I0328 18:02:45.010710 140342252849024 tpu_estimator.py:553] Shutdown TPU system.










    






INFO:tensorflow:Finished evaluation at 2019-03-28-18:02:45










    






I0328 18:02:45.812334 140342252849024 evaluation.py:277] Finished evaluation at 2019-03-28-18:02:45










    






INFO:tensorflow:Saving dict for global step 343: eval_accuracy = 0.86764705, eval_loss = 0.6635489, global_step = 343, loss = 0.5285201










    






I0328 18:02:45.815102 140342252849024 estimator.py:1979] Saving dict for global step 343: eval_accuracy = 0.86764705, eval_loss = 0.6635489, global_step = 343, loss = 0.5285201










    






INFO:tensorflow:Saving 'checkpoint_path' summary for global step 343: gs://YOUR_BUCKET/bert-tfhub/models/MRPC/model.ckpt-343










    






I0328 18:02:49.287359 140342252849024 estimator.py:2039] Saving 'checkpoint_path' summary for global step 343: gs://YOUR_BUCKET/bert-tfhub/models/MRPC/model.ckpt-343










    






INFO:tensorflow:evaluation_loop marked as finished










    






I0328 18:02:49.619682 140342252849024 error_handling.py:93] evaluation_loop marked as finished










    






***** Finished evaluation at 2019-03-28 18:02:49.621812 *****
***** Eval results *****
  eval_accuracy = 0.86764705
  eval_loss = 0.6635489
  global_step = 343
  loss = 0.5285201

















In [0]:



    


def model_predict(estimator):
  # Make predictions on a subset of eval examples
  prediction_examples = processor.get_dev_examples(TASK_DATA_DIR)[:PREDICT_BATCH_SIZE]
  input_features = run_classifier.convert_examples_to_features(prediction_examples, label_list, MAX_SEQ_LENGTH, tokenizer)
  predict_input_fn = run_classifier.input_fn_builder(features=input_features, seq_length=MAX_SEQ_LENGTH, is_training=False, drop_remainder=True)
  predictions = estimator.predict(predict_input_fn)

  for example, prediction in zip(prediction_examples, predictions):
    print('text_a: %s\ntext_b: %s\nlabel:%s\nprediction:%s\n' % (example.text_a, example.text_b, str(example.label), prediction['probabilities']))



    











In [13]:



    


model_predict(estimator_from_tfhub)



    


















    






INFO:tensorflow:Writing example 0 of 8










    






I0328 18:03:41.113085 140342252849024 run_classifier.py:774] Writing example 0 of 8










    






INFO:tensorflow:*** Example ***










    






I0328 18:03:41.121951 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: dev-1










    






I0328 18:03:41.123759 140342252849024 run_classifier.py:462] guid: dev-1










    






INFO:tensorflow:tokens: [CLS] he said the foods ##er ##vic ##e pie business doesn ' t fit the company ' s long - term growth strategy . [SEP] " the foods ##er ##vic ##e pie business does not fit our long - term growth strategy . [SEP]










    






I0328 18:03:41.126170 140342252849024 run_classifier.py:464] tokens: [CLS] he said the foods ##er ##vic ##e pie business doesn ' t fit the company ' s long - term growth strategy . [SEP] " the foods ##er ##vic ##e pie business does not fit our long - term growth strategy . [SEP]










    






INFO:tensorflow:input_ids: 101 2002 2056 1996 9440 2121 7903 2063 11345 2449 2987 1005 1056 4906 1996 2194 1005 1055 2146 1011 2744 3930 5656 1012 102 1000 1996 9440 2121 7903 2063 11345 2449 2515 2025 4906 2256 2146 1011 2744 3930 5656 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:03:41.130502 140342252849024 run_classifier.py:465] input_ids: 101 2002 2056 1996 9440 2121 7903 2063 11345 2449 2987 1005 1056 4906 1996 2194 1005 1055 2146 1011 2744 3930 5656 1012 102 1000 1996 9440 2121 7903 2063 11345 2449 2515 2025 4906 2256 2146 1011 2744 3930 5656 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:03:41.135733 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:03:41.148722 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 1 (id = 1)










    






I0328 18:03:41.152383 140342252849024 run_classifier.py:468] label: 1 (id = 1)










    






INFO:tensorflow:*** Example ***










    






I0328 18:03:41.161146 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: dev-2










    






I0328 18:03:41.165202 140342252849024 run_classifier.py:462] guid: dev-2










    






INFO:tensorflow:tokens: [CLS] magna ##relli said ra ##cic ##ot hated the iraqi regime and looked forward to using his long years of training in the war . [SEP] his wife said he was " 100 percent behind george bush " and looked forward to using his years of training in the war . [SEP]










    






I0328 18:03:41.167545 140342252849024 run_classifier.py:464] tokens: [CLS] magna ##relli said ra ##cic ##ot hated the iraqi regime and looked forward to using his long years of training in the war . [SEP] his wife said he was " 100 percent behind george bush " and looked forward to using his years of training in the war . [SEP]










    






INFO:tensorflow:input_ids: 101 20201 22948 2056 10958 19053 4140 6283 1996 8956 6939 1998 2246 2830 2000 2478 2010 2146 2086 1997 2731 1999 1996 2162 1012 102 2010 2564 2056 2002 2001 1000 2531 3867 2369 2577 5747 1000 1998 2246 2830 2000 2478 2010 2086 1997 2731 1999 1996 2162 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:03:41.171585 140342252849024 run_classifier.py:465] input_ids: 101 20201 22948 2056 10958 19053 4140 6283 1996 8956 6939 1998 2246 2830 2000 2478 2010 2146 2086 1997 2731 1999 1996 2162 1012 102 2010 2564 2056 2002 2001 1000 2531 3867 2369 2577 5747 1000 1998 2246 2830 2000 2478 2010 2086 1997 2731 1999 1996 2162 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:03:41.174519 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:03:41.177199 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 0 (id = 0)










    






I0328 18:03:41.179187 140342252849024 run_classifier.py:468] label: 0 (id = 0)










    






INFO:tensorflow:*** Example ***










    






I0328 18:03:41.190357 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: dev-3










    






I0328 18:03:41.192526 140342252849024 run_classifier.py:462] guid: dev-3










    






INFO:tensorflow:tokens: [CLS] the dollar was at 116 . 92 yen against the yen , flat on the session , and at 1 . 289 ##1 against the swiss fran ##c , also flat . [SEP] the dollar was at 116 . 78 yen jp ##y = , virtually flat on the session , and at 1 . 287 ##1 against the swiss fran ##c ch ##f = , down 0 . 1 percent . [SEP]










    






I0328 18:03:41.197390 140342252849024 run_classifier.py:464] tokens: [CLS] the dollar was at 116 . 92 yen against the yen , flat on the session , and at 1 . 289 ##1 against the swiss fran ##c , also flat . [SEP] the dollar was at 116 . 78 yen jp ##y = , virtually flat on the session , and at 1 . 287 ##1 against the swiss fran ##c ch ##f = , down 0 . 1 percent . [SEP]










    






INFO:tensorflow:input_ids: 101 1996 7922 2001 2012 12904 1012 6227 18371 2114 1996 18371 1010 4257 2006 1996 5219 1010 1998 2012 1015 1012 27054 2487 2114 1996 5364 23151 2278 1010 2036 4257 1012 102 1996 7922 2001 2012 12904 1012 6275 18371 16545 2100 1027 1010 8990 4257 2006 1996 5219 1010 1998 2012 1015 1012 23090 2487 2114 1996 5364 23151 2278 10381 2546 1027 1010 2091 1014 1012 1015 3867 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:03:41.202018 140342252849024 run_classifier.py:465] input_ids: 101 1996 7922 2001 2012 12904 1012 6227 18371 2114 1996 18371 1010 4257 2006 1996 5219 1010 1998 2012 1015 1012 27054 2487 2114 1996 5364 23151 2278 1010 2036 4257 1012 102 1996 7922 2001 2012 12904 1012 6275 18371 16545 2100 1027 1010 8990 4257 2006 1996 5219 1010 1998 2012 1015 1012 23090 2487 2114 1996 5364 23151 2278 10381 2546 1027 1010 2091 1014 1012 1015 3867 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:03:41.206455 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:03:41.210074 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 0 (id = 0)










    






I0328 18:03:41.213917 140342252849024 run_classifier.py:468] label: 0 (id = 0)










    






INFO:tensorflow:*** Example ***










    






I0328 18:03:41.218997 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: dev-4










    






I0328 18:03:41.224253 140342252849024 run_classifier.py:462] guid: dev-4










    






INFO:tensorflow:tokens: [CLS] the afl - ci ##o is waiting until october to decide if it will end ##ors ##e a candidate . [SEP] the afl - ci ##o announced wednesday that it will decide in october whether to end ##ors ##e a candidate before the primaries . [SEP]










    






I0328 18:03:41.227671 140342252849024 run_classifier.py:464] tokens: [CLS] the afl - ci ##o is waiting until october to decide if it will end ##ors ##e a candidate . [SEP] the afl - ci ##o announced wednesday that it will decide in october whether to end ##ors ##e a candidate before the primaries . [SEP]










    






INFO:tensorflow:input_ids: 101 1996 10028 1011 25022 2080 2003 3403 2127 2255 2000 5630 2065 2009 2097 2203 5668 2063 1037 4018 1012 102 1996 10028 1011 25022 2080 2623 9317 2008 2009 2097 5630 1999 2255 3251 2000 2203 5668 2063 1037 4018 2077 1996 27419 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:03:41.231556 140342252849024 run_classifier.py:465] input_ids: 101 1996 10028 1011 25022 2080 2003 3403 2127 2255 2000 5630 2065 2009 2097 2203 5668 2063 1037 4018 1012 102 1996 10028 1011 25022 2080 2623 9317 2008 2009 2097 5630 1999 2255 3251 2000 2203 5668 2063 1037 4018 2077 1996 27419 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:03:41.235036 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:03:41.238193 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 1 (id = 1)










    






I0328 18:03:41.241272 140342252849024 run_classifier.py:468] label: 1 (id = 1)










    






INFO:tensorflow:*** Example ***










    






I0328 18:03:41.245405 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: dev-5










    






I0328 18:03:41.248059 140342252849024 run_classifier.py:462] guid: dev-5










    






INFO:tensorflow:tokens: [CLS] no dates have been set for the civil or the criminal trial . [SEP] no dates have been set for the criminal or civil cases , but shan ##ley has pleaded not guilty . [SEP]










    






I0328 18:03:41.251023 140342252849024 run_classifier.py:464] tokens: [CLS] no dates have been set for the civil or the criminal trial . [SEP] no dates have been set for the criminal or civil cases , but shan ##ley has pleaded not guilty . [SEP]










    






INFO:tensorflow:input_ids: 101 2053 5246 2031 2042 2275 2005 1996 2942 2030 1996 4735 3979 1012 102 2053 5246 2031 2042 2275 2005 1996 4735 2030 2942 3572 1010 2021 17137 3051 2038 12254 2025 5905 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:03:41.254218 140342252849024 run_classifier.py:465] input_ids: 101 2053 5246 2031 2042 2275 2005 1996 2942 2030 1996 4735 3979 1012 102 2053 5246 2031 2042 2275 2005 1996 4735 2030 2942 3572 1010 2021 17137 3051 2038 12254 2025 5905 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:03:41.257383 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:03:41.260523 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 0 (id = 0)










    






I0328 18:03:41.263534 140342252849024 run_classifier.py:468] label: 0 (id = 0)










    






INFO:tensorflow:Calling model_fn.










    






I0328 18:03:41.431937 140342252849024 estimator.py:1111] Calling model_fn.










    






INFO:tensorflow:*** Features ***










    






I0328 18:03:41.640449 140342252849024 run_classifier_with_tfhub.py:94] *** Features ***










    






INFO:tensorflow:  name = input_ids, shape = (1, 128)










    






I0328 18:03:41.642913 140342252849024 run_classifier_with_tfhub.py:96]   name = input_ids, shape = (1, 128)










    






INFO:tensorflow:  name = input_mask, shape = (1, 128)










    






I0328 18:03:41.645038 140342252849024 run_classifier_with_tfhub.py:96]   name = input_mask, shape = (1, 128)










    






INFO:tensorflow:  name = label_ids, shape = (1,)










    






I0328 18:03:41.647103 140342252849024 run_classifier_with_tfhub.py:96]   name = label_ids, shape = (1,)










    






INFO:tensorflow:  name = segment_ids, shape = (1, 128)










    






I0328 18:03:41.649264 140342252849024 run_classifier_with_tfhub.py:96]   name = segment_ids, shape = (1, 128)










    






ERROR:tensorflow:Operation of type Placeholder (module_apply_tokens/input_ids) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






E0328 18:03:47.668058 140342252849024 tpu.py:330] Operation of type Placeholder (module_apply_tokens/input_ids) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






ERROR:tensorflow:Operation of type Placeholder (module_apply_tokens/input_mask) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






E0328 18:03:47.671020 140342252849024 tpu.py:330] Operation of type Placeholder (module_apply_tokens/input_mask) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






ERROR:tensorflow:Operation of type Placeholder (module_apply_tokens/segment_ids) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






E0328 18:03:47.679236 140342252849024 tpu.py:330] Operation of type Placeholder (module_apply_tokens/segment_ids) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






ERROR:tensorflow:Operation of type Placeholder (module_apply_tokens/mlm_positions) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






E0328 18:03:47.682481 140342252849024 tpu.py:330] Operation of type Placeholder (module_apply_tokens/mlm_positions) is not supported on the TPU. Execution will fail if this op is used in the graph. 










    






INFO:tensorflow:Saver not created because there are no variables in the graph to restore










    






I0328 18:03:48.326836 140342252849024 saver.py:1483] Saver not created because there are no variables in the graph to restore










    






INFO:tensorflow:Done calling model_fn.










    






I0328 18:03:48.994716 140342252849024 estimator.py:1113] Done calling model_fn.










    






INFO:tensorflow:TPU job name worker










    






I0328 18:03:49.005714 140342252849024 tpu_estimator.py:447] TPU job name worker










    






INFO:tensorflow:Graph was finalized.










    






I0328 18:03:49.605780 140342252849024 monitored_session.py:222] Graph was finalized.










    






INFO:tensorflow:Restoring parameters from gs://YOUR_BUCKET/bert-tfhub/models/MRPC/model.ckpt-343










    






I0328 18:03:49.689134 140342252849024 saver.py:1270] Restoring parameters from gs://YOUR_BUCKET/bert-tfhub/models/MRPC/model.ckpt-343










    






INFO:tensorflow:Running local_init_op.










    






I0328 18:03:51.591142 140342252849024 session_manager.py:491] Running local_init_op.










    






INFO:tensorflow:Done running local_init_op.










    






I0328 18:03:51.864667 140342252849024 session_manager.py:493] Done running local_init_op.










    






INFO:tensorflow:Init TPU system










    






I0328 18:03:52.487024 140342252849024 tpu_estimator.py:504] Init TPU system










    






INFO:tensorflow:Initialized TPU in 10 seconds










    






I0328 18:04:02.875528 140342252849024 tpu_estimator.py:510] Initialized TPU in 10 seconds










    






INFO:tensorflow:Starting infeed thread controller.










    






I0328 18:04:02.879256 140340944819968 tpu_estimator.py:463] Starting infeed thread controller.










    






INFO:tensorflow:Starting outfeed thread controller.










    






I0328 18:04:02.881857 140340926732032 tpu_estimator.py:482] Starting outfeed thread controller.










    






INFO:tensorflow:Initialized dataset iterators in 0 seconds










    






I0328 18:04:03.217654 140342252849024 util.py:51] Initialized dataset iterators in 0 seconds










    






INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.










    






I0328 18:04:03.502789 140342252849024 tpu_estimator.py:536] Enqueue next (1) batch(es) of data to infeed.










    






INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.










    






I0328 18:04:03.505891 140342252849024 tpu_estimator.py:540] Dequeue next (1) batch(es) of data from outfeed.










    






text_a: He said the foodservice pie business doesn 't fit the company 's long-term growth strategy .
text_b: " The foodservice pie business does not fit our long-term growth strategy .
label:1
prediction:[0.00109948 0.99890053]

text_a: Magnarelli said Racicot hated the Iraqi regime and looked forward to using his long years of training in the war .
text_b: His wife said he was " 100 percent behind George Bush " and looked forward to using his years of training in the war .
label:0
prediction:[0.99797004 0.00202989]

text_a: The dollar was at 116.92 yen against the yen , flat on the session , and at 1.2891 against the Swiss franc , also flat .
text_b: The dollar was at 116.78 yen JPY = , virtually flat on the session , and at 1.2871 against the Swiss franc CHF = , down 0.1 percent .
label:0
prediction:[0.98365766 0.01634231]

text_a: The AFL-CIO is waiting until October to decide if it will endorse a candidate .
text_b: The AFL-CIO announced Wednesday that it will decide in October whether to endorse a candidate before the primaries .
label:1
prediction:[0.00116696 0.998833  ]

text_a: No dates have been set for the civil or the criminal trial .
text_b: No dates have been set for the criminal or civil cases , but Shanley has pleaded not guilty .
label:0
prediction:[0.9981337  0.00186629]

text_a: Wal-Mart said it would check all of its million-plus domestic workers to ensure they were legally employed .
text_b: It has also said it would review all of its domestic employees more than 1 million to ensure they have legal status .
label:1
prediction:[0.00122184 0.99877816]

text_a: While dioxin levels in the environment were up last year , they have dropped by 75 percent since the 1970s , said Caswell .
text_b: The Institute said dioxin levels in the environment have fallen by as much as 76 percent since the 1970s .
label:0
prediction:[0.00143446 0.9985656 ]

text_a: This integrates with Rational PurifyPlus and allows developers to work in supported versions of Java , Visual C # and Visual Basic .NET.
text_b: IBM said the Rational products were also integrated with Rational PurifyPlus , which allows developers to work in Java , Visual C # and VisualBasic .Net.
label:1
prediction:[0.00113128 0.9988687 ]

INFO:tensorflow:prediction_loop marked as finished

Fine-tune and run predictions on a pre-trained BERT model from checkpoints

Alternatively, you can also load pre-trained BERT models from saved checkpoints.





In [14]:



    


# Setup task specific model and TPU running config.
BERT_PRETRAINED_DIR = 'gs://cloud-tpu-checkpoints/bert/' + BERT_MODEL 
print('***** BERT pretrained directory: {} *****'.format(BERT_PRETRAINED_DIR))
!gsutil ls $BERT_PRETRAINED_DIR

CONFIG_FILE = os.path.join(BERT_PRETRAINED_DIR, 'bert_config.json')
INIT_CHECKPOINT = os.path.join(BERT_PRETRAINED_DIR, 'bert_model.ckpt')

model_fn = run_classifier.model_fn_builder(
  bert_config=modeling.BertConfig.from_json_file(CONFIG_FILE),
  num_labels=len(label_list),
  init_checkpoint=INIT_CHECKPOINT,
  learning_rate=LEARNING_RATE,
  num_train_steps=num_train_steps,
  num_warmup_steps=num_warmup_steps,
  use_tpu=True,
  use_one_hot_embeddings=True
)

OUTPUT_DIR = OUTPUT_DIR.replace('bert-tfhub', 'bert-checkpoints')
tf.gfile.MakeDirs(OUTPUT_DIR)

estimator_from_checkpoints = tf.contrib.tpu.TPUEstimator(
  use_tpu=True,
  model_fn=model_fn,
  config=get_run_config(OUTPUT_DIR),
  train_batch_size=TRAIN_BATCH_SIZE,
  eval_batch_size=EVAL_BATCH_SIZE,
  predict_batch_size=PREDICT_BATCH_SIZE,
)



    


















    






***** BERT pretrained directory: gs://cloud-tpu-checkpoints/bert/uncased_L-12_H-768_A-12 *****
gs://cloud-tpu-checkpoints/bert/uncased_L-12_H-768_A-12/bert_config.json
gs://cloud-tpu-checkpoints/bert/uncased_L-12_H-768_A-12/bert_model.ckpt.data-00000-of-00001
gs://cloud-tpu-checkpoints/bert/uncased_L-12_H-768_A-12/bert_model.ckpt.index
gs://cloud-tpu-checkpoints/bert/uncased_L-12_H-768_A-12/bert_model.ckpt.meta
gs://cloud-tpu-checkpoints/bert/uncased_L-12_H-768_A-12/checkpoint
gs://cloud-tpu-checkpoints/bert/uncased_L-12_H-768_A-12/vocab.txt
WARNING:tensorflow:Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7fa3b7fe3ae8>) includes params argument, but params are not passed to Estimator.










    






W0328 18:04:45.658415 140342252849024 estimator.py:1924] Estimator's model_fn (<function model_fn_builder.<locals>.model_fn at 0x7fa3b7fe3ae8>) includes params argument, but params are not passed to Estimator.










    






INFO:tensorflow:Using config: {'_model_dir': 'gs://YOUR_BUCKET/bert-checkpoints/models/MRPC', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
cluster_def {
  job {
    name: "worker"
    tasks {
      key: 0
      value: "10.114.88.226:8470"
    }
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fa3b79914e0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': 'grpc://10.114.88.226:8470', '_evaluation_master': 'grpc://10.114.88.226:8470', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None), '_cluster': <tensorflow.python.distribute.cluster_resolver.tpu_cluster_resolver.TPUClusterResolver object at 0x7fa3b0834748>}










    






I0328 18:04:45.668935 140342252849024 estimator.py:201] Using config: {'_model_dir': 'gs://YOUR_BUCKET/bert-checkpoints/models/MRPC', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
cluster_def {
  job {
    name: "worker"
    tasks {
      key: 0
      value: "10.114.88.226:8470"
    }
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fa3b79914e0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': 'grpc://10.114.88.226:8470', '_evaluation_master': 'grpc://10.114.88.226:8470', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None), '_cluster': <tensorflow.python.distribute.cluster_resolver.tpu_cluster_resolver.TPUClusterResolver object at 0x7fa3b0834748>}










    






INFO:tensorflow:_TPUContext: eval_on_tpu True










    






I0328 18:04:45.675429 140342252849024 tpu_context.py:202] _TPUContext: eval_on_tpu True

Now, you can repeat the training, evaluation, and prediction steps.





In [15]:



    


model_train(estimator_from_checkpoints)



    


















    






MRPC/CoLA on BERT base model normally takes about 2-3 minutes. Please wait...
INFO:tensorflow:Writing example 0 of 3668










    






I0328 18:04:55.535114 140342252849024 run_classifier.py:774] Writing example 0 of 3668










    






INFO:tensorflow:*** Example ***










    






I0328 18:04:55.540609 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: train-1










    






I0328 18:04:55.546897 140342252849024 run_classifier.py:462] guid: train-1










    






INFO:tensorflow:tokens: [CLS] am ##ro ##zi accused his brother , whom he called " the witness " , of deliberately di ##stor ##ting his evidence . [SEP] referring to him as only " the witness " , am ##ro ##zi accused his brother of deliberately di ##stor ##ting his evidence . [SEP]










    






I0328 18:04:55.550664 140342252849024 run_classifier.py:464] tokens: [CLS] am ##ro ##zi accused his brother , whom he called " the witness " , of deliberately di ##stor ##ting his evidence . [SEP] referring to him as only " the witness " , am ##ro ##zi accused his brother of deliberately di ##stor ##ting his evidence . [SEP]










    






INFO:tensorflow:input_ids: 101 2572 3217 5831 5496 2010 2567 1010 3183 2002 2170 1000 1996 7409 1000 1010 1997 9969 4487 23809 3436 2010 3350 1012 102 7727 2000 2032 2004 2069 1000 1996 7409 1000 1010 2572 3217 5831 5496 2010 2567 1997 9969 4487 23809 3436 2010 3350 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:04:55.554611 140342252849024 run_classifier.py:465] input_ids: 101 2572 3217 5831 5496 2010 2567 1010 3183 2002 2170 1000 1996 7409 1000 1010 1997 9969 4487 23809 3436 2010 3350 1012 102 7727 2000 2032 2004 2069 1000 1996 7409 1000 1010 2572 3217 5831 5496 2010 2567 1997 9969 4487 23809 3436 2010 3350 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:04:55.559763 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:04:55.569917 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 1 (id = 1)










    






I0328 18:04:55.575988 140342252849024 run_classifier.py:468] label: 1 (id = 1)










    






INFO:tensorflow:*** Example ***










    






I0328 18:04:55.583806 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: train-2










    






I0328 18:04:55.586417 140342252849024 run_classifier.py:462] guid: train-2










    






INFO:tensorflow:tokens: [CLS] yu ##ca ##ip ##a owned dominic ##k ' s before selling the chain to safe ##way in 1998 for $ 2 . 5 billion . [SEP] yu ##ca ##ip ##a bought dominic ##k ' s in 1995 for $ 69 ##3 million and sold it to safe ##way for $ 1 . 8 billion in 1998 . [SEP]










    






I0328 18:04:55.588699 140342252849024 run_classifier.py:464] tokens: [CLS] yu ##ca ##ip ##a owned dominic ##k ' s before selling the chain to safe ##way in 1998 for $ 2 . 5 billion . [SEP] yu ##ca ##ip ##a bought dominic ##k ' s in 1995 for $ 69 ##3 million and sold it to safe ##way for $ 1 . 8 billion in 1998 . [SEP]










    






INFO:tensorflow:input_ids: 101 9805 3540 11514 2050 3079 11282 2243 1005 1055 2077 4855 1996 4677 2000 3647 4576 1999 2687 2005 1002 1016 1012 1019 4551 1012 102 9805 3540 11514 2050 4149 11282 2243 1005 1055 1999 2786 2005 1002 6353 2509 2454 1998 2853 2009 2000 3647 4576 2005 1002 1015 1012 1022 4551 1999 2687 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:04:55.590543 140342252849024 run_classifier.py:465] input_ids: 101 9805 3540 11514 2050 3079 11282 2243 1005 1055 2077 4855 1996 4677 2000 3647 4576 1999 2687 2005 1002 1016 1012 1019 4551 1012 102 9805 3540 11514 2050 4149 11282 2243 1005 1055 1999 2786 2005 1002 6353 2509 2454 1998 2853 2009 2000 3647 4576 2005 1002 1015 1012 1022 4551 1999 2687 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:04:55.592910 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:04:55.594666 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 0 (id = 0)










    






I0328 18:04:55.596426 140342252849024 run_classifier.py:468] label: 0 (id = 0)










    






INFO:tensorflow:*** Example ***










    






I0328 18:04:55.603716 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: train-3










    






I0328 18:04:55.605922 140342252849024 run_classifier.py:462] guid: train-3










    






INFO:tensorflow:tokens: [CLS] they had published an advertisement on the internet on june 10 , offering the cargo for sale , he added . [SEP] on june 10 , the ship ' s owners had published an advertisement on the internet , offering the explosives for sale . [SEP]










    






I0328 18:04:55.607858 140342252849024 run_classifier.py:464] tokens: [CLS] they had published an advertisement on the internet on june 10 , offering the cargo for sale , he added . [SEP] on june 10 , the ship ' s owners had published an advertisement on the internet , offering the explosives for sale . [SEP]










    






INFO:tensorflow:input_ids: 101 2027 2018 2405 2019 15147 2006 1996 4274 2006 2238 2184 1010 5378 1996 6636 2005 5096 1010 2002 2794 1012 102 2006 2238 2184 1010 1996 2911 1005 1055 5608 2018 2405 2019 15147 2006 1996 4274 1010 5378 1996 14792 2005 5096 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:04:55.609957 140342252849024 run_classifier.py:465] input_ids: 101 2027 2018 2405 2019 15147 2006 1996 4274 2006 2238 2184 1010 5378 1996 6636 2005 5096 1010 2002 2794 1012 102 2006 2238 2184 1010 1996 2911 1005 1055 5608 2018 2405 2019 15147 2006 1996 4274 1010 5378 1996 14792 2005 5096 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:04:55.614890 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:04:55.616701 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 1 (id = 1)










    






I0328 18:04:55.618228 140342252849024 run_classifier.py:468] label: 1 (id = 1)










    






INFO:tensorflow:*** Example ***










    






I0328 18:04:55.621234 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: train-4










    






I0328 18:04:55.623008 140342252849024 run_classifier.py:462] guid: train-4










    






INFO:tensorflow:tokens: [CLS] around 03 ##35 gm ##t , tab shares were up 19 cents , or 4 . 4 % , at a $ 4 . 56 , having earlier set a record high of a $ 4 . 57 . [SEP] tab shares jumped 20 cents , or 4 . 6 % , to set a record closing high at a $ 4 . 57 . [SEP]










    






I0328 18:04:55.629560 140342252849024 run_classifier.py:464] tokens: [CLS] around 03 ##35 gm ##t , tab shares were up 19 cents , or 4 . 4 % , at a $ 4 . 56 , having earlier set a record high of a $ 4 . 57 . [SEP] tab shares jumped 20 cents , or 4 . 6 % , to set a record closing high at a $ 4 . 57 . [SEP]










    






INFO:tensorflow:input_ids: 101 2105 6021 19481 13938 2102 1010 21628 6661 2020 2039 2539 16653 1010 2030 1018 1012 1018 1003 1010 2012 1037 1002 1018 1012 5179 1010 2383 3041 2275 1037 2501 2152 1997 1037 1002 1018 1012 5401 1012 102 21628 6661 5598 2322 16653 1010 2030 1018 1012 1020 1003 1010 2000 2275 1037 2501 5494 2152 2012 1037 1002 1018 1012 5401 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:04:55.631920 140342252849024 run_classifier.py:465] input_ids: 101 2105 6021 19481 13938 2102 1010 21628 6661 2020 2039 2539 16653 1010 2030 1018 1012 1018 1003 1010 2012 1037 1002 1018 1012 5179 1010 2383 3041 2275 1037 2501 2152 1997 1037 1002 1018 1012 5401 1012 102 21628 6661 5598 2322 16653 1010 2030 1018 1012 1020 1003 1010 2000 2275 1037 2501 5494 2152 2012 1037 1002 1018 1012 5401 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:04:55.644842 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:04:55.647881 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 0 (id = 0)










    






I0328 18:04:55.650696 140342252849024 run_classifier.py:468] label: 0 (id = 0)










    






INFO:tensorflow:*** Example ***










    






I0328 18:04:55.654719 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: train-5










    






I0328 18:04:55.657314 140342252849024 run_classifier.py:462] guid: train-5










    






INFO:tensorflow:tokens: [CLS] the stock rose $ 2 . 11 , or about 11 percent , to close friday at $ 21 . 51 on the new york stock exchange . [SEP] pg & e corp . shares jumped $ 1 . 63 or 8 percent to $ 21 . 03 on the new york stock exchange on friday . [SEP]










    






I0328 18:04:55.659839 140342252849024 run_classifier.py:464] tokens: [CLS] the stock rose $ 2 . 11 , or about 11 percent , to close friday at $ 21 . 51 on the new york stock exchange . [SEP] pg & e corp . shares jumped $ 1 . 63 or 8 percent to $ 21 . 03 on the new york stock exchange on friday . [SEP]










    






INFO:tensorflow:input_ids: 101 1996 4518 3123 1002 1016 1012 2340 1010 2030 2055 2340 3867 1010 2000 2485 5958 2012 1002 2538 1012 4868 2006 1996 2047 2259 4518 3863 1012 102 18720 1004 1041 13058 1012 6661 5598 1002 1015 1012 6191 2030 1022 3867 2000 1002 2538 1012 6021 2006 1996 2047 2259 4518 3863 2006 5958 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:04:55.663161 140342252849024 run_classifier.py:465] input_ids: 101 1996 4518 3123 1002 1016 1012 2340 1010 2030 2055 2340 3867 1010 2000 2485 5958 2012 1002 2538 1012 4868 2006 1996 2047 2259 4518 3863 1012 102 18720 1004 1041 13058 1012 6661 5598 1002 1015 1012 6191 2030 1022 3867 2000 1002 2538 1012 6021 2006 1996 2047 2259 4518 3863 2006 5958 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:04:55.667357 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:04:55.669975 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 1 (id = 1)










    






I0328 18:04:55.673620 140342252849024 run_classifier.py:468] label: 1 (id = 1)










    






***** Started training at 2019-03-28 18:04:59.941973 *****
  Num examples = 3668
  Batch size = 32
INFO:tensorflow:  Num steps = 343










    






I0328 18:04:59.942914 140342252849024 <ipython-input-7-108f6f632247>:9]   Num steps = 343










    






INFO:tensorflow:Querying Tensorflow master (grpc://10.114.88.226:8470) for TPU system metadata.










    






I0328 18:05:00.097110 140342252849024 tpu_system_metadata.py:59] Querying Tensorflow master (grpc://10.114.88.226:8470) for TPU system metadata.










    






INFO:tensorflow:Found TPU system:










    






I0328 18:05:00.117680 140342252849024 tpu_system_metadata.py:120] Found TPU system:










    






INFO:tensorflow:*** Num TPU Cores: 8










    






I0328 18:05:00.120908 140342252849024 tpu_system_metadata.py:121] *** Num TPU Cores: 8










    






INFO:tensorflow:*** Num TPU Workers: 1










    






I0328 18:05:00.125142 140342252849024 tpu_system_metadata.py:122] *** Num TPU Workers: 1










    






INFO:tensorflow:*** Num TPU Cores Per Worker: 8










    






I0328 18:05:00.129628 140342252849024 tpu_system_metadata.py:124] *** Num TPU Cores Per Worker: 8










    






INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, -1, 3020137555896628722)










    






I0328 18:05:00.133469 140342252849024 tpu_system_metadata.py:126] *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:CPU:0, CPU, -1, 3020137555896628722)










    






INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 17179869184, 15223542790104685402)










    






I0328 18:05:00.137553 140342252849024 tpu_system_metadata.py:126] *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 17179869184, 15223542790104685402)










    






INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 17179869184, 1843375914324221839)










    






I0328 18:05:00.141189 140342252849024 tpu_system_metadata.py:126] *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:0, TPU, 17179869184, 1843375914324221839)










    






INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 17179869184, 9894822734449577370)










    






I0328 18:05:00.144656 140342252849024 tpu_system_metadata.py:126] *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:1, TPU, 17179869184, 9894822734449577370)










    






INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 17179869184, 14380569740864114478)










    






I0328 18:05:00.149079 140342252849024 tpu_system_metadata.py:126] *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:2, TPU, 17179869184, 14380569740864114478)










    






INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 17179869184, 17084474974895924727)










    






I0328 18:05:00.152604 140342252849024 tpu_system_metadata.py:126] *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:3, TPU, 17179869184, 17084474974895924727)










    






INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 17179869184, 7110320513217349863)










    






I0328 18:05:00.156504 140342252849024 tpu_system_metadata.py:126] *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:4, TPU, 17179869184, 7110320513217349863)










    






INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 17179869184, 11587941208496677740)










    






I0328 18:05:00.159977 140342252849024 tpu_system_metadata.py:126] *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:5, TPU, 17179869184, 11587941208496677740)










    






INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 17179869184, 10553633026221058370)










    






I0328 18:05:00.163847 140342252849024 tpu_system_metadata.py:126] *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:6, TPU, 17179869184, 10553633026221058370)










    






INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 17179869184, 9279467740370141958)










    






I0328 18:05:00.168128 140342252849024 tpu_system_metadata.py:126] *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU:7, TPU, 17179869184, 9279467740370141958)










    






INFO:tensorflow:*** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 17179869184, 2228303855717980436)










    






I0328 18:05:00.172176 140342252849024 tpu_system_metadata.py:126] *** Available Device: _DeviceAttributes(/job:worker/replica:0/task:0/device:TPU_SYSTEM:0, TPU_SYSTEM, 17179869184, 2228303855717980436)










    






INFO:tensorflow:Calling model_fn.










    






I0328 18:05:00.224345 140342252849024 estimator.py:1111] Calling model_fn.










    






INFO:tensorflow:*** Features ***










    






I0328 18:05:02.516797 140342252849024 run_classifier.py:627] *** Features ***










    






INFO:tensorflow:  name = input_ids, shape = (4, 128)










    






I0328 18:05:02.519099 140342252849024 run_classifier.py:629]   name = input_ids, shape = (4, 128)










    






INFO:tensorflow:  name = input_mask, shape = (4, 128)










    






I0328 18:05:02.521341 140342252849024 run_classifier.py:629]   name = input_mask, shape = (4, 128)










    






INFO:tensorflow:  name = label_ids, shape = (4,)










    






I0328 18:05:02.532980 140342252849024 run_classifier.py:629]   name = label_ids, shape = (4,)










    






INFO:tensorflow:  name = segment_ids, shape = (4, 128)










    






I0328 18:05:02.536545 140342252849024 run_classifier.py:629]   name = segment_ids, shape = (4, 128)










    






WARNING:tensorflow:From bert_repo/modeling.py:671: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.dense instead.










    






W0328 18:05:02.682061 140342252849024 deprecation.py:323] From bert_repo/modeling.py:671: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.dense instead.










    






INFO:tensorflow:**** Trainable Variables ****










    






I0328 18:05:06.714067 140342252849024 run_classifier.py:663] **** Trainable Variables ****










    






INFO:tensorflow:  name = bert/embeddings/word_embeddings:0, shape = (30522, 768), *INIT_FROM_CKPT*










    






I0328 18:05:06.721744 140342252849024 run_classifier.py:669]   name = bert/embeddings/word_embeddings:0, shape = (30522, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/embeddings/token_type_embeddings:0, shape = (2, 768), *INIT_FROM_CKPT*










    






I0328 18:05:06.726054 140342252849024 run_classifier.py:669]   name = bert/embeddings/token_type_embeddings:0, shape = (2, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/embeddings/position_embeddings:0, shape = (512, 768), *INIT_FROM_CKPT*










    






I0328 18:05:06.730526 140342252849024 run_classifier.py:669]   name = bert/embeddings/position_embeddings:0, shape = (512, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/embeddings/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.736248 140342252849024 run_classifier.py:669]   name = bert/embeddings/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/embeddings/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.741278 140342252849024 run_classifier.py:669]   name = bert/embeddings/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:06.746249 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.751816 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:06.760060 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.766595 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:06.773247 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.778477 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:06.784694 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.790246 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.795325 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.807510 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:05:06.810704 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:05:06.815002 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:05:06.819494 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.824499 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.828748 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.832954 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:06.837995 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.841996 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:06.846843 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.851677 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:06.854423 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.861060 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:06.866105 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.870515 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.874437 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.879356 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:05:06.883686 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:05:06.889334 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:05:06.893544 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.898211 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.902364 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.910530 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:06.912998 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.915054 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:06.919115 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.923844 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:06.928172 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.932953 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:06.937655 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.941922 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.947440 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.954761 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:05:06.959302 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:05:06.963307 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:05:06.968447 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.973113 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.977387 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.980476 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:06.983308 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.985826 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:06.989023 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.991263 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:06.993323 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:06.995579 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:06.997941 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.000304 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.002704 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.005290 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:05:07.007716 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:05:07.010972 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.013359 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.016290 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.019014 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.021372 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.025074 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.027445 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.029966 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.032356 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.034811 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.039101 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.042121 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.045274 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.048369 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.051246 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:05:07.054186 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:05:07.057096 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.060274 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.063208 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.066642 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.069826 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.072484 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.076253 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.080281 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.083036 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.086344 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.089022 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.092370 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.095581 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.099692 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.104001 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:05:07.106338 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:05:07.109644 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.112673 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.116206 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.118413 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.121369 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.124219 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.126268 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.128882 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.131964 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.134999 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.137931 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.140959 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.143760 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.148051 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.151268 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:05:07.154150 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:05:07.156741 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.160168 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.164381 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.167316 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.170557 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.174921 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.177916 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.189308 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.192266 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.196341 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.199903 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.213517 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.217218 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.220183 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.223106 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:05:07.226730 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:05:07.229518 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.232555 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.235052 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.238057 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.240724 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.244205 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.247169 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.250312 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.253015 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.256021 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.258648 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.261716 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.264732 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.266756 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.268639 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:05:07.270730 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:05:07.274443 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.277174 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.280239 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.282575 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.285261 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.287965 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.290886 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.294781 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.297497 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.300189 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.303539 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.306574 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.309624 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.312353 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.315087 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:05:07.317754 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:05:07.320631 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.323016 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.326446 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.329148 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.332097 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.334444 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.337329 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.340749 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.345029 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.347413 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.350351 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.353719 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.356094 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.359432 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.362445 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:05:07.365732 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:05:07.368464 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.371211 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.375015 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.377716 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.380410 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.383285 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.385693 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.389121 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.392160 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.394874 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.397759 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.401733 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.404174 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.407024 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.409589 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:05:07.412536 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:05:07.415087 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.418801 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.420981 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.423915 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.426108 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/pooler/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:05:07.429377 140342252849024 run_classifier.py:669]   name = bert/pooler/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/pooler/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:05:07.432390 140342252849024 run_classifier.py:669]   name = bert/pooler/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = output_weights:0, shape = (2, 768)










    






I0328 18:05:07.434767 140342252849024 run_classifier.py:669]   name = output_weights:0, shape = (2, 768)










    






INFO:tensorflow:  name = output_bias:0, shape = (2,)










    






I0328 18:05:07.437483 140342252849024 run_classifier.py:669]   name = output_bias:0, shape = (2,)










    






INFO:tensorflow:Create CheckpointSaverHook.










    






I0328 18:05:23.834862 140342252849024 basic_session_run_hooks.py:527] Create CheckpointSaverHook.










    






INFO:tensorflow:Done calling model_fn.










    






I0328 18:05:24.276604 140342252849024 estimator.py:1113] Done calling model_fn.










    






INFO:tensorflow:TPU job name worker










    






I0328 18:05:27.278962 140342252849024 tpu_estimator.py:447] TPU job name worker










    






INFO:tensorflow:Graph was finalized.










    






I0328 18:05:29.207826 140342252849024 monitored_session.py:222] Graph was finalized.










    






INFO:tensorflow:Running local_init_op.










    






I0328 18:05:39.613986 140342252849024 session_manager.py:491] Running local_init_op.










    






INFO:tensorflow:Done running local_init_op.










    






I0328 18:05:40.436935 140342252849024 session_manager.py:493] Done running local_init_op.










    






INFO:tensorflow:Saving checkpoints for 0 into gs://YOUR_BUCKET/bert-checkpoints/models/MRPC/model.ckpt.










    






I0328 18:05:52.873016 140342252849024 basic_session_run_hooks.py:594] Saving checkpoints for 0 into gs://YOUR_BUCKET/bert-checkpoints/models/MRPC/model.ckpt.










    






INFO:tensorflow:Initialized dataset iterators in 0 seconds










    






I0328 18:06:15.972548 140342252849024 util.py:51] Initialized dataset iterators in 0 seconds










    






INFO:tensorflow:Installing graceful shutdown hook.










    






I0328 18:06:15.977824 140342252849024 session_support.py:345] Installing graceful shutdown hook.










    






INFO:tensorflow:Creating heartbeat manager for ['/job:worker/replica:0/task:0/device:CPU:0']










    






I0328 18:06:15.990210 140342252849024 session_support.py:102] Creating heartbeat manager for ['/job:worker/replica:0/task:0/device:CPU:0']










    






INFO:tensorflow:Configuring worker heartbeat: shutdown_mode: WAIT_FOR_COORDINATOR











    






I0328 18:06:16.013316 140342252849024 session_support.py:130] Configuring worker heartbeat: shutdown_mode: WAIT_FOR_COORDINATOR











    






INFO:tensorflow:Init TPU system










    






I0328 18:06:16.029163 140342252849024 tpu_estimator.py:504] Init TPU system










    






INFO:tensorflow:Initialized TPU in 7 seconds










    






I0328 18:06:23.396520 140342252849024 tpu_estimator.py:510] Initialized TPU in 7 seconds










    






INFO:tensorflow:Starting infeed thread controller.










    






I0328 18:06:23.405627 140340633626368 tpu_estimator.py:463] Starting infeed thread controller.










    






INFO:tensorflow:Starting outfeed thread controller.










    






I0328 18:06:23.410192 140340625233664 tpu_estimator.py:482] Starting outfeed thread controller.










    






INFO:tensorflow:Enqueue next (343) batch(es) of data to infeed.










    






I0328 18:06:24.204131 140342252849024 tpu_estimator.py:536] Enqueue next (343) batch(es) of data to infeed.










    






INFO:tensorflow:Dequeue next (343) batch(es) of data from outfeed.










    






I0328 18:06:24.206908 140342252849024 tpu_estimator.py:540] Dequeue next (343) batch(es) of data from outfeed.










    






INFO:tensorflow:loss = 0.0070027537, step = 343










    






I0328 18:07:29.553707 140342252849024 basic_session_run_hooks.py:249] loss = 0.0070027537, step = 343










    






INFO:tensorflow:Saving checkpoints for 343 into gs://YOUR_BUCKET/bert-checkpoints/models/MRPC/model.ckpt.










    






I0328 18:07:29.560395 140342252849024 basic_session_run_hooks.py:594] Saving checkpoints for 343 into gs://YOUR_BUCKET/bert-checkpoints/models/MRPC/model.ckpt.










    






INFO:tensorflow:Stop infeed thread controller










    






I0328 18:07:55.322841 140342252849024 tpu_estimator.py:545] Stop infeed thread controller










    






INFO:tensorflow:Shutting down InfeedController thread.










    






I0328 18:07:55.329241 140342252849024 tpu_estimator.py:392] Shutting down InfeedController thread.










    






INFO:tensorflow:InfeedController received shutdown signal, stopping.










    






I0328 18:07:55.334971 140340633626368 tpu_estimator.py:387] InfeedController received shutdown signal, stopping.










    






INFO:tensorflow:Infeed thread finished, shutting down.










    






I0328 18:07:55.338541 140340633626368 tpu_estimator.py:479] Infeed thread finished, shutting down.










    






INFO:tensorflow:infeed marked as finished










    






I0328 18:07:55.341921 140342252849024 error_handling.py:93] infeed marked as finished










    






INFO:tensorflow:Stop output thread controller










    






I0328 18:07:55.344388 140342252849024 tpu_estimator.py:549] Stop output thread controller










    






INFO:tensorflow:Shutting down OutfeedController thread.










    






I0328 18:07:55.349071 140342252849024 tpu_estimator.py:392] Shutting down OutfeedController thread.










    






INFO:tensorflow:OutfeedController received shutdown signal, stopping.










    






I0328 18:07:55.353745 140340625233664 tpu_estimator.py:387] OutfeedController received shutdown signal, stopping.










    






INFO:tensorflow:Outfeed thread finished, shutting down.










    






I0328 18:07:55.355992 140340625233664 tpu_estimator.py:488] Outfeed thread finished, shutting down.










    






INFO:tensorflow:outfeed marked as finished










    






I0328 18:07:55.359511 140342252849024 error_handling.py:93] outfeed marked as finished










    






INFO:tensorflow:Shutdown TPU system.










    






I0328 18:07:55.362388 140342252849024 tpu_estimator.py:553] Shutdown TPU system.










    






INFO:tensorflow:Loss for final step: 0.0070027537.










    






I0328 18:07:56.611001 140342252849024 estimator.py:359] Loss for final step: 0.0070027537.










    






INFO:tensorflow:training_loop marked as finished










    






I0328 18:07:56.615934 140342252849024 error_handling.py:93] training_loop marked as finished










    






***** Finished training at 2019-03-28 18:07:56.624281 *****

















In [16]:



    


model_eval(estimator_from_checkpoints)



    


















    






INFO:tensorflow:Writing example 0 of 408










    






I0328 18:08:29.290884 140342252849024 run_classifier.py:774] Writing example 0 of 408










    






INFO:tensorflow:*** Example ***










    






I0328 18:08:29.294555 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: dev-1










    






I0328 18:08:29.297458 140342252849024 run_classifier.py:462] guid: dev-1










    






INFO:tensorflow:tokens: [CLS] he said the foods ##er ##vic ##e pie business doesn ' t fit the company ' s long - term growth strategy . [SEP] " the foods ##er ##vic ##e pie business does not fit our long - term growth strategy . [SEP]










    






I0328 18:08:29.301168 140342252849024 run_classifier.py:464] tokens: [CLS] he said the foods ##er ##vic ##e pie business doesn ' t fit the company ' s long - term growth strategy . [SEP] " the foods ##er ##vic ##e pie business does not fit our long - term growth strategy . [SEP]










    






INFO:tensorflow:input_ids: 101 2002 2056 1996 9440 2121 7903 2063 11345 2449 2987 1005 1056 4906 1996 2194 1005 1055 2146 1011 2744 3930 5656 1012 102 1000 1996 9440 2121 7903 2063 11345 2449 2515 2025 4906 2256 2146 1011 2744 3930 5656 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:08:29.303488 140342252849024 run_classifier.py:465] input_ids: 101 2002 2056 1996 9440 2121 7903 2063 11345 2449 2987 1005 1056 4906 1996 2194 1005 1055 2146 1011 2744 3930 5656 1012 102 1000 1996 9440 2121 7903 2063 11345 2449 2515 2025 4906 2256 2146 1011 2744 3930 5656 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:08:29.306126 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:08:29.308491 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 1 (id = 1)










    






I0328 18:08:29.310660 140342252849024 run_classifier.py:468] label: 1 (id = 1)










    






INFO:tensorflow:*** Example ***










    






I0328 18:08:29.315461 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: dev-2










    






I0328 18:08:29.317679 140342252849024 run_classifier.py:462] guid: dev-2










    






INFO:tensorflow:tokens: [CLS] magna ##relli said ra ##cic ##ot hated the iraqi regime and looked forward to using his long years of training in the war . [SEP] his wife said he was " 100 percent behind george bush " and looked forward to using his years of training in the war . [SEP]










    






I0328 18:08:29.321001 140342252849024 run_classifier.py:464] tokens: [CLS] magna ##relli said ra ##cic ##ot hated the iraqi regime and looked forward to using his long years of training in the war . [SEP] his wife said he was " 100 percent behind george bush " and looked forward to using his years of training in the war . [SEP]










    






INFO:tensorflow:input_ids: 101 20201 22948 2056 10958 19053 4140 6283 1996 8956 6939 1998 2246 2830 2000 2478 2010 2146 2086 1997 2731 1999 1996 2162 1012 102 2010 2564 2056 2002 2001 1000 2531 3867 2369 2577 5747 1000 1998 2246 2830 2000 2478 2010 2086 1997 2731 1999 1996 2162 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:08:29.323409 140342252849024 run_classifier.py:465] input_ids: 101 20201 22948 2056 10958 19053 4140 6283 1996 8956 6939 1998 2246 2830 2000 2478 2010 2146 2086 1997 2731 1999 1996 2162 1012 102 2010 2564 2056 2002 2001 1000 2531 3867 2369 2577 5747 1000 1998 2246 2830 2000 2478 2010 2086 1997 2731 1999 1996 2162 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:08:29.326866 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:08:29.329257 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 0 (id = 0)










    






I0328 18:08:29.332798 140342252849024 run_classifier.py:468] label: 0 (id = 0)










    






INFO:tensorflow:*** Example ***










    






I0328 18:08:29.338916 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: dev-3










    






I0328 18:08:29.341611 140342252849024 run_classifier.py:462] guid: dev-3










    






INFO:tensorflow:tokens: [CLS] the dollar was at 116 . 92 yen against the yen , flat on the session , and at 1 . 289 ##1 against the swiss fran ##c , also flat . [SEP] the dollar was at 116 . 78 yen jp ##y = , virtually flat on the session , and at 1 . 287 ##1 against the swiss fran ##c ch ##f = , down 0 . 1 percent . [SEP]










    






I0328 18:08:29.344743 140342252849024 run_classifier.py:464] tokens: [CLS] the dollar was at 116 . 92 yen against the yen , flat on the session , and at 1 . 289 ##1 against the swiss fran ##c , also flat . [SEP] the dollar was at 116 . 78 yen jp ##y = , virtually flat on the session , and at 1 . 287 ##1 against the swiss fran ##c ch ##f = , down 0 . 1 percent . [SEP]










    






INFO:tensorflow:input_ids: 101 1996 7922 2001 2012 12904 1012 6227 18371 2114 1996 18371 1010 4257 2006 1996 5219 1010 1998 2012 1015 1012 27054 2487 2114 1996 5364 23151 2278 1010 2036 4257 1012 102 1996 7922 2001 2012 12904 1012 6275 18371 16545 2100 1027 1010 8990 4257 2006 1996 5219 1010 1998 2012 1015 1012 23090 2487 2114 1996 5364 23151 2278 10381 2546 1027 1010 2091 1014 1012 1015 3867 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:08:29.347177 140342252849024 run_classifier.py:465] input_ids: 101 1996 7922 2001 2012 12904 1012 6227 18371 2114 1996 18371 1010 4257 2006 1996 5219 1010 1998 2012 1015 1012 27054 2487 2114 1996 5364 23151 2278 1010 2036 4257 1012 102 1996 7922 2001 2012 12904 1012 6275 18371 16545 2100 1027 1010 8990 4257 2006 1996 5219 1010 1998 2012 1015 1012 23090 2487 2114 1996 5364 23151 2278 10381 2546 1027 1010 2091 1014 1012 1015 3867 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:08:29.349680 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:08:29.353568 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 0 (id = 0)










    






I0328 18:08:29.355954 140342252849024 run_classifier.py:468] label: 0 (id = 0)










    






INFO:tensorflow:*** Example ***










    






I0328 18:08:29.359657 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: dev-4










    






I0328 18:08:29.363342 140342252849024 run_classifier.py:462] guid: dev-4










    






INFO:tensorflow:tokens: [CLS] the afl - ci ##o is waiting until october to decide if it will end ##ors ##e a candidate . [SEP] the afl - ci ##o announced wednesday that it will decide in october whether to end ##ors ##e a candidate before the primaries . [SEP]










    






I0328 18:08:29.365685 140342252849024 run_classifier.py:464] tokens: [CLS] the afl - ci ##o is waiting until october to decide if it will end ##ors ##e a candidate . [SEP] the afl - ci ##o announced wednesday that it will decide in october whether to end ##ors ##e a candidate before the primaries . [SEP]










    






INFO:tensorflow:input_ids: 101 1996 10028 1011 25022 2080 2003 3403 2127 2255 2000 5630 2065 2009 2097 2203 5668 2063 1037 4018 1012 102 1996 10028 1011 25022 2080 2623 9317 2008 2009 2097 5630 1999 2255 3251 2000 2203 5668 2063 1037 4018 2077 1996 27419 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:08:29.368891 140342252849024 run_classifier.py:465] input_ids: 101 1996 10028 1011 25022 2080 2003 3403 2127 2255 2000 5630 2065 2009 2097 2203 5668 2063 1037 4018 1012 102 1996 10028 1011 25022 2080 2623 9317 2008 2009 2097 5630 1999 2255 3251 2000 2203 5668 2063 1037 4018 2077 1996 27419 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:08:29.371611 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:08:29.373966 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 1 (id = 1)










    






I0328 18:08:29.376606 140342252849024 run_classifier.py:468] label: 1 (id = 1)










    






INFO:tensorflow:*** Example ***










    






I0328 18:08:29.379866 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: dev-5










    






I0328 18:08:29.382511 140342252849024 run_classifier.py:462] guid: dev-5










    






INFO:tensorflow:tokens: [CLS] no dates have been set for the civil or the criminal trial . [SEP] no dates have been set for the criminal or civil cases , but shan ##ley has pleaded not guilty . [SEP]










    






I0328 18:08:29.385606 140342252849024 run_classifier.py:464] tokens: [CLS] no dates have been set for the civil or the criminal trial . [SEP] no dates have been set for the criminal or civil cases , but shan ##ley has pleaded not guilty . [SEP]










    






INFO:tensorflow:input_ids: 101 2053 5246 2031 2042 2275 2005 1996 2942 2030 1996 4735 3979 1012 102 2053 5246 2031 2042 2275 2005 1996 4735 2030 2942 3572 1010 2021 17137 3051 2038 12254 2025 5905 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:08:29.388680 140342252849024 run_classifier.py:465] input_ids: 101 2053 5246 2031 2042 2275 2005 1996 2942 2030 1996 4735 3979 1012 102 2053 5246 2031 2042 2275 2005 1996 4735 2030 2942 3572 1010 2021 17137 3051 2038 12254 2025 5905 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:08:29.391760 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:08:29.394418 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 0 (id = 0)










    






I0328 18:08:29.396791 140342252849024 run_classifier.py:468] label: 0 (id = 0)










    






***** Started evaluation at 2019-03-28 18:08:29.896624 *****
  Num examples = 408
  Batch size = 8
INFO:tensorflow:Calling model_fn.










    






I0328 18:08:30.135971 140342252849024 estimator.py:1111] Calling model_fn.










    






INFO:tensorflow:*** Features ***










    






I0328 18:08:30.468494 140342252849024 run_classifier.py:627] *** Features ***










    






INFO:tensorflow:  name = input_ids, shape = (1, 128)










    






I0328 18:08:30.472352 140342252849024 run_classifier.py:629]   name = input_ids, shape = (1, 128)










    






INFO:tensorflow:  name = input_mask, shape = (1, 128)










    






I0328 18:08:30.474865 140342252849024 run_classifier.py:629]   name = input_mask, shape = (1, 128)










    






INFO:tensorflow:  name = label_ids, shape = (1,)










    






I0328 18:08:30.476619 140342252849024 run_classifier.py:629]   name = label_ids, shape = (1,)










    






INFO:tensorflow:  name = segment_ids, shape = (1, 128)










    






I0328 18:08:30.478551 140342252849024 run_classifier.py:629]   name = segment_ids, shape = (1, 128)










    






INFO:tensorflow:**** Trainable Variables ****










    






I0328 18:08:34.930666 140342252849024 run_classifier.py:663] **** Trainable Variables ****










    






INFO:tensorflow:  name = bert/embeddings/word_embeddings:0, shape = (30522, 768), *INIT_FROM_CKPT*










    






I0328 18:08:34.933112 140342252849024 run_classifier.py:669]   name = bert/embeddings/word_embeddings:0, shape = (30522, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/embeddings/token_type_embeddings:0, shape = (2, 768), *INIT_FROM_CKPT*










    






I0328 18:08:34.944250 140342252849024 run_classifier.py:669]   name = bert/embeddings/token_type_embeddings:0, shape = (2, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/embeddings/position_embeddings:0, shape = (512, 768), *INIT_FROM_CKPT*










    






I0328 18:08:34.949510 140342252849024 run_classifier.py:669]   name = bert/embeddings/position_embeddings:0, shape = (512, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/embeddings/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:34.956188 140342252849024 run_classifier.py:669]   name = bert/embeddings/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/embeddings/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:34.961345 140342252849024 run_classifier.py:669]   name = bert/embeddings/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:34.966374 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:34.968989 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:34.971570 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:34.975316 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:34.977545 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:34.981194 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:34.984221 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:34.987038 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:34.990913 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:34.994204 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:08:34.997058 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:08:35.001154 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.008656 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.011514 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.015535 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.018846 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.021831 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.025609 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.028908 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.031621 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.035448 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.039496 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.043406 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.047753 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.051549 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.056871 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:08:35.061164 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:08:35.065517 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.070128 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.080112 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.082617 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.087204 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.092362 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.097181 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.101622 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.106539 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.110664 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.115644 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.122073 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.126379 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.131755 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.137169 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:08:35.142477 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:08:35.147870 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.160803 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.166306 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.171946 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.178140 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.183742 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.191657 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.197242 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.201961 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.208306 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.213749 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.219347 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.224180 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.229151 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.233356 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:08:35.237444 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:08:35.241319 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.244627 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.246647 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.253925 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.258820 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.262720 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.267793 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.272687 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.276571 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.281380 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.288484 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.292406 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.297094 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.300463 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.304189 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:08:35.308437 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:08:35.312669 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.317061 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.320202 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.324209 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.329385 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.336377 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.338609 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.342877 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.347451 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.351763 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.353690 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.357832 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.363008 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.368316 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.373997 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:08:35.390368 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:08:35.395058 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.401025 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.405577 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.409591 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.416696 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.421100 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.425030 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.429227 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.434773 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.438227 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.442845 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.447171 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.451530 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.457512 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.463207 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:08:35.466449 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:08:35.470533 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.473840 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.478681 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.482624 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.486095 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.489797 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.494722 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.497803 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.501204 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.503841 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.506658 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.509189 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.512007 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.514420 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.517626 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:08:35.520459 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:08:35.522950 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.525353 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.527995 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.530408 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.533775 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.536364 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.538707 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.541361 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.543747 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.547076 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.549540 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.552896 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.555040 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.558121 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.560395 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:08:35.563224 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:08:35.565766 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.568119 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.570418 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.572737 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.575192 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.577419 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.579659 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.582430 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.584472 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.586820 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.589996 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.592251 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.595412 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.597576 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.599792 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:08:35.601955 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:08:35.605096 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.607434 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.609787 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.612257 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.615039 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.617275 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.619778 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.622038 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.624315 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.626583 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.629442 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.631489 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.635103 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.637356 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.640200 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:08:35.642646 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:08:35.644885 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.648331 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.650622 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.653185 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.655478 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.657781 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.660387 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.662796 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.666097 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.668762 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.671361 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.675264 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.678123 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.680541 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.683714 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:08:35.689280 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:08:35.691461 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.694651 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.697245 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.699611 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.703086 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/pooler/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:08:35.705245 140342252849024 run_classifier.py:669]   name = bert/pooler/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/pooler/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:08:35.707941 140342252849024 run_classifier.py:669]   name = bert/pooler/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = output_weights:0, shape = (2, 768)










    






I0328 18:08:35.710869 140342252849024 run_classifier.py:669]   name = output_weights:0, shape = (2, 768)










    






INFO:tensorflow:  name = output_bias:0, shape = (2,)










    






I0328 18:08:35.713490 140342252849024 run_classifier.py:669]   name = output_bias:0, shape = (2,)










    






INFO:tensorflow:Done calling model_fn.










    






I0328 18:08:37.683892 140342252849024 estimator.py:1113] Done calling model_fn.










    






INFO:tensorflow:Starting evaluation at 2019-03-28T18:08:37Z










    






I0328 18:08:37.715115 140342252849024 evaluation.py:257] Starting evaluation at 2019-03-28T18:08:37Z










    






INFO:tensorflow:TPU job name worker










    






I0328 18:08:37.717103 140342252849024 tpu_estimator.py:447] TPU job name worker










    






INFO:tensorflow:Graph was finalized.










    






I0328 18:08:38.792811 140342252849024 monitored_session.py:222] Graph was finalized.










    






INFO:tensorflow:Restoring parameters from gs://YOUR_BUCKET/bert-checkpoints/models/MRPC/model.ckpt-343










    






I0328 18:08:38.856900 140342252849024 saver.py:1270] Restoring parameters from gs://YOUR_BUCKET/bert-checkpoints/models/MRPC/model.ckpt-343










    






INFO:tensorflow:Running local_init_op.










    






I0328 18:08:46.423724 140342252849024 session_manager.py:491] Running local_init_op.










    






INFO:tensorflow:Done running local_init_op.










    






I0328 18:08:46.612194 140342252849024 session_manager.py:493] Done running local_init_op.










    






INFO:tensorflow:Init TPU system










    






I0328 18:08:47.047445 140342252849024 tpu_estimator.py:504] Init TPU system










    






INFO:tensorflow:Initialized TPU in 7 seconds










    






I0328 18:08:54.687232 140342252849024 tpu_estimator.py:510] Initialized TPU in 7 seconds










    






INFO:tensorflow:Starting infeed thread controller.










    






I0328 18:08:54.690903 140340697868032 tpu_estimator.py:463] Starting infeed thread controller.










    






INFO:tensorflow:Starting outfeed thread controller.










    






I0328 18:08:54.695691 140340662204160 tpu_estimator.py:482] Starting outfeed thread controller.










    






INFO:tensorflow:Initialized dataset iterators in 0 seconds










    






I0328 18:08:54.885627 140342252849024 util.py:51] Initialized dataset iterators in 0 seconds










    






INFO:tensorflow:Enqueue next (51) batch(es) of data to infeed.










    






I0328 18:08:55.086502 140342252849024 tpu_estimator.py:536] Enqueue next (51) batch(es) of data to infeed.










    






INFO:tensorflow:Dequeue next (51) batch(es) of data from outfeed.










    






I0328 18:08:55.090766 140342252849024 tpu_estimator.py:540] Dequeue next (51) batch(es) of data from outfeed.










    






INFO:tensorflow:Evaluation [51/51]










    






I0328 18:08:59.010277 140342252849024 evaluation.py:169] Evaluation [51/51]










    






INFO:tensorflow:Stop infeed thread controller










    






I0328 18:08:59.018233 140342252849024 tpu_estimator.py:545] Stop infeed thread controller










    






INFO:tensorflow:Shutting down InfeedController thread.










    






I0328 18:08:59.021522 140342252849024 tpu_estimator.py:392] Shutting down InfeedController thread.










    






INFO:tensorflow:InfeedController received shutdown signal, stopping.










    






I0328 18:08:59.023180 140340697868032 tpu_estimator.py:387] InfeedController received shutdown signal, stopping.










    






INFO:tensorflow:Infeed thread finished, shutting down.










    






I0328 18:08:59.024757 140340697868032 tpu_estimator.py:479] Infeed thread finished, shutting down.










    






INFO:tensorflow:infeed marked as finished










    






I0328 18:08:59.026638 140342252849024 error_handling.py:93] infeed marked as finished










    






INFO:tensorflow:Stop output thread controller










    






I0328 18:08:59.028192 140342252849024 tpu_estimator.py:549] Stop output thread controller










    






INFO:tensorflow:Shutting down OutfeedController thread.










    






I0328 18:08:59.029755 140342252849024 tpu_estimator.py:392] Shutting down OutfeedController thread.










    






INFO:tensorflow:OutfeedController received shutdown signal, stopping.










    






I0328 18:08:59.031406 140340662204160 tpu_estimator.py:387] OutfeedController received shutdown signal, stopping.










    






INFO:tensorflow:Outfeed thread finished, shutting down.










    






I0328 18:08:59.033022 140340662204160 tpu_estimator.py:488] Outfeed thread finished, shutting down.










    






INFO:tensorflow:outfeed marked as finished










    






I0328 18:08:59.034819 140342252849024 error_handling.py:93] outfeed marked as finished










    






INFO:tensorflow:Shutdown TPU system.










    






I0328 18:08:59.036807 140342252849024 tpu_estimator.py:553] Shutdown TPU system.










    






INFO:tensorflow:Finished evaluation at 2019-03-28-18:08:59










    






I0328 18:08:59.589336 140342252849024 evaluation.py:277] Finished evaluation at 2019-03-28-18:08:59










    






INFO:tensorflow:Saving dict for global step 343: eval_accuracy = 0.8137255, eval_loss = 0.8239641, global_step = 343, loss = 0.78950214










    






I0328 18:08:59.595096 140342252849024 estimator.py:1979] Saving dict for global step 343: eval_accuracy = 0.8137255, eval_loss = 0.8239641, global_step = 343, loss = 0.78950214










    






INFO:tensorflow:Saving 'checkpoint_path' summary for global step 343: gs://YOUR_BUCKET/bert-checkpoints/models/MRPC/model.ckpt-343










    






I0328 18:09:01.965670 140342252849024 estimator.py:2039] Saving 'checkpoint_path' summary for global step 343: gs://YOUR_BUCKET/bert-checkpoints/models/MRPC/model.ckpt-343










    






INFO:tensorflow:evaluation_loop marked as finished










    






I0328 18:09:02.363112 140342252849024 error_handling.py:93] evaluation_loop marked as finished










    






***** Finished evaluation at 2019-03-28 18:09:02.365279 *****
***** Eval results *****
  eval_accuracy = 0.8137255
  eval_loss = 0.8239641
  global_step = 343
  loss = 0.78950214

















In [17]:



    


model_predict(estimator_from_checkpoints)



    


















    






INFO:tensorflow:Writing example 0 of 8










    






I0328 18:09:42.543784 140342252849024 run_classifier.py:774] Writing example 0 of 8










    






INFO:tensorflow:*** Example ***










    






I0328 18:09:42.547711 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: dev-1










    






I0328 18:09:42.550762 140342252849024 run_classifier.py:462] guid: dev-1










    






INFO:tensorflow:tokens: [CLS] he said the foods ##er ##vic ##e pie business doesn ' t fit the company ' s long - term growth strategy . [SEP] " the foods ##er ##vic ##e pie business does not fit our long - term growth strategy . [SEP]










    






I0328 18:09:42.553687 140342252849024 run_classifier.py:464] tokens: [CLS] he said the foods ##er ##vic ##e pie business doesn ' t fit the company ' s long - term growth strategy . [SEP] " the foods ##er ##vic ##e pie business does not fit our long - term growth strategy . [SEP]










    






INFO:tensorflow:input_ids: 101 2002 2056 1996 9440 2121 7903 2063 11345 2449 2987 1005 1056 4906 1996 2194 1005 1055 2146 1011 2744 3930 5656 1012 102 1000 1996 9440 2121 7903 2063 11345 2449 2515 2025 4906 2256 2146 1011 2744 3930 5656 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:09:42.557337 140342252849024 run_classifier.py:465] input_ids: 101 2002 2056 1996 9440 2121 7903 2063 11345 2449 2987 1005 1056 4906 1996 2194 1005 1055 2146 1011 2744 3930 5656 1012 102 1000 1996 9440 2121 7903 2063 11345 2449 2515 2025 4906 2256 2146 1011 2744 3930 5656 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:09:42.561140 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:09:42.565555 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 1 (id = 1)










    






I0328 18:09:42.568085 140342252849024 run_classifier.py:468] label: 1 (id = 1)










    






INFO:tensorflow:*** Example ***










    






I0328 18:09:42.574437 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: dev-2










    






I0328 18:09:42.577828 140342252849024 run_classifier.py:462] guid: dev-2










    






INFO:tensorflow:tokens: [CLS] magna ##relli said ra ##cic ##ot hated the iraqi regime and looked forward to using his long years of training in the war . [SEP] his wife said he was " 100 percent behind george bush " and looked forward to using his years of training in the war . [SEP]










    






I0328 18:09:42.580848 140342252849024 run_classifier.py:464] tokens: [CLS] magna ##relli said ra ##cic ##ot hated the iraqi regime and looked forward to using his long years of training in the war . [SEP] his wife said he was " 100 percent behind george bush " and looked forward to using his years of training in the war . [SEP]










    






INFO:tensorflow:input_ids: 101 20201 22948 2056 10958 19053 4140 6283 1996 8956 6939 1998 2246 2830 2000 2478 2010 2146 2086 1997 2731 1999 1996 2162 1012 102 2010 2564 2056 2002 2001 1000 2531 3867 2369 2577 5747 1000 1998 2246 2830 2000 2478 2010 2086 1997 2731 1999 1996 2162 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:09:42.584676 140342252849024 run_classifier.py:465] input_ids: 101 20201 22948 2056 10958 19053 4140 6283 1996 8956 6939 1998 2246 2830 2000 2478 2010 2146 2086 1997 2731 1999 1996 2162 1012 102 2010 2564 2056 2002 2001 1000 2531 3867 2369 2577 5747 1000 1998 2246 2830 2000 2478 2010 2086 1997 2731 1999 1996 2162 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:09:42.586907 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:09:42.589719 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 0 (id = 0)










    






I0328 18:09:42.592014 140342252849024 run_classifier.py:468] label: 0 (id = 0)










    






INFO:tensorflow:*** Example ***










    






I0328 18:09:42.595603 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: dev-3










    






I0328 18:09:42.598916 140342252849024 run_classifier.py:462] guid: dev-3










    






INFO:tensorflow:tokens: [CLS] the dollar was at 116 . 92 yen against the yen , flat on the session , and at 1 . 289 ##1 against the swiss fran ##c , also flat . [SEP] the dollar was at 116 . 78 yen jp ##y = , virtually flat on the session , and at 1 . 287 ##1 against the swiss fran ##c ch ##f = , down 0 . 1 percent . [SEP]










    






I0328 18:09:42.601310 140342252849024 run_classifier.py:464] tokens: [CLS] the dollar was at 116 . 92 yen against the yen , flat on the session , and at 1 . 289 ##1 against the swiss fran ##c , also flat . [SEP] the dollar was at 116 . 78 yen jp ##y = , virtually flat on the session , and at 1 . 287 ##1 against the swiss fran ##c ch ##f = , down 0 . 1 percent . [SEP]










    






INFO:tensorflow:input_ids: 101 1996 7922 2001 2012 12904 1012 6227 18371 2114 1996 18371 1010 4257 2006 1996 5219 1010 1998 2012 1015 1012 27054 2487 2114 1996 5364 23151 2278 1010 2036 4257 1012 102 1996 7922 2001 2012 12904 1012 6275 18371 16545 2100 1027 1010 8990 4257 2006 1996 5219 1010 1998 2012 1015 1012 23090 2487 2114 1996 5364 23151 2278 10381 2546 1027 1010 2091 1014 1012 1015 3867 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:09:42.609267 140342252849024 run_classifier.py:465] input_ids: 101 1996 7922 2001 2012 12904 1012 6227 18371 2114 1996 18371 1010 4257 2006 1996 5219 1010 1998 2012 1015 1012 27054 2487 2114 1996 5364 23151 2278 1010 2036 4257 1012 102 1996 7922 2001 2012 12904 1012 6275 18371 16545 2100 1027 1010 8990 4257 2006 1996 5219 1010 1998 2012 1015 1012 23090 2487 2114 1996 5364 23151 2278 10381 2546 1027 1010 2091 1014 1012 1015 3867 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:09:42.612076 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:09:42.614373 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 0 (id = 0)










    






I0328 18:09:42.616106 140342252849024 run_classifier.py:468] label: 0 (id = 0)










    






INFO:tensorflow:*** Example ***










    






I0328 18:09:42.620203 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: dev-4










    






I0328 18:09:42.623185 140342252849024 run_classifier.py:462] guid: dev-4










    






INFO:tensorflow:tokens: [CLS] the afl - ci ##o is waiting until october to decide if it will end ##ors ##e a candidate . [SEP] the afl - ci ##o announced wednesday that it will decide in october whether to end ##ors ##e a candidate before the primaries . [SEP]










    






I0328 18:09:42.625893 140342252849024 run_classifier.py:464] tokens: [CLS] the afl - ci ##o is waiting until october to decide if it will end ##ors ##e a candidate . [SEP] the afl - ci ##o announced wednesday that it will decide in october whether to end ##ors ##e a candidate before the primaries . [SEP]










    






INFO:tensorflow:input_ids: 101 1996 10028 1011 25022 2080 2003 3403 2127 2255 2000 5630 2065 2009 2097 2203 5668 2063 1037 4018 1012 102 1996 10028 1011 25022 2080 2623 9317 2008 2009 2097 5630 1999 2255 3251 2000 2203 5668 2063 1037 4018 2077 1996 27419 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:09:42.628221 140342252849024 run_classifier.py:465] input_ids: 101 1996 10028 1011 25022 2080 2003 3403 2127 2255 2000 5630 2065 2009 2097 2203 5668 2063 1037 4018 1012 102 1996 10028 1011 25022 2080 2623 9317 2008 2009 2097 5630 1999 2255 3251 2000 2203 5668 2063 1037 4018 2077 1996 27419 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:09:42.632727 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:09:42.635007 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 1 (id = 1)










    






I0328 18:09:42.637812 140342252849024 run_classifier.py:468] label: 1 (id = 1)










    






INFO:tensorflow:*** Example ***










    






I0328 18:09:42.642343 140342252849024 run_classifier.py:461] *** Example ***










    






INFO:tensorflow:guid: dev-5










    






I0328 18:09:42.645550 140342252849024 run_classifier.py:462] guid: dev-5










    






INFO:tensorflow:tokens: [CLS] no dates have been set for the civil or the criminal trial . [SEP] no dates have been set for the criminal or civil cases , but shan ##ley has pleaded not guilty . [SEP]










    






I0328 18:09:42.648330 140342252849024 run_classifier.py:464] tokens: [CLS] no dates have been set for the civil or the criminal trial . [SEP] no dates have been set for the criminal or civil cases , but shan ##ley has pleaded not guilty . [SEP]










    






INFO:tensorflow:input_ids: 101 2053 5246 2031 2042 2275 2005 1996 2942 2030 1996 4735 3979 1012 102 2053 5246 2031 2042 2275 2005 1996 4735 2030 2942 3572 1010 2021 17137 3051 2038 12254 2025 5905 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:09:42.650661 140342252849024 run_classifier.py:465] input_ids: 101 2053 5246 2031 2042 2275 2005 1996 2942 2030 1996 4735 3979 1012 102 2053 5246 2031 2042 2275 2005 1996 4735 2030 2942 3572 1010 2021 17137 3051 2038 12254 2025 5905 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:09:42.653470 140342252849024 run_classifier.py:466] input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






I0328 18:09:42.655730 140342252849024 run_classifier.py:467] segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0










    






INFO:tensorflow:label: 0 (id = 0)










    






I0328 18:09:42.658500 140342252849024 run_classifier.py:468] label: 0 (id = 0)










    






INFO:tensorflow:Calling model_fn.










    






I0328 18:09:42.887690 140342252849024 estimator.py:1111] Calling model_fn.










    






INFO:tensorflow:*** Features ***










    






I0328 18:09:43.080179 140342252849024 run_classifier.py:627] *** Features ***










    






INFO:tensorflow:  name = input_ids, shape = (1, 128)










    






I0328 18:09:43.088006 140342252849024 run_classifier.py:629]   name = input_ids, shape = (1, 128)










    






INFO:tensorflow:  name = input_mask, shape = (1, 128)










    






I0328 18:09:43.096208 140342252849024 run_classifier.py:629]   name = input_mask, shape = (1, 128)










    






INFO:tensorflow:  name = label_ids, shape = (1,)










    






I0328 18:09:43.103522 140342252849024 run_classifier.py:629]   name = label_ids, shape = (1,)










    






INFO:tensorflow:  name = segment_ids, shape = (1, 128)










    






I0328 18:09:43.107226 140342252849024 run_classifier.py:629]   name = segment_ids, shape = (1, 128)










    






INFO:tensorflow:**** Trainable Variables ****










    






I0328 18:09:46.301699 140342252849024 run_classifier.py:663] **** Trainable Variables ****










    






INFO:tensorflow:  name = bert/embeddings/word_embeddings:0, shape = (30522, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.303973 140342252849024 run_classifier.py:669]   name = bert/embeddings/word_embeddings:0, shape = (30522, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/embeddings/token_type_embeddings:0, shape = (2, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.313265 140342252849024 run_classifier.py:669]   name = bert/embeddings/token_type_embeddings:0, shape = (2, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/embeddings/position_embeddings:0, shape = (512, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.317430 140342252849024 run_classifier.py:669]   name = bert/embeddings/position_embeddings:0, shape = (512, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/embeddings/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.321765 140342252849024 run_classifier.py:669]   name = bert/embeddings/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/embeddings/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.326072 140342252849024 run_classifier.py:669]   name = bert/embeddings/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.330779 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.333064 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.336601 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.340407 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.343188 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.346713 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.349225 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.363087 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.367160 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.369868 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:09:46.374012 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:09:46.377884 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.382111 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.385379 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.388869 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_0/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.392647 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_0/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.395893 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.399075 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.402270 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.405372 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.408465 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.411661 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.414755 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.417858 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.420938 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.424105 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:09:46.427273 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:09:46.430072 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.433114 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.437773 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.441319 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_1/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.444790 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_1/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.448274 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.451744 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.455790 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.459544 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.463448 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.466542 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.469533 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.471924 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.474876 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.477544 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:09:46.480422 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:09:46.483295 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.485698 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.488709 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.491691 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_2/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.494310 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_2/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.497205 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.499852 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.502381 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.505746 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.509076 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.512781 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.516092 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.519376 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.523582 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.527877 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:09:46.532019 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:09:46.536346 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.540779 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.544047 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.547618 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_3/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.551290 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_3/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.555361 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.558569 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.561749 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.564456 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.567626 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.569723 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.571722 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.573711 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.576571 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.579058 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:09:46.581938 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:09:46.584037 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.586356 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.589459 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.591811 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_4/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.594111 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_4/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.597102 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.599733 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.601904 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.604269 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.606068 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.608819 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.611078 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.613336 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.615607 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.619085 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:09:46.621872 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:09:46.624923 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.627609 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.629702 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.632010 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_5/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.634467 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_5/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.637341 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.639658 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.642677 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.644940 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.647429 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.649695 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.652338 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.654657 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.656944 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.659775 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:09:46.662292 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:09:46.664640 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.667972 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.670702 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.673571 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_6/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.676699 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_6/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.679326 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.682324 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.685144 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.687604 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.690398 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.693022 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.695914 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.698217 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.701189 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.703768 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:09:46.706351 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:09:46.708421 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.710739 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.713716 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.716275 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_7/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.720611 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_7/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.722781 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.725121 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.728065 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.730404 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.732681 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.735210 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.737574 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.740870 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.742918 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.745364 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:09:46.749043 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:09:46.751715 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.753807 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.756139 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.759760 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_8/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.762698 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_8/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.765141 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.767543 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.769685 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.772532 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.774675 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.777576 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.780618 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.783388 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.785891 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.788779 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:09:46.791039 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:09:46.793609 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.795879 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.798954 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.801198 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_9/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.803493 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_9/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.806077 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.808104 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.809810 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.811765 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.814300 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.816848 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.818702 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.821313 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.823703 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.826022 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:09:46.828295 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:09:46.831164 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.833700 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.836013 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.838809 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_10/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.841579 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_10/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.843890 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.846538 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.848896 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.851267 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.853848 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.856163 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.858525 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.860873 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.864148 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.866300 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






I0328 18:09:46.868679 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






I0328 18:09:46.871296 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.875169 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.877859 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.880488 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/encoder/layer_11/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.882997 140342252849024 run_classifier.py:669]   name = bert/encoder/layer_11/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/pooler/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






I0328 18:09:46.885303 140342252849024 run_classifier.py:669]   name = bert/pooler/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = bert/pooler/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






I0328 18:09:46.889101 140342252849024 run_classifier.py:669]   name = bert/pooler/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*










    






INFO:tensorflow:  name = output_weights:0, shape = (2, 768)










    






I0328 18:09:46.893280 140342252849024 run_classifier.py:669]   name = output_weights:0, shape = (2, 768)










    






INFO:tensorflow:  name = output_bias:0, shape = (2,)










    






I0328 18:09:46.897313 140342252849024 run_classifier.py:669]   name = output_bias:0, shape = (2,)










    






INFO:tensorflow:Done calling model_fn.










    






I0328 18:09:49.495923 140342252849024 estimator.py:1113] Done calling model_fn.










    






INFO:tensorflow:TPU job name worker










    






I0328 18:09:49.508721 140342252849024 tpu_estimator.py:447] TPU job name worker










    






INFO:tensorflow:Graph was finalized.










    






I0328 18:09:50.185844 140342252849024 monitored_session.py:222] Graph was finalized.










    






INFO:tensorflow:Restoring parameters from gs://YOUR_BUCKET/bert-checkpoints/models/MRPC/model.ckpt-343










    






I0328 18:09:50.262846 140342252849024 saver.py:1270] Restoring parameters from gs://YOUR_BUCKET/bert-checkpoints/models/MRPC/model.ckpt-343










    






INFO:tensorflow:Running local_init_op.










    






I0328 18:09:51.612142 140342252849024 session_manager.py:491] Running local_init_op.










    






INFO:tensorflow:Done running local_init_op.










    






I0328 18:09:51.813796 140342252849024 session_manager.py:493] Done running local_init_op.










    






INFO:tensorflow:Init TPU system










    






I0328 18:09:52.278981 140342252849024 tpu_estimator.py:504] Init TPU system










    






INFO:tensorflow:Initialized TPU in 10 seconds










    






I0328 18:10:02.660590 140342252849024 tpu_estimator.py:510] Initialized TPU in 10 seconds










    






INFO:tensorflow:Starting infeed thread controller.










    






I0328 18:10:02.667446 140340662204160 tpu_estimator.py:463] Starting infeed thread controller.










    






INFO:tensorflow:Starting outfeed thread controller.










    






I0328 18:10:02.679363 140340635723520 tpu_estimator.py:482] Starting outfeed thread controller.










    






INFO:tensorflow:Initialized dataset iterators in 0 seconds










    






I0328 18:10:02.918637 140342252849024 util.py:51] Initialized dataset iterators in 0 seconds










    






INFO:tensorflow:Enqueue next (1) batch(es) of data to infeed.










    






I0328 18:10:03.104757 140342252849024 tpu_estimator.py:536] Enqueue next (1) batch(es) of data to infeed.










    






INFO:tensorflow:Dequeue next (1) batch(es) of data from outfeed.










    






I0328 18:10:03.107872 140342252849024 tpu_estimator.py:540] Dequeue next (1) batch(es) of data from outfeed.










    






text_a: He said the foodservice pie business doesn 't fit the company 's long-term growth strategy .
text_b: " The foodservice pie business does not fit our long-term growth strategy .
label:1
prediction:[0.00160909 0.9983909 ]

text_a: Magnarelli said Racicot hated the Iraqi regime and looked forward to using his long years of training in the war .
text_b: His wife said he was " 100 percent behind George Bush " and looked forward to using his years of training in the war .
label:0
prediction:[0.99653023 0.00346978]

text_a: The dollar was at 116.92 yen against the yen , flat on the session , and at 1.2891 against the Swiss franc , also flat .
text_b: The dollar was at 116.78 yen JPY = , virtually flat on the session , and at 1.2871 against the Swiss franc CHF = , down 0.1 percent .
label:0
prediction:[0.99027705 0.009723  ]

text_a: The AFL-CIO is waiting until October to decide if it will endorse a candidate .
text_b: The AFL-CIO announced Wednesday that it will decide in October whether to endorse a candidate before the primaries .
label:1
prediction:[0.00168661 0.9983134 ]

text_a: No dates have been set for the civil or the criminal trial .
text_b: No dates have been set for the criminal or civil cases , but Shanley has pleaded not guilty .
label:0
prediction:[0.99469006 0.00531   ]

text_a: Wal-Mart said it would check all of its million-plus domestic workers to ensure they were legally employed .
text_b: It has also said it would review all of its domestic employees more than 1 million to ensure they have legal status .
label:1
prediction:[0.00595403 0.994046  ]

text_a: While dioxin levels in the environment were up last year , they have dropped by 75 percent since the 1970s , said Caswell .
text_b: The Institute said dioxin levels in the environment have fallen by as much as 76 percent since the 1970s .
label:0
prediction:[0.00515148 0.99484843]

text_a: This integrates with Rational PurifyPlus and allows developers to work in supported versions of Java , Visual C # and Visual Basic .NET.
text_b: IBM said the Rational products were also integrated with Rational PurifyPlus , which allows developers to work in Java , Visual C # and VisualBasic .Net.
label:1
prediction:[0.00127064 0.9987294 ]

INFO:tensorflow:prediction_loop marked as finished










    






I0328 18:10:07.010930 140342252849024 error_handling.py:93] prediction_loop marked as finished

What's next

  • Learn about Cloud TPUs that Google designed and optimized specifically to speed up and scale up ML workloads for training and inference and to enable ML engineers and researchers to iterate more quickly.
  • Explore the range of Cloud TPU tutorials and Colabs to find other examples that can be used when implementing your ML project.

On Google Cloud Platform, in addition to GPUs and TPUs available on pre-configured deep learning VMs, you will find AutoML(beta) for training custom models without writing code and Cloud ML Engine which will allows you to run parallel trainings and hyperparameter tuning of your custom models on powerful distributed hardware.

Content source: tensorflow/tpu

Similar notebooks:

notebook.community | gallery | about