In [1]:
# %load /Users/facai/Study/book_notes/preconfig.py
%matplotlib inline

import matplotlib.pyplot as plt
import seaborn as sns
sns.set(color_codes=True)
sns.set(font='SimHei', font_scale=2.5)

from IPython.display import Image

TensorFlow 2.0资料汇总

RFC management

RFC: Variables in TensorFlow 2.0 #11

  • drawbacks:

    1. impossible-to-reason-about semantics => RefVariable -> ResourceVariable
    2. reliance on global scopes, and reliance on global collections. => keras, object oriented
  • main changes:

    1. get_variable => tf.Variable + scoped factory function
    2. remove variable_scope => name_scope
      • graph.variable_scope_stack => module-global weak dict[graph] = variable_scope_stack
    3. custom_getters: variable_creator_scope
    4. tf.assign* will be removed.
  • feedback:

    1. scope, too magic?
    2. slice: a[1].assign(5)
    3. restore from proto (graph def) or other python object: say, tf.Variable.from_proto.
    4. scope和Variable搭配,粒度太粗: 比如ResourceVariableScope下面,能创建PartitionVariable么? 参数不一致下,必须感知,如Resource和Partition。
    5. tf.make_template: 依赖于variable_scope来共享variable,机制要重写?
  • TODO:

    1. check scoped constructor functions: like variable_scope.variable
      # 封装到variable_scope.variable_createor_scope:
      with ops.get_default_graph()._variable_creator_scope(custom_creator):
        # 封装进tf.Variable:
        previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
        for getter in ops.get_default_graph()._variable_creator_stack:  # pylint: disable=protected-access
          previous_getter = _make_getter(getter, previous_getter)
      
    2. check implementation: PartitionedVariable, MirroredVariable, _UnreadVariable, CastVariable, etc 猜测:
      • subclass要override init方法
      • sublcass要搭配对应的creator function
      • 使用时要显示配置variable creator scope
        with variable_creator_scope(custom_creator_function):
        tf.Variable(args)
        
        参考RefVariable, ResourceVariable查看
    3. check tf.Variable(*args, **kwargs), use __init__ or __call__. => metaclass behavior
    4. check low-level implement of RefVariable, ResourceVariable
      • 确保读写顺序
      • Optimizer don't support Tensor: RefVariable
    5. Checkpointable class and API

RFC: New tf.print #14

  1. tf.Print => tf.print, or tf.strings.format
    • print在python 2里是statement,不能重载
  2. identity op => depensencies
    t = xxx
    with tf.control_dependencies([tf.print(xxxx)]):
      t_2 = 2 * t
    
  3. support log level.
  4. stateful => ensure order, and not being pruned.
  5. support nested structure.

problem:

  • c++ standard out / error => notebook kernel's console output
  • device: cpu:0 by default (avoid GPU crash)
  • 使用仍然不如常见的print直觉

RFC: Unify RNN interface #15

Unify TF RNN and Keras RNN: port functionalities from TF RNN to Keras.

  • 统一API:RNN Layer; RNN Cell
    1. gate order: 统一, IFCO, ICFO
  • RNN Cell in tf.contrib.rnn: 只迁移少部份
  • NVidia CuDNN: 性能好,但功能受限制
    if activation == 'tan' and dropout == 0 and use_bias == True: 
        self.could_use_cudnn = True
      else:
        self.could_use_cudnn = False
    

RFC: TensorFlow API symbols and namespaces #16

  • namespace, endpoint
  • @tf_export: TODO: 弄清机制
    @tf_export('keras.metrics.mean_squared_error',
               'keras.metrics.mse',
               'keras.metrics.MSE',
               'keras.losses.mean_squared_error',
               'keras.losses.mse',
               'keras.losses.MSE')
    def mean_squared_error(y_true, y_pred):
      return K.mean(math_ops.square(y_pred - y_true), axis=-1)
    
  • additional namespaces
    • tf.losses => tf.keras.losses
    • tf.metrics => tf.keras.metrics
    • tf.layers => tf.keras.layers
  • deprecated namespaces
    • tf.logging
    • tf.manip

RFC: Deprecate Collections #17

常用三种用法:

  1. 搜集变量或者op => 各自追踪
  2. 利用它来序列化数据:SaveModel
  3. 利用它来维持状态:SharedEmbeddingColumns => 全局变量

queue => tf.data

variable

class VariableTracker(object):
  def __init__(self):
    self.variables = []

  def variable_tracker(self, next_creator, **kwargs):
    v = next_creator(**kwargs)
    self.variables.append(v)
    return v

VariableTracker tracker
with tf.variable_creator_scope(tracker.variable_tracker):
  ...
  a = tf.Variable(0)
  ...

assert tracker.variables == [a]

update

def model_fn(features, labels, mode, params, config):
  logits = ...
  batch_norm = tf.BatchNormalization(momentum=0.99)
  logits = batch_norm(logits)

  train_op = 
  train_op = tf.group(train_op, *batch_norm.updates)
  return tf.EstimatorSpec(..., train_op=train_op, ...)

Table: 还未确定

  • keras: track_table
  • table_creator_scope
  • tf.data: iterator, 初始化耗时serving

Summary, Condition => V2 design

RFC: Sunset tf.contrib #18

处理:

  1. moving to core: symbols should be prefixed with experimental.
  2. moving to a seperate repository

    • tensorflow/addons: layer, metric, loss, optimizer, op or kernel

      人员组成

      • Google
        • Martin Wicke: tf owner
        • Edd Wilder-James: administrative questions
        • Karmel Allison: TensorFlow technical contact
        • Scott Zhu: contrib.recurrent and contrib.rnn
        • Goldie Gadde
      • Community
        • Sean Morgan: project lead
        • Armando Fandango
        • Andreas Madsen: sparsemax
        • Facai Yan
        • Srivatsa P

      10月底会有第一次kickoff

    • tensorflow/network
    • tensorflow/scientific
  3. deleting

RFC: Functions, not sessions in 2.0 #20

  • tf.contrib.eager.defun 或者 Defun class

obejctive:

  • graph + session => function
  • state: tf object == python object
  • export: Checkpoint
  • eager execution by default
import tensorflow as tf

@tf.function
def compute_z1(x, y):
  return tf.add(x, y)

@tf.function
def compute_z0(x):
  return compute_z1(x, tf.square(x))

z0 = compute_z0(2.)
z1 = compute_z1(2., 2.)

机制:@tf.function: function => class: trace_cache_key 缓存graph

特性:

  1. For W, b, and c, the lifetime of the Python objects and the runtime state are tied together.

    W = tf.Variable(
      tf.glorot_uniform_initializer()(
        (10, 10)))
    b = tf.Variable(tf.zeros(10))
    c = tf.Variable(0)
    
    @tf.function
    def f(x):
      c.assign_add(1)
      return tf.matmul(x, W) + b
    
    print(f(make_input_value())
    assert int(c) == 1
    
  2. automatically insert control dependencies to ensure stateful operations follow graph construction order.
    • the intention here is to avoid observable differences from program order
      a = tf.Variable(1.0)
      b = tf.Variable(1.0)
      @tf.function
      def f():
      a.assign(2.0)
      b.assign(3.0)
      return a + b
      print(f())
      
  3. cache graph:

    • input argument: type, shape, dtype
    • "context": eg: device
    • Too many traces

      @tf.function
       def f(x):
         return tf.square(x)
      
       f(tf.constant(1, dtype=tf.int32))
       f(tf.constant(1.0, dtype=tf.float32))
       f(2.0)  # use tf.constant instead.
       f(3.0)
      
      1. gc + weak reference
      2. input sigatures
        @tf.function(input_signature=((tf.float32, [None]))
        def f(x): return tf.add(x, 1.)
      3. log an error: call / total >= threshold
  4. class

    class ScalarModel(object):
      def __init__(self):
        self.v = tf.Variable(0)
    
      @tf.function
      def increment(self, amount):
        self.v.assign_add(amount)
    
  5. tf.compat.v1.wrap_function: Transitioning from 1.x
  6. Serialization: Exporting SavedModel/GraphDefs
    • graph = f.graph_function((tf.float32, (None, 10)).graph
  7. Derived/Related Graphs

    @tf.function
    def f(x):
      return tf.square(x)
    
    @tf.function
    def g(x):
      return tf.square(f(x))
    
    g(2.0) # 16.0
    
  8. Distributed Execution: details will be discussed separately

限制:

  1. 只能初始化变量一次(后面是reuse):
    • weak references to these created variables
    • local tf.Variable in function: not supported yet => investigated independently.
  2. Python control flow: data-dependent control flow, not supported => AutoGraph
    def f(x, y):
      if tf.equal(y, 0.0):
        return y
      return x / y
    
  3. summaries
  4. Supporting structured inputs?

其他:

  1. eager executution: symbolic Tensor inside function, while concrete Tensor object outside function.

feedback:

  1. 引入太多复杂性

TODO:

  1. 查看Defun class实现细节

In [ ]: