

In [1]:

    
# %load /Users/facai/Study/book_notes/preconfig.py
%matplotlib inline

import matplotlib.pyplot as plt
import seaborn as sns
sns.set(color_codes=True)
sns.set(font='SimHei', font_scale=2.5)

from IPython.display import Image

TensorFlow 2.0资料汇总

RFC management

RFC: Variables in TensorFlow 2.0 #11

drawbacks:
1. impossible-to-reason-about semantics => RefVariable -> ResourceVariable
2. reliance on global scopes, and reliance on global collections. => keras, object oriented
main changes:
1. get_variable => tf.Variable + scoped factory function
2. remove variable_scope => name_scope
  - graph.variable_scope_stack => module-global weak dict[graph] = variable_scope_stack
3. custom_getters: variable_creator_scope
4. tf.assign* will be removed.
feedback:
1. scope, too magic?
2. slice: a[1].assign(5)
3. restore from proto (graph def) or other python object: say, tf.Variable.from_proto.
4. scope和Variable搭配，粒度太粗: 比如ResourceVariableScope下面，能创建PartitionVariable么? 参数不一致下，必须感知，如Resource和Partition。
5. tf.make_template: 依赖于variable_scope来共享variable，机制要重写？
TODO:
1. check scoped constructor functions: like variable_scope.variable
```
# 封装到variable_scope.variable_createor_scope:
with ops.get_default_graph()._variable_creator_scope(custom_creator):
  # 封装进tf.Variable:
  previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
  for getter in ops.get_default_graph()._variable_creator_stack:  # pylint: disable=protected-access
    previous_getter = _make_getter(getter, previous_getter)
```
2. check implementation: PartitionedVariable, MirroredVariable, _UnreadVariable, CastVariable, etc 猜测：
  - subclass要override init方法
  - sublcass要搭配对应的creator function
  - 使用时要显示配置variable creator scope
```
with variable_creator_scope(custom_creator_function):
tf.Variable(args)
```
    参考RefVariable, ResourceVariable查看
3. check tf.Variable(*args, **kwargs), use __init__ or __call__. => metaclass behavior
4. check low-level implement of RefVariable, ResourceVariable
  - 确保读写顺序
  - Optimizer don't support Tensor: RefVariable
5. Checkpointable class and API

RFC: New tf.print #14

tf.Print => tf.print, or tf.strings.format
- print在python 2里是statement，不能重载

identity op => depensencies

t = xxx
with tf.control_dependencies([tf.print(xxxx)]):
  t_2 = 2 * t

support log level.
stateful => ensure order, and not being pruned.
support nested structure.

problem:

c++ standard out / error => notebook kernel's console output
device: cpu:0 by default (avoid GPU crash)
使用仍然不如常见的print直觉

RFC: Unify RNN interface #15

Unify TF RNN and Keras RNN: port functionalities from TF RNN to Keras.

统一API：RNN Layer; RNN Cell
1. gate order: 统一， IFCO, ICFO
RNN Cell in tf.contrib.rnn: 只迁移少部份

NVidia CuDNN: 性能好，但功能受限制

if activation == 'tan' and dropout == 0 and use_bias == True: 
    self.could_use_cudnn = True
  else:
    self.could_use_cudnn = False

RFC: TensorFlow API symbols and namespaces #16

namespace, endpoint

@tf_export: TODO: 弄清机制

@tf_export('keras.metrics.mean_squared_error',
           'keras.metrics.mse',
           'keras.metrics.MSE',
           'keras.losses.mean_squared_error',
           'keras.losses.mse',
           'keras.losses.MSE')
def mean_squared_error(y_true, y_pred):
  return K.mean(math_ops.square(y_pred - y_true), axis=-1)

additional namespaces
- tf.losses => tf.keras.losses
- tf.metrics => tf.keras.metrics
- tf.layers => tf.keras.layers
deprecated namespaces
- tf.logging
- tf.manip

RFC: Deprecate Collections #17

常用三种用法：

搜集变量或者op => 各自追踪
利用它来序列化数据：SaveModel
利用它来维持状态：SharedEmbeddingColumns => 全局变量

queue => tf.data

variable

class VariableTracker(object):
  def __init__(self):
    self.variables = []

  def variable_tracker(self, next_creator, **kwargs):
    v = next_creator(**kwargs)
    self.variables.append(v)
    return v

VariableTracker tracker
with tf.variable_creator_scope(tracker.variable_tracker):
  ...
  a = tf.Variable(0)
  ...

assert tracker.variables == [a]

update

def model_fn(features, labels, mode, params, config):
  logits = ...
  batch_norm = tf.BatchNormalization(momentum=0.99)
  logits = batch_norm(logits)

  train_op = …
  train_op = tf.group(train_op, *batch_norm.updates)
  return tf.EstimatorSpec(..., train_op=train_op, ...)

Table: 还未确定

keras: track_table
table_creator_scope
tf.data: iterator, 初始化耗时serving

Summary, Condition => V2 design

RFC: Sunset tf.contrib #18

处理：

moving to core: symbols should be prefixed with experimental.
moving to a seperate repository
- tensorflow/addons: layer, metric, loss, optimizer, op or kernel
  
  人员组成
  - Google
    - Martin Wicke: tf owner
    - Edd Wilder-James: administrative questions
    - Karmel Allison: TensorFlow technical contact
    - Scott Zhu: contrib.recurrent and contrib.rnn
    - Goldie Gadde
  - Community
    - Sean Morgan: project lead
    - Armando Fandango
    - Andreas Madsen: sparsemax
    - Facai Yan
    - Srivatsa P
  10月底会有第一次kickoff
- tensorflow/network
- tensorflow/scientific
deleting

RFC: Functions, not sessions in 2.0 #20

tf.contrib.eager.defun 或者 Defun class

obejctive:

graph + session => function
state: tf object == python object
export: Checkpoint
eager execution by default

import tensorflow as tf

@tf.function
def compute_z1(x, y):
  return tf.add(x, y)

@tf.function
def compute_z0(x):
  return compute_z1(x, tf.square(x))

z0 = compute_z0(2.)
z1 = compute_z1(2., 2.)

机制：@tf.function: function => class: trace_cache_key 缓存graph

特性：

For W, b, and c, the lifetime of the Python objects and the runtime state are tied together.

W = tf.Variable(
  tf.glorot_uniform_initializer()(
    (10, 10)))
b = tf.Variable(tf.zeros(10))
c = tf.Variable(0)

@tf.function
def f(x):
  c.assign_add(1)
  return tf.matmul(x, W) + b

print(f(make_input_value())
assert int(c) == 1

automatically insert control dependencies to ensure stateful operations follow graph construction order.
- the intention here is to avoid observable differences from program order
```
a = tf.Variable(1.0)
b = tf.Variable(1.0)
@tf.function
def f():
a.assign(2.0)
b.assign(3.0)
return a + b
print(f())
```

cache graph:

input argument: type, shape, dtype
"context": eg: device

Too many traces

@tf.function
 def f(x):
   return tf.square(x)

 f(tf.constant(1, dtype=tf.int32))
 f(tf.constant(1.0, dtype=tf.float32))
 f(2.0)  # use tf.constant instead.
 f(3.0)

gc + weak reference

input sigatures

@tf.function(input_signature=((tf.float32, [None]))
def f(x): return tf.add(x, 1.)

log an error: call / total >= threshold

class

class ScalarModel(object):
  def __init__(self):
    self.v = tf.Variable(0)

  @tf.function
  def increment(self, amount):
    self.v.assign_add(amount)

tf.compat.v1.wrap_function: Transitioning from 1.x
Serialization: Exporting SavedModel/GraphDefs
- graph = f.graph_function((tf.float32, (None, 10)).graph

Derived/Related Graphs

@tf.function
def f(x):
  return tf.square(x)

@tf.function
def g(x):
  return tf.square(f(x))

g(2.0) # 16.0

Distributed Execution: details will be discussed separately

限制：

只能初始化变量一次（后面是reuse):
- weak references to these created variables
- local tf.Variable in function: not supported yet => investigated independently.

Python control flow: data-dependent control flow, not supported => AutoGraph

def f(x, y):
  if tf.equal(y, 0.0):
    return y
  return x / y

summaries
Supporting structured inputs?

其他：

eager executution: symbolic Tensor inside function, while concrete Tensor object outside function.

feedback:

引入太多复杂性

TODO:

查看Defun class实现细节



In [ ]: