In [ ]:
#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

Note: This is an archived TF1 notebook. These are configured to run in TF2's compatbility mode but will run in TF1 as well. To use TF1 in Colab, use the magic.
In [ ]:
import math
import tensorflow.compat.v1 as tf
Your data comes in many shapes; your tensors should too. Ragged tensors are the TensorFlow equivalent of nested variablelength lists. They make it easy to store and process data with nonuniform shapes, including:
Ragged tensors are supported by more than a hundred TensorFlow operations,
including math operations (such as tf.add
and tf.reduce_mean
), array operations
(such as tf.concat
and tf.tile
), string manipulation ops (such as
tf.substr
), and many others:
In [ ]:
digits = tf.ragged.constant([[3, 1, 4, 1], [], [5, 9, 2], [6], []])
words = tf.ragged.constant([["So", "long"], ["thanks", "for", "all", "the", "fish"]])
print(tf.add(digits, 3))
print(tf.reduce_mean(digits, axis=1))
print(tf.concat([digits, [[5, 3]]], axis=0))
print(tf.tile(digits, [1, 2]))
print(tf.strings.substr(words, 0, 2))
There are also a number of methods and operations that are
specific to ragged tensors, including factory methods, conversion methods,
and valuemapping operations.
For a list of supported ops, see the tf.ragged
package
documentation.
As with normal tensors, you can use Pythonstyle indexing to access specific slices of a ragged tensor. For more information, see the section on Indexing below.
In [ ]:
print(digits[0]) # First row
In [ ]:
print(digits[:, :2]) # First two values in each row.
In [ ]:
print(digits[:, 2:]) # Last two values in each row.
And just like normal tensors, you can use Python arithmetic and comparison operators to perform elementwise operations. For more information, see the section on Overloaded Operators below.
In [ ]:
print(digits + 3)
In [ ]:
print(digits + tf.ragged.constant([[1, 2, 3, 4], [], [5, 6, 7], [8], []]))
If you need to perform an elementwise transformation to the values of a RaggedTensor
, you can use tf.ragged.map_flat_values
, which takes a function plus one or more arguments, and applies the function to transform the RaggedTensor
's values.
In [ ]:
times_two_plus_one = lambda x: x * 2 + 1
print(tf.ragged.map_flat_values(times_two_plus_one, digits))
In [ ]:
sentences = tf.ragged.constant([
["Let's", "build", "some", "ragged", "tensors", "!"],
["We", "can", "use", "tf.ragged.constant", "."]])
print(sentences)
In [ ]:
paragraphs = tf.ragged.constant([
[['I', 'have', 'a', 'cat'], ['His', 'name', 'is', 'Mat']],
[['Do', 'you', 'want', 'to', 'come', 'visit'], ["I'm", 'free', 'tomorrow']],
])
print(paragraphs)
Ragged tensors can also be constructed by pairing flat values tensors with
rowpartitioning tensors indicating how those values should be divided into
rows, using factory classmethods such as tf.RaggedTensor.from_value_rowids
,
tf.RaggedTensor.from_row_lengths
, and
tf.RaggedTensor.from_row_splits
.
tf.RaggedTensor.from_value_rowids
If you know which row each value belongs in, then you can build a RaggedTensor
using a value_rowids
rowpartitioning tensor:
In [ ]:
print(tf.RaggedTensor.from_value_rowids(
values=[3, 1, 4, 1, 5, 9, 2, 6],
value_rowids=[0, 0, 0, 0, 2, 2, 2, 3]))
In [ ]:
print(tf.RaggedTensor.from_row_lengths(
values=[3, 1, 4, 1, 5, 9, 2, 6],
row_lengths=[4, 0, 3, 1]))
In [ ]:
print(tf.RaggedTensor.from_row_splits(
values=[3, 1, 4, 1, 5, 9, 2, 6],
row_splits=[0, 4, 4, 7, 8]))
See the tf.RaggedTensor
class documentation for a full list of factory methods.
In [ ]:
print(tf.ragged.constant([["Hi"], ["How", "are", "you"]])) # ok: type=string, rank=2
In [ ]:
print(tf.ragged.constant([[[1, 2], [3]], [[4, 5]]])) # ok: type=int32, rank=3
In [ ]:
try:
tf.ragged.constant([["one", "two"], [3, 4]]) # bad: multiple types
except ValueError as exception:
print(exception)
In [ ]:
try:
tf.ragged.constant(["A", ["B", "C"]]) # bad: multiple nesting depths
except ValueError as exception:
print(exception)
The following example demonstrates how RaggedTensor
s can be used to construct
and combine unigram and bigram embeddings for a batch of variablelength
queries, using special markers for the beginning and end of each sentence.
For more details on the ops used in this example, see the tf.ragged
package documentation.
In [ ]:
queries = tf.ragged.constant([['Who', 'is', 'Dan', 'Smith'],
['Pause'],
['Will', 'it', 'rain', 'later', 'today']])
# Create an embedding table.
num_buckets = 1024
embedding_size = 4
embedding_table = tf.Variable(
tf.truncated_normal([num_buckets, embedding_size],
stddev=1.0 / math.sqrt(embedding_size)))
# Look up the embedding for each word.
word_buckets = tf.strings.to_hash_bucket_fast(queries, num_buckets)
word_embeddings = tf.ragged.map_flat_values(
tf.nn.embedding_lookup, embedding_table, word_buckets) # ①
# Add markers to the beginning and end of each sentence.
marker = tf.fill([queries.nrows(), 1], '#')
padded = tf.concat([marker, queries, marker], axis=1) # ②
# Build word bigrams & look up embeddings.
bigrams = tf.string_join([padded[:, :1], padded[:, 1:]], separator='+') # ③
bigram_buckets = tf.strings.to_hash_bucket_fast(bigrams, num_buckets)
bigram_embeddings = tf.ragged.map_flat_values(
tf.nn.embedding_lookup, embedding_table, bigram_buckets) # ④
# Find the average embedding for each sentence
all_embeddings = tf.concat([word_embeddings, bigram_embeddings], axis=1) # ⑤
avg_embedding = tf.reduce_mean(all_embeddings, axis=1) # ⑥
print(avg_embedding)
A ragged tensor is a tensor with one or more ragged dimensions,
which are dimensions whose slices may have different lengths. For example, the
inner (column) dimension of rt=[[3, 1, 4, 1], [], [5, 9, 2], [6], []]
is
ragged, since the column slices (rt[0, :]
, ..., rt[4, :]
) have different
lengths. Dimensions whose slices all have the same length are called uniform
dimensions.
The outermost dimension of a ragged tensor is always uniform, since it consists
of a single slice (and so there is no possibility for differing slice lengths).
In addition to the uniform outermost dimension, ragged tensors may also have
uniform inner dimensions. For example, we might store the word embeddings for
each word in a batch of sentences using a ragged tensor with shape
[num_sentences, (num_words), embedding_size]
, where the parentheses around
(num_words)
indicate that the dimension is ragged.
Ragged tensors may have multiple ragged dimensions. For example, we could store
a batch of structured text documents using a tensor with shape [num_documents,
(num_paragraphs), (num_sentences), (num_words)]
(where again parentheses are
used to indicate ragged dimensions).
The shape of a ragged tensor is currently restricted to have the following form:
Note: These restrictions are a consequence of the current implementation, and we may relax them in the future.
The total number of dimensions in a ragged tensor is called its rank, and
the number of ragged dimensions in a ragged tensor is called its ragged
rank. In graph execution mode (i.e., noneager mode), a tensor's ragged rank
is fixed at creation time: it can't depend
on runtime values, and can't vary dynamically for different session runs.
A potentially ragged tensor is a value that might be
either a tf.Tensor
or a tf.RaggedTensor
. The
ragged rank of a tf.Tensor
is defined to be zero.
When describing the shape of a RaggedTensor, ragged dimensions are indicated by
enclosing them in parentheses. For example, as we saw above, the shape of a 3D
RaggedTensor that stores word embeddings for each word in a batch of sentences
can be written as [num_sentences, (num_words), embedding_size]
.
The RaggedTensor.shape
attribute returns a tf.TensorShape
for a
ragged tensor, where ragged dimensions have size None
:
In [ ]:
tf.ragged.constant([["Hi"], ["How", "are", "you"]]).shape
The method tf.RaggedTensor.bounding_shape
can be used to find a tight
bounding shape for a given RaggedTensor
:
In [ ]:
print(tf.ragged.constant([["Hi"], ["How", "are", "you"]]).bounding_shape())
A ragged tensor should not be thought of as a type of sparse tensor, but rather as a dense tensor with an irregular shape.
As an illustrative example, consider how array operations such as concat
,
stack
, and tile
are defined for ragged vs. sparse tensors. Concatenating
ragged tensors joins each row to form a single row with the combined length:
In [ ]:
ragged_x = tf.ragged.constant([["John"], ["a", "big", "dog"], ["my", "cat"]])
ragged_y = tf.ragged.constant([["fell", "asleep"], ["barked"], ["is", "fuzzy"]])
print(tf.concat([ragged_x, ragged_y], axis=1))
But concatenating sparse tensors is equivalent to concatenating the corresponding dense tensors, as illustrated by the following example (where Ø indicates missing values):
In [ ]:
sparse_x = ragged_x.to_sparse()
sparse_y = ragged_y.to_sparse()
sparse_result = tf.sparse.concat(sp_inputs=[sparse_x, sparse_y], axis=1)
print(tf.sparse.to_dense(sparse_result, ''))
For another example of why this distinction is important, consider the
definition of “the mean value of each row” for an op such as tf.reduce_mean
.
For a ragged tensor, the mean value for a row is the sum of the
row’s values divided by the row’s width.
But for a sparse tensor, the mean value for a row is the sum of the
row’s values divided by the sparse tensor’s overall width (which is
greater than or equal to the width of the longest row).
In [ ]:
x = tf.ragged.constant([[1, 2], [3], [4, 5, 6]])
y = tf.ragged.constant([[1, 1], [2], [3, 3, 3]])
print(x + y)
Since the overloaded operators perform elementwise computations, the inputs to all binary operations must have the same shape, or be broadcastable to the same shape. In the simplest broadcasting case, a single scalar is combined elementwise with each value in a ragged tensor:
In [ ]:
x = tf.ragged.constant([[1, 2], [3], [4, 5, 6]])
print(x + 3)
For a discussion of more advanced cases, see the section on Broadcasting.
Ragged tensors overload the same set of operators as normal Tensor
s: the unary
operators 
, ~
, and abs()
; and the binary operators +
, 
, *
, /
,
//
, %
, **
, &
, 
, ^
, <
, <=
, >
, and >=
. Note that, as with
standard Tensor
s, binary ==
is not overloaded; you can use
tf.equal()
to check elementwise equality.
In [ ]:
queries = tf.ragged.constant(
[['Who', 'is', 'George', 'Washington'],
['What', 'is', 'the', 'weather', 'tomorrow'],
['Goodnight']])
print(queries[1])
In [ ]:
print(queries[1, 2]) # A single word
In [ ]:
print(queries[1:]) # Everything but the first row
In [ ]:
print(queries[:, :3]) # The first 3 words of each query
In [ ]:
print(queries[:, 2:]) # The last 2 words of each query
In [ ]:
rt = tf.ragged.constant([[[1, 2, 3], [4]],
[[5], [], [6]],
[[7]],
[[8, 9], [10]]])
In [ ]:
print(rt[1]) # Second row (2D RaggedTensor)
In [ ]:
print(rt[3, 0]) # First element of fourth row (1D Tensor)
In [ ]:
print(rt[:, 1:3]) # Items 13 of each row (3D RaggedTensor)
In [ ]:
print(rt[:, 1:]) # Last item of each row (3D RaggedTensor)
RaggedTensor
s supports multidimensional indexing and slicing, with one
restriction: indexing into a ragged dimension is not allowed. This case is
problematic because the indicated value may exist in some rows but not others.
In such cases, it's not obvious whether we should (1) raise an IndexError
; (2)
use a default value; or (3) skip that value and return a tensor with fewer rows
than we started with. Following the
guiding principles of Python
("In the face
of ambiguity, refuse the temptation to guess" ), we currently disallow this
operation.
In [ ]:
ragged_sentences = tf.ragged.constant([
['Hi'], ['Welcome', 'to', 'the', 'fair'], ['Have', 'fun']])
print(ragged_sentences.to_tensor(default_value=''))
In [ ]:
print(ragged_sentences.to_sparse())
In [ ]:
x = [[1, 3, 1, 1], [2, 1, 1, 1], [4, 5, 8, 9]]
print(tf.RaggedTensor.from_tensor(x, padding=1))
In [ ]:
st = tf.SparseTensor(indices=[[0, 0], [2, 0], [2, 1]],
values=['a', 'b', 'c'],
dense_shape=[3, 3])
print(tf.RaggedTensor.from_sparse(st))
In [ ]:
rt = tf.ragged.constant([[1, 2], [3, 4, 5], [6], [], [7]])
print(rt.to_list())
EagerTensor
. You can then use
the numpy()
method to access the value directly.
In [ ]:
print(rt[1].numpy())
tf.RaggedTensor.values
and
tf.RaggedTensor.row_splits
properties, or rowparitioning methods such as tf.RaggedTensor.row_lengths()
and tf.RaggedTensor.value_rowids()
.
In [ ]:
print(rt.values)
In [ ]:
print(rt.row_splits)
In [ ]:
with tf.Session() as session:
rt = tf.ragged.constant([[1, 2], [3, 4, 5], [6], [], [7]])
rt_value = session.run(rt)
The resulting value will be a
tf.ragged.RaggedTensorValue
instance. To access the values contained in a RaggedTensorValue
, you can:
tf.ragged.RaggedTensorValue.to_list()
method, which converts the RaggedTensorValue
to a Python list
.
In [ ]:
print(rt_value.to_list())
tf.ragged.RaggedTensorValue.values
and
tf.ragged.RaggedTensorValue.row_splits
properties.
In [ ]:
print(rt_value.values)
In [ ]:
print(rt_value.row_splits)
In [ ]:
tf.enable_eager_execution() # Resume eager execution mode.
Broadcasting is the process of making tensors with different shapes have compatible shapes for elementwise operations. For more background on broadcasting, see:
tf.broadcast_dynamic_shape
tf.broadcast_to
The basic steps for broadcasting two inputs x
and y
to have compatible
shapes are:
If x
and y
do not have the same number of dimensions, then add outer
dimensions (with size 1) until they do.
For each dimension where x
and y
have different sizes:
If x
or y
have size 1
in dimension d
, then repeat its values
across dimension d
to match the other input's size.
Otherwise, raise an exception (x
and y
are not broadcast
compatible).
In [ ]:
# x (2D ragged): 2 x (num_rows)
# y (scalar)
# result (2D ragged): 2 x (num_rows)
x = tf.ragged.constant([[1, 2], [3]])
y = 3
print(x + y)
In [ ]:
# x (2d ragged): 3 x (num_rows)
# y (2d tensor): 3 x 1
# Result (2d ragged): 3 x (num_rows)
x = tf.ragged.constant(
[[10, 87, 12],
[19, 53],
[12, 32]])
y = [[1000], [2000], [3000]]
print(x + y)
In [ ]:
# x (3d ragged): 2 x (r1) x 2
# y (2d ragged): 1 x 1
# Result (3d ragged): 2 x (r1) x 2
x = tf.ragged.constant(
[[[1, 2], [3, 4], [5, 6]],
[[7, 8]]],
ragged_rank=1)
y = tf.constant([[10]])
print(x + y)
In [ ]:
# x (3d ragged): 2 x (r1) x (r2) x 1
# y (1d tensor): 3
# Result (3d ragged): 2 x (r1) x (r2) x 3
x = tf.ragged.constant(
[
[
[[1], [2]],
[],
[[3]],
[[4]],
],
[
[[5], [6]],
[[7]]
]
],
ragged_rank=2)
y = tf.constant([10, 20, 30])
print(x + y)
Here are some examples of shapes that do not broadcast:
In [ ]:
# x (2d ragged): 3 x (r1)
# y (2d tensor): 3 x 4 # trailing dimensions do not match
x = tf.ragged.constant([[1, 2], [3, 4, 5, 6], [7]])
y = tf.constant([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]])
try:
x + y
except tf.errors.InvalidArgumentError as exception:
print(exception)
In [ ]:
# x (2d ragged): 3 x (r1)
# y (2d ragged): 3 x (r2) # ragged dimensions do not match.
x = tf.ragged.constant([[1, 2, 3], [4], [5, 6]])
y = tf.ragged.constant([[10, 20], [30, 40], [50]])
try:
x + y
except tf.errors.InvalidArgumentError as exception:
print(exception)
In [ ]:
# x (3d ragged): 3 x (r1) x 2
# y (3d ragged): 3 x (r1) x 3 # trailing dimensions do not match
x = tf.ragged.constant([[[1, 2], [3, 4], [5, 6]],
[[7, 8], [9, 10]]])
y = tf.ragged.constant([[[1, 2, 0], [3, 4, 0], [5, 6, 0]],
[[7, 8, 0], [9, 10, 0]]])
try:
x + y
except tf.errors.InvalidArgumentError as exception:
print(exception)
Ragged tensors are encoded using the RaggedTensor
class. Internally, each
RaggedTensor
consists of:
values
tensor, which concatenates the variablelength rows into a
flattened list.row_splits
vector, which indicates how those flattened values are
divided into rows. In particular, the values for row rt[i]
are stored in
the slice rt.values[rt.row_splits[i]:rt.row_splits[i+1]]
.
In [ ]:
rt = tf.RaggedTensor.from_row_splits(
values=[3, 1, 4, 1, 5, 9, 2],
row_splits=[0, 4, 4, 6, 7])
print(rt)
In [ ]:
rt = tf.RaggedTensor.from_row_splits(
values=tf.RaggedTensor.from_row_splits(
values=[10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
row_splits=[0, 3, 3, 5, 9, 10]),
row_splits=[0, 1, 1, 5])
print(rt)
print("Shape: {}".format(rt.shape))
print("Number of ragged dimensions: {}".format(rt.ragged_rank))
The factory function tf.RaggedTensor.from_nested_row_splits
may be used to construct a
RaggedTensor with multiple ragged dimensions directly, by providing a list of
row_splits
tensors:
In [ ]:
rt = tf.RaggedTensor.from_nested_row_splits(
flat_values=[10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
nested_row_splits=([0, 1, 1, 5], [0, 3, 3, 5, 9, 10]))
print(rt)
In [ ]:
rt = tf.RaggedTensor.from_row_splits(
values=[[1, 3], [0, 0], [1, 3], [5, 3], [3, 3], [1, 2]],
row_splits=[0, 3, 4, 6])
print(rt)
print("Shape: {}".format(rt.shape))
print("Number of ragged dimensions: {}".format(rt.ragged_rank))
The RaggedTensor
class uses row_splits
as the primary mechanism to store
information about how the values are partitioned into rows. However,
RaggedTensor
also provides support for four alternative rowpartitioning
schemes, which can be more convenient to use depending on how your data is
formatted. Internally, RaggedTensor
uses these additional schemes to improve
efficiency in some contexts.
For example, the following ragged tensors are equivalent:
In [ ]:
values = [3, 1, 4, 1, 5, 9, 2, 6]
print(tf.RaggedTensor.from_row_splits(values, row_splits=[0, 4, 4, 7, 8, 8]))
print(tf.RaggedTensor.from_row_lengths(values, row_lengths=[4, 0, 3, 1, 0]))
print(tf.RaggedTensor.from_row_starts(values, row_starts=[0, 4, 4, 7, 8]))
print(tf.RaggedTensor.from_row_limits(values, row_limits=[4, 4, 7, 8, 8]))
print(tf.RaggedTensor.from_value_rowids(
values, value_rowids=[0, 0, 0, 0, 2, 2, 2, 3], nrows=5))
The RaggedTensor class defines methods which can be used to construct each of these rowpartitioning tensors.
In [ ]:
rt = tf.ragged.constant([[3, 1, 4, 1], [], [5, 9, 2], [6], []])
print(" values: {}".format(rt.values))
print(" row_splits: {}".format(rt.row_splits))
print(" row_lengths: {}".format(rt.row_lengths()))
print(" row_starts: {}".format(rt.row_starts()))
print(" row_limits: {}".format(rt.row_limits()))
print("value_rowids: {}".format(rt.value_rowids()))
(Note that tf.RaggedTensor.values
and tf.RaggedTensors.row_splits
are properties, while the remaining rowpartitioning accessors are all methods. This reflects the fact that the row_splits
are the primary underlying representation, and the other rowpartitioning tensors must be computed.)
Some of the advantages and disadvantages of the different rowpartitioning schemes are:
Efficient indexing:
The row_splits
, row_starts
, and row_limits
schemes all enable
constanttime indexing into ragged tensors. The value_rowids
and
row_lengths
schemes do not.
Small encoding size:
The value_rowids
scheme is more efficient when storing ragged tensors that
have a large number of empty rows, since the size of the tensor depends only
on the total number of values. On the other hand, the other four encodings
are more efficient when storing ragged tensors with longer rows, since they
require only one scalar value for each row.
Efficient concatenation:
The row_lengths
scheme is more efficient when concatenating ragged
tensors, since row lengths do not change when two tensors are concatenated
together (but row splits and row indices do).
Compatibility:
The value_rowids
scheme matches the
segmentation
format used by operations such as tf.segment_sum
. The row_limits
scheme
matches the format used by ops such as tf.sequence_mask
.