RStudio AI Blog: Easy PixelCNN with tfprobability

We’ve seen fairly a number of examples of unsupervised studying (or self-supervised studying, to decide on the extra right however much less
well-liked time period) on this weblog.

Often, these concerned Variational Autoencoders (VAEs), whose enchantment lies in them permitting to mannequin a latent house of
underlying, impartial (ideally) elements that decide the seen options. A doable draw back might be the inferior
high quality of generated samples. Generative Adversarial Networks (GANs) are one other well-liked method. Conceptually, these are
extremely engaging resulting from their game-theoretic framing. However, they are often troublesome to coach. PixelCNN variants, on the
different hand – we’ll subsume all of them right here below PixelCNN – are usually identified for his or her good outcomes. They appear to contain
some extra alchemy although. Under these circumstances, what may very well be extra welcome than a straightforward method of experimenting with
them? Through TensorFlow Probability (TFP) and its R wrapper, tfprobability, we now have
such a method.

This submit first provides an introduction to PixelCNN, concentrating on high-level ideas (leaving the main points for the curious
to look them up within the respective papers). We’ll then present an instance of utilizing tfprobability to experiment with the TFP
implementation.

PixelCNN ideas

Autoregressivity, or: We want (some) order

The fundamental concept in PixelCNN is autoregressivity. Each pixel is modeled as relying on all prior pixels. Formally:

[p(mathbf{x}) = prod_{i}p(x_i|x_0, x_1, …, x_{i-1})]

Now wait a second – what even are prior pixels? Last I noticed one photographs had been two-dimensional. So this implies we’ve got to impose
an order on the pixels. Commonly this can be raster scan order: row after row, from left to proper. But when coping with
colour photographs, there’s one thing else: At every place, we even have three depth values, one for every of purple, inexperienced,
and blue. The authentic PixelCNN paper(Oord, Kalchbrenner, and Kavukcuoglu 2016) carried via autoregressivity right here as properly, with a pixel’s depth for
purple relying on simply prior pixels, these for inexperienced relying on these identical prior pixels however moreover, the present worth
for purple, and people for blue relying on the prior pixels in addition to the present values for purple and inexperienced.

[p(x_i|mathbf{x}<i) = p(x_{i,R}|mathbf{x}<i) p(x_{i,G}|mathbf{x}<i, x_{i,R}) p(x_{i,B}|mathbf{x}<i, x_{i,R}, x_{i,G})]

Here, the variant carried out in TFP, PixelCNN++(Salimans et al. 2017) , introduces a simplification; it factorizes the joint
distribution in a much less compute-intensive method.

Technically, then, we all know how autoregressivity is realized; intuitively, it could nonetheless appear stunning that imposing a raster
scan order “just works” (to me, at the very least, it’s). Maybe that is a type of factors the place compute energy efficiently
compensates for lack of an equal of a cognitive prior.

Masking, or: Where to not look

Now, PixelCNN ends in “CNN” for a cause – as common in picture processing, convolutional layers (or blocks thereof) are
concerned. But – is it not the very nature of a convolution that it computes a mean of some types, trying, for every
output pixel, not simply on the corresponding enter but in addition, at its spatial (or temporal) environment? How does that rhyme
with the look-at-just-prior-pixels technique?

Surprisingly, this downside is simpler to resolve than it sounds. When making use of the convolutional kernel, simply multiply with a
masks that zeroes out any “forbidden pixels” – like on this instance for a 5×5 kernel, the place we’re about to compute the
convolved worth for row 3, column 3:

[left[begin{array}
{rrr}
1 & 1 & 1 & 1 & 1
1 & 1 & 1 & 1 & 1
1 & 1 & 1 & 0 & 0
0 & 0 & 0 & 0 & 0
0 & 0 & 0 & 0 & 0
end{array}right]
]

This makes the algorithm sincere, however introduces a unique downside: With every successive convolutional layer consuming its
predecessor’s output, there’s a repeatedly rising blind spot (so-called in analogy to the blind spot on the retina, however
positioned within the prime proper) of pixels which can be by no means seen by the algorithm. Van den Oord et al. (2016)(Oord et al. 2016) repair this
through the use of two completely different convolutional stacks, one continuing from prime to backside, the opposite from left to proper.

Conditioning, or: Show me a kitten

So far, we’ve all the time talked about “generating images” in a purely generic method. But the actual attraction lies in creating
samples of some specified kind – one of many lessons we’ve been coaching on, or orthogonal data fed into the community.
This is the place PixelCNN turns into Conditional PixelCNN(Oord et al. 2016), and it is usually the place that feeling of magic resurfaces.
Again, as “general math” it’s not laborious to conceive. Here, (mathbf{h}) is the extra enter we’re conditioning on:

[p(mathbf{x}| mathbf{h}) = prod_{i}p(x_i|x_0, x_1, …, x_{i-1}, mathbf{h})]

But how does this translate into neural community operations? It’s simply one other matrix multiplication ((V^T mathbf{h})) added
to the convolutional outputs ((W mathbf{x})).

[mathbf{y} = tanh(W_{k,f} mathbf{x} + V^T_{k,f} mathbf{h}) odot sigma(W_{k,g} mathbf{x} + V^T_{k,g} mathbf{h})]

(If you’re questioning concerning the second half on the best, after the Hadamard product signal – we gained’t go into particulars, however in a
nutshell, it’s one other modification launched by (Oord et al. 2016), a switch of the “gating” precept from recurrent neural
networks, similar to GRUs and LSTMs, to the convolutional setting.)

So we see what goes into the choice of a pixel worth to pattern. But how is that call truly made?

Logistic combination chance , or: No pixel is an island

Again, that is the place the TFP implementation doesn’t observe the unique paper, however the latter PixelCNN++ one. Originally,
pixels had been modeled as discrete values, selected by a softmax over 256 (0-255) doable values. (That this truly labored
looks like one other occasion of deep studying magic. Imagine: In this mannequin, 254 is as removed from 255 as it’s from 0.)

In distinction, PixelCNN++ assumes an underlying steady distribution of colour depth, and rounds to the closest integer.
That underlying distribution is a combination of logistic distributions, thus permitting for multimodality:

[nu sim sum_{i} pi_i logistic(mu_i, sigma_i)]

Overall structure and the PixelCNN distribution

Overall, PixelCNN++, as described in (Salimans et al. 2017), consists of six blocks. The blocks collectively make up a UNet-like
construction, successively downsizing the enter after which, upsampling once more:

In TFP’s PixelCNN distribution, the variety of blocks is configurable as num_hierarchies, the default being 3.

Each block consists of a customizable variety of layers, referred to as ResNet layers as a result of residual connection (seen on the
proper) complementing the convolutional operations within the horizontal stack:

In TFP, the variety of these layers per block is configurable as num_resnet.

num_resnet and num_hierarchies are the parameters you’re most certainly to experiment with, however there are a number of extra you’ll be able to
try within the documentation. The variety of logistic
distributions within the combination can be configurable, however from my experiments it’s greatest to maintain that quantity somewhat low to keep away from
producing NaNs throughout coaching.

Let’s now see a whole instance.

End-to-end instance

Our playground can be QuickDraw, a dataset – nonetheless rising –
obtained by asking individuals to attract some object in at most twenty seconds, utilizing the mouse. (To see for your self, simply try
the web site). As of at this time, there are greater than a fifty million cases, from 345
completely different lessons.

First and foremost, these information had been chosen to take a break from MNIST and its variants. But identical to these (and plenty of extra!),
QuickDraw might be obtained, in tfdatasets-ready type, through tfds, the R wrapper to
TensorFlow datasets. In distinction to the MNIST “family” although, the “real samples” are themselves extremely irregular, and sometimes
even lacking important components. So to anchor judgment, when displaying generated samples we all the time present eight precise drawings
with them.

Preparing the information

The dataset being gigantic, we instruct tfds to load the primary 500,000 drawings “only.”

To velocity up coaching additional, we then zoom in on twenty lessons. This successfully leaves us with ~ 1,100 – 1,500 drawings per
class.

# bee, bicycle, broccoli, butterfly, cactus,
# frog, guitar, lightning, penguin, pizza,
# rollerskates, sea turtle, sheep, snowflake, solar,
# swan, The Eiffel Tower, tractor, prepare, tree
lessons <- c(26, 29, 43, 49, 50,
             125, 134, 172, 218, 225,
             246, 255, 258, 271, 295,
             296, 308, 320, 322, 323
)

classes_tensor <- tf$solid(lessons, tf$int64)

train_ds <- train_ds %>%
  dataset_filter(
    perform(document) tf$reduce_any(tf$equal(classes_tensor, document$label), -1L)
  )

The PixelCNN distribution expects values within the vary from 0 to 255 – no normalization required. Preprocessing then consists
of simply casting pixels and labels every to float:

preprocess <- perform(document) {
  document$picture <- tf$solid(document$picture, tf$float32) 
  document$label <- tf$solid(document$label, tf$float32)
  listing(tuple(document$picture, document$label))
}

batch_size <- 32

prepare <- train_ds %>%
  dataset_map(preprocess) %>%
  dataset_shuffle(10000) %>%
  dataset_batch(batch_size)

Creating the mannequin

We now use tfd_pixel_cnn to outline what would be the
loglikelihood utilized by the mannequin.

dist <- tfd_pixel_cnn(
  image_shape = c(28, 28, 1),
  conditional_shape = listing(),
  num_resnet = 5,
  num_hierarchies = 3,
  num_filters = 128,
  num_logistic_mix = 5,
  dropout_p =.5
)

image_input <- layer_input(form = c(28, 28, 1))
label_input <- layer_input(form = listing())
log_prob <- dist %>% tfd_log_prob(image_input, conditional_input = label_input)

This customized loglikelihood is added as a loss to the mannequin, after which, the mannequin is compiled with simply an optimizer
specification solely. During coaching, loss first decreased shortly, however enhancements from later epochs had been smaller.

mannequin <- keras_model(inputs = listing(image_input, label_input), outputs = log_prob)
mannequin$add_loss(-tf$reduce_mean(log_prob))
mannequin$compile(optimizer = optimizer_adam(lr = .001))

mannequin %>% match(prepare, epochs = 10)

To collectively show actual and faux photographs:

for (i in lessons) {
  
  real_images <- train_ds %>%
    dataset_filter(
      perform(document) document$label == tf$solid(i, tf$int64)
    ) %>% 
    dataset_take(8) %>%
    dataset_batch(8)
  it <- as_iterator(real_images)
  real_images <- iter_next(it)
  real_images <- real_images$picture %>% as.array()
  real_images <- real_images[ , , , 1]/255
  
  generated_images <- dist %>% tfd_sample(8, conditional_input = i)
  generated_images <- generated_images %>% as.array()
  generated_images <- generated_images[ , , , 1]/255
  
  photographs <- abind::abind(real_images, generated_images, alongside = 1)
  png(paste0("draw_", i, ".png"), width = 8 * 28 * 10, peak = 2 * 28 * 10)
  par(mfrow = c(2, 8), mar = c(0, 0, 0, 0))
  photographs %>%
    purrr::array_tree(1) %>%
    purrr::map(as.raster) %>%
    purrr::iwalk(plot)
  dev.off()
}

From our twenty lessons, right here’s a alternative of six, every exhibiting actual drawings within the prime row, and faux ones under.

We in all probability wouldn’t confuse the primary and second rows, however then, the precise human drawings exhibit monumental variation, too.
And nobody ever stated PixelCNN was an structure for idea studying. Feel free to mess around with different datasets of your
alternative – TFP’s PixelCNN distribution makes it straightforward.

Wrapping up

In this submit, we had tfprobability / TFP do all of the heavy lifting for us, and so, may give attention to the underlying ideas.
Depending in your inclinations, this may be a perfect state of affairs – you don’t lose sight of the forest for the timber. On the
different hand: Should you discover that altering the supplied parameters doesn’t obtain what you need, you might have a reference
implementation to start out from. So regardless of the consequence, the addition of such higher-level performance to TFP is a win for the
customers. (If you’re a TFP developer studying this: Yes, we’d like extra :-)).

To everybody although, thanks for studying!