So, how come we are able to use TensorStream from R?

Which pc language is most carefully related to TensorStream? While on the TensorStream for R weblog, we’d in fact like the reply to be R, chances are high it’s Python (although TensorStream has official bindings for C++, Swift, Javascript, Java, and Go as effectively).

So why is it you may outline a Keras mannequin as

library(keras)
mannequin <- keras_model_sequential() %>%
  layer_dense(models = 32, activation = "relu") %>%
  layer_dense(models = 1)

(good with %>%s and all!) – then prepare and consider it, get predictions and plot them, all that with out ever leaving R?

The quick reply is, you’ve keras, tensorflow and reticulate put in.
reticulate embeds a Python session inside the R course of. A single course of means a single deal with area: The identical objects exist, and might be operated upon, no matter whether or not they’re seen by R or by Python. On that foundation, tensorflow and keras then wrap the respective Python libraries and allow you to write R code that, in actual fact, seems like R.

This publish first elaborates a bit on the quick reply. We then go deeper into what occurs within the background.

One word on terminology earlier than we bounce in: On the R facet, we’re making a transparent distinction between the packages keras and tensorflow. For Python we’re going to use TensorStream and Keras interchangeably. Historically, these have been totally different, and TensorStream was generally considered one potential backend to run Keras on, moreover the pioneering, now discontinued Theano, and CNTK. Standalone Keras does nonetheless exist, however latest work has been, and is being, completed in tf.keras. Of course, this makes Python Keras a subset of Python TensorStream, however all examples on this publish will use that subset so we are able to use each to seek advice from the identical factor.

So keras, tensorflow, reticulate, what are they for?

Firstly, nothing of this may be potential with out reticulate. reticulate is an R package deal designed to permit seemless interoperability between R and Python. If we completely wished, we might assemble a Keras mannequin like this:

<class 'tensorflow.python.keras.engine.sequential.Sequential'>

We might go on including layers …

m$add(tf$keras$layers$Dense(32, "relu"))
m$add(tf$keras$layers$Dense(1))
m$layers

[[1]]
<tensorflow.python.keras.layers.core.Dense>

[[2]]
<tensorflow.python.keras.layers.core.Dense>

But who would need to? If this have been the one means, it’d be much less cumbersome to instantly write Python as a substitute. Plus, as a person you’d must know the whole Python-side module construction (now the place do optimizers dwell, at present: tf.keras.optimizers, tf.optimizers …?), and sustain with all path and identify adjustments within the Python API.

This is the place keras comes into play. keras is the place the TensorStream-specific usability, re-usability, and comfort options dwell.
Functionality supplied by keras spans the entire vary between boilerplate-avoidance over enabling elegant, R-like idioms to offering technique of superior characteristic utilization. As an instance for the primary two, think about layer_dense which, amongst others, converts its models argument to an integer, and takes arguments in an order that permit it to be “pipe-added” to a mannequin: Instead of

mannequin <- keras_model_sequential()
mannequin$add(layer_dense(models = 32L))

we are able to simply say

mannequin <- keras_model_sequential()
mannequin %>% layer_dense(models = 32)

While these are good to have, there’s extra. Advanced performance in (Python) Keras largely depends upon the power to subclass objects. One instance is customized callbacks. If you have been utilizing Python, you’d must subclass tf.keras.callbacks.Callback. From R, you may create an R6 class inheriting from KerasCallback, like so

CustomizedCallback <- R6::R6Class("CustomizedCallback",
    inherit = KerasCallback,
    public = listing(
      on_train_begin = operate(logs) {
        # do one thing
      },
      on_train_end = operate(logs) {
        # do one thing
      }
    )
  )

This is as a result of keras defines an precise Python class, RCallback, and maps your R6 class’ strategies to it.
Another instance is customized fashions, launched on this weblog a couple of 12 months in the past.
These fashions might be educated with customized coaching loops. In R, you utilize keras_model_custom to create one, for instance, like this:

m <- keras_model_custom(identify = "mymodel", operate(self) {
  self$dense1 <- layer_dense(models = 32, activation = "relu")
  self$dense2 <- layer_dense(models = 10, activation = "softmax")
  
  operate(inputs, masks = NULL) {
    self$dense1(inputs) %>%
      self$dense2()
  }
})

Here, keras will make sure that an precise Python object is created which subclasses tf.keras.Model and when known as, runs the above nameless operate().

So that’s keras. What concerning the tensorflow package deal? As a person you solely want it when it’s a must to do superior stuff, like configure TensorStream machine utilization or (in TF 1.x) entry components of the Graph or the Session. Internally, it’s utilized by keras closely. Essential inner performance contains, e.g., implementations of S3 strategies, like print, [ or +, on Tensors, so you can operate on them like on R vectors.

Now that we know what each of the packages is “for”, let’s dig deeper into what makes this possible.

Show me the magic: reticulate

Instead of exposing the topic top-down, we follow a by-example approach, building up complexity as we go. We’ll have three scenarios.

First, we assume we already have a Python object (that has been constructed in whatever way) and need to convert that to R. Then, we’ll investigate how we can create a Python object, calling its constructor. Finally, we go the other way round: We ask how we can pass an R function to Python for later usage.

Scenario 1: R-to-Python conversion

Let’s assume we have created a Python object in the global namespace, like this:

So: There is a variable, called x, with value 1, living in Python world. Now how do we bring this thing into R?

We know the main entry point to conversion is py_to_r, defined as a generic in conversion.R:

py_to_r <- function(x) {
  ensure_python_initialized()
  UseMethod("py_to_r")
}

… with the default implementation calling a function named py_ref_to_r:

#' @export
py_to_r.default <- function(x) {
  [...]
  x <- py_ref_to_r(x)
  [...]
}

To discover out extra about what’s going on, debugging on the R stage received’t get us far. We begin gdb so we are able to set breakpoints in C++ features:

$ R -d gdb

GNU gdb (GDB) Fedora 8.3-6.fc30
[... some more gdb saying hello ...]
Reading symbols from /usr/lib64/R/bin/exec/R...
Reading symbols from /usr/lib/debug/usr/lib64/R/bin/exec/R-3.6.0-1.fc30.x86_64.debug...

Now begin R, load reticulate, and execute the project we’re going to presuppose:

(gdb) run
Starting program: /usr/lib64/R/bin/exec/R 
[...]
R model 3.6.0 (2019-04-26) -- "Planting of a Tree"
Copyright (C) 2019 The R Foundation for Statistical Computing
[...]
> library(reticulate)
> py_run_string("x = 1")

So that arrange our situation, the Python object (named x) we need to convert to R. Now, use Ctrl-C to “escape” to gdb, set a breakpoint in py_to_r and sort c to get again to R:

(gdb) b py_to_r
Breakpoint 1 at 0x7fffe48315d0 (2 areas)
(gdb) c

Now what are we going to see once we entry that x?

> py$x

Thread 1 "R" hit Breakpoint 1, 0x00007fffe48315d0 in py_to_r(libpython::_object*, bool)@plt () from /dwelling/key/R/x86_64-redhat-linux-gnu-library/3.6/reticulate/libs/reticulate.so

Here are the related (for our investigation) frames of the backtrace:

Thread 1 "R" hit Breakpoint 3, 0x00007fffe48315d0 in py_to_r(libpython::_object*, bool)@plt () from /dwelling/key/R/x86_64-redhat-linux-gnu-library/3.6/reticulate/libs/reticulate.so
(gdb) bt
#0  0x00007fffe48315d0 in py_to_r(libpython::_object*, bool)@plt () from /dwelling/key/R/x86_64-redhat-linux-gnu-library/3.6/reticulate/libs/reticulate.so
#1  0x00007fffe48588a0 in py_ref_to_r_with_convert (x=..., convert=true) at reticulate_types.h:32
#2  0x00007fffe4858963 in py_ref_to_r (x=...) at /dwelling/key/R/x86_64-redhat-linux-gnu-library/3.6/Rcpp/embody/RcppFrequent.h:120
#3  0x00007fffe483d7a9 in _reticulate_py_ref_to_r (xSEXP=0x55555daa7e50) at /dwelling/key/R/x86_64-redhat-linux-gnu-library/3.6/Rcpp/embody/Rcpp/as.h:151
...
...
#14 0x00007ffff7cc5fc7 in Rf_usemethod (generic=0x55555757ce70 "py_to_r", obj=obj@entry=0x55555daa7e50, name=name@entry=0x55555a0fe198, args=args@entry=0x55555557c4e0, 
    rho=rho@entry=0x55555dab2ed0, callrho=0x55555dab48d8, defrho=0x5555575a4068, ans=0x7fffffff69e8) at objects.c:486

We’ve eliminated a number of intermediate frames associated to (R-level) methodology dispatch.

As we already noticed within the supply code, py_to_r.default will delegate to a way known as py_ref_to_r, which we see seems in #2. But what’s _reticulate_py_ref_to_r in #3, the body just under? Here is the place the magic, unseen by the person, begins.

Let’s have a look at this from a chicken’s eye’s view. To translate an object from one language to a different, we have to discover a frequent floor, that’s, a 3rd language “spoken” by each of them. In the case of R and Python (in addition to in a number of different instances) this can be C / C++. So assuming we’re going to write a C operate to speak to Python, how can we use this operate in R?

While R customers have the power to name into C instantly, utilizing .Call or .External , that is made far more handy by Rcpp : You simply write your C++ operate, and Rcpp takes care of compilation and supplies the glue code essential to name this operate from R.

So py_ref_to_r actually is written in C++:

// [[Rcpp::export]]
SEXP py_ref_to_r(PyObjectRef x) {
  return py_ref_to_r_with_convert(x, x.convert());
}

however the remark // [[Rcpp::export]] tells Rcpp to generate an R wrapper, py_ref_to_R, that itself calls a C++ wrapper, _reticulate_py_ref_to_r …

py_ref_to_r <- operate(x) {
  .Call(`_reticulate_py_ref_to_r`, x)
}

which lastly wraps the “real” factor, the C++ operate py_ref_to_R we noticed above.

Via py_ref_to_r_with_convert in #1, a one-liner that extracts an object’s “convert” characteristic (see under)

// [[Rcpp::export]]
SEXP py_ref_to_r_with_convert(PyObjectRef x, bool convert) {
  return py_to_r(x, convert);
}

we lastly arrive at py_to_r in #0.

Before we have a look at that, let’s ponder that C/C++ “bridge” from the opposite facet – Python.
While strictly, Python is a language specification, its reference implementation is CPython, with a core written in C and far more performance constructed on high in Python. In CPython, each Python object (together with integers or different numeric sorts) is a PyObject. PyObjects are allotted by way of and operated on utilizing pointers; most C API features return a pointer to at least one, PyObject *.

So that is what we anticipate to work with, from R. What then is PyObjectRef doing in py_ref_to_r?
PyObjectRef is just not a part of the C API, it’s a part of the performance launched by reticulate to handle Python objects. Its fundamental objective is to verify the Python object is mechanically cleaned up when the R object (an Rcpp::Environment) goes out of scope.
Why use an R surroundings to wrap the Python-level pointer? This is as a result of R environments can have finalizers: features which can be known as earlier than objects are rubbish collected.
We use this R-level finalizer to make sure the Python-side object will get finalized as effectively:

Rcpp::RObject xptr = R_MakeExternalPtr((void*) object, R_NilValue, R_NilValue);
R_RegisterCFinalizer(xptr, python_object_finalize);

python_object_finalize is fascinating, because it tells us one thing essential about Python – about CPython, to be exact: To discover out if an object continues to be wanted, or might be rubbish collected, it makes use of reference counting, thus inserting on the person the burden of appropriately incrementing and decrementing references in accordance with language semantics.

inline void python_object_finalize(SEXP object) {
  PyObject* pyObject = (PyObject*)R_ExternalPtrAddr(object);
  if (pyObject != NULL)
    Py_DecRef(pyObject);
}

Resuming on PyObjectRef, word that it additionally shops the “convert” characteristic of the Python object, used to find out whether or not that object needs to be transformed to R mechanically.

Back to py_to_r. This one now actually will get to work with (a pointer to the) Python object,

SEXP py_to_r(PyObject* x, bool convert) {
  //...
}

and – however wait. Didn’t py_ref_to_r_with_convert cross it a PyObjectRef? So how come it receives a PyObject as a substitute? This is as a result of PyObjectRef inherits from Rcpp::Environment, and its implicit conversion operator is used to extract the Python object from the Environment. Concretely, that operator tells the compiler {that a} PyObjectRef can be utilized as if it have been a PyObject* in some ideas, and the related code specifies how one can convert from PyObjectRef to PyObject*:

operator PyObject*() const {
  return get();
}

PyObject* get() const {
  SEXP pyObject = getFromEnvironment("pyobj");
  if (pyObject != R_NilValue) {
    PyObject* obj = (PyObject*)R_ExternalPtrAddr(pyObject);
    if (obj != NULL)
      return obj;
  }
  Rcpp::cease("Unable to entry object (object is from earlier session and is now invalid)");
}

So py_to_r works with a pointer to a Python object and returns what we would like, an R object (a SEXP).
The operate checks for the kind of the article, after which makes use of Rcpp to assemble the sufficient R object, in our case, an integer:

else if (scalarType == INTSXP)
  return IntegerVector::create(PyInt_AsLengthy(x));

For different objects, sometimes there’s extra motion required; however primarily, the operate is “just” a giant if–else tree.

So this was situation 1: changing a Python object to R. Now in situation 2, we assume we nonetheless have to create that Python object.

Scenario 2:

As this situation is significantly extra advanced than the earlier one, we are going to explicitly think about some points and omit others. Importantly, we’ll not go into module loading, which might deserve separate therapy of its personal. Instead, we attempt to shed a lightweight on what’s concerned utilizing a concrete instance: the ever present, in keras code, keras_model_sequential(). All this R operate does is

operate(layers = NULL, identify = NULL) {
  keras$fashions$Sequential(layers = layers, identify = identify)
}

How can keras$fashions$Sequential() give us an object? When in Python, you run the equal

tf.keras.fashions.Sequential()

this calls the constructor, that’s, the __init__ methodology of the category:

class Sequential(coaching.Model):
  def __init__(self, layers=None, identify=None):
    # ...
  # ...

So this time, earlier than – as all the time, ultimately – getting an R object again from Python, we have to name that constructor, that’s, a Python callable. (Python callables subsume features, constructors, and objects created from a category that has a name methodology.)

So when py_to_r, inspecting its argument’s sort, sees it’s a Python callable (wrapped in a PyObjectRef, the reticulate-specific subclass of Rcpp::Environment we talked about above), it wraps it (the PyObjectRef) in an R operate, utilizing Rcpp:

Rcpp::Function f = py_callable_as_function(pyFunc, convert);

The cpython-side motion begins when py_callable_as_function then calls py_call_impl. py_call_impl executes the precise name and returns an R object, a SEXP. Now chances are you’ll be asking, how does the Python runtime realize it shouldn’t deallocate that object, now that its work is finished? This is taken of by the identical PyObjectRef class used to wrap cases of PyObject *: It can wrap SEXPs as effectively.

While much more might be mentioned about what occurs earlier than we lastly get to work with that Sequential mannequin from R, let’s cease right here and have a look at our third situation.

Scenario 3: Calling R from Python

Not surprisingly, generally we have to cross R callbacks to Python. An instance are R knowledge mills that can be utilized with keras fashions .

In common, for R objects to be handed to Python, the method is considerably reverse to what we described in instance 1. Say we sort:

This assigns 1 to a variable a within the python fundamental module.
To allow project, reticulate supplies an implementation of the S3 generic $<-, $<-.python.builtin.object, which delegates to py_set_attr, which then calls py_set_attr_impl – one more C++ operate exported by way of Rcpp.

Let’s concentrate on a special facet right here, although. A prerequisite for the project to occur is getting that 1 transformed to Python. (We’re utilizing the best potential instance, clearly; however you may think about this getting much more advanced if the article isn’t a easy quantity).

For our “minimal example”, we see a stacktrace like the next

#0 0x00007fffe4832010 in r_to_py_cpp(Rcpp::RObject_Impl<Rcpp::PreserveStorage>, bool)@plt () from /dwelling/key/R/x86_64-redhat-linux-gnu-library/3.6/reticulate/libs/reticulate.so
#1  0x00007fffe4854f38 in r_to_py_impl (object=..., convert=convert@entry=true) at /dwelling/key/R/x86_64-redhat-linux-gnu-library/3.6/Rcpp/embody/RcppFrequent.h:120
#2  0x00007fffe48418f3 in _reticulate_r_to_py_impl (objectSEXP=0x55555ec88fa8, convertSEXP=<optimized out>) at /dwelling/key/R/x86_64-redhat-linux-gnu-library/3.6/Rcpp/embody/Rcpp/as.h:151
...
#12 0x00007ffff7cc5c03 in dispatchMethod (sxp=0x55555d0cf1a0, dotClass=<optimized out>, cptr=cptr@entry=0x7ffffffeaae0, methodology=methodology@entry=0x55555bfe06c0, 
    generic=0x555557634458 "r_to_py", rho=0x55555d1d98a8, callrho=0x5555555af2d0, defrho=0x555557947430, op=<optimized out>, op=<optimized out>) at objects.c:436
#13 0x00007ffff7cc5fc7 in Rf_usemethod (generic=0x555557634458 "r_to_py", obj=obj@entry=0x55555ec88fa8, name=name@entry=0x55555c0317b8, args=args@entry=0x55555557cc60, 
    rho=rho@entry=0x55555d1d98a8, callrho=0x5555555af2d0, defrho=0x555557947430, ans=0x7ffffffe9928) at objects.c:486

Whereas r_to_py is a generic (like py_to_r above), r_to_py_impl is wrapped by Rcpp and r_to_py_cpp is a C++ operate that branches on the kind of the article – principally the counterpart of the C++ r_to_py.

In addition to that common course of, there’s extra happening once we name an R operate from Python. As Python doesn’t “speak” R, we have to wrap the R operate in CPython – principally, we’re extending Python right here! How to do that is described within the official Extending Python Guide.

In official phrases, what reticulate does it embed and prolong Python.
Embed, as a result of it helps you to use Python from inside R. Extend, as a result of to allow Python to name again into R it must wrap R features in C, so Python can perceive them.

As a part of the previous, the specified Python is loaded (Py_Initialize()); as a part of the latter, two features are outlined in a brand new module named rpycall, that can be loaded when Python itself is loaded.

PyImport_AppendInittab("rpycall", &initializeRPYCall);

These strategies are call_r_function, utilized by default, and call_python_function_on_main_thread, utilized in instances the place we’d like to verify the R operate is known as on the principle thread:

PyMethodDef RPYCallStrategies[] = {
   METH_KEYWORDS, "Call an R operate" ,
   METH_KEYWORDS, "Call a Python operate on the principle thread" ,
  { NULL, NULL, 0, NULL }
};

call_python_function_on_main_thread is particularly fascinating. The R runtime is single-threaded; whereas the CPython implementation of Python successfully is as effectively, as a result of Global Interpreter Lock, this isn’t mechanically the case when different implementations are used, or C is used instantly. So call_python_function_on_main_thread makes positive that until we are able to execute on the principle thread, we wait.

That’s it for our three “spotlights on reticulate”.

Wrapup

It goes with out saying that there’s loads about reticulate we didn’t cowl on this article, akin to reminiscence administration, initialization, or specifics of knowledge conversion. Nonetheless, we hope we have been in a position to shed a bit of sunshine on the magic concerned in calling TensorStream from R.

R is a concise and chic language, however to a excessive diploma its energy comes from its packages, together with those who will let you name into, and work together with, the surface world, akin to deep studying frameworks or distributed processing engines. In this publish, it was a particular pleasure to concentrate on a central constructing block that makes a lot of this potential: reticulate.

Thanks for studying!