The Anatomy of Deep Learning Frameworks* *Everything you wanted to know about DL Frameworks but were afraid to ask $whoami Master’s Student in CS @ ETH Zurich, bachelors ● from BITS Pilani Contributor to Theano ● Working on DL for Astronomy and GANs ● space.ml, Anatomy of Deep Learning Frameworks ● What’s this talk about? Understanding the internals of DL frameworks ● Common components of all DLFs ● Old wine in a new bottle ● How you could DIY ● Why do you need to know this? All DL work done in DLFs, no vanilla code ● TF, Keras, Theano backbone of DL research ● Any sufficiently advanced technology is ● indistinguishable from magic --Arthur C. Clarke Not voodoo ● Simple concepts, complex implementation details ● Once you know it, you get more control ● Frameworks Galore Many names, same concepts Why do we have so many frameworks? ● – Because, why not? – Theano – MILA, Torch – FB, CNTK – MS, TF – Google… – Does something well Are they all really that different? ● – NO – We’ll cover this in the talk Bare-bones DL Framework Components of any DL framework ● – Tensors – Operations – Computation Graph – Auto-differentiation – Fast and Efficient floating pt. Operations, GPU support BLAS, cuBLAS, cuDNN ● Example in TensorFlow Tensor: tf.Tensor ● Ops: tf.Operation ● Graph: tf.Graph ● Autodiff: tf.gradients et al. ● CuDNN, BLAS – See install notes ● Example in Theano Tensor: theano.tensor ● Ops: theano.* ● Graph: theano.gof.graph ● Autodiff: theano.tensor.grad ● CuDNN / BLAS: GPU Backend ● Tensors Tensors – Mathematical objects ● Simply, N-Dimensional Arrays, like numpy.ndarray ● lingua franca in DL frameworks ● Data → Input Tensors → DNN → Output Tensors → ● Results Clean abstraction, allows use in different scenarios ● DNN sees only tensors, not images, audio, text... ●