TensorFlow Tutorial

TensorFlow is a powerful library for numerical computation, particularly well suited and fine-tuned for large–scale Machine Learning ( but you could use it for anything else that requires heavy calculations). The Google Brain team developed it, and it powers many of Google’s large-scale services, such as Google cloud speech, Google Photos, and Google Search.

It was open-sourced in November 2015, and it is now the most popular Deep Learning library ( in terms of citations in papers, adoption in companies, stars on GitHub, etc.). Countless projects use Tensorflow for all sorts of Machine Learning tasks, such as image classification, natural language processing, recommender systems, and time series forecasting.

Features of TensorFlow

  • Its core is very similar to Numpy, but with GPU support.
  • It supports distributed computing (across multiple devices and servers).
  • It includes a kind of just –in –time (JIT) Compiler that allows it to optimize computations for speed and memory usage. It works by extracting the computation graph from a python function, then optimizing it (e.g., by pruning unused nodes), and finally running it efficiently (e.g., by automatically running independent operations in parallel).
  • Computation graphs can be exported to a portable format so that you can train a Tensorflow model in one environment (e.g., using python on Linux) and run it in another (e.g., using java on an Android device).
  • It implements auto diff (see chapter 10 and Appendix D) and provides some excellent optimizers, such as RMSProp and Nadam, to quickly minimize all sorts of loss functions.

TensorFlow offers many more features built on top of these core features; the most important is, of course, tf.keras, but it also has data loading and preprocessing ops (tf.data, tf. Io, etc.), image processing ops (tf.image), signal processing ops ( tf.signal ), and more.

TensorFlow Tutorial

Before moving forward, I will import some libraries, that we need to operate with TensorFlow.

# TensorFlow ≥2.0 is required import tensorflow as tf from tensorflow import keras assert tf.__version__ >= "2.0" # Common imports import numpy as np import os # to make this notebook's output stable across runs np.random.seed(42) tf.random.set_seed(42) # To plot pretty figures %matplotlib inline import matplotlib as mpl import matplotlib.pyplot as plt mpl.rc('axes', labelsize=14) mpl.rc('xtick', labelsize=12) mpl.rc('ytick', labelsize=12)

The API of TensorFlow revolves around tensors, which flow from operation to operation, hence the name TensorFlow. A tensor is usually a multidimensional array (exactly like a numpy ndarray), but it can hold a scalar ( a simple value such as 42). These tensors will be important when we create custom cost functions, custom metrics, custom layers, and more, so let’s see how to create and manipulate them.

Also, read – Introduction to Reinforcement Learning.

Tensors and Operations

You can create a tensor with tf.constant(). For example, here is a tensor representing a matrix with two rows and three columns of floats:

tf.constant([[1., 2., 3.], [4., 5., 6.]]) # matrix
<tf.Tensor: shape=(2, 3), dtype=float32, numpy= array([[1., 2., 3.], [4., 5., 6.]], dtype=float32)>
tf.constant(42) # scalar
<tf.Tensor: shape=(), dtype=int32, numpy=42>

Just like an ndarray, a tf.Tensor has a shape and data type (dtype):

t = tf.constant([[1., 2., 3.], [4., 5., 6.]]) t.shape
TensorShape([2, 3])

Indexing works much like in Numpy:

t[:, 1:]
<tf.Tensor: shape=(2, 2), dtype=float32, numpy= array([[2., 3.], [5., 6.]], dtype=float32)>
t[..., 1, tf.newaxis]
<tf.Tensor: shape=(2, 1), dtype=float32, numpy= array([[2.], [5.]], dtype=float32)>

Most importantly, all sorts of tensor operations are available:

t + 10
<tf.Tensor: shape=(2, 3), dtype=float32, numpy= array([[11., 12., 13.], [14., 15., 16.]], dtype=float32)>
<tf.Tensor: shape=(2, 3), dtype=float32, numpy= array([[ 1., 4., 9.], [16., 25., 36.]], dtype=float32)>
t @ tf.transpose(t)
<tf.Tensor: shape=(2, 2), dtype=float32, numpy= array([[14., 32.], [32., 77.]], dtype=float32)>

Tensors and NumPy

Tensors play nice with NumPy: you can create a tensor from a NumPy array, and vice versa. You can even apply TensorFlow operations to NumPy arrays and NumPy operations to tensors:

a = np.array([2., 4., 5.]) tf.constant(a)
<tf.Tensor: shape=(3,), dtype=float64, numpy=array([2., 4., 5.])>
array([[1., 2., 3.], [4., 5., 6.]], dtype=float32)
<tf.Tensor: shape=(3,), dtype=float64, numpy=array([ 4., 16., 25.])>
array([[1., 2., 3.], [4., 5., 6.]], dtype=float32)

Notice that NumPy uses 64-bit precision by default, while TensorFlow uses 32-bit. This is because 32-bit precision is generally ,ore than enough for neural networks, plus it runs faster and uses less RAM. So when you create a tensor from a NumPy array, make sure to set dtype=tf.float32.

Customizing Models and Training Algorithms in TensorFlow

Let’s start by creating a custom loss function, which is a simple common use case. I will start by loading and preparing the California housing dataset. I will first load it, then split it into a training set, a validation set and a test set, and finally we scale it:

from sklearn.datasets import fetch_california_housing from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler housing = fetch_california_housing() X_train_full, X_test, y_train_full, y_test = train_test_split( housing.data, housing.target.reshape(-1, 1), random_state=42) X_train, X_valid, y_train, y_valid = train_test_split( X_train_full, y_train_full, random_state=42) scaler = StandardScaler() X_train_scaled = scaler.fit_transform(X_train) X_valid_scaled = scaler.transform(X_valid) X_test_scaled = scaler.transform(X_test)

Custom Loss Functions

Suppose you want to train a regression model, but your training set is a bit noisy. Of course, you start by trying to clean up your dataset by removing or fixing the outliers, but that turns out to be insufficient, your dataset is still noisy. Which loss function you should use? This is probably the best time to use the Huber loss insted of the good MSE.

The Huber loss is not currently part of the official Keras API but is available in tf.keras. But let’s pretend it’s not there. Just create a function that takes the labels and predictions as arguments, and use TensorFlow operations to compute every instance’s loss:

def huber_fn(y_true, y_pred): error = y_true - y_pred is_small_error = tf.abs(error) < 1 squared_loss = tf.square(error) / 2 linear_loss = tf.abs(error) - 0.5 return tf.where(is_small_error, squared_loss, linear_loss)

Now, lets visualize our model after the Huber loss function:

plt.figure(figsize=(8, 3.5)) z = np.linspace(-4, 4, 200) plt.plot(z, huber_fn(0, z), "b-", linewidth=2, label="huber($z$)") plt.plot(z, z**2 / 2, "b:", linewidth=1, label=r"$frac{1}{2}z^2$") plt.plot([-1, -1], [0, huber_fn(0., -1.)], "r--") plt.plot([1, 1], [0, huber_fn(0., 1.)], "r--") plt.gca().axhline(y=0, color='k') plt.gca().axvline(x=0, color='k') plt.axis([-4, 4, 0, 4]) plt.grid(True) plt.xlabel("$z$") plt.legend(fontsize=14) plt.title("Huber loss", fontsize=14) plt.show()

Now you can use the Huber loss function when you compile the keras model, then train your model:

input_shape = X_train.shape[1:] model = keras.models.Sequential([ keras.layers.Dense(30, activation="selu", kernel_initializer="lecun_normal", input_shape=input_shape), keras.layers.Dense(1), ]) model.compile(loss=huber_fn, optimizer="nadam", metrics=["mae"]) model.fit(X_train_scaled, y_train, epochs=2, validation_data=(X_valid_scaled, y_valid))
Train on 11610 samples, validate on 3870 samples Epoch 1/2 11610/11610 [==============================] - 1s 73us/sample - loss: 0.6247 - mae: 0.9969 - val_loss: 0.2884 - val_mae: 0.5913 Epoch 2/2 11610/11610 [==============================] - 0s 32us/sample - loss: 0.2193 - mae: 0.5176 - val_loss: 0.2361 - val_mae: 0.5232
<tensorflow.python.keras.callbacks.History at 0x7f9fa6099f10>

Saving and Loading Tensorflow Models

Saving a model containing a custom loss function works fine, as keras saves the name of the function. Whenever you load it, you will need to provide a dictionary that maps the function name to the actual function. More generally, when you load a model containing objects, you need to map the names of the objects:

model.save("my_model_with_a_custom_loss.h5") model = keras.models.load_model("my_model_with_a_custom_loss.h5", custom_objects={"huber_fn": huber_fn}) model.fit(X_train_scaled, y_train, epochs=2, validation_data=(X_valid_scaled, y_valid))
Train on 11610 samples, validate on 3870 samples Epoch 1/2 11610/11610 [==============================] - 1s 55us/sample - loss: 0.2056 - mae: 0.4982 - val_loss: 0.2170 - val_mae: 0.5037 Epoch 2/2 11610/11610 [==============================] - 0s 35us/sample - loss: 0.2006 - mae: 0.4911 - val_loss: 0.2097 - val_mae: 0.4908
<tensorflow.python.keras.callbacks.History at 0x7f9f800da650>

With the current implementations, any error between -1 and 1 is considered “small.” But what if you want a different threshold? One solution to create a function that creates a configured loss function:

def create_huber(threshold=1.0): def huber_fn(y_true, y_pred): error = y_true - y_pred is_small_error = tf.abs(error) < threshold squared_loss = tf.square(error) / 2 linear_loss = threshold * tf.abs(error) - threshold**2 / 2 return tf.where(is_small_error, squared_loss, linear_loss) return huber_fn model.compile(loss=create_huber(2.0), optimizer="nadam", metrics=["mae"]) model.fit(X_train_scaled, y_train, epochs=2, validation_data=(X_valid_scaled, y_valid))
Train on 11610 samples, validate on 3870 samples Epoch 1/2 11610/11610 [==============================] - 1s 69us/sample - loss: 0.2229 - mae: 0.4893 - val_loss: 0.2525 - val_mae: 0.4973 Epoch 2/2 11610/11610 [==============================] - 0s 32us/sample - loss: 0.2189 - mae: 0.4856 - val_loss: 0.2338 - val_mae: 0.4765
<tensorflow.python.keras.callbacks.History at 0x7f9f50093a50>
model.save("my_model_with_a_custom_loss_threshold_2.h5") model = keras.models.load_model("my_model_with_a_custom_loss_threshold_2.h5", custom_objects={"huber_fn": create_huber(2.0)}) model.fit(X_train_scaled, y_train, epochs=2, validation_data=(X_valid_scaled, y_valid))
Train on 11610 samples, validate on 3870 samples Epoch 1/2 11610/11610 [==============================] - 1s 58us/sample - loss: 0.2148 - mae: 0.4796 - val_loss: 0.2111 - val_mae: 0.4713 Epoch 2/2 11610/11610 [==============================] - 0s 34us/sample - loss: 0.2123 - mae: 0.4775 - val_loss: 0.1970 - val_mae: 0.4534
<tensorflow.python.keras.callbacks.History at 0x7f9fa61cce90>

Also, read – 10 Machine Learning Projects to Boost your Portfolio.

I hope you liked this article on TensorFlow Tutorial. Feel free to ask questions on this topic or any other topic that you want in the comments section below.

Receive Daily Newsletters

Leave a Reply