TensorFlow is a powerful library for numerical computation, particularly well suited and fine-tuned for large–scale Machine Learning ( but you could use it for anything else that requires heavy calculations). The Google Brain team developed it, and it powers many of Google’s large-scale services, such as Google cloud speech, Google Photos, and Google Search.
It was open-sourced in November 2015, and it is now the most popular Deep Learning library ( in terms of citations in papers, adoption in companies, stars on GitHub, etc.). Countless projects use Tensorflow for all sorts of Machine Learning tasks, such as image classification, natural language processing, recommender systems, and time series forecasting.
Features of TensorFlow
- Its core is very similar to Numpy, but with GPU support.
- It supports distributed computing (across multiple devices and servers).
- It includes a kind of just –in –time (JIT) Compiler that allows it to optimize computations for speed and memory usage. It works by extracting the computation graph from a python function, then optimizing it (e.g., by pruning unused nodes), and finally running it efficiently (e.g., by automatically running independent operations in parallel).
- Computation graphs can be exported to a portable format so that you can train a Tensorflow model in one environment (e.g., using python on Linux) and run it in another (e.g., using java on an Android device).
- It implements auto diff (see chapter 10 and Appendix D) and provides some excellent optimizers, such as RMSProp and Nadam, to quickly minimize all sorts of loss functions.
TensorFlow offers many more features built on top of these core features; the most important is, of course, tf.keras, but it also has data loading and preprocessing ops (tf.data, tf. Io, etc.), image processing ops (tf.image), signal processing ops ( tf.signal ), and more.
TensorFlow Tutorial
Before moving forward, I will import some libraries, that we need to operate with TensorFlow.
# TensorFlow ≥2.0 is required
import tensorflow as tf
from tensorflow import keras
assert tf.__version__ >= "2.0"
# Common imports
import numpy as np
import os
# to make this notebook's output stable across runs
np.random.seed(42)
tf.random.set_seed(42)
# To plot pretty figures
%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
mpl.rc('axes', labelsize=14)
mpl.rc('xtick', labelsize=12)
mpl.rc('ytick', labelsize=12)Code language: Python (python)The API of TensorFlow revolves around tensors, which flow from operation to operation, hence the name TensorFlow. A tensor is usually a multidimensional array (exactly like a numpy ndarray), but it can hold a scalar ( a simple value such as 42). These tensors will be important when we create custom cost functions, custom metrics, custom layers, and more, so let’s see how to create and manipulate them.
Also, read – Introduction to Reinforcement Learning.
Tensors and Operations
You can create a tensor with tf.constant(). For example, here is a tensor representing a matrix with two rows and three columns of floats:
tf.constant([[1., 2., 3.], [4., 5., 6.]]) # matrixCode language: Python (python)<tf.Tensor: shape=(2, 3), dtype=float32, numpy= array([[1., 2., 3.], [4., 5., 6.]], dtype=float32)>
tf.constant(42) # scalarCode language: Python (python)<tf.Tensor: shape=(), dtype=int32, numpy=42>
Just like an ndarray, a tf.Tensor has a shape and data type (dtype):
t = tf.constant([[1., 2., 3.], [4., 5., 6.]])
t.shapeCode language: Python (python)TensorShape([2, 3])
t.dtypeCode language: Python (python)tf.float32
Indexing works much like in Numpy:
t[:, 1:]Code language: Python (python)<tf.Tensor: shape=(2, 2), dtype=float32, numpy= array([[2., 3.], [5., 6.]], dtype=float32)>
t[..., 1, tf.newaxis]Code language: Python (python)<tf.Tensor: shape=(2, 1), dtype=float32, numpy= array([[2.], [5.]], dtype=float32)>
Most importantly, all sorts of tensor operations are available:
t + 10Code language: Python (python)<tf.Tensor: shape=(2, 3), dtype=float32, numpy= array([[11., 12., 13.], [14., 15., 16.]], dtype=float32)>
tf.square(t)Code language: Python (python)<tf.Tensor: shape=(2, 3), dtype=float32, numpy= array([[ 1., 4., 9.], [16., 25., 36.]], dtype=float32)>
t @ tf.transpose(t)Code language: Python (python)<tf.Tensor: shape=(2, 2), dtype=float32, numpy= array([[14., 32.], [32., 77.]], dtype=float32)>
Tensors and NumPy
Tensors play nice with NumPy: you can create a tensor from a NumPy array, and vice versa. You can even apply TensorFlow operations to NumPy arrays and NumPy operations to tensors:
a = np.array([2., 4., 5.])
tf.constant(a)Code language: Python (python)<tf.Tensor: shape=(3,), dtype=float64, numpy=array([2., 4., 5.])>
t.numpy()Code language: Python (python)array([[1., 2., 3.], [4., 5., 6.]], dtype=float32)
tf.square(a)Code language: Python (python)<tf.Tensor: shape=(3,), dtype=float64, numpy=array([ 4., 16., 25.])>
np.array(t)Code language: Python (python)array([[1., 2., 3.], [4., 5., 6.]], dtype=float32)
Notice that NumPy uses 64-bit precision by default, while TensorFlow uses 32-bit. This is because 32-bit precision is generally ,ore than enough for neural networks, plus it runs faster and uses less RAM. So when you create a tensor from a NumPy array, make sure to set dtype=tf.float32.
Customizing Models and Training Algorithms in TensorFlow
Let’s start by creating a custom loss function, which is a simple common use case. I will start by loading and preparing the California housing dataset. I will first load it, then split it into a training set, a validation set and a test set, and finally we scale it:
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
housing = fetch_california_housing()
X_train_full, X_test, y_train_full, y_test = train_test_split(
housing.data, housing.target.reshape(-1, 1), random_state=42)
X_train, X_valid, y_train, y_valid = train_test_split(
X_train_full, y_train_full, random_state=42)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_valid_scaled = scaler.transform(X_valid)
X_test_scaled = scaler.transform(X_test)Code language: Python (python)Custom Loss Functions
Suppose you want to train a regression model, but your training set is a bit noisy. Of course, you start by trying to clean up your dataset by removing or fixing the outliers, but that turns out to be insufficient, your dataset is still noisy. Which loss function you should use? This is probably the best time to use the Huber loss instead of the good MSE.
The Huber loss is not currently part of the official Keras API but is available in tf.keras. But let’s pretend it’s not there. Just create a function that takes the labels and predictions as arguments, and use TensorFlow operations to compute every instance’s loss:
def huber_fn(y_true, y_pred):
error = y_true - y_pred
is_small_error = tf.abs(error) < 1
squared_loss = tf.square(error) / 2
linear_loss = tf.abs(error) - 0.5
return tf.where(is_small_error, squared_loss, linear_loss)Code language: Python (python)Now, lets visualize our model after the Huber loss function:
plt.figure(figsize=(8, 3.5))
z = np.linspace(-4, 4, 200)
plt.plot(z, huber_fn(0, z), "b-", linewidth=2, label="huber($z$)")
plt.plot(z, z**2 / 2, "b:", linewidth=1, label=r"$frac{1}{2}z^2$")
plt.plot([-1, -1], [0, huber_fn(0., -1.)], "r--")
plt.plot([1, 1], [0, huber_fn(0., 1.)], "r--")
plt.gca().axhline(y=0, color='k')
plt.gca().axvline(x=0, color='k')
plt.axis([-4, 4, 0, 4])
plt.grid(True)
plt.xlabel("$z$")
plt.legend(fontsize=14)
plt.title("Huber loss", fontsize=14)
plt.show()Code language: Python (python)
Now you can use the Huber loss function when you compile the keras model, then train your model:
input_shape = X_train.shape[1:]
model = keras.models.Sequential([
keras.layers.Dense(30, activation="selu", kernel_initializer="lecun_normal",
input_shape=input_shape),
keras.layers.Dense(1),
])
model.compile(loss=huber_fn, optimizer="nadam", metrics=["mae"])
model.fit(X_train_scaled, y_train, epochs=2,
validation_data=(X_valid_scaled, y_valid))Code language: Python (python)Train on 11610 samples, validate on 3870 samples Epoch 1/2 11610/11610 [==============================] - 1s 73us/sample - loss: 0.6247 - mae: 0.9969 - val_loss: 0.2884 - val_mae: 0.5913 Epoch 2/2 11610/11610 [==============================] - 0s 32us/sample - loss: 0.2193 - mae: 0.5176 - val_loss: 0.2361 - val_mae: 0.5232 <tensorflow.python.keras.callbacks.History at 0x7f9fa6099f10>
Saving and Loading Tensorflow Models
Saving a model containing a custom loss function works fine, as keras saves the name of the function. Whenever you load it, you will need to provide a dictionary that maps the function name to the actual function. More generally, when you load a model containing objects, you need to map the names of the objects:
model.save("my_model_with_a_custom_loss.h5")
model = keras.models.load_model("my_model_with_a_custom_loss.h5",
custom_objects={"huber_fn": huber_fn})
model.fit(X_train_scaled, y_train, epochs=2,
validation_data=(X_valid_scaled, y_valid))Code language: Python (python)Train on 11610 samples, validate on 3870 samples Epoch 1/2 11610/11610 [==============================] - 1s 55us/sample - loss: 0.2056 - mae: 0.4982 - val_loss: 0.2170 - val_mae: 0.5037 Epoch 2/2 11610/11610 [==============================] - 0s 35us/sample - loss: 0.2006 - mae: 0.4911 - val_loss: 0.2097 - val_mae: 0.4908 <tensorflow.python.keras.callbacks.History at 0x7f9f800da650>
With the current implementations, any error between -1 and 1 is considered “small.” But what if you want a different threshold? One solution to create a function that creates a configured loss function:
def create_huber(threshold=1.0):
def huber_fn(y_true, y_pred):
error = y_true - y_pred
is_small_error = tf.abs(error) < threshold
squared_loss = tf.square(error) / 2
linear_loss = threshold * tf.abs(error) - threshold**2 / 2
return tf.where(is_small_error, squared_loss, linear_loss)
return huber_fn
model.compile(loss=create_huber(2.0), optimizer="nadam", metrics=["mae"])
model.fit(X_train_scaled, y_train, epochs=2,
validation_data=(X_valid_scaled, y_valid))Code language: Python (python)Train on 11610 samples, validate on 3870 samples Epoch 1/2 11610/11610 [==============================] - 1s 69us/sample - loss: 0.2229 - mae: 0.4893 - val_loss: 0.2525 - val_mae: 0.4973 Epoch 2/2 11610/11610 [==============================] - 0s 32us/sample - loss: 0.2189 - mae: 0.4856 - val_loss: 0.2338 - val_mae: 0.4765 <tensorflow.python.keras.callbacks.History at 0x7f9f50093a50>
model.save("my_model_with_a_custom_loss_threshold_2.h5")
model = keras.models.load_model("my_model_with_a_custom_loss_threshold_2.h5",
custom_objects={"huber_fn": create_huber(2.0)})
model.fit(X_train_scaled, y_train, epochs=2,
validation_data=(X_valid_scaled, y_valid))Code language: Python (python)Train on 11610 samples, validate on 3870 samples Epoch 1/2 11610/11610 [==============================] - 1s 58us/sample - loss: 0.2148 - mae: 0.4796 - val_loss: 0.2111 - val_mae: 0.4713 Epoch 2/2 11610/11610 [==============================] - 0s 34us/sample - loss: 0.2123 - mae: 0.4775 - val_loss: 0.1970 - val_mae: 0.4534 <tensorflow.python.keras.callbacks.History at 0x7f9fa61cce90>
Also, read – 10 Machine Learning Projects to Boost your Portfolio.
I hope you liked this article on TensorFlow Tutorial. Feel free to ask questions on this topic or any other topic that you want in the comments section below.
