Adversarial Learning with Keras and TensorFlow (Part 2): Implementing the Neural Structured Learning (NSL) Framework and Building a Data Pipeline

---

Table of Contents

---

Adversarial Learning with Keras and TensorFlow (Part 2): Implementing the Neural Structured Learning (NSL) Framework and Building a Data Pipeline

In this tutorial, you will learn about the TensorFlow NSL (neural structured learning) framework and how it can enhance our training pipeline and allow us to train better and more robust models. Specifically, we will take a deep dive into the salient features of the NSL framework and understand how it can be used to make models robust to adversarial attacks and perturbations with the help of adversarial training.

Furthermore, we will start building our end-to-end adversarial learning application and discuss details about the dataset we will use and implement the data pipeline using Keras and TensorFlow.

This lesson is the 2nd of a 4-part series on Adversarial Learning:

1. Adversarial Learning with Keras and TensorFlow (Part 1): Overview of Adversarial Learning
2. Adversarial Learning with Keras and TensorFlow (Part 2): Implementing the Neural Structured Learning (NSL) Framework and Building a Data Pipeline (this tutorial)
3. Adversarial Learning with Keras and TensorFlow (Part 3): Exploring Adversarial Attacks Using Neural Structured Learning (NSL)
4. Adversarial Learning with Keras and TensorFlow (Part 4): Enhancing Adversarial Defense and Comparing Models Trained With and Without Neural Structured Learning (NSL)

To learn about the TensorFlow NSL framework, just keep reading.

---

Adversarial Learning with Keras and TensorFlow (Part 2): Implementing the Neural Structured Learning (NSL) Framework and Building a Data Pipeline

The TensorFlow NSL framework allows neural networks to learn with structured data. It provides a simple, easy-to-use API that seamlessly integrates with any computer vision project workflow and enables us to train better and more robust networks. Basically, NSL tries to augment the workflow of a network with additional structured signals, which help to regularize the training and allow us to train better and more reliable models.

Let us try to understand in detail how the NSL framework works.

Figure 1 (yellow) shows a typical neural network workflow for classifying cats and dogs. Given an input dog image, the network computes features and activations and later outputs the predicted class for the input image (i.e., dog here).

Now, let us understand how the NSL framework tries to enhance this workflow.

The NSL framework takes two inputs (Figure 1, red):

• Input images or features
• Structured signals

The structured signal here refers to other images that are similar or related to the current input. Let us try to understand this with the dog vs. cat classification example.

As shown in Figure 1, the NSL framework takes the input dog image and a structure. The structure is a graph representing similarities between other images in our dataset and our input image. Notice that the graph in the figure above contains images of other bulldogs that are similar to our input dog image.

Given the input image and the structure (i.e., other related or similar images to the input), NSL tries to jointly optimize the input features and the structured signal to regularize the neural network training. Let us try to understand this with an example.

Figure 2 shows a typical NSL training pipeline.

Suppose we have a bulldog image (i.e., the input) and other bulldog images in the dataset (i.e., structured signal) depicting bulldogs of different skin tones, sizes, poses, and backgrounds. We call them neighbor samples.

The NSL framework takes both of these and creates a batch with the original input sample and the neighbor samples (Figure 2). These are then passed through the neural network to get sample embeddings (corresponding to our input image) and neighbor embeddings (corresponding to our neighbor samples). Both of these embeddings are then used to compute loss with the ground-truth label, and the loss is back-propagated.

Notice that in the above example, we enabled the network to learn that all images of a bulldog with different skin tones, sizes, poses, and backgrounds belong to the same ground-truth class (i.e., bulldog). This regularizes our model and allows it to become robust to different variations of bulldog images, helping it to correctly recognize a bulldog at test time in varied poses, colors, backgrounds, etc.

Thus, the NSL framework makes use of the structured signals to help us train more robust neural network systems.

---

Adversarial Learning with NSL

In the above example, we discussed how the NSL framework allows us to use the input image and the structured signals to learn better networks. The structured signal in the above example was a set of bulldog images (similar to the input image) with varying skin tones, sizes, poses, and backgrounds. Notice these structured examples were based on the similarity of class (bulldog) between samples and neighbors.

Naturally, the question arises: Can we have other structured signals as inputs? Of course, yes !!!!!!

The NSL framework allows us to use structures that align with our use case and the final object of the system we are building. Let us try to dig a little deeper and understand this.

In the previous tutorial of this series, we discussed how adversarial learning can make models robust to adversarial examples. Basically, this involved training or fine-tuning models on both original and adversarial examples of a class. Can the NSL framework help to achieve this goal for our use case.

Of course, yes!!!! Let us take the case where the structured signals to our NSL pipeline are adversarial examples. In this case, we have 2 inputs:

• original input image
• engineered adversarial example for that image (i.e., structured signal).

Here, the structured signal is related to the input sample by the following relation: It is the corresponding adversarial sample of the input.

Note, as we discussed above in Figure 2, this means that we can train our model on both the images and their corresponding adversarial examples. This will enable the network to learn that original images of a class and adversarially perturbed images from that class should be assigned the same label and belong to the same ground-truth class. This regularizes our model and allows it to become robust (e.g., adversarial examples and attacks), which will make its predictions more reliable.

As we have discussed earlier in this tutorial series, we will use this ability of the TensorFlow NSL framework and implement an end-to-end pipeline to show how this framework can be used to make models robust to adversarial attacks.

---

CIFAR-10 Dataset

We will be using the CIFAR-10 dataset (Canadian Institute For Advanced Research) for the purpose of building our adversarial learning application. The CIFAR-10 dataset, containing 60,000 32×32 color images from 10 classes, include the following: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.

The dataset is balanced and consists of 6,000 images from each class and is divided into 50,000 training images and 10,000 test images.

---

Configuring Your Development Environment

To follow this guide, you need to have the TensorFlow and OpenCV library installed on your system.

Luckily, TensorFlow and OpenCV are pip-installable:

$ pip install tensorflow
$ pip install opencv-contrib-python

If you need help configuring your development environment for OpenCV, we highly recommend that you read our pip install OpenCV guide — it will have you up and running in minutes.

---

Need Help Configuring Your Development Environment?

All that said, are you:

• Short on time?
• Learning on your employer’s administratively locked system?
• Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
• Ready to run the code immediately on your Windows, macOS, or Linux system?

Then join PyImageSearch University today!

Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides pre-configured to run on Google Colab’s ecosystem right in your web browser! No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

---

Project Structure

We first need to review our project directory structure.

Start by accessing this tutorial’s “Downloads” section to retrieve the source code and example images.

From there, take a look at the directory structure:

├── demo.py
├── inference.py
├── output
├── pyimagesearch
│   ├── __init__.py
│   ├── callbacks.py
│   ├── config.py
│   ├── data.py
│   ├── model.py
│   ├── robust.py
│   └── visualization.py
└── train.py

Given above is the directory which we discussed in detail in the first part of this series.

In this tutorial, we will discuss the following files:

• config.py: contains the code for our initial parameter configurations
• data.py: implements the data pipeline
• callbacks.py: implements the callback routine

---

Configuration File

We start by discussing the parameter configurations we will use for this tutorial series.

We open our config.py file, which stores the initial parameter configurations.

# import the necessary packages
import os

# define the data configurations
INPUT_SHAPE = [32, 32, 3]
NUM_CLASSES = 10
CLASSES = ["airplane", "automobile", "bird", "cat", "deer", "dog",
	"frog", "horse", "ship", "truck"]

# define the training configurations
LR_START = 1e-5
LR_MAX = 1e-3
LR_RAMPUP_EPOCHS = 5
LR_SUSTAIN_EPOCHS = 0
LR_STEP_DECAY = 0.75
BATCH_SIZE = 32
EPOCHS = 100

# define the adversary configurations
ADV_MULTIPLIER = 0.60
ADV_STEP_SIZE = 0.01
ADV_GRAD_NORM = "infinity"
PGD_ITERATION = 5
PGD_EPSILON = 2.0 / 255.0
CLIP_VALUE_MIN = 0.0
CLIP_VALUE_MAX = 1.0

# define the image and label names
IMAGE_INPUT_NAME = "image"
LABEL_INPUT_NAME = "label"

# define the output paths for models and plots
OUTPUT_PATH = "output"
DATASET_PLOT_PATH = os.path.join(OUTPUT_PATH, "data_viz.png")
BASE_MODEL_PATH = os.path.join(OUTPUT_PATH, "base_model")
FGSM_MODEL_PATH = os.path.join(OUTPUT_PATH, "fgsm_model")
PGD_MODEL_PATH = os.path.join(OUTPUT_PATH, "pgd_model")
BASE_MODEL_FGSM_PLOT_PATH = os.path.join(OUTPUT_PATH,
	"base_model_fgsm_plot.png")
BASE_MODEL_PGD_PLOT_PATH = os.path.join(OUTPUT_PATH,
	"base_model_pgd_plot.png")
FGSM_MODEL_FGSM_PLOT_PATH = os.path.join(OUTPUT_PATH,
	"fgsm_model_fgsm_plot.png")
FGSM_MODEL_PGD_PLOT_PATH = os.path.join(OUTPUT_PATH,
	"fgsm_model_pgd_plot.png")
PGD_MODEL_FGSM_PLOT_PATH = os.path.join(OUTPUT_PATH,
	"pgd_model_fgsm_plot.png")
PGD_MODEL_PGD_PLOT_PATH = os.path.join(OUTPUT_PATH,
	"pgd_model_pgd_plot.png")

First, we import the os module on Line 2, which allows us to use file-handling functionalities, as we will see later in this post.

Then, on Lines 5-8, we define the following data configuration parameters:

• INPUT_SHAPE: the dimensions of our input images
• NUM_CLASSES: number of classes in the CIFAR-10 dataset
• CLASSES: list of all class names in the CIFAR-10 dataset

Now, we go ahead and define the following parameters we will use for training our models:

• learning rate related parameters (Lines 11-15)
• batch size (Line 16)
• total number of epochs (Line 17)

Note that we will discuss more about the different learning rate parameters while discussing the training procedure in the upcoming tutorials of this series.

Next, we define the adversary parameters and configurations (Lines 20-26), which include the following:

• ADV_MULTIPLIER: multiplier for our adversarial attack
• ADV_STEP_SIZE: step size
• ADV_GRAD_NORM: the type of norm our attack will be based

Furthermore, we define the number of iterations we will use for our projected gradient descent (PGD) attack and the epsilon value (i.e., PGD_ITERATION and PGD_EPSILON), which we discussed in the previous tutorial. Additionally, we define the minimum and maximum clipping values (i.e., CLIP_VALUE_MIN and CLIP_VALUE_MAX), as shown on Lines 25 and 26.

Now that we have defined the dataset-, training-, and adversary-related parameters, let us go ahead and define parameters for images and labels and construct the paths where the outputs will be stored.

On Lines 29 and 30, we define the image and label parameters (i.e., IMAGE_INPUT_NAME and LABEL_INPUT_NAME).

Finally, we define the output paths and locations where the final outputs, models, and plots will be stored. On Line 33, we define OUTPUT_PATH, which points to the folder where the outputs will be stored.

Next, on Lines 34-37, we construct the following paths that will be stored within the OUTPUT_PATH folder:

• DATASET_PLOT_PATH: data visualization
• BASE_MODEL_PATH: base model
• FGSM_MODEL_PATH: fast gradient sign method (FGSM) model
• PGD_MODEL_PATH: PGD model

Additionally, we construct the paths to store the plots corresponding to different models that we will use in this tutorial series (Lines 38-49):

• BASE_MODEL_FGSM_PLOT_PATH
• BASE_MODEL_PGD_PLOT_PATH
• FGSM_MODEL_FGSM_PLOT_PATH
• FGSM_MODEL_PGD_PLOT_PATH
• PGD_MODEL_FGSM_PLOT_PATH
• PGD_MODEL_PGD_PLOT_PATH

---

Building a Data Pipeline

Now that we have completed defining the initial parameter configurations, it is time to start building our data pipeline.

As we discussed above, we will use the CIFAR-10 dataset for this tutorial series. The TensorFlow API allows us to seamlessly download and prepare the CIFAR-10 dataset so that we can use it in our computer vision projects.

Let us open the data.py file and start our discussion.

# import the necessary packages
from tensorflow.keras.datasets import cifar10
import tensorflow as tf

def get_cifar_data(trainVal=True):
	# check if we are looking to load the training and validation data
	if trainVal:
		# get the CIFAR-10 train dataset and split it into train and
		# val
		(xTrain, yTrain), (_, _) = cifar10.load_data()
		(xTrain, yTrain), (xVal, yVal) = (
			(xTrain[:40000], yTrain[:40000]),
			(xTrain[40000:], yTrain[40000:]),
		)

		# build the train and val dataset from the NumPy arrays
		trainDs = tf.data.Dataset.from_tensor_slices((xTrain, yTrain))
		valDs = tf.data.Dataset.from_tensor_slices((xVal, yVal))

		# return the train and val dataset
		return (trainDs, valDs)

	# otherwise, we are looking to load the testing data
	else:
		# get the CIFAR-10 test dataset and build a TensorFlow
		# Dataset from the NumPy arrays
		(_, _), (xTest, yTest) = cifar10.load_data()
		testDs = tf.data.Dataset.from_tensor_slices((xTest, yTest))

		# return the test dataset
		return testDs

def preprocess_image(image, label):
	# cast the image to float32 type and scale the pixel intensity
	# to be in the [0, 1] range
	image = tf.cast(image, dtype=tf.float32) / 255.0

	# return the image and the label
	return (image, label)

def build_data_pipeline(ds, batchSize, train=True):
	# check if we are building training data input pipeline
	if train:
		# build the train data input pipeline
		ds = (ds
			.map(preprocess_image)
			.shuffle(batchSize * 2)
			.batch(batchSize)
			.prefetch(tf.data.AUTOTUNE)
		)

	# otherwise, we are building validation or testing data input
	# pipeline
	else:
		# build the val or test data input pipeline
		ds = (ds
			.map(preprocess_image)
			.batch(batchSize)
			.prefetch(tf.data.AUTOTUNE)
		)

	# return the data input pipeline
	return ds

We start by importing the cifar10 dataset from tensorflow.keras.datasets and the tensorflow library (Lines 2 and 3).

On Lines 5-31, we define the get_cifar_data function, which loads the CIFAR-10 dataset, splits it into train, validation, and test subsets, and returns the data subset as per our requirement. Note that it takes the trainVal parameter as the input argument and returns the training and validation sets when this parameter is True or the test data subset when this parameter is False.

On Line 7, we check if the trainVal parameter is True, and if yes, we load the cifar10 dataset using the cifar10.load_data() function (Line 10).

Note that this function returns the train set and the test set for the CIFAR-10 dataset. On Line 10, we only unpack the train set and store it as (xTrain, yTrain).

Next, we split our training dataset into train and validation data subsets. For this, we keep the first 40,000 examples as the train subset (i.e., (xTrain, yTrain)) and everything else as the val subset (i.e., (xVal, yVal)) (Lines 11-14).

Now that we have the train and val subsets as NumPy arrays, we are ready to create TensorFlow datasets. On Lines 16 and 17, we use the tf.data.Dataset.from_tensor_slices function to convert the (xTrain, yTrain) NumPy arrays to the training dataset (i.e., trainDs) and the (xVal, yVal) NumPy arrays to the validation dataset (i.e., valDs), respectively.

Finally, on Line 21, we return the training and validation datasets.

Next, we implement code for the case where we need the test dataset (i.e., the trainVal input argument is False). On Line 27, we use the cifar10.load_data() function as we saw above, but here we unpack the test subset as (xTest, yTest).

Then, on Line 28, we use the tf.data.Dataset.from_tensor_slices function, as we saw earlier, to convert our test subset to the TensorFlow test dataset (i.e., testDs). Finally, we return our test dataset on Line 31.

Now that we have discussed our function to load the CIFAR-10 dataset, let us implement the functions which will allow us to preprocess our images and build our data pipeline.

On Lines 33-39, we implement the preprocess_image function, which takes as an argument the image and corresponding label and processes it to a format that our model expects.

Specifically, we take the input image and cast it to the float32 data type as shown and normalize the pixel values between 0 and 1 by dividing by 255.0 (Line 36).

Finally, we return the preprocessed image and label tuple (Line 39).

Let us now implement our build_data_pipeline function (Lines 41-63), which will allow us to construct our data pipeline and get data samples for training or testing our models.

This function takes as arguments the dataset (i.e., ds), the batch size (i.e., batchSize), and a boolean (i.e., train), which specifies if we are building the data pipeline for the training or inference phase (Line 41).

In case we want to build a data pipeline for the training phase (i.e., the train argument is True), we will take the input dataset (i.e., ds) and use the functionalities provided by TensorFlow to process the data samples and create batches (Lines 45-50).

Specifically, we use the following functionalities:

• map: allows us to apply the preprocess_image function to our input data samples (Line 46)
• shuffle: randomly samples elements from a buffer of elements with buffer size = batchSize * 2 (Line 47)
• batch: takes as input the batchSize and allows us to sample batches of data samples (Line 48)
• prefetch: directs TensorFlow to prepare later elements while current elements are being processed (Line 49)

On the other hand, in case we want to build a data pipeline for the inference phase (i.e., the train argument is False), we use the map, batch, and prefetch functionalities as we did above to process the test dataset (Lines 56-60). Note that we do not use the shuffle functionality since we only need to shuffle samples during training, and this is not required during the inference stage.

Finally, we return the processed data (i.e., ds), as shown on Line 63.

---

Building Callbacks

During the training process of our adversarial application, we will need to use callbacks, which will allow us to tune the learning rate during the training process and implement an early stopping strategy to avoid overfitting.

Let us open the callbacks.py file and implement our training callbacks in TensorFlow.

# import the necessary packages
from tensorflow.keras.callbacks import EarlyStopping
from tensorflow.keras.callbacks import LearningRateScheduler

def get_lr_schedule_callback(lrStart, lrMax, lrRampupEpochs,
	lrSustainEpochs, lrStepDecay):
	# build the learning rate function
	def lrfn(epoch):
		if epoch < lrRampupEpochs:
			lr = (lrMax - lrStart) / lrRampupEpochs * epoch + lrStart
		elif epoch < lrRampupEpochs + lrSustainEpochs:
			lr = lrMax
		else:
			lr = lrMax * lrStepDecay**(
				(epoch - lrRampupEpochs - lrSustainEpochs)//10
			)
		return lr
	
	# build the learning rate schedule callback and return it
	lrScheduleCallback = LearningRateScheduler(lrfn, verbose=True)
	return lrScheduleCallback

def get_early_stopping_callback():
	# build the early stopping callback and return it
	earlyStopCallback = EarlyStopping(monitor="val_loss", patience=5,
		restore_best_weights=True)
	return earlyStopCallback

We start by importing the necessary modules, such as the EarlyStopping and LearningRateScheduler callbacks from tensorflow.keras.callbacks (Lines 2 and 3).

Next, we start with the get_lr_schedule_callback function, which will allow us to build our callback for learning rate scheduling.

This function takes the following as input (Lines 5 and 6):

• lrStart: starting learning rate
• lrMax: maximum learning rate we will reach during training
• lrRampupEpochs: number of epochs we want to ramp up the learning rate
• lrSustainEpochs: number of epochs we want to sustain the learning rate
• lrStepDecay: learning rate decay parameter

Now, let us discuss the logic behind the learning rate scheduler that this callback implements. We start defining the lrfn function (Lines 8-17), which takes as input the particular epoch and returns the learning rate for that particular epoch as per our scheduler scheme.

Specifically, we first check if the current epoch is less than the lrRampupEpochs (Line 9). If that is true, our current learning rate (i.e., lr) is given by the expression lrStart + (lrMax - lrStart) / lrRampupEpochs * epoch (Line 10), which takes the base learning rate as lrStart and divides the interval between maximum learning rate (i.e., lrMax) and starting learning rate (i.e., lrStart) by the total number of ramp-up epochs (i.e., lrRampupEpochs) to distribute the rise over lrRampupEpochs evenly.

Next, we check if the current epoch is less than the sum of ramp-up and sustain epochs (i.e., lrRampupEpochs + lrSustainEpochs), and if that is true, we simply sustain the learning rate (i.e., keep it constant) at lrMax (Lines 11 and 12).

Otherwise, we start gradually decaying (or reducing) the learning rate starting from lrMax and using the decay rate (i.e., lrStepDecay), as shown on Lines 14-16.

On Line 17, we return the learning rate.

Finally, on Line 20, we wrap our scheduling logic (i.e., lrfn function) using the LearningRateScheduler callback and return our final callback (i.e., lrScheduleCallback) on Line 21.

Now that we have completed implementing the learning rate scheduler callback, we will discuss the early stopping callback. This will enable us to stop training the model based on the validation loss to avoid overfitting.

We discuss the get_early_stopping_callback function, which implements the early stopping callback (Lines 23-27).

On Lines 25 and 26, we use the EarlyStopping callback (that we imported from tensorflow.keras.callbacks) and direct it to monitor (or keep track of) the validation loss. Note that we use a patience of 5 epochs and set the restore_best_weights=True, which means that the training will be automatically stopped if the validation loss does not decrease for 5 continuous epochs and the best model weights (i.e., a model with the least validation loss) will be restored.

We return our earlyStopCallback on Line 27.

---

---

Summary

In this tutorial, we learned about the TensorFlow NSL framework and how it can help us train better deep learning models.

Specifically, we took an in-depth look at the NSL framework workflow. We discussed how it can be used to regularize our models and make them robust by learning from adversarial examples during training.

Furthermore, we started building our NSL-based adversarial learning application and implemented the data pipeline and callbacks.

---

Citation Information

Chandhok, S. “Adversarial Learning with Keras and TensorFlow (Part 2): Implementing the Neural Structured Learning (NSL) Framework and Building a Data Pipeline,” PyImageSearch, P. Chugh, A. R. Gosthipaty, S. Huot, K. Kidriavsteva, and R. Raha, eds., 2024, https://pyimg.co/bfzma

@incollection{Chandhok_2024_ALwKTF-pt2,
  author = {Shivam Chandhok},
  title = {Adversarial Learning with Keras and TensorFlow (Part 2): Implementing the Neural Structured Learning (NSL) Framework and Building a Data Pipeline},
  booktitle = {PyImageSearch},
  editor = {Puneet Chugh and Aritra Roy Gosthipaty and Susan Huot and Kseniia Kidriavsteva and Ritwik Raha},
  year = {2024},
  url = {https://pyimg.co/bfzma},
}

---

---

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!