Neural Machine Translation with Luong’s Attention Using TensorFlow and Keras

---

Table of Contents

---

Neural Machine Translation with Luong’s Attention Using TensorFlow and Keras

The previous tutorial on Neural Machine Translation is where we first covered attention (Bahdanau’s attention). Not only did we cover the math behind it, but we also went hands-on training a TensorFlow model from scratch. If you haven’t read it yet, be sure to read that before you go through this tutorial.

In this tutorial, you will learn how Neural Machine Translation can be achieved using Luong’s attention and how it enhances the attention mechanism introduced in Bahdanau’s attention.

Enough of Google Translate. We also have an interactive demo where you can see the trained model in action.

This lesson is the last in a 2-part series on NLP 103:

1. Neural Machine Translation with Bahdanau’s Attention Using TensorFlow and Keras
2. Neural Machine Translation with Luong’s Attention Using TensorFlow and Keras (this tutorial)

To learn how to apply Luong’s attention to the Neural Machine Translation task, just keep reading.

---

Neural Machine Translation with Luong’s Attention Using TensorFlow and Keras

---

Introduction

In the academic paper, Effective Approaches to Attention-Based Neural Machine Translation, Luong et al. provide more effective approaches to building attention. We have to note here that the basic intuition behind attention does not change. The additions provided by Luong et al. are iterative changes that make the process of attention simpler and more efficient.

Building upon the work of Bahdanau et al., Luong et al. add their own twist to the normal attention architecture, suggesting subtle changes to break through the limitations of the old architecture.

Note: We will not be diving too deep into the entire code walkthrough since that has already been covered in the Bahdanau’s attention tutorial. This tutorial will cover the significant changes and how to implement them in TensorFlow and Keras.

---

Configuring Your Development Environment

To follow this guide, you need to have the tensorflow and tensorflow-text installed on your system.

Luckily, TensorFlow is pip-installable:

$ pip install tensorflow==2.8.0
$ pip install tensorflow-text==2.8.0

---

Having Problems Configuring Your Development Environment?

All that said, are you:

• Short on time?
• Learning on your employer’s administratively locked system?
• Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
• Ready to run the code right now on your Windows, macOS, or Linux system?

Then join PyImageSearch University today!

Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides that are pre-configured to run on Google Colab’s ecosystem right in your web browser! No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

---

Project Structure

We first review our project directory structure.

Start by accessing the “Downloads” section of this tutorial to retrieve the source code and example images.

From there, take a look at the directory structure:

├── download.sh
├── inference.py
├── output
│   └── loss.png
├── pyimagesearch
│   ├── config.py
│   ├── dataset.py
│   ├── __init__.py
│   ├── loss.py
│   ├── models.py
│   ├── schedule.py
│   └── translator.py
├── requirements.txt
└── train.py

In the pyimagesearch directory, we have:

• config.py: The configuration file for the task
• dataset.py: The utilities for the dataset pipeline
• loss.py: Holds the code snippet for the losses needed to train the model
• models.py: Encoder and Decoder for the translation model
• schedule.py: The learning rate scheduler for the training pipeline
• translator.py: The train and inference models

In the core directory, we have four scripts:

• download.sh: A shell script to download the training data
• requirements.txt: The python packages that are required for this tutorial
• train.py: The script run to train the model
• inference.py: The inference script

---

Luong’s Attention

Luong et al. suggest some small but necessary changes to the architecture of the decoder. We will first talk about the encoder, the attention layer, and then the decoder. While discussing the architecture, we will also compare it with that of Bahdanau et al. Consider the diagram shown in Figure 2.

---

Encoder Architecture

In this paper, the authors opt for a unidirectional (instead of bidirectional as in Bahdanau’s implementation) Recurrent Neural architecture for the encoder. Unidirectional RNNs speed up the computation.

For the encoder we choose a Gated Recurrent Unit (GRU) which takes the present input and past hidden state as input while processing them into the present hidden state .

After passing the entire source sentence to the encoder, we have a set of all the hidden states.

---

Decoder Architecture

At each time step, , in the decoding phase, the main motive is to capture the present hidden state of the decoder, , and then to derive a context vector, , that captures relevant source-side information.

Specifically, given the target hidden state, , and the source-side context vector, , we employ a simple concatenation layer to combine the information from both vectors to produce an attentional hidden state as follows:

The attention vector, , is then fed through the softmax layer to produce the probability of the next decoder word.

Enough of mathematical jargon, let’s focus only on the part where the authors have proposed changes to the attention layer.

Bahdanau goes from:

Luong goes from:

---

Input Feeding Approach

With the present proposal, the authors discovered that they were not feeding the attention to the decoder’s recurrent units. This meant that the decoding system did not know which parts of the source sentence were attended to at the previous step.

With that in mind, they now propose to provide the attention to the next decoder unit along with the input and the hidden states. This proved to be a game-changer. While Bahdanau’s model already had this mechanism installed inside of it, Luong’s model had to do it explicitly.

Figure 3 shows the entire encoding and decoding mechanism envisioned by Luong et al.

---

Implementing Luong’s Attention

Since all the methods and functions are the same as Bahdanau’s attention, we will discuss Luong’s attention module in this tutorial. The entire code walkthrough is covered in the previous blog post of this series.

If you haven’t already, do be sure to read it before proceeding further. And if you have done so already, let us open models.py inside the pyimagesearch directory and go through it.

class LuongAttention(Layer):
    def __init__(self, attnUnits, **kwargs):
        super().__init__(**kwargs)
        # initialize the attention units
        self.attnUnits = attnUnits
    
    def build(self, inputShape):
        # build the additive attention layer
        self.attention = Attention()

    def get_config(self):
        # return the configuration of the layer
        return {
            "attnUnits": self.attnUnits,
        }

    def call(self, hiddenStateEnc, hiddenStateDec, mask):
        # grab the source and target mask
        sourceMask = mask[0]
        targetMask = mask[1]

        # apply attention to align the representations
        (contextVector, attentionWeights) = self.attention(
            inputs=[hiddenStateDec, hiddenStateEnc, hiddenStateEnc],
            mask=[targetMask, sourceMask],
            return_attention_scores=True
        )

        # return the context vector and the attention weights
        return (contextVector, attentionWeights)

We begin by initializing the attention units on Line 80 inside the __init__ function of the LuongAttention class.

Next, on Lines 82-84, we build the attention layer inside the build function. We return the attention units inside the get_config function (Lines 86-90).

On Line 92, we define the call method. First, we extract the sourceMask and targetMask on Lines 94 and 95.

On Lines 98-102, we apply attention to align the representations. We use the neat attention API from TensorFlow to achieve this. You can read more about it from the official TensorFlow documentation here.

Finally on Line 105, we return the contextVector and attentionWeights.

---

Summary

This tutorial walked us through the specific ways Luong’s attention improved the task of Neural Machine Translation. We also learned how to implement the attention module simply using Keras and TensorFlow.

The Bahdanau and Luong attentions are neither state-of-the-art techniques nor widely used in NLP systems. But they do provide a stepping stone and a concrete foundation for what comes next.

We will be back with another tutorial on attention and its uses, but till that time, here’s a little food for thought:

“What happens if we only use attention in an architecture? ?”

---

Citation Information

A. R. Gosthipaty and R. Raha. “Neural Machine Translation with Luong’s Attention Using TensorFlow and Keras,” PyImageSearch, P. Chugh, S. Huot, K. Kidriavsteva, and A. Thanki, eds., 2022, https://pyimg.co/tpf3l

@incollection{ARG-RR_2022_Luong,
  author = {Aritra Roy Gosthipaty and Ritwik Raha},
  title = {Neural Machine Translation with {Luong’s} Attention Using {TensorFlow} and {Keras}},
  booktitle = {PyImageSearch},
  editor = {Puneet Chugh and Susan Huot and Kseniia Kidriavsteva and Abhishek Thanki},
  year = {2022},
  note = {https://pyimg.co/tpf3l},
}

---

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!