The use of this site and the content contained therein is governed by the Terms of Use. When you use this site you acknowledge that you have read the Terms of Use and that you accept and will be bound by the terms hereof and such terms as may be modified from time to time.
All text, graphics, audio, design and other works on the site are the copyrighted works of nasscom unless otherwise indicated. All rights reserved.
Content on the site is for personal use only and may be downloaded provided the material is kept intact and there is no violation of the copyrights, trademarks, and other proprietary rights. Any alteration of the material or use of the material contained in the site for any other purpose is a violation of the copyright of nasscom and / or its affiliates or associates or of its third-party information providers. This material cannot be copied, reproduced, republished, uploaded, posted, transmitted or distributed in any way for non-personal use without obtaining the prior permission from nasscom.
The nasscom Members login is for the reference of only registered nasscom Member Companies.
nasscom reserves the right to modify the terms of use of any service without any liability. nasscom reserves the right to take all measures necessary to prevent access to any service or termination of service if the terms of use are not complied with or are contravened or there is any violation of copyright, trademark or other proprietary right.
From time to time nasscom may supplement these terms of use with additional terms pertaining to specific content (additional terms). Such additional terms are hereby incorporated by reference into these Terms of Use.
Disclaimer
The Company information provided on the nasscom web site is as per data collected by companies. nasscom is not liable on the authenticity of such data.
nasscom has exercised due diligence in checking the correctness and authenticity of the information contained in the site, but nasscom or any of its affiliates or associates or employees shall not be in any way responsible for any loss or damage that may arise to any person from any inadvertent error in the information contained in this site. The information from or through this site is provided "as is" and all warranties express or implied of any kind, regarding any matter pertaining to any service or channel, including without limitation the implied warranties of merchantability, fitness for a particular purpose, and non-infringement are disclaimed. nasscom and its affiliates and associates shall not be liable, at any time, for any failure of performance, error, omission, interruption, deletion, defect, delay in operation or transmission, computer virus, communications line failure, theft or destruction or unauthorised access to, alteration of, or use of information contained on the site. No representations, warranties or guarantees whatsoever are made as to the accuracy, adequacy, reliability, completeness, suitability or applicability of the information to a particular situation.
nasscom or its affiliates or associates or its employees do not provide any judgments or warranty in respect of the authenticity or correctness of the content of other services or sites to which links are provided. A link to another service or site is not an endorsement of any products or services on such site or the site.
The content provided is for information purposes alone and does not substitute for specific advice whether investment, legal, taxation or otherwise. nasscom disclaims all liability for damages caused by use of content on the site.
All responsibility and liability for any damages caused by downloading of any data is disclaimed.
nasscom reserves the right to modify, suspend / cancel, or discontinue any or all sections, or service at any time without notice.
For any grievances under the Information Technology Act 2000, please get in touch with Grievance Officer, Mr. Anirban Mandal at data-query@nasscom.in.
In today's interconnected world, seamless communication across languages is more important than ever before. Neural Machine Translation (NMT) has emerged as a powerful tool to bridge language barriers, and Transformer models have revolutionized the field with their ability to achieve state-of-the-art results.
This blog embarks on a thrilling journey into the exciting world of NMT with Transformer models, powered by the robust infrastructure of E2E's Cloud GPU servers. We'll delve deeper than just code snippets, uncovering the theoretical foundations that make Transformers tick. We'll explore the intricacies of their implementation using TensorFlow, the popular deep-learning framework. But, most importantly, we'll bridge the gap between theory and practice, guiding you through the process of deploying your own NMT system on E2E's cloud platform. By the end of this exploration, you'll not only gain a profound understanding of Transformer-based NMT, but will also be equipped with the practical knowledge needed to harness its power for real-world applications. So, buckle up, language enthusiasts and tech adventurers alike, as we embark on this exciting quest to conquer the frontiers of machine translation!
Transformers: Unleashing the Power of Attention
The Transformer revolutionized machine translation with its unique architecture, built upon the concept of attention. Introduced in the groundbreaking paper ‘Attention is All You Need’, Transformers replaced traditional CNNs and RNNs with this powerful mechanism. Unlike its predecessors, attention allows each word to ‘attend’ to all other words in the sentence simultaneously, capturing intricate relationships and context across the entire sequence.
Think of it like a classroom discussion: instead of each student waiting for their turn to speak, everyone can participate simultaneously, enriching the conversation with diverse perspectives. Similarly, Transformers process words in parallel, leading to faster and more efficient information flow.
This parallelization isn't just about speed; it unlocks the ability to capture long-range dependencies. Unlike RNNs, where information fades with distance, Transformers can directly connect distant words, understanding how seemingly unrelated parts contribute to overall meaning. Imagine analyzing a complex sentence with multiple clauses and references. Transformers can seamlessly navigate these connections, achieving superior translation accuracy.
Also, Transformers make no assumptions about the order of elements, making them ideal for tasks beyond language like analyzing game scenarios where the spatial arrangement of objects is crucial. By harnessing the power of attention, Transformers have become the champions in various natural language processing tasks, offering unparalleled performance and versatility.
E2E’s GPU Cloud: An Overview
The most effective approach to grasp Neural Machine Translation involves hands-on experience, where the environment you choose for practice plays a pivotal role in mastering complex architectures. Amidst numerous GPU cloud service providers available, selecting the right one can notably enhance both cost-efficiency and productivity. Fortunately, after thorough research, I've identified E2E Cloud as the optimal choice, offering a balance between cost-effectiveness and accessibility. Moreover, it provides readily available setups for all required environments, expediting projects by saving valuable time. For this hands-on session, I utilized the TIR-AI Platform within E2E cloud. To embark on a similar journey, you can initiate the process by following this link: https://www.e2enetworks.com/blog/how-to-use-jupyter-notebooks-on-e2e-networks .
Boost your training efficiency with NVIDIA NGC pipelines and E2E’s Cloud GPUs. Leveraging pre-built, optimized pipelines from NGC can significantly accelerate your Transformer model training process. E2E's Cloud GPUs are specifically designed to harness the power of NGC, providing seamless compatibility and maximizing performance. This potent combination makes E2E a top choice for developers seeking to unlock the full potential of Transformer-based machine translation while minimizing training time and resources.
Let’s Play: Crafting a Transformer Model for Seamless Portuguese-to-English Translation
Ditch the dictionary and build your own AI-powered language bridge! This tutorial teaches you how to craft a Transformer model for seamless Portuguese-to-English translation.
To employ the needed packages for NMT, installation can be accomplished via the Python package installer, PIP. In a Jupyter notebook, utilize the magic command as illustrated below:
# some examples
for pt_examples, en_examples in train_examples.batch(3).take(1):
print('> Examples in Portuguese:')
for pt in pt_examples.numpy():
print(pt.decode('utf-8'))
print()
print('> Examples in English:')
for en in en_examples.numpy():
print(en.decode('utf-8'))
Before diving into the exciting world of machine translation with Transformer models, it's crucial to understand how we prepare the text for these powerful algorithms. This is where tokenization comes in.
Think of tokenization as a linguistic chef carefully chopping up a sentence into smaller pieces, called tokens. These tokens can be individual words, smaller pieces of words (subwords), or even individual characters, depending on the chosen method.
In our case, we're using a special type of tokenizer called a subword tokenizer. This tool is specifically designed to optimize text for language models like Transformers. Why subwords? Because they offer a sweet spot between individual words and characters:
More granular than words: Subwords can capture smaller nuances within words, like prefixes and suffixes, which are crucial for accurate translation.
Less numerous than characters: Unlike individual characters, subwords create a manageable vocabulary size, making the training process more efficient.
To handle both Portuguese and English effectively, we've employed two separate BertTokenizer objects, each trained on its respective language. This ensures each language is treated with the appropriate understanding of its unique grammar and vocabulary.
# some examples
model_name = 'ted_hrlr_translate_pt_en_converter'
tf.keras.utils.get_file(
f'{model_name}.zip',
f'https://storage.googleapis.com/download.tensorflow.org/models/{model_name}.zip',
cache_dir='.', cache_subdir='', extract=True
)
tokenizers = tf.saved_model.load(model_name)
MAX_TOKENS=128
BUFFER_SIZE = 20000
BATCH_SIZE = 256
def prepare_batch(pt, en):
pt = tokenizers.pt.tokenize(pt) # Output is ragged.
pt = pt[:, :MAX_TOKENS] # Trim to MAX_TOKENS.
pt = pt.to_tensor() # Convert to 0-padded dense Tensor
en = tokenizers.en.tokenize(en)
en = en[:, :(MAX_TOKENS+1)]
en_inputs = en[:, :-1].to_tensor() # Drop the [END] tokens
en_labels = en[:, 1:].to_tensor() # Drop the [START] tokens
return (pt, en_inputs), en_labels
def make_batches(ds):
return (
ds
.shuffle(BUFFER_SIZE)
.batch(BATCH_SIZE)
.map(prepare_batch, tf.data.AUTOTUNE)
.prefetch(buffer_size=tf.data.AUTOTUNE))
train_batches = make_batches(train_examples)
val_batches = make_batches(val_examples)
for (pt, en), en_labels in train_batches.take(1):
break
print(pt.shape)
print(en.shape)
print(en_labels.shape)
While the paper ‘Attention Is All You Need’ offers a deep dive into the theoretical underpinnings of the Transformer architecture, let's embark on a more practical journey. Forget dense equations and academic jargon - get ready to code this powerhouse architecture yourself!
This visual below provides a high-level overview of the Transformer's structure, but the true excitement lies in bringing it to life. We'll break down the key components, understand their interactions, and then translate that understanding into actual lines of code. Imagine, by the end of this exploration, you'll possess your very own functional Transformer model, ready to tackle natural language tasks with remarkable power! So, are you ready to embark on this coding adventure? Get your coding tools ready, and let's unlock the mysteries of the Transformer together!
## Attention layers
class BaseAttention(tf.keras.layers.Layer):
def __init__(self, **kwargs):
super().__init__()
self.mha = tf.keras.layers.MultiHeadAttention(**kwargs)
self.layernorm = tf.keras.layers.LayerNormalization()
self.add = tf.keras.layers.Add()
class CrossAttention(BaseAttention):
def call(self, x, context):
attn_output, attn_scores = self.mha(
query=x,
key=context,
value=context,
return_attention_scores=True)
# Cache the attention scores for plotting later.
self.last_attn_scores = attn_scores
x = self.add([x, attn_output])
x = self.layernorm(x)
return x
class GlobalSelfAttention(BaseAttention):
def call(self, x):
attn_output = self.mha(
query=x,
value=x,
key=x)
x = self.add([x, attn_output])
x = self.layernorm(x)
return x
class CausalSelfAttention(BaseAttention):
def call(self, x):
attn_output = self.mha(
query=x,
value=x,
key=x,
use_causal_mask = True)
x = self.add([x, attn_output])
x = self.layernorm(x)
return x
Feed Forward Layer:
# Feed Forward Block
class FeedForward(tf.keras.layers.Layer):
def __init__(self, d_model, dff, dropout_rate=0.1):
super().__init__()
self.seq = tf.keras.Sequential([
tf.keras.layers.Dense(dff, activation='relu'),
tf.keras.layers.Dense(d_model),
tf.keras.layers.Dropout(dropout_rate)
])
self.add = tf.keras.layers.Add()
self.layer_norm = tf.keras.layers.LayerNormalization()
def call(self, x):
x = self.add([x, self.seq(x)])
x = self.layer_norm(x)
return x
Encoder:
# Encoder
class EncoderLayer(tf.keras.layers.Layer):
def __init__(self,*, d_model, num_heads, dff, dropout_rate=0.1):
super().__init__()
self.self_attention = GlobalSelfAttention(
num_heads=num_heads,
key_dim=d_model,
dropout=dropout_rate)
self.ffn = FeedForward(d_model, dff)
def call(self, x):
x = self.self_attention(x)
x = self.ffn(x)
return x
class Encoder(tf.keras.layers.Layer):
def __init__(self, *, num_layers, d_model, num_heads,
dff, vocab_size, dropout_rate=0.1):
super().__init__()
self.d_model = d_model
self.num_layers = num_layers
self.pos_embedding = PositionalEmbedding(
vocab_size=vocab_size, d_model=d_model)
self.enc_layers = [
EncoderLayer(d_model=d_model,
num_heads=num_heads,
dff=dff,
dropout_rate=dropout_rate)
for _ in range(num_layers)]
self.dropout = tf.keras.layers.Dropout(dropout_rate)
def call(self, x):
# `x` is token-IDs shape: (batch, seq_len)
x = self.pos_embedding(x) # Shape `(batch_size, seq_len, d_model)`.
# Add dropout.
x = self.dropout(x)
for i in range(self.num_layers):
x = self.enc_layers[i](x)
return x # Shape `(batch_size, seq_len, d_model)`.
Decoder:
# Decoder
class DecoderLayer(tf.keras.layers.Layer):
def __init__(self,
*,
d_model,
num_heads,
dff,
dropout_rate=0.1):
super(DecoderLayer, self).__init__()
self.causal_self_attention = CausalSelfAttention(
num_heads=num_heads,
key_dim=d_model,
dropout=dropout_rate)
self.cross_attention = CrossAttention(
num_heads=num_heads,
key_dim=d_model,
dropout=dropout_rate)
self.ffn = FeedForward(d_model, dff)
def call(self, x, context):
x = self.causal_self_attention(x=x)
x = self.cross_attention(x=x, context=context)
# Cache the last attention scores for plotting later
self.last_attn_scores = self.cross_attention.last_attn_scores
x = self.ffn(x) # Shape `(batch_size, seq_len, d_model)`.
return x
class Decoder(tf.keras.layers.Layer):
def __init__(self, *, num_layers, d_model, num_heads, dff, vocab_size,
dropout_rate=0.1):
super(Decoder, self).__init__()
self.d_model = d_model
self.num_layers = num_layers
self.pos_embedding = PositionalEmbedding(vocab_size=vocab_size,
d_model=d_model)
self.dropout = tf.keras.layers.Dropout(dropout_rate)
self.dec_layers = [
DecoderLayer(d_model=d_model, num_heads=num_heads,
dff=dff, dropout_rate=dropout_rate)
for _ in range(num_layers)]
self.last_attn_scores = None
def call(self, x, context):
# `x` is token-IDs shape (batch, target_seq_len)
x = self.pos_embedding(x) # (batch_size, target_seq_len, d_model)
x = self.dropout(x)
for i in range(self.num_layers):
x = self.dec_layers[i](x, context)
self.last_attn_scores = self.dec_layers[-1].last_attn_scores
# The shape of x is (batch_size, target_seq_len, d_model).
return x
The Final Transformer Model (tying up all the pieces):
## Final Transformer Architecture
class Transformer(tf.keras.Model):
def __init__(self, *, num_layers, d_model, num_heads, dff,
input_vocab_size, target_vocab_size, dropout_rate=0.1):
super().__init__()
self.encoder = Encoder(num_layers=num_layers, d_model=d_model,
num_heads=num_heads, dff=dff,
vocab_size=input_vocab_size,
dropout_rate=dropout_rate)
self.decoder = Decoder(num_layers=num_layers, d_model=d_model,
num_heads=num_heads, dff=dff,
vocab_size=target_vocab_size,
dropout_rate=dropout_rate)
self.final_layer = tf.keras.layers.Dense(target_vocab_size)
def call(self, inputs):
# To use a Keras model with `.fit` you must pass all your inputs in the
# first argument.
context, x = inputs
context = self.encoder(context) # (batch_size, context_len, d_model)
x = self.decoder(x, context) # (batch_size, target_len, d_model)
# Final linear layer output.
logits = self.final_layer(x) # (batch_size, target_len, target_vocab_size)
try:
# Drop the keras mask, so it doesn't scale the losses/metrics.
# b/250038731
del logits._keras_mask
except AttributeError:
pass
# Return the final output and the attention weights.
return logits
num_layers = 3
d_model = 128
dff = 512
num_heads = 8
dropout_rate = 0.1
transformer = Transformer(
num_layers=num_layers,
d_model=d_model,
num_heads=num_heads,
dff=dff,
input_vocab_size=tokenizers.pt.get_vocab_size().numpy(),
target_vocab_size=tokenizers.en.get_vocab_size().numpy(),
dropout_rate=dropout_rate)
Congratulations! Your Transformer model has successfully navigated the training journey. Now, the thrill of putting it to the test arrives! Prepare to witness its translation prowess in action.
But the exploration doesn't stop there. We delve deeper to comprehend the inner workings of the model. By visualizing the attention heads, we gain insights into how it processes languages, identifies crucial connections, and ultimately generates translations.
Imagine unlocking a secret window into the model's thought process, observing how it analyzes each word, its relationship to others, and how that understanding shapes the translated output. This unveils the intricate dance of attention, allowing us to appreciate the model's brilliance and identify potential areas for further improvement.
class Translator(tf.Module):
def __init__(self, tokenizers, transformer):
self.tokenizers = tokenizers
self.transformer = transformer
def __call__(self, sentence, max_length=MAX_TOKENS):
# The input sentence is Portuguese, hence adding the `[START]` and `[END]` tokens.
assert isinstance(sentence, tf.Tensor)
if len(sentence.shape) == 0:
sentence = sentence[tf.newaxis]
sentence = self.tokenizers.pt.tokenize(sentence).to_tensor()
encoder_input = sentence
# As the output language is English, initialize the output with the
# English `[START]` token.
start_end = self.tokenizers.en.tokenize([''])[0]
start = start_end[0][tf.newaxis]
end = start_end[1][tf.newaxis]
# `tf.TensorArray` is required here (instead of a Python list), so that the
# dynamic-loop can be traced by `tf.function`.
output_array = tf.TensorArray(dtype=tf.int64, size=0, dynamic_size=True)
output_array = output_array.write(0, start)
for i in tf.range(max_length):
output = tf.transpose(output_array.stack())
predictions = self.transformer([encoder_input, output], training=False)
# Select the last token from the `seq_len` dimension.
predictions = predictions[:, -1:, :] # Shape `(batch_size, 1, vocab_size)`.
predicted_id = tf.argmax(predictions, axis=-1)
# Concatenate the `predicted_id` to the output which is given to the
# decoder as its input.
output_array = output_array.write(i+1, predicted_id[0])
if predicted_id == end:
break
output = tf.transpose(output_array.stack())
# The output shape is `(1, tokens)`.
text = tokenizers.en.detokenize(output)[0] # Shape: `()`.
tokens = tokenizers.en.lookup(output)[0]
# `tf.function` prevents us from using the attention_weights that were
# calculated on the last iteration of the loop.
# So, recalculate them outside the loop.
self.transformer([encoder_input, output[:,:-1]], training=False)
attention_weights = self.transformer.decoder.last_attn_scores
return text, tokens, attention_weights
translator = Translator(tokenizers, transformer)
def print_translation(sentence, tokens, ground_truth):
print(f'{"Input:":15s}: {sentence}')
print(f'{"Prediction":15s}: {tokens.numpy().decode("utf-8")}')
print(f'{"Ground truth":15s}: {ground_truth}')
sentence = 'este é o primeiro livro que eu fiz.'
ground_truth = "this is the first book i've ever done."
translated_text, translated_tokens, attention_weights = translator(
tf.constant(sentence))
print_translation(sentence, translated_text, ground_truth)
Let’s visualize the attention heads for the above example:
def plot_attention_head(in_tokens, translated_tokens, attention):
# The model didn't generate `` in the output. Skip it.
translated_tokens = translated_tokens[1:]
ax = plt.gca()
ax.matshow(attention)
ax.set_xticks(range(len(in_tokens)))
ax.set_yticks(range(len(translated_tokens)))
labels = [label.decode('utf-8') for label in in_tokens.numpy()]
ax.set_xticklabels(
labels, rotation=90)
labels = [label.decode('utf-8') for label in translated_tokens.numpy()]
ax.set_yticklabels(labels)
head = 0
# Shape: `(batch=1, num_heads, seq_len_q, seq_len_k)`.
attention_heads = tf.squeeze(attention_weights, 0)
attention = attention_heads[head]
in_tokens = tf.convert_to_tensor([sentence])
in_tokens = tokenizers.pt.tokenize(in_tokens).to_tensor()
in_tokens = tokenizers.pt.lookup(in_tokens)[0]
plot_attention_head(in_tokens, translated_tokens, attention)
While exploring, one attention head offered a glimpse into the model's inner workings – but it's merely a single piece of the puzzle. To truly grasp its intricate language processing, we need to unveil the grand tapestry of all attention heads.
Think of it like trying to understand a complex painting by examining just one brushstroke. By studying the interplay of all attention heads, we gain a holistic view of how the model analyzes relationships between words, identifies key connections, and ultimately guides the translation process.
Each head acts as a unique lens, focusing on different aspects of the input sentence. It's by combining these diverse perspectives that the model paints a full picture of the meaning and generates nuanced translations.
sentence = 'Eu li sobre triceratops na enciclopédia.'
ground_truth = 'I read about triceratops in the encyclopedia.'
translated_text, translated_tokens, attention_weights = translator(
tf.constant(sentence))
print_translation(sentence, translated_text, ground_truth)
plot_attention_weights(sentence, translated_tokens, attention_weights[0])
Conclusion
This exploration has taken us on a thrilling journey through the world of Transformer-based machine translation. You've witnessed the power of attention, delved into the model's inner workings, and gained valuable insights into its translation prowess.
But remember, this is just the beginning. The true potential of your NMT model lies in its ability to scale and translate real-world data efficiently. This is where the combined power of E2E Cloud GPUs and NVIDIA NGC comes into play.
E2E's Cloud GPUs offer a robust and scalable platform specifically designed for AI workloads like NMT. The GPUs, coupled with the optimized pipelines and tools available through NVIDIA NGC, significantly accelerate training and inference, allowing you to handle larger datasets and achieve faster translation speeds.
Imagine translating massive volumes of text, powering real-time communication platforms, or enabling multilingual content creation – all with the efficiency and scalability provided by E2E and NGC.
So, don't let your exploration end here. Leverage the power of GPUs to push the boundaries of machine translation, unlock new possibilities, and bridge the gap between languages like never before.
That the contents of third-party articles/blogs published here on the website, and the interpretation of all information in the article/blogs such as data, maps, numbers, opinions etc. displayed in the article/blogs and views or the opinions expressed within the content are solely of the author's; and do not reflect the opinions and beliefs of NASSCOM or its affiliates in any manner. NASSCOM does not take any liability w.r.t. content in any manner and will not be liable in any manner whatsoever for any kind of liability arising out of any act, error or omission. The contents of third-party article/blogs published, are provided solely as convenience; and the presence of these articles/blogs should not, under any circumstances, be considered as an endorsement of the contents by NASSCOM in any manner; and if you chose to access these articles/blogs , you do so at your own risk.
2023 had seen a surge of interest in Large Language Models in India. This thriving curiosity was because of the mass level adoption of ChatGPT whose first version used GPT model. Experts stated that GPT 3.5 was largely trained on English language…
There are a lot of changes occurring around the world concerning technology and advancement. The autonomous vehicle system is one of the incredible advances introduced worldwide in recent years. This technology has been implemented in various…
The advent of wearable technology has revolutionized the way people monitor their health and wellness. What started as simple fitness trackers has now evolved into sophisticated devices that play a crucial role in personalized healthcare.
These…
The transportation sector is much-needed sector for the globe as it carries goods, people, products, etc from one place to another or you can say from one corner of the world to another corner of the world. It is one of the fast-growing sectors as…
The difference between Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) lies in their methodologies and purposes in training machine learning models, particularly in natural language processing.
Supervised Fine-…
At present, Artificial Intelligence (AI) is changing the game in various industries, and asset management is certainly not left behind. As asset managers aim to incorporate ESG factors into their investment strategies, AI has gained prominence owing…
