Summarizing Text With facebook/bart-large-cnn Model

In this short tutorial about summarizing text we worked on Google Colab, so go get an account here if you don’t have one already.

Here’s a link to this small project, where we basically summarized a chunk of text with a Facebook/bart-large-cnn model. BART is particularly effective when fine-tuned for text generation, such as summarization, translation but also works well for comprehension tasks, for example text classification and question answering. This particular checkpoint has been fine-tuned on CNN Daily Mail, a large collection of text-summary pairs.

First, we need to install transformers and torch:

!pip install transformers 
!pip install torch

Next, the following code is using the Transformers library, which is commonly used for natural language processing tasks like text classification, translation, and question answering, among others:

from transformers.utils import logging
logging.set_verbosity_error()

The line from transformers.utils import logging imports the logging module from the Transformers library. This module provides utilities for logging information during the execution of the code, which can be useful for debugging and monitoring.

The next line logging.set_verbosity_error() sets the logging verbosity level to “error”. This means that only error messages will be displayed in the log output, suppressing lower severity messages like warnings or informational messages. This can be helpful if you want to reduce the amount of information displayed during the execution of your code, especially if you’re primarily interested in errors or critical issues.

Next, this code snippet imports two libraries: transformers and torch.

from transformers import pipeline 
import torch

from transformers import pipeline: This line imports the pipeline module from the transformers library. The transformers library is a popular open-source library developed by Hugging Face, primarily used for natural language processing (NLP) tasks such as text classification, named entity recognition, question answering, and more. The pipeline module allows you to easily construct pipelines for various NLP tasks, abstracting away much of the complexity involved in setting up and using pre-trained models for these tasks.

import torch: This line imports the torch library. torch is the core library for PyTorch, an open-source machine learning framework primarily developed by Facebook’s AI Research lab (FAIR). PyTorch is widely used for building and training deep learning models, including those for NLP tasks. In the context of using the transformers library, PyTorch is often used as the backend for implementing neural network architectures and performing computations efficiently, especially on GPUs. By importing torch, you have access to its functionalities, including tensor operations, neural network building blocks, and GPU acceleration.

Overall, this code sets up a summarization pipeline using a specific pre-trained BART model, configured to use 16-bit floating-point precision for computations. This pipeline can then be used to generate summaries of input text efficiently:

summarizer = pipeline(task="summarization",
                      model="facebook/bart-large-cnn",
                      torch_dtype=torch.bfloat16)

pipeline: This function is used to create a processing pipeline for various natural language processing (NLP) tasks. It simplifies the process of using pre-trained models for specific tasks.

task=”summarization”: This argument specifies the task that the pipeline will perform, which in this case is summarization. The pipeline will be configured to generate summaries of input text.

model=”facebook/bart-large-cnn”: This argument specifies the pre-trained model that the pipeline will use for summarization. In this case, it’s using a model called “facebook/bart-large-cnn”. This model is a variant of BART (Bidirectional and Auto-Regressive Transformers), which is a transformer-based neural network architecture designed for sequence-to-sequence tasks like summarization. The “large” in the model name indicates that it’s a larger version of the BART model, likely with more parameters and higher capacity for capturing complex relationships in text. The “cnn” suffix indicates that it has been fine-tuned for summarization tasks, but most of all the model had been fine-tuned on CNN Daily Mail.

torch_dtype=torch.bfloat16: This argument specifies the data type to be used for computations within the pipeline. In this case, it’s set to torch.bfloat16, which is a 16-bit floating-point representation provided by PyTorch. Using bfloat16 can help reduce memory usage and accelerate computations, especially on hardware that supports it, such as certain GPUs.

The following text is about Barcelona and the text is passed through a variable “text”

text = """Barcelona is a city on the northeastern coast of Spain.
It is the capital and largest city of the autonomous community of Catalonia, as well as the second-most
populous municipality of Spain. With a population of 1.6 million within city limits, its urban area extends
to numerous neighbouring municipalities within the province of Barcelona and is home to around 4.8 million people,
making it the fifth most populous urban area in the European Union after Paris, the Ruhr area, Madrid and Milan.
It is one of the largest metropolises on the Mediterranean Sea, located on the coast between the mouths of the
rivers Llobregat and Besòs, bounded to the west by the Serra de Collserola mountain range."""

The summarization pipeline will take the input text, process it, and generate a summary within the specified length constraints. The resulting summary will be stored in the summary variable for further processing or display:

summary = summarizer(text,
                     min_length=15,
                     max_length=90)

summary: This line declares a variable named summary, which will hold the output of the summarization process.

summarizer(text, min_length=15, max_length=90): This line calls the summarizer pipeline created earlier with the input text to be summarized. The pipeline will process the input text and generate a summary.

text: This is the input text that you want to summarize.

min_length=15: This argument specifies the minimum length of the summary to be generated. In this case, the minimum length is set to 15 tokens. This ensures that the generated summary is not too short.

max_length=90: This argument specifies the maximum length of the summ

The last code snippet gives us the results of the summary:

summary

In conclusion

in conclusion, the tutorial effectively guides you through the process of setting up a summarization pipeline using a pre-trained BART model and showcases its application on a specific example. It highlights the simplicity and power of Hugging Face’s Transformers library for NLP tasks.