running inference#
Now that you have a sense of how things work on the HF website, we are going to practice running inference on Google Colab.
Our goal is to create a text generator, using Python code, taking the following steps:
Will use the model, “gpt-neo-125m”, importing this model into the colab coding space.
Then we will write code that processes an input text to generate an output, a continuation.
Finally, we will import a dataset from the library and practice running inference with it.
We’ll talk about some programming concepts along the way, like variables and data types, and how to access data from different types and structures. We will grapple with a new data type, a dict, and how to access or manipulate data from that type.
Installations#
on google colab#
First, on the toolbar, where it says RAM DISK, change the hardware accelator to GPU.
Then, download the below libraries to your colab environment:
%% capture
%pip install transformers trl
on jupyter#
Run this in your terminal to download and install transformers with:
conda install transformers datasets
pip3 install trl torch
If installing with conda doesn’t work, you can install all packages using just pip:
pip3 install transformers datasets trl torch
And if that doesn’t work (perhaps because you’re on a CPU-only system), try:
pip install transformers[torch] datasets trl
Read more about installations here: https://huggingface.co/docs/transformers/installation
# to check that you've installed correctly, run:
!python3 -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('Filipa is the coolest prof ever'))"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'transformers'
After installing, go back to the models page. Search for gpt-neo, select 125m. On the top right, click on “Use in Transformers.” Copy that code, and paste it to your notebook.
from transformers import pipeline
# if you have a GPU (Mac M1 chip)
pipe = pipeline("text-generation", model="EleutherAI/gpt-neo-125m", device = 1)
# if you do not have a GPU
# pipe = pipeline("text-generation", model="EleutherAI/gpt-neo-125m")
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
Cell In[2], line 1
----> 1 from transformers import pipeline
3 # if you have a GPU (Mac M1 chip)
4 pipe = pipeline("text-generation", model="EleutherAI/gpt-neo-125m", device = 1)
ModuleNotFoundError: No module named 'transformers'
Here we have a function, called pipeline(), which takes parameters (a
fancy word for input).
The parameters specify the task and the model that we will be using.
We save the function to a variable called pipe, which we will later
use to process our prompt.
inference#
Now we are going to “run inference.”
First, we will type up a prompt, and save it to a variable prompt. Then we will pass that prompt to the pipe variable that we created before, saving the output to a new variable, called output.
prompt = "Hello, my name is Filipa and"
pipe(prompt, max_length = 50)
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
[{'generated_text': "Hello, my name is Filipa and I'm a newbie in the world of web development. I'm a newbie in the world of web development. I'm a newbie in the world of web development. I'm a newbie in"}]
# saving the output
output = pipe(prompt, max_length = 50)
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
Now let’s look at the response, and inspect the data structure contained within it, which is a list.
list is a collection of objects, or bits of information. So our output is saved as this collection type of object.
output
[{'generated_text': "Hello, my name is Filipa and I'm a newbie in the world of web development. I'm a newbie in the world of web development. I'm a newbie in the world of web development. I'm a newbie in"}]
type(output)
list
What if we wanted to extract just the output text, not the rest of the data, how would we go about it? We use list indexing. When we check the type, we find out the first item of the list is inside another data type, a dict.
output[0]
{'generated_text': "Hello, my name is Filipa and I'm a newbie in the world of web development. I'm a newbie in the world of web development. I'm a newbie in the world of web development. I'm a newbie in"}
type(output[0])
dict
To get items from a dict, you use a different method, accessing them by their keys.
output[0]['generated_text']
"Hello, my name is Filipa and I'm a newbie in the world of web development. I'm a newbie in the world of web development. I'm a newbie in the world of web development. I'm a newbie in"
accessing data from datasets:#
Now we will practice what we’ve learned about accessing data on the Datasets library from HF.
import datasets
# install the library and import dataset loader
# %%capture
# !pip install datasets
from datasets import load_dataset
# load the dataset and its subset
dataset = load_dataset("gofilipa/bedtime_stories")
# check the dataset object
dataset
DatasetDict({
train: Dataset({
features: ['stories'],
num_rows: 199
})
})
type(dataset)
datasets.dataset_dict.DatasetDict
# how do we get items from a dict? by the key
dataset['train']
Dataset({
features: ['stories'],
num_rows: 199
})
# how would we get the second row from this dataset?
dataset['train']['stories'][0]
'The stars twinkled in the night sky as little Eva lay in her bed dreaming of adventures. She imagined flying on the back of a dragon across the clouds, soaring over the tallest mountains. Just then, Eva heard a gentle whisper that seemed to come from everywhere. It said, "It\'s time for beddy-byes now, dream sweetly little one." Eva smiled and snuggled into her cozy blankets, dreaming of more magical places.'
outputs = []
for i in dataset['train']['stories'][:5]:
out = pipe(i, max_new_tokens=100)
outputs.append(out)
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
outputs
[[{'generated_text': 'The stars twinkled in the night sky as little Eva lay in her bed dreaming of adventures. She imagined flying on the back of a dragon across the clouds, soaring over the tallest mountains. Just then, Eva heard a gentle whisper that seemed to come from everywhere. It said, "It\'s time for beddy-byes now, dream sweetly little one." Eva smiled and snuggled into her cozy blankets, dreaming of more magical places.\n\nThe next morning, Eva woke up with a start. She was in the middle of a dream, and she was in the middle of a dream. She was in the middle of a dream. She was in the middle of a dream. She was in the middle of a dream. She was in the middle of a dream. She was in the middle of a dream. She was in the middle of a dream. She was in the middle of a dream. She was in the middle'}],
[{'generated_text': "Once upon a time there was a little girl named Sam. Every night she snuggled into her warm bed for a cozy sleep. One night, while snuggling with her teddy bear, Sam heard a gentle voice coming from the window. She got up to investigate and saw a soft, glowing light outside. It was a fairy with a big smile who asked Sam to join her on a magical adventure. They flew around the night sky, visiting different planets and galaxies. When the sun began to rise they returned back to Sam's window. The magical fairy gave Sam a hug and Sam went back to bed filled with wonderful memories.\n\nThe next morning, Sam woke up with a strange feeling. She was in a dream. She was in a dream. She was in a dream. She was in a dream. She was in a dream. She was in a dream. She was in a dream. She was in a dream. She was in a dream. She was in a dream. She was in a dream. She was in a dream. She was in a dream. She was in a dream. She was"}],
[{'generated_text': "Alice was so excited for her first sleepover at her best friend's house. After playing all night and eating delicious snacks, they both got into their pajamas and laid down in the bed. They discussed all their favorite stories until their eyes got heavy and they both drifted off to dreamland while the stars shimmered above them. As they slowly fell asleep, Alice and her friend were filled with joy knowing they'd have more adventures to come.\n\nAlice was so excited for her first sleepover at her best friend's house. After playing all night and eating delicious snacks, they both got into their pajamas and lay down in the bed. They discussed all their favorite stories until their eyes got heavy and they both drifted off to dreamland while the stars shimmered above them. As they slowly fell asleep, Alice and her friend were filled with joy knowing they'd have more adventures to come.\n\nAlice was so excited for"}],
[{'generated_text': 'Once upon a time, there lived two best friends, a squirrel and a rabbit. Every day they spent playing together in the meadow, chasing each other around the tall grass and laughing with glee. At night, they snuggled together in a cozy nook of a tree, dreaming about their next adventure. One night, when the stars shone like little diamonds in the sky, they heard a magical song from a mysterious musician. As they followed the music, they discovered a small pond, full of fireflies that lit up the water and filled the air with even more wondrous tunes. From that day forward, every night before bed, they’d sit by the pond and listen to the enchanting song, until their dreams were over.\n\nThe next day, they’d wake up and find themselves in the middle of a forest, surrounded by a huge, beautiful, and beautiful world. They’d sit on the ground and listen to the song, and then they’d sit on the grass and dream about the world.\n\nThe next day, they’d wake up and find themselves in the middle of a forest, surrounded by a huge, beautiful, and beautiful world.'}],
[{'generated_text': 'Max loved the stars. Every night he would lay in bed and count each and every one. One night, he wished on the brightest star. When the morning came, a surprise was waiting for him at his doorstep! Max opened the package to find a jar of stars – millions of them! He now had all the stars he ever wanted!\n\nHe took the jar and opened it. The stars were there, but they were not there. He opened the jar and found a tiny star. He opened the jar and found a tiny star. He opened the jar and found a tiny star. He opened the jar and found a tiny star. He opened the jar and found a tiny star. He opened the jar and found a tiny star. He opened the jar and found a tiny star. He opened the jar and found a tiny star'}]]
outs = []
for i in outputs:
out = i[0]['generated_text']
outs.append(out)
outs
['The stars twinkled in the night sky as little Eva lay in her bed dreaming of adventures. She imagined flying on the back of a dragon across the clouds, soaring over the tallest mountains. Just then, Eva heard a gentle whisper that seemed to come from everywhere. It said, "It\'s time for beddy-byes now, dream sweetly little one." Eva smiled and snuggled into her cozy blankets, dreaming of more magical places.\n\nThe next morning, Eva woke up with a start. She was in the middle of a dream, and she was in the middle of a dream. She was in the middle of a dream. She was in the middle of a dream. She was in the middle of a dream. She was in the middle of a dream. She was in the middle of a dream. She was in the middle of a dream. She was in the middle of a dream. She was in the middle',
"Once upon a time there was a little girl named Sam. Every night she snuggled into her warm bed for a cozy sleep. One night, while snuggling with her teddy bear, Sam heard a gentle voice coming from the window. She got up to investigate and saw a soft, glowing light outside. It was a fairy with a big smile who asked Sam to join her on a magical adventure. They flew around the night sky, visiting different planets and galaxies. When the sun began to rise they returned back to Sam's window. The magical fairy gave Sam a hug and Sam went back to bed filled with wonderful memories.\n\nThe next morning, Sam woke up with a strange feeling. She was in a dream. She was in a dream. She was in a dream. She was in a dream. She was in a dream. She was in a dream. She was in a dream. She was in a dream. She was in a dream. She was in a dream. She was in a dream. She was in a dream. She was in a dream. She was in a dream. She was",
"Alice was so excited for her first sleepover at her best friend's house. After playing all night and eating delicious snacks, they both got into their pajamas and laid down in the bed. They discussed all their favorite stories until their eyes got heavy and they both drifted off to dreamland while the stars shimmered above them. As they slowly fell asleep, Alice and her friend were filled with joy knowing they'd have more adventures to come.\n\nAlice was so excited for her first sleepover at her best friend's house. After playing all night and eating delicious snacks, they both got into their pajamas and lay down in the bed. They discussed all their favorite stories until their eyes got heavy and they both drifted off to dreamland while the stars shimmered above them. As they slowly fell asleep, Alice and her friend were filled with joy knowing they'd have more adventures to come.\n\nAlice was so excited for",
'Once upon a time, there lived two best friends, a squirrel and a rabbit. Every day they spent playing together in the meadow, chasing each other around the tall grass and laughing with glee. At night, they snuggled together in a cozy nook of a tree, dreaming about their next adventure. One night, when the stars shone like little diamonds in the sky, they heard a magical song from a mysterious musician. As they followed the music, they discovered a small pond, full of fireflies that lit up the water and filled the air with even more wondrous tunes. From that day forward, every night before bed, they’d sit by the pond and listen to the enchanting song, until their dreams were over.\n\nThe next day, they’d wake up and find themselves in the middle of a forest, surrounded by a huge, beautiful, and beautiful world. They’d sit on the ground and listen to the song, and then they’d sit on the grass and dream about the world.\n\nThe next day, they’d wake up and find themselves in the middle of a forest, surrounded by a huge, beautiful, and beautiful world.',
'Max loved the stars. Every night he would lay in bed and count each and every one. One night, he wished on the brightest star. When the morning came, a surprise was waiting for him at his doorstep! Max opened the package to find a jar of stars – millions of them! He now had all the stars he ever wanted!\n\nHe took the jar and opened it. The stars were there, but they were not there. He opened the jar and found a tiny star. He opened the jar and found a tiny star. He opened the jar and found a tiny star. He opened the jar and found a tiny star. He opened the jar and found a tiny star. He opened the jar and found a tiny star. He opened the jar and found a tiny star. He opened the jar and found a tiny star']