At a Glance

Name: LLM Foundations: Get started with tokenization
Price: Free CAD
Rating: 4.5 (41 reviews)
Author: sina_nazeri, kang_wang

Tokenization is a preprocessing technique in natural language processing (NLP) that converts text to structured data so a computer can understand human language. It breaks down unstructured text data into smaller units called tokens. A single token can range from a single character or individual word to much larger textual units.

Tokenization is a stage in text-mining pipelines that converts raw text data into a structured format for machine processing. It's a required step for other preprocessing techniques, so it's usually one of the first preprocessing steps in NLP pipelines. In this project, you’ll learn how to tokenize raw text data for use in machine learning models and NLP tasks. You'll use the Python natural language toolkit (NLTK) to convert .txt files to tokens at different levels of granularity using an open-access text file sourced largely from Project Gutenberg.

This project is based on the IBM Developer tutorial Tokenizing text in Python, by Jacob Murel (Ph.D).

A Look at the Project Ahead

Introduction to tokenization concepts in text processing.
Exploring different methods and libraries for tokenizing text in Python.
Practical examples and exercises to apply tokenization techniques.

What You'll Need

A basic knowledge of Python and a browser.

Offered By: IBMSkillsNetwork

LLM Foundations: Get started with tokenization

At a Glance

A Look at the Project Ahead

What You'll Need