Text Preprocessing Techniques You Must Know for NLP

Introduction

What is Text Preprocessing?

Text Preprocessing is a method that converts text into numerical vector.

Why we used text preprocessing?

Machine learning algorithms cannot understand raw text directly. Therefore, we use text preprocessing to convert text into numerical or mathematical values

Steps of text preprocessing

Step 1: Lowercasing - Convert all text in to lower case 

Example: "Hi My Name Is Rajat" → "hi my name is rajat"

Step 2: Stop Words- It Remove common words which add little meaning (e.g., "is" , "the" , "and")

Example: "hi my name is rajat and i am learning AI/ML" →  "my name rajat learning AI/ML"

Step 3: Stemming- It is used for reducing words to their root form from which it helps group similar words together so that the machine treats them as the same feature.

            "Taste ,Tasty ,Tasteful" → "Tasty(Root word)"

Example: Original: "I like the taste of this tasty and tasteful cake"

                 After Stemming: "I like the taste of this taste and taste cake"

       

Step 4: Tokenization- Splitting sentences into smaller part of words  

Example: "I am learning AI and ML" → "I" , "am" , "learning" , "AI" , "and"  , "ML"

Step 5: Lemmatization- Reducing Words to their dictionary form (not python dictionary)  

Original Word- "Running", "better", "Studies" → Lemming Words - "Run", "good", "study".

Example: Original: "I am running and studying AI daily"

                 After Lemmatization: "I am run and study AI daily"

Converting Text into Numbers (Vectorization)

Methods of Vectorization

1. Bag Of Words(Bow):

Bag of Words (BoW) is a method used to count how many times each word appears in a document. It converts text into a numerical vector based on word frequency, ignoring grammar and word order.

For Example:  Doc1: This Product is very good and affordable.
                        Doc2: This Product is good with afforfable price

Step 1: Build Vocabulary

             Vocabulary = [this, product, is, very, good, and, affordable, with, price]

step 2: 

Document	this	product	is	very	good	and	affordable	with	price
Doc1	1	1	1	1	1	1	1	0	0
Doc2	1	1	1	0	1	0	1	1	1

Each document is now representing with the numerical vector

word in document is not present it will get 0.

Advantages of Bow:

Simple and Easy to Implement

• Very straightforward to understand and code.
• Good for beginners learning NLP and text classification.

Disadvantages of Bow:

• it will ignore the word order like "very good" and "good very" it will count as same
• No Semantic Understanding like Words with similar meaning are treated as completely different

          Example: "happy" and "joyful" → Count as different vectors

2. TF/IDF 

TF-Term Frequency / IDF- Inverse Document Frequency

How much i understand about this is used for checking the word importance

like "is", "the", "and more like this get low score and rare words get high score

Now how to find the TF/IDF

Formulas: TF=Number of time terms repeated / Total Number of Terms

                  IDF= log(Total Number of Documents/Number of documents containing the word) 

                  TF × IDF

lets take one example :

doc1= "I am Computer Science Engineer"

doc2="Engineer builds computer systems"

So TF (Engineer) = 1/5 = 0.20

     IDF (Engineer) = 2/2= 0

     TF * IDF= 0.20*0 = 0