Natural Language Processing

Introduction to NLP

Enables machines to understand and generate human language.
Use Case: sentiment analysis, translation, recommender system etc.
Challenges: ambiguity, context understanding, language variability.
Python's nltk Library is used for NLP tasks. pip install nltk
nltk provides tools for tokenization, stemming, stopword removal, POS tagging etc.

Text Preprocessing Steps

Tokenization: Split text into words or sentences.
Stemming: Reduce words to their root form (e.g. running => run).
Stopword Removal: Remove common words that do not add much meaning (e.g. the, is).
Vectorization: Convert text into numerical features using techniques like Bag of Words or TF-IDF.

Example of Text Preprocessing Techniques and their Effects:

Step	Effect on Text 1	Effect on Text 2
Original Text	Python is a programming language.	I love learning new languages!
Tokenization	['Python', 'is', 'a', 'programming', 'language']	['I', 'love', 'learning', 'new', 'languages']
Stemming	['python', 'is', 'a', 'program', 'languag']	['i', 'love', 'learn', 'new', 'languag']
Remove Stopword	['python', 'program', 'languag']	['love', 'learn', 'new', 'languag']
Vocabulary Creation		['python', 'program', 'languag', 'love', 'learn', 'new']
Vectorization	[1, 1, 1, 0, 0, 0]	[0, 0, 1, 1, 1, 1]

Example of Text Preprocessing Techniques and their Effects.

POS Tagging

POS Tagging: Assign part-of-speech tags to each word (e.g. noun, verb).
Helps in understanding grammatical structure and meaning.
nltk provides pos_tag function for POS tagging.
Example: from nltk import word_tokenize from nltk import pos_tag tokens = word_tokenize("Python is great!") pos_tags = pos_tag(tokens) print(pos_tags) # [("Python", "NNP"), ("is", "VBZ"), ("great", "JJ")]

Cosine Similarity

Cosine Similarity: Measures the cosine of the angle between two vectors.
Used to determine the similarity between documents.
Value ranges from -1 (opposite) to 1 (identical).
Example: from sklearn.metrics.pairwise import cosine_similarity vector1 = [[1, 0, 1]] vector2 = [[0, 1, 1]] similarity = cosine_similarity(vector1, vector2) print(similarity) # [[0.5]]

Practice QuestionsNot started

Job Recommendation System
Question 1 of 2
Write a Python code that takes input of user's CV and recommends job category. 1. Load job_description_details.txt into Pandas DataFrame. 2. Assign title and description as two series. 3. Use TfidfVectorizer to convert text data into numerical vectors. 4. Ask user for resume and transform into vector using the fitted vectorizer. 5. Calculate cosine similarity between user CV and each job description. 6. Recommend the job category with highest similarity score.
Ad Placeholder336x280 • Slot 9287725823
Transportation Complaint Classification
Question 2 of 2
Write a Python program that classifies transportation complaints into categories such as Driver, Vehicle, Fare, Route, App, and Safety. 1. Load transportation_complaints.csv into a Pandas DataFrame. 2. Assign the text column as the input and category column as the target. 3. Split the dataset into training and testing sets using an 80/20 split and random_state=42. 4. Create a Scikit-learn Pipeline containing TfidfVectorizer and SVC. 5. Use GridSearchCV to tune the TF-IDF ngram_range and SVC hyperparameters such as C and kernel. 6. Train the GridSearchCV model using the training data. 7. Store the best model using best_model = grid_search.best_estimator_. 8. Display the best hyperparameters and the model accuracy on the test data. 9. Use the best model to classify this new complaint: "The driver was speeding and I felt unsafe during the trip." 10. Display the predicted category.
Ad Placeholder336x280 • Slot 9287725823