Natural Language Processing

✕

Introduction to NLP

  • Enables machines to understand and generate human language.
  • Use Case: sentiment analysis, translation, recommender system etc.
  • Challenges: ambiguity, context understanding, language variability.
  • Python's nltk Library is used for NLP tasks. pip install nltk
  • nltk provides tools for tokenization, stemming, stopword removal, POS tagging etc.

Text Preprocessing Steps

  • Tokenization: Split text into words or sentences.
  • Stemming: Reduce words to their root form (e.g. running => run).
  • Stopword Removal: Remove common words that do not add much meaning (e.g. the, is).
  • Vectorization: Convert text into numerical features using techniques like Bag of Words or TF-IDF.
Example of Text Preprocessing Techniques and their Effects:
StepEffect on Text 1Effect on Text 2
Original TextPython is a programming language.I love learning new languages!
Tokenization['Python', 'is', 'a', 'programming', 'language']['I', 'love', 'learning', 'new', 'languages']
Stemming['python', 'is', 'a', 'program', 'languag']['i', 'love', 'learn', 'new', 'languag']
Remove Stopword['python', 'program', 'languag']['love', 'learn', 'new', 'languag']
Vocabulary Creation['python', 'program', 'languag', 'love', 'learn', 'new']
Vectorization[1, 1, 1, 0, 0, 0][0, 0, 1, 1, 1, 1]
Example of Text Preprocessing Techniques and their Effects.

POS Tagging

  • POS Tagging: Assign part-of-speech tags to each word (e.g. noun, verb).
  • Helps in understanding grammatical structure and meaning.
  • nltk provides pos_tag function for POS tagging.
  • Example: from nltk import word_tokenize from nltk import pos_tag tokens = word_tokenize("Python is great!") pos_tags = pos_tag(tokens) print(pos_tags) # [("Python", "NNP"), ("is", "VBZ"), ("great", "JJ")]

Cosine Similarity

  • Cosine Similarity: Measures the cosine of the angle between two vectors.
  • Used to determine the similarity between documents.
  • Value ranges from -1 (opposite) to 1 (identical).
  • Example: from sklearn.metrics.pairwise import cosine_similarity vector1 = [[1, 0, 1]] vector2 = [[0, 1, 1]] similarity = cosine_similarity(vector1, vector2) print(similarity) # [[0.5]]

Practice QuestionsNot started

  1. Job Recommendation System

    Question 1 of 1

    • Write a Python code that takes input of user's CV and recommends job category. 1. Load job_description.txt into Pandas DataFrame. 2. Assign title and description as two series. 3. Use TfidfVectorizer to convert text data into numerical vectors. 4. Ask user for resume and transform into vector using the fitted vectorizer. 5. Calculate cosine similarity between user CV and each job description. 6. Recommend the job category with highest similarity score.