{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tools: Working with Text\n",
    "\n",
    "## Introduction\n",
    "Text is a common form of data, and it is important to know how to work with it. In this notebook, we will cover some of the most common tools and techniques for working with text data in Python.\n",
    "\n",
    "This notebook is just meant to be a quick introduction to some of the most common tools and techniques for working with text data in Python. There are many other tools and techniques available, so be sure to explore further if you are interested in learning more."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What to Install\n",
    "\n",
    "To begin working with Python and text as data, you'll need to set up a few essential tools. Below is a list of recommended software to install, along with brief explanations of why they’re useful and how they fit into the workflow:\n",
    "\n",
    "### 1. Visual Studio Code (VS Code)\n",
    "[VS Code](https://code.visualstudio.com/) is a free, open-source, and highly customizable code editor developed by Microsoft. It supports many programming languages, including Python, and offers extensions that make coding more efficient and enjoyable.\n",
    "- **Why use it?** \n",
    "  VS Code provides a rich environment with features like syntax highlighting, autocompletion, debugging, integrated Git control, and customizable extensions. It’s lightweight yet powerful, making it ideal for beginners and advanced users alike.\n",
    "- **Installation Instructions:**\n",
    "  - Download and install VS Code from [the official website](https://code.visualstudio.com/Download).\n",
    "  - After installation, open VS Code and install the **Python extension** (which will provide features like syntax checking, code completion, and running Python scripts directly from the editor).\n",
    "\n",
    "### 2. Python\n",
    "[Python](https://www.python.org/) is the programming language you'll be using throughout the course. It’s widely used for data analysis, machine learning, and natural language processing, and has a simple syntax that makes it accessible for beginners.\n",
    "- **Why use it?** \n",
    "  Python is not only beginner-friendly, but also comes with a wide range of libraries for working with text, data, and machine learning, such as `pandas`, `numpy`, `matplotlib` and `scikit-learn`.\n",
    "- **Installation Instructions:**\n",
    "  - Python is pre-installed on many systems, but you may need to install it manually if it’s not already available.\n",
    "  - Download and install the latest version of Python from [python.org](https://www.python.org/downloads/).\n",
    "  - Ensure that you check the option to **Add Python to PATH** during installation, so that you can run Python from the command line.\n",
    "\n",
    "### 3. Jupyter Notebook (via Anaconda)\n",
    "[Jupyter Notebook](https://jupyter.org/) is an open-source tool that allows you to create and share documents that contain live code, equations, visualizations, and narrative text. It’s especially useful for data analysis, experimentation, and learning how to work with Python.\n",
    "- **Why use it?**\n",
    "  Jupyter provides an interactive coding environment where you can run Python code in separate cells, visualize data, and write explanations all in the same document. It’s widely used in data science and academia for its ability to combine code and explanations seamlessly.\n",
    "- **Installation Instructions:**\n",
    "  - The easiest way to install Jupyter Notebook, along with other important libraries like `numpy`, `pandas`, and `matplotlib`, is by downloading **Anaconda**.\n",
    "  - [Download Anaconda](https://www.anaconda.com/products/individual) and install it on your system. Anaconda comes with Jupyter Notebook and several key data science libraries pre-installed, simplifying the setup process.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python basics\n",
    "\n",
    "### Variables and Data Types\n",
    "\n",
    "In Python, variables are used to store data. The data stored can be of various types, such as:\n",
    "\n",
    "- Integers: Whole numbers\n",
    "- Floats: Decimal numbers\n",
    "- Strings: Text data\n",
    "- Booleans: True or False values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Assigning different data types to variables\n",
    "integer_var = 42\n",
    "float_var = 3.14\n",
    "string_var = \"Hello, Python!\"\n",
    "boolean_var = True\n",
    "\n",
    "# Printing the variables\n",
    "print(\"Integer:\", integer_var)\n",
    "print(\"Float:\", float_var)\n",
    "print(\"String:\", string_var)\n",
    "print(\"Boolean:\", boolean_var)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic String Operations\n",
    "\n",
    "Strings are sequences of characters and are essential for text analysis. Here are some basic operations you can perform on strings:\n",
    "\n",
    "- Concatenation: Combining strings using `+`\n",
    "- Repetition: Repeating strings using `*`\n",
    "- Indexing: Accessing individual characters\n",
    "- Slicing: Accessing substrings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# String concatenation\n",
    "greeting = \"Hello\"\n",
    "name = \"Alice\"\n",
    "message = greeting + \", \" + name + \"!\"\n",
    "print(\"Concatenated String:\", message)\n",
    "\n",
    "# String repetition\n",
    "repeat_str = \"ha\" * 3\n",
    "print(\"Repeated String:\", repeat_str)\n",
    "\n",
    "# String indexing and slicing\n",
    "sample_text = \"Python\"\n",
    "print(\"First character:\", sample_text[0])\n",
    "print(\"Last two characters:\", sample_text[-2:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lists and Loops\n",
    "\n",
    "Lists are ordered collections of items, which can be of different data types. Loops allow you to iterate over collections, which is very useful in text processing.\n",
    "\n",
    "- Lists: Created using square brackets `[]`\n",
    "- For Loops: Iterate over each item in a collection\n",
    "- List Comprehensions: Concise way to create lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a list of words\n",
    "words_list = [\"Python\", \"is\", \"great\", \"for\", \"text\", \"analysis\"]\n",
    "\n",
    "# Using a for loop to iterate over the list\n",
    "for word in words_list:\n",
    "    print(word)\n",
    "\n",
    "# Using list comprehension to get the length of each word\n",
    "word_lengths = [len(word) for word in words_list]\n",
    "print(\"Word Lengths:\", word_lengths)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python basics for text processing\n",
    "\n",
    "- Key Python concepts (variables, data types, loops, functions, etc.)\n",
    "- Working with strings (slicing, concatenation, formatting, methods like `split`, `strip`, `replace`, `lower`,  etc.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining a function to clean and process text\n",
    "def clean_text(text):\n",
    "    # Remove leading and trailing whitespace\n",
    "    text = text.strip()\n",
    "    # Convert text to lowercase\n",
    "    text = text.lower()\n",
    "    # Replace punctuation with nothing\n",
    "    text = text.replace('.', '').replace(',', '')\n",
    "    return text\n",
    "\n",
    "# Sample text\n",
    "raw_text = \"   Hello, World! Welcome to Python text processing.   \"\n",
    "\n",
    "raw_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using the function\n",
    "processed_text = clean_text(raw_text)\n",
    "print(\"Processed Text:\", processed_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Splitting the text into words\n",
    "words = processed_text.split()\n",
    "print(\"List of Words:\", words)\n",
    "\n",
    "# Replacing a word in the text\n",
    "replaced_text = processed_text.replace(\"python\", \"advanced Python\")\n",
    "print(\"Replaced Text:\", replaced_text)\n",
    "\n",
    "# Converting text to uppercase\n",
    "upper_text = processed_text.upper()\n",
    "print(\"Uppercase Text:\", upper_text)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Regular Expressions (Regex) in Python\n",
    "\n",
    "Regular expressions (often shortened to \"regex\") are a powerful tool for matching patterns in text. They allow us to search, extract, and manipulate strings based on defined patterns, making them extremely useful when working with text data. Python’s built-in `re` library provides a wide range of functions to work with regular expressions.\n",
    "\n",
    "Some of the most commonly used functions in the `re` library include:\n",
    "\n",
    "- `re.search()`: Searches a string for the first location where a pattern matches.\n",
    "- `re.match()`: Checks if the beginning of a string matches a pattern.\n",
    "- `re.findall()`: Returns all non-overlapping matches of a pattern in a string.\n",
    "- `re.sub()`: Substitutes occurrences of a pattern in a string with another string.\n",
    "\n",
    "Below is a basic example of how to use the `re` library in Python to find a pattern in a text string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the regular expressions library\n",
    "import re\n",
    "\n",
    "# Example string\n",
    "text = \"The rain in Spain falls mainly on the plain.\"\n",
    "\n",
    "# Find all words that end with \"ain\"\n",
    "pattern = r'\\b\\w*ain\\b'\n",
    "matches = re.findall(pattern, text)\n",
    "\n",
    "# Print the result\n",
    "print(\"Words that match the pattern:\", matches)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Text Data in Pandas\n",
    "\n",
    "Pandas is a powerful data manipulation library in Python that makes it easy to work with structured data, such as CSV files or Excel sheets. \n",
    "\n",
    "Pandas gives easy-to-use data structures and data analysis tools for the Python programming language. It is widely used in data science, machine learning, and other related fields.\n",
    "\n",
    "However, it also has robust capabilities for working with text data. \n",
    "\n",
    "Some key functions and methods for working with text in Pandas include:\n",
    "\n",
    "- `.str`: Allows vectorized string operations, making it easy to apply text transformations across entire columns.\n",
    "- `str.contains()`: Checks whether a pattern or substring exists in each element of the Series.\n",
    "- `str.replace()`: Replaces occurrences of a pattern or substring with another string.\n",
    "- `str.split()`: Splits strings based on a delimiter.\n",
    "- `str.extract()`: Extracts substrings based on regular expressions.\n",
    "\n",
    "By leveraging these methods, you can clean, manipulate, and analyze text data directly within Pandas DataFrames. Below is a basic example demonstrating how to work with text data in a Pandas DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "\n",
    "# Create a sample DataFrame with text data\n",
    "data = {\n",
    "    'Name': ['Alice Smith', 'Bob Johnson', 'Charlie Brown', 'Diana Prince'],\n",
    "    'Occupation': ['Data Scientist', 'Software Developer', 'Data Analyst', 'AI Researcher'],\n",
    "    'Location': ['New York, USA', 'London, UK', 'San Francisco, USA', 'Oslo, Norway']\n",
    "}\n",
    "\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use .str methods to manipulate text data\n",
    "\n",
    "# 1. Split the 'Name' column into First Name and Last Name\n",
    "df['First Name'] = df['Name'].str.split().str[0]\n",
    "df['Last Name'] = df['Name'].str.split().str[1]\n",
    "\n",
    "# 2. Check if the 'Location' column contains 'USA'\n",
    "df['In USA'] = df['Location'].str.contains('USA')\n",
    "\n",
    "# 3. Replace 'Data' with 'ML' in the 'Occupation' column\n",
    "df['Occupation'] = df['Occupation'].str.replace('Data', 'ML')\n",
    "\n",
    "# Display the modified DataFrame\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Text Libraries in Python\n",
    "\n",
    "Python has a rich ecosystem of libraries designed to help process, analyze, and model text data. Here are three of the most commonly used libraries in the field of Natural Language Processing (NLP):\n",
    "\n",
    "### 5.1 `nltk` (Natural Language Toolkit)\n",
    "`nltk` is one of the oldest and most comprehensive libraries for text processing and natural language processing in Python. It provides tools for working with tokenization, stemming, tagging, parsing, and much more. It also comes with a vast collection of datasets and corpora, making it an excellent starting point for beginners.\n",
    "\n",
    "### 5.2 `scikit-learn`\n",
    "`scikit-learn` is a popular machine learning library in Python. While it’s not dedicated to NLP, it provides a number of key tools for working with text, such as `CountVectorizer` and `TfidfVectorizer`, which are essential for converting raw text into numerical representations that machine learning models can work with.\n",
    "\n",
    "### 5.3 `spaCy`\n",
    "`spaCy` is a modern, fast, and industrial-strength NLP library designed for large-scale text processing. It excels at tasks such as tokenization, part-of-speech tagging, named entity recognition (NER), and dependency parsing. `spaCy` is known for its performance and ease of use, especially in production environments where speed is critical.\n",
    "\n",
    "Below are examples of how to get started with each library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# nltk Example (Tokenization)\n",
    "\n",
    "import nltk\n",
    "from nltk.tokenize import word_tokenize, sent_tokenize\n",
    "\n",
    "# Download necessary NLTK data (only needs to be done once)\n",
    "nltk.download('punkt')\n",
    "nltk.download('punkt_tab')\n",
    "\n",
    "# Example text\n",
    "text = \"Natural language processing with Python is fun. It's exciting to explore text data!\"\n",
    "\n",
    "# Sentence tokenization\n",
    "sentences = sent_tokenize(text)\n",
    "print(\"Sentences:\", sentences)\n",
    "\n",
    "# Word tokenization\n",
    "words = word_tokenize(text)\n",
    "print(\"Words:\", words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# scikit-learn Example (Vectorization)\n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# Example text data\n",
    "texts = [\n",
    "    \"I love programming in Python.\",\n",
    "    \"Python is great for machine learning.\",\n",
    "    \"I enjoy working with text data.\"\n",
    "]\n",
    "\n",
    "# Initialize the CountVectorizer\n",
    "vectorizer = CountVectorizer()\n",
    "\n",
    "# Fit and transform the text data to a bag-of-words model\n",
    "X = vectorizer.fit_transform(texts)\n",
    "\n",
    "# Convert the result to an array\n",
    "print(\"Bag of Words (BoW) representation:\\n\", X.toarray())\n",
    "\n",
    "# Display the vocabulary\n",
    "print(\"Vocabulary:\", vectorizer.get_feature_names_out())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# spaCy Example (Named Entity Recognition)\n",
    "\n",
    "\n",
    "import spacy\n",
    "\n",
    "# Load the spaCy English model\n",
    "# python -m spacy download en_core_web_sm\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "# Example text\n",
    "text = \"Apple is looking at buying U.K. startup for $1 billion.\"\n",
    "\n",
    "# Process the text with spaCy\n",
    "doc = nlp(text)\n",
    "\n",
    "# Extract named entities\n",
    "for entity in doc.ents:\n",
    "    print(f\"Entity: {entity.text}, Label: {entity.label_}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Data Cleaning and Preprocessing\n",
    "\n",
    "When working with text data, raw text often contains noise that can interfere with analysis or machine learning tasks. Data cleaning and preprocessing are essential steps to convert raw text into a more useful and consistent format. Proper preprocessing ensures that the text is in a form that machine learning models or analytical tools can understand and process efficiently.\n",
    "\n",
    "### 6.1 Why Preprocessing is Important\n",
    "\n",
    "- **Cleaning**: Raw text may contain irrelevant characters, inconsistent formats, or noisy data that needs to be removed to improve analysis.\n",
    "- **Standardizing**: Text data often contains variations in case, spelling, or formatting. Standardization ensures consistency across the dataset.\n",
    "- **Normalizing**: Normalization involves reducing words to their root forms to group variations of a word together (e.g., \"running\" → \"run\").\n",
    "\n",
    "### 6.2 Common Preprocessing Steps\n",
    "\n",
    "Here are some common preprocessing steps used to clean and prepare text data:\n",
    "\n",
    "- **Removing Punctuation**: Punctuation marks are typically irrelevant for text analysis or machine learning models.\n",
    "- **Lowercasing**: Converting all characters to lowercase ensures that \"Apple\" and \"apple\" are treated the same.\n",
    "- **Removing Stopwords**: Stopwords (like \"and\", \"the\", \"is\") are common words that don't carry much semantic meaning and are often removed.\n",
    "- **Stemming**: Reduces words to their base or root form (e.g., \"running\" → \"run\") by chopping off suffixes.\n",
    "- **Lemmatization**: Similar to stemming but more sophisticated, it reduces words to their base form based on vocabulary and context (e.g., \"better\" → \"good\").\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#  Basic Text Cleaning (Lowercasing and Removing Punctuation)\n",
    "\n",
    "import string\n",
    "\n",
    "# Example text\n",
    "text = \"Hello! This is a sample text, full of punctuations. Let's clean it up.\"\n",
    "\n",
    "# Convert to lowercase\n",
    "text_clean = text.lower()\n",
    "\n",
    "# Remove punctuation\n",
    "text_clean = text_clean.translate(str.maketrans(\"\", \"\", string.punctuation))\n",
    "\n",
    "print(\"Cleaned Text:\", text_clean)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Removing Stopwords with nltk\n",
    "\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "# Example text\n",
    "text_clean = \"this is a simple example showing how to remove stopwords from text\"\n",
    "\n",
    "# Define stopwords in English\n",
    "stop_words = set(stopwords.words('english'))\n",
    "\n",
    "# Remove stopwords\n",
    "words = text_clean.split()\n",
    "filtered_text = [word for word in words if word not in stop_words]\n",
    "\n",
    "print(\"Text after removing stopwords:\", \" \".join(filtered_text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stemming and Lemmatization with nltk and spaCy\n",
    "\n",
    "import nltk\n",
    "from nltk.stem import PorterStemmer\n",
    "import spacy\n",
    "\n",
    "# Download NLTK's wordnet data (needed for lemmatization)\n",
    "nltk.download('wordnet')\n",
    "nltk.download('omw-1.4')\n",
    "\n",
    "# Initialize the Porter Stemmer\n",
    "stemmer = PorterStemmer()\n",
    "\n",
    "# Load spaCy's English model for lemmatization\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "\n",
    "# Example text\n",
    "text = \"The cats are running faster than the dogs.\"\n",
    "\n",
    "# Stemming with NLTK\n",
    "stemmed_words = [stemmer.stem(word) for word in text.split()]\n",
    "print(\"Stemmed Words:\", stemmed_words)\n",
    "\n",
    "# Lemmatization with spaCy\n",
    "doc = nlp(text)\n",
    "lemmatized_words = [token.lemma_ for token in doc]\n",
    "print(\"Lemmatized Words:\", lemmatized_words)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (nlp)",
   "language": "python",
   "name": "nlp"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}