textattack-nlp-transformer/docs/examples/1_Introduction_and_Transformations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The TextAttack🐙 ecosystem: search, transformations, and constraints\n",
    "\n",
    "An attack in TextAttack consists of four parts.\n",
    "\n",
    "### Goal function\n",
    "\n",
    "The **goal function** determines if the attack is successful or not. One common goal function is **untargeted classification**, where the attack tries to perturb an input to change its classification. \n",
    "\n",
    "### Search method\n",
    "The **search method** explores the space of potential transformations and tries to locate a successful perturbation. Greedy search, beam search, and brute-force search are all examples of search methods.\n",
    "\n",
    "### Transformation\n",
    "A **transformation** takes a text input and transforms it, replacing words or phrases with similar ones, while trying not to change the meaning. Paraphrase and synonym substitution are two broad classes of transformations.\n",
    "\n",
    "### Constraints\n",
    "Finally, **constraints** determine whether or not a given transformation is valid. Transformations don't perfectly preserve syntax or semantics, so additional constraints can increase the probability that these qualities are preserved from the source to adversarial example. There are many types of constraints: overlap constraints that measure edit distance, syntactical  constraints check part-of-speech and grammar errors, and semantic constraints like language models and sentence encoders."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A custom transformation\n",
    "\n",
    "This lesson explains how to create a custom transformation. In TextAttack, many transformations involve *word swaps*: they take a word and try and find suitable substitutes. Some attacks focus on replacing characters with neighboring characters to create \"typos\" (these don't intend to preserve the grammaticality of inputs). Other attacks rely on semantics: they take a word and try to replace it with semantic equivalents.\n",
    "\n",
    "\n",
    "### Banana word swap 🍌\n",
    "\n",
    "As an introduction to writing transformations for TextAttack, we're going to try a very simple transformation: one that replaces any given word with the word 'banana'. In TextAttack, there's an abstract `WordSwap` class that handles the heavy lifting of breaking sentences into words and avoiding replacement of stopwords. We can extend `WordSwap` and implement a single method, `_get_replacement_words`, to indicate to replace each word with 'banana'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from textattack.transformations.word_swap import WordSwap\n",
    "\n",
    "class BananaWordSwap(WordSwap):\n",
    "    \"\"\" Transforms an input by replacing any word with 'banana'.\n",
    "    \"\"\"\n",
    "    \n",
    "    # We don't need a constructor, since our class doesn't require any parameters.\n",
    "\n",
    "    def _get_replacement_words(self, word):\n",
    "        \"\"\" Returns 'banana', no matter what 'word' was originally.\n",
    "        \n",
    "            Returns a list with one item, since `_get_replacement_words` is intended to\n",
    "                return a list of candidate replacement words.\n",
    "        \"\"\"\n",
    "        return ['banana']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Using our transformation\n",
    "\n",
    "Now we have the transformation chosen, but we're missing a few other things. To complete the attack, we need to choose the **search method** and **constraints**. And to use the attack, we need a **goal function**, a **model** and a **dataset**. (The goal function indicates the task our model performs – in this case, classification – and the type of attack – in this case, we'll perform an untargeted attack.)\n",
    "\n",
    "### Creating the goal function, model, and dataset\n",
    "We are performing an untargeted attack on a classification model, so we'll use the `UntargetedClassification` class. For the model, let's use an LSTM trained for news classification on the AG News dataset. Luckily, TextAttack comes with 1000 text samples from some popular datasets, as well as pretrained models for those datasets. So we don't have to train our own model, or procure someone else's. We can just use the built-in datasets and models for this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34;1mtextattack\u001b[0m: Goal function <class 'textattack.goal_functions.classification.untargeted_classification.UntargetedClassification'> matches model LSTMForAGNewsClassification.\n"
     ]
    }
   ],
   "source": [
    "# Import the dataset.\n",
    "from textattack.datasets.classification import AGNews\n",
    "# Create the model.\n",
    "from textattack.models.classification.lstm import LSTMForAGNewsClassification\n",
    "model = LSTMForAGNewsClassification()\n",
    "# Create the goal function using the model.\n",
    "from textattack.goal_functions import UntargetedClassification\n",
    "goal_function = UntargetedClassification(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating the attack\n",
    "Let's keep it simple: let's use a greedy search method, and let's not use any constraints for now. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from textattack.search_methods import GreedySearch\n",
    "from textattack.constraints.semantics import RepeatModification, StopwordModification\n",
    "from textattack.shared import Attack\n",
    "\n",
    "# We're going to use our Banana word swap class as the attack transformation.\n",
    "transformation = BananaWordSwap() \n",
    "# We'll constrain modificaiton of already modified indices and stopwords\n",
    "constraints = [RepeatModification(),\n",
    "               StopwordModification()]\n",
    "# We'll use the Greedy search method\n",
    "search_method = GreedySearch()\n",
    "# Now, let's make the attack from the 4 components:\n",
    "attack = Attack(goal_function, constraints, transformation, search_method)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's print our attack to see all the parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attack(\n",
      "  (search_method): GreedySearch\n",
      "  (goal_function):  UntargetedClassification\n",
      "  (transformation):  BananaWordSwap\n",
      "  (constraints): \n",
      "    (0): RepeatModification\n",
      "    (1): StopwordModification\n",
      "  (is_black_box):  True\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "print(attack)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the attack\n",
    "\n",
    "Let's use our attack to attack 10 samples (by setting `num_examples` to 10). Additionally, we set `attack_n` to `True`, which indicates that we should attack 10 samples, no matter what. If the model mispredicts a sample already, it isn't attacked; since `attack_n` is `True`, if a sample is mispredicted, we'll take try the next thing in the dataset, and continue until `num_examples` attacks have been completed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "12it [00:00, 13.71it/s]                       \n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm # tqdm provides us a nice progress bar.\n",
    "from textattack.loggers import CSVLogger # tracks a dataframe for us.\n",
    "\n",
    "results_iterable = attack.attack_dataset(AGNews(), num_examples=10, attack_n=True)\n",
    "results = []\n",
    "\n",
    "logger = CSVLogger(color_method='html')\n",
    "\n",
    "for result in tqdm(results_iterable, total=10):\n",
    "    logger.log_attack_result(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing attack results\n",
    "\n",
    "We are logging `AttackResult` objects using a `CSVLogger`. This logger stores all attack results in a dataframe, which we can easily access and display. Since we set `color_method` to `'html'`, the attack results will display their differences, in color, in HTML. Using `IPython` utilities and `pandas`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>original_text</th>\n",
       "      <th>perturbed_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Thirst, Fear and <font color = red>Bribes</font> on Desert Escape from <font color = red>Africa</font> AGADEZ, Niger (Reuters) - Customs officers in this dusty Saharan town turned a blind eye as yet another creaking <font color = red>truck</font> <font color = red>piled</font> with <font color = red>grain</font>, <font color = red>smuggled</font> <font color = red>cigarettes</font> and <font color = red>dozens</font> of <font color = red>migrants</font> heading for Europe rumbled off into the desert.</td>\n",
       "      <td>Thirst, Fear and <font color = blue>banana</font> on Desert Escape from <font color = blue>banana</font> AGADEZ, Niger (Reuters) - Customs officers in this dusty Saharan town turned a blind eye as yet another creaking <font color = blue>banana</font> <font color = blue>banana</font> with <font color = blue>banana</font>, <font color = blue>banana</font> <font color = blue>banana</font> and <font color = blue>banana</font> of <font color = blue>banana</font> heading for Europe rumbled off into the desert.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Toshiba 20 TV freaks out, sends distress signal See what happens when your warranty runs out?. In this case, a 20 Toshiba owned by Chris van Rossman started sending out the international distress signal at 121.</td>\n",
       "      <td>Toshiba 20 TV freaks out, sends distress signal See what happens when your warranty runs out?. In this case, a 20 Toshiba owned by Chris van Rossman started sending out the international distress signal at 121.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>British hostage tried fleeing before death: report The portrait of Ken Bigley, who was murdered in Iraq October 7, stands in front of the congregation during a service at Liverpool #39;s Roman Catholic Cathedral on October 10.</td>\n",
       "      <td>British hostage tried fleeing before death: report The portrait of Ken Bigley, who was murdered in Iraq October 7, stands in front of the congregation during a service at Liverpool #39;s Roman Catholic Cathedral on October 10.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td><font color = purple>Keychain</font> clicker kills TVs <font color = purple>Discrete</font> <font color = purple>device</font> <font color = purple>turns</font> off <font color = purple>televisions</font>, creating a little peace and quiet. Until the yelling starts.</td>\n",
       "      <td><font color = blue>banana</font> clicker kills TVs <font color = blue>banana</font> <font color = blue>banana</font> <font color = blue>banana</font> off <font color = blue>banana</font>, creating a little peace and quiet. Until the yelling starts.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td><font color = red>Cleric</font> <font color = red>returns</font> to <font color = red>Iraq</font>, orders march on <font color = red>Najaf</font> <font color = red>Powerful</font> Shiite <font color = red>leader</font> <font color = red>says</font> he <font color = red>plans</font> to lead a <font color = red>mass</font> <font color = red>demonstration</font> <font color = red>today</font> to <font color = red>end</font> <font color = red>fighting</font>. <font color = red>Iraqi</font> <font color = red>hostage</font>: <font color = red>Militants</font> <font color = red>said</font> <font color = red>Wednesday</font> they had <font color = red>kidnapped</font> the <font color = red>brother</font>-in-<font color = red>law</font> of <font color = red>Iraqi</font> <font color = red>Defense</font> Minister Hazem <font color = red>Shaalan</font></td>\n",
       "      <td><font color = blue>banana</font> <font color = blue>banana</font> to <font color = blue>banana</font>, orders march on <font color = blue>banana</font> <font color = blue>banana</font> Shiite <font color = blue>banana</font> <font color = blue>banana</font> he <font color = blue>banana</font> to lead a <font color = blue>banana</font> <font color = blue>banana</font> <font color = blue>banana</font> to <font color = blue>banana</font> <font color = blue>banana</font>. <font color = blue>banana</font> <font color = blue>banana</font>: <font color = blue>banana</font> <font color = blue>banana</font> <font color = blue>banana</font> they had <font color = blue>banana</font> the <font color = blue>banana</font>-in-<font color = blue>banana</font> of <font color = blue>banana</font> <font color = blue>banana</font> Minister Hazem <font color = blue>banana</font></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td><font color = green>Hewitt</font> <font color = green>Beats</font> Roddick to Reach Masters Final HOUSTON (Reuters) - A fired-up Lleyton Hewitt defused hard-hitting Andy Roddick 6-3, 6-2 on Saturday, scurrying into the final of the <font color = green>Masters</font> <font color = green>Cup</font> for the <font color = green>third</font> <font color = green>time</font> in <font color = green>four</font> <font color = green>years</font>.</td>\n",
       "      <td><font color = blue>banana</font> <font color = blue>banana</font> Roddick to Reach Masters Final HOUSTON (Reuters) - A fired-up Lleyton Hewitt defused hard-hitting Andy Roddick 6-3, 6-2 on Saturday, scurrying into the final of the <font color = blue>banana</font> <font color = blue>banana</font> for the <font color = blue>banana</font> <font color = blue>banana</font> in <font color = blue>banana</font> <font color = blue>banana</font>.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td><font color = blue>Despite</font> booming economy, no concrete <font color = blue>move</font> on <font color = blue>debt</font> <font color = blue>relief</font> (AFP) AFP - Senior <font color = blue>finance</font> <font color = blue>officials</font> have <font color = blue>hailed</font> a <font color = blue>robust</font> <font color = blue>global</font> economic <font color = blue>recovery</font>, albeit one threatened by surging oil prices, but made little headway pushing China toward currency reform and took no firm steps to ease the debt of the world's poorest nations.</td>\n",
       "      <td><font color = red>banana</font> booming economy, no concrete <font color = red>banana</font> on <font color = red>banana</font> <font color = red>banana</font> (AFP) AFP - Senior <font color = red>banana</font> <font color = red>banana</font> have <font color = red>banana</font> a <font color = red>banana</font> <font color = red>banana</font> economic <font color = red>banana</font>, albeit one threatened by surging oil prices, but made little headway pushing China toward currency reform and took no firm steps to ease the debt of the world's poorest nations.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td><font color = red>Ethiopian</font> court sentences 3 former rebels to death for mass murders (Canadian Press) <font color = red>Canadian</font> <font color = red>Press</font> - <font color = red>ADDIS</font> <font color = red>ABABA</font>, <font color = red>Ethiopia</font> (<font color = red>AP</font>) - A court has sentenced three <font color = red>former</font> <font color = red>rebels</font> to <font color = red>death</font> for <font color = red>killing</font> dozens of people while rebel <font color = red>factions</font> <font color = red>jockeyed</font> for <font color = red>power</font> more than a <font color = red>decade</font> <font color = red>ago</font>, a government <font color = red>spokesman</font> <font color = red>said</font> <font color = red>Thursday</font>.</td>\n",
       "      <td><font color = purple>banana</font> court sentences 3 former rebels to death for mass murders (Canadian Press) <font color = purple>banana</font> <font color = purple>banana</font> - <font color = purple>banana</font> <font color = purple>banana</font>, <font color = purple>banana</font> (<font color = purple>banana</font>) - A court has sentenced three <font color = purple>banana</font> <font color = purple>banana</font> to <font color = purple>banana</font> for <font color = purple>banana</font> dozens of people while rebel <font color = purple>banana</font> <font color = purple>banana</font> for <font color = purple>banana</font> more than a <font color = purple>banana</font> <font color = purple>banana</font>, a government <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font>.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Just a close call for closer No need to check with your primary care provider about the origin of that nervous tic you woke up with this morning, on the first full <font color = green>day</font> of <font color = green>autumn</font>. <font color = green>Ninth</font>-inning home runs <font color = green>allowed</font> in three consecutive games by Sox closer Keith Foulke, who also was tagged with <font color = green>blown</font> <font color = green>saves</font> in each of the last two <font color = green>games</font>, were enough to leave ...</td>\n",
       "      <td>Just a close call for closer No need to check with your primary care provider about the origin of that nervous tic you woke up with this morning, on the first full <font color = blue>banana</font> of <font color = blue>banana</font>. <font color = blue>banana</font>-inning home runs <font color = blue>banana</font> in three consecutive games by Sox closer Keith Foulke, who also was tagged with <font color = blue>banana</font> <font color = blue>banana</font> in each of the last two <font color = blue>banana</font>, were enough to leave ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td><font color = purple>Study</font>: Wild Monkeys Resort to <font color = purple>Use</font> of Tools WASHINGTON - <font color = purple>Wild</font> <font color = purple>South</font> <font color = purple>American</font> monkeys routinely use fist-sized rocks to crack open seeds and to dig in dry Brazilian soil for grubs and edible tubers, researchers report in the journal <font color = purple>Science</font>.</td>\n",
       "      <td><font color = blue>banana</font>: Wild Monkeys Resort to <font color = blue>banana</font> of Tools WASHINGTON - <font color = blue>banana</font> <font color = blue>banana</font> <font color = blue>banana</font> monkeys routinely use fist-sized rocks to crack open seeds and to dig in dry Brazilian soil for grubs and edible tubers, researchers report in the journal <font color = blue>banana</font>.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Bryant's Request to Seal Evidence Opposed (AP) AP - The prosecutor who charged <font color = green>Kobe</font> Bryant with felony sexual assault has joined news organizations in opposing an attempt by the <font color = green>NBA</font> star's attorney to permanently seal evidence and documents in the case.</td>\n",
       "      <td>Bryant's Request to Seal Evidence Opposed (AP) AP - The prosecutor who charged <font color = blue>banana</font> Bryant with felony sexual assault has joined news organizations in opposing an attempt by the <font color = blue>banana</font> star's attorney to permanently seal evidence and documents in the case.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td><font color = green>Eagles</font> have lift off Crystal Palace were triumphant <font color = green>last</font> night (<font color = green>Oct</font> 4) over 10-man Fulham at Selhurst Park, lifting themselves off the bottom of the Premiership.</td>\n",
       "      <td><font color = blue>banana</font> have lift off Crystal Palace were triumphant <font color = blue>banana</font> night (<font color = blue>banana</font> 4) over 10-man Fulham at Selhurst Park, lifting themselves off the bottom of the Premiership.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.options.display.max_colwidth = 480 # increase colum width so we can actually read the examples\n",
    "\n",
    "from IPython.core.display import display, HTML\n",
    "display(HTML(logger.df[['original_text', 'perturbed_text']].to_html(escape=False)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Conclusion 🍌\n",
    "\n",
    "We can examine these examples for a good idea of how many words had to be changed to \"banana\" to change the prediction score from the correct class to another class. The examples without perturbed words were originally misclassified, so they were skipped by the attack. Looks like some examples needed only a single \"banana\", while others needed up to 17 \"banana\" substitutions to change the class score. Wow!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}