textattack-nlp-transformer/docs/examples/1_Introduction_and_Transformations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The TextAttack ecosystem: search, transformations, and constraints\n",
    "\n",
    "An attack in TextAttack consists of four parts.\n",
    "\n",
    "### Goal function\n",
    "\n",
    "The **goal function** determines if the attack is successful or not. One common goal function is **untargeted classification**, where the attack tries to perturb an input to change its classification. \n",
    "\n",
    "### Search method\n",
    "The **search method** explores the space of potential transformations and tries to locate a successful perturbation. Greedy search, beam search, and brute-force search are all examples of search methods.\n",
    "\n",
    "### Transformation\n",
    "A **transformation** takes a text input and transforms it, for example replacing words or phrases with similar ones, while trying not to change the meaning. Paraphrase and synonym substitution are two broad classes of transformations.\n",
    "\n",
    "### Constraints\n",
    "Finally, **constraints** determine whether or not a given transformation is valid. Transformations don't perfectly preserve syntax or semantics, so additional constraints can increase the probability that these qualities are preserved from the source to adversarial example. There are many types of constraints: overlap constraints that measure edit distance, syntactical  constraints check part-of-speech and grammar errors, and semantic constraints like language models and sentence encoders."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A custom transformation\n",
    "\n",
    "This lesson explains how to create a custom transformation. In TextAttack, many transformations involve *word swaps*: they take a word and try and find suitable substitutes. Some attacks focus on replacing characters with neighboring characters to create \"typos\" (these don't intend to preserve the grammaticality of inputs). Other attacks rely on semantics: they take a word and try to replace it with semantic equivalents.\n",
    "\n",
    "\n",
    "### Banana word swap \n",
    "\n",
    "As an introduction to writing transformations for TextAttack, we're going to try a very simple transformation: one that replaces any given word with the word 'banana'. In TextAttack, there's an abstract `WordSwap` class that handles the heavy lifting of breaking sentences into words and avoiding replacement of stopwords. We can extend `WordSwap` and implement a single method, `_get_replacement_words`, to indicate to replace each word with 'banana'. 🍌"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from textattack.transformations import WordSwap\n",
    "\n",
    "class BananaWordSwap(WordSwap):\n",
    "    \"\"\" Transforms an input by replacing any word with 'banana'.\n",
    "    \"\"\"\n",
    "    \n",
    "    # We don't need a constructor, since our class doesn't require any parameters.\n",
    "\n",
    "    def _get_replacement_words(self, word):\n",
    "        \"\"\" Returns 'banana', no matter what 'word' was originally.\n",
    "        \n",
    "            Returns a list with one item, since `_get_replacement_words` is intended to\n",
    "                return a list of candidate replacement words.\n",
    "        \"\"\"\n",
    "        return ['banana']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Using our transformation\n",
    "\n",
    "Now we have the transformation chosen, but we're missing a few other things. To complete the attack, we need to choose the **search method** and **constraints**. And to use the attack, we need a **goal function**, a **model** and a **dataset**. (The goal function indicates the task our model performs – in this case, classification – and the type of attack – in this case, we'll perform an untargeted attack.)\n",
    "\n",
    "### Creating the goal function, model, and dataset\n",
    "We are performing an untargeted attack on a classification model, so we'll use the `UntargetedClassification` class. For the model, let's use an LSTM trained for news classification on the AG News dataset. Luckily, TextAttack comes with 1000 text samples from some popular datasets, as well as pretrained models for those datasets. So we don't have to train our own model, or procure someone else's. We can just use the built-in datasets and models for this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34;1mtextattack\u001b[0m: Goal function <class 'textattack.goal_functions.classification.untargeted_classification.UntargetedClassification'> matches model LSTMForAGNewsClassification.\n"
     ]
    }
   ],
   "source": [
    "# Import the dataset.\n",
    "from textattack.datasets.classification import AGNews\n",
    "# Create the model.\n",
    "from textattack.models.classification.lstm import LSTMForAGNewsClassification\n",
    "model = LSTMForAGNewsClassification()\n",
    "# Create the goal function using the model.\n",
    "from textattack.goal_functions import UntargetedClassification\n",
    "goal_function = UntargetedClassification(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating the attack\n",
    "Let's keep it simple: let's use a greedy search method, and let's not use any constraints for now. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from textattack.search_methods import GreedySearch\n",
    "from textattack.constraints.pre_transformation import RepeatModification, StopwordModification\n",
    "from textattack.shared import Attack\n",
    "\n",
    "# We're going to use our Banana word swap class as the attack transformation.\n",
    "transformation = BananaWordSwap() \n",
    "# We'll constrain modification of already modified indices and stopwords\n",
    "constraints = [RepeatModification(),\n",
    "               StopwordModification()]\n",
    "# We'll use the Greedy search method\n",
    "search_method = GreedySearch()\n",
    "# Now, let's make the attack from the 4 components:\n",
    "attack = Attack(goal_function, constraints, transformation, search_method)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's print our attack to see all the parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attack(\n",
      "  (search_method): GreedySearch\n",
      "  (goal_function):  UntargetedClassification\n",
      "  (transformation):  BananaWordSwap\n",
      "  (constraints): \n",
      "    (0): RepeatModification\n",
      "    (1): StopwordModification\n",
      "  (is_black_box):  True\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "print(attack)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the attack\n",
    "\n",
    "Let's use our attack to attack 10 samples (by setting `num_examples` to 10). Additionally, we set `attack_n` to `True`, which indicates that we should attack 10 samples, no matter what. If the model mispredicts a sample already, it isn't attacked; since `attack_n` is `True`, if a sample is mispredicted, we'll try the next example in the dataset, and continue until `num_examples` attacks have been completed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "12it [00:00, 19.61it/s]                       \n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm # tqdm provides us a nice progress bar.\n",
    "from textattack.loggers import CSVLogger # tracks a dataframe for us.\n",
    "\n",
    "results_iterable = attack.attack_dataset(AGNews(), num_examples=10, attack_n=True)\n",
    "results = []\n",
    "\n",
    "logger = CSVLogger(color_method='html')\n",
    "\n",
    "for result in tqdm(results_iterable, total=10):\n",
    "    logger.log_attack_result(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing attack results\n",
    "\n",
    "We are logging `AttackResult` objects using a `CSVLogger`. This logger stores all attack results in a dataframe, which we can easily access and display. Since we set `color_method` to `'html'`, the attack results will display their differences, in color, in HTML. Using `IPython` utilities and `pandas`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>original_text</th>\n",
       "      <th>perturbed_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Thirst, Fear and Bribes on <font color = red>Desert</font> Escape from Africa <font color = red>AGADEZ</font>, <font color = red>Niger</font> (Reuters) - Customs officers in this dusty Saharan town turned a blind eye as yet another creaking truck piled with grain, smuggled cigarettes and dozens of migrants heading for Europe rumbled off into the desert.</td>\n",
       "      <td>Thirst, Fear and Bribes on <font color = blue>banana</font> Escape from Africa <font color = blue>banana</font>, <font color = blue>banana</font> (Reuters) - Customs officers in this dusty Saharan town turned a blind eye as yet another creaking truck piled with grain, smuggled cigarettes and dozens of migrants heading for Europe rumbled off into the desert.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Toshiba 20 TV freaks out, sends distress signal See what happens when your warranty runs out?. In this case, a 20 Toshiba owned by Chris van Rossman started sending out the international distress signal at 121.</td>\n",
       "      <td>Toshiba 20 TV freaks out, sends distress signal See what happens when your warranty runs out?. In this case, a 20 Toshiba owned by Chris van Rossman started sending out the international distress signal at 121.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>British hostage tried fleeing before death: report The portrait of Ken Bigley, who was murdered in Iraq October 7, stands in front of the congregation during a service at Liverpool #39;s Roman Catholic Cathedral on October 10.</td>\n",
       "      <td>British hostage tried fleeing before death: report The portrait of Ken Bigley, who was murdered in Iraq October 7, stands in front of the congregation during a service at Liverpool #39;s Roman Catholic Cathedral on October 10.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td><font color = purple>Keychain</font> <font color = purple>clicker</font> kills TVs Discrete device turns off televisions, creating a little peace and quiet. Until the yelling starts.</td>\n",
       "      <td><font color = blue>banana</font> <font color = blue>banana</font> kills TVs Discrete device turns off televisions, creating a little peace and quiet. Until the yelling starts.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td><font color = red>Cleric</font> returns to <font color = red>Iraq</font>, orders march on Najaf <font color = red>Powerful</font> Shiite leader says he plans to <font color = red>lead</font> a mass demonstration today to end fighting. <font color = red>Iraqi</font> <font color = red>hostage</font>: <font color = red>Militants</font> <font color = red>said</font> Wednesday they had <font color = red>kidnapped</font> the brother-in-law of <font color = red>Iraqi</font> Defense <font color = red>Minister</font> Hazem Shaalan</td>\n",
       "      <td><font color = blue>banana</font> returns to <font color = blue>banana</font>, orders march on Najaf <font color = blue>banana</font> Shiite leader says he plans to <font color = blue>banana</font> a mass demonstration today to end fighting. <font color = blue>banana</font> <font color = blue>banana</font>: <font color = blue>banana</font> <font color = blue>banana</font> Wednesday they had <font color = blue>banana</font> the brother-in-law of <font color = blue>banana</font> Defense <font color = blue>banana</font> Hazem Shaalan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td><font color = green>Hewitt</font> Beats Roddick to <font color = green>Reach</font> Masters Final HOUSTON (Reuters) - <font color = green>A</font> fired-up Lleyton Hewitt defused hard-hitting Andy Roddick 6-3, 6-2 on Saturday, scurrying into the final of the <font color = green>Masters</font> Cup for the third time in four years.</td>\n",
       "      <td><font color = blue>banana</font> Beats Roddick to <font color = blue>banana</font> Masters Final HOUSTON (Reuters) - <font color = blue>banana</font> fired-up Lleyton Hewitt defused hard-hitting Andy Roddick 6-3, 6-2 on Saturday, scurrying into the final of the <font color = blue>banana</font> Cup for the third time in four years.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td><font color = blue>Despite</font> <font color = blue>booming</font> economy, no concrete move on <font color = blue>debt</font> relief (AFP) AFP - Senior finance officials have hailed a robust global <font color = blue>economic</font> recovery, albeit one threatened by surging <font color = blue>oil</font> prices, but made little headway pushing China toward currency reform and took no firm <font color = blue>steps</font> to ease the debt of the world's poorest nations.</td>\n",
       "      <td><font color = red>banana</font> <font color = red>banana</font> economy, no concrete move on <font color = red>banana</font> relief (AFP) AFP - Senior finance officials have hailed a robust global <font color = red>banana</font> recovery, albeit one threatened by surging <font color = red>banana</font> prices, but made little headway pushing China toward currency reform and took no firm <font color = red>banana</font> to ease the debt of the world's poorest nations.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td><font color = red>Ethiopian</font> court sentences 3 former <font color = red>rebels</font> to death for mass murders (Canadian <font color = red>Press</font>) Canadian <font color = red>Press</font> - ADDIS ABABA, <font color = red>Ethiopia</font> (AP) - A court has sentenced three former <font color = red>rebels</font> to death for <font color = red>killing</font> dozens of people while rebel factions jockeyed for power more than a decade ago, a government spokesman said Thursday.</td>\n",
       "      <td><font color = blue>banana</font> court sentences 3 former <font color = blue>banana</font> to death for mass murders (Canadian <font color = blue>banana</font>) Canadian <font color = blue>banana</font> - ADDIS ABABA, <font color = blue>banana</font> (AP) - A court has sentenced three former <font color = blue>banana</font> to death for <font color = blue>banana</font> dozens of people while rebel factions jockeyed for power more than a decade ago, a government spokesman said Thursday.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Just a close call for closer No need to check with your primary care <font color = green>provider</font> about the origin of that <font color = green>nervous</font> <font color = green>tic</font> you woke up with this morning, on the first full day of autumn. Ninth-inning home runs allowed in three consecutive <font color = green>games</font> by Sox closer <font color = green>Keith</font> Foulke, who also was tagged with blown saves in each of the last two <font color = green>games</font>, were enough to leave ...</td>\n",
       "      <td>Just a close call for closer No need to check with your primary care <font color = blue>banana</font> about the origin of that <font color = blue>banana</font> <font color = blue>banana</font> you woke up with this morning, on the first full day of autumn. Ninth-inning home runs allowed in three consecutive <font color = blue>banana</font> by Sox closer <font color = blue>banana</font> Foulke, who also was tagged with blown saves in each of the last two <font color = blue>banana</font>, were enough to leave ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td><font color = purple>Study</font>: Wild <font color = purple>Monkeys</font> Resort to Use of Tools WASHINGTON - Wild South American monkeys routinely use fist-sized rocks to crack open seeds and to dig in dry Brazilian soil for grubs and edible tubers, researchers report in the journal Science.</td>\n",
       "      <td><font color = blue>banana</font>: Wild <font color = blue>banana</font> Resort to Use of Tools WASHINGTON - Wild South American monkeys routinely use fist-sized rocks to crack open seeds and to dig in dry Brazilian soil for grubs and edible tubers, researchers report in the journal Science.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Bryant's Request to Seal Evidence Opposed (AP) AP - <font color = green>The</font> prosecutor who charged Kobe Bryant with felony sexual assault has joined news organizations in opposing an attempt by the <font color = green>NBA</font> star's attorney to permanently seal evidence and documents in the case.</td>\n",
       "      <td>Bryant's Request to Seal Evidence Opposed (AP) AP - <font color = blue>banana</font> prosecutor who charged Kobe Bryant with felony sexual assault has joined news organizations in opposing an attempt by the <font color = blue>banana</font> star's attorney to permanently seal evidence and documents in the case.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Eagles have lift off Crystal Palace were triumphant last night (Oct 4) over 10-<font color = green>man</font> Fulham at Selhurst Park, lifting themselves off the <font color = green>bottom</font> of the <font color = green>Premiership</font>.</td>\n",
       "      <td>Eagles have lift off Crystal Palace were triumphant last night (Oct 4) over 10-<font color = purple>banana</font> Fulham at Selhurst Park, lifting themselves off the <font color = purple>banana</font> of the <font color = purple>banana</font>.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.options.display.max_colwidth = 480 # increase colum width so we can actually read the examples\n",
    "\n",
    "from IPython.core.display import display, HTML\n",
    "display(HTML(logger.df[['original_text', 'perturbed_text']].to_html(escape=False)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Conclusion n",
    "\n",
    "We can examine these examples for a good idea of how many words had to be changed to \"banana\" to change the prediction score from the correct class to another class. The examples without perturbed words were originally misclassified, so they were skipped by the attack. Looks like some examples needed only a couple \"banana\"s, while others needed up to 17 \"banana\" substitutions to change the class score. Wow! 🍌"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}