Merge pull request #539 from QData/fix-'textattack'-has-no-attribute-'shared'-bug

Fix logger initiation bug

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,31 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Run following command `textattack ...`
+2. Run following code ...
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots or Traceback**
+If applicable, add screenshots to help explain your problem. Also, copy and paste tracebacks produced by the bug. 
+
+**System Information (please complete the following information):**
+ - OS: [e.g. MacOS, Linux, Windows]
+ - Library versions (e.g. `torch==1.7.0, transformers==3.3.0`)
+ - Textattack version
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/custom.md

@@ -0,0 +1,10 @@
+---
+name: Custom issue template
+about: Describe this issue template's purpose here.
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.github/pull_request_template.md

@@ -0,0 +1,22 @@
+# What does this PR do?
+
+## Summary
+*Example: This PR adds [CLARE](https://arxiv.org/abs/2009.07502) attack, which uses distilled RoBERTa masked language model to perform word swaps, word insertions, word merges (which is where we combine two adjacent words and replace it with another word) in a greedy manner.  s*
+
+## Additions
+- *Example: Added `clare` recipe as `textattack.attack_recipes.CLARE2020`.*
+
+## Changes
+- *Example: `WordSwapMaskedLM` has been updated to have a minimum confidence score cutoff and batching has been added for faster performance.*
+
+## Deletions
+- *Example: Remove unnecessary files under `textattack.models...`*
+
+## Checklist
+- [  ] The title of your pull request should be a summary of its contribution.
+- [  ] Please write detailed description of what parts have been newly added and what parts have been modified. Please also explain why certain changes were made.
+- [  ] If your pull request addresses an issue, please mention the issue number in the pull request description to make sure they are linked (and people consulting the issue know you   are working on it)
+- [  ] To indicate a work in progress please mark it as a draft on Github.
+- [  ] Make sure existing tests pass.
+- [  ] Add relevant tests. No quality testing = no merge.
+- [  ] All public methods must have informative docstrings that work nicely with sphinx. For new modules/files, please add/modify the appropriate `.rst` file in `TextAttack/docs/apidoc`.'

.github/workflows/codeql-analysis.yml

@@ -0,0 +1,67 @@
+# For most projects, this workflow file will not need changing; you simply need
+# to commit it to your repository.
+#
+# You may wish to alter this file to override the set of languages analyzed,
+# or to provide custom queries or build logic.
+#
+# ******** NOTE ********
+# We have attempted to detect the languages in your repository. Please check
+# the `language` matrix defined below to confirm you have the correct set of
+# supported CodeQL languages.
+#
+name: "CodeQL"
+
+on:
+  push:
+    branches: [ master, master* ]
+  pull_request:
+    # The branches below must be a subset of the branches above
+    branches: [ master ]
+  schedule:
+    - cron: '24 1 * * 0'
+
+jobs:
+  analyze:
+    name: Analyze
+    runs-on: ubuntu-latest
+
+    strategy:
+      fail-fast: false
+      matrix:
+        language: [ 'python' ]
+        # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python' ]
+        # Learn more:
+        # https://docs.github.com/en/free-pro-team@latest/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-code-scanning#changing-the-languages-that-are-analyzed
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v2
+
+    # Initializes the CodeQL tools for scanning.
+    - name: Initialize CodeQL
+      uses: github/codeql-action/init@v1
+      with:
+        languages: ${{ matrix.language }}
+        # If you wish to specify custom queries, you can do so here or in a config file.
+        # By default, queries listed here will override any specified in a config file.
+        # Prefix the list here with "+" to use these queries and those in the config file.
+        # queries: ./path/to/local/query, your-org/your-repo/queries@main
+
+    # Autobuild attempts to build any compiled languages  (C/C++, C#, or Java).
+    # If this step fails, then you should remove it and run the build manually (see below)
+    - name: Autobuild
+      uses: github/codeql-action/autobuild@v1
+
+    # ℹ️ Command-line programs to run using the OS shell.
+    # 📚 https://git.io/JvXDl
+
+    # ✏️ If the Autobuild fails above, remove it and uncomment the following three lines
+    #    and modify them (or add more) to build your code if your project
+    #    uses a compiled language
+
+    #- run: |
+    #   make bootstrap
+    #   make release
+
+    - name: Perform CodeQL Analysis
+      uses: github/codeql-action/analyze@v1

.github/workflows/make-docs.yml

@@ -31,7 +31,7 @@ jobs:
         python -m pip install --upgrade pip setuptools wheel # update python
         pip install ipython --upgrade # needed for Github for whatever reason
         python setup.py install_egg_info # Workaround https://github.com/pypa/pip/issues/4537
-        pip install -e . ".[dev]" # This should install all packages for development
+        pip install -e .[dev]
         pip install jupyter 'ipykernel<5.0.0' 'ipython<7.0.0' # ipykernel workaround: github.com/jupyter/notebook/issues/4050
     - name: Build docs with Sphinx and check for errors
       run: |

.github/workflows/publish-to-pypi.yml

@@ -5,7 +5,7 @@ name: Upload Python Package to PyPI
 
 on:
   release:
-    types: [created]
+    types: [published]
 
 jobs:
   deploy:

.github/workflows/run-pytest.yml

@@ -31,7 +31,15 @@ jobs:
         python setup.py install_egg_info # Workaround https://github.com/pypa/pip/issues/4537
         pip install -e .[dev]
         pip freeze
+    - name: Free disk space
+      run: |
+        sudo apt-get remove mysql-client libmysqlclient-dev -y >/dev/null 2>&1
+        sudo apt-get remove php* -y >/dev/null 2>&1
+        sudo apt-get autoremove -y >/dev/null 2>&1
+        sudo apt-get autoclean -y >/dev/null 2>&1
+        sudo rm -rf /usr/local/lib/android >/dev/null 2>&1
+        docker rmi $(docker image ls -aq) >/dev/null 2>&1
     - name: Test with pytest
       run: |
-        pytest tests -vx --dist=loadfile -n auto
+        pytest tests -v
         

.gitignore

@@ -36,10 +36,14 @@ dist/
 # Weights & Biases outputs
 wandb/
 
+# Tensorboard logs
+runs/
+
 # checkpoints
 checkpoints/
 
 # vim
 *.swp
 
-.vscode
+.vscode
+*.csv

.readthedocs.yml

@@ -17,3 +17,5 @@ python:
       - requirements: requirements.txt
       - method: pip
         path: .
+        extra_requirements:
+          - docs

Makefile

@@ -20,7 +20,7 @@ docs-check: FORCE ## Builds docs using Sphinx. If there is an error, exit with a
 	sphinx-build -b html docs docs/_build/html -W
 
 docs-auto: FORCE ## Build docs using Sphinx and run hotreload server using Sphinx autobuild.
-	sphinx-autobuild docs docs/_build/html -H 0.0.0.0 -p 8765
+	sphinx-autobuild docs docs/_build/html --port 8765
 
 all: format lint docs-check test ## Format, lint, and test. 
 

README.md

@@ -24,7 +24,7 @@
 
 TextAttack is a Python framework for adversarial attacks, data augmentation, and model training in NLP.
 
-> If you're looking for information about TextAttack's menagerie of pre-trained models, you might want the [TextAttack Model Zoo](textattack/models/README.md) readme.
+> If you're looking for information about TextAttack's menagerie of pre-trained models, you might want the [TextAttack Model Zoo](https://textattack.readthedocs.io/en/latest/3recipes/models.html) page.
 
 ## Slack Channel
 
@@ -58,6 +58,8 @@ or via python module (`python -m textattack ...`).
 
 ## Usage
 
+### Help: `textattack --help`
+
 TextAttack's main features can all be accessed via the `textattack` command. Two very
 common commands are `textattack attack <args>`, and `textattack augment <args>`. You can see more
 information about all commands using 
@@ -69,13 +71,18 @@ or a specific command using, for example,
 textattack attack --help
 ```
 
-The [`examples/`](examples/) folder includes scripts showing common TextAttack usage for training models, running attacks, and augmenting a CSV file. The [documentation website](https://textattack.readthedocs.io/en/latest) contains walkthroughs explaining basic usage of TextAttack, including building a custom transformation and a custom constraint..
+The [`examples/`](examples/) folder includes scripts showing common TextAttack usage for training models, running attacks, and augmenting a CSV file. 
 
-### Running Attacks
+
+The [documentation website](https://textattack.readthedocs.io/en/latest) contains walkthroughs explaining basic usage of TextAttack, including building a custom transformation and a custom constraint..
+
+
+
+### Running Attacks: `textattack attack --help`
 
 The easiest way to try out an attack is via the command-line interface, `textattack attack`. 
 
-> **Tip:** If your machine has multiple GPUs, you can distribute the attack across them using the `--parallel` option. For some attacks, this can really help performance.
+> **Tip:** If your machine has multiple GPUs, you can distribute the attack across them using the `--parallel` option. For some attacks, this can really help performance. (If you want to attack Keras models in parallel, please check out `examples/attack/attack_keras_parallel.py` instead)
 
 Here are some concrete examples:
 
@@ -86,7 +93,7 @@ textattack attack --recipe textfooler --model bert-base-uncased-mr --num-example
 
 *DeepWordBug on DistilBERT trained on the Quora Question Pairs paraphrase identification dataset*: 
 ```bash
-textattack attack --model distilbert-base-uncased-qqp --recipe deepwordbug --num-examples 100
+textattack attack --model distilbert-base-uncased-cola --recipe deepwordbug --num-examples 100
 ```
 
 *Beam search with beam width 4 and word embedding transformation and untargeted goal function on an LSTM*:
@@ -99,61 +106,192 @@ textattack attack --model lstm-mr --num-examples 20 \
 
 > **Tip:** Instead of specifying a dataset and number of examples, you can pass `--interactive` to attack samples inputted by the user.
 
-### Attacks and Papers Implemented ("Attack Recipes")
+### Attacks and Papers Implemented ("Attack Recipes"): `textattack attack --recipe [recipe_name]`
 
 We include attack recipes which implement attacks from the literature. You can list attack recipes using `textattack list attack-recipes`.
 
 To run an attack recipe: `textattack attack --recipe [recipe_name]`
 
-Attacks on classification tasks, like sentiment classification and entailment:
-- **alzantot**: Genetic algorithm attack from (["Generating Natural Language Adversarial Examples" (Alzantot et al., 2018)](https://arxiv.org/abs/1804.07998)).
-- **bae**: BERT masked language model transformation attack from (["BAE: BERT-based Adversarial Examples for Text Classification" (Garg & Ramakrishnan, 2019)](https://arxiv.org/abs/2004.01970)).
-- **bert-attack**: BERT masked language model transformation attack with subword replacements (["BERT-ATTACK: Adversarial Attack Against BERT Using BERT" (Li et al., 2020)](https://arxiv.org/abs/2004.09984)).
-- **checklist**: Invariance testing implemented in CheckList that contract, extend, and substitutes name entities. (["Beyond Accuracy: Behavioral
-    Testing of NLP models with CheckList" (Ribeiro et al., 2020)](https://arxiv.org/abs/2005.04118)).
-- **clare (*coming soon*)**: Greedy attack with word swap, insertion, and merge transformations using RoBERTa masked language model. (["Contextualized Perturbation for Textual Adversarial Attack" (Li et al., 2020)](https://arxiv.org/abs/2009.07502)).
-- **faster-alzantot**: modified, faster version of the Alzantot et al. genetic algorithm, from (["Certified Robustness to Adversarial Word Substitutions" (Jia et al., 2019)](https://arxiv.org/abs/1909.00986)).
-- **deepwordbug**: Greedy replace-1 scoring and multi-transformation character-swap attack (["Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers" (Gao et al., 2018)](https://arxiv.org/abs/1801.04354)).
-- **hotflip**: Beam search and gradient-based word swap (["HotFlip: White-Box Adversarial Examples for Text Classification" (Ebrahimi et al., 2017)](https://arxiv.org/abs/1712.06751)).
-- **iga**: Improved genetic algorithm attack from (["Natural Language Adversarial Attacks and Defenses in Word Level (Wang et al., 2019)"](https://arxiv.org/abs/1909.06723)
-- **input-reduction**: Reducing the input while maintaining the prediction through word importance ranking (["Pathologies of Neural Models Make Interpretation Difficult" (Feng et al., 2018)](https://arxiv.org/pdf/1804.07781.pdf)).
-- **kuleshov**: Greedy search and counterfitted embedding swap (["Adversarial Examples for Natural Language Classification Problems" (Kuleshov et al., 2018)](https://openreview.net/pdf?id=r1QZ3zbAZ)).
-- **pruthi**: Character-based attack that simulates common typos (["Combating Adversarial Misspellings with Robust Word Recognition" (Pruthi et al., 2019)](https://arxiv.org/abs/1905.11268)
-- **pso**: Particle swarm optimization and HowNet synonym swap (["Word-level Textual Adversarial Attacking as Combinatorial Optimization" (Zang et al., 2020)](https://www.aclweb.org/anthology/2020.acl-main.540/)).
-- **pwws**: Greedy attack with word importance ranking based on word saliency and synonym swap scores (["Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency" (Ren et al., 2019)](https://www.aclweb.org/anthology/P19-1103/)).
-- **textbugger**: Greedy attack with word importance ranking and a combination of synonym and character-based swaps ([(["TextBugger: Generating Adversarial Text Against Real-world Applications" (Li et al., 2018)](https://arxiv.org/abs/1812.05271)).
-- **textfooler**: Greedy attack with word importance ranking and counter-fitted embedding swap (["Is Bert Really Robust?" (Jin et al., 2019)](https://arxiv.org/abs/1907.11932)).
+<img src="docs/_static/imgs/overview.png" alt="TextAttack Overview" style="display: block; margin: 0 auto;" />
 
-Attacks on sequence-to-sequence models:
-- **morpheus**: Greedy attack that replaces words with their inflections with the goal of minimizing BLEU score (["It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations"](https://www.aclweb.org/anthology/2020.acl-main.263.pdf)
-- **seq2sick**: Greedy attack with goal of changing every word in the output translation. Currently implemented as black-box with plans to change to white-box as done in paper (["Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples" (Cheng et al., 2018)](https://arxiv.org/abs/1803.01128)).
+<table  style="width:100%" border="1">
+<thead>
+<tr class="header">
+<th><strong>Attack Recipe Name</strong></th>
+<th><strong>Goal Function</strong></th>
+<th><strong>ConstraintsEnforced</strong></th>
+<th><strong>Transformation</strong></th>
+<th><strong>Search Method</strong></th>
+<th><strong>Main Idea</strong></th>
+</tr>
+</thead>
+<tbody>
+  <tr><td style="text-align: center;" colspan="6"><strong><br>Attacks on classification tasks, like sentiment classification and entailment:<br></strong></td></tr>
 
-Following table illustrates the comparison of the attack models.
+<tr>
+<td><code>a2t</code> 
+<span class="citation" data-cites="yoo2021a2t"></span></td>
+<td><sub>Untargeted {Classification, Entailment}</sub></td>
+<td><sub>Percentage of words perturbed, Word embedding distance, DistilBERT sentence encoding cosine similarity, part-of-speech consistency</sub></td>
+<td><sub>Counter-fitted word embedding swap (or) BERT Masked Token Prediction</sub></td>
+<td><sub>Greedy-WIR (gradient)</sub></td>
+<td ><sub>from (["Towards Improving Adversarial Training of NLP Models" (Yoo et al., 2021)](https://arxiv.org/abs/2109.00544))</sub></td>
+</tr>
+<tr>
+<td><code>alzantot</code>  <span class="citation" data-cites="Alzantot2018GeneratingNL Jia2019CertifiedRT"></span></td>
+<td><sub>Untargeted {Classification, Entailment}</sub></td>
+<td><sub>Percentage of words perturbed, Language Model perplexity, Word embedding distance</sub></td>
+<td><sub>Counter-fitted word embedding swap</sub></td>
+<td><sub>Genetic Algorithm</sub></td>
+<td ><sub>from (["Generating Natural Language Adversarial Examples" (Alzantot et al., 2018)](https://arxiv.org/abs/1804.07998))</sub></td>
+</tr>
+<tr>
+<td><code>bae</code> <span class="citation" data-cites="garg2020bae"></span></td>
+<td><sub>Untargeted Classification</sub></td>
+<td><sub>USE sentence encoding cosine similarity</sub></td>
+<td><sub>BERT Masked Token Prediction</sub></td>
+<td><sub>Greedy-WIR</sub></td>
+<td ><sub>BERT masked language model transformation attack from (["BAE: BERT-based Adversarial Examples for Text Classification" (Garg & Ramakrishnan, 2019)](https://arxiv.org/abs/2004.01970)). </td>
+</tr>
+<tr>
+<td><code>bert-attack</code> <span class="citation" data-cites="li2020bertattack"></span></td>
+<td><sub>Untargeted Classification</td>
+<td><sub>USE sentence encoding cosine similarity, Maximum number of words perturbed</td>
+<td><sub>BERT Masked Token Prediction (with subword expansion)</td>
+<td><sub>Greedy-WIR</sub></td>
+<td ><sub> (["BERT-ATTACK: Adversarial Attack Against BERT Using BERT" (Li et al., 2020)](https://arxiv.org/abs/2004.09984))</sub></td>
+</tr>
+<tr>
+<td><code>checklist</code> <span class="citation" data-cites="Gao2018BlackBoxGO"></span></td>
+<td><sub>{Untargeted, Targeted} Classification</sub></td>
+<td><sub>checklist distance</sub></td>
+<td><sub>contract, extend, and substitutes name entities</sub></td>
+<td><sub>Greedy-WIR</sub></td>
+<td ><sub>Invariance testing implemented in CheckList . (["Beyond Accuracy: Behavioral Testing of NLP models with CheckList" (Ribeiro et al., 2020)](https://arxiv.org/abs/2005.04118))</sub></td>
+</tr>
+<tr>
+<td> <code>clare</code> <span class="citation" data-cites="Alzantot2018GeneratingNL Jia2019CertifiedRT"></span></td>
+<td><sub>Untargeted {Classification, Entailment}</sub></td>
+<td><sub>USE sentence encoding cosine similarity</sub></td>
+<td><sub>RoBERTa Masked Prediction for token swap, insert and merge</sub></td>
+<td><sub>Greedy</sub></td>
+<td ><sub>["Contextualized Perturbation for Textual Adversarial Attack" (Li et al., 2020)](https://arxiv.org/abs/2009.07502))</sub></td>
+</tr>
+<tr>
+<td><code>deepwordbug</code> <span class="citation" data-cites="Gao2018BlackBoxGO"></span></td>
+<td><sub>{Untargeted, Targeted} Classification</sub></td>
+<td><sub>Levenshtein edit distance</sub></td>
+<td><sub>{Character Insertion, Character Deletion, Neighboring Character Swap, Character Substitution}</sub></td>
+<td><sub>Greedy-WIR</sub></td>
+<td ><sub>Greedy replace-1 scoring and multi-transformation character-swap attack (["Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers" (Gao et al., 2018)](https://arxiv.org/abs/1801.04354)</sub></td>
+</tr>
+<tr>
+<td> <code>fast-alzantot</code> <span class="citation" data-cites="Alzantot2018GeneratingNL Jia2019CertifiedRT"></span></td>
+<td><sub>Untargeted {Classification, Entailment}</sub></td>
+<td><sub>Percentage of words perturbed, Language Model perplexity, Word embedding distance</sub></td>
+<td><sub>Counter-fitted word embedding swap</sub></td>
+<td><sub>Genetic Algorithm</sub></td>
+<td ><sub>Modified, faster version of the Alzantot et al. genetic algorithm, from (["Certified Robustness to Adversarial Word Substitutions" (Jia et al., 2019)](https://arxiv.org/abs/1909.00986))</sub></td>
+</tr>
+<tr>
+<td><code>hotflip</code> (word swap) <span class="citation" data-cites="Ebrahimi2017HotFlipWA"></span></td>
+<td><sub>Untargeted Classification</sub></td>
+<td><sub>Word Embedding Cosine Similarity, Part-of-speech match, Number of words perturbed</sub></td>
+<td><sub>Gradient-Based Word Swap</sub></td>
+<td><sub>Beam search</sub></td>
+<td ><sub> (["HotFlip: White-Box Adversarial Examples for Text Classification" (Ebrahimi et al., 2017)](https://arxiv.org/abs/1712.06751))</sub></td>
+</tr>
+<tr>
+<td><code>iga</code> <span class="citation" data-cites="iga-wang2019natural"></span></td>
+<td><sub>Untargeted {Classification, Entailment}</sub></td>
+<td><sub>Percentage of words perturbed, Word embedding distance</sub></td>
+<td><sub>Counter-fitted word embedding swap</sub></td>
+<td><sub>Genetic Algorithm</sub></td>
+<td ><sub>Improved genetic algorithm -based word substitution from (["Natural Language Adversarial Attacks and Defenses in Word Level (Wang et al., 2019)"](https://arxiv.org/abs/1909.06723)</sub></td>
+</tr>
+<tr>
+<td><code>input-reduction</code> <span class="citation" data-cites="feng2018pathologies"></span></td>
+<td><sub>Input Reduction</sub></td>
+<td></td>
+<td><sub>Word deletion</sub></td>
+<td><sub>Greedy-WIR</sub></td>
+<td ><sub>Greedy attack with word importance ranking , Reducing the input while maintaining the prediction through word importance ranking (["Pathologies of Neural Models Make Interpretation Difficult" (Feng et al., 2018)](https://arxiv.org/pdf/1804.07781.pdf))</sub></td>
+</tr>
+<tr>
+<td><code>kuleshov</code> <span class="citation" data-cites="Kuleshov2018AdversarialEF"></span></td>
+<td><sub>Untargeted Classification</sub></td>
+<td><sub>Thought vector encoding cosine similarity, Language model similarity probability</sub></td>
+<td><sub>Counter-fitted word embedding swap</sub></td>
+<td><sub>Greedy word swap</sub></td>
+<td ><sub>(["Adversarial Examples for Natural Language Classification Problems" (Kuleshov et al., 2018)](https://openreview.net/pdf?id=r1QZ3zbAZ)) </sub></td>
+</tr>
+<tr>
+<td><code>pruthi</code> <span class="citation" data-cites="pruthi2019combating"></span></td>
+<td><sub>Untargeted Classification</sub></td>
+<td><sub>Minimum word length, Maximum number of words perturbed</sub></td>
+<td><sub>{Neighboring Character Swap, Character Deletion, Character Insertion, Keyboard-Based Character Swap}</sub></td>
+<td><sub>Greedy search</sub></td>
+<td ><sub>simulates common typos (["Combating Adversarial Misspellings with Robust Word Recognition" (Pruthi et al., 2019)](https://arxiv.org/abs/1905.11268) </sub></td>
+</tr>
+<tr>
+<td><code>pso</code> <span class="citation" data-cites="pso-zang-etal-2020-word"></span></td>
+<td><sub>Untargeted Classification</sub></td>
+<td></td>
+<td><sub>HowNet Word Swap</sub></td>
+<td><sub>Particle Swarm Optimization</sub></td>
+<td ><sub>(["Word-level Textual Adversarial Attacking as Combinatorial Optimization" (Zang et al., 2020)](https://www.aclweb.org/anthology/2020.acl-main.540/)) </sub></td>
+</tr>
+<tr>
+<td><code>pwws</code> <span class="citation" data-cites="pwws-ren-etal-2019-generating"></span></td>
+<td><sub>Untargeted Classification</sub></td>
+<td></td>
+<td><sub>WordNet-based synonym swap</sub></td>
+<td><sub>Greedy-WIR (saliency)</sub></td>
+<td ><sub>Greedy attack with word importance ranking based on word saliency and synonym swap scores (["Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency" (Ren et al., 2019)](https://www.aclweb.org/anthology/P19-1103/))</sub> </td>
+</tr>
+<tr>
+<td><code>textbugger</code> : (black-box) <span class="citation" data-cites="Li2019TextBuggerGA"></span></td>
+<td><sub>Untargeted Classification</sub></td>
+<td><sub>USE sentence encoding cosine similarity</sub></td>
+<td><sub>{Character Insertion, Character Deletion, Neighboring Character Swap, Character Substitution}</sub></td>
+<td><sub>Greedy-WIR</sub></td>
+<td ><sub>([(["TextBugger: Generating Adversarial Text Against Real-world Applications" (Li et al., 2018)](https://arxiv.org/abs/1812.05271)).</sub></td>
+</tr>
+<tr>
+<td><code>textfooler</code> <span class="citation" data-cites="Jin2019TextFooler"></span></td>
+<td><sub>Untargeted {Classification, Entailment}</sub></td>
+<td><sub>Word Embedding Distance, Part-of-speech match, USE sentence encoding cosine similarity</sub></td>
+<td><sub>Counter-fitted word embedding swap</sub></td>
+<td><sub>Greedy-WIR</sub></td>
+<td ><sub>Greedy attack with word importance ranking  (["Is Bert Really Robust?" (Jin et al., 2019)](https://arxiv.org/abs/1907.11932))</sub> </td>
+</tr>
 
-Attacks on classification tasks
-| Attack Recipe(s)                  |  Accessibility  | Perturbation | Main Idea                                                                                         |
-| :-------------------------------: | :-------------: | :----------: | :-----------------------------------------------------------------------------------------------: |
-| Alzantot Genetic Algorithm        |      Score      |     Word     | Genetic algorithm-based word substitution                                                         |
-| BAE*                              |      Score      |     Word     | BERT masked language model transformation attack                                                  |
-| Faster Alzantot Genetic Algorithm |      Score      |     Word     | Genetic algorithm-based word substitution(faster version)                                         |
-| Improved Genetic Algorithm        |      Score      |     Word     | Improved genetic algorithm-based word substitution        		                                     |
-| Input Reduction*                  |    Gradient     |     Word     | Reducing the input while maintaining the prediction through word importance ranking               |
-| Kuleshov							            |      Score      |     Word     | Greedy search and counterfitted embedding swap     		                                           |
-| Particle Swarm Optimization       |      Score      |     Word     | Particle Swarm Optimization-based word substitution                                               |
-| TextFooler                        |      Score      |     Word     | Greedy attack with word importance ranking and counter-fitted embedding swap                      |
-| PWWS                              |      Score      |     Word     | Greedy attack with word importance ranking based on word saliency and synonym swap scores         |
-| TextBugger                        | Gradient, Score |  Word+Char   | Greedy attack with word importance ranking and a combination of synonym and character-based swaps |
-| HotFlip                           |    Gradient     |  Word, Char  | Beam search and gradient-based word swap				                                                   |
-| BERT-Attack*                      |      Score      |  Word, Char  | BERT masked language model transformation attack with subword replacements                        |
-| CheckList*                        |      Score      |  Word, Char  | Invariance testing that contract, extend, and substitutes name entities.                          |
-| DeepWordBug                       |      Score      |     Char     | Greedy replace-1 scoring and multi-transformation character-swap attack                           |
-| pruthi                            |      Score      |     Char     | Character-based attack that simulates common typos		                                             |
+<tr><td style="text-align: center;" colspan="6"><strong><br>Attacks on sequence-to-sequence models: <br></strong></td></tr>
+
+<tr>
+<td><code>morpheus</code> <span class="citation" data-cites="morpheus-tan-etal-2020-morphin"></span></td>
+<td><sub>Minimum BLEU Score</sub> </td>
+<td></td>
+<td><sub>Inflection Word Swap</sub> </td>
+<td><sub>Greedy search</sub> </td>
+<td ><sub>Greedy to replace words with their inflections with the goal of minimizing BLEU score (["It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations"](https://www.aclweb.org/anthology/2020.acl-main.263.pdf)</sub> </td>
+</tr>
+
+</tr>
+<tr>
+<td><code>seq2sick</code> :(black-box) <span class="citation" data-cites="cheng2018seq2sick"></span></td>
+<td><sub>Non-overlapping output</sub> </td>
+<td></td>
+<td><sub>Counter-fitted word embedding swap</sub> </td>
+<td><sub>Greedy-WIR</sub></td>
+<td ><sub>Greedy attack with goal of changing every word in the output translation. Currently implemented as black-box with plans to change to white-box as done in paper (["Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples" (Cheng et al., 2018)](https://arxiv.org/abs/1803.01128)) </sub>  </td>
+</tr>
+
+
+</tbody>
+</font>
+</table>
 
-Attacks on sequence-to-sequence models:
-| Attack Recipe(s)                  |  Accessibility  | Perturbation | Main Idea                                                                                         |
-| :-------------------------------: | :-------------: | :----------: | :-----------------------------------------------------------------------------------------------: |
-| Seq2Sick                          |      Score      |     Word     | Greedy attack with goal of changing every word in the output translation. 					     |
-| MORPHEUS                          |      Score      |     Word     | Greedy attack that replaces words with their inflections with the goal of minimizing BLEU score   |
 
 
 #### Recipe Usage Examples
@@ -170,16 +308,18 @@ textattack attack --model bert-base-uncased-sst2 --recipe textfooler --num-examp
  textattack attack --model t5-en-de --recipe seq2sick --num-examples 100
 ```
 
-### Augmenting Text
+### Augmenting Text: `textattack augment`
 
 Many of the components of TextAttack are useful for data augmentation. The `textattack.Augmenter` class
-uses a transformation and a list of constraints to augment data. We also offer five built-in recipes
+uses a transformation and a list of constraints to augment data. We also offer  built-in recipes
 for data augmentation:
-- `textattack.WordNetAugmenter` augments text by replacing words with WordNet synonyms
-- `textattack.EmbeddingAugmenter` augments text by replacing words with neighbors in the counter-fitted embedding space, with a constraint to ensure their cosine similarity is at least 0.8
-- `textattack.CharSwapAugmenter` augments text by substituting, deleting, inserting, and swapping adjacent characters
-- `textattack.EasyDataAugmenter` augments text with a combination of word insertions, substitutions and deletions.
-- `textattack.CheckListAugmenter` augments text by contraction/extension and by substituting names, locations, numbers.
+- `wordnet` augments text by replacing words with WordNet synonyms
+- `embedding` augments text by replacing words with neighbors in the counter-fitted embedding space, with a constraint to ensure their cosine similarity is at least 0.8
+- `charswap` augments text by substituting, deleting, inserting, and swapping adjacent characters
+- `eda` augments text with a combination of word insertions, substitutions and deletions.
+- `checklist` augments text by contraction/extension and by substituting names, locations, numbers.
+- `clare` augments text by replacing, inserting, and merging with a pre-trained masked language model.
+
 
 #### Augmentation Command-Line Interface
 The easiest way to use our data augmentation tools is with `textattack augment <args>`. `textattack augment`
@@ -198,10 +338,16 @@ For example, given the following as `examples.csv`:
 "it's a mystery how the movie could be released in this condition .", 0
 ```
 
-The command `textattack augment --csv examples.csv --input-column text --recipe embedding --pct-words-to-swap .1 --transformations-per-example 2 --exclude-original`
+The command 
+```bash
+textattack augment --input-csv examples.csv --output-csv output.csv  --input-column text --recipe embedding --pct-words-to-swap .1 --transformations-per-example 2 --exclude-original
+```
 will augment the `text` column by altering 10% of each example's words, generating twice as many augmentations as original inputs, and exclude the original inputs from the
 output CSV. (All of this will be saved to `augment.csv` by default.)
 
+> **Tip:** Just as running attacks interactively, you can also pass `--interactive` to augment samples inputted by the user to quickly try out different augmentation recipes!
+
+
 After augmentation, here are the contents of `augment.csv`:
 ```csv
 text,label
@@ -244,7 +390,7 @@ You can also create your own augmenter from scratch by importing transformations
 ['What I cannot creae, I do not understand.', 'What I cannot creat, I do not understand.', 'What I cannot create, I do not nderstand.', 'What I cannot create, I do nt understand.', 'Wht I cannot create, I do not understand.']
 ```
 
-### Training Models
+### Training Models:  `textattack train`
 
 Our model training code is available via `textattack train` to help you train LSTMs,
 CNNs, and `transformers` models using TextAttack out-of-the-box. Datasets are
@@ -253,44 +399,45 @@ automatically loaded using the `datasets` package.
 #### Training Examples
 *Train our default LSTM for 50 epochs on the Yelp Polarity dataset:*
 ```bash
-textattack train --model lstm --dataset yelp_polarity --batch-size 64 --epochs 50 --learning-rate 1e-5
+textattack train --model-name-or-path lstm --dataset yelp_polarity  --epochs 50 --learning-rate 1e-5
 ```
 
-The training process has data augmentation built-in:
-```bash
-textattack train --model lstm --dataset rotten_tomatoes --augment eda --pct-words-to-swap .1 --transformations-per-example 4
-```
-This uses the `EasyDataAugmenter` recipe to augment the `rotten_tomatoes` dataset before training.
 
 *Fine-Tune `bert-base` on the `CoLA` dataset for 5 epochs**:
 ```bash
-textattack train --model bert-base-uncased --dataset glue^cola --batch-size 32 --epochs 5
+textattack train --model-name-or-path bert-base-uncased --dataset glue^cola --per-device-train-batch-size 8 --epochs 5
 ```
 
-### `textattack peek-dataset`
 
-To take a closer look at a dataset, use `textattack peek-dataset`. TextAttack will print some cursory statistics about the inputs and outputs from the dataset. For example, `textattack peek-dataset --dataset-from-huggingface snli` will show information about the SNLI dataset from the NLP package.
+### To check datasets: `textattack peek-dataset`
+
+To take a closer look at a dataset, use `textattack peek-dataset`. TextAttack will print some cursory statistics about the inputs and outputs from the dataset. For example, 
+```bash
+textattack peek-dataset --dataset-from-huggingface snli
+```
+will show information about the SNLI dataset from the NLP package.
 
 
-### `textattack list`
+### To list functional components: `textattack list`
 
 There are lots of pieces in TextAttack, and it can be difficult to keep track of all of them. You can use `textattack list` to list components, for example, pretrained models (`textattack list models`) or available search methods (`textattack list search-methods`).
 
+
 ## Design
 
-### AttackedText
 
-To allow for word replacement after a sequence has been tokenized, we include an `AttackedText` object
-which maintains both a list of tokens and the original text, with punctuation. We use this object in favor of a list of words or just raw text.
+### Models 
 
-### Models and Datasets
+TextAttack is model-agnostic!  You can use `TextAttack` to analyze any model that outputs IDs, tensors, or strings. To help users, TextAttack includes pre-trained models for different common NLP tasks. This makes it easier for
+users to get started with TextAttack. It also enables a more fair comparison of attacks from
+the literature.
 
-TextAttack is model-agnostic! You can use `TextAttack` to analyze any model that outputs IDs, tensors, or strings.
 
-#### Built-in Models
+
+#### Built-in Models and Datasets
 
 TextAttack also comes built-in with models and datasets. Our command-line interface will automatically match the correct 
-dataset to the correct model. We include various pre-trained models for each of the nine [GLUE](https://gluebenchmark.com/) 
+dataset to the correct model. We include 82 different (Oct 2020) pre-trained models for each of the nine [GLUE](https://gluebenchmark.com/) 
 tasks, as well as some common datasets for classification, translation, and summarization. 
 
 A list of available pretrained models and their validation accuracies is available at
@@ -324,7 +471,7 @@ create a short file that loads them as variables `model` and `tokenizer`.  The `
 be able to transform string inputs to lists or tensors of IDs using a method called `encode()`. The
 model must take inputs via the `__call__` method.
 
-##### Model from a file
+##### Custom Model from a file
 To experiment with a model you've trained, you could create the following file
 and name it `my_model.py`:
 
@@ -335,6 +482,11 @@ tokenizer = load_your_tokenizer_with_custom_code() # replace this line with your
 
 Then, run an attack with the argument `--model-from-file my_model.py`. The model and tokenizer will be loaded automatically.
 
+
+
+### Custom Datasets
+
+
 #### Dataset from a file
 
 Loading a dataset from a file is very similar to loading a model from a file. A 'dataset' is any iterable of `(input, output)` pairs.
@@ -346,27 +498,86 @@ dataset = [('Today was....', 1), ('This movie is...', 0), ...]
 
 You can then run attacks on samples from this dataset by adding the argument `--dataset-from-file my_dataset.py`.
 
-### Attacks
 
-The `attack_one` method in an `Attack` takes as input an `AttackedText`, and outputs either a `SuccessfulAttackResult` if it succeeds or a `FailedAttackResult` if it fails. We formulate an attack as consisting of four components: a **goal function** which determines if the attack has succeeded, **constraints** defining which perturbations are valid, a **transformation** that generates potential modifications given an input, and a **search method** which traverses through the search space of possible perturbations. 
 
-### Goal Functions
+#### Dataset loading via other mechanism, see: [more details at here](https://textattack.readthedocs.io/en/latest/api/datasets.html)
+
+```python
+import textattack
+my_dataset = [("text",label),....]
+new_dataset = textattack.datasets.Dataset(my_dataset)
+```
+
+
+
+#### Dataset via AttackedText class
+
+To allow for word replacement after a sequence has been tokenized, we include an `AttackedText` object
+which maintains both a list of tokens and the original text, with punctuation. We use this object in favor of a list of words or just raw text.
+
+
+### Attacks and how to design a new attack 
+
+
+We formulate an attack as consisting of four components: a **goal function** which determines if the attack has succeeded, **constraints** defining which perturbations are valid, a **transformation** that generates potential modifications given an input, and a **search method** which traverses through the search space of possible perturbations. The attack attempts to perturb an input text such that the model output fulfills the goal function (i.e., indicating whether the attack is successful) and the perturbation adheres to the set of constraints (e.g., grammar constraint, semantic similarity constraint). A search method is used to find a sequence of transformations that produce a successful adversarial example.
+
+
+This modular design unifies adversarial attack methods into one system, enables us to easily assemble attacks from the literature while re-using components that are shared across attacks. We provides clean, readable implementations of 16 adversarial attack recipes from the literature (see above table). For the first time, these attacks can be benchmarked, compared, and analyzed in a standardized setting.
+
+
+TextAttack is model-agnostic - meaning it can run attacks on models implemented in any deep learning framework. Model objects must be able to take a string (or list of strings) and return an output that can be processed by the goal function. For example, machine translation models take a list of strings as input and produce a list of strings as output. Classification and entailment models return an array of scores. As long as the user's model meets this specification, the model is fit to use with TextAttack.
+
+
+
+#### Goal Functions
 
 A `GoalFunction` takes as input an `AttackedText` object, scores it, and determines whether the attack has succeeded, returning a `GoalFunctionResult`.
 
-### Constraints
+#### Constraints
 
 A `Constraint` takes as input a current `AttackedText`, and a list of transformed `AttackedText`s. For each transformed option, it returns a boolean representing whether the constraint is met.
 
-### Transformations
+#### Transformations
 
 A `Transformation` takes as input an `AttackedText` and returns a list of possible transformed `AttackedText`s. For example, a transformation might return all possible synonym replacements.
 
-### Search Methods
+#### Search Methods
 
 A `SearchMethod` takes as input an initial `GoalFunctionResult` and returns a final `GoalFunctionResult` The search is given access to the `get_transformations` function, which takes as input an `AttackedText` object and outputs a list of possible transformations filtered by meeting all of the attack’s constraints. A search consists of successive calls to `get_transformations` until the search succeeds (determined using `get_goal_results`) or is exhausted.
 
 
+## On Benchmarking Attacks
+
+- See our analysis paper: Searching for a Search Method: Benchmarking Search Algorithms for Generating NLP Adversarial Examples at [EMNLP BlackBoxNLP](https://arxiv.org/abs/2009.06368). 
+
+- As we emphasized in the above paper, we don't recommend to directly compare Attack Recipes out of the box. 
+
+- This comment is due to that attack recipes in the recent literature used different ways or thresholds in setting up their constraints. Without the constraint space held constant, an increase in attack success rate could come from an improved search or transformation method or a less restrictive search space. 
+
+- Our Github on benchmarking scripts and results:  [TextAttack-Search-Benchmark Github](https://github.com/QData/TextAttack-Search-Benchmark)
+
+
+## On Quality of Generated Adversarial Examples in Natural Language
+
+- Our analysis Paper in [EMNLP Findings](https://arxiv.org/abs/2004.14174)
+- We analyze the generated adversarial examples of two state-of-the-art synonym substitution attacks. We find that their perturbations often do not preserve semantics, and 38% introduce grammatical errors. Human surveys reveal that to successfully preserve semantics, we need to significantly increase the minimum cosine similarities between the embeddings of swapped words and between the sentence encodings of original and perturbed sentences.With constraints adjusted to better preserve semantics and grammaticality, the attack success rate drops by over 70 percentage points.
+- Our Github on Reevaluation results: [Reevaluating-NLP-Adversarial-Examples Github](https://github.com/QData/Reevaluating-NLP-Adversarial-Examples)
+- As we have emphasized in this analysis paper, we recommend researchers and users to be EXTREMELY mindful on the quality of generated adversarial examples in natural language 
+- We recommend the field to use human-evaluation derived thresholds for setting up constraints 
+
+
+
+## Multi-lingual Support
+
+
+- see example code: [https://github.com/QData/TextAttack/blob/master/examples/attack/attack_camembert.py](https://github.com/QData/TextAttack/blob/master/examples/attack/attack_camembert.py) for using our framework to attack French-BERT. 
+
+- see tutorial notebook: [https://textattack.readthedocs.io/en/latest/2notebook/Example_4_CamemBERT.html](https://textattack.readthedocs.io/en/latest/2notebook/Example_4_CamemBERT.html) for using our framework to attack French-BERT. 
+
+- See [README_ZH.md](https://github.com/QData/TextAttack/blob/master/README_ZH.md) for our README in Chinese 
+
+
+
 ## Contributing to TextAttack
 
 We welcome suggestions and contributions! Submit an issue or pull request and we will do our best to respond in a timely manner. TextAttack is currently in an "alpha" stage in which we are working to improve its capabilities and design.
@@ -378,13 +589,12 @@ See [CONTRIBUTING.md](https://github.com/QData/TextAttack/blob/master/CONTRIBUTI
 If you use TextAttack for your research, please cite [TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP](https://arxiv.org/abs/2005.05909).
 
 ```bibtex
-@misc{morris2020textattack,
-    title={TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP},
-    author={John X. Morris and Eli Lifland and Jin Yong Yoo and Jake Grigsby and Di Jin and Yanjun Qi},
-    year={2020},
-    eprint={2005.05909},
-    archivePrefix={arXiv},
-    primaryClass={cs.CL}
+@inproceedings{morris2020textattack,
+  title={TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP},
+  author={Morris, John and Lifland, Eli and Yoo, Jin Yong and Grigsby, Jake and Jin, Di and Qi, Yanjun},
+  booktitle={Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
+  pages={119--126},
+  year={2020}
 }
 ```
 

README_ZH.md

@@ -0,0 +1,515 @@
+<h1 align="center">TextAttack 🐙</h1>
+
+<p align="center">为 NLP 模型生成对抗样本</p>
+
+<p align="center">
+  <a href="https://textattack.readthedocs.io/">[TextAttack 的 ReadTheDocs 文档]</a> 
+  <br> <br>
+  <a href="#简介">简介</a> •
+  <a href="#环境配置">环境配置</a> •
+  <a href="#使用方法textattack---help">使用方法</a> •
+  <a href="#设计模式">设计模式</a> 
+  <br> <br>
+  <a target="_blank">
+    <img src="https://github.com/QData/TextAttack/workflows/Github%20PyTest/badge.svg" alt="Github Runner Covergae Status">
+  </a>
+  <a href="https://badge.fury.io/py/textattack">
+    <img src="https://badge.fury.io/py/textattack.svg" alt="PyPI version" height="18">
+  </a>
+</p>
+
+<img src="http://jackxmorris.com/files/textattack.gif" alt="TextAttack Demo GIF" style="display: block; margin: 0 auto;" />
+
+## 简介
+
+TextAttack 是一个可以实行自然语言处理的 Python 框架，用于方便快捷地进行对抗攻击，增强数据，以及训练模型。
+
+> 如果你在寻找 TextAttacks 支持的预训练模型，请访问 [TextAttack Model Zoo](https://textattack.readthedocs.io/en/latest/3recipes/models.html)。
+
+## Slack 频道
+
+加入[TextAttack Slack](https://join.slack.com/t/textattack/shared_invite/zt-huomtd9z-KqdHBPPu2rOP~Z8q3~urgg) 频道，获取在线帮助与更新提示！
+
+### *选择 TextAttack 的原因*
+
+1. **深入理解 NLP 模型**： 通过使用各种对抗攻击，观察模型的表现
+2. **研究与开发 NLP 对抗攻击**： 在你的项目中使用 TextAttack 的框架与组件库
+3. **进行数据增强**： 提升模型的泛化性与鲁棒性
+3. **训练 NLP 模型**： 只需一行命令，轻松训练模型 (包括下载所有的依赖资源！)
+
+## 环境配置
+
+### 安装
+
+支持 Python 3.6 及以上。支持 CPU ，使用兼容 CUDA 的 GPU ，还可以大幅度提高代码运行速度。使用 pip 轻松安装 TextAttack:
+
+```bash
+pip install textattack
+```
+
+当 TextAttack 安装完成，可以通过命令行 (`textattack ...`)
+或者通过 python 模块 (`python -m textattack ...`) 运行 TextAttack。
+
+> **小提醒**：TextAttack 默认将文件下载保存在 `~/.cache/textattack/` 路径。这些文件包括预训练模型，数据集，以及配置文件 `config.yaml`。若需更改缓存路径，可以通过设置环境变量 `TA_CACHE_DIR`。(例如: `TA_CACHE_DIR=/tmp/ textattack attack ...`).
+
+## 使用方法：`textattack --help`
+
+TextAttack 的主要功能均可通过 `textattack` 命令运行。常用的两个命令为 `textattack attack <args>` 和 `textattack augment <args>`。你可以通过如下命令获取关于所有命令的介绍：
+```bash
+textattack --help 
+```
+或者获取具体命令的用法，例如：
+```bash
+textattack attack --help
+```
+
+文件夹 [`examples/`](examples/) 里是一些示例脚本，展示了 TextAttack 的常用方法，包括训练模型，对抗攻击，以及数据增强。[文档网站](https://textattack.readthedocs.io/en/latest) 中有 TextAttack 基本用法的详尽说明与示例，包括自定义攻击的变换与约束。
+
+### 运行对抗攻击：`textattack attack --help`
+
+尝试运行对抗攻击，最快捷的方法是通过命令行接口：`textattack attack` 
+
+> **小提醒**：如果你的机器有多个 GPU，可以通过 `--parallel` 参数将对抗攻击分布在多个 GPU 上。这对一些攻击策略的性能提升巨大。
+
+下面是几个具体的例子：
+
+*对 MR 情感分类数据集上训练的 BERT 模型进行 TextFooler 攻击*: 
+
+```bash
+textattack attack --recipe textfooler --model bert-base-uncased-mr --num-examples 100
+```
+
+*对 Quora 问句对数据集上训练的 DistilBERT 模型进行 DeepWordBug 攻击*: 
+
+```bash
+textattack attack --model distilbert-base-uncased-cola --recipe deepwordbug --num-examples 100
+```
+
+*对 MR 数据集上训练的 LSTM 模型：设置束搜索宽度为 4，使用词嵌入转换进行无目标攻击*:
+
+```bash
+textattack attack --model lstm-mr --num-examples 20 \
+ --search-method beam-search^beam_width=4 --transformation word-swap-embedding \
+ --constraints repeat stopword max-words-perturbed^max_num_words=2 embedding^min_cos_sim=0.8 part-of-speech \
+ --goal-function untargeted-classification
+```
+
+> **小提醒**：除了设置具体的数据集与样本数量，你还可以通过传入 `--interactive` 参数，对用户输入的文本进行攻击。
+
+### 攻击策略：`textattack attack --recipe [recipe_name]`
+
+我们实现了一些文献中的攻击策略（Attack recipe）。使用 `textattack list attack-recipes` 命令可以列出所有内置的攻击策略。
+
+运行攻击策略：`textattack attack --recipe [recipe_name]`
+
+
+<img src="docs/_static/imgs/overview.png" alt="TextAttack Overview" style="display: block; margin: 0 auto;" />
+
+<table>
+<thead>
+<tr class="header">
+<th><strong>—————— 攻击策略 ——————</strong></th>
+<th><strong>—————— 目标函数 ——————</strong></th>
+<th><strong>—————— 约束条件 ——————</strong></th>
+<th><strong>—————— 变换方式 ——————</strong></th>
+<th><strong>——————— 搜索方法 ———————</strong></th>
+<th><strong>主要思想</strong></th>
+</tr>
+</thead>
+<tbody>
+  <tr><td colspan="6"><strong><br>对于分类任务的攻击策略，例如情感分类和文本蕴含任务：<br></strong></td></tr>
+
+<tr>
+<td><code>alzantot</code>  <span class="citation" data-cites="Alzantot2018GeneratingNL Jia2019CertifiedRT"></span></td>
+<td><sub>无目标<br/>{分类，蕴含}</sub></td>
+<td><sub>被扰动词的比例，语言模型的困惑度，词嵌入的距离</sub></td>
+<td><sub>Counter-fitted 词嵌入替换</sub></td>
+<td><sub>遗传算法</sub></td>
+<td ><sub>来自 (["Generating Natural Language Adversarial Examples" (Alzantot et al., 2018)](https://arxiv.org/abs/1804.07998))</sub></td>
+</tr>
+<tr>
+<td><code>bae</code> <span class="citation" data-cites="garg2020bae"></span></td>
+<td><sub>无目标<br/>分类</sub></td>
+<td><sub>USE 通用句子编码向量的 cosine 相似度</sub></td>
+<td><sub>BERT 遮罩词预测</sub></td>
+<td><sub>对 WIR 的贪心搜索</sub></td>
+<td><sub>使用 BERT 语言模型作为变换的攻击方法，来自 (["BAE: BERT-based Adversarial Examples for Text Classification" (Garg & Ramakrishnan, 2019)](https://arxiv.org/abs/2004.01970)). </sub></td>
+</tr>
+<tr>
+<td><code>bert-attack</code> <span class="citation" data-cites="li2020bertattack"></span></td>
+<td><sub>无目标<br/>分类</sub></td>
+<td><sub>USE 通用句子编码向量的 cosine 相似度, 被扰动词的最大数量</sub></td>
+<td><sub>BERT 遮罩词预测 (包括对 subword 的扩充)</sub></td>
+<td><sub>对 WIR 的贪心搜索</sub></td>
+<td ><sub> (["BERT-ATTACK: Adversarial Attack Against BERT Using BERT" (Li et al., 2020)](https://arxiv.org/abs/2004.09984))</sub></td>
+</tr>
+<tr>
+<td><code>checklist</code> <span class="citation" data-cites="Gao2018BlackBoxGO"></span></td>
+<td><sub>{无目标，有目标}<br/>分类</sub></td>
+<td><sub>checklist 距离</sub></td>
+<td><sub>简写，扩写，以及命名实体替换</sub></td>
+<td><sub>对 WIR 的贪心搜索</sub></td>
+<td ><sub>CheckList 中实现的不变性检验(["Beyond Accuracy: Behavioral Testing of NLP models with CheckList" (Ribeiro et al., 2020)](https://arxiv.org/abs/2005.04118))</sub></td>
+</tr>
+<tr>
+<td> <code>clare (*coming soon*)</code> <span class="citation" data-cites="Alzantot2018GeneratingNL Jia2019CertifiedRT"></span></td>
+<td><sub>无目标<br/>{分类，蕴含}</sub></td>
+<td><sub>RoBERTa 掩码语言模型</sub></td>
+<td><sub>词的替换，插入，合并</sub></td>
+<td><sub>贪心搜索</sub></td>
+<td ><sub>["Contextualized Perturbation for Textual Adversarial Attack" (Li et al., 2020)](https://arxiv.org/abs/2009.07502))</sub></td>
+</tr>
+<tr>
+<td><code>deepwordbug</code> <span class="citation" data-cites="Gao2018BlackBoxGO"></span></td>
+<td><sub>{无目标，有目标}<br/>分类</sub></td>
+<td><sub>Levenshtein 编辑距离</sub></td>
+<td><sub>{字符的插入，删除，替换，以及临近字符交换}</sub></td>
+<td><sub>对 WIR 的贪心搜索</sub></td>
+<td ><sub>贪心搜索 replace-1 分数，多种变换的字符交换式的攻击 (["Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers" (Gao et al., 2018)](https://arxiv.org/abs/1801.04354)</sub></td>
+</tr>
+<tr>
+<td> <code>fast-alzantot</code> <span class="citation" data-cites="Alzantot2018GeneratingNL Jia2019CertifiedRT"></span></td>
+<td><sub>无目标<br/>{分类，蕴含}</sub></td>
+<td><sub>被扰动词的比例，语言模型的困惑度，词嵌入的距离</sub></td>
+<td><sub>Counter-fitted 词嵌入替换</sub></td>
+<td><sub>遗传算法</sub></td>
+<td ><sub>改进过的更快的 Alzantot et al. 遗传算法, 来自 (["Certified Robustness to Adversarial Word Substitutions" (Jia et al., 2019)](https://arxiv.org/abs/1909.00986))</sub></td>
+</tr>
+<tr>
+<td><code>hotflip</code> (word swap) <span class="citation" data-cites="Ebrahimi2017HotFlipWA"></span></td>
+<td><sub>无目标<br/>分类</sub></td>
+<td><sub>词嵌入的 cosine 相似度，词性的匹配，被扰动词的数量</sub></td>
+<td><sub>基于梯度的词的交换</sub></td>
+<td><sub>束搜索</sub></td>
+<td ><sub> (["HotFlip: White-Box Adversarial Examples for Text Classification" (Ebrahimi et al., 2017)](https://arxiv.org/abs/1712.06751))</sub></td>
+</tr>
+<tr>
+<td><code>iga</code> <span class="citation" data-cites="iga-wang2019natural"></span></td>
+<td><sub>无目标<br/>{分类，蕴含}</sub></td>
+<td><sub>被扰动词的比例，词嵌入的距离</sub></td>
+<td><sub>Counter-fitted 词嵌入替换</sub></td>
+<td><sub>遗传算法</sub></td>
+<td ><sub>改进的基于遗传算法的词替换，来自 (["Natural Language Adversarial Attacks and Defenses in Word Level (Wang et al., 2019)"](https://arxiv.org/abs/1909.06723)</sub></td>
+</tr>
+<tr>
+<td><code>input-reduction</code> <span class="citation" data-cites="feng2018pathologies"></span></td>
+<td><sub>输入归约</sub></td>
+<td></td>
+<td><sub>词的删除</sub></td>
+<td><sub>对 WIR 的贪心搜索</sub></td>
+<td ><sub>基于词重要性排序的贪心攻击方法，在缩减输入词的同时保持预测结果不变 (["Pathologies of Neural Models Make Interpretation Difficult" (Feng et al., 2018)](https://arxiv.org/pdf/1804.07781.pdf))</sub></td>
+</tr>
+<tr>
+<td><code>kuleshov</code> <span class="citation" data-cites="Kuleshov2018AdversarialEF"></span></td>
+<td><sub>无目标<br/>分类</sub></td>
+<td><sub>Thought vector 编码的 cosine 相似度, 语言模型给出的相似度概率</sub></td>
+<td><sub>Counter-fitted 词嵌入替换</sub></td>
+<td><sub>贪心的词的替换</sub></td>
+<td ><sub>(["Adversarial Examples for Natural Language Classification Problems" (Kuleshov et al., 2018)](https://openreview.net/pdf?id=r1QZ3zbAZ)) </sub></td>
+</tr>
+<tr>
+<td><code>pruthi</code> <span class="citation" data-cites="pruthi2019combating"></span></td>
+<td><sub>无目标<br/>分类</sub></td>
+<td><sub>词的最短长度，被扰动词的最大数量</sub></td>
+<td><sub>{临近字符替换，字符的插入与删除，基于键盘字符位置的字符替换}</sub></td>
+<td><sub>贪心搜索</sub></td>
+<td ><sub>模拟常见的打字错误 (["Combating Adversarial Misspellings with Robust Word Recognition" (Pruthi et al., 2019)](https://arxiv.org/abs/1905.11268) </sub></td>
+</tr>
+<tr>
+<td><code>pso</code> <span class="citation" data-cites="pso-zang-etal-2020-word"></span></td>
+<td><sub>无目标<br/>分类</sub></td>
+<td></td>
+<td><sub>基于 HowNet 的词替换</sub></td>
+<td><sub>粒子群优化算法</sub></td>
+<td ><sub>(["Word-level Textual Adversarial Attacking as Combinatorial Optimization" (Zang et al., 2020)](https://www.aclweb.org/anthology/2020.acl-main.540/)) </sub></td>
+</tr>
+<tr>
+<td><code>pwws</code> <span class="citation" data-cites="pwws-ren-etal-2019-generating"></span></td>
+<td><sub>无目标<br/>分类</sub></td>
+<td></td>
+<td><sub>基于 WordNet 的同义词替换</sub></td>
+<td><sub>对 WIR 的贪心搜索</sub></td>
+<td ><sub>贪心的攻击方法，基于词重要性排序，词的显著性，以及同义词替换分数(["Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency" (Ren et al., 2019)](https://www.aclweb.org/anthology/P19-1103/))</sub> </td>
+</tr>
+<tr>
+<td><code>textbugger</code> : (black-box) <span class="citation" data-cites="Li2019TextBuggerGA"></span></td>
+<td><sub>无目标<br/>分类</sub></td>
+<td><sub>USE 通用句子编码向量的 cosine 相似度</sub></td>
+<td><sub>{字符的插入、删除、替换，以及临近字符交换}</sub></td>
+<td><sub>对 WIR 的贪心搜索</sub></td>
+<td ><sub>([(["TextBugger: Generating Adversarial Text Against Real-world Applications" (Li et al., 2018)](https://arxiv.org/abs/1812.05271)).</sub></td>
+</tr>
+<tr>
+<td><code>textfooler</code> <span class="citation" data-cites="Jin2019TextFooler"></span></td>
+<td><sub>无目标<br/>{分类，蕴含}</sub></td>
+<td><sub>词嵌入的距离，词性的匹配，USE 通用句子编码向量的 cosine 相似度</sub></td>
+<td><sub>Counter-fitted 词嵌入替换</sub></td>
+<td><sub>对 WIR 的贪心搜索</sub></td>
+<td ><sub>对词重要性排序的贪心攻击方法(["Is Bert Really Robust?" (Jin et al., 2019)](https://arxiv.org/abs/1907.11932))</sub> </td>
+</tr>
+
+<tr><td colspan="6"><strong><br>对 seq2seq 模型的攻击策略：<br></strong></td></tr>
+
+<tr>
+<td><code>morpheus</code> <span class="citation" data-cites="morpheus-tan-etal-2020-morphin"></span></td>
+<td><sub>最小 BLEU 分数</sub> </td>
+<td></td>
+<td><sub>词的屈折变化</sub> </td>
+<td><sub>贪心搜索</sub> </td>
+<td ><sub>贪心的用词的屈折变化进行替换，来最小化 BLEU 分数(["It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations"](https://www.aclweb.org/anthology/2020.acl-main.263.pdf)</sub> </td>
+</tr>
+
+</tr>
+<tr>
+<td><code>seq2sick</code> :(black-box) <span class="citation" data-cites="cheng2018seq2sick"></span></td>
+<td><sub>翻译结果无重叠</sub> </td>
+<td></td>
+<td><sub>Counter-fitted 词嵌入替换</sub> </td>
+<td><sub>对 WIR 的贪心搜索</sub></td>
+<td ><sub>贪心攻击方法，以改变全部的翻译结果为目标。目前实现的是黑盒攻击，计划改为与论文中一样的白盒攻击(["Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples" (Cheng et al., 2018)](https://arxiv.org/abs/1803.01128)) </sub>  </td>
+</tr>
+
+</tbody>
+</font>
+</table>
+
+> WIR 为 word word importance ranking 的缩写，即词重要性排序。
+
+
+
+#### 运行攻击的例子
+
+下面是几个样例，在命令行中验证上述实现的攻击方法:
+
+*对在 SST-2 上精调的 BERT 模型进行 TextFooler 攻击：*
+```bash
+textattack attack --model bert-base-uncased-sst2 --recipe textfooler --num-examples 10
+```
+
+*对用于英语-德语翻译的 T2 模型进行 seq2sick (黑盒) 攻击：*
+```bash
+ textattack attack --model t5-en-de --recipe seq2sick --num-examples 100
+```
+
+### 增强文本数据：`textattack augment`
+
+TextAttack 的组件中，有很多易用的数据增强工具。`textattack.Augmenter` 类使用 *变换* 与一系列的 *约束* 进行数据增强。我们提供了 5 中内置的数据增强策略：
+- `wordnet` 通过基于 WordNet 同义词替换的方式增强文本
+- `embedding` 通过邻近词替换的方式增强文本，使用 counter-fitted 词嵌入空间中的邻近词进行替换，约束二者的 cosine 相似度不低于 0.8
+- `charswap` 通过字符的增删改，以及临近字符交换的方式增强文本
+- `eda` 通过对词的增删改来增强文本
+- `checklist` 通过简写，扩写以及对实体、地点、数字的替换来增强文本
+- `clare` 使用 pre-trained masked language model, 通过对词的增删改来增强文本
+
+
+
+#### 数据增强的命令行接口
+使用 textattack 来进行数据增强，最快捷的方法是通过 `textattack augment <args>` 命令行接口。 `textattack augment` 使用 CSV 文件作为输入，在参数中设置需要增强的文本列，每个样本允许改变的比例，以及对于每个输入样本生成多少个增强样本。输出的结果保存为与输入文件格式一致的 CSV 文件，结果文件中为对指定的文本列生成的增强样本。
+
+比如，对于下面这个 `examples.csv` 文件:
+
+```csv
+"text",label
+"the rock is destined to be the 21st century's new conan and that he's going to make a splash even greater than arnold schwarzenegger , jean- claud van damme or steven segal.", 1
+"the gorgeously elaborate continuation of 'the lord of the rings' trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .", 1
+"take care of my cat offers a refreshingly different slice of asian cinema .", 1
+"a technically well-made suspenser . . . but its abrupt drop in iq points as it races to the finish line proves simply too discouraging to let slide .", 0
+"it's a mystery how the movie could be released in this condition .", 0
+```
+
+使用命令 `textattack augment --input-csv examples.csv --output-csv output.csv  --input-column text --recipe embedding --pct-words-to-swap .1 --transformations-per-example 2 --exclude-original`
+会增强 `text` 列，约束对样本中 10% 的词进行修改，生成输入数据两倍的样本，同时结果文件中不保存 csv 文件的原始输入。(默认所有结果将会保存在 `augment.csv` 文件中)
+
+数据增强后，下面是 `augment.csv` 文件的内容:
+```csv
+text,label
+"the rock is destined to be the 21st century's newest conan and that he's gonna to make a splashing even stronger than arnold schwarzenegger , jean- claud van damme or steven segal.",1
+"the rock is destined to be the 21tk century's novel conan and that he's going to make a splat even greater than arnold schwarzenegger , jean- claud van damme or stevens segal.",1
+the gorgeously elaborate continuation of 'the lord of the rings' trilogy is so huge that a column of expression significant adequately describe co-writer/director pedro jackson's expanded vision of j . rs . r . tolkien's middle-earth .,1
+the gorgeously elaborate continuation of 'the lordy of the piercings' trilogy is so huge that a column of mots cannot adequately describe co-novelist/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .,1
+take care of my cat offerings a pleasantly several slice of asia cinema .,1
+taking care of my cat offers a pleasantly different slice of asiatic kino .,1
+a technically good-made suspenser . . . but its abrupt drop in iq points as it races to the finish bloodline proves straightforward too disheartening to let slide .,0
+a technically well-made suspenser . . . but its abrupt drop in iq dot as it races to the finish line demonstrates simply too disheartening to leave slide .,0
+it's a enigma how the film wo be releases in this condition .,0
+it's a enigma how the filmmaking wo be publicized in this condition .,0
+```
+
+在 'embedding' 增强策略中，使用 counterfitted 词嵌入空间的最近邻来增强数据。
+
+#### 数据增强的 Python 接口
+除了使用命令行接口，你还可以在自己的代码中导入 `Augmenter` 来进行动态的数据增强。所有的 `Augmenter` 对象都实现了 `augment` 和 `augment_many` 方法，用于对单个 string 和一个 list 的 string 进行数据增强。下面是在 python 脚本中使用 `EmbeddingAugmenter` 的例子：
+
+```python
+>>> from textattack.augmentation import EmbeddingAugmenter
+>>> augmenter = EmbeddingAugmenter()
+>>> s = 'What I cannot create, I do not understand.'
+>>> augmenter.augment(s)
+['What I notable create, I do not understand.', 'What I significant create, I do not understand.', 'What I cannot engender, I do not understand.', 'What I cannot creating, I do not understand.', 'What I cannot creations, I do not understand.', 'What I cannot create, I do not comprehend.', 'What I cannot create, I do not fathom.', 'What I cannot create, I do not understanding.', 'What I cannot create, I do not understands.', 'What I cannot create, I do not understood.', 'What I cannot create, I do not realise.']
+```
+你还可以通过从 `textattack.transformations` 和 `textattack.constraints` 导入 *变换* 与 *约束* 来从头创建自己的数据增强方法。下面是一个使用  `WordSwapRandomCharacterDeletion` *变换* 进行数据增强的例子：
+
+```python
+>>> from textattack.transformations import WordSwapRandomCharacterDeletion
+>>> from textattack.transformations import CompositeTransformation
+>>> from textattack.augmentation import Augmenter
+>>> transformation = CompositeTransformation([WordSwapRandomCharacterDeletion()])
+>>> augmenter = Augmenter(transformation=transformation, transformations_per_example=5)
+>>> s = 'What I cannot create, I do not understand.'
+>>> augmenter.augment(s)
+['What I cannot creae, I do not understand.', 'What I cannot creat, I do not understand.', 'What I cannot create, I do not nderstand.', 'What I cannot create, I do nt understand.', 'Wht I cannot create, I do not understand.']
+```
+
+### 训练模型：`textattack train`
+
+通过 `textattack train` 可以便捷地使用 TextAttack 框架来训练 LSTM，CNN，以及 `transofrmers` 模型。数据集会通过 `datasets` 包自动加载。
+
+#### 运行训练的例子
+*在 Yelp 分类数据集上对 TextAttack 中默认的 LSTM 模型训练 50 个 epoch：*
+```bash
+textattack train --model-name-or-path lstm --dataset yelp_polarity  --epochs 50 --learning-rate 1e-5
+```
+
+
+*在 `CoLA` 数据集上对 `bert-base` 模型精调 5 个 epoch：*
+```bash
+textattack train --model-name-or-path bert-base-uncased --dataset glue^cola --per-device-train-batch-size 8 --epochs 5
+```
+
+
+### 检查数据集：`textattack peek-dataset`
+
+使用 `textattack peek-dataset` 可以进一步的观察数据。TextAttack 会打印出数据集粗略的统计信息，包括数据样例，输入文本的统计信息以及标签分布。比如，运行 `textattack peek-dataset --dataset-from-huggingface snli` 命令，会打印指定 NLP 包中 SNLI 数据集的统计信息。
+
+
+### 列出功能组件：`textattack list`
+
+TextAttack 中有很多组件，有时很难跟进所有组件的情况。你可以使用 `textattack list` 列出所有的组件。比如，列出预训练模型  (`textattack list models`)，或是列出可用的搜索方法 (`textattack list search-methods`)。
+
+
+## 设计模式
+
+
+### 模型
+
+TextAttack 不依赖具体模型！你可以使用 TextAttack 来分析任何模型，只要模型的输出是 ID，张量，或者字符串。为了方便使用，TextAttack 内置了常见 NLP 任务的各种预训练模型。你可以轻松愉悦地上手 TextAttack。同时还可以更公平的比较不同文献的 attack 策略。
+
+
+
+#### 内置的模型
+
+TextAttack 提供了各种内置模型和数据集。使用 TextAttack 命令行接口，可以自动匹配模型和数据集。
+我们为 [GLUE](https://gluebenchmark.com/) 中的九个任务内置了多种预训练模型，并且还内置了很多常见的分类任务、翻译任务和摘要任务的数据集。
+
+[textattack/models/README.md](textattack/models/README.md) 这个列表包含可用的预训练模型以及这些模型的准确率。你还可以通过 `textattack attack --help` 查看完整列表，包括所有的内置模型与数据集。
+
+下面是一个使用内置模型的例子（SST-2 数据集会自动的加载）：
+```bash
+textattack attack --model roberta-base-sst2 --recipe textfooler --num-examples 10
+```
+
+#### HuggingFace 支持 ：`transformers` 模型和 `datasets` 数据集
+
+TextAttack 兼容 [`transformers` 预训练模型](https://huggingface.co/models) 
+和 [`datasets` 数据集](https://github.com/huggingface/datasets)! 下面是一个例子，加载数据集并攻击相应预训练模型：
+
+```bash
+textattack attack --model-from-huggingface distilbert-base-uncased-finetuned-sst-2-english --dataset-from-huggingface glue^sst2 --recipe deepwordbug --num-examples 10
+```
+
+你还可以通过 `--model-from-huggingface` 参数探索更多支持的预训练模型，或是通过 
+`--dataset-from-huggingface` 参数指定其他数据集。
+
+
+#### 加载本地文件中的模型与数据集
+
+你可以快捷地对本地模型或数据样本进行攻击：创建一个简单的文件就可以加载预训练模型，然后在文件中可以通过对象 `model` 与 `tokenizer` 对象加载模型。`tokenizer` 对象必须实现 `encode()` 方法，该方法将输入字符串转为一个列表或一个 ID 张量。`model` 对象必须通过实现 `__call__` 方法来加载模型。
+
+##### 使用本地模型
+对于你已经训练完成的模型，可以通过创建下面这样的文件，将其命名为 `my_model.py`:
+
+```python
+model = load_your_model_with_custom_code() # replace this line with your model loading code
+tokenizer = load_your_tokenizer_with_custom_code() # replace this line with your tokenizer loading code
+```
+
+然后，在运行攻击时指定参数 `--model-from-file my_model.py`，就可以自动载入你的模型与分词器。
+
+### 数据集
+
+#### 使用本地数据集
+
+加载本地数据集与加载本地预训练模型的方法相似。`dataset` 对象可以是任意可迭代的`(input, output)` 对。下面这个例子演示了如何在 `my_dataset.py` 脚本中加载一个情感分类数据集：
+
+```python
+dataset = [('Today was....', 1), ('This movie is...', 0), ...]
+```
+
+然后，在运行攻击时指定参数 `--dataset-from-file my_dataset.py`，就可以对这个本地数据集进行攻击。
+
+#### 通过 AttackedText 类调用数据集
+
+为了对分词后的句子运行攻击方法，我们设计了 `AttackedText` 对象。它同时维护 token 列表与含有标点符号的原始文本。我们使用这个对象来处理原始的与分词后的文本。
+
+#### 通过 Data Frames 调用数据集(*即將上線*)
+
+
+### 何为攻击 & 如何设计新的攻击 
+
+
+我们将攻击划分并定义为四个组成部分：**目标函数** 定义怎样的攻击是一次成功的攻击，**约束条件** 定义怎样的扰动是可行的，**变换规则** 对输入文本生成一系列可行的扰动结果，**搜索方法** 在搜索空间中遍历所有可行的扰动结果。每一次攻击都尝试对输入的文本添加扰动，使其通过目标函数（即判断攻击是否成功），并且扰动要符合约束（如语法约束，语义相似性约束）。最后用搜索方法在所有可行的变换结果中，挑选出优质的对抗样本。
+
+
+这种模块化的设计可以将各种对抗攻击策略整合在一个系统里。这使得我们可以方便地将文献中的方法集成在一起，同时复用攻击策略之间相同的部分。我们已经实现了 16 种简明易读的攻击策略（见上表）。史上首次！各种攻击方法终于可以在标准的设置下作为基准方法，进行比较与分析。
+
+
+TextAttack 是不依赖具体模型的，这意味着可以对任何深度学习框架训练的模型进行攻击。只要被攻击的模型可以读取字符串（或一组字符串），并根据目标函数返回一个结果。比如说，机器翻译模型读取一句话，返回一句对应的翻译结果。分类或蕴含任务的模型输入字符串，返回一组分数。只要你的模型满足这两点，就可以使用 TextAttack 进行攻击。
+
+
+
+### 目标函数
+
+目标函数 `GoalFunction` 以 `AttackedText` 对象作为输入，为输入对象打分，并且判别这次攻击是否满足目标函数定义的成功条件，返回一个 `GoalFunctionResult` 对象。
+
+### 约束条件
+
+约束条件 `Constraint` 以 `AttackedText` 对象作为输入，返回一个变换后的 `AttackedText` 列表。对于每条变换，返回一个布尔值表示这条变换是否满足约束条件。
+
+### 变换规则
+
+变换规则 `Transformation` 以 `AttackedText` 对象作为输入，返回对于 `AttackedText` 所有可行变换的列表。例如，一个变换规则可以是返回所有可能的同义词替换结果。
+
+### 搜索方法
+
+搜索方法 `SearchMethod` 以初始的 `GoalFunctionResult` 作为输入，返回最终的 `GoalFunctionResult`。`get_transformations` 方法，以一个 `AttackedText` 对象作为输入，返还所有符合约束条件的变换结果。搜索方法不断地调用 `get_transformations` 函数，直到攻击成功 (由 `get_goal_results` 决定) 或搜索结束。
+
+### 公平比较攻击策略（Benchmarking Attacks）
+
+- 详细情况参见我们的分析文章：Searching for a Search Method: Benchmarking Search Algorithms for Generating NLP Adversarial Examples at [EMNLP BlackBoxNLP](https://arxiv.org/abs/2009.06368). 
+
+- 正如我们在上面的文章中所强调的，我们不推荐在对攻击策略没有约束的情况下直接进行比较。
+
+- 对这点进行强调，是由于最近的文献中在设置约束时使用了不同的方法或者阈值。在不固定约束空间时，攻击成功率的增加可能是源于改进的搜索方法或变换方式，又或是降低了对搜索空间的约束。
+
+## 帮助改进 TextAttack
+
+我们欢迎任何建议与改进！请提交 Issues（议题）和 Pull requests（拉取请求），我们会竭尽所能的做出即时反馈。TextAttack 当前处于 "alpha" 版本，我们仍在完善它的设计与功能。
+
+关于提交建议与改进的详细指引，查看 [CONTRIBUTING.md](https://github.com/QData/TextAttack/blob/master/CONTRIBUTING.md) 。
+
+## 引用 TextAttack
+
+如果 TextAttack 对你的研究工作有所帮助，欢迎在论文中引用 [TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP](https://arxiv.org/abs/2005.05909)。
+
+```bibtex
+@misc{morris2020textattack,
+    title={TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP},
+    author={John X. Morris and Eli Lifland and Jin Yong Yoo and Jake Grigsby and Di Jin and Yanjun Qi},
+    year={2020},
+    eprint={2005.05909},
+    archivePrefix={arXiv},
+    primaryClass={cs.CL}
+}
+```
+
+

docs/0_get_started/basic-Intro.rst

@@ -17,10 +17,6 @@ Where should I start?
 This is a great question, and one we get a lot. First of all, almost everything in TextAttack can be done in two ways: via the command-line or via the Python API. If you're looking to integrate TextAttack into an existing project, the Python API is likely for you. If you'd prefer to use built-in functionality end-to-end (training a model, running an adversarial attack, augmenting a CSV) then you can just use the command-line API.
 
 
-
-
-For future developers, visit the :ref:`Installation <installation>` page for more details about installing TextAttack onto your own computer. To start making contributions, read the detailed instructions `here <https://github.com/QData/TextAttack/blob/master/CONTRIBUTING.md>`__.
-
 TextAttack does three things very well:
 
 1. Adversarial attacks (Python: ``textattack.shared.Attack``, Bash: ``textattack attack``)
@@ -61,3 +57,4 @@ TextAttack has some other features that make it a pleasure to use:
 - :ref:`Pre-trained Models <models>` for testing attacks and evaluating constraints
 - :ref:`Visualization options   <loggers>` like Weights & Biases and Visdom
 - :ref:`AttackedText   <attacked_text>`, a utility class for strings that includes tools for tokenizing and editing text
+

docs/0_get_started/command_line_usage.md

@@ -9,6 +9,13 @@ just about anything TextAttack offers in a single bash command.
 > can access all the same functionality by prepending `python -m` to the command
 > (`python -m textattack ...`).
 
+
+> The [`examples/`](https://github.com/QData/TextAttack/tree/master/examples) folder includes scripts showing common TextAttack usage for training models, running attacks, and augmenting a CSV file. 
+
+
+> The [documentation website](https://textattack.readthedocs.io/en/latest) contains walkthroughs explaining basic usage of TextAttack, including building a custom transformation and a custom constraint..
+
+
 To see all available commands, type `textattack --help`. This page explains
 some of the most important functionalities of textattack: NLP data augmentation,
 adversarial attacks, and training and evaluating models.
@@ -33,7 +40,7 @@ For example, given the following as `examples.csv`:
 
 The command: 
 ```
-textattack augment --csv examples.csv --input-column text --recipe eda --pct-words-to-swap .1 \
+textattack augment --input-csv examples.csv --output-csv output.csv  --input-column text --recipe eda --pct-words-to-swap .1 \
 --transformations-per-example 2 --exclude-original
 ``` 
 will augment the `text` column with 10% of words edited per augmentation, twice as many augmentations as original inputs, and exclude the original inputs from the
@@ -134,3 +141,5 @@ see some basic information about the dataset.
 For example, use `textattack peek-dataset --dataset-from-huggingface glue^mrpc` to see
 information about the MRPC dataset (from the GLUE set of datasets). This will
 print statistics like the number of labels, average number of words, etc.
+
+

docs/0_get_started/installation.md

@@ -0,0 +1,80 @@
+Installation
+==============
+
+To use TextAttack, you must be running Python 3.6 or above. A CUDA-compatible GPU is optional but will greatly improve speed. 
+
+We recommend installing TextAttack in a virtual environment (check out this [guide](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/)).
+
+There are two ways to install TextAttack. If you want to simply use as it is, install via `pip`. If you want to make any changes and play around, install it from source.
+
+## Install with pip
+Simply run
+
+    pip install textattack 
+
+## Install from Source
+To install TextAttack from source, first clone the repo by running
+
+    git clone https://github.com/QData/TextAttack.git
+    cd TextAttack
+
+Then, install it using `pip`.
+
+    pip install -e . 
+
+To install TextAttack for further development, please run this instead.
+
+    pip install -e .[dev]
+
+This installs additional dependencies required for development.
+
+
+## Optional Dependencies
+For quick installation, TextAttack only installs esssential packages as dependencies (e.g. Transformers, PyTorch). However, you might need to install additional packages to run certain attacks or features.
+For example, Tensorflow and Tensorflow Hub are required to use the TextFooler attack, which was proposed in [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://arxiv.org/abs/1907.11932) by Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolov.
+
+If you attempting to use a feature that requires additional dependencies, TextAttack will let you know which ones you need to install.
+
+However, during installation step, you can also install them together with TextAttack.
+You can install Tensorflow and its related packages by running
+
+    pip install textattack[tensorflow]
+
+You can also install other miscallenous optional dependencies by running
+
+    pip install textattack[optional]
+
+To install both groups of packages, run
+
+    pip install textattack[tensorflow,optional]
+
+
+
+## FAQ on installation
+
+For many of the dependent library issues, the following command is the first you could try: 
+```bash
+pip install --force-reinstall textattack
+```
+
+OR 
+```bash
+pip install textattack[tensorflow,optional]
+```
+
+
+Besides, we highly recommend you to use virtual environment for textattack use, 
+see [information here](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#removing-an-environment). Here is one conda example: 
+
+```bash
+conda create -n textattackenv python=3.7
+conda activate textattackenv
+conda env list
+```
+
+If you want to use the most-up-to-date version of textattack (normally with newer bug fixes), you can run the following: 
+```bash
+git clone https://github.com/QData/TextAttack.git
+cd TextAttack
+pip install .[dev]
+```

docs/0_get_started/quick_api_tour.rst

@@ -0,0 +1,38 @@
+Quick Tour
+==========================
+
+Let us have a quick look at how TextAttack can be used to carry out adversarial attack.
+
+Attacking a BERT model
+------------------------------
+Let us attack a BERT model fine-tuned for sentimental classification task. We are going to use a model that has already been fine-tuned on IMDB dataset using the Transformers library. 
+
+.. code-block::
+
+    >>> import transformers
+    >>> model = transformers.AutoModelForSequenceClassification.from_pretrained("textattack/bert-base-uncased-imdb")
+    >>> tokenizer = transformers.AutoTokenizer.from_pretrained("textattack/bert-base-uncased-imdb")
+
+
+TextAttack requires both the model and the tokenizer to be wrapped by a :class:`~transformers.models.wrapper.ModelWrapper` class that implements the forward pass operation given a list of input texts. For models provided by Transformers library, we can also simply use :class:`~transformers.models.wrapper.HuggingFaceModelWrapper` class which implements both the forward pass and tokenization.
+
+.. code-block::
+
+    >>> import textattack
+    >>> model_wrapper = textattack.models.wrappers.HuggingFaceModelWrapper(model, tokenizer)
+
+Next, let's build the attack that we want to use. TextAttack provides prebuilt attacks in the form of :class:`~transformers.attack_recipes.AttackRecipe`. For this example, we will use :ref:TextFooler attack 
+
+
+.. code-block::
+
+    >>> dataset = textattack.datasets.HuggingFaceDataset("imdb", split="test")
+    >>> attack = textattack.attack_recipes.TextFoolerJin2019.build(model_wrapper)
+    >>> # Attack 20 samples with CSV logging and checkpoint saved every 5 interval
+    >>> # Attack 20 samples with CSV logging and checkpoint saved every 5 interval
+    >>> attack_args = textattack.AttackArgs(num_examples=20, log_to_csv="log.csv", checkpoint_interval=5, checkpoint_dir="checkpoints", disable_stdout=True)
+    >>> attacker = textattack.Attacker(attack, dataset, attack_args)
+    >>> attacker.attack_dataset()
+
+
+.. image:: ../_static/imgs/overview.png

docs/1start/FAQ.md

@@ -0,0 +1,157 @@
+Frequently Asked Questions
+========================================
+
+## Via Slack: Where to Ask Questions: 
+
+For help and realtime updates related to TextAttack, please [join the TextAttack Slack](https://join.slack.com/t/textattack/shared_invite/zt-huomtd9z-KqdHBPPu2rOP~Z8q3~urgg)!
+
+
+## Via CLI: `--help`
+
++ Easiest self help:   `textattack --help`
++ More concrete self help: 
+  - `textattack attack --help`  
+  - `textattack augment --help`
+  - `textattack train --help`
+  - `textattack peek-dataset --help`
+  - `textattack list`, e.g., `textattack list search-methods`
+
+
+## Via our papers: More details on results  
++ [references](https://textattack.readthedocs.io/en/latest/1start/references.html)
+
+
+## Via readthedocs: More details on APIs
++ [complete API reference on TextAttack](https://textattack.readthedocs.io/en/latest/apidoc/textattack.html) 
+
+
+## More Concrete Questions: 
+
+
+### 0. For many of the dependent library issues, the following command is the first you could try: 
+```bash
+pip install --force-reinstall textattack
+```
+
+OR 
+```bash
+pip install textattack[tensorflow,optional]
+```
+
+
+Besides, we highly recommend you to use virtual environment for textattack use, 
+see [information here](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#removing-an-environment). Here is one conda example: 
+
+```bash
+conda create -n textattackenv python=3.7
+conda activate textattackenv
+conda env list
+```
+
+If you want to use the most-up-to-date version of textattack (normally with newer bug fixes), you can run the following: 
+```bash
+git clone https://github.com/QData/TextAttack.git
+cd TextAttack
+pip install .[dev]
+```
+
+
+### 1. How to Train
+
+For example, you can *Train our default LSTM for 50 epochs on the Yelp Polarity dataset:*
+```bash
+textattack train --model-name-or-path lstm --dataset yelp_polarity  --epochs 50 --learning-rate 1e-5
+```
+
+
+*Fine-Tune `bert-base` on the `CoLA` dataset for 5 epochs*:
+```bash
+textattack train --model-name-or-path bert-base-uncased --dataset glue^cola --per-device-train-batch-size 8 --epochs 5
+```
+
+
+### 2. Use Custom  Models  
+
+TextAttack is model-agnostic!  You can use `TextAttack` to analyze any model that outputs IDs, tensors, or strings. To help users, TextAttack includes pre-trained models for different common NLP tasks. This makes it easier for
+users to get started with TextAttack. It also enables a more fair comparison of attacks from the literature. A list of available pretrained models and their validation accuracies is available at [HERE](https://textattack.readthedocs.io/en/latest/3recipes/models.html).
+
+
+You can easily try out an attack on a local model you prefer. To attack a pre-trained model, create a short file that loads them as variables `model` and `tokenizer`.  The `tokenizer` must
+be able to transform string inputs to lists or tensors of IDs using a method called `encode()`. The
+model must take inputs via the `__call__` method.
+
+##### Model from a file
+To experiment with a model you've trained, you could create the following file
+and name it `my_model.py`:
+
+```python
+model = load_your_model_with_custom_code() # replace this line with your model loading code
+tokenizer = load_your_tokenizer_with_custom_code() # replace this line with your tokenizer loading code
+```
+
+Then, run an attack with the argument `--model-from-file my_model.py`. The model and tokenizer will be loaded automatically.
+
+TextAttack is model-agnostic - meaning it can run attacks on models implemented in any deep learning framework. Model objects must be able to take a string (or list of strings) and return an output that can be processed by the goal function. For example, machine translation models take a list of strings as input and produce a list of strings as output. Classification and entailment models return an array of scores. As long as the user's model meets this specification, the model is fit to use with TextAttack.
+
+
+### 3. Use Custom Datasets 
+
+
+#### From a file
+
+Loading a dataset from a file is very similar to loading a model from a file. A 'dataset' is any iterable of `(input, output)` pairs.
+The following example would load a sentiment classification dataset from file `my_dataset.py`:
+
+```python
+dataset = [('Today was....', 1), ('This movie is...', 0), ...]
+```
+
+You can then run attacks on samples from this dataset by adding the argument `--dataset-from-file my_dataset.py`.
+
+
+
+#### Dataset loading via other mechanism, see: [more details at here](https://textattack.readthedocs.io/en/latest/api/datasets.html)
+
+```python
+import textattack
+my_dataset = [("text",label),....]
+new_dataset = textattack.datasets.Dataset(my_dataset)
+```
+
+
+#### Custom Dataset via AttackedText class
+
+To allow for word replacement after a sequence has been tokenized, we include an `AttackedText` object
+which maintains both a list of tokens and the original text, with punctuation. We use this object in favor of a list of words or just raw text.
+
+
+
+### 4. Benchmarking Attacks
+
+- See our analysis paper: Searching for a Search Method: Benchmarking Search Algorithms for Generating NLP Adversarial Examples at [EMNLP BlackBoxNLP](https://arxiv.org/abs/2009.06368). 
+
+- As we emphasized in the above paper, we don't recommend to directly compare Attack Recipes out of the box. 
+
+- This comment is due to that attack recipes in the recent literature used different ways or thresholds in setting up their constraints. Without the constraint space held constant, an increase in attack success rate could from an improved search or transformation method or a less restrictive search space. 
+
+
+### 5. Create Custom or New Attacks
+
+The `attack_one` method in an `Attack` takes as input an `AttackedText`, and outputs either a `SuccessfulAttackResult` if it succeeds or a `FailedAttackResult` if it fails. 
+
+- [Here is an example of using TextAttack to create a new attack method](https://github.com/jxmorris12/second-order-adversarial-examples) 
+
+
+We formulate an attack as consisting of four components: a **goal function** which determines if the attack has succeeded, **constraints** defining which perturbations are valid, a **transformation** that generates potential modifications given an input, and a **search method** which traverses through the search space of possible perturbations. The attack attempts to perturb an input text such that the model output fulfills the goal function (i.e., indicating whether the attack is successful) and the perturbation adheres to the set of constraints (e.g., grammar constraint, semantic similarity constraint). A search method is used to find a sequence of transformations that produce a successful adversarial example.
+
+
+This modular design unifies adversarial attack methods into one system, enables us to easily assemble attacks from the literature while re-using components that are shared across attacks. We provides clean, readable implementations of 16 adversarial attack recipes from the literature (see [our tool paper](https://arxiv.org/abs/2005.05909) and [our benchmark search paper](https://arxiv.org/abs/2009.06368)). For the first time, these attacks can be benchmarked, compared, and analyzed in a standardized setting.
+
+
+
+### 6. The attacking is too slow 
+
+
+- **Tip:** If your machine has multiple GPUs, you can distribute the attack across them using the `--parallel` option. For some attacks, this can really help performance.
+
+- If you want to attack Keras models in parallel, please check out `examples/attack/attack_keras_parallel.py` instead. (This is a hotfix for issues caused by a recent update of Keras in TF)

docs/1start/api-design-tips.md

@@ -7,6 +7,24 @@ Lessons learned in designing TextAttack
 
 TextAttack is an open-source Python toolkit for adversarial attacks, adversarial training, and data augmentation in NLP. TextAttack unites 15+ papers from the NLP adversarial attack literature into a single shared framework, with many components reused across attacks. This framework allows both researchers and developers to test and study the weaknesses of their NLP models. 
 
+
+## Presentations on TextAttack 
+
+### 2020: Jack Morris' summary tutorial talk on TextAttack 
+
+- On Jul 31, 2020, Jack Morries gave an invited talk at Weights & Biases research salon on " TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP"
+
+- [Youtube Talk link](https://www.youtube.com/watch?v=22Q3f7Fb110)
+
+
+### 2021: Dr. Qi's summary tutorial talk on TextAttack 
+
+- On April 14 2021, Prof. Qi gave an invited talk at the UVA Human and Machine Intelligence Seminar on "Generalizing Adversarial Examples to Natural Language Processing"
+
+- [TalkSlide](https://qdata.github.io/qdata-page/pic/20210414-HMI-textAttack.pdf)
+
+
+
 ## Challenges in Design
 
 
@@ -17,7 +35,6 @@ One of the challenges for building such tools is that the tool should be flexibl
 
 We provide the following broad advice to help other future developers create user-friendly NLP libraries in Python:
 - To become model-agnostic, implement a model wrapper class: a model is anything that takes string input(s) and returns a prediction.
-- To become model-agnostic, implement a model wrapper class.
 - To become data-agnostic, take dataset inputs as (input, output) pairs, where each model input is represented as an OrderedDict.
 - Do not plan for inputs (tensors, lists, etc.) to be a certain size or shape unless explicitly necessary.
 - Centralize common text operations, like parsing and string-level operations, in one class.
@@ -29,6 +46,17 @@ We provide the following broad advice to help other future developers create use
  Our modular and extendable design allows us to reuse many components to offer 15+ different adversarial attack methods proposed by literature. Our model-agnostic and dataset-agnostic design allows users to easily run adversarial attacks against their own models built using any deep learning framework. We hope that our lessons from developing TextAttack will help others create user-friendly open-source NLP libraries.
 
 
+## TextAttack flowchart
+
+![TextAttack flowchart](/_static/imgs/intro/textattack_components.png)
+
+
++ Here is a summary diagram of TextAttack Ecosystem
+
+![diagram](/_static/imgs/intro/textattack_ecosystem.png)
+
+
+
 ## More Details in Reference
 
 ```

docs/1start/attacks4Components.md

@@ -0,0 +1,245 @@
+Four Components of TextAttack Attacks
+========================================
+
+To unify adversarial attack methods into one system, We formulate an attack as consisting of four components: a **goal function** which determines if the attack has succeeded, **constraints** defining which perturbations are valid, a **transformation** that generates potential modifications given an input, and a **search method** which traverses through the search space of possible perturbations.  The attack attempts to perturb an input text such that the model output fulfills the goal function (i.e., indicating whether the attack is successful) and the perturbation adheres to the set of constraints (e.g., grammar constraint, semantic similarity constraint). A search method is used to find a sequence of transformations that produce a successful adversarial example.
+
+
+
+This modular design enables us to easily assemble attacks from the literature while re-using components that are shared across attacks. TextAttack provides clean, readable implementations of 16 adversarial attacks from the literature. For the first time, these attacks can be benchmarked, compared, and analyzed in a standardized setting.
+
+
+- Two examples showing four components of two SOTA attacks
+![two-categorized-attacks](/_static/imgs/intro/01-categorized-attacks.png)
+
+
+- You can create one new attack (in one line of code!!!) from composing members of four components we proposed, for instance: 
+
+```bash 
+# Shows how to build an attack from components and use it on a pre-trained model on the Yelp dataset.
+textattack attack --attack-n --model bert-base-uncased-yelp --num-examples 8 \
+    --goal-function untargeted-classification \
+    --transformation word-swap-wordnet \
+    --constraints edit-distance^12 max-words-perturbed^max_percent=0.75 repeat stopword \
+    --search greedy
+```
+
+### Goal Functions
+
+A `GoalFunction` takes as input an `AttackedText` object, scores it, and determines whether the attack has succeeded, returning a `GoalFunctionResult`.
+
+### Constraints
+
+A `Constraint` takes as input a current `AttackedText`, and a list of transformed `AttackedText`s. For each transformed option, it returns a boolean representing whether the constraint is met.
+
+### Transformations
+
+A `Transformation` takes as input an `AttackedText` and returns a list of possible transformed `AttackedText`s. For example, a transformation might return all possible synonym replacements.
+
+### Search Methods
+
+A `SearchMethod` takes as input an initial `GoalFunctionResult` and returns a final `GoalFunctionResult` The search is given access to the `get_transformations` function, which takes as input an `AttackedText` object and outputs a list of possible transformations filtered by meeting all of the attack’s constraints. A search consists of successive calls to `get_transformations` until the search succeeds (determined using `get_goal_results`) or is exhausted.
+
+
+
+### On Benchmarking Attack Recipes
+
+- Please read our analysis paper: Searching for a Search Method: Benchmarking Search Algorithms for Generating NLP Adversarial Examples at [EMNLP BlackBoxNLP](https://arxiv.org/abs/2009.06368). 
+
+- As we emphasized in the above paper, we don't recommend to directly compare Attack Recipes out of the box. 
+
+- This is due to that attack recipes in the recent literature used different ways or thresholds in setting up their constraints. Without the constraint space held constant, an increase in attack success rate could come from an improved search or a better transformation method or a less restrictive search space. 
+
+
+
+### Four components in Attack Recipes we have implemented 
+
+
+- TextAttack provides clean, readable implementations of 16 adversarial attacks from the literature.
+
+- To run an attack recipe: `textattack attack --recipe [recipe_name]`
+
+
+
+<table  style="width:100%" border="1">
+<thead>
+<tr class="header">
+<th style="text-align: left;"><strong>Attack Recipe Name</strong></th>
+<th style="text-align: left;"><strong>Goal Function</strong></th>
+<th style="text-align: left; width:130px" ><strong>Constraints-Enforced</strong></th>
+<th style="text-align: left;"><strong>Transformation</strong></th>
+<th style="text-align: left;"><strong>Search Method</strong></th>
+<th style="text-align: left;"><strong>Main Idea</strong></th>
+</tr>
+</thead>
+<tbody>
+  <tr><td style="text-align: center;" colspan="6"><strong><br>Attacks on classification tasks, like sentiment classification and entailment:<br></strong></td></tr>
+
+<tr class="even">
+<td style="text-align: left;"><code>alzantot</code>  <span class="citation" data-cites="Alzantot2018GeneratingNL Jia2019CertifiedRT"></span></td>
+<td style="text-align: left;"><sub>Untargeted {Classification, Entailment}</sub></td>
+<td style="text-align: left;"><sub>Percentage of words perturbed, Language Model perplexity, Word embedding distance</sub></td>
+<td style="text-align: left;"><sub>Counter-fitted word embedding swap</sub></td>
+<td style="text-align: left;"><sub>Genetic Algorithm</sub></td>
+<td ><sub>from (["Generating Natural Language Adversarial Examples" (Alzantot et al., 2018)](https://arxiv.org/abs/1804.07998))</sub></td>
+</tr>
+<tr class="odd">
+<td style="text-align: left;"><code>bae</code> <span class="citation" data-cites="garg2020bae"></span></td>
+<td style="text-align: left;"><sub>Untargeted Classification</sub></td>
+<td style="text-align: left;"><sub>USE sentence encoding cosine similarity</sub></td>
+<td style="text-align: left;"><sub>BERT Masked Token Prediction</sub></td>
+<td style="text-align: left;"><sub>Greedy-WIR</sub></td>
+<td ><sub>BERT masked language model transformation attack from (["BAE: BERT-based Adversarial Examples for Text Classification" (Garg & Ramakrishnan, 2019)](https://arxiv.org/abs/2004.01970)). </td>
+</tr>
+<tr class="even">
+<td style="text-align: left;"><code>bert-attack</code> <span class="citation" data-cites="li2020bertattack"></span></td>
+<td style="text-align: left;"><sub>Untargeted Classification</td>
+<td style="text-align: left;"><sub>USE sentence encoding cosine similarity, Maximum number of words perturbed</td>
+<td style="text-align: left;"><sub>BERT Masked Token Prediction (with subword expansion)</td>
+<td style="text-align: left;"><sub>Greedy-WIR</sub></td>
+<td ><sub> (["BERT-ATTACK: Adversarial Attack Against BERT Using BERT" (Li et al., 2020)](https://arxiv.org/abs/2004.09984))</sub></td>
+</tr>
+<tr class="odd">
+<td style="text-align: left;"><code>checklist</code> <span class="citation" data-cites="Gao2018BlackBoxGO"></span></td>
+<td style="text-align: left;"><sub>{Untargeted, Targeted} Classification</sub></td>
+<td style="text-align: left;"><sub>checklist distance</sub></td>
+<td style="text-align: left;"><sub>contract, extend, and substitutes name entities</sub></td>
+<td style="text-align: left;"><sub>Greedy-WIR</sub></td>
+<td ><sub>Invariance testing implemented in CheckList . (["Beyond Accuracy: Behavioral Testing of NLP models with CheckList" (Ribeiro et al., 2020)](https://arxiv.org/abs/2005.04118))</sub></td>
+</tr>
+<tr>
+<td> <code>clare</code> <span class="citation" data-cites="Alzantot2018GeneratingNL Jia2019CertifiedRT"></span></td>
+<td><sub>Untargeted {Classification, Entailment}</sub></td>
+<td><sub>USE sentence encoding cosine similarity</sub></td>
+<td><sub>RoBERTa Masked Prediction for token swap, insert and merge</sub></td>
+<td><sub>Greedy</sub></td>
+<td ><sub>["Contextualized Perturbation for Textual Adversarial Attack" (Li et al., 2020)](https://arxiv.org/abs/2009.07502))</sub></td>
+</tr>
+<tr class="odd">
+<td style="text-align: left;"><code>deepwordbug</code> <span class="citation" data-cites="Gao2018BlackBoxGO"></span></td>
+<td style="text-align: left;"><sub>{Untargeted, Targeted} Classification</sub></td>
+<td style="text-align: left;"><sub>Levenshtein edit distance</sub></td>
+<td style="text-align: left;"><sub>{Character Insertion, Character Deletion, Neighboring Character Swap, Character Substitution}</sub></td>
+<td style="text-align: left;"><sub>Greedy-WIR</sub></td>
+<td ><sub>Greedy replace-1 scoring and multi-transformation character-swap attack (["Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers" (Gao et al., 2018)](https://arxiv.org/abs/1801.04354)</sub></td>
+</tr>
+<tr class="even">
+<td style="text-align: left;"> <code>fast-alzantot</code> <span class="citation" data-cites="Alzantot2018GeneratingNL Jia2019CertifiedRT"></span></td>
+<td style="text-align: left;"><sub>Untargeted {Classification, Entailment}</sub></td>
+<td style="text-align: left;"><sub>Percentage of words perturbed, Language Model perplexity, Word embedding distance</sub></td>
+<td style="text-align: left;"><sub>Counter-fitted word embedding swap</sub></td>
+<td style="text-align: left;"><sub>Genetic Algorithm</sub></td>
+<td ><sub>Modified, faster version of the Alzantot et al. genetic algorithm, from (["Certified Robustness to Adversarial Word Substitutions" (Jia et al., 2019)](https://arxiv.org/abs/1909.00986))</sub></td>
+</tr>
+<tr class="even">
+<td style="text-align: left;"><code>hotflip</code> (word swap) <span class="citation" data-cites="Ebrahimi2017HotFlipWA"></span></td>
+<td style="text-align: left;"><sub>Untargeted Classification</sub></td>
+<td style="text-align: left;"><sub>Word Embedding Cosine Similarity, Part-of-speech match, Number of words perturbed</sub></td>
+<td style="text-align: left;"><sub>Gradient-Based Word Swap</sub></td>
+<td style="text-align: left;"><sub>Beam search</sub></td>
+<td ><sub> (["HotFlip: White-Box Adversarial Examples for Text Classification" (Ebrahimi et al., 2017)](https://arxiv.org/abs/1712.06751))</sub></td>
+</tr>
+<tr class="odd">
+<td style="text-align: left;"><code>iga</code> <span class="citation" data-cites="iga-wang2019natural"></span></td>
+<td style="text-align: left;"><sub>Untargeted {Classification, Entailment}</sub></td>
+<td style="text-align: left;"><sub>Percentage of words perturbed, Word embedding distance</sub></td>
+<td style="text-align: left;"><sub>Counter-fitted word embedding swap</sub></td>
+<td style="text-align: left;"><sub>Genetic Algorithm</sub></td>
+<td ><sub>Improved genetic algorithm -based word substitution from (["Natural Language Adversarial Attacks and Defenses in Word Level (Wang et al., 2019)"](https://arxiv.org/abs/1909.06723)</sub></td>
+</tr>
+<tr class="even">
+<td style="text-align: left;"><code>input-reduction</code> <span class="citation" data-cites="feng2018pathologies"></span></td>
+<td style="text-align: left;"><sub>Input Reduction</sub></td>
+<td style="text-align: left;"></td>
+<td style="text-align: left;"><sub>Word deletion</sub></td>
+<td style="text-align: left;"><sub>Greedy-WIR</sub></td>
+<td ><sub>Greedy attack with word importance ranking , Reducing the input while maintaining the prediction through word importance ranking (["Pathologies of Neural Models Make Interpretation Difficult" (Feng et al., 2018)](https://arxiv.org/pdf/1804.07781.pdf))</sub></td>
+</tr>
+<tr class="odd">
+<td style="text-align: left;"><code>kuleshov</code> <span class="citation" data-cites="Kuleshov2018AdversarialEF"></span></td>
+<td style="text-align: left;"><sub>Untargeted Classification</sub></td>
+<td style="text-align: left;"><sub>Thought vector encoding cosine similarity, Language model similarity probability</sub></td>
+<td style="text-align: left;"><sub>Counter-fitted word embedding swap</sub></td>
+<td style="text-align: left;"><sub>Greedy word swap</sub></td>
+<td ><sub>(["Adversarial Examples for Natural Language Classification Problems" (Kuleshov et al., 2018)](https://openreview.net/pdf?id=r1QZ3zbAZ)) </sub></td>
+</tr>
+<tr class="even">
+<td style="text-align: left;"><code>pruthi</code> <span class="citation" data-cites="pruthi2019combating"></span></td>
+<td style="text-align: left;"><sub>Untargeted Classification</sub></td>
+<td style="text-align: left;"><sub>Minimum word length, Maximum number of words perturbed</sub></td>
+<td style="text-align: left;"><sub>{Neighboring Character Swap, Character Deletion, Character Insertion, Keyboard-Based Character Swap}</sub></td>
+<td style="text-align: left;"><sub>Greedy search</sub></td>
+<td ><sub>simulates common typos (["Combating Adversarial Misspellings with Robust Word Recognition" (Pruthi et al., 2019)](https://arxiv.org/abs/1905.11268) </sub></td>
+</tr>
+<tr class="odd">
+<td style="text-align: left;"><code>pso</code> <span class="citation" data-cites="pso-zang-etal-2020-word"></span></td>
+<td style="text-align: left;"><sub>Untargeted Classification</sub></td>
+<td style="text-align: left;"></td>
+<td style="text-align: left;"><sub>HowNet Word Swap</sub></td>
+<td style="text-align: left;"><sub>Particle Swarm Optimization</sub></td>
+<td ><sub>(["Word-level Textual Adversarial Attacking as Combinatorial Optimization" (Zang et al., 2020)](https://www.aclweb.org/anthology/2020.acl-main.540/)) </sub></td>
+</tr>
+<tr class="even">
+<td style="text-align: left;"><code>pwws</code> <span class="citation" data-cites="pwws-ren-etal-2019-generating"></span></td>
+<td style="text-align: left;"><sub>Untargeted Classification</sub></td>
+<td style="text-align: left;"></td>
+<td style="text-align: left;"><sub>WordNet-based synonym swap</sub></td>
+<td style="text-align: left;"><sub>Greedy-WIR (saliency)</sub></td>
+<td ><sub>Greedy attack with word importance ranking based on word saliency and synonym swap scores (["Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency" (Ren et al., 2019)](https://www.aclweb.org/anthology/P19-1103/))</sub> </td>
+</tr>
+<tr class="even">
+<td style="text-align: left;"><code>textbugger</code> : (black-box) <span class="citation" data-cites="Li2019TextBuggerGA"></span></td>
+<td style="text-align: left;"><sub>Untargeted Classification</sub></td>
+<td style="text-align: left;"><sub>USE sentence encoding cosine similarity</sub></td>
+<td style="text-align: left;"><sub>{Character Insertion, Character Deletion, Neighboring Character Swap, Character Substitution}</sub></td>
+<td style="text-align: left;"><sub>Greedy-WIR</sub></td>
+<td ><sub>([(["TextBugger: Generating Adversarial Text Against Real-world Applications" (Li et al., 2018)](https://arxiv.org/abs/1812.05271)).</sub></td>
+</tr>
+<tr class="odd">
+<td style="text-align: left;"><code>textfooler</code> <span class="citation" data-cites="Jin2019TextFooler"></span></td>
+<td style="text-align: left;"><sub>Untargeted {Classification, Entailment}</sub></td>
+<td style="text-align: left;"><sub>Word Embedding Distance, Part-of-speech match, USE sentence encoding cosine similarity</sub></td>
+<td style="text-align: left;"><sub>Counter-fitted word embedding swap</sub></td>
+<td style="text-align: left;"><sub>Greedy-WIR</sub></td>
+<td ><sub>Greedy attack with word importance ranking  (["Is Bert Really Robust?" (Jin et al., 2019)](https://arxiv.org/abs/1907.11932))</sub> </td>
+</tr>
+
+<tr><td style="text-align: center;" colspan="6"><strong><br>Attacks on sequence-to-sequence models: <br></strong></td></tr>
+
+<tr class="odd">
+<td style="text-align: left;"><code>morpheus</code> <span class="citation" data-cites="morpheus-tan-etal-2020-morphin"></span></td>
+<td style="text-align: left;"><sub>Minimum BLEU Score</sub> </td>
+<td style="text-align: left;"></td>
+<td style="text-align: left;"><sub>Inflection Word Swap</sub> </td>
+<td style="text-align: left;"><sub>Greedy search</sub> </td>
+<td ><sub>Greedy to replace words with their inflections with the goal of minimizing BLEU score (["It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations"](https://www.aclweb.org/anthology/2020.acl-main.263.pdf)</sub> </td>
+</tr>
+
+</tr>
+<tr class="odd">
+<td style="text-align: left;"><code>seq2sick</code> :(black-box) <span class="citation" data-cites="cheng2018seq2sick"></span></td>
+<td style="text-align: left;"><sub>Non-overlapping output</sub> </td>
+<td style="text-align: left;"></td>
+<td style="text-align: left;"><sub>Counter-fitted word embedding swap</sub> </td>
+<td style="text-align: left;"><sub>Greedy-WIR</sub></td>
+<td ><sub>Greedy attack with goal of changing every word in the output translation. Currently implemented as black-box with plans to change to white-box as done in paper (["Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples" (Cheng et al., 2018)](https://arxiv.org/abs/1803.01128)) </sub>  </td>
+</tr>
+
+
+</tbody>
+</font>
+</table>
+
+
+
+- Citations
+
+```
+@misc{morris2020textattack,
+    title={TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP},
+    author={John X. Morris and Eli Lifland and Jin Yong Yoo and Jake Grigsby and Di Jin and Yanjun Qi},
+    year={2020},
+    eprint={2005.05909},
+    archivePrefix={arXiv},
+    primaryClass={cs.CL}
+}
+```

docs/1start/benchmark-search.md

@@ -1,9 +1,8 @@
-
 Benchmarking Search Algorithms for Generating NLP Adversarial Examples
 =========================================================================
 
 
-*This documentation page was adapted from Our Paper in [EMNLP BlackNLP](https://arxiv.org/abs/2009.06368).*
+*This documentation page was adapted from Our Paper in [EMNLP BlackBoxNLP](https://arxiv.org/abs/2009.06368).*
 
 
 ### Title: Searching for a Search Method: Benchmarking Search Algorithms for Generating NLP Adversarial Examples
@@ -12,7 +11,7 @@ Benchmarking Search Algorithms for Generating NLP Adversarial Examples
 - Abstract:  We study the behavior of several black-box search algorithms used for generating adversarial examples for natural language processing (NLP) tasks. We perform a fine-grained analysis of three elements relevant to search: search algorithm, search space, and search budget. When new search methods are proposed in past work, the attack search space is often modified alongside the search method. Without ablation studies benchmarking the search algorithm change with the search space held constant, an increase in attack success rate could from an improved search method or a less restrictive search space. Additionally, many previous studies fail to properly consider the search algorithms' run-time cost, which is essential for downstream tasks like adversarial training. Our experiments provide a reproducible benchmark of search algorithms across a variety of search spaces and query budgets to guide future research in adversarial NLP. Based on our experiments, we recommend greedy attacks with word importance ranking when under a time constraint or attacking long inputs, and either beam search or particle swarm optimization otherwise. 
 
 
-###  Citations: 
++ Citations: 
 ```
 @misc{yoo2020searching,
       title={Searching for a Search Method: Benchmarking Search Algorithms for Generating NLP Adversarial Examples}, 
@@ -24,18 +23,26 @@ Benchmarking Search Algorithms for Generating NLP Adversarial Examples
 }
 ```
 
+### Our search benchmarking result Github 
 
+TextAttack-Search-Benchmark Github [https://github.com/QData/TextAttack-Search-Benchmark](https://github.com/QData/TextAttack-Search-Benchmark)
 
 ### Our benchmarking results on comparing search methods used in the past attacks. 
 
 
-![Table1](/_static/imgs/benchmark/search-example.pdf)
+![search-example](/_static/imgs/benchmark/search-example.pdf)
 
-![Table1](/_static/imgs/benchmark/search-table2.png)
-![Table1](/_static/imgs/benchmark/search-table31.png)
-![Table1](/_static/imgs/benchmark/search-table32.png)
-![Table1](/_static/imgs/benchmark/search-fig1.png)
-![Table1](/_static/imgs/benchmark/search-fig2.png)
+![Table2](/_static/imgs/benchmark/search-table2.png)
+![Table31](/_static/imgs/benchmark/search-table31.png)
+![Table32](/_static/imgs/benchmark/search-table32.png)
+![Figure1](/_static/imgs/benchmark/search-fig1.png)
+![Figure2](/_static/imgs/benchmark/search-fig2.png)
 
 ![Table1](/_static/imgs/benchmark/search-table1.png)
 
+
+### Benchmarking Attack Recipes 
+
+- As we emphasized in the above paper, we don't recommend to directly compare Attack Recipes out of the box. 
+
+- This is due to that attack recipes in the recent literature used different ways or thresholds in setting up their constraints. Without the constraint space held constant, an increase in attack success rate could come from an improved search or transformation method or a less restrictive search space. 

docs/1start/installation.rst

@@ -1,20 +0,0 @@
-
-.. _installation:
-
-
-Installation
-==============
-
-To use TextAttack, you must be running Python 3.6+. Tensorflow needs to be installed for users, and Java needs to be installed for developers. A CUDA-compatible GPU is optional but will greatly improve speed. To install, simply run::
-
-    pip install textattack 
-
-You're now all set to use TextAttack! Try running an attack from the command line::
-
-    textattack attack --recipe textfooler --model bert-base-uncased-mr --num-examples 10
-
-This will run an attack using the TextFooler_ recipe, attacking BERT fine-tuned on the MR dataset. It will attack the first 10 samples. Once everything downloads and starts running, you should see attack results print to ``stdout``.
-
-Read on for more information on TextAttack, including how to use it from a Python script (``import textattack``).
-
-.. _TextFooler: https://arxiv.org/abs/1907.11932

docs/1start/multilingual-visualization.md

@@ -0,0 +1,68 @@
+TextAttack Extended Functions (Multilingual)
+============================================
+
+
+
+## Textattack Supports Multiple Model Types besides huggingface models and our textattack models: 
+
+- Example attacking TensorFlow models @ [https://textattack.readthedocs.io/en/latest/2notebook/Example_0_tensorflow.html](https://textattack.readthedocs.io/en/latest/2notebook/Example_0_tensorflow.html)
+- Example attacking scikit-learn models @ [https://textattack.readthedocs.io/en/latest/2notebook/Example_1_sklearn.html](https://textattack.readthedocs.io/en/latest/2notebook/Example_1_sklearn.html)
+- Example attacking AllenNLP models @ [https://textattack.readthedocs.io/en/latest/2notebook/Example_2_allennlp.html](https://textattack.readthedocs.io/en/latest/2notebook/Example_2_allennlp.html)
+- Example attacking Kera models @ [https://textattack.readthedocs.io/en/latest/2notebook/Example_3_Keras.html](https://textattack.readthedocs.io/en/latest/2notebook/Example_3_Keras.html)
+
+
+## Multilingual Supports
+
+
+- see tutorial notebook for using our framework to attack French-BERT.: [https://textattack.readthedocs.io/en/latest/2notebook/Example_4_CamemBERT.html](https://textattack.readthedocs.io/en/latest/2notebook/Example_4_CamemBERT.html)  
+
+- see example code for using our framework to attack French-BERT: [https://github.com/QData/TextAttack/blob/master/examples/attack/attack_camembert.py](https://github.com/QData/TextAttack/blob/master/examples/attack/attack_camembert.py) . 
+
+
+
+## User defined custom inputs and models 
+
+
+### Custom Datasets:  Dataset from a file
+
+Loading a dataset from a file is very similar to loading a model from a file. A 'dataset' is any iterable of `(input, output)` pairs.
+The following example would load a sentiment classification dataset from file `my_dataset.py`:
+
+```python
+dataset = [('Today was....', 1), ('This movie is...', 0), ...]
+```
+
+You can then run attacks on samples from this dataset by adding the argument `--dataset-from-file my_dataset.py`.
+
+
+#### Custom Model:  from a file
+To experiment with a model you've trained, you could create the following file
+and name it `my_model.py`:
+
+```python
+model = load_your_model_with_custom_code() # replace this line with your model loading code
+tokenizer = load_your_tokenizer_with_custom_code() # replace this line with your tokenizer loading code
+```
+
+Then, run an attack with the argument `--model-from-file my_model.py`. The model and tokenizer will be loaded automatically.
+
+
+
+## User defined Custom attack components 
+
+The [documentation website](https://textattack.readthedocs.io/en/latest) contains walkthroughs explaining basic usage of TextAttack, including building a custom transformation and a custom constraint..
+
+- custom transformation example @ [https://textattack.readthedocs.io/en/latest/2notebook/1_Introduction_and_Transformations.html](https://textattack.readthedocs.io/en/latest/2notebook/1_Introduction_and_Transformations.html)
+
+- custome constraint example @[https://textattack.readthedocs.io/en/latest/2notebook/2_Constraints.html#A-custom-constraint](https://textattack.readthedocs.io/en/latest/2notebook/2_Constraints.html#A-custom-constraint)
+
+
+
+
+## Visulizing TextAttack generated Examples; 
+
+
+- You can visualize the generated adversarial examples vs. see examples, following visualization ways we provided here: [https://textattack.readthedocs.io/en/latest/2notebook/2_Constraints.html](https://textattack.readthedocs.io/en/latest/2notebook/2_Constraints.html)
+
+- If you have webapp, we have also built a new WebDemo [TextAttack-WebDemo Github](https://github.com/QData/TextAttack-WebDemo) for visualizing generated adversarial examples from textattack.. 
+

docs/1start/quality-SOTA-recipes.md

@@ -0,0 +1,45 @@
+On Quality of Generated Adversarial Examples and How to Set Attack Contraints
+==============================================================================
+
+
+### Title: Reevaluating Adversarial Examples in Natural Language
+
+- Paper [EMNLP Findings](https://arxiv.org/abs/2004.14174)
+
+- Abstract:  State-of-the-art attacks on NLP models lack a shared definition of a what constitutes a successful attack. We distill ideas from past work into a unified framework: a successful natural language adversarial example is a perturbation that fools the model and follows some linguistic constraints. We then analyze the outputs of two state-of-the-art synonym substitution attacks. We find that their perturbations often do not preserve semantics, and 38% introduce grammatical errors. Human surveys reveal that to successfully preserve semantics, we need to significantly increase the minimum cosine similarities between the embeddings of swapped words and between the sentence encodings of original and perturbed sentences.With constraints adjusted to better preserve semantics and grammaticality, the attack success rate drops by over 70 percentage points.
+
+
+### Our Github on Reevaluation: [Reevaluating-NLP-Adversarial-Examples Github](https://github.com/QData/Reevaluating-NLP-Adversarial-Examples)
+
+
+- Citations
+```
+@misc{morris2020reevaluating,
+      title={Reevaluating Adversarial Examples in Natural Language}, 
+      author={John X. Morris and Eli Lifland and Jack Lanchantin and Yangfeng Ji and Yanjun Qi},
+      year={2020},
+      eprint={2004.14174},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```
+
+
+### Some of our evaluation results on quality of two SOTA attack recipes
+
+- As we have emphasized in this paper, we recommend researchers and users to be EXTREMELY mindful on the quality of generated adversarial examples in natural language 
+- We recommend the field to use human-evaluation derived thresholds for setting up constraints 
+
+
+![Table3](/_static/imgs/benchmark/table3.png)
+![Table4](/_static/imgs/benchmark/table4.png)
+
+
+### Some of our evaluation results on how to set constraints to evaluate NLP model's adversarial robustness
+
+![Table5](/_static/imgs/benchmark/table5-main.png)
+![Table7](/_static/imgs/benchmark/table7.png)
+
+
+
+![Table9](/_static/imgs/benchmark/table9.png)

docs/1start/references.md

@@ -1,19 +1,20 @@
 How to Cite TextAttack  
 ===========================
 
-## Main Paper:  TextAttack: A Framework for Adversarial Attacks in Natural Language Processing
+## Main Paper:  TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP
 
 - Paper [EMNLP Demo](https://arxiv.org/abs/2005.05909)
 
+- Abstract: An adversarial example is an input designed to fool a machine learning model. While there has been substantial research using adversarial attacks to analyze NLP models, each attack is implemented in its own code repository. It remains challenging to develop NLP attacks and utilize them to improve model performance. This paper introduces TextAttack, a Python framework for adversarial attacks, data augmentation, and adversarial training in NLP. TextAttack builds attacks from four components: a goal function, a set of constraints, a transformation, and a search method. TextAttack's modular design enables researchers to easily construct attacks from combinations of novel and existing components. TextAttack provides implementations of 16 adversarial attacks from the literature and supports a variety of models and datasets, including BERT and other transformers, and all GLUE tasks. TextAttack also includes data augmentation and adversarial training modules for using components of adversarial attacks to improve model accuracy and robustness. TextAttack is democratizing NLP: anyone can try data augmentation and adversarial training on any model or dataset, with just a few lines of code. Code and tutorials are available at this site.
 
-- Abstract: TextAttack is a library for generating natural language adversarial examples to fool natural language processing (NLP) models. TextAttack builds attacks from four components: a search method, goal function, transformation, and a set of constraints. Researchers can use these components to easily assemble new attacks. Individual components can be isolated and compared for easier ablation studies. TextAttack currently supports attacks on models trained for text classification and entailment across a variety of datasets. Additionally, TextAttack's modular design makes it easily extensible to new NLP tasks, models, and attack strategies. 
+### Our Github on TextAttack: [https://github.com/QData/TextAttack](https://github.com/QData/TextAttack)
 
 - Citations
 
 ```
 @misc{morris2020textattack,
-    title={TextAttack: A Framework for Adversarial Attacks in Natural Language Processing},
-    author={John X. Morris and Eli Lifland and Jin Yong Yoo and Yanjun Qi},
+    title={TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP},
+    author={John X. Morris and Eli Lifland and Jin Yong Yoo and Jake Grigsby and Di Jin and Yanjun Qi},
     year={2020},
     eprint={2005.05909},
     archivePrefix={arXiv},
@@ -28,6 +29,7 @@ How to Cite TextAttack
 
 - Abstract:  State-of-the-art attacks on NLP models lack a shared definition of a what constitutes a successful attack. We distill ideas from past work into a unified framework: a successful natural language adversarial example is a perturbation that fools the model and follows some linguistic constraints. We then analyze the outputs of two state-of-the-art synonym substitution attacks. We find that their perturbations often do not preserve semantics, and 38% introduce grammatical errors. Human surveys reveal that to successfully preserve semantics, we need to significantly increase the minimum cosine similarities between the embeddings of swapped words and between the sentence encodings of original and perturbed sentences.With constraints adjusted to better preserve semantics and grammaticality, the attack success rate drops by over 70 percentage points.
 
+### Our Github on Reevaluation: [Reevaluating-NLP-Adversarial-Examples Github](https://github.com/QData/Reevaluating-NLP-Adversarial-Examples)
 
 - Citations
 ```
@@ -43,10 +45,12 @@ How to Cite TextAttack
 
 ## Our Analysis paper: Searching for a Search Method: Benchmarking Search Algorithms for Generating NLP Adversarial Examples
 
-- Paper [EMNLP BlackNLP](https://arxiv.org/abs/2009.06368)
+- Paper [EMNLP BlackBoxNLP](https://arxiv.org/abs/2009.06368)
 
 - Abstract:  We study the behavior of several black-box search algorithms used for generating adversarial examples for natural language processing (NLP) tasks. We perform a fine-grained analysis of three elements relevant to search: search algorithm, search space, and search budget. When new search methods are proposed in past work, the attack search space is often modified alongside the search method. Without ablation studies benchmarking the search algorithm change with the search space held constant, an increase in attack success rate could from an improved search method or a less restrictive search space. Additionally, many previous studies fail to properly consider the search algorithms' run-time cost, which is essential for downstream tasks like adversarial training. Our experiments provide a reproducible benchmark of search algorithms across a variety of search spaces and query budgets to guide future research in adversarial NLP. Based on our experiments, we recommend greedy attacks with word importance ranking when under a time constraint or attacking long inputs, and either beam search or particle swarm optimization otherwise. 
 
+### Our Github on benchmarking:  [TextAttack-Search-Benchmark Github](https://github.com/QData/TextAttack-Search-Benchmark)
+
 
 - Citations: 
 ```
@@ -59,4 +63,3 @@ How to Cite TextAttack
       primaryClass={cs.CL}
 }
 ```
-

docs/1start/support.md

@@ -224,3 +224,10 @@ You can run TextAttack tests with `pytest`. Just type `make test`.
 
 
 #### This guide was heavily inspired by the awesome [transformers guide to contributing](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md)
+
++ More details on how we design our APIs at [API-Design-Tips](https://textattack.readthedocs.io/en/latest/1start/api-design-tips.html)
++ Here is a summary diagram of TextAttack Ecosystem
+
+![diagram](/_static/imgs/intro/textattack_ecosystem.png)
+
+

docs/1start/what_is_an_adversarial_attack.md

@@ -70,7 +70,7 @@ TextAttack attack recipes that fall under this category: deepwordbug, hotflip, p
 
 Some NLP models are trained to measure semantic similarity. Adversarial attacks based on the notion of semantic indistinguishability typically use another NLP model to enforce that perturbations are grammatically valid and semantically similar to the original input.
 
-TextAttack attack recipes that fall under this category: alzantot, bae, bert-attack, faster-alzantot, iga, kuleshov, pso, pwws, textbugger\*, textfooler
+TextAttack attack recipes that fall under this category: alzantot, bae, bert-attack, fast-alzantot, iga, kuleshov, pso, pwws, textbugger\*, textfooler
 
 \*The textbugger attack generates perturbations using both typo-like character edits and synonym substitutions. It could be considered to use both definitions of indistinguishability.
 
@@ -87,6 +87,7 @@ The following figure shows an overview of the main functionality of TextAttack:
 ![TextAttack flowchart](/_static/imgs/intro/textattack_components.png)
 
 
+
 ## The future of adversarial attacks in NLP
 
 We are excited to see the impact that TextAttack has on the NLP research community! One thing we would like to see research in is the combination of components from various papers. TextAttack makes it easy to run ablation studies to compare the effects of swapping out, say, search method from paper A with the search method from paper B, without making any other changes. (And these tests can be run across dozens of pre-trained models and datasets with no downloads!)

docs/2notebook/1_Introduction_and_Transformations.ipynb

@@ -2,14 +2,18 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "metadata": {},
+   "metadata": {
+    "id": "xK7B3NnYaPR6"
+   },
    "source": [
     "# The TextAttack ecosystem: search, transformations, and constraints"
    ]
   },
   {
    "cell_type": "markdown",
-   "metadata": {},
+   "metadata": {
+    "id": "9rY3w9b2aPSG"
+   },
    "source": [
     "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/QData/TextAttack/blob/master/docs/2notebook/1_Introduction_and_Transformations.ipynb)\n",
     "\n",
@@ -18,7 +22,18 @@
   },
   {
    "cell_type": "markdown",
-   "metadata": {},
+   "metadata": {
+    "id": "urhoEHXJf8YK"
+   },
+   "source": [
+    "Installation of  Attack-api branch"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "HTe13zUKaPSH"
+   },
    "source": [
     "An attack in TextAttack consists of four parts.\n",
     "\n",
@@ -38,7 +53,9 @@
   },
   {
    "cell_type": "markdown",
-   "metadata": {},
+   "metadata": {
+    "id": "tiXXNJO4aPSI"
+   },
    "source": [
     "### A custom transformation\n",
     "\n",
@@ -53,7 +70,9 @@
   {
    "cell_type": "code",
    "execution_count": 1,
-   "metadata": {},
+   "metadata": {
+    "id": "8r7zviXkaPSJ"
+   },
    "outputs": [],
    "source": [
     "from textattack.transformations import WordSwap\n",
@@ -76,7 +95,7 @@
   {
    "cell_type": "markdown",
    "metadata": {
-    "collapsed": true
+    "id": "RHGvZxenaPSJ"
    },
    "source": [
     "### Using our transformation\n",
@@ -89,159 +108,33 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
-   "metadata": {},
+   "execution_count": 2,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "wREwoDkMaPSK",
+    "outputId": "4a8f74c7-c51a-4216-8435-be52d2165d4c"
+   },
    "outputs": [
     {
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "\u001b[34;1mtextattack\u001b[0m: Goal function <class 'textattack.goal_functions.classification.untargeted_classification.UntargetedClassification'> compatible with model BertForSequenceClassification.\n"
-     ]
-    },
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "b537c513e8b3410eb2f7e3ec5df851fc",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=3939.0, style=ProgressStyle(description…"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n"
-     ]
-    },
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "4f3b600b1f1b4a4da538f43582846964",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=2486.0, style=ProgressStyle(description…"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "Using custom data configuration default\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n",
-      "Downloading and preparing dataset ag_news/default (download: 29.88 MiB, generated: 30.23 MiB, total: 60.10 MiB) to /u/edl9cy/.cache/huggingface/datasets/ag_news/default/0.0.0...\n"
-     ]
-    },
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "df8846bd027a457891dd665e3fd4156f",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=11045148.0, style=ProgressStyle(descrip…"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n"
-     ]
-    },
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "e3a3710421f6423ba77fb3276b3240af",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=751209.0, style=ProgressStyle(descripti…"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "\n"
-     ]
-    },
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    },
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "\u001b[34;1mtextattack\u001b[0m: Loading \u001b[94mnlp\u001b[0m dataset \u001b[94mag_news\u001b[0m, split \u001b[94mtest\u001b[0m.\n"
-     ]
-    },
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Dataset ag_news downloaded and prepared to /u/edl9cy/.cache/huggingface/datasets/ag_news/default/0.0.0. Subsequent calls will reuse this data.\n"
+      "textattack: Unknown if model of class <class 'transformers.models.bert.modeling_bert.BertForSequenceClassification'> compatible with goal function <class 'textattack.goal_functions.classification.untargeted_classification.UntargetedClassification'>.\n",
+      "Using custom data configuration default\n",
+      "Reusing dataset ag_news (/p/qdata/jy2ma/.cache/textattack/datasets/ag_news/default/0.0.0/0eeeaaa5fb6dffd81458e293dfea1adba2881ffcbdc3fb56baeb5a892566c29a)\n",
+      "textattack: Loading \u001b[94mdatasets\u001b[0m dataset \u001b[94mag_news\u001b[0m, split \u001b[94mtest\u001b[0m.\n"
      ]
     }
    ],
    "source": [
     "# Import the model\n",
     "import transformers\n",
-    "from textattack.models.tokenizers import AutoTokenizer\n",
     "from textattack.models.wrappers import HuggingFaceModelWrapper\n",
     "\n",
     "model = transformers.AutoModelForSequenceClassification.from_pretrained(\"textattack/bert-base-uncased-ag-news\")\n",
-    "tokenizer = AutoTokenizer(\"textattack/bert-base-uncased-ag-news\")\n",
+    "tokenizer = transformers.AutoTokenizer.from_pretrained(\"textattack/bert-base-uncased-ag-news\")\n",
     "\n",
     "model_wrapper = HuggingFaceModelWrapper(model, tokenizer)\n",
     "\n",
@@ -256,7 +149,9 @@
   },
   {
    "cell_type": "markdown",
-   "metadata": {},
+   "metadata": {
+    "id": "sfGMvqcTaPSN"
+   },
    "source": [
     "### Creating the attack\n",
     "Let's keep it simple: let's use a greedy search method, and let's not use any constraints for now. "
@@ -264,13 +159,15 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
-   "metadata": {},
+   "execution_count": 3,
+   "metadata": {
+    "id": "nSAHSoI_aPSO"
+   },
    "outputs": [],
    "source": [
     "from textattack.search_methods import GreedySearch\n",
     "from textattack.constraints.pre_transformation import RepeatModification, StopwordModification\n",
-    "from textattack.shared import Attack\n",
+    "from textattack import Attack\n",
     "\n",
     "# We're going to use our Banana word swap class as the attack transformation.\n",
     "transformation = BananaWordSwap() \n",
@@ -285,15 +182,23 @@
   },
   {
    "cell_type": "markdown",
-   "metadata": {},
+   "metadata": {
+    "id": "PqrHaZOaaPSO"
+   },
    "source": [
     "Let's print our attack to see all the parameters:"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
-   "metadata": {},
+   "execution_count": 4,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "d2qYOr0maPSP",
+    "outputId": "7266dc40-fc6c-4c78-90a8-8150e8fb5d8e"
+   },
    "outputs": [
     {
      "name": "stdout",
@@ -315,9 +220,34 @@
     "print(attack)"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "m97uyJxDh1wq",
+    "outputId": "87ca8836-9781-4c5d-85f2-7ffbf4a7ef80"
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "(OrderedDict([('text', \"Fears for T N pension after talks Unions representing workers at Turner   Newall say they are 'disappointed' after talks with stricken parent firm Federal Mogul.\")]), 2)\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(dataset[0])"
+   ]
+  },
   {
    "cell_type": "markdown",
-   "metadata": {},
+   "metadata": {
+    "id": "GYKoVFuXaPSP"
+   },
    "source": [
     "### Using the attack\n",
     "\n",
@@ -326,23 +256,263 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
-   "metadata": {},
+   "execution_count": 6,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "LyokhnFtaPSQ",
+    "outputId": "d8a43c4f-1551-40c9-d031-a42b429ed33d"
+   },
    "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "  0%|          | 0/10 [00:00<?, ?it/s]"
+     ]
+    },
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "1 of 10 successes complete.\n",
-      "2 of 10 successes complete.\n",
-      "3 of 10 successes complete.\n",
-      "4 of 10 successes complete.\n",
-      "5 of 10 successes complete.\n",
-      "6 of 10 successes complete.\n",
-      "7 of 10 successes complete.\n",
-      "8 of 10 successes complete.\n",
-      "9 of 10 successes complete.\n",
-      "10 of 10 successes complete.\n"
+      "Attack(\n",
+      "  (search_method): GreedySearch\n",
+      "  (goal_function):  UntargetedClassification\n",
+      "  (transformation):  BananaWordSwap\n",
+      "  (constraints): \n",
+      "    (0): RepeatModification\n",
+      "    (1): StopwordModification\n",
+      "  (is_black_box):  True\n",
+      ") \n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 1 / 0 / 0 / 1:  10%|█         | 1/10 [00:01<00:14,  1.57s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 1 ---------------------------------------------\n",
+      "\u001b[94mBusiness (100%)\u001b[0m --> \u001b[91mWorld (89%)\u001b[0m\n",
+      "\n",
+      "Fears for T N \u001b[94mpension\u001b[0m after \u001b[94mtalks\u001b[0m \u001b[94mUnions\u001b[0m representing \u001b[94mworkers\u001b[0m at Turner   Newall say they are '\u001b[94mdisappointed'\u001b[0m after talks with stricken parent firm Federal \u001b[94mMogul\u001b[0m.\n",
+      "\n",
+      "Fears for T N \u001b[91mbanana\u001b[0m after \u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m representing \u001b[91mbanana\u001b[0m at Turner   Newall say they are '\u001b[91mbanana\u001b[0m after talks with stricken parent firm Federal \u001b[91mbanana\u001b[0m.\n",
+      "\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 2 / 0 / 0 / 2:  20%|██        | 2/10 [00:13<00:53,  6.68s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 2 ---------------------------------------------\n",
+      "\u001b[35mSci/tech (100%)\u001b[0m --> \u001b[91mWorld (64%)\u001b[0m\n",
+      "\n",
+      "The Race is On: Second Private \u001b[35mTeam\u001b[0m Sets Launch \u001b[35mDate\u001b[0m for \u001b[35mHuman\u001b[0m \u001b[35mSpaceflight\u001b[0m (\u001b[35mSPACE\u001b[0m.\u001b[35mcom\u001b[0m) \u001b[35mSPACE\u001b[0m.\u001b[35mcom\u001b[0m - \u001b[35mTORONTO\u001b[0m, \u001b[35mCanada\u001b[0m -- \u001b[35mA\u001b[0m \u001b[35msecond\u001b[0m\\\u001b[35mteam\u001b[0m of rocketeers \u001b[35mcompeting\u001b[0m for the  #36;10 million Ansari X \u001b[35mPrize\u001b[0m, a \u001b[35mcontest\u001b[0m for\\\u001b[35mprivately\u001b[0m funded \u001b[35msuborbital\u001b[0m \u001b[35mspace\u001b[0m \u001b[35mflight\u001b[0m, has officially \u001b[35mannounced\u001b[0m the first\\\u001b[35mlaunch\u001b[0m date for its \u001b[35mmanned\u001b[0m rocket.\n",
+      "\n",
+      "The Race is On: Second Private \u001b[91mbanana\u001b[0m Sets Launch \u001b[91mbanana\u001b[0m for \u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m (\u001b[91mbanana\u001b[0m.\u001b[91mbanana\u001b[0m) \u001b[91mbanana\u001b[0m.\u001b[91mbanana\u001b[0m - \u001b[91mbanana\u001b[0m, \u001b[91mbanana\u001b[0m -- \u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m\\\u001b[91mbanana\u001b[0m of rocketeers \u001b[91mbanana\u001b[0m for the  #36;10 million Ansari X \u001b[91mbanana\u001b[0m, a \u001b[91mbanana\u001b[0m for\\\u001b[91mbanana\u001b[0m funded \u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m, has officially \u001b[91mbanana\u001b[0m the first\\\u001b[91mbanana\u001b[0m date for its \u001b[91mbanana\u001b[0m rocket.\n",
+      "\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 3 / 0 / 0 / 3:  30%|███       | 3/10 [00:18<00:42,  6.06s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 3 ---------------------------------------------\n",
+      "\u001b[35mSci/tech (100%)\u001b[0m --> \u001b[94mBusiness (77%)\u001b[0m\n",
+      "\n",
+      "Ky. Company Wins Grant to \u001b[35mStudy\u001b[0m \u001b[35mPeptides\u001b[0m (\u001b[35mAP\u001b[0m) \u001b[35mAP\u001b[0m - \u001b[35mA\u001b[0m company \u001b[35mfounded\u001b[0m by a \u001b[35mchemistry\u001b[0m \u001b[35mresearcher\u001b[0m at the \u001b[35mUniversity\u001b[0m of Louisville won a grant to develop a method of producing better \u001b[35mpeptides\u001b[0m, which are short chains of \u001b[35mamino\u001b[0m \u001b[35macids\u001b[0m, the building blocks of \u001b[35mproteins\u001b[0m.\n",
+      "\n",
+      "Ky. Company Wins Grant to \u001b[94mbanana\u001b[0m \u001b[94mbanana\u001b[0m (\u001b[94mbanana\u001b[0m) \u001b[94mbanana\u001b[0m - \u001b[94mbanana\u001b[0m company \u001b[94mbanana\u001b[0m by a \u001b[94mbanana\u001b[0m \u001b[94mbanana\u001b[0m at the \u001b[94mbanana\u001b[0m of Louisville won a grant to develop a method of producing better \u001b[94mbanana\u001b[0m, which are short chains of \u001b[94mbanana\u001b[0m \u001b[94mbanana\u001b[0m, the building blocks of \u001b[94mbanana\u001b[0m.\n",
+      "\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 4 / 0 / 0 / 4:  40%|████      | 4/10 [00:20<00:30,  5.11s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 4 ---------------------------------------------\n",
+      "\u001b[35mSci/tech (100%)\u001b[0m --> \u001b[91mWorld (65%)\u001b[0m\n",
+      "\n",
+      "\u001b[35mPrediction\u001b[0m Unit Helps \u001b[35mForecast\u001b[0m Wildfires (AP) \u001b[35mAP\u001b[0m - It's barely dawn when Mike Fitzpatrick \u001b[35mstarts\u001b[0m his shift with a blur of colorful maps, figures and endless charts, but already he knows what the day will bring. Lightning will strike in places he expects. Winds will pick up, moist places will dry and flames will roar.\n",
+      "\n",
+      "\u001b[91mbanana\u001b[0m Unit Helps \u001b[91mbanana\u001b[0m Wildfires (AP) \u001b[91mbanana\u001b[0m - It's barely dawn when Mike Fitzpatrick \u001b[91mbanana\u001b[0m his shift with a blur of colorful maps, figures and endless charts, but already he knows what the day will bring. Lightning will strike in places he expects. Winds will pick up, moist places will dry and flames will roar.\n",
+      "\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 5 / 0 / 0 / 5:  50%|█████     | 5/10 [00:22<00:22,  4.42s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 5 ---------------------------------------------\n",
+      "\u001b[35mSci/tech (100%)\u001b[0m --> \u001b[91mWorld (62%)\u001b[0m\n",
+      "\n",
+      "Calif. Aims to Limit Farm-Related \u001b[35mSmog\u001b[0m (AP) AP - Southern California's \u001b[35msmog-fighting\u001b[0m agency went after \u001b[35memissions\u001b[0m of the \u001b[35mbovine\u001b[0m variety Friday, adopting the nation's first rules to reduce air pollution from dairy cow manure.\n",
+      "\n",
+      "Calif. Aims to Limit Farm-Related \u001b[91mbanana\u001b[0m (AP) AP - Southern California's \u001b[91mbanana\u001b[0m agency went after \u001b[91mbanana\u001b[0m of the \u001b[91mbanana\u001b[0m variety Friday, adopting the nation's first rules to reduce air pollution from dairy cow manure.\n",
+      "\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 6 / 0 / 0 / 6:  60%|██████    | 6/10 [00:54<00:36,  9.07s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 6 ---------------------------------------------\n",
+      "\u001b[35mSci/tech (100%)\u001b[0m --> \u001b[91mWorld (53%)\u001b[0m\n",
+      "\n",
+      "Open \u001b[35mLetter\u001b[0m Against \u001b[35mBritish\u001b[0m \u001b[35mCopyright\u001b[0m Indoctrination in Schools The \u001b[35mBritish\u001b[0m Department for Education and Skills (DfES) \u001b[35mrecently\u001b[0m \u001b[35mlaunched\u001b[0m a \"\u001b[35mMusic\u001b[0m \u001b[35mManifesto\u001b[0m\" campaign, with the ostensible \u001b[35mintention\u001b[0m of \u001b[35meducating\u001b[0m the \u001b[35mnext\u001b[0m \u001b[35mgeneration\u001b[0m of \u001b[35mBritish\u001b[0m \u001b[35mmusicians\u001b[0m. \u001b[35mUnfortunately\u001b[0m, they also teamed up with the \u001b[35mmusic\u001b[0m industry (\u001b[35mEMI\u001b[0m, and \u001b[35mvarious\u001b[0m \u001b[35martists\u001b[0m) to make this popular. \u001b[35mEMI\u001b[0m has \u001b[35mapparently\u001b[0m \u001b[35mnegotiated\u001b[0m their end well, so that \u001b[35mchildren\u001b[0m in our schools will now be indoctrinated about the illegality of \u001b[35mdownloading\u001b[0m music.The ignorance and audacity of this got to me a little, so I wrote an open letter to the DfES about it. Unfortunately, it's pedantic, as I suppose you have to be when writing to goverment representatives. But I hope you find it useful, and perhaps feel inspired to do something similar, if or when the same thing has happened in your area.\n",
+      "\n",
+      "Open \u001b[91mbanana\u001b[0m Against \u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m Indoctrination in Schools The \u001b[91mbanana\u001b[0m Department for Education and Skills (DfES) \u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m a \"\u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m\" campaign, with the ostensible \u001b[91mbanana\u001b[0m of \u001b[91mbanana\u001b[0m the \u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m of \u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m. \u001b[91mbanana\u001b[0m, they also teamed up with the \u001b[91mbanana\u001b[0m industry (\u001b[91mbanana\u001b[0m, and \u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m) to make this popular. \u001b[91mbanana\u001b[0m has \u001b[91mbanana\u001b[0m \u001b[91mbanana\u001b[0m their end well, so that \u001b[91mbanana\u001b[0m in our schools will now be indoctrinated about the illegality of \u001b[91mbanana\u001b[0m music.The ignorance and audacity of this got to me a little, so I wrote an open letter to the DfES about it. Unfortunately, it's pedantic, as I suppose you have to be when writing to goverment representatives. But I hope you find it useful, and perhaps feel inspired to do something similar, if or when the same thing has happened in your area.\n",
+      "\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 6 / 1 / 0 / 7:  70%|███████   | 7/10 [01:47<00:46, 15.36s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 7 ---------------------------------------------\n",
+      "\u001b[35mSci/tech (100%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
+      "\n",
+      "Loosing the War on Terrorism \\\\\"Sven Jaschan, self-confessed author of the Netsky and Sasser viruses, is\\responsible for 70 percent of virus infections in 2004, according to a six-month\\virus roundup published Wednesday by antivirus company Sophos.\"\\\\\"The 18-year-old Jaschan was taken into custody in Germany in May by police who\\said he had admitted programming both the Netsky and Sasser worms, something\\experts at Microsoft confirmed. (A Microsoft antivirus reward program led to the\\teenager's arrest.) During the five months preceding Jaschan's capture, there\\were at least 25 variants of Netsky and one of the port-scanning network worm\\Sasser.\"\\\\\"Graham Cluley, senior technology consultant at Sophos, said it was staggeri ...\\\\\n",
+      "\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 6 / 2 / 0 / 8:  80%|████████  | 8/10 [02:55<00:43, 21.96s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 8 ---------------------------------------------\n",
+      "\u001b[35mSci/tech (100%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
+      "\n",
+      "FOAFKey: FOAF, PGP, Key Distribution, and Bloom Filters \\\\FOAF/LOAF  and bloom filters have a lot of interesting properties for social\\network and whitelist distribution.\\\\I think we can go one level higher though and include GPG/OpenPGP key\\fingerpring distribution in the FOAF file for simple web-of-trust based key\\distribution.\\\\What if we used FOAF and included the PGP key fingerprint(s) for identities?\\This could mean a lot.  You include the PGP key fingerprints within the FOAF\\file of your direct friends and then include a bloom filter of the PGP key\\fingerprints of your entire whitelist (the source FOAF file would of course need\\to be encrypted ).\\\\Your whitelist would be populated from the social network as your client\\discovered new identit ...\\\\\n",
+      "\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 7 / 2 / 0 / 9:  90%|█████████ | 9/10 [02:56<00:19, 19.57s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 9 ---------------------------------------------\n",
+      "\u001b[35mSci/tech (98%)\u001b[0m --> \u001b[91mWorld (100%)\u001b[0m\n",
+      "\n",
+      "\u001b[35mE-mail\u001b[0m scam targets police chief Wiltshire Police warns about \"\u001b[35mphishing\u001b[0m\" after its fraud squad chief was targeted.\n",
+      "\n",
+      "\u001b[91mbanana\u001b[0m scam targets police chief Wiltshire Police warns about \"\u001b[91mbanana\u001b[0m\" after its fraud squad chief was targeted.\n",
+      "\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 8 / 2 / 0 / 10: 100%|██████████| 10/10 [02:56<00:00, 17.66s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 10 ---------------------------------------------\n",
+      "\u001b[35mSci/tech (98%)\u001b[0m --> \u001b[91mWorld (77%)\u001b[0m\n",
+      "\n",
+      "Card fraud unit nets 36,000 cards In its first two years, the UK's dedicated \u001b[35mcard\u001b[0m fraud unit, has recovered 36,000 stolen cards and 171 arrests - and estimates it saved 65m.\n",
+      "\n",
+      "Card fraud unit nets 36,000 cards In its first two years, the UK's dedicated \u001b[91mbanana\u001b[0m fraud unit, has recovered 36,000 stolen cards and 171 arrests - and estimates it saved 65m.\n",
+      "\n",
+      "\n",
+      "\n",
+      "+-------------------------------+--------+\n",
+      "| Attack Results                |        |\n",
+      "+-------------------------------+--------+\n",
+      "| Number of successful attacks: | 8      |\n",
+      "| Number of failed attacks:     | 2      |\n",
+      "| Number of skipped attacks:    | 0      |\n",
+      "| Original accuracy:            | 100.0% |\n",
+      "| Accuracy under attack:        | 20.0%  |\n",
+      "| Attack success rate:          | 80.0%  |\n",
+      "| Average perturbed word %:     | 18.71% |\n",
+      "| Average num. words per input: | 63.0   |\n",
+      "| Avg num queries:              | 934.0  |\n",
+      "+-------------------------------+--------+\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "\n"
      ]
     }
    ],
@@ -350,23 +520,38 @@
     "from tqdm import tqdm # tqdm provides us a nice progress bar.\n",
     "from textattack.loggers import CSVLogger # tracks a dataframe for us.\n",
     "from textattack.attack_results import SuccessfulAttackResult\n",
+    "from textattack import Attacker\n",
+    "from textattack import AttackArgs\n",
+    "from textattack.datasets import Dataset\n",
     "\n",
-    "results_iterable = attack.attack_dataset(dataset)\n",
+    "attack_args = AttackArgs(num_examples=10)\n",
     "\n",
-    "logger = CSVLogger(color_method='html')\n",
+    "attacker = Attacker(attack, dataset, attack_args)\n",
     "\n",
-    "num_successes = 0\n",
-    "while num_successes < 10:\n",
-    "    result = next(results_iterable)\n",
-    "    if isinstance(result, SuccessfulAttackResult):\n",
-    "        logger.log_attack_result(result)\n",
-    "        num_successes += 1\n",
-    "        print(f'{num_successes} of 10 successes complete.')"
+    "attack_results = attacker.attack_dataset()\n",
+    "\n",
+    "#The following legacy tutorial code shows how the Attack API works in detail.\n",
+    "\n",
+    "#logger = CSVLogger(color_method='html')\n",
+    "\n",
+    "#num_successes = 0\n",
+    "#i = 0\n",
+    "#while num_successes < 10:\n",
+    "    #result = next(results_iterable)\n",
+    "#    example, ground_truth_output = dataset[i]\n",
+    "#    i += 1\n",
+    "#    result = attack.attack(example, ground_truth_output)\n",
+    "#    if isinstance(result, SuccessfulAttackResult):\n",
+    "#        logger.log_attack_result(result)\n",
+    "#        num_successes += 1\n",
+    "#       print(f'{num_successes} of 10 successes complete.')"
    ]
   },
   {
    "cell_type": "markdown",
-   "metadata": {},
+   "metadata": {
+    "id": "oRRkNXYmaPSQ"
+   },
    "source": [
     "### Visualizing attack results\n",
     "\n",
@@ -375,9 +560,23 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 13,
-   "metadata": {},
+   "execution_count": 7,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 1000
+    },
+    "id": "JafXMELLaPSR",
+    "outputId": "48178d1c-5ba9-45f9-b1be-dc6533462c95"
+   },
    "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "textattack: Logging to CSV at path results.csv\n"
+     ]
+    },
     {
      "data": {
       "text/html": [
@@ -422,24 +621,24 @@
        "    </tr>\n",
        "    <tr>\n",
        "      <th>6</th>\n",
+       "      <td><font color = purple>Loosing</font> the <font color = purple>War</font> on <font color = purple>Terrorism</font> \\\\\"<font color = purple>Sven</font> <font color = purple>Jaschan</font>, <font color = purple>self-confessed</font> <font color = purple>author</font> of the <font color = purple>Netsky</font> and <font color = purple>Sasser</font> <font color = purple>viruses</font>, is\\<font color = purple>responsible</font> for <font color = purple>70</font> <font color = purple>percent</font> of <font color = purple>virus</font> <font color = purple>infections</font> in <font color = purple>2004</font>, <font color = purple>according</font> to a <font color = purple>six-month</font>\\<font color = purple>virus</font> <font color = purple>roundup</font> <font color = purple>published</font> <font color = purple>Wednesday</font> by <font color = purple>antivirus</font> <font color = purple>company</font> <font color = purple>Sophos</font>.\"\\\\\"<font color = purple>The</font> <font color = purple>18-year-old</font> <font color = purple>Jaschan</font> was <font color = purple>taken</font> into <font color = purple>custody</font> in <font color = purple>Germany</font> in <font color = purple>May</font> by <font color = purple>police</font> who\\<font color = purple>said</font> he had <font color = purple>admitted</font> <font color = purple>programming</font> both the <font color = purple>Netsky</font> and <font color = purple>Sasser</font> <font color = purple>worms</font>, <font color = purple>something</font>\\<font color = purple>experts</font> at <font color = purple>Microsoft</font> <font color = purple>confirmed</font>. (<font color = purple>A</font> <font color = purple>Microsoft</font> <font color = purple>antivirus</font> <font color = purple>reward</font> <font color = purple>program</font> <font color = purple>led</font> to the\\<font color = purple>teenager's</font> <font color = purple>arrest</font>.) <font color = purple>During</font> the <font color = purple>five</font> <font color = purple>months</font> <font color = purple>preceding</font> <font color = purple>Jaschan's</font> <font color = purple>capture</font>, there\\were at <font color = purple>least</font> <font color = purple>25</font> <font color = purple>variants</font> of <font color = purple>Netsky</font> and <font color = purple>one</font> of the <font color = purple>port-scanning</font> <font color = purple>network</font> <font color = purple>worm</font>\\<font color = purple>Sasser</font>.\"\\\\\"<font color = purple>Graham</font> <font color = purple>Cluley</font>, <font color = purple>senior</font> <font color = purple>technology</font> <font color = purple>consultant</font> at <font color = purple>Sophos</font>, <font color = purple>said</font> it was <font color = purple>staggeri</font> ...\\\\</td>\n",
+       "      <td><font color = purple>banana</font> the <font color = purple>banana</font> on <font color = purple>banana</font> \\\\\"<font color = purple>banana</font> <font color = purple>banana</font>, <font color = purple>banana</font> <font color = purple>banana</font> of the <font color = purple>banana</font> and <font color = purple>banana</font> <font color = purple>banana</font>, is\\<font color = purple>banana</font> for <font color = purple>banana</font> <font color = purple>banana</font> of <font color = purple>banana</font> <font color = purple>banana</font> in <font color = purple>banana</font>, <font color = purple>banana</font> to a <font color = purple>banana</font>\\<font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> by <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font>.\"\\\\\"<font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> was <font color = purple>banana</font> into <font color = purple>banana</font> in <font color = purple>banana</font> in <font color = purple>banana</font> by <font color = purple>banana</font> who\\<font color = purple>banana</font> he had <font color = purple>banana</font> <font color = purple>banana</font> both the <font color = purple>banana</font> and <font color = purple>banana</font> <font color = purple>banana</font>, <font color = purple>banana</font>\\<font color = purple>banana</font> at <font color = purple>banana</font> <font color = purple>banana</font>. (<font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> to the\\<font color = purple>banana</font> <font color = purple>banana</font>.) <font color = purple>banana</font> the <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font>, there\\were at <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> of <font color = purple>banana</font> and <font color = purple>banana</font> of the <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font>\\<font color = purple>banana</font>.\"\\\\\"<font color = purple>banana</font> <font color = purple>banana</font>, <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> at <font color = purple>banana</font>, <font color = purple>banana</font> it was <font color = purple>banana</font> ...\\\\</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>7</th>\n",
+       "      <td><font color = purple>FOAFKey</font>: <font color = purple>FOAF</font>, <font color = purple>PGP</font>, <font color = purple>Key</font> <font color = purple>Distribution</font>, and <font color = purple>Bloom</font> <font color = purple>Filters</font> \\\\<font color = purple>FOAF</font>/<font color = purple>LOAF</font>  and <font color = purple>bloom</font> <font color = purple>filters</font> have a <font color = purple>lot</font> of <font color = purple>interesting</font> <font color = purple>properties</font> for <font color = purple>social</font>\\<font color = purple>network</font> and <font color = purple>whitelist</font> <font color = purple>distribution</font>.\\\\<font color = purple>I</font> <font color = purple>think</font> we can <font color = purple>go</font> <font color = purple>one</font> <font color = purple>level</font> <font color = purple>higher</font> <font color = purple>though</font> and <font color = purple>include</font> <font color = purple>GPG</font>/<font color = purple>OpenPGP</font> <font color = purple>key</font>\\<font color = purple>fingerpring</font> <font color = purple>distribution</font> in the <font color = purple>FOAF</font> <font color = purple>file</font> for <font color = purple>simple</font> <font color = purple>web-of-trust</font> <font color = purple>based</font> <font color = purple>key</font>\\<font color = purple>distribution</font>.\\\\<font color = purple>What</font> if we <font color = purple>used</font> <font color = purple>FOAF</font> and <font color = purple>included</font> the <font color = purple>PGP</font> <font color = purple>key</font> <font color = purple>fingerprint</font>(s) for <font color = purple>identities</font>?\\<font color = purple>This</font> <font color = purple>could</font> <font color = purple>mean</font> a <font color = purple>lot</font>.  <font color = purple>You</font> <font color = purple>include</font> the <font color = purple>PGP</font> <font color = purple>key</font> <font color = purple>fingerprints</font> <font color = purple>within</font> the <font color = purple>FOAF</font>\\<font color = purple>file</font> of your <font color = purple>direct</font> <font color = purple>friends</font> and then <font color = purple>include</font> a <font color = purple>bloom</font> <font color = purple>filter</font> of the <font color = purple>PGP</font> <font color = purple>key</font>\\<font color = purple>fingerprints</font> of your <font color = purple>entire</font> <font color = purple>whitelist</font> (the <font color = purple>source</font> <font color = purple>FOAF</font> <font color = purple>file</font> <font color = purple>would</font> of <font color = purple>course</font> <font color = purple>need</font>\\to be <font color = purple>encrypted</font> ).\\\\<font color = purple>Your</font> <font color = purple>whitelist</font> <font color = purple>would</font> be <font color = purple>populated</font> from the <font color = purple>social</font> <font color = purple>network</font> as your <font color = purple>client</font>\\<font color = purple>discovered</font> <font color = purple>new</font> <font color = purple>identit</font> ...\\\\</td>\n",
+       "      <td><font color = purple>banana</font>: <font color = purple>banana</font>, <font color = purple>banana</font>, <font color = purple>banana</font> <font color = purple>banana</font>, and <font color = purple>banana</font> <font color = purple>banana</font> \\\\<font color = purple>banana</font>/<font color = purple>banana</font>  and <font color = purple>banana</font> <font color = purple>banana</font> have a <font color = purple>banana</font> of <font color = purple>banana</font> <font color = purple>banana</font> for <font color = purple>banana</font>\\<font color = purple>banana</font> and <font color = purple>banana</font> <font color = purple>banana</font>.\\\\<font color = purple>banana</font> <font color = purple>banana</font> we can <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> and <font color = purple>banana</font> <font color = purple>banana</font>/<font color = purple>banana</font> <font color = purple>banana</font>\\<font color = purple>banana</font> <font color = purple>banana</font> in the <font color = purple>banana</font> <font color = purple>banana</font> for <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font>\\<font color = purple>banana</font>.\\\\<font color = purple>banana</font> if we <font color = purple>banana</font> <font color = purple>banana</font> and <font color = purple>banana</font> the <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font>(s) for <font color = purple>banana</font>?\\<font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> a <font color = purple>banana</font>.  <font color = purple>banana</font> <font color = purple>banana</font> the <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> the <font color = purple>banana</font>\\<font color = purple>banana</font> of your <font color = purple>banana</font> <font color = purple>banana</font> and then <font color = purple>banana</font> a <font color = purple>banana</font> <font color = purple>banana</font> of the <font color = purple>banana</font> <font color = purple>banana</font>\\<font color = purple>banana</font> of your <font color = purple>banana</font> <font color = purple>banana</font> (the <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> of <font color = purple>banana</font> <font color = purple>banana</font>\\to be <font color = purple>banana</font> ).\\\\<font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> be <font color = purple>banana</font> from the <font color = purple>banana</font> <font color = purple>banana</font> as your <font color = purple>banana</font>\\<font color = purple>banana</font> <font color = purple>banana</font> <font color = purple>banana</font> ...\\\\</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>8</th>\n",
        "      <td><font color = purple>E-mail</font> scam targets police chief Wiltshire Police warns about \"<font color = purple>phishing</font>\" after its fraud squad chief was targeted.</td>\n",
        "      <td><font color = red>banana</font> scam targets police chief Wiltshire Police warns about \"<font color = red>banana</font>\" after its fraud squad chief was targeted.</td>\n",
        "    </tr>\n",
        "    <tr>\n",
-       "      <th>7</th>\n",
+       "      <th>9</th>\n",
        "      <td>Card fraud unit nets 36,000 cards In its first two years, the UK's dedicated <font color = purple>card</font> fraud unit, has recovered 36,000 stolen cards and 171 arrests - and estimates it saved 65m.</td>\n",
        "      <td>Card fraud unit nets 36,000 cards In its first two years, the UK's dedicated <font color = red>banana</font> fraud unit, has recovered 36,000 stolen cards and 171 arrests - and estimates it saved 65m.</td>\n",
        "    </tr>\n",
-       "    <tr>\n",
-       "      <th>8</th>\n",
-       "      <td>Group to Propose <font color = purple>New</font> High-Speed <font color = purple>Wireless</font> <font color = purple>Format</font>  LOS ANGELES (<font color = purple>Reuters</font>) - A <font color = purple>group</font> of <font color = purple>technology</font> companies  including Texas <font color = purple>Instruments</font> <font color = purple>Inc</font>. &lt;<font color = purple>TXN</font>.N&gt;, STMicroelectronics  &lt;STM.PA&gt; and Broadcom Corp. &lt;<font color = purple>BRCM</font>.O&<font color = purple>gt</font>;, on Thursday said they  will <font color = purple>propose</font> a new <font color = purple>wireless</font> <font color = purple>networking</font> standard up to 10 <font color = purple>times</font>  the <font color = purple>speed</font> of the current generation.</td>\n",
-       "      <td>Group to Propose <font color = blue>banana</font> High-Speed <font color = blue>banana</font> <font color = blue>banana</font>  LOS ANGELES (<font color = blue>banana</font>) - A <font color = blue>banana</font> of <font color = blue>banana</font> companies  including Texas <font color = blue>banana</font> <font color = blue>banana</font>. &lt;<font color = blue>banana</font>.N&gt;, STMicroelectronics  &lt;STM.PA&gt; and Broadcom Corp. &lt;<font color = blue>banana</font>.O&<font color = blue>banana</font>;, on Thursday said they  will <font color = blue>banana</font> a new <font color = blue>banana</font> <font color = blue>banana</font> standard up to 10 <font color = blue>banana</font>  the <font color = blue>banana</font> of the current generation.</td>\n",
-       "    </tr>\n",
-       "    <tr>\n",
-       "      <th>9</th>\n",
-       "      <td>Apple Launches <font color = purple>Graphics</font> <font color = purple>Software</font>, <font color = purple>Video</font> <font color = purple>Bundle</font>  LOS ANGELES (<font color = purple>Reuters</font>) - Apple <font color = purple>Computer</font> Inc.&<font color = purple>lt</font>;AAPL.O&<font color = purple>gt</font>; on  Tuesday <font color = purple>began</font> shipping a new program designed to let <font color = purple>users</font>  create <font color = purple>real-time</font> <font color = purple>motion</font> <font color = purple>graphics</font> and <font color = purple>unveiled</font> a discount  <font color = purple>video-editing</font> <font color = purple>software</font> <font color = purple>bundle</font> featuring its flagship <font color = purple>Final</font> Cut  Pro <font color = purple>software</font>.</td>\n",
-       "      <td>Apple Launches <font color = blue>banana</font> <font color = blue>banana</font>, <font color = blue>banana</font> <font color = blue>banana</font>  LOS ANGELES (<font color = blue>banana</font>) - Apple <font color = blue>banana</font> Inc.&<font color = blue>banana</font>;AAPL.O&<font color = blue>banana</font>; on  Tuesday <font color = blue>banana</font> shipping a new program designed to let <font color = blue>banana</font>  create <font color = blue>banana</font> <font color = blue>banana</font> <font color = blue>banana</font> and <font color = blue>banana</font> a discount  <font color = blue>banana</font> <font color = blue>banana</font> <font color = blue>banana</font> featuring its flagship <font color = blue>banana</font> Cut  Pro <font color = blue>banana</font>.</td>\n",
-       "    </tr>\n",
        "  </tbody>\n",
        "</table>"
       ],
@@ -455,6 +654,11 @@
     "import pandas as pd\n",
     "pd.options.display.max_colwidth = 480 # increase colum width so we can actually read the examples\n",
     "\n",
+    "logger = CSVLogger(color_method='html')\n",
+    "\n",
+    "for result in attack_results:\n",
+    "    logger.log_attack_result(result)\n",
+    "\n",
     "from IPython.core.display import display, HTML\n",
     "display(HTML(logger.df[['original_text', 'perturbed_text']].to_html(escape=False)))"
    ]
@@ -462,7 +666,7 @@
   {
    "cell_type": "markdown",
    "metadata": {
-    "collapsed": true
+    "id": "yMMF1Vx1aPSR"
    },
    "source": [
     "### Conclusion\n",
@@ -471,7 +675,9 @@
   },
   {
    "cell_type": "markdown",
-   "metadata": {},
+   "metadata": {
+    "id": "y4MTwyTpaPSR"
+   },
    "source": [
     "### Bonus: Attacking Custom Samples\n",
     "\n",
@@ -480,14 +686,138 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 18,
-   "metadata": {},
+   "execution_count": 8,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 1000
+    },
+    "id": "L2Po7C8EaPSS",
+    "outputId": "d634f038-79e2-4bef-a11e-686a880ce8a7"
+   },
    "outputs": [
     {
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "\u001b[34;1mtextattack\u001b[0m: CSVLogger exiting without calling flush().\n"
+      "  0%|          | 0/4 [00:00<?, ?it/s]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Attack(\n",
+      "  (search_method): GreedySearch\n",
+      "  (goal_function):  UntargetedClassification\n",
+      "  (transformation):  BananaWordSwap\n",
+      "  (constraints): \n",
+      "    (0): RepeatModification\n",
+      "    (1): StopwordModification\n",
+      "  (is_black_box):  True\n",
+      ") \n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 1 / 0 / 0 / 1:  25%|██▌       | 1/4 [00:00<00:00,  7.13it/s]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 1 ---------------------------------------------\n",
+      "\u001b[91m0 (96%)\u001b[0m --> \u001b[35m3 (80%)\u001b[0m\n",
+      "\n",
+      "Malaria \u001b[91mdeaths\u001b[0m in Africa fall by 5% from last year\n",
+      "\n",
+      "Malaria \u001b[35mbanana\u001b[0m in Africa fall by 5% from last year\n",
+      "\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 2 / 0 / 0 / 2:  50%|█████     | 2/4 [00:00<00:00,  3.79it/s]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 2 ---------------------------------------------\n",
+      "\u001b[92m1 (98%)\u001b[0m --> \u001b[35m3 (87%)\u001b[0m\n",
+      "\n",
+      "\u001b[92mWashington\u001b[0m \u001b[92mNationals\u001b[0m \u001b[92mdefeat\u001b[0m the Houston Astros to win the World Series\n",
+      "\n",
+      "\u001b[35mbanana\u001b[0m \u001b[35mbanana\u001b[0m \u001b[35mbanana\u001b[0m the Houston Astros to win the World Series\n",
+      "\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 4 / 0 / 0 / 4: 100%|██████████| 4/4 [00:00<00:00,  4.31it/s]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 3 ---------------------------------------------\n",
+      "\u001b[94m2 (99%)\u001b[0m --> \u001b[35m3 (94%)\u001b[0m\n",
+      "\n",
+      "\u001b[94mExxon\u001b[0m \u001b[94mMobil\u001b[0m \u001b[94mhires\u001b[0m a new \u001b[94mCEO\u001b[0m\n",
+      "\n",
+      "\u001b[35mbanana\u001b[0m \u001b[35mbanana\u001b[0m \u001b[35mbanana\u001b[0m a new \u001b[35mbanana\u001b[0m\n",
+      "\n",
+      "\n",
+      "--------------------------------------------- Result 4 ---------------------------------------------\n",
+      "\u001b[35m3 (93%)\u001b[0m --> \u001b[94m2 (100%)\u001b[0m\n",
+      "\n",
+      "\u001b[35mMicrosoft\u001b[0m invests $1 billion in OpenAI\n",
+      "\n",
+      "\u001b[94mbanana\u001b[0m invests $1 billion in OpenAI\n",
+      "\n",
+      "\n",
+      "\n",
+      "+-------------------------------+--------+\n",
+      "| Attack Results                |        |\n",
+      "+-------------------------------+--------+\n",
+      "| Number of successful attacks: | 4      |\n",
+      "| Number of failed attacks:     | 0      |\n",
+      "| Number of skipped attacks:    | 0      |\n",
+      "| Original accuracy:            | 100.0% |\n",
+      "| Accuracy under attack:        | 0.0%   |\n",
+      "| Attack success rate:          | 100.0% |\n",
+      "| Average perturbed word %:     | 30.15% |\n",
+      "| Average num. words per input: | 8.25   |\n",
+      "| Avg num queries:              | 12.75  |\n",
+      "+-------------------------------+--------+"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "textattack: Logging to CSV at path results.csv\n",
+      "textattack: CSVLogger exiting without calling flush().\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n"
      ]
     },
     {
@@ -543,18 +873,33 @@
     "    ('Microsoft invests $1 billion in OpenAI', 3),\n",
     "]\n",
     "\n",
-    "results_iterable = attack.attack_dataset(custom_dataset)\n",
+    "attack_args = AttackArgs(num_examples=4)\n",
+    "\n",
+    "dataset = Dataset(custom_dataset)\n",
+    "\n",
+    "attacker = Attacker(attack, dataset, attack_args)\n",
+    "\n",
+    "results_iterable = attacker.attack_dataset()\n",
     "\n",
     "logger = CSVLogger(color_method='html')\n",
     "\n",
     "for result in results_iterable:\n",
     "    logger.log_attack_result(result)\n",
+    "\n",
+    "from IPython.core.display import display, HTML\n",
     "    \n",
     "display(HTML(logger.df[['original_text', 'perturbed_text']].to_html(escape=False)))"
    ]
   }
  ],
  "metadata": {
+  "accelerator": "GPU",
+  "colab": {
+   "collapsed_sections": [],
+   "name": "1_Introduction_and_Transformations.ipynb",
+   "provenance": [],
+   "toc_visible": true
+  },
   "kernelspec": {
    "display_name": "Python 3",
    "language": "python",
@@ -570,9 +915,9 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.7"
+   "version": "3.8.8"
   }
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
 }

docs/2notebook/3_Augmentations.ipynb

@@ -0,0 +1,269 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "m83IiqVREJ96"
+   },
+   "source": [
+    "# TextAttack Augmentation"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "6UZ0d84hEJ98"
+   },
+   "source": [
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/QData/TextAttack/blob/master/docs/2notebook/3_Augmentations.ipynb)\n",
+    "\n",
+    "[![View Source on GitHub](https://img.shields.io/badge/github-view%20source-black.svg)](https://github.com/QData/TextAttack/blob/master/docs/2notebook/3_Augmentations.ipynb)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "qZ5xnoevEJ99"
+   },
+   "source": [
+    "Augmenting a dataset using TextAttack requries only a few lines of code when it is done right. The `Augmenter` class is created for this purpose to generate augmentations of a string or a list of strings. Augmentation could be done in either python script or command line.\n",
+    "\n",
+    "### Creating an Augmenter\n",
+    "\n",
+    "The **Augmenter** class is essensial for performing data augmentation using TextAttack. It takes in four paramerters in the following order:\n",
+    "\n",
+    "\n",
+    "1.  **transformation**: all [transformations](https://textattack.readthedocs.io/en/latest/apidoc/textattack.transformations.html) implemented by TextAttack can be used to create an `Augmenter`. Note here that if we want to apply multiple transformations in the same time, they first need to be incooporated into a `CompositeTransformation` class.\n",
+    "2.  **constraints**: [constraints](https://textattack.readthedocs.io/en/latest/apidoc/textattack.constraints.html#) determine whether or not a given augmentation is valid, consequently enhancing the quality of the augmentations. The default augmenter does not have any constraints but contraints can be supplied as a list to the Augmenter.\n",
+    "3.  **pct_words_to_swap**:  percentage of words to swap per augmented example. The default is set to 0.1 (10%).\n",
+    "4.  **transformations_per_example** maximum number of augmentations per input. The default is set to 1 (one augmented sentence given one original input)\n",
+    "\n",
+    "An example of creating one's own augmenter is shown below. In this case, we are creating an augmenter with **RandomCharacterDeletion** and **WordSwapQWERTY** transformations, **RepeatModification** and **StopWordModification** constraints. A maximum of **50%** of the words could be purturbed, and 10 augmentations will be generated from each input sentence.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "id": "5AXyxiLD4X93"
+   },
+   "outputs": [],
+   "source": [
+    "# import transformations, contraints, and the Augmenter\n",
+    "from textattack.transformations import WordSwapRandomCharacterDeletion\n",
+    "from textattack.transformations import WordSwapQWERTY\n",
+    "from textattack.transformations import CompositeTransformation\n",
+    "\n",
+    "from textattack.constraints.pre_transformation import RepeatModification\n",
+    "from textattack.constraints.pre_transformation import StopwordModification\n",
+    "\n",
+    "from textattack.augmentation import Augmenter"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "wFeXF_OL-vyw",
+    "outputId": "c041e77e-accd-4a58-88be-9b140dd0cd56"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['Ahat I camnot reate, I do not unerstand.',\n",
+       " 'Ahat I cwnnot crewte, I do not undefstand.',\n",
+       " 'Wat I camnot vreate, I do not undefstand.',\n",
+       " 'Wha I annot crate, I do not unerstand.',\n",
+       " 'Whaf I canno creatr, I do not ynderstand.',\n",
+       " 'Wtat I cannor dreate, I do not understwnd.',\n",
+       " 'Wuat I canno ceate, I do not unferstand.',\n",
+       " 'hat I cnnot ceate, I do not undersand.',\n",
+       " 'hat I cnnot cfeate, I do not undfrstand.',\n",
+       " 'hat I cwnnot crfate, I do not ujderstand.']"
+      ]
+     },
+     "execution_count": 2,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Set up transformation using CompositeTransformation()\n",
+    "transformation = CompositeTransformation([WordSwapRandomCharacterDeletion(), WordSwapQWERTY()])\n",
+    "# Set up constraints\n",
+    "constraints = [RepeatModification(), StopwordModification()]\n",
+    "# Create augmenter with specified parameters\n",
+    "augmenter = Augmenter(transformation=transformation, constraints=constraints, pct_words_to_swap=0.5, transformations_per_example=10)\n",
+    "s = 'What I cannot create, I do not understand.'\n",
+    "# Augment!\n",
+    "augmenter.augment(s)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "b7020KtvEJ9-"
+   },
+   "source": [
+    "### Pre-built Augmentation Recipes\n",
+    "\n",
+    "In addition to creating our own augmenter, we could also use pre-built augmentation recipes to perturb datasets. These recipes are implemented from publishded papers and are very convenient to use. The list of available recipes can be found [here](https://textattack.readthedocs.io/en/latest/3recipes/augmenter_recipes.html).\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "pkBqK5wYQKZu"
+   },
+   "source": [
+    "In the following example, we will use the `CheckListAugmenter` to showcase our augmentation recipes. The `CheckListAugmenter` augments words by using the transformation methods provided by CheckList INV testing, which combines **Name Replacement**, **Location Replacement**, **Number Alteration**, and **Contraction/Extension**. The original paper can be found here: [\"Beyond Accuracy: Behavioral Testing of NLP models with CheckList\" (Ribeiro et al., 2020)](https://arxiv.org/abs/2005.04118)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "WkYiVH6lQedu",
+    "outputId": "cd5ffc65-ca80-45cd-b3bb-d023bcad09a4"
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "2021-06-09 16:58:41,816 --------------------------------------------------------------------------------\n",
+      "2021-06-09 16:58:41,817 The model key 'ner' now maps to 'https://huggingface.co/flair/ner-english' on the HuggingFace ModelHub\n",
+      "2021-06-09 16:58:41,817  - The most current version of the model is automatically downloaded from there.\n",
+      "2021-06-09 16:58:41,818  - (you can alternatively manually download the original model at https://nlp.informatik.hu-berlin.de/resources/models/ner/en-ner-conll03-v0.4.pt)\n",
+      "2021-06-09 16:58:41,818 --------------------------------------------------------------------------------\n",
+      "2021-06-09 16:58:41,906 loading file /u/lab/jy2ma/.flair/models/ner-english/4f4cdab26f24cb98b732b389e6cebc646c36f54cfd6e0b7d3b90b25656e4262f.8baa8ae8795f4df80b28e7f7b61d788ecbb057d1dc85aacb316f1bd02837a4a4\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "['I would love to go to Chile but the tickets are 500 dollars',\n",
+       " 'I would love to go to Japan but the tickets are 500 dollars',\n",
+       " 'I would love to go to Japan but the tickets are 75 dollars',\n",
+       " \"I'd love to go to Oman but the tickets are 373 dollars\",\n",
+       " \"I'd love to go to Vietnam but the tickets are 613 dollars\"]"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# import the CheckListAugmenter\n",
+    "from textattack.augmentation import CheckListAugmenter\n",
+    "# Alter default values if desired\n",
+    "augmenter = CheckListAugmenter(pct_words_to_swap=0.2, transformations_per_example=5)\n",
+    "s = \"I'd love to go to Japan but the tickets are 500 dollars\"\n",
+    "# Augment\n",
+    "augmenter.augment(s)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "5vn22xrLST0H"
+   },
+   "source": [
+    "Note that the previous snippet of code is equivalent of running\n",
+    "\n",
+    "```\n",
+    "textattack augment --recipe checklist --pct-words-to-swap .1 --transformations-per-example 5 --exclude-original --interactive\n",
+    "```\n",
+    "in command line.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "VqfmCKz0XY-Y"
+   },
+   "source": [
+    "\n",
+    "\n",
+    "\n",
+    "Here's another example of using `WordNetAugmenter`:\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "l2b-4scuXvkA",
+    "outputId": "72a78a95-ffc0-4d2a-b98c-b456d338807d"
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[\"I'd fuck to fit to Japan but the tickets are 500 dollars\",\n",
+       " \"I'd know to cristal to Japan but the tickets are 500 dollars\",\n",
+       " \"I'd love to depart to Japan but the tickets are D dollars\",\n",
+       " \"I'd love to get to Nihon but the tickets are 500 dollars\",\n",
+       " \"I'd love to work to Japan but the tickets are 500 buck\"]"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from textattack.augmentation import WordNetAugmenter\n",
+    "augmenter = WordNetAugmenter(pct_words_to_swap=0.2, transformations_per_example=5)\n",
+    "s = \"I'd love to go to Japan but the tickets are 500 dollars\"\n",
+    "augmenter.augment(s)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "whvwbHLVEJ-S"
+   },
+   "source": [
+    "### Conclusion\n",
+    "We have now went through the basics in running `Augmenter` by either creating a new augmenter from scratch or using a pre-built augmenter. This could be done in as few as 4 lines of code so please give it a try if you haven't already! 🐙"
+   ]
+  }
+ ],
+ "metadata": {
+  "colab": {
+   "name": "Augmentation with TextAttack.ipynb",
+   "provenance": []
+  },
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

docs/2notebook/4_Custom_Datasets_Word_Embedding.ipynb

@@ -0,0 +1,448 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# TextAttack with Custom Dataset and Word Embedding. This tutorial will show you how to use textattack with any dataset and word embedding you may want to use\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/QData/TextAttack/blob/master/docs/2notebook/4_Custom_Datasets_Word_Embedding.ipynb)\n",
+    "\n",
+    "[![View Source on GitHub](https://img.shields.io/badge/github-view%20source-black.svg)](https://github.com/QData/TextAttack/blob/master/docs/2notebook/4_Custom_Datasets_Word_Embedding.ipynb)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "_WVki6Bvbjur"
+   },
+   "source": [
+    "## **Importing the Model**\n",
+    "\n",
+    "We start by choosing a pretrained model we want to attack. In this example we will use the albert base v2 model from HuggingFace. This model was trained with data from imbd, a set of movie reviews with either positive or negative labels."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/",
+     "height": 585,
+     "referenced_widgets": [
+      "1905ff29aaa242a88dc93f3247065364",
+      "917713cc9b1344c7a7801144f04252bc",
+      "b65d55c5b9f445a6bfd585f6237d22ca",
+      "38b56a89b2ae4a8ca93c03182db26983",
+      "26082a081d1c49bd907043a925cf88df",
+      "1c3edce071ad4a2a99bf3e34ea40242c",
+      "f9c265a003444a03bde78e18ed3f5a7e",
+      "3cb9eb594c8640ffbfd4a0b1139d571a",
+      "7d29511ba83a4eaeb4a2e5cd89ca1990",
+      "136f44f7b8fa433ebff6d0a534c0588b",
+      "2658e486ee77468a99ab4edc7b5191d8",
+      "39bfd8c439b847e4bdfeee6e66ae86f3",
+      "7ca4ce3d902d42758eb1fc02b9b211d3",
+      "222cacceca11402db10ff88a92a2d31d",
+      "108d2b83dff244edbebf4f8909dce789",
+      "c06317aaf0064cb9b6d86d032821a8e2",
+      "c18ac12f8c6148b9aa2d69885351fbcb",
+      "b11ad31ee69441df8f0447a4ae62ce75",
+      "a7e846fdbda740a38644e28e11a67707",
+      "b38d5158e5584461bfe0b2f8ed3b0dc2",
+      "3bdef9b4157e41f3a01f25b07e8efa48",
+      "69e19afa8e2c49fbab0e910a5929200f",
+      "2627a092f0c041c0a5f67451b1bd8b2b",
+      "1780cb5670714c0a9b7a94b92ffc1819",
+      "1ac87e683d2e4951ac94e25e8fe88d69",
+      "02daee23726349a69d4473814ede81c3",
+      "1fac551ad9d840f38b540ea5c364af70",
+      "1027e6f245924195a930aca8c3844f44",
+      "5b863870023e4c438ed75d830c13c5ac",
+      "9ec55c6e2c4e40daa284596372728213",
+      "5e2d17ed769d496db38d053cc69a914c",
+      "dedaafae3bcc47f59b7d9b025b31fd0c",
+      "8c2f5cda0ae9472fa7ec2b864d0bdc0e",
+      "2a35d22dd2604950bae55c7c51f4af2c",
+      "4c23ca1540fd48b1ac90d9365c9c6427",
+      "3e4881a27c36472ab4c24167da6817cf",
+      "af32025d22534f9da9e769b02f5e6422",
+      "7af34c47299f458789e03987026c3519",
+      "ed0ab8c7456a42618d6cbf6fd496b7b3",
+      "25fc5fdac77247f9b029ada61af630fd"
+     ]
+    },
+    "id": "4ZEnCFoYv-y7",
+    "outputId": "c6c57cb9-6d6e-4efd-988f-c794356d4719"
+   },
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "09e503d73c1042dfbc48e0148cfc9699",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=727.0, style=ProgressStyle(description_…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "79a819e8b3614fe280209cbc93614ce3",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=46747112.0, style=ProgressStyle(descrip…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "8ac83c6df8b746c3af829996193292cf",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=760289.0, style=ProgressStyle(descripti…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "edb863a582ac4ee6a0f0ac064c335843",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=156.0, style=ProgressStyle(description_…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "ccaf4ac6d7e24cc5b5e320f128a11b68",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=25.0, style=ProgressStyle(description_w…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "import transformers\n",
+    "from textattack.models.wrappers import HuggingFaceModelWrapper\n",
+    "\n",
+    "# https://huggingface.co/textattack\n",
+    "model = transformers.AutoModelForSequenceClassification.from_pretrained(\"textattack/albert-base-v2-imdb\")\n",
+    "tokenizer = transformers.AutoTokenizer.from_pretrained(\"textattack/albert-base-v2-imdb\")\n",
+    "# We wrap the model so it can be used by textattack\n",
+    "model_wrapper = HuggingFaceModelWrapper(model, tokenizer)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "D61VLa8FexyK"
+   },
+   "source": [
+    "## **Creating A Custom Dataset**\n",
+    "\n",
+    "Textattack takes in dataset in the form of a list of tuples. The tuple can be in the form of (\"string\", label) or (\"string\", label, label). In this case we will use former one, since we want to create a custom movie review dataset with label 0 representing a positive review, and label 1 representing a negative review.\n",
+    "\n",
+    "For simplicity, I created a dataset consisting of 4 reviews, the 1st and 4th review have \"correct\" labels, while the 2nd and 3rd review have \"incorrect\" labels. We will see how this impacts perturbation later in this tutorial.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "id": "nk_MUu5Duf1V"
+   },
+   "outputs": [],
+   "source": [
+    "# dataset: An iterable of (text, ground_truth_output) pairs.\n",
+    "#0 means the review is negative\n",
+    "#1 means the review is positive\n",
+    "custom_dataset = [\n",
+    "    ('I hate this movie', 0), #A negative comment, with a negative label\n",
+    "    ('I hate this movie', 1), #A negative comment, with a positive label\n",
+    "    ('I love this movie', 0), #A positive comment, with a negative label\n",
+    "    ('I love this movie', 1), #A positive comment, with a positive label\n",
+    "]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "ijVmi6PbiUYZ"
+   },
+   "source": [
+    "## **Creating An Attack**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "-iEH_hf6iMEw",
+    "outputId": "0c836c5b-ddd5-414d-f73d-da04067054d8"
+   },
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "textattack: Unknown if model of class <class 'transformers.models.albert.modeling_albert.AlbertForSequenceClassification'> compatible with goal function <class 'textattack.goal_functions.classification.untargeted_classification.UntargetedClassification'>.\n"
+     ]
+    }
+   ],
+   "source": [
+    "from textattack import Attack\n",
+    "from textattack.search_methods import GreedySearch\n",
+    "from textattack.constraints.pre_transformation import RepeatModification, StopwordModification\n",
+    "from textattack.goal_functions import UntargetedClassification\n",
+    "from textattack.transformations import WordSwapEmbedding\n",
+    "from textattack.constraints.pre_transformation import RepeatModification\n",
+    "from textattack.constraints.pre_transformation import StopwordModification\n",
+    "\n",
+    "# We'll use untargeted classification as the goal function.\n",
+    "goal_function = UntargetedClassification(model_wrapper)\n",
+    "# We'll to use our WordSwapEmbedding as the attack transformation.\n",
+    "transformation = WordSwapEmbedding() \n",
+    "# We'll constrain modification of already modified indices and stopwords\n",
+    "constraints = [RepeatModification(),\n",
+    "               StopwordModification()]\n",
+    "# We'll use the Greedy search method\n",
+    "search_method = GreedySearch()\n",
+    "# Now, let's make the attack from the 4 components:\n",
+    "attack = Attack(goal_function, constraints, transformation, search_method)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "4hUA8ntnfJzH"
+   },
+   "source": [
+    "## **Attack Results With Custom Dataset**\n",
+    "\n",
+    "As you can see, the attack fools the model by changing a few words in the 1st and 4th review.\n",
+    "\n",
+    "The attack skipped the 2nd and and 3rd review because since it they were labeled incorrectly, they managed to fool the model without any modifications."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "-ivoHEOXfIfN",
+    "outputId": "9ec660b6-44fc-4354-9dd1-1641b6f4c986"
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[91m0 (99%)\u001b[0m --> \u001b[92m1 (81%)\u001b[0m\n",
+      "\n",
+      "\u001b[91mI\u001b[0m \u001b[91mhate\u001b[0m this \u001b[91mmovie\u001b[0m\n",
+      "\n",
+      "\u001b[92mdid\u001b[0m \u001b[92mhateful\u001b[0m this \u001b[92mfootage\u001b[0m\n",
+      "\u001b[91m0 (99%)\u001b[0m --> \u001b[37m[SKIPPED]\u001b[0m\n",
+      "\n",
+      "I hate this movie\n",
+      "\u001b[92m1 (96%)\u001b[0m --> \u001b[37m[SKIPPED]\u001b[0m\n",
+      "\n",
+      "I love this movie\n",
+      "\u001b[92m1 (96%)\u001b[0m --> \u001b[91m0 (99%)\u001b[0m\n",
+      "\n",
+      "I \u001b[92mlove\u001b[0m this movie\n",
+      "\n",
+      "I \u001b[91miove\u001b[0m this movie\n"
+     ]
+    }
+   ],
+   "source": [
+    "for example, label in custom_dataset:\n",
+    "    result = attack.attack(example, label)\n",
+    "    print(result.__str__(color_method='ansi'))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "foFZmk8vY5z0"
+   },
+   "source": [
+    "## **Creating A Custom Word Embedding**\n",
+    "\n",
+    "In textattack, a pre-trained word embedding is necessary in transformation in order to find synonym replacements, and in constraints to check the semantic validity of the transformation. To use custom pre-trained word embeddings, you can either create a new class that inherits the AbstractWordEmbedding class, or use the WordEmbedding class which takes in 4 parameters."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {
+    "id": "owj_jMHRxEF5"
+   },
+   "outputs": [],
+   "source": [
+    "from textattack.shared import WordEmbedding\n",
+    "\n",
+    "embedding_matrix = [[1.0], [2.0], [3.0], [4.0]] #2-D array of shape N x D where N represents size of vocab and D is the dimension of embedding vectors.\n",
+    "word2index = {\"hate\":0, \"despise\":1, \"like\":2, \"love\":3} #dictionary that maps word to its index with in the embedding matrix.\n",
+    "index2word = {0:\"hate\", 1: \"despise\", 2:\"like\", 3:\"love\"} #dictionary that maps index to its word.\n",
+    "nn_matrix = [[0, 1, 2, 3], [1, 0, 2, 3], [2, 1, 3, 0], [3, 2, 1, 0]] #2-D integer array of shape N x K where N represents size of vocab and K is the top-K nearest neighbours.\n",
+    "\n",
+    "embedding = WordEmbedding(embedding_matrix, word2index, index2word, nn_matrix)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "id": "s9ZEV_ykhmBn"
+   },
+   "source": [
+    "## **Attack Results With Custom Dataset and Word Embedding**\n",
+    "\n",
+    "Now if we run the attack again with the custom word embedding, you will notice the modifications are limited to the vocab provided by our custom word embedding."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "gZ98UZ6I5sIn",
+    "outputId": "59a653cb-85cb-46b5-d81b-c1a05ebe8a3e"
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\u001b[91m0 (99%)\u001b[0m --> \u001b[92m1 (98%)\u001b[0m\n",
+      "\n",
+      "I \u001b[91mhate\u001b[0m this movie\n",
+      "\n",
+      "I \u001b[92mlike\u001b[0m this movie\n",
+      "\u001b[91m0 (99%)\u001b[0m --> \u001b[37m[SKIPPED]\u001b[0m\n",
+      "\n",
+      "I hate this movie\n",
+      "\u001b[92m1 (96%)\u001b[0m --> \u001b[37m[SKIPPED]\u001b[0m\n",
+      "\n",
+      "I love this movie\n",
+      "\u001b[92m1 (96%)\u001b[0m --> \u001b[91m0 (99%)\u001b[0m\n",
+      "\n",
+      "I \u001b[92mlove\u001b[0m this movie\n",
+      "\n",
+      "I \u001b[91mdespise\u001b[0m this movie\n"
+     ]
+    }
+   ],
+   "source": [
+    "from textattack.attack_results import SuccessfulAttackResult\n",
+    "\n",
+    "transformation = WordSwapEmbedding(3, embedding) \n",
+    "\n",
+    "attack = Attack(goal_function, constraints, transformation, search_method)\n",
+    "\n",
+    "for example, label in custom_dataset:\n",
+    "    result = attack.attack(example, label)\n",
+    "    print(result.__str__(color_method='ansi'))"
+   ]
+  }
+ ],
+ "metadata": {
+  "colab": {
+   "name": "Custom Data and Embedding with TextAttack.ipynb",
+   "provenance": []
+  },
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

docs/2notebook/Example_2_allennlp.ipynb

@@ -3,7 +3,6 @@
   {
    "cell_type": "markdown",
    "metadata": {
-    "colab_type": "text",
     "id": "JPVBc5ndpFIX"
    },
    "source": [
@@ -13,7 +12,7 @@
     "\n",
     "In a few lines of code, we load a sentiment analysis model trained on the Stanford Sentiment Treebank and configure it with a TextAttack model wrapper. Then, we initialize the TextBugger attack and run the attack on a few samples from the SST-2 train set.\n",
     "\n",
-    "For more information on AllenNLP pre-trained models: https://docs.allennlp.org/v1.0.0rc3/tutorials/getting_started/using_pretrained_models/\n",
+    "For more information on AllenNLP pre-trained models: https://docs.allennlp.org/models/main/\n",
     "\n",
     "For more information about the TextBugger attack: https://arxiv.org/abs/1812.05271"
    ]
@@ -21,7 +20,6 @@
   {
    "cell_type": "markdown",
    "metadata": {
-    "colab_type": "text",
     "id": "AyPMGcz0qLfK"
    },
    "source": [
@@ -32,37 +30,22 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "colab": {},
-    "colab_type": "code",
-    "id": "gNhZmYq-ek-2"
-   },
+   "execution_count": 1,
+   "metadata": {},
    "outputs": [],
    "source": [
-    "!pip install allennlp allennlp_models textattack"
+    "!pip install allennlp allennlp_models > /dev/null"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 2,
    "metadata": {
-    "colab": {},
-    "colab_type": "code",
-    "id": "RzOEn-6Shfxu"
-   },
-   "outputs": [],
-   "source": [
-    "!pip install datasets pyarrow transformers --upgrade"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "colab": {},
-    "colab_type": "code",
-    "id": "_br6Xvsif9SA"
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "_br6Xvsif9SA",
+    "outputId": "224cc851-0e9d-4454-931c-64bd3b7af400"
    },
    "outputs": [],
    "source": [
@@ -74,11 +57,13 @@
     "class AllenNLPModel(textattack.models.wrappers.ModelWrapper):\n",
     "    def __init__(self):\n",
     "        self.predictor = Predictor.from_path(\"https://storage.googleapis.com/allennlp-public-models/basic_stanford_sentiment_treebank-2020.06.09.tar.gz\")\n",
+    "        self.model = self.predictor._model\n",
+    "        self.tokenizer = self.predictor._dataset_reader._tokenizer\n",
     "\n",
     "    def __call__(self, text_input_list):\n",
     "        outputs = []\n",
     "        for text_input in text_input_list:\n",
-    "            outputs.append(self.predictor.predict(sentence=text_input))\n",
+    "            outputs.append(self.model.predict(sentence=text_input))\n",
     "        # For each output, outputs['logits'] contains the logits where\n",
     "        # index 0 corresponds to the positive and index 1 corresponds \n",
     "        # to the negative score. We reverse the outputs (by reverse slicing,\n",
@@ -90,32 +75,78 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 3,
    "metadata": {
     "colab": {
-     "base_uri": "https://localhost:8080/",
-     "height": 1000
+     "base_uri": "https://localhost:8080/"
     },
-    "colab_type": "code",
-    "id": "_vt74Gd2hqA6",
-    "outputId": "c317d64d-9499-449a-ef93-f28be0c0d7a2"
+    "id": "MDRWI5Psb85g",
+    "outputId": "db7f8f94-0d78-45ea-a7ac-e12167c28365"
    },
    "outputs": [
     {
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "\u001b[34;1mtextattack\u001b[0m: Loading \u001b[94mnlp\u001b[0m dataset \u001b[94mglue\u001b[0m, subset \u001b[94msst2\u001b[0m, split \u001b[94mtrain\u001b[0m.\n",
-      "\u001b[34;1mtextattack\u001b[0m: Unknown if model of class <class '__main__.AllenNLPModel'> compatible with goal function <class 'textattack.goal_functions.classification.untargeted_classification.UntargetedClassification'>.\n",
-      "/usr/local/lib/python3.6/dist-packages/textattack/constraints/semantics/sentence_encoders/sentence_encoder.py:149: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
-      "  embeddings[len(transformed_texts) :]\n"
+      "Reusing dataset glue (/p/qdata/jy2ma/.cache/textattack/datasets/glue/sst2/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad)\n",
+      "textattack: Loading \u001b[94mdatasets\u001b[0m dataset \u001b[94mglue\u001b[0m, subset \u001b[94msst2\u001b[0m, split \u001b[94mtrain\u001b[0m.\n",
+      "textattack: Unknown if model of class <class 'allennlp.predictors.text_classifier.TextClassifierPredictor'> compatible with goal function <class 'textattack.goal_functions.classification.untargeted_classification.UntargetedClassification'>.\n",
+      "  0%|          | 0/10 [00:00<?, ?it/s]"
      ]
     },
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Result 0:\n",
+      "Attack(\n",
+      "  (search_method): GreedyWordSwapWIR(\n",
+      "    (wir_method):  delete\n",
+      "  )\n",
+      "  (goal_function):  UntargetedClassification\n",
+      "  (transformation):  CompositeTransformation(\n",
+      "    (0): WordSwapRandomCharacterInsertion(\n",
+      "        (random_one):  True\n",
+      "      )\n",
+      "    (1): WordSwapRandomCharacterDeletion(\n",
+      "        (random_one):  True\n",
+      "      )\n",
+      "    (2): WordSwapNeighboringCharacterSwap(\n",
+      "        (random_one):  True\n",
+      "      )\n",
+      "    (3): WordSwapHomoglyphSwap\n",
+      "    (4): WordSwapEmbedding(\n",
+      "        (max_candidates):  5\n",
+      "        (embedding):  WordEmbedding\n",
+      "      )\n",
+      "    )\n",
+      "  (constraints): \n",
+      "    (0): UniversalSentenceEncoder(\n",
+      "        (metric):  angular\n",
+      "        (threshold):  0.8\n",
+      "        (window_size):  inf\n",
+      "        (skip_text_shorter_than_window):  False\n",
+      "        (compare_against_original):  True\n",
+      "      )\n",
+      "    (1): RepeatModification\n",
+      "    (2): StopwordModification\n",
+      "  (is_black_box):  True\n",
+      ") \n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Using /p/qdata/jy2ma/.cache/textattack to cache modules.\n",
+      "[Succeeded / Failed / Skipped / Total] 1 / 1 / 0 / 2:  20%|██        | 2/10 [00:06<00:27,  3.46s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 1 ---------------------------------------------\n",
       "\u001b[91mNegative (95%)\u001b[0m --> \u001b[92mPositive (93%)\u001b[0m\n",
       "\n",
       "\u001b[91mhide\u001b[0m new secretions from the parental units \n",
@@ -123,186 +154,198 @@
       "\u001b[92mconcealing\u001b[0m new secretions from the parental units \n",
       "\n",
       "\n",
-      "\n",
-      "Result 1:\n",
+      "--------------------------------------------- Result 2 ---------------------------------------------\n",
       "\u001b[91mNegative (96%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
       "\n",
       "contains no wit , only labored gags \n",
       "\n",
-      "\n",
-      "\n",
-      "Result 2:\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 1 / 2 / 1 / 4:  40%|████      | 4/10 [00:07<00:10,  1.80s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 3 ---------------------------------------------\n",
       "\u001b[92mPositive (100%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
       "\n",
       "that loves its characters and communicates something rather beautiful about human nature \n",
       "\n",
       "\n",
-      "\n",
-      "Result 3:\n",
+      "--------------------------------------------- Result 4 ---------------------------------------------\n",
       "\u001b[92mPositive (82%)\u001b[0m --> \u001b[37m[SKIPPED]\u001b[0m\n",
       "\n",
       "remains utterly satisfied to remain the same throughout \n",
       "\n",
-      "\n",
-      "\n",
-      "Result 4:\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 2 / 2 / 1 / 5:  50%|█████     | 5/10 [00:07<00:07,  1.52s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 5 ---------------------------------------------\n",
       "\u001b[91mNegative (98%)\u001b[0m --> \u001b[92mPositive (52%)\u001b[0m\n",
       "\n",
       "on the \u001b[91mworst\u001b[0m \u001b[91mrevenge-of-the-nerds\u001b[0m clichés the filmmakers could \u001b[91mdredge\u001b[0m up \n",
       "\n",
-      "on the \u001b[92mpire\u001b[0m \u001b[92mrеvenge-of-the-nerds\u001b[0m clichés the filmmakers could \u001b[92mdragging\u001b[0m up \n",
+      "on the \u001b[92mpire\u001b[0m \u001b[92mreveng-of-the-nerds\u001b[0m clichés the filmmakers could \u001b[92mdragging\u001b[0m up \n",
       "\n",
-      "\n",
-      "\n",
-      "Result 5:\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 2 / 3 / 1 / 6:  60%|██████    | 6/10 [00:07<00:05,  1.32s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 6 ---------------------------------------------\n",
       "\u001b[91mNegative (99%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
       "\n",
       "that 's far too tragic to merit such superficial treatment \n",
       "\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 3 / 4 / 1 / 8:  80%|████████  | 8/10 [00:09<00:02,  1.13s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 7 ---------------------------------------------\n",
+      "\u001b[92mPositive (98%)\u001b[0m --> \u001b[91mNegative (62%)\u001b[0m\n",
+      "\n",
+      "\u001b[92mdemonstrates\u001b[0m that the \u001b[92mdirector\u001b[0m of such \u001b[92mhollywood\u001b[0m blockbusters as patriot games can still \u001b[92mturn\u001b[0m out a \u001b[92msmall\u001b[0m , personal \u001b[92mfilm\u001b[0m with an emotional \u001b[92mwallop\u001b[0m . \n",
+      "\n",
+      "\u001b[91mshows\u001b[0m that the \u001b[91mdirectors\u001b[0m of such \u001b[91mtinseltown\u001b[0m blockbusters as patriot games can still \u001b[91mturning\u001b[0m out a \u001b[91mtiny\u001b[0m , personal \u001b[91mmovies\u001b[0m with an emotional \u001b[91mbatting\u001b[0m . \n",
       "\n",
       "\n",
-      "Result 6:\n",
-      "\u001b[92mPositive (98%)\u001b[0m --> \u001b[91mNegative (50%)\u001b[0m\n",
-      "\n",
-      "demonstrates that the \u001b[92mdirector\u001b[0m of such \u001b[92mhollywood\u001b[0m blockbusters as patriot \u001b[92mgames\u001b[0m can still turn out a \u001b[92msmall\u001b[0m , personal \u001b[92mfilm\u001b[0m with an \u001b[92memotional\u001b[0m \u001b[92mwallop\u001b[0m . \n",
-      "\n",
-      "demonstrates that the \u001b[91mdirectors\u001b[0m of such \u001b[91mtinseltown\u001b[0m blockbusters as patriot \u001b[91mgame\u001b[0m can still turn out a \u001b[91mtiny\u001b[0m , personal \u001b[91mmovie\u001b[0m with an \u001b[91msentimental\u001b[0m \u001b[91mbatting\u001b[0m . \n",
-      "\n",
-      "\n",
-      "\n",
-      "Result 7:\n",
+      "--------------------------------------------- Result 8 ---------------------------------------------\n",
       "\u001b[92mPositive (90%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
       "\n",
       "of saucy \n",
       "\n",
-      "\n",
-      "\n",
-      "Result 8:\n",
-      "\u001b[91mNegative (99%)\u001b[0m --> \u001b[92mPositive (83%)\u001b[0m\n",
-      "\n",
-      "a \u001b[91mdepressed\u001b[0m \u001b[91mfifteen-year-old\u001b[0m 's suicidal poetry \n",
-      "\n",
-      "a \u001b[92mdepr\u001b[0m \u001b[92messed\u001b[0m \u001b[92mfifteeny-ear-old\u001b[0m 's suicidal poetry \n",
-      "\n",
-      "\n",
-      "\n",
-      "Result 9:\n",
-      "\u001b[92mPositive (79%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
-      "\n",
-      "are more deeply thought through than in most ` right-thinking ' films \n",
-      "\n",
-      "\n",
-      "\n",
-      "Result 10:\n",
-      "\u001b[91mNegative (97%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
-      "\n",
-      "goes to absurd lengths \n",
-      "\n",
-      "\n",
-      "\n",
-      "Result 11:\n",
-      "\u001b[91mNegative (94%)\u001b[0m --> \u001b[92mPositive (51%)\u001b[0m\n",
-      "\n",
-      "for those \u001b[91mmoviegoers\u001b[0m who \u001b[91mcomplain\u001b[0m that ` they do \u001b[91mn't\u001b[0m make movies like they used to anymore \n",
-      "\n",
-      "for those \u001b[92mmovieg\u001b[0m \u001b[92moers\u001b[0m who \u001b[92mcompl\u001b[0m \u001b[92main\u001b[0m that ` they do \u001b[92mnt\u001b[0m make movies like they used to anymore \n",
-      "\n",
-      "\n",
-      "\n",
-      "Result 12:\n",
-      "\u001b[91mNegative (92%)\u001b[0m --> \u001b[92mPositive (85%)\u001b[0m\n",
-      "\n",
-      "the part where \u001b[91mnothing\u001b[0m 's happening , \n",
-      "\n",
-      "the part where \u001b[92mnothin\u001b[0m 's happening , \n",
-      "\n",
-      "\n",
-      "\n",
-      "Result 13:\n",
-      "\u001b[91mNegative (97%)\u001b[0m --> \u001b[92mPositive (90%)\u001b[0m\n",
-      "\n",
-      "saw how \u001b[91mbad\u001b[0m this movie was \n",
-      "\n",
-      "saw how \u001b[92minclement\u001b[0m this movie was \n",
-      "\n",
-      "\n",
-      "\n",
-      "Result 14:\n",
-      "\u001b[91mNegative (73%)\u001b[0m --> \u001b[92mPositive (84%)\u001b[0m\n",
-      "\n",
-      "lend some dignity to a \u001b[91mdumb\u001b[0m story \n",
-      "\n",
-      "lend some dignity to a \u001b[92mdaft\u001b[0m story \n",
-      "\n",
-      "\n",
-      "\n",
-      "Result 15:\n",
-      "\u001b[92mPositive (99%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
-      "\n",
-      "the greatest musicians \n",
-      "\n",
-      "\n",
-      "\n",
-      "Result 16:\n",
-      "\u001b[91mNegative (98%)\u001b[0m --> \u001b[92mPositive (99%)\u001b[0m\n",
-      "\n",
-      "\u001b[91mcold\u001b[0m movie \n",
-      "\n",
-      "\u001b[92mcolder\u001b[0m movie \n",
-      "\n",
-      "\n",
-      "\n",
-      "Result 17:\n",
-      "\u001b[92mPositive (87%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
-      "\n",
-      "with his usual intelligence and subtlety \n",
-      "\n",
-      "\n",
-      "\n",
-      "Result 18:\n",
-      "\u001b[91mNegative (99%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
-      "\n",
-      "redundant concept \n",
-      "\n",
-      "\n",
-      "\n",
-      "Result 19:\n",
-      "\u001b[92mPositive (93%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
-      "\n",
-      "swimming is above all about a young woman 's face , and by casting an actress whose face projects that woman 's doubts and yearnings , it succeeds . \n",
-      "\n",
-      "\n",
-      "\n",
       "\n"
      ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 4 / 5 / 1 / 10: 100%|██████████| 10/10 [00:09<00:00,  1.06it/s]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 9 ---------------------------------------------\n",
+      "\u001b[91mNegative (99%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
+      "\n",
+      "a depressed fifteen-year-old 's suicidal poetry \n",
+      "\n",
+      "\n",
+      "--------------------------------------------- Result 10 ---------------------------------------------\n",
+      "\u001b[92mPositive (79%)\u001b[0m --> \u001b[91mNegative (65%)\u001b[0m\n",
+      "\n",
+      "are more \u001b[92mdeeply\u001b[0m thought through than in most ` right-thinking ' films \n",
+      "\n",
+      "are more \u001b[91mseriously\u001b[0m thought through than in most ` right-thinking ' films \n",
+      "\n",
+      "\n",
+      "\n",
+      "+-------------------------------+--------+\n",
+      "| Attack Results                |        |\n",
+      "+-------------------------------+--------+\n",
+      "| Number of successful attacks: | 4      |\n",
+      "| Number of failed attacks:     | 5      |\n",
+      "| Number of skipped attacks:    | 1      |\n",
+      "| Original accuracy:            | 90.0%  |\n",
+      "| Accuracy under attack:        | 50.0%  |\n",
+      "| Attack success rate:          | 44.44% |\n",
+      "| Average perturbed word %:     | 20.95% |\n",
+      "| Average num. words per input: | 9.5    |\n",
+      "| Avg num queries:              | 34.67  |\n",
+      "+-------------------------------+--------+\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "[<textattack.attack_results.successful_attack_result.SuccessfulAttackResult at 0x7fb68d0028b0>,\n",
+       " <textattack.attack_results.failed_attack_result.FailedAttackResult at 0x7fb685f0dbb0>,\n",
+       " <textattack.attack_results.failed_attack_result.FailedAttackResult at 0x7fb689188040>,\n",
+       " <textattack.attack_results.skipped_attack_result.SkippedAttackResult at 0x7fb695031250>,\n",
+       " <textattack.attack_results.successful_attack_result.SuccessfulAttackResult at 0x7fb695031760>,\n",
+       " <textattack.attack_results.failed_attack_result.FailedAttackResult at 0x7fb694b7abb0>,\n",
+       " <textattack.attack_results.successful_attack_result.SuccessfulAttackResult at 0x7fb67cd36df0>,\n",
+       " <textattack.attack_results.failed_attack_result.FailedAttackResult at 0x7fb694b7a880>,\n",
+       " <textattack.attack_results.failed_attack_result.FailedAttackResult at 0x7fb694b7a790>,\n",
+       " <textattack.attack_results.successful_attack_result.SuccessfulAttackResult at 0x7fb689ab1be0>]"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
     }
    ],
    "source": [
     "from textattack.datasets import HuggingFaceDataset\n",
     "from textattack.attack_recipes import TextBuggerLi2018\n",
+    "from textattack.attacker import Attacker\n",
+    "\n",
     "\n",
     "dataset = HuggingFaceDataset(\"glue\", \"sst2\", \"train\")\n",
-    "attack = TextBuggerLi2018(model_wrapper)\n",
+    "attack = TextBuggerLi2018.build(model_wrapper)\n",
     "\n",
-    "results = list(attack.attack_dataset(dataset, indices=range(20)))\n",
-    "for idx, result in enumerate(results):\n",
-    "    print(f'Result {idx}:')\n",
-    "    print(result.__str__(color_method='ansi'))\n",
-    "    print('\\n')\n",
-    "print()"
+    "attacker = Attacker(attack, dataset)\n",
+    "attacker.attack_dataset()"
    ]
   }
  ],
  "metadata": {
+  "accelerator": "GPU",
   "colab": {
    "collapsed_sections": [],
    "name": "[TextAttack] Model Example: AllenNLP",
    "provenance": []
   },
   "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
+   "name": "python379jvsc74a57bd00aa23297d40f12761ebb1c384bf2965d5ecbdef2f9c005ee7346b9ec0bcc5588",
+   "display_name": "Python 3.7.9 64-bit ('pytorch-gpu': pyenv)"
   },
   "language_info": {
    "codemirror_mode": {
@@ -314,9 +357,9 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.7"
+   "version": "3.8.8"
   }
  },
  "nbformat": 4,
- "nbformat_minor": 1
+ "nbformat_minor": 4
 }

docs/2notebook/Example_4_CamemBERT.ipynb

@@ -2,7 +2,9 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "metadata": {},
+   "metadata": {
+    "id": "koVcufVBD9uv"
+   },
    "source": [
     "# Multi-language attacks\n",
     "\n",
@@ -19,7 +21,9 @@
   },
   {
    "cell_type": "markdown",
-   "metadata": {},
+   "metadata": {
+    "id": "Abd2C3zJD9u4"
+   },
    "source": [
     "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/QData/TextAttack/blob/master/docs/2notebook/Example_4_CamemBERT.ipynb)\n",
     "\n",
@@ -29,13 +33,16 @@
   {
    "cell_type": "code",
    "execution_count": 1,
-   "metadata": {},
+   "metadata": {
+    "id": "-fnSUl8ND9u5"
+   },
    "outputs": [],
    "source": [
     "from textattack.attack_recipes import PWWSRen2019\n",
     "from textattack.datasets import HuggingFaceDataset\n",
     "from textattack.models.wrappers import ModelWrapper\n",
     "from transformers import AutoTokenizer, TFAutoModelForSequenceClassification, pipeline\n",
+    "from textattack import Attacker\n",
     "\n",
     "import numpy as np\n",
     "\n",
@@ -55,10 +62,10 @@
     "        \n",
     "            [[0.218262017, 0.7817379832267761]\n",
     "    \"\"\"\n",
-    "    def __init__(self, pipeline):\n",
-    "        self.pipeline = pipeline\n",
+    "    def __init__(self, model):\n",
+    "        self.model = model#pipeline = pipeline\n",
     "    def __call__(self, text_inputs):\n",
-    "        raw_outputs = self.pipeline(text_inputs)\n",
+    "        raw_outputs = self.model(text_inputs)\n",
     "        outputs = []\n",
     "        for output in raw_outputs:\n",
     "            score = output['score']\n",
@@ -71,111 +78,500 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 2,
    "metadata": {
-    "scrolled": true
+    "colab": {
+     "base_uri": "https://localhost:8080/"
+    },
+    "id": "i2WPtwO9D9u6",
+    "outputId": "2f5e8fab-1047-417d-c90c-b9238b2886a4",
+    "scrolled": true,
+    "tags": []
    },
    "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "99f2f220b210403eaaf82004365bb30b",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=445132512.0, style=ProgressStyle(descri…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n"
+     ]
+    },
     {
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "All model checkpoint weights were used when initializing TFCamembertForSequenceClassification.\n",
+      "All model checkpoint layers were used when initializing TFCamembertForSequenceClassification.\n",
       "\n",
-      "All the weights of TFCamembertForSequenceClassification were initialized from the model checkpoint at tblard/tf-allocine.\n",
-      "If your task is similar to the task the model of the ckeckpoint was trained on, you can already use TFCamembertForSequenceClassification for predictions without further training.\n",
-      "\u001b[34;1mtextattack\u001b[0m: Unknown if model of class <class '__main__.HuggingFaceSentimentAnalysisPipelineWrapper'> compatible with goal function <class 'textattack.goal_functions.classification.untargeted_classification.UntargetedClassification'>.\n",
-      "\u001b[34;1mtextattack\u001b[0m: Loading \u001b[94mnlp\u001b[0m dataset \u001b[94mallocine\u001b[0m, split \u001b[94mtest\u001b[0m.\n"
+      "All the layers of TFCamembertForSequenceClassification were initialized from the model checkpoint at tblard/tf-allocine.\n",
+      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFCamembertForSequenceClassification for predictions without further training.\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "584bb087e19b46c3a97a69f7bdd25c8d",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=810912.0, style=ProgressStyle(descripti…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "8ea1879230924bf985f07737c7979d8a",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=210.0, style=ProgressStyle(description_…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "d27e420e82004ebd8628adbc5ed4e883",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=2.0, style=ProgressStyle(description_wi…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "textattack: Unknown if model of class <class 'transformers.pipelines.text_classification.TextClassificationPipeline'> compatible with goal function <class 'textattack.goal_functions.classification.untargeted_classification.UntargetedClassification'>.\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "c01c2a4b2ef949018c400cfbbd8ab96c",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=1167.0, style=ProgressStyle(description…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "1606f0a088f444b48e36a7c12156aa12",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=813.0, style=ProgressStyle(description_…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "Downloading and preparing dataset allocine_dataset/allocine (download: 63.54 MiB, generated: 109.12 MiB, post-processed: Unknown size, total: 172.66 MiB) to /p/qdata/jy2ma/.cache/textattack/datasets/allocine_dataset/allocine/1.0.0/d7a2c05d4ab7254d411130aa8b47ae2a094af074e120fc8d46ec0beed909e896...\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "b90894c25b9841fc9e3e458b6a82ddd9",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=66625305.0, style=ProgressStyle(descrip…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "HBox(children=(FloatProgress(value=1.0, bar_style='info', layout=Layout(width='20px'), max=1.0), HTML(value=''…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "HBox(children=(FloatProgress(value=1.0, bar_style='info', layout=Layout(width='20px'), max=1.0), HTML(value=''…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "HBox(children=(FloatProgress(value=1.0, bar_style='info', layout=Layout(width='20px'), max=1.0), HTML(value=''…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "textattack: Loading \u001b[94mdatasets\u001b[0m dataset \u001b[94mallocine\u001b[0m, split \u001b[94mtest\u001b[0m.\n",
+      "  0%|          | 0/10 [00:00<?, ?it/s]"
      ]
     },
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "xxxxxxxxxxxxxxxxxxxx Result 1 xxxxxxxxxxxxxxxxxxxx\n",
+      "Dataset allocine_dataset downloaded and prepared to /p/qdata/jy2ma/.cache/textattack/datasets/allocine_dataset/allocine/1.0.0/d7a2c05d4ab7254d411130aa8b47ae2a094af074e120fc8d46ec0beed909e896. Subsequent calls will reuse this data.\n",
+      "Attack(\n",
+      "  (search_method): GreedyWordSwapWIR(\n",
+      "    (wir_method):  weighted-saliency\n",
+      "  )\n",
+      "  (goal_function):  UntargetedClassification\n",
+      "  (transformation):  WordSwapWordNet\n",
+      "  (constraints): \n",
+      "    (0): RepeatModification\n",
+      "    (1): StopwordModification\n",
+      "  (is_black_box):  True\n",
+      ") \n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 1 / 0 / 0 / 1:  10%|█         | 1/10 [00:18<02:42, 18.01s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 1 ---------------------------------------------\n",
       "\u001b[92mPositive (100%)\u001b[0m --> \u001b[91mNegative (53%)\u001b[0m\n",
       "\n",
       "\u001b[92mMagnifique\u001b[0m épopée, une \u001b[92mbelle\u001b[0m \u001b[92mhistoire\u001b[0m, touchante avec des acteurs \u001b[92mqui\u001b[0m interprètent \u001b[92mtrès\u001b[0m \u001b[92mbien\u001b[0m leur rôles (Mel Gibson, Heath Ledger, Jason Isaacs...), le genre \u001b[92mde\u001b[0m \u001b[92mfilm\u001b[0m \u001b[92mqui\u001b[0m \u001b[92mse\u001b[0m savoure \u001b[92men\u001b[0m \u001b[92mfamille\u001b[0m! :)\n",
       "\n",
       "\u001b[91mbonnard\u001b[0m épopée, une \u001b[91mbeau\u001b[0m \u001b[91mbobard\u001b[0m, touchante avec des acteurs \u001b[91mlequel\u001b[0m interprètent \u001b[91mmême\u001b[0m \u001b[91macceptablement\u001b[0m leur rôles (Mel Gibson, Heath Ledger, Jason Isaacs...), le genre \u001b[91mgale\u001b[0m \u001b[91mpellicule\u001b[0m \u001b[91mOMS\u001b[0m \u001b[91mConcepteur\u001b[0m savoure \u001b[91mun\u001b[0m \u001b[91msyndicat\u001b[0m! :)\n",
       "\n",
-      "xxxxxxxxxxxxxxxxxxxx Result 2 xxxxxxxxxxxxxxxxxxxx\n",
-      "\u001b[91mNegative (94%)\u001b[0m --> \u001b[92mPositive (91%)\u001b[0m\n",
-      "\n",
-      "Je n'ai pas aimé mais pourtant je lui mets \u001b[91m2\u001b[0m étoiles car l'expérience est louable. Rien de conventionnel ici. Une visite E.T. mais jonchée d'idées /- originales. Le soucis, tout ceci avait-il vraiment sa place dans un film de S.F. tirant sur l'horreur ? Voici un film qui, à l'inverse de tant d'autres qui y ont droit, mériterait peut-être un remake.\n",
-      "\n",
-      "Je n'ai pas aimé mais pourtant je lui mets \u001b[92m4\u001b[0m étoiles car l'expérience est louable. Rien de conventionnel ici. Une visite E.T. mais jonchée d'idées /- originales. Le soucis, tout ceci avait-il vraiment sa place dans un film de S.F. tirant sur l'horreur ? Voici un film qui, à l'inverse de tant d'autres qui y ont droit, mériterait peut-être un remake.\n",
-      "\n",
-      "xxxxxxxxxxxxxxxxxxxx Result 3 xxxxxxxxxxxxxxxxxxxx\n",
-      "\u001b[92mPositive (85%)\u001b[0m --> \u001b[91mNegative (91%)\u001b[0m\n",
-      "\n",
-      "Un \u001b[92mdessin\u001b[0m animé qui brille par sa féerie et ses chansons.\n",
-      "\n",
-      "Un \u001b[91mbrouillon\u001b[0m animé qui brille par sa féerie et ses chansons.\n",
-      "\n",
-      "xxxxxxxxxxxxxxxxxxxx Result 4 xxxxxxxxxxxxxxxxxxxx\n",
-      "\u001b[91mNegative (100%)\u001b[0m --> \u001b[92mPositive (80%)\u001b[0m\n",
-      "\n",
-      "\u001b[91mSi\u001b[0m c'est là le renouveau du cinéma français, c'est tout \u001b[91mde\u001b[0m même foutrement chiant. \u001b[91mSi\u001b[0m l'objet est \u001b[91mtrès\u001b[0m stylisé et la tension palpable, le film paraît \u001b[91mplutôt\u001b[0m \u001b[91mcreux\u001b[0m.\n",
-      "\n",
-      "\u001b[92maussi\u001b[0m c'est là le renouveau du cinéma français, c'est tout \u001b[92mabolir\u001b[0m même foutrement chiant. \u001b[92mtellement\u001b[0m l'objet est \u001b[92mprodigieusement\u001b[0m stylisé et la tension palpable, le film paraît \u001b[92mpeu\u001b[0m \u001b[92mtrou\u001b[0m.\n",
-      "\n",
-      "xxxxxxxxxxxxxxxxxxxx Result 5 xxxxxxxxxxxxxxxxxxxx\n",
-      "\u001b[91mNegative (100%)\u001b[0m --> \u001b[92mPositive (51%)\u001b[0m\n",
-      "\n",
-      "Et \u001b[91mpourtant\u001b[0m on s’\u001b[91men\u001b[0m Doutait !\u001b[91mSecond\u001b[0m \u001b[91mvolet\u001b[0m \u001b[91mtrès\u001b[0m \u001b[91mmauvais\u001b[0m, sans \u001b[91mfraîcheur\u001b[0m et particulièrement lourdingue. Quel \u001b[91mdommage\u001b[0m.\n",
-      "\n",
-      "Et \u001b[92mfin\u001b[0m on s’\u001b[92mpostérieurement\u001b[0m Doutait !\u001b[92mmoment\u001b[0m \u001b[92mchapitre\u001b[0m \u001b[92mincroyablement\u001b[0m \u001b[92mdifficile\u001b[0m, sans \u001b[92mimpudence\u001b[0m et particulièrement lourdingue. Quel \u001b[92mprix\u001b[0m.\n",
-      "\n",
-      "xxxxxxxxxxxxxxxxxxxx Result 6 xxxxxxxxxxxxxxxxxxxx\n",
-      "\u001b[92mPositive (100%)\u001b[0m --> \u001b[91mNegative (50%)\u001b[0m\n",
-      "\n",
-      "Vous reprendrez bien un peu d'été ? Ce film je le voyais comme un mélange de Rohmer et de Rozier, un film de vacances, j'adore ça, un truc beau et pur qui dit des choses sur la vie, l'amour, les filles, les vacances. Un film qui se regarde en sirotant une boisson fraîche en écoutant les grillons ! Sauf qu'en fait \u001b[92mnon\u001b[0m ! On a un film foutraque au \u001b[92mpossible\u001b[0m qui reprend les codes justement de Rohmer voir Godard, enfin la Nouvelle Vague en général dans sa première partie (jusqu'à même finir sur une partie qui ressemblerait à du Kusturica), mais en beaucoup plus léger et décalé. Le film n'en a rien à foutre de rien, il ose tout, n'a peur de rien et ça c'est \u001b[92mbon\u001b[0m. C'est sans doute le film le plus \u001b[92mdrôle\u001b[0m de 2013, mais tout \u001b[92msimplement\u001b[0m l'un des meilleurs tout \u001b[92mcourt\u001b[0m. Le film qui nous sort des dialogues qui pourraient sortir d'un mauvais Godard (oxymore) sur un ton what the fuckesque… raconte des anecdotes débiles au souhait face caméra… et pourtant, il y a quelque chose dans ce film survolté. Il y a du beau. Ces scènes dans la neige, c'est tendre, c'est beau, ça tranche avec le reste et ça donne du coeur à l'amourette, ça aide à le faire paraître comme une évidence. Et puis on a cette scène que je trouve sublime qui m'a profondément émue, cette scène où le docteur Placenta devient tout à coup sérieux et parle de cette date où chaque année il repense à cette fille et au fait qu'une année de plus le sépare d'elle. C'est horrible comme concept et pourtant tellement vrai et sincère. C'est vraiment \u001b[92mtroublant\u001b[0m. Et encore une fois la scène d'avant est très drôle et là, un petit moment de douceur avant de repartir sur le train effréné ! Et il y a ces fesses… Et le plus beau c'est qu'à la fin Vimala Pons a un petit air d'Anna Karina ! Film fout, étonnant, percutant, drôle, beau, triste ! C'est foutrement cool !\n",
-      "\n",
-      "Vous reprendrez bien un peu d'été ? Ce film je le voyais comme un mélange de Rohmer et de Rozier, un film de vacances, j'adore ça, un truc beau et pur qui dit des choses sur la vie, l'amour, les filles, les vacances. Un film qui se regarde en sirotant une boisson fraîche en écoutant les grillons ! Sauf qu'en fait \u001b[91mniet\u001b[0m ! On a un film foutraque au \u001b[91mexécutable\u001b[0m qui reprend les codes justement de Rohmer voir Godard, enfin la Nouvelle Vague en général dans sa première partie (jusqu'à même finir sur une partie qui ressemblerait à du Kusturica), mais en beaucoup plus léger et décalé. Le film n'en a rien à foutre de rien, il ose tout, n'a peur de rien et ça c'est \u001b[91mlisse\u001b[0m. C'est sans doute le film le plus \u001b[91mridicule\u001b[0m de 2013, mais tout \u001b[91msauf\u001b[0m l'un des meilleurs tout \u001b[91minsuffisant\u001b[0m. Le film qui nous sort des dialogues qui pourraient sortir d'un mauvais Godard (oxymore) sur un ton what the fuckesque… raconte des anecdotes débiles au souhait face caméra… et pourtant, il y a quelque chose dans ce film survolté. Il y a du beau. Ces scènes dans la neige, c'est tendre, c'est beau, ça tranche avec le reste et ça donne du coeur à l'amourette, ça aide à le faire paraître comme une évidence. Et puis on a cette scène que je trouve sublime qui m'a profondément émue, cette scène où le docteur Placenta devient tout à coup sérieux et parle de cette date où chaque année il repense à cette fille et au fait qu'une année de plus le sépare d'elle. C'est horrible comme concept et pourtant tellement vrai et sincère. C'est vraiment \u001b[91mennuyeux\u001b[0m. Et encore une fois la scène d'avant est très drôle et là, un petit moment de douceur avant de repartir sur le train effréné ! Et il y a ces fesses… Et le plus beau c'est qu'à la fin Vimala Pons a un petit air d'Anna Karina ! Film fout, étonnant, percutant, drôle, beau, triste ! C'est foutrement cool !\n",
-      "\n",
-      "xxxxxxxxxxxxxxxxxxxx Result 7 xxxxxxxxxxxxxxxxxxxx\n",
-      "\u001b[92mPositive (55%)\u001b[0m --> \u001b[91mNegative (88%)\u001b[0m\n",
-      "\n",
-      "Bon c'est \u001b[92mpas\u001b[0m un grand film mais on passe un bon moment avec ses ado à la recherche de l'orgasme. Y'a que les Allemands pour faire des films aussi barge ! :-)\n",
-      "\n",
-      "Bon c'est \u001b[91mniet\u001b[0m un grand film mais on passe un bon moment avec ses ado à la recherche de l'orgasme. Y'a que les Allemands pour faire des films aussi barge ! :-)\n",
-      "\n",
-      "xxxxxxxxxxxxxxxxxxxx Result 8 xxxxxxxxxxxxxxxxxxxx\n",
-      "\u001b[92mPositive (100%)\u001b[0m --> \u001b[91mNegative (97%)\u001b[0m\n",
-      "\n",
-      "\u001b[92mTerrible\u001b[0m histoire que ces êtres sans amour, ces êtres lisses et frustres qui passent à côté de leur vie. Quelle leçon Monsieur Brizé! Vous avez tout dit, tout filmé jusqu'au moindre détail. \u001b[92mtout\u001b[0m est beau et terrifiant jusqu'à la scène finale qui nous liquéfie, un Vincent Lindon regardant la vie fixement sans oser la toucher ni la prendre dans ses bras, une Hélène Vincent qui attend, qui attend... Mon Dieu Monsieur Brizé, continuez....\n",
-      "\n",
-      "\u001b[91mméprisable\u001b[0m histoire que ces êtres sans amour, ces êtres lisses et frustres qui passent à côté de leur vie. Quelle leçon Monsieur Brizé! Vous avez tout dit, tout filmé jusqu'au moindre détail. \u001b[91mrien\u001b[0m est beau et terrifiant jusqu'à la scène finale qui nous liquéfie, un Vincent Lindon regardant la vie fixement sans oser la toucher ni la prendre dans ses bras, une Hélène Vincent qui attend, qui attend... Mon Dieu Monsieur Brizé, continuez....\n",
       "\n"
      ]
     },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 2 / 0 / 0 / 2:  20%|██        | 2/10 [00:57<03:50, 28.86s/it]"
+     ]
+    },
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "xxxxxxxxxxxxxxxxxxxx Result 9 xxxxxxxxxxxxxxxxxxxx\n",
+      "--------------------------------------------- Result 2 ---------------------------------------------\n",
+      "\u001b[91mNegative (94%)\u001b[0m --> \u001b[92mPositive (91%)\u001b[0m\n",
+      "\n",
+      "Je n'ai pas aimé mais pourtant je lui mets \u001b[91m2\u001b[0m étoiles car l'expérience est louable. Rien de conventionnel ici. Une visite E.T. mais jonchée d'idées /- originales. Le soucis, tout ceci avait-il vraiment sa place dans un film de S.F. tirant sur l'horreur ? Voici un film qui, à l'inverse de tant d'autres qui y ont droit, mériterait peut-être un remake.\n",
+      "\n",
+      "Je n'ai pas aimé mais pourtant je lui mets \u001b[92m4\u001b[0m étoiles car l'expérience est louable. Rien de conventionnel ici. Une visite E.T. mais jonchée d'idées /- originales. Le soucis, tout ceci avait-il vraiment sa place dans un film de S.F. tirant sur l'horreur ? Voici un film qui, à l'inverse de tant d'autres qui y ont droit, mériterait peut-être un remake.\n",
+      "\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 3 / 0 / 0 / 3:  30%|███       | 3/10 [00:59<02:18, 19.74s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 3 ---------------------------------------------\n",
+      "\u001b[92mPositive (85%)\u001b[0m --> \u001b[91mNegative (91%)\u001b[0m\n",
+      "\n",
+      "Un \u001b[92mdessin\u001b[0m animé qui brille par sa féerie et ses chansons.\n",
+      "\n",
+      "Un \u001b[91mbrouillon\u001b[0m animé qui brille par sa féerie et ses chansons.\n",
+      "\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 4 / 0 / 0 / 4:  40%|████      | 4/10 [01:09<01:44, 17.37s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 4 ---------------------------------------------\n",
+      "\u001b[91mNegative (100%)\u001b[0m --> \u001b[92mPositive (80%)\u001b[0m\n",
+      "\n",
+      "\u001b[91mSi\u001b[0m c'est là le renouveau du cinéma français, c'est tout \u001b[91mde\u001b[0m même foutrement chiant. \u001b[91mSi\u001b[0m l'objet est \u001b[91mtrès\u001b[0m stylisé et la tension palpable, le film paraît \u001b[91mplutôt\u001b[0m \u001b[91mcreux\u001b[0m.\n",
+      "\n",
+      "\u001b[92maussi\u001b[0m c'est là le renouveau du cinéma français, c'est tout \u001b[92mabolir\u001b[0m même foutrement chiant. \u001b[92mtellement\u001b[0m l'objet est \u001b[92mprodigieusement\u001b[0m stylisé et la tension palpable, le film paraît \u001b[92mpeu\u001b[0m \u001b[92mtrou\u001b[0m.\n",
+      "\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 5 / 0 / 0 / 5:  50%|█████     | 5/10 [01:15<01:15, 15.03s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 5 ---------------------------------------------\n",
+      "\u001b[91mNegative (100%)\u001b[0m --> \u001b[92mPositive (51%)\u001b[0m\n",
+      "\n",
+      "Et \u001b[91mpourtant\u001b[0m on s’\u001b[91men\u001b[0m Doutait !\u001b[91mSecond\u001b[0m \u001b[91mvolet\u001b[0m \u001b[91mtrès\u001b[0m \u001b[91mmauvais\u001b[0m, sans \u001b[91mfraîcheur\u001b[0m et particulièrement lourdingue. Quel \u001b[91mdommage\u001b[0m.\n",
+      "\n",
+      "Et \u001b[92mfin\u001b[0m on s’\u001b[92mpostérieurement\u001b[0m Doutait !\u001b[92mmoment\u001b[0m \u001b[92mchapitre\u001b[0m \u001b[92mincroyablement\u001b[0m \u001b[92mdifficile\u001b[0m, sans \u001b[92mimpudence\u001b[0m et particulièrement lourdingue. Quel \u001b[92mprix\u001b[0m.\n",
+      "\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 6 / 0 / 0 / 6:  60%|██████    | 6/10 [23:02<15:21, 230.43s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 6 ---------------------------------------------\n",
+      "\u001b[92mPositive (100%)\u001b[0m --> \u001b[91mNegative (50%)\u001b[0m\n",
+      "\n",
+      "Vous reprendrez bien un peu d'été ? Ce film je le voyais comme un mélange de Rohmer et de Rozier, un film de vacances, j'adore ça, un truc beau et pur qui dit des choses sur la vie, l'amour, les filles, les vacances. Un film qui se regarde en sirotant une boisson fraîche en écoutant les grillons ! Sauf qu'en fait \u001b[92mnon\u001b[0m ! On a un film foutraque au \u001b[92mpossible\u001b[0m qui reprend les codes justement de Rohmer voir Godard, enfin la Nouvelle Vague en général dans sa première partie (jusqu'à même finir sur une partie qui ressemblerait à du Kusturica), mais en beaucoup plus léger et décalé. Le film n'en a rien à foutre de rien, il ose tout, n'a peur de rien et ça c'est \u001b[92mbon\u001b[0m. C'est sans doute le film le plus \u001b[92mdrôle\u001b[0m de 2013, mais tout \u001b[92msimplement\u001b[0m l'un des meilleurs tout \u001b[92mcourt\u001b[0m. Le film qui nous sort des dialogues qui pourraient sortir d'un mauvais Godard (oxymore) sur un ton what the fuckesque… raconte des anecdotes débiles au souhait face caméra… et pourtant, il y a quelque chose dans ce film survolté. Il y a du beau. Ces scènes dans la neige, c'est tendre, c'est beau, ça tranche avec le reste et ça donne du coeur à l'amourette, ça aide à le faire paraître comme une évidence. Et puis on a cette scène que je trouve sublime qui m'a profondément émue, cette scène où le docteur Placenta devient tout à coup sérieux et parle de cette date où chaque année il repense à cette fille et au fait qu'une année de plus le sépare d'elle. C'est horrible comme concept et pourtant tellement vrai et sincère. C'est vraiment \u001b[92mtroublant\u001b[0m. Et encore une fois la scène d'avant est très drôle et là, un petit moment de douceur avant de repartir sur le train effréné ! Et il y a ces fesses… Et le plus beau c'est qu'à la fin Vimala Pons a un petit air d'Anna Karina ! Film fout, étonnant, percutant, drôle, beau, triste ! C'est foutrement cool !\n",
+      "\n",
+      "Vous reprendrez bien un peu d'été ? Ce film je le voyais comme un mélange de Rohmer et de Rozier, un film de vacances, j'adore ça, un truc beau et pur qui dit des choses sur la vie, l'amour, les filles, les vacances. Un film qui se regarde en sirotant une boisson fraîche en écoutant les grillons ! Sauf qu'en fait \u001b[91mniet\u001b[0m ! On a un film foutraque au \u001b[91mexécutable\u001b[0m qui reprend les codes justement de Rohmer voir Godard, enfin la Nouvelle Vague en général dans sa première partie (jusqu'à même finir sur une partie qui ressemblerait à du Kusturica), mais en beaucoup plus léger et décalé. Le film n'en a rien à foutre de rien, il ose tout, n'a peur de rien et ça c'est \u001b[91mlisse\u001b[0m. C'est sans doute le film le plus \u001b[91mridicule\u001b[0m de 2013, mais tout \u001b[91msauf\u001b[0m l'un des meilleurs tout \u001b[91minsuffisant\u001b[0m. Le film qui nous sort des dialogues qui pourraient sortir d'un mauvais Godard (oxymore) sur un ton what the fuckesque… raconte des anecdotes débiles au souhait face caméra… et pourtant, il y a quelque chose dans ce film survolté. Il y a du beau. Ces scènes dans la neige, c'est tendre, c'est beau, ça tranche avec le reste et ça donne du coeur à l'amourette, ça aide à le faire paraître comme une évidence. Et puis on a cette scène que je trouve sublime qui m'a profondément émue, cette scène où le docteur Placenta devient tout à coup sérieux et parle de cette date où chaque année il repense à cette fille et au fait qu'une année de plus le sépare d'elle. C'est horrible comme concept et pourtant tellement vrai et sincère. C'est vraiment \u001b[91mennuyeux\u001b[0m. Et encore une fois la scène d'avant est très drôle et là, un petit moment de douceur avant de repartir sur le train effréné ! Et il y a ces fesses… Et le plus beau c'est qu'à la fin Vimala Pons a un petit air d'Anna Karina ! Film fout, étonnant, percutant, drôle, beau, triste ! C'est foutrement cool !\n",
+      "\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 7 / 0 / 0 / 7:  70%|███████   | 7/10 [23:19<09:59, 199.87s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 7 ---------------------------------------------\n",
+      "\u001b[92mPositive (55%)\u001b[0m --> \u001b[91mNegative (88%)\u001b[0m\n",
+      "\n",
+      "Bon c'est \u001b[92mpas\u001b[0m un grand film mais on passe un bon moment avec ses ado à la recherche de l'orgasme. Y'a que les Allemands pour faire des films aussi barge ! :-)\n",
+      "\n",
+      "Bon c'est \u001b[91mniet\u001b[0m un grand film mais on passe un bon moment avec ses ado à la recherche de l'orgasme. Y'a que les Allemands pour faire des films aussi barge ! :-)\n",
+      "\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 8 / 0 / 0 / 8:  80%|████████  | 8/10 [24:03<06:00, 180.39s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 8 ---------------------------------------------\n",
+      "\u001b[92mPositive (100%)\u001b[0m --> \u001b[91mNegative (97%)\u001b[0m\n",
+      "\n",
+      "\u001b[92mTerrible\u001b[0m histoire que ces êtres sans amour, ces êtres lisses et frustres qui passent à côté de leur vie. Quelle leçon Monsieur Brizé! Vous avez tout dit, tout filmé jusqu'au moindre détail. \u001b[92mtout\u001b[0m est beau et terrifiant jusqu'à la scène finale qui nous liquéfie, un Vincent Lindon regardant la vie fixement sans oser la toucher ni la prendre dans ses bras, une Hélène Vincent qui attend, qui attend... Mon Dieu Monsieur Brizé, continuez....\n",
+      "\n",
+      "\u001b[91mméprisable\u001b[0m histoire que ces êtres sans amour, ces êtres lisses et frustres qui passent à côté de leur vie. Quelle leçon Monsieur Brizé! Vous avez tout dit, tout filmé jusqu'au moindre détail. \u001b[91mrien\u001b[0m est beau et terrifiant jusqu'à la scène finale qui nous liquéfie, un Vincent Lindon regardant la vie fixement sans oser la toucher ni la prendre dans ses bras, une Hélène Vincent qui attend, qui attend... Mon Dieu Monsieur Brizé, continuez....\n",
+      "\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 9 / 0 / 0 / 9:  90%|█████████ | 9/10 [24:13<02:41, 161.53s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 9 ---------------------------------------------\n",
       "\u001b[92mPositive (100%)\u001b[0m --> \u001b[91mNegative (54%)\u001b[0m\n",
       "\n",
       "Un \u001b[92mtrès\u001b[0m joli \u001b[92mfilm\u001b[0m, qui ressemble à un téléfilm mais qui a le mérite d'être émouvant et proche de ses personnages. Magimel est \u001b[92mvraiment\u001b[0m très \u001b[92mbon\u001b[0m et l'histoire est touchante\n",
       "\n",
       "Un \u001b[91mplus\u001b[0m joli \u001b[91mfeuil\u001b[0m, qui ressemble à un téléfilm mais qui a le mérite d'être émouvant et proche de ses personnages. Magimel est \u001b[91mabsolument\u001b[0m très \u001b[91mlisse\u001b[0m et l'histoire est touchante\n",
       "\n",
-      "xxxxxxxxxxxxxxxxxxxx Result 10 xxxxxxxxxxxxxxxxxxxx\n",
+      "\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "[Succeeded / Failed / Skipped / Total] 10 / 0 / 0 / 10: 100%|██████████| 10/10 [28:30<00:00, 171.04s/it]"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "--------------------------------------------- Result 10 ---------------------------------------------\n",
       "\u001b[91mNegative (100%)\u001b[0m --> \u001b[92mPositive (51%)\u001b[0m\n",
       "\n",
       "Mais comment certaines personnes ont pus lui \u001b[91mmettre\u001b[0m 5/5 et \u001b[91mdonc\u001b[0m dire indirectement \u001b[91mque\u001b[0m c'est un chef-d'œuvre ??? Et comment a-t-il fait pour sortir au cinéma et non en DTV ??? C'est pas un film que l'on regarde dans une salle obscur ça, pour moi ça ressemble plus à un téléfilm que l'on visionne un dimanche pluvieux \u001b[91mpour\u001b[0m que les enfants arrête de nous casser les pieds ! \u001b[91mEt\u001b[0m puis, le \u001b[91mscénario\u001b[0m avec le chien que devient le meilleur ami du gosse, c'est du vu et revu (un cliché) ! L'acteur principal est quant à lui aussi agaçant que son personnage ! Les suites ont l'air \u001b[91maussi\u001b[0m mauvaises que Buddy Star des Paniers étant donné que l'histoire est quasiment la même (pour moi ça c'est pas des suites, c'est \u001b[91mplutôt\u001b[0m une succession \u001b[91mde\u001b[0m petits reboots inutiles). \u001b[91mReste\u001b[0m regardable pour les moins de 10 ans (et encore, même moi à 6 ans, je n'aurais pas aimé).\n",
       "\n",
       "Mais comment certaines personnes ont pus lui \u001b[92mformuler\u001b[0m 5/5 et \u001b[92md'où\u001b[0m dire indirectement \u001b[92mcar\u001b[0m c'est un chef-d'œuvre ??? Et comment a-t-il fait pour sortir au cinéma et non en DTV ??? C'est pas un film que l'on regarde dans une salle obscur ça, pour moi ça ressemble plus à un téléfilm que l'on visionne un dimanche pluvieux \u001b[92mat\u001b[0m que les enfants arrête de nous casser les pieds ! \u001b[92mpoids\u001b[0m puis, le \u001b[92mfigure\u001b[0m avec le chien que devient le meilleur ami du gosse, c'est du vu et revu (un cliché) ! L'acteur principal est quant à lui aussi agaçant que son personnage ! Les suites ont l'air \u001b[92mmaintenant\u001b[0m mauvaises que Buddy Star des Paniers étant donné que l'histoire est quasiment la même (pour moi ça c'est pas des suites, c'est \u001b[92mpeu\u001b[0m une succession \u001b[92mdu\u001b[0m petits reboots inutiles). \u001b[92mrelique\u001b[0m regardable pour les moins de 10 ans (et encore, même moi à 6 ans, je n'aurais pas aimé).\n",
       "\n",
-      "xxxxxxxxxxxxxxxxxxxx Result 11 xxxxxxxxxxxxxxxxxxxx\n",
-      "\u001b[92mPositive (100%)\u001b[0m --> \u001b[91mNegative (53%)\u001b[0m\n",
       "\n",
-      "LE film de mon enfance , il a un peu vieilli maintenant , mais l'ours reste toujours impressionnant, il est bien réel contrairement au film 'the Revenant\" . Ce n'est surement pas un chef-d'œuvre mais je le trouve bien réalise , captivant , beaux et accompagné d'une superbe musique. Le gros points noir c'est la facilité qu'ils ont a créer des peaux , des pièges , et rester longtemps sans manger....mais on oublie assez vite ces erreurs grâce a un casting sympathique et aux décors naturels. Un \u001b[92mvieux\u001b[0m film mais qui reste \u001b[92mtoujours\u001b[0m un \u001b[92mbon\u001b[0m \u001b[92mfilm\u001b[0m.\n",
       "\n",
-      "LE film de mon enfance , il a un peu vieilli maintenant , mais l'ours reste toujours impressionnant, il est bien réel contrairement au film 'the Revenant\" . Ce n'est surement pas un chef-d'œuvre mais je le trouve bien réalise , captivant , beaux et accompagné d'une superbe musique. Le gros points noir c'est la facilité qu'ils ont a créer des peaux , des pièges , et rester longtemps sans manger....mais on oublie assez vite ces erreurs grâce a un casting sympathique et aux décors naturels. Un \u001b[91mbancal\u001b[0m film mais qui reste \u001b[91mdéfinitivement\u001b[0m un \u001b[91mpassable\u001b[0m \u001b[91mpellicule\u001b[0m.\n",
+      "+-------------------------------+--------+\n",
+      "| Attack Results                |        |\n",
+      "+-------------------------------+--------+\n",
+      "| Number of successful attacks: | 10     |\n",
+      "| Number of failed attacks:     | 0      |\n",
+      "| Number of skipped attacks:    | 0      |\n",
+      "| Original accuracy:            | 100.0% |\n",
+      "| Accuracy under attack:        | 0.0%   |\n",
+      "| Attack success rate:          | 100.0% |\n",
+      "| Average perturbed word %:     | 14.73% |\n",
+      "| Average num. words per input: | 76.4   |\n",
+      "| Avg num queries:              | 904.4  |\n",
+      "+-------------------------------+--------+\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
       "\n"
      ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "[<textattack.attack_results.successful_attack_result.SuccessfulAttackResult at 0x7f9d3cb55b80>,\n",
+       " <textattack.attack_results.successful_attack_result.SuccessfulAttackResult at 0x7f9d43fc5d90>,\n",
+       " <textattack.attack_results.successful_attack_result.SuccessfulAttackResult at 0x7f9d39840df0>,\n",
+       " <textattack.attack_results.successful_attack_result.SuccessfulAttackResult at 0x7f9d3241a160>,\n",
+       " <textattack.attack_results.successful_attack_result.SuccessfulAttackResult at 0x7f9d398405b0>,\n",
+       " <textattack.attack_results.successful_attack_result.SuccessfulAttackResult at 0x7f9d47ce17f0>,\n",
+       " <textattack.attack_results.successful_attack_result.SuccessfulAttackResult at 0x7f9d3db79040>,\n",
+       " <textattack.attack_results.successful_attack_result.SuccessfulAttackResult at 0x7f9d3f8e3730>,\n",
+       " <textattack.attack_results.successful_attack_result.SuccessfulAttackResult at 0x7f9d33152f10>,\n",
+       " <textattack.attack_results.successful_attack_result.SuccessfulAttackResult at 0x7f9e5d43aeb0>]"
+      ]
+     },
+     "execution_count": 2,
+     "metadata": {},
+     "output_type": "execute_result"
     }
    ],
    "source": [
@@ -198,18 +594,23 @@
     "recipe.transformation.language = 'fra'\n",
     "\n",
     "dataset = HuggingFaceDataset('allocine', split='test')\n",
-    "for idx, result in enumerate(recipe.attack_dataset(dataset, indices=range(11))):\n",
-    "    print(('x' * 20), f'Result {idx+1}', ('x' * 20))\n",
-    "    print(result.__str__(color_method='ansi'))\n",
-    "    print()\n"
+    "\n",
+    "attacker = Attacker(recipe, dataset)\n",
+    "attacker.attack_dataset()\n"
    ]
   }
  ],
  "metadata": {
+  "accelerator": "GPU",
+  "colab": {
+   "collapsed_sections": [],
+   "name": "Example_4_CamemBERT.ipynb",
+   "provenance": []
+  },
   "kernelspec": {
-   "display_name": "torch",
+   "display_name": "Python 3",
    "language": "python",
-   "name": "build_central"
+   "name": "python3"
   },
   "language_info": {
    "codemirror_mode": {
@@ -221,7 +622,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.3"
+   "version": "3.8.8"
   }
  },
  "nbformat": 4,

docs/3recipes/attack_recipes.rst

@@ -1,5 +1,5 @@
-Attack Recipes
-===============
+Attack Recipes API
+==================
 
 We provide a number of pre-built attack recipes, which correspond to attacks from the literature. To run an attack recipe from the command line, run::
 
@@ -20,124 +20,112 @@ Attacks on classification models
 #################################
 
 
-Alzantot Genetic Algorithm (Generating Natural Language Adversarial Examples)
-***********************************************************************************
+1. Alzantot Genetic Algorithm (Generating Natural Language Adversarial Examples)
+2. Faster Alzantot Genetic Algorithm (Certified Robustness to Adversarial Word Substitutions)
+3. BAE (BAE: BERT-Based Adversarial Examples)
+4. BERT-Attack: (BERT-Attack: Adversarial Attack Against BERT Using BERT)
+5. CheckList: (Beyond Accuracy: Behavioral Testing of NLP models with CheckList)
+6. DeepWordBug (Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers)
+7. HotFlip (HotFlip: White-Box Adversarial Examples for Text Classification)
+8. Improved Genetic Algorithm (Natural Language Adversarial Attacks and Defenses in Word Level)
+9. Input Reduction (Pathologies of Neural Models Make Interpretations Difficult)
+10. Kuleshov (Adversarial Examples for Natural Language Classification Problems)
+11. Particle Swarm Optimization (Word-level Textual Adversarial Attacking as Combinatorial Optimization)
+12. PWWS (Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency)
+13. TextFooler (Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment)
+14. TextBugger (TextBugger: Generating Adversarial Text Against Real-world Applications)
 
-.. warning::
-    This attack uses a very slow language model. Consider using the ``fast-alzantot``
-    recipe instead.
 
 
 .. automodule:: textattack.attack_recipes.genetic_algorithm_alzantot_2018
    :members:
    :noindex:
 
-Faster Alzantot Genetic Algorithm (Certified Robustness to Adversarial Word Substitutions)
-**********************************************************************************************
 
 .. automodule:: textattack.attack_recipes.faster_genetic_algorithm_jia_2019
    :members:
    :noindex:
 
-BAE (BAE: BERT-Based Adversarial Examples)
-*********************************************
 
 .. automodule:: textattack.attack_recipes.bae_garg_2019
    :members:
    :noindex:
 
-BERT-Attack: (BERT-Attack: Adversarial Attack Against BERT Using BERT)
-*************************************************************************
+
 
 .. automodule:: textattack.attack_recipes.bert_attack_li_2020
    :members:
    :noindex:
 
 
-CheckList: (Beyond Accuracy: Behavioral Testing of NLP models with CheckList)
-*******************************************************************************
 
 .. automodule:: textattack.attack_recipes.checklist_ribeiro_2020
    :members:
    :noindex:
 
-DeepWordBug (Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers)
-******************************************************************************************************
+
 
 .. automodule:: textattack.attack_recipes.deepwordbug_gao_2018
    :members:
    :noindex:
 
-HotFlip (HotFlip: White-Box Adversarial Examples for Text Classification)
-******************************************************************************
 
 .. automodule:: textattack.attack_recipes.hotflip_ebrahimi_2017
    :members:
    :noindex:
 
-Improved Genetic Algorithm (Natural Language Adversarial Attacks and Defenses in Word Level)
-*************************************************************************************************
 
 .. automodule:: textattack.attack_recipes.iga_wang_2019
    :members:
    :noindex:
 
-Input Reduction (Pathologies of Neural Models Make Interpretations Difficult)
-************************************************************************************
 
 .. automodule:: textattack.attack_recipes.input_reduction_feng_2018
    :members:
    :noindex:
 
 
-Kuleshov (Adversarial Examples for Natural Language Classification Problems)
-******************************************************************************
 
 .. automodule:: textattack.attack_recipes.kuleshov_2017
    :members:
    :noindex:
 
-Particle Swarm Optimization (Word-level Textual Adversarial Attacking as Combinatorial Optimization)
-*****************************************************************************************************
+
 
 .. automodule:: textattack.attack_recipes.pso_zang_2020
    :members:
    :noindex:
 
-PWWS (Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency)
-***************************************************************************************************
+
 
 .. automodule:: textattack.attack_recipes.pwws_ren_2019
    :members:
    :noindex:
 
-TextFooler (Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment)
-************************************************************************************************************************
+
 
 .. automodule:: textattack.attack_recipes.textfooler_jin_2019
    :members:
    :noindex:
 
 
-TextBugger (TextBugger: Generating Adversarial Text Against Real-world Applications)
-****************************************************************************************
 
 .. automodule:: textattack.attack_recipes.textbugger_li_2018
    :members:
    :noindex:
 
+
 Attacks on sequence-to-sequence models
 ############################################
 
-MORPHEUS (It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations)
-*****************************************************************************************************
+15. MORPHEUS (It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations)
+16. Seq2Sick (Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples)
+
 
 .. automodule:: textattack.attack_recipes.morpheus_tan_2020
    :members:
    :noindex:
 
-Seq2Sick (Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples)
-*********************************************************************************************************
 
 .. automodule:: textattack.attack_recipes.seq2sick_cheng_2018_blackbox
    :members:

docs/3recipes/attack_recipes_cmd.md

@@ -0,0 +1,242 @@
+# Attack Recipes CommandLine Use
+
+We provide a number of pre-built attack recipes, which correspond to attacks from the literature. 
+
+
+## Help: `textattack --help`
+
+TextAttack's main features can all be accessed via the `textattack` command. Two very
+common commands are `textattack attack <args>`, and `textattack augment <args>`. You can see more
+information about all commands using 
+```bash
+textattack --help 
+```
+or a specific command using, for example,
+```bash
+textattack attack --help
+```
+
+The [`examples/`](https://github.com/QData/TextAttack/tree/master/examples) folder includes scripts showing common TextAttack usage for training models, running attacks, and augmenting a CSV file. 
+
+
+The [documentation website](https://textattack.readthedocs.io/en/latest) contains walkthroughs explaining basic usage of TextAttack, including building a custom transformation and a custom constraint..
+
+## Running Attacks: `textattack attack --help`
+
+The easiest way to try out an attack is via the command-line interface, `textattack attack`. 
+
+> **Tip:** If your machine has multiple GPUs, you can distribute the attack across them using the `--parallel` option. For some attacks, this can really help performance.
+
+Here are some concrete examples:
+
+*TextFooler on BERT trained on the MR sentiment classification dataset*: 
+```bash
+textattack attack --recipe textfooler --model bert-base-uncased-mr --num-examples 100
+```
+
+*DeepWordBug on DistilBERT trained on the Quora Question Pairs paraphrase identification dataset*: 
+```bash
+textattack attack --model distilbert-base-uncased-cola --recipe deepwordbug --num-examples 100
+```
+
+*Beam search with beam width 4 and word embedding transformation and untargeted goal function on an LSTM*:
+```bash
+textattack attack --model lstm-mr --num-examples 20 \
+ --search-method beam-search^beam_width=4 --transformation word-swap-embedding \
+ --constraints repeat stopword max-words-perturbed^max_num_words=2 embedding^min_cos_sim=0.8 part-of-speech \
+ --goal-function untargeted-classification
+```
+
+> **Tip:** Instead of specifying a dataset and number of examples, you can pass `--interactive` to attack samples inputted by the user.
+
+## Attacks and Papers Implemented ("Attack Recipes"): `textattack attack --recipe [recipe_name]`
+
+We include attack recipes which implement attacks from the literature. You can list attack recipes using `textattack list attack-recipes`.
+
+To run an attack recipe: `textattack attack --recipe [recipe_name]`
+
+
+<table  style="width:100%" border="1">
+<thead>
+<tr class="header">
+<th><strong>Attack Recipe Name</strong></th>
+<th><strong>Goal Function</strong></th>
+<th><strong>ConstraintsEnforced</strong></th>
+<th><strong>Transformation</strong></th>
+<th><strong>Search Method</strong></th>
+<th><strong>Main Idea</strong></th>
+</tr>
+</thead>
+<tbody>
+  <tr><td style="text-align: center;" colspan="6"><strong><br>Attacks on classification tasks, like sentiment classification and entailment:<br></strong></td></tr>
+
+<tr>
+<td><code>alzantot</code>  <span class="citation" data-cites="Alzantot2018GeneratingNL Jia2019CertifiedRT"></span></td>
+<td><sub>Untargeted {Classification, Entailment}</sub></td>
+<td><sub>Percentage of words perturbed, Language Model perplexity, Word embedding distance</sub></td>
+<td><sub>Counter-fitted word embedding swap</sub></td>
+<td><sub>Genetic Algorithm</sub></td>
+<td ><sub>from (["Generating Natural Language Adversarial Examples" (Alzantot et al., 2018)](https://arxiv.org/abs/1804.07998))</sub></td>
+</tr>
+<tr>
+<td><code>bae</code> <span class="citation" data-cites="garg2020bae"></span></td>
+<td><sub>Untargeted Classification</sub></td>
+<td><sub>USE sentence encoding cosine similarity</sub></td>
+<td><sub>BERT Masked Token Prediction</sub></td>
+<td><sub>Greedy-WIR</sub></td>
+<td ><sub>BERT masked language model transformation attack from (["BAE: BERT-based Adversarial Examples for Text Classification" (Garg & Ramakrishnan, 2019)](https://arxiv.org/abs/2004.01970)). </td>
+</tr>
+<tr>
+<td><code>bert-attack</code> <span class="citation" data-cites="li2020bertattack"></span></td>
+<td><sub>Untargeted Classification</td>
+<td><sub>USE sentence encoding cosine similarity, Maximum number of words perturbed</td>
+<td><sub>BERT Masked Token Prediction (with subword expansion)</td>
+<td><sub>Greedy-WIR</sub></td>
+<td ><sub> (["BERT-ATTACK: Adversarial Attack Against BERT Using BERT" (Li et al., 2020)](https://arxiv.org/abs/2004.09984))</sub></td>
+</tr>
+<tr>
+<td><code>checklist</code> <span class="citation" data-cites="Gao2018BlackBoxGO"></span></td>
+<td><sub>{Untargeted, Targeted} Classification</sub></td>
+<td><sub>checklist distance</sub></td>
+<td><sub>contract, extend, and substitutes name entities</sub></td>
+<td><sub>Greedy-WIR</sub></td>
+<td ><sub>Invariance testing implemented in CheckList . (["Beyond Accuracy: Behavioral Testing of NLP models with CheckList" (Ribeiro et al., 2020)](https://arxiv.org/abs/2005.04118))</sub></td>
+</tr>
+<tr>
+<td> <code>clare</code> <span class="citation" data-cites="Alzantot2018GeneratingNL Jia2019CertifiedRT"></span></td>
+<td><sub>Untargeted {Classification, Entailment}</sub></td>
+<td><sub>USE sentence encoding cosine similarity</sub></td>
+<td><sub>RoBERTa Masked Prediction for token swap, insert and merge</sub></td>
+<td><sub>Greedy</sub></td>
+<td ><sub>["Contextualized Perturbation for Textual Adversarial Attack" (Li et al., 2020)](https://arxiv.org/abs/2009.07502))</sub></td>
+</tr>
+<tr>
+<td><code>deepwordbug</code> <span class="citation" data-cites="Gao2018BlackBoxGO"></span></td>
+<td><sub>{Untargeted, Targeted} Classification</sub></td>
+<td><sub>Levenshtein edit distance</sub></td>
+<td><sub>{Character Insertion, Character Deletion, Neighboring Character Swap, Character Substitution}</sub></td>
+<td><sub>Greedy-WIR</sub></td>
+<td ><sub>Greedy replace-1 scoring and multi-transformation character-swap attack (["Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers" (Gao et al., 2018)](https://arxiv.org/abs/1801.04354)</sub></td>
+</tr>
+<tr>
+<td> <code>fast-alzantot</code> <span class="citation" data-cites="Alzantot2018GeneratingNL Jia2019CertifiedRT"></span></td>
+<td><sub>Untargeted {Classification, Entailment}</sub></td>
+<td><sub>Percentage of words perturbed, Language Model perplexity, Word embedding distance</sub></td>
+<td><sub>Counter-fitted word embedding swap</sub></td>
+<td><sub>Genetic Algorithm</sub></td>
+<td ><sub>Modified, faster version of the Alzantot et al. genetic algorithm, from (["Certified Robustness to Adversarial Word Substitutions" (Jia et al., 2019)](https://arxiv.org/abs/1909.00986))</sub></td>
+</tr>
+<tr>
+<td><code>hotflip</code> (word swap) <span class="citation" data-cites="Ebrahimi2017HotFlipWA"></span></td>
+<td><sub>Untargeted Classification</sub></td>
+<td><sub>Word Embedding Cosine Similarity, Part-of-speech match, Number of words perturbed</sub></td>
+<td><sub>Gradient-Based Word Swap</sub></td>
+<td><sub>Beam search</sub></td>
+<td ><sub> (["HotFlip: White-Box Adversarial Examples for Text Classification" (Ebrahimi et al., 2017)](https://arxiv.org/abs/1712.06751))</sub></td>
+</tr>
+<tr>
+<td><code>iga</code> <span class="citation" data-cites="iga-wang2019natural"></span></td>
+<td><sub>Untargeted {Classification, Entailment}</sub></td>
+<td><sub>Percentage of words perturbed, Word embedding distance</sub></td>
+<td><sub>Counter-fitted word embedding swap</sub></td>
+<td><sub>Genetic Algorithm</sub></td>
+<td ><sub>Improved genetic algorithm -based word substitution from (["Natural Language Adversarial Attacks and Defenses in Word Level (Wang et al., 2019)"](https://arxiv.org/abs/1909.06723)</sub></td>
+</tr>
+<tr>
+<td><code>input-reduction</code> <span class="citation" data-cites="feng2018pathologies"></span></td>
+<td><sub>Input Reduction</sub></td>
+<td></td>
+<td><sub>Word deletion</sub></td>
+<td><sub>Greedy-WIR</sub></td>
+<td ><sub>Greedy attack with word importance ranking , Reducing the input while maintaining the prediction through word importance ranking (["Pathologies of Neural Models Make Interpretation Difficult" (Feng et al., 2018)](https://arxiv.org/pdf/1804.07781.pdf))</sub></td>
+</tr>
+<tr>
+<td><code>kuleshov</code> <span class="citation" data-cites="Kuleshov2018AdversarialEF"></span></td>
+<td><sub>Untargeted Classification</sub></td>
+<td><sub>Thought vector encoding cosine similarity, Language model similarity probability</sub></td>
+<td><sub>Counter-fitted word embedding swap</sub></td>
+<td><sub>Greedy word swap</sub></td>
+<td ><sub>(["Adversarial Examples for Natural Language Classification Problems" (Kuleshov et al., 2018)](https://openreview.net/pdf?id=r1QZ3zbAZ)) </sub></td>
+</tr>
+<tr>
+<td><code>pruthi</code> <span class="citation" data-cites="pruthi2019combating"></span></td>
+<td><sub>Untargeted Classification</sub></td>
+<td><sub>Minimum word length, Maximum number of words perturbed</sub></td>
+<td><sub>{Neighboring Character Swap, Character Deletion, Character Insertion, Keyboard-Based Character Swap}</sub></td>
+<td><sub>Greedy search</sub></td>
+<td ><sub>simulates common typos (["Combating Adversarial Misspellings with Robust Word Recognition" (Pruthi et al., 2019)](https://arxiv.org/abs/1905.11268) </sub></td>
+</tr>
+<tr>
+<td><code>pso</code> <span class="citation" data-cites="pso-zang-etal-2020-word"></span></td>
+<td><sub>Untargeted Classification</sub></td>
+<td></td>
+<td><sub>HowNet Word Swap</sub></td>
+<td><sub>Particle Swarm Optimization</sub></td>
+<td ><sub>(["Word-level Textual Adversarial Attacking as Combinatorial Optimization" (Zang et al., 2020)](https://www.aclweb.org/anthology/2020.acl-main.540/)) </sub></td>
+</tr>
+<tr>
+<td><code>pwws</code> <span class="citation" data-cites="pwws-ren-etal-2019-generating"></span></td>
+<td><sub>Untargeted Classification</sub></td>
+<td></td>
+<td><sub>WordNet-based synonym swap</sub></td>
+<td><sub>Greedy-WIR (saliency)</sub></td>
+<td ><sub>Greedy attack with word importance ranking based on word saliency and synonym swap scores (["Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency" (Ren et al., 2019)](https://www.aclweb.org/anthology/P19-1103/))</sub> </td>
+</tr>
+<tr>
+<td><code>textbugger</code> : (black-box) <span class="citation" data-cites="Li2019TextBuggerGA"></span></td>
+<td><sub>Untargeted Classification</sub></td>
+<td><sub>USE sentence encoding cosine similarity</sub></td>
+<td><sub>{Character Insertion, Character Deletion, Neighboring Character Swap, Character Substitution}</sub></td>
+<td><sub>Greedy-WIR</sub></td>
+<td ><sub>([(["TextBugger: Generating Adversarial Text Against Real-world Applications" (Li et al., 2018)](https://arxiv.org/abs/1812.05271)).</sub></td>
+</tr>
+<tr>
+<td><code>textfooler</code> <span class="citation" data-cites="Jin2019TextFooler"></span></td>
+<td><sub>Untargeted {Classification, Entailment}</sub></td>
+<td><sub>Word Embedding Distance, Part-of-speech match, USE sentence encoding cosine similarity</sub></td>
+<td><sub>Counter-fitted word embedding swap</sub></td>
+<td><sub>Greedy-WIR</sub></td>
+<td ><sub>Greedy attack with word importance ranking  (["Is Bert Really Robust?" (Jin et al., 2019)](https://arxiv.org/abs/1907.11932))</sub> </td>
+</tr>
+
+<tr><td style="text-align: center;" colspan="6"><strong><br>Attacks on sequence-to-sequence models: <br></strong></td></tr>
+
+<tr>
+<td><code>morpheus</code> <span class="citation" data-cites="morpheus-tan-etal-2020-morphin"></span></td>
+<td><sub>Minimum BLEU Score</sub> </td>
+<td></td>
+<td><sub>Inflection Word Swap</sub> </td>
+<td><sub>Greedy search</sub> </td>
+<td ><sub>Greedy to replace words with their inflections with the goal of minimizing BLEU score (["It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations"](https://www.aclweb.org/anthology/2020.acl-main.263.pdf)</sub> </td>
+</tr>
+
+</tr>
+<tr>
+<td><code>seq2sick</code> :(black-box) <span class="citation" data-cites="cheng2018seq2sick"></span></td>
+<td><sub>Non-overlapping output</sub> </td>
+<td></td>
+<td><sub>Counter-fitted word embedding swap</sub> </td>
+<td><sub>Greedy-WIR</sub></td>
+<td ><sub>Greedy attack with goal of changing every word in the output translation. Currently implemented as black-box with plans to change to white-box as done in paper (["Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples" (Cheng et al., 2018)](https://arxiv.org/abs/1803.01128)) </sub>  </td>
+</tr>
+
+
+</tbody>
+</font>
+</table>
+
+
+
+## Recipe Usage Examples
+
+Here are some examples of testing attacks from the literature from the command-line:
+
+*TextFooler against BERT fine-tuned on SST-2:*
+```bash
+textattack attack --model bert-base-uncased-sst2 --recipe textfooler --num-examples 10
+```
+
+*seq2sick (black-box) against T5 fine-tuned for English-German translation:*
+```bash
+ textattack attack --model t5-en-de --recipe seq2sick --num-examples 100
+```

docs/3recipes/augmenter_recipes.rst

@@ -1,8 +1,36 @@
-======================
-Augmenter Recipes
-======================
+Augmenter Recipes API 
+=====================
 
-Transformations and constraints can be used for simple NLP data augmentations. Here is a list of recipes for NLP data augmentations
+Summary: Transformations and constraints can be used for simple NLP data augmentations. 
+
+In addition to the command-line interface, you can augment text dynamically by importing the
+`Augmenter` in your own code. All `Augmenter` objects implement `augment` and `augment_many` to generate augmentations
+of a string or a list of strings. Here's an example of how to use the `EmbeddingAugmenter` in a python script:
+
+.. code-block:: python
+
+   >>> from textattack.augmentation import EmbeddingAugmenter
+   >>> augmenter = EmbeddingAugmenter()
+   >>> s = 'What I cannot create, I do not understand.'
+   >>> augmenter.augment(s)
+   ['What I notable create, I do not understand.', 'What I significant create, I do not understand.', 'What I cannot engender, I do not understand.', 'What I cannot creating, I do not understand.', 'What I cannot creations, I do not understand.', 'What I cannot create, I do not comprehend.', 'What I cannot create, I do not fathom.', 'What I cannot create, I do not understanding.', 'What I cannot create, I do not understands.', 'What I cannot create, I do not understood.', 'What I cannot create, I do not realise.']
+
+You can also create your own augmenter from scratch by importing transformations/constraints from `textattack.transformations` and `textattack.constraints`. Here's an example that generates augmentations of a string using `WordSwapRandomCharacterDeletion`:
+
+.. code-block:: python
+
+   >>> from textattack.transformations import WordSwapRandomCharacterDeletion
+   >>> from textattack.transformations import CompositeTransformation
+   >>> from textattack.augmentation import Augmenter
+   >>> transformation = CompositeTransformation([WordSwapRandomCharacterDeletion()])
+   >>> augmenter = Augmenter(transformation=transformation, transformations_per_example=5)
+   >>> s = 'What I cannot create, I do not understand.'
+   >>> augmenter.augment(s)
+   ['What I cannot creae, I do not understand.', 'What I cannot creat, I do not understand.', 'What I cannot create, I do not nderstand.', 'What I cannot create, I do nt understand.', 'Wht I cannot create, I do not understand.']
+
+
+
+Here is a list of recipes for NLP data augmentations
 
 .. automodule:: textattack.augmentation.recipes
    :members:

docs/3recipes/augmenter_recipes_cmd.md

@@ -0,0 +1,67 @@
+# Augmenter Recipes CommandLine Use 
+
+Transformations and constraints can be used for simple NLP data augmentations. 
+
+The [`examples/`](https://github.com/QData/TextAttack/tree/master/examples) folder includes scripts showing common TextAttack usage for training models, running attacks, and augmenting a CSV file. 
+
+The [documentation website](https://textattack.readthedocs.io/en/latest) contains walkthroughs explaining basic usage of TextAttack, including building a custom transformation and a custom constraint..
+
+
+### Augmenting Text: `textattack augment`
+
+Many of the components of TextAttack are useful for data augmentation. The `textattack.Augmenter` class
+uses a transformation and a list of constraints to augment data. We also offer  built-in recipes
+for data augmentation:
+- `wordnet` augments text by replacing words with WordNet synonyms
+- `embedding` augments text by replacing words with neighbors in the counter-fitted embedding space, with a constraint to ensure their cosine similarity is at least 0.8
+- `charswap` augments text by substituting, deleting, inserting, and swapping adjacent characters
+- `eda` augments text with a combination of word insertions, substitutions and deletions.
+- `checklist` augments text by contraction/extension and by substituting names, locations, numbers.
+- `clare` augments text by replacing, inserting, and merging with a pre-trained masked language model.
+
+
+### Augmentation Command-Line Interface
+The easiest way to use our data augmentation tools is with `textattack augment <args>`. 
+
+`textattack augment`
+takes an input CSV file, the "text" column to augment, along with the number of words to change per augmentation
+and the number of augmentations per input example. It outputs a CSV in the same format with all the augmented examples in the proper columns.
+
+> For instance, when given the following as `examples.csv`:
+
+```
+"text",label
+"the rock is destined to be the 21st century's new conan and that he's going to make a splash even greater than arnold schwarzenegger , jean- claud van damme or steven segal.", 1
+"the gorgeously elaborate continuation of 'the lord of the rings' trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .", 1
+"take care of my cat offers a refreshingly different slice of asian cinema .", 1
+"a technically well-made suspenser . . . but its abrupt drop in iq points as it races to the finish line proves simply too discouraging to let slide .", 0
+"it's a mystery how the movie could be released in this condition .", 0
+```
+
+The command
+```
+textattack augment --input-csv examples.csv --output-csv output.csv  --input-column text --recipe embedding --pct-words-to-swap .1 --transformations-per-example 2 --exclude-original
+```
+will augment the `text` column by altering 10% of each example's words, generating twice as many augmentations as original inputs, and exclude the original inputs from the
+output CSV. (All of this will be saved to `augment.csv` by default.)
+
+> **Tip:** Just as running attacks interactively, you can also pass `--interactive` to augment samples inputted by the user to quickly try out different augmentation recipes!
+
+
+After augmentation, here are the contents of `augment.csv`:
+```
+text,label
+"the rock is destined to be the 21st century's newest conan and that he's gonna to make a splashing even stronger than arnold schwarzenegger , jean- claud van damme or steven segal.",1
+"the rock is destined to be the 21tk century's novel conan and that he's going to make a splat even greater than arnold schwarzenegger , jean- claud van damme or stevens segal.",1
+the gorgeously elaborate continuation of 'the lord of the rings' trilogy is so huge that a column of expression significant adequately describe co-writer/director pedro jackson's expanded vision of j . rs . r . tolkien's middle-earth .,1
+the gorgeously elaborate continuation of 'the lordy of the piercings' trilogy is so huge that a column of mots cannot adequately describe co-novelist/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .,1
+take care of my cat offerings a pleasantly several slice of asia cinema .,1
+taking care of my cat offers a pleasantly different slice of asiatic kino .,1
+a technically good-made suspenser . . . but its abrupt drop in iq points as it races to the finish bloodline proves straightforward too disheartening to let slide .,0
+a technically well-made suspenser . . . but its abrupt drop in iq dot as it races to the finish line demonstrates simply too disheartening to leave slide .,0
+it's a enigma how the film wo be releases in this condition .,0
+it's a enigma how the filmmaking wo be publicized in this condition .,0
+```
+
+The 'embedding' augmentation recipe uses counterfitted embedding nearest-neighbors to augment data.
+

docs/3recipes/models.md

@@ -0,0 +1,496 @@
+# TextAttack Model Zoo
+
+TextAttack is model-agnostic - meaning it can run attacks on models implemented in any deep learning framework. Model objects must be able to take a string (or list of strings) and return an output that can be processed by the goal function. For example, machine translation models take a list of strings as input and produce a list of strings as output. Classification and entailment models return an array of scores. As long as the user's model meets this specification, the model is fit to use with TextAttack.
+
+
+
+To help users, TextAttack includes pre-trained models for different common NLP tasks. This makes it easier for
+users to get started with TextAttack. It also enables a more fair comparison of attacks from
+the literature.
+
+
+## Available Models
+
+### TextAttack Models
+TextAttack has two build-in model types, a 1-layer bidirectional LSTM with a hidden
+state size of 150 (`lstm`), and a WordCNN with 3 window sizes
+(3, 4, 5) and 100 filters for the window size (`cnn`). Both models set dropout
+to 0.3 and use a base of the 200-dimensional GLoVE embeddings.
+
+### `transformers` Models
+Along with the `lstm` and `cnn`, you can theoretically fine-tune any model based
+in the huggingface [transformers](https://github.com/huggingface/transformers/)
+repo. Just type the model name (like `bert-base-cased`) and it will be automatically
+loaded.
+
+Here are some models from transformers that have worked well for us:
+- `bert-base-uncased` and `bert-base-cased`
+- `distilbert-base-uncased` and `distilbert-base-cased`
+- `albert-base-v2`
+- `roberta-base`
+- `xlnet-base-cased`
+
+
+## Evaluation Results of Available Models
+
+All evaluation results were obtained using `textattack eval` to evaluate models on their default
+test dataset (test set, if labels are available, otherwise, eval/validation set). You can use
+this command to verify the accuracies for yourself: for example, `textattack eval --model roberta-base-mr`.
+
+
+The LSTM and wordCNN models' code is available in `textattack.models.helpers`. All other models are transformers
+imported from the [`transformers`](https://github.com/huggingface/transformers/) package. To list evaluate all 
+TextAttack pretrained models, invoke `textattack eval` without specifying a model: `textattack eval --num-examples 1000`.
+All evaluations shown are on the full validation or test set up to 1000 examples.
+
+
+### `LSTM`
+
+<section>
+
+- AG News (`lstm-ag-news`)
+    - `datasets` dataset `ag_news`, split `test`
+        - Correct/Whole: 914/1000
+        - Accuracy: 91.4%
+- IMDB (`lstm-imdb`)
+    - `datasets` dataset `imdb`, split `test`
+        - Correct/Whole: 883/1000
+        - Accuracy: 88.30%
+- Movie Reviews [Rotten Tomatoes] (`lstm-mr`)
+    - `datasets` dataset `rotten_tomatoes`, split `validation`
+        - Correct/Whole: 807/1000 
+        - Accuracy: 80.70%
+    - `datasets` dataset `rotten_tomatoes`, split `test`
+        - Correct/Whole: 781/1000
+        - Accuracy: 78.10%
+- SST-2 (`lstm-sst2`)
+    - `datasets` dataset `glue`, subset `sst2`, split `validation`
+        - Correct/Whole: 737/872 
+        - Accuracy: 84.52%
+- Yelp Polarity (`lstm-yelp`)
+    - `datasets` dataset `yelp_polarity`, split `test`
+        - Correct/Whole: 922/1000
+        - Accuracy: 92.20%
+
+</section> 
+
+### `wordCNN`
+
+<section>
+
+
+- AG News (`cnn-ag-news`)
+    - `datasets` dataset `ag_news`, split `test`
+        - Correct/Whole: 910/1000
+        - Accuracy: 91.00%
+- IMDB (`cnn-imdb`)
+    - `datasets` dataset `imdb`, split `test`
+        - Correct/Whole: 863/1000
+        - Accuracy: 86.30%
+- Movie Reviews [Rotten Tomatoes] (`cnn-mr`)
+    - `datasets` dataset `rotten_tomatoes`, split `validation`
+        - Correct/Whole: 794/1000 
+        - Accuracy: 79.40%
+    - `datasets` dataset `rotten_tomatoes`, split `test`
+        - Correct/Whole: 768/1000
+        - Accuracy: 76.80%
+- SST-2 (`cnn-sst2`)
+    - `datasets` dataset `glue`, subset `sst2`, split `validation`
+        - Correct/Whole: 721/872 
+        - Accuracy: 82.68%
+- Yelp Polarity (`cnn-yelp`)
+    - `datasets` dataset `yelp_polarity`, split `test`
+        - Correct/Whole: 913/1000
+        - Accuracy: 91.30%
+
+</section>
+
+
+### `albert-base-v2`
+
+<section>
+
+- AG News (`albert-base-v2-ag-news`)
+    - `datasets` dataset `ag_news`, split `test`
+        - Correct/Whole: 943/1000
+        - Accuracy: 94.30%
+- CoLA (`albert-base-v2-cola`)
+    - `datasets` dataset `glue`, subset `cola`, split `validation`
+        - Correct/Whole: 829/1000
+        - Accuracy: 82.90%
+- IMDB (`albert-base-v2-imdb`)
+    - `datasets` dataset `imdb`, split `test`
+        - Correct/Whole: 913/1000
+        - Accuracy: 91.30%
+- Movie Reviews [Rotten Tomatoes] (`albert-base-v2-mr`)
+    - `datasets` dataset `rotten_tomatoes`, split `validation`
+        - Correct/Whole: 882/1000
+        - Accuracy: 88.20%
+    - `datasets` dataset `rotten_tomatoes`, split `test`
+        - Correct/Whole: 851/1000
+        - Accuracy: 85.10%
+- Quora Question Pairs (`albert-base-v2-qqp`)
+    - `datasets` dataset `glue`, subset `qqp`, split `validation`
+        - Correct/Whole: 914/1000
+        - Accuracy: 91.40%
+- Recognizing Textual Entailment (`albert-base-v2-rte`)
+    - `datasets` dataset `glue`, subset `rte`, split `validation`
+        - Correct/Whole: 211/277 
+        - Accuracy: 76.17%
+- SNLI (`albert-base-v2-snli`)
+    - `datasets` dataset `snli`, split `test`
+        - Correct/Whole: 883/1000
+        - Accuracy: 88.30%
+- SST-2 (`albert-base-v2-sst2`)
+    - `datasets` dataset `glue`, subset `sst2`, split `validation`
+        - Correct/Whole: 807/872
+        - Accuracy: 92.55%)
+- STS-b (`albert-base-v2-stsb`)
+    - `datasets` dataset `glue`, subset `stsb`, split `validation`
+    - Pearson correlation: 0.9041359738552746
+    - Spearman correlation: 0.8995912861209745
+- WNLI (`albert-base-v2-wnli`)
+    - `datasets` dataset `glue`, subset `wnli`, split `validation`
+        - Correct/Whole: 42/71
+        - Accuracy: 59.15%
+- Yelp Polarity (`albert-base-v2-yelp`)
+    - `datasets` dataset `yelp_polarity`, split `test`
+        - Correct/Whole: 963/1000
+        - Accuracy: 96.30%
+
+</section>
+
+### `bert-base-uncased`
+
+<section>
+
+- AG News (`bert-base-uncased-ag-news`)
+    - `datasets` dataset `ag_news`, split `test`
+        - Correct/Whole: 942/1000
+        - Accuracy: 94.20%
+- CoLA (`bert-base-uncased-cola`)
+    - `datasets` dataset `glue`, subset `cola`, split `validation`
+        - Correct/Whole: 812/1000
+        - Accuracy: 81.20%
+- IMDB (`bert-base-uncased-imdb`)
+    - `datasets` dataset `imdb`, split `test`
+        - Correct/Whole: 919/1000
+        - Accuracy: 91.90%
+- MNLI matched (`bert-base-uncased-mnli`)
+    - `datasets` dataset `glue`, subset `mnli`, split `validation_matched`
+        - Correct/Whole: 840/1000
+        - Accuracy: 84.00%
+- Movie Reviews [Rotten Tomatoes] (`bert-base-uncased-mr`)
+    - `datasets` dataset `rotten_tomatoes`, split `validation`
+        - Correct/Whole: 876/1000
+        - Accuracy: 87.60%
+    - `datasets` dataset `rotten_tomatoes`, split `test`
+        - Correct/Whole: 838/1000
+        - Accuracy: 83.80%
+- MRPC (`bert-base-uncased-mrpc`)
+    - `datasets` dataset `glue`, subset `mrpc`, split `validation`
+        - Correct/Whole: 358/408
+        - Accuracy: 87.75%
+- QNLI (`bert-base-uncased-qnli`)
+    - `datasets` dataset `glue`, subset `qnli`, split `validation`
+        - Correct/Whole: 904/1000
+        - Accuracy: 90.40%
+- Quora Question Pairs (`bert-base-uncased-qqp`)
+    - `datasets` dataset `glue`, subset `qqp`, split `validation`
+        - Correct/Whole: 924/1000
+        - Accuracy: 92.40%
+- Recognizing Textual Entailment (`bert-base-uncased-rte`)
+    - `datasets` dataset `glue`, subset `rte`, split `validation`
+        - Correct/Whole: 201/277 
+        - Accuracy: 72.56%
+- SNLI (`bert-base-uncased-snli`)
+    - `datasets` dataset `snli`, split `test`
+        - Correct/Whole: 894/1000
+        - Accuracy: 89.40%
+- SST-2 (`bert-base-uncased-sst2`)
+    - `datasets` dataset `glue`, subset `sst2`, split `validation`
+        - Correct/Whole: 806/872
+        - Accuracy: 92.43%)
+- STS-b (`bert-base-uncased-stsb`)
+    - `datasets` dataset `glue`, subset `stsb`, split `validation`
+    - Pearson correlation: 0.8775458937815515
+    - Spearman correlation: 0.8773251339980935
+- WNLI (`bert-base-uncased-wnli`)
+    - `datasets` dataset `glue`, subset `wnli`, split `validation`
+        - Correct/Whole: 40/71
+        - Accuracy: 56.34%
+- Yelp Polarity (`bert-base-uncased-yelp`)
+    - `datasets` dataset `yelp_polarity`, split `test`
+        - Correct/Whole: 963/1000
+        - Accuracy: 96.30%
+
+</section>
+
+### `distilbert-base-cased`
+
+<section>
+
+
+- CoLA (`distilbert-base-cased-cola`)
+    - `datasets` dataset `glue`, subset `cola`, split `validation`
+        - Correct/Whole: 786/1000
+        - Accuracy: 78.60%
+- MRPC (`distilbert-base-cased-mrpc`)
+    - `datasets` dataset `glue`, subset `mrpc`, split `validation`
+        - Correct/Whole: 320/408
+        - Accuracy: 78.43%
+- Quora Question Pairs (`distilbert-base-cased-qqp`)
+    - `datasets` dataset `glue`, subset `qqp`, split `validation`
+        - Correct/Whole: 908/1000
+        - Accuracy: 90.80%
+- SNLI (`distilbert-base-cased-snli`)
+    - `datasets` dataset `snli`, split `test`
+        - Correct/Whole: 861/1000
+        - Accuracy: 86.10%
+- SST-2 (`distilbert-base-cased-sst2`)
+    - `datasets` dataset `glue`, subset `sst2`, split `validation`
+        - Correct/Whole: 785/872
+        - Accuracy: 90.02%)
+- STS-b (`distilbert-base-cased-stsb`)
+    - `datasets` dataset `glue`, subset `stsb`, split `validation`
+    - Pearson correlation: 0.8421540899520146
+    - Spearman correlation: 0.8407155030382939
+
+</section>
+
+### `distilbert-base-uncased`
+
+<section>
+
+- AG News (`distilbert-base-uncased-ag-news`)
+    - `datasets` dataset `ag_news`, split `test`
+        - Correct/Whole: 944/1000
+        - Accuracy: 94.40%
+- CoLA (`distilbert-base-uncased-cola`)
+    - `datasets` dataset `glue`, subset `cola`, split `validation`
+        - Correct/Whole: 786/1000
+        - Accuracy: 78.60%
+- IMDB (`distilbert-base-uncased-imdb`)
+    - `datasets` dataset `imdb`, split `test`
+        - Correct/Whole: 903/1000
+        - Accuracy: 90.30%
+- MNLI matched (`distilbert-base-uncased-mnli`)
+    - `datasets` dataset `glue`, subset `mnli`, split `validation_matched`
+        - Correct/Whole: 817/1000
+        - Accuracy: 81.70%
+- MRPC (`distilbert-base-uncased-mrpc`)
+    - `datasets` dataset `glue`, subset `mrpc`, split `validation`
+        - Correct/Whole: 350/408
+        - Accuracy: 85.78%
+- QNLI (`distilbert-base-uncased-qnli`)
+    - `datasets` dataset `glue`, subset `qnli`, split `validation`
+        - Correct/Whole: 860/1000
+        - Accuracy: 86.00%
+- Recognizing Textual Entailment (`distilbert-base-uncased-rte`)
+    - `datasets` dataset `glue`, subset `rte`, split `validation`
+        - Correct/Whole: 180/277 
+        - Accuracy: 64.98%
+- STS-b (`distilbert-base-uncased-stsb`)
+    - `datasets` dataset `glue`, subset `stsb`, split `validation`
+    - Pearson correlation: 0.8421540899520146
+    - Spearman correlation: 0.8407155030382939
+- WNLI (`distilbert-base-uncased-wnli`)
+    - `datasets` dataset `glue`, subset `wnli`, split `validation`
+        - Correct/Whole: 40/71
+        - Accuracy: 56.34%
+
+</section>
+
+### `roberta-base`
+
+<section>
+
+- AG News (`roberta-base-ag-news`)
+    - `datasets` dataset `ag_news`, split `test`
+        - Correct/Whole: 947/1000
+        - Accuracy: 94.70%
+- CoLA (`roberta-base-cola`)
+    - `datasets` dataset `glue`, subset `cola`, split `validation`
+        - Correct/Whole: 857/1000
+        - Accuracy: 85.70%
+- IMDB (`roberta-base-imdb`)
+    - `datasets` dataset `imdb`, split `test`
+        - Correct/Whole: 941/1000
+        - Accuracy: 94.10%
+- Movie Reviews [Rotten Tomatoes] (`roberta-base-mr`)
+    - `datasets` dataset `rotten_tomatoes`, split `validation`
+        - Correct/Whole: 899/1000
+        - Accuracy: 89.90%
+    - `datasets` dataset `rotten_tomatoes`, split `test`
+        - Correct/Whole: 883/1000
+        - Accuracy: 88.30%
+- MRPC (`roberta-base-mrpc`)
+    - `datasets` dataset `glue`, subset `mrpc`, split `validation`
+        - Correct/Whole: 371/408
+        - Accuracy: 91.18%
+- QNLI (`roberta-base-qnli`)
+    - `datasets` dataset `glue`, subset `qnli`, split `validation`
+        - Correct/Whole: 917/1000
+        - Accuracy: 91.70%
+- Recognizing Textual Entailment (`roberta-base-rte`)
+    - `datasets` dataset `glue`, subset `rte`, split `validation`
+        - Correct/Whole: 217/277 
+        - Accuracy: 78.34%
+- SST-2 (`roberta-base-sst2`)
+    - `datasets` dataset `glue`, subset `sst2`, split `validation`
+        - Correct/Whole: 820/872
+        - Accuracy: 94.04%)
+- STS-b (`roberta-base-stsb`)
+    - `datasets` dataset `glue`, subset `stsb`, split `validation`
+    - Pearson correlation: 0.906067852162708
+    - Spearman correlation: 0.9025045272903051
+- WNLI (`roberta-base-wnli`)
+    - `datasets` dataset `glue`, subset `wnli`, split `validation`
+        - Correct/Whole: 40/71
+        - Accuracy: 56.34%
+
+</section>
+
+### `xlnet-base-cased`
+
+<section>
+
+- CoLA (`xlnet-base-cased-cola`)
+    - `datasets` dataset `glue`, subset `cola`, split `validation`
+        - Correct/Whole: 800/1000
+        - Accuracy: 80.00%
+- IMDB (`xlnet-base-cased-imdb`)
+    - `datasets` dataset `imdb`, split `test`
+        - Correct/Whole: 957/1000
+        - Accuracy: 95.70%
+- Movie Reviews [Rotten Tomatoes] (`xlnet-base-cased-mr`)
+    - `datasets` dataset `rotten_tomatoes`, split `validation`
+        - Correct/Whole: 908/1000
+        - Accuracy: 90.80%
+    - `datasets` dataset `rotten_tomatoes`, split `test`
+        - Correct/Whole: 876/1000
+        - Accuracy: 87.60%
+- MRPC (`xlnet-base-cased-mrpc`)
+    - `datasets` dataset `glue`, subset `mrpc`, split `validation`
+        - Correct/Whole: 363/408
+        - Accuracy: 88.97%
+- Recognizing Textual Entailment (`xlnet-base-cased-rte`)
+    - `datasets` dataset `glue`, subset `rte`, split `validation`
+        - Correct/Whole: 196/277 
+        - Accuracy: 70.76%
+- STS-b (`xlnet-base-cased-stsb`)
+    - `datasets` dataset `glue`, subset `stsb`, split `validation`
+    - Pearson correlation: 0.883111673280641
+    - Spearman correlation: 0.8773439961182335
+- WNLI (`xlnet-base-cased-wnli`)
+    - `datasets` dataset `glue`, subset `wnli`, split `validation`
+        - Correct/Whole: 41/71
+        - Accuracy: 57.75%
+
+</section>
+
+
+## How we have trained the TextAttack Models
+
+
+- By Oct 2020, TextAttack provides users with 82 pre-trained TextAttack models, including word-level LSTM, word-level CNN, BERT, and other transformer based models pre-trained on various datasets provided by [HuggingFace](https://github.com/huggingface/nlp/). 
+
+- Since TextAttack is integrated with the  [https://github.com/huggingface/nlp/](https://github.com/huggingface/nlp) library, it can automatically load the test or validation data set for the corresponding pre-trained model. While the literature has mainly focused on classification and entailment, TextAttack's pretrained models enable research on the robustness of models across all GLUE tasks.
+
+- We host all TextAttack Models at huggingface Model Hub: [https://huggingface.co/textattack](https://huggingface.co/textattack)
+
+
+## Training details for each TextAttack Model 
+
+
+All of our models have model cards on the HuggingFace model hub. So for now, the easiest way to figure this out is as follows:
+
+
+- Please Go to our page on the model hub: [https://huggingface.co/textattack](https://huggingface.co/textattack)
+
+- Find the model you're looking for and select its page, for instance: [https://huggingface.co/textattack/roberta-base-imdb](https://huggingface.co/textattack/roberta-base-imdb)
+
+- Scroll down to the end of the page, looking for **model card** section. Here it is the details of the model training for that specific TextAttack model. 
+
+- BTW: For each of our transformers, we selected the best out of a grid search over a bunch of possible hyperparameters. So the model training hyperparemeter actually varies from model to model.
+
+
+
+
+## More details on TextAttack fine-tuned NLP models (details on target NLP task, input type, output type, SOTA results on paperswithcode; model card on huggingface):
+
+
+
+
+Fine-tuned Model                         |  NLP Task                                       |  Input type                                   |  Output Type                                        |  paperswithcode.com SOTA                                                       |  huggingface.co Model Card
+--------------|-----------------|--------------------|--------------------|--------------------------|-------------------------------
+albert-base-v2-CoLA                      |  linguistic acceptability                       |  single sentences                             |  binary (1=acceptable/ 0=unacceptable)               |  <sub><sup>https://paperswithcode.com/sota/linguistic-acceptability-on-cola  </sub></sup>            |  <sub><sup>https://huggingface.co/textattack/albert-base-v2-CoLA </sub></sup>
+bert-base-uncased-CoLA                   |  linguistic acceptability                       |  single sentences                             |  binary (1=acceptable/ 0=unacceptable)               |  none yet                                                                      |  <sub><sup>https://huggingface.co/textattack/bert-base-uncased-CoLA </sub></sup>
+distilbert-base-cased-CoLA               |  linguistic acceptability                       |  single sentences                             |  binary (1=acceptable/ 0=unacceptable)               | <sub><sup> https://paperswithcode.com/sota/linguistic-acceptability-on-cola  </sub></sup>            |  <sub><sup>https://huggingface.co/textattack/distilbert-base-cased-CoLA </sub></sup>
+distilbert-base-uncased-CoLA             |  linguistic acceptability                       |  single sentences                             |  binary (1=acceptable/ 0=unacceptable)               | <sub><sup> https://paperswithcode.com/sota/linguistic-acceptability-on-cola     </sub></sup>         |  <sub><sup>https://huggingface.co/textattack/distilbert-base-uncased-CoLA </sub></sup>
+roberta-base-CoLA                        |  linguistic acceptability                       |  single sentences                             |  binary (1=acceptable/ 0=unacceptable)               | <sub><sup> https://paperswithcode.com/sota/linguistic-acceptability-on-cola  </sub></sup>            | <sub><sup> https://huggingface.co/textattack/roberta-base-CoLA </sub></sup>
+xlnet-base-cased-CoLA                    |  linguistic acceptability                       |  single sentences                             |  binary (1=acceptable/ 0=unacceptable)               | <sub><sup> https://paperswithcode.com/sota/linguistic-acceptability-on-cola   </sub></sup>            |  <sub><sup>https://huggingface.co/textattack/xlnet-base-cased-CoLA </sub></sup> 
+albert-base-v2-RTE                       |  natural language inference                     |  sentence pairs (1 premise and 1 hypothesis)  |  binary(0=entailed/1=not entailed)                  | <sub><sup> https://paperswithcode.com/sota/natural-language-inference-on-rte     </sub></sup>         | <sub><sup> https://huggingface.co/textattack/albert-base-v2-RTE </sub></sup> 
+albert-base-v2-snli                      |  natural language inference                     |  sentence pairs                               |  accuracy (0=entailment, 1=neutral,2=contradiction)  |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/albert-base-v2-snli </sub></sup>  
+albert-base-v2-WNLI                      |  natural language inference                     |  sentence pairs                               |  binary                                             | <sub><sup>  https://paperswithcode.com/sota/natural-language-inference-on-wnli  </sub></sup>           | <sub><sup> https://huggingface.co/textattack/albert-base-v2-WNLI</sub></sup> 
+bert-base-uncased-MNLI                   |  natural language inference                     |  sentence pairs (1 premise and 1 hypothesis)  |  accuracy (0=entailment, 1=neutral,2=contradiction)  |  none yet                                                                      |  <sub><sup> https://huggingface.co/textattack/bert-base-uncased-MNLI  </sub></sup>
+bert-base-uncased-QNLI                   |  natural language inference                     |  question/answer pairs                        |  binary (1=unanswerable/ 0=answerable)               |  none yet                                                                      |<sub><sup>  https://huggingface.co/textattack/bert-base-uncased-QNLI </sub></sup>
+bert-base-uncased-RTE                    |  natural language inference                     |  sentence pairs (1 premise and 1 hypothesis)  |  binary(0=entailed/1=not entailed)                  |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/bert-base-uncased-RTE </sub></sup>
+bert-base-uncased-snli                   |  natural language inference                     |  sentence pairs                               |  accuracy (0=entailment, 1=neutral,2=contradiction)  |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/bert-base-uncased-snli </sub></sup>
+bert-base-uncased-WNLI                   |  natural language inference                     |  sentence pairs                               |  binary                                             |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/bert-base-uncased-WNLI </sub></sup>
+distilbert-base-cased-snli               |  natural language inference                     |  sentence pairs                               |  accuracy (0=entailment, 1=neutral,2=contradiction)  |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/distilbert-base-cased-snli </sub></sup>
+distilbert-base-uncased-MNLI             |  natural language inference                     |  sentence pairs (1 premise and 1 hypothesis)  |  accuracy (0=entailment,1=neutral, 2=contradiction)  |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/distilbert-base-uncased-MNLI </sub></sup>
+distilbert-base-uncased-RTE              |  natural language inference                     |  sentence pairs (1 premise and 1 hypothesis)  |  binary(0=entailed/1=not entailed)                  | <sub><sup> https://paperswithcode.com/sota/natural-language-inference-on-rte   </sub></sup>          | <sub><sup> https://huggingface.co/textattack/distilbert-base-uncased-RTE</sub></sup>
+distilbert-base-uncased-WNLI             |  natural language inference                     |  sentence pairs                               |  binary                                             | <sub><sup> https://paperswithcode.com/sota/natural-language-inference-on-wnli    </sub></sup>        | <sub><sup> https://huggingface.co/textattack/distilbert-base-uncased-WNLI </sub></sup>
+roberta-base-QNLI                        |  natural language inference                     |  question/answer pairs                        |  binary (1=unanswerable/ 0=answerable)               | <sub><sup> https://paperswithcode.com/sota/natural-language-inference-on-qnli   </sub></sup>         | <sub><sup>  https://huggingface.co/textattack/roberta-base-QNLI </sub></sup>
+roberta-base-RTE                         |  natural language inference                     |  sentence pairs (1 premise and 1 hypothesis)  |  binary(0=entailed/1=not entailed)                  | <sub><sup>  https://paperswithcode.com/sota/natural-language-inference-on-rte  </sub></sup>           | <sub><sup> https://huggingface.co/textattack/roberta-base-RTE</sub></sup>
+roberta-base-WNLI                        |  natural language inference                     |  sentence pairs                               |  binary                                             | <sub><sup> https://paperswithcode.com/sota/natural-language-inference-on-wnli </sub></sup>           |  https://huggingface.co/textattack/roberta-base-WNLI </sub></sup>
+xlnet-base-cased-RTE                     |  natural language inference                     |  sentence pairs (1 premise and 1 hypothesis)  |  binary(0=entailed/1=not entailed)                  | <sub><sup>  https://paperswithcode.com/sota/ </sub></sup>natural-language-inference-on-rte             | <sub><sup> https://huggingface.co/textattack/xlnet-base-cased-RTE </sub></sup>
+xlnet-base-cased-WNLI                    |  natural language inference                     |  sentence pairs                               |  binary                                             |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/xlnet-base-cased-WNLI </sub></sup>
+albert-base-v2-QQP                       |  paraphase similarity                           |  question pairs                               |  binary (1=similar/0=not similar)                   | <sub><sup> https://paperswithcode.com/sota/question-answering-on-quora-question-pairs  </sub></sup>  | <sub><sup> https://huggingface.co/textattack/albert-base-v2-QQP</sub></sup>
+bert-base-uncased-QQP                    |  paraphase similarity                           |  question pairs                               |  binary (1=similar/0=not similar)                   | <sub><sup> https://paperswithcode.com/sota/question-answering-on-quora-question-pairs  </sub></sup>  | <sub><sup> https://huggingface.co/textattack/bert-base-uncased-QQP </sub></sup>
+distilbert-base-uncased-QNLI             |  question answering/natural language inference  |  question/answer pairs                        |  binary (1=unanswerable/ 0=answerable)               | <sub><sup> https://paperswithcode.com/sota/natural-language-inference-on-qnli   </sub></sup>         | <sub><sup> https://huggingface.co/textattack/distilbert-base-uncased-QNLI </sub></sup>
+distilbert-base-cased-QQP                |  question answering/paraphase similarity        |  question pairs                               |  binary (1=similar/ 0=not similar)                   | <sub><sup> https://paperswithcode.com/sota/question-answering-on-quora-question-pairs  </sub></sup>  | <sub><sup> https://huggingface.co/textattack/distilbert-base-cased-QQP </sub></sup>
+albert-base-v2-STS-B                     |  semantic textual similarity                    |  sentence pairs                               |  similarity (0.0 to 5.0)                            | <sub><sup> https://paperswithcode.com/sota/semantic-textual-similarity-on-sts-benchmark </sub></sup> | <sub><sup> https://huggingface.co/textattack/albert-base-v2-STS-B </sub></sup>
+bert-base-uncased-MRPC                   |  semantic textual similarity                    |  sentence pairs                               |  binary (1=similar/0=not similar)                   |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/bert-base-uncased-MRPC </sub></sup>
+bert-base-uncased-STS-B                  |  semantic textual similarity                    |  sentence pairs                               |  similarity (0.0 to 5.0)                            |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/bert-base-uncased-STS-B </sub></sup>
+distilbert-base-cased-MRPC               |  semantic textual similarity                    |  sentence pairs                               |  binary (1=similar/0=not similar)                   | <sub><sup> https://paperswithcode.com/sota/semantic-textual-similarity-on-mrpc  </sub></sup>         | <sub><sup> https://huggingface.co/textattack/distilbert-base-cased-MRPC </sub></sup>
+distilbert-base-cased-STS-B              |  semantic textual similarity                    |  sentence pairs                               |  similarity (0.0 to 5.0)                            | <sub><sup> https://paperswithcode.com/sota/semantic-textual-similarity-on-sts-benchmark </sub></sup> | <sub><sup> https://huggingface.co/textattack/distilbert-base-cased-STS-B </sub></sup>
+distilbert-base-uncased-MRPC             |  semantic textual similarity                    |  sentence pairs                               |  binary (1=similar/0=not similar)                   | <sub><sup> https://paperswithcode.com/sota/semantic-textual-similarity-on-mrpc   </sub></sup>        | <sub><sup> https://huggingface.co/textattack/distilbert-base-uncased-MRPC</sub></sup>
+roberta-base-MRPC                        |  semantic textual similarity                    |  sentence pairs                               |  binary (1=similar/0=not similar)                   | <sub><sup> https://paperswithcode.com/sota/semantic-textual-similarity-on-mrpc </sub></sup>          | <sub><sup> https://huggingface.co/textattack/roberta-base-MRPC </sub></sup>
+roberta-base-STS-B                       |  semantic textual similarity                    |  sentence pairs                               |  similarity (0.0 to 5.0)                            | <sub><sup> https://paperswithcode.com/sota/semantic-textual-similarity-on-sts-benchmark </sub></sup> | <sub><sup> https://huggingface.co/textattack/roberta-base-STS-B </sub></sup>
+xlnet-base-cased-MRPC                    |  semantic textual similarity                    |  sentence pairs                               |  binary (1=similar/0=not similar)                   | <sub><sup> https://paperswithcode.com/sota/semantic-textual-similarity-on-mrpc </sub></sup>          | <sub><sup> https://huggingface.co/textattack/xlnet-base-cased-MRPC </sub></sup>
+xlnet-base-cased-STS-B                   |  semantic textual similarity                    |  sentence pairs                               |  similarity (0.0 to 5.0)                            | <sub><sup> https://paperswithcode.com/sota/semantic-textual-similarity-on-sts-benchmark </sub></sup> | <sub><sup> https://huggingface.co/textattack/xlnet-base-cased-STS-B </sub></sup>
+albert-base-v2-imdb                      |  sentiment analysis                             |  movie reviews                                |  binary (1=good/0=bad)                              |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/albert-base-v2-imdb </sub></sup>
+albert-base-v2-rotten-tomatoes           |  sentiment analysis                             |  movie reviews                                |  binary (1=good/0=bad)                              |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/albert-base-v2-rotten-tomatoes </sub></sup>
+albert-base-v2-SST-2                     |  sentiment analysis                             |  phrases                                      |  accuracy (0.0000 to 1.0000)                        | <sub><sup> https://paperswithcode.com/sota/sentiment-analysis-on-sst-2-binary  </sub></sup>          | <sub><sup> https://huggingface.co/textattack/albert-base-v2-SST-2 </sub></sup>
+albert-base-v2-yelp-polarity             |  sentiment analysis                             |  yelp reviews                                 |  binary (1=good/0=bad)                              |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/albert-base-v2-yelp-polarity </sub></sup>
+bert-base-uncased-imdb                   |  sentiment analysis                             |  movie reviews                                |  binary (1=good/0=bad)                              |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/bert-base-uncased-imdb </sub></sup>
+bert-base-uncased-rotten-tomatoes        |  sentiment analysis                             |  movie reviews                                |  binary (1=good/0=bad)                              |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/bert-base-uncased-rotten-tomatoes </sub></sup>
+bert-base-uncased-SST-2                  |  sentiment analysis                             |  phrases                                      |  accuracy (0.0000 to 1.0000)                        | <sub><sup> https://paperswithcode.com/sota/sentiment-analysis-on-sst-2-binary </sub></sup>           | <sub><sup> https://huggingface.co/textattack/bert-base-uncased-SST-2 </sub></sup>
+bert-base-uncased-yelp-polarity          |  sentiment analysis                             |  yelp reviews                                 |  binary (1=good/0=bad)                              | <sub><sup> https://paperswithcode.com/sota/sentiment-analysis-on-yelp-binary  </sub></sup>           | <sub><sup> https://huggingface.co/textattack/bert-base-uncased-yelp-polarity </sub></sup>
+cnn-imdb                                 |  sentiment analysis                             |  movie reviews                                |  binary (1=good/0=bad)                              | <sub><sup> https://paperswithcode.com/sota/sentiment-analysis-on-imdb  </sub></sup>                  |  none
+cnn-mr                                   |  sentiment analysis                             |  movie reviews                                |  binary (1=good/0=bad)                              |  none yet                                                                      |  none
+cnn-sst2                                 |  sentiment analysis                             |  phrases                                      |  accuracy (0.0000 to 1.0000)                        |  <sub><sup> https://paperswithcode.com/sota/sentiment-analysis-on-sst-2-binary </sub></sup>           |  none
+cnn-yelp                                 |  sentiment analysis                             |  yelp reviews                                 |  binary (1=good/0=bad)                              | <sub><sup> https://paperswithcode.com/sota/sentiment-analysis-on-yelp-binary </sub></sup>            |  none
+distilbert-base-cased-SST-2              |  sentiment analysis                             |  phrases                                      |  accuracy (0.0000 to 1.0000)                        | <sub><sup> https://paperswithcode.com/sota/sentiment-analysis-on-sst-2-binary </sub></sup>           | <sub><sup> https://huggingface.co/textattack/distilbert-base-cased-SST-2 </sub></sup>
+distilbert-base-uncased-imdb             |  sentiment analysis                             |  movie reviews                                |  binary (1=good/0=bad)                              | <sub><sup> https://paperswithcode.com/sota/sentiment-analysis-on-imdb</sub></sup>                    | <sub><sup> https://huggingface.co/textattack/distilbert-base-uncased-imdb </sub></sup>
+distilbert-base-uncased-rotten-tomatoes  |  sentiment analysis                             |  movie reviews                                |  binary (1=good/0=bad)                              |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/distilbert-base-uncased-rotten-tomatoes </sub></sup>
+lstm-imdb                                |  sentiment analysis                             |  movie reviews                                |  binary (1=good/0=bad)                              | <sub><sup> https://paperswithcode.com/sota/sentiment-analysis-on-imdb </sub></sup>                    |  none
+lstm-mr                                  |  sentiment analysis                             |  movie reviews                                |  binary (1=good/0=bad)                              |  none yet                                                                      |  none
+lstm-sst2                                |  sentiment analysis                             |  phrases                                      |  accuracy (0.0000 to 1.0000)                        |  none yet                                                                      |  none
+lstm-yelp                                |  sentiment analysis                             |  yelp reviews                                 |  binary (1=good/0=bad)                              |  none yet                                                                      |  none
+roberta-base-imdb                        |  sentiment analysis                             |  movie reviews                                |  binary (1=good/0=bad)                              |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/roberta-base-imdb </sub></sup>
+roberta-base-rotten-tomatoes             |  sentiment analysis                             |  movie reviews                                |  binary (1=good/0=bad)                              |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/roberta-base-rotten-tomatoes </sub></sup>
+roberta-base-SST-2                       |  sentiment analysis                             |  phrases                                      |  accuracy (0.0000 to 1.0000)                        | <sub><sup> https://paperswithcode.com/sota/sentiment-analysis-on-sst-2-binary  </sub></sup>          | <sub><sup> https://huggingface.co/textattack/roberta-base-SST-2 </sub></sup>
+xlnet-base-cased-imdb                    |  sentiment analysis                             |  movie reviews                                |  binary (1=good/0=bad)                              |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/xlnet-base-cased-imdb </sub></sup>
+xlnet-base-cased-rotten-tomatoes         |  sentiment analysis                             |  movie reviews                                |  binary (1=good/0=bad)                              |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/xlnet-base-cased-rotten-tomatoes </sub></sup>
+albert-base-v2-ag-news                   |  text classification                            |  news articles                                |  news category                                      |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/albert-base-v2-ag-news </sub></sup>
+bert-base-uncased-ag-news                |  text classification                            |  news articles                                |  news category                                      |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/bert-base-uncased-ag-news </sub></sup>
+cnn-ag-news                              |  text classification                            |  news articles                                |  news category                                      | <sub><sup> https://paperswithcode.com/sota/text-classification-on-ag-news  </sub></sup>              |  none
+distilbert-base-uncased-ag-news          |  text classification                            |  news articles                                |  news category                                      |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/distilbert-base-uncased-ag-news </sub></sup>
+lstm-ag-news                             |  text classification                            |  news articles                                |  news category                                      | <sub><sup> https://paperswithcode.com/sota/text-classification-on-ag-news  </sub></sup>              |  none
+roberta-base-ag-news                     |  text classification                            |  news articles                                |  news category                                      |  none yet                                                                      | <sub><sup> https://huggingface.co/textattack/roberta-base-ag-news </sub></sup>
+
+
+

docs/api/attack.rst

@@ -0,0 +1,26 @@
+Attack API Reference
+=======================
+
+Attack
+------------
+Attack is composed of four components:
+
+- `Goal Functions <../attacks/goal_function.html>`__ stipulate the goal of the attack, like to change the prediction score of a classification model, or to change all of the words in a translation output.
+- `Constraints <../attacks/constraint.html>`__ determine if a potential perturbation is valid with respect to the original input.
+- `Transformations <../attacks/transformation.html>`__ take a text input and transform it by inserting and deleting characters, words, and/or phrases.
+- `Search Methods <../attacks/search_method.html>`__ explore the space of possible **transformations** within the defined **constraints** and attempt to find a successful perturbation which satisfies the **goal function**.
+
+The :class:`~textattack.Attack` class represents an adversarial attack composed of a goal function, search method, transformation, and constraints.
+
+.. autoclass:: textattack.Attack
+   :members:
+
+AttackRecipe
+-------------
+Attack recipe is a subclass of :class:`~textattack.Attack` class that has a special method :meth:`build` which 
+returns a pre-built :class:`~textattack.Attack` that correspond to attacks from the literature.
+
+ 
+
+.. autoclass:: textattack.attack_recipes.AttackRecipe
+   :members:

docs/api/attack_results.rst

@@ -0,0 +1,27 @@
+Attack Result API Reference
+============================
+
+AttackResult
+-------------
+.. autoclass:: textattack.attack_results.AttackResult
+   :members:
+
+SuccessfulAttackResult
+-----------------------
+.. autoclass:: textattack.attack_results.SuccessfulAttackResult
+   :members:
+
+FailedAttackResult
+-----------------------
+.. autoclass:: textattack.attack_results.FailedAttackResult
+   :members:
+
+SkippedAttackResult
+-----------------------
+.. autoclass:: textattack.attack_results.SkippedAttackResult
+   :members:
+
+MaximizedAttackResult
+-----------------------
+.. autoclass:: textattack.attack_results.MaximizedAttackResult
+   :members:

docs/api/attacker.rst

@@ -0,0 +1,19 @@
+Attacker API Reference
+=======================
+
+Attacker
+-------------
+While :class:`~textattack.Attack` is the main class used to carry out the adversarial attack, it is only useful for attacking one example at a time.
+It lacks features that support attacking multiple samples in parallel (i.e. multi-GPU), saving checkpoints, or logging results to text file, CSV file, or wandb.
+:class:`~textattack.Attacker` provides these features in an easy-to-use API.
+
+.. autoclass:: textattack.Attacker
+   :members:
+
+
+AttackArgs
+-------------
+:class:`~textattack.AttackArgs` represents arguments to be passed to :class:`~textattack.Attacker`, such as number of examples to attack, interval at which to save checkpoints, logging details.
+
+.. autoclass:: textattack.AttackArgs
+   :members:

docs/api/constraints.rst

@@ -0,0 +1,13 @@
+Constraints API Reference
+============================
+
+Constraint
+------------
+.. autoclass:: textattack.constraints.Constraint
+   :members:
+
+PreTransformationConstraint
+-----------------------------
+.. autoclass:: textattack.constraints.PreTransformationConstraint
+    :members:
+

docs/api/datasets.rst

@@ -0,0 +1,16 @@
+Datasets API Reference
+=============================
+Dataset class define the dataset object used to for carrying out attacks, augmentation, and training.
+:class:`~textattack.datasets.Dataset` class is the most basic class that could be used to wrap a list of input and output pairs.
+To load datasets from text, CSV, or JSON files, we recommend using 🤗 Datasets library to first
+load it as a :obj:`datasets.Dataset` object and then pass it to TextAttack's :class:`~textattack.datasets.HuggingFaceDataset` class.
+
+Dataset
+----------
+.. autoclass:: textattack.datasets.Dataset
+   :members: __getitem__, __len__
+
+HuggingFaceDataset
+-------------------
+.. autoclass:: textattack.datasets.HuggingFaceDataset
+   :members: __getitem__, __len__

docs/api/goal_functions.rst

@@ -0,0 +1,46 @@
+Goal Functions API Reference
+============================
+
+:class:`~textattack.goal_functions.GoalFunction` determines both the conditions under which the attack is successful (in terms of the model outputs)
+and the heuristic score that we want to maximize when searching for the solution.
+
+GoalFunction
+------------
+.. autoclass:: textattack.goal_functions.GoalFunction
+   :members:
+
+ClassificationGoalFunction
+--------------------------
+.. autoclass:: textattack.goal_functions.ClassificationGoalFunction
+   :members:
+
+TargetedClassification
+----------------------
+.. autoclass:: textattack.goal_functions.TargetedClassification
+   :members:
+
+UntargetedClassification
+------------------------
+.. autoclass:: textattack.goal_functions.UntargetedClassification
+   :members:
+
+InputReduction
+--------------
+.. autoclass:: textattack.goal_functions.InputReduction
+   :members:
+
+TextToTextGoalFunction
+-----------------------
+.. autoclass:: textattack.goal_functions.TextToTextGoalFunction
+   :members:
+
+MinimizeBleu
+-------------
+.. autoclass:: textattack.goal_functions.MinimizeBleu
+   :members:
+
+NonOverlappingOutput
+----------------------
+.. autoclass:: textattack.goal_functions.NonOverlappingOutput
+   :members:
+

docs/api/search_methods.rst

@@ -0,0 +1,46 @@
+Search Methods API Reference
+============================
+
+:class:`~textattack.search_methods.SearchMethod` attempts to find the optimal set of perturbations that will produce an adversarial example.
+Finding such optimal perturbations becomes a combinatorial optimization problem, and search methods are typically heuristic search algorithms designed
+to solve the underlying combinatorial problem.
+
+More in-depth study of search algorithms for NLP adversarial attacks can be found in the following work 
+`Searching for a Search Method: Benchmarking Search Algorithms for Generating NLP Adversarial Examples <https://arxiv.org/abs/2009.06368>`_
+by Jin Yong Yoo, John X. Morris, Eli Lifland, and Yanjun Qi.
+
+SearchMethod
+------------
+.. autoclass:: textattack.search_methods.SearchMethod
+   :members:
+
+BeamSearch
+------------
+.. autoclass:: textattack.search_methods.BeamSearch
+   :members:
+
+GreedySearch
+------------
+.. autoclass:: textattack.search_methods.GreedySearch
+   :members:
+
+GreedyWordSwapWIR
+------------------
+.. autoclass:: textattack.search_methods.GreedyWordSwapWIR
+   :members:
+
+AlzantotGeneticAlgorithm
+-------------------------
+.. autoclass:: textattack.search_methods.AlzantotGeneticAlgorithm
+   :members:
+
+ImprovedGeneticAlgorithm
+-------------------------
+.. autoclass:: textattack.search_methods.ImprovedGeneticAlgorithm
+   :members:
+
+ParticleSwarmOptimization
+--------------------------
+.. autoclass:: textattack.search_methods.ParticleSwarmOptimization
+   :members:
+

docs/api/trainer.rst

@@ -0,0 +1,60 @@
+Training API Reference
+==========================
+
+Trainer
+------------
+The :class:`~textattack.Trainer` class provides an API for adversarial training with features builtin for standard use cases.
+It is designed to be similar to the :obj:`Trainer` class provided by 🤗  Transformers library. 
+Custom behaviors can be added by subclassing the class and overriding these methods:
+
+- :meth:`training_step`: Peform a single training step. Override this for custom forward pass or custom loss.
+- :meth:`evaluate_step`: Peform a single evaluation step. Override this for custom foward pass.
+- :meth:`get_train_dataloader`: Creates the PyTorch DataLoader for training. Override this for custom batch setup.
+- :meth:`get_eval_dataloader`: Creates the PyTorch DataLoader for evaluation. Override this for custom batch setup.
+- :meth:`get_optimizer_and_scheduler`: Creates the optimizer and scheduler for training. Override this for custom optimizer and scheduler.
+
+The pseudocode for how training is done:
+
+.. code-block::
+
+   train_preds = []
+   train_targets = []
+   for batch in train_dataloader:
+      loss, preds, targets = training_step(model, tokenizer, batch)
+      train_preds.append(preds)
+      train_targets.append(targets)
+
+      # clear gradients
+      optimizer.zero_grad()
+
+      # backward
+      loss.backward()
+
+      # update parameters
+      optimizer.step()
+      if scheduler:
+         scheduler.step()
+
+   # Calculate training accuracy using `train_preds` and `train_targets`
+
+   eval_preds = []
+   eval_targets = []
+   for batch in eval_dataloader:
+      loss, preds, targets = training_step(model, tokenizer, batch)
+      eval_preds.append(preds)
+      eval_targets.append(targets)
+
+   # Calculate eval accuracy using `eval_preds` and `eval_targets`
+  
+
+.. autoclass:: textattack.Trainer
+   :members:
+
+
+TrainingArgs
+-------------
+Training arguments to be passed to :class:`~textattack.Trainer` class.
+
+.. autoclass:: textattack.TrainingArgs
+   :members:
+

docs/apidoc/textattack.attack_recipes.rst

@@ -7,6 +7,7 @@ textattack.attack\_recipes package
    :show-inheritance:
 
 
+
 .. automodule:: textattack.attack_recipes.attack_recipe
    :members:
    :undoc-members:
@@ -31,6 +32,12 @@ textattack.attack\_recipes package
    :show-inheritance:
 
 
+.. automodule:: textattack.attack_recipes.clare_li_2020
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
 .. automodule:: textattack.attack_recipes.deepwordbug_gao_2018
    :members:
    :undoc-members:

docs/apidoc/textattack.commands.attack.rst

@@ -1,44 +0,0 @@
-textattack.commands.attack package
-==================================
-
-.. automodule:: textattack.commands.attack
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-
-.. automodule:: textattack.commands.attack.attack_args
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.commands.attack.attack_args_helpers
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.commands.attack.attack_command
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.commands.attack.attack_resume_command
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.commands.attack.run_attack_parallel
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.commands.attack.run_attack_single_threaded
-   :members:
-   :undoc-members:
-   :show-inheritance:

docs/apidoc/textattack.commands.eval_model.rst

@@ -1,14 +0,0 @@
-textattack.commands.eval\_model package
-=======================================
-
-.. automodule:: textattack.commands.eval_model
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-
-.. automodule:: textattack.commands.eval_model.eval_model_command
-   :members:
-   :undoc-members:
-   :show-inheritance:

docs/apidoc/textattack.commands.rst

@@ -6,37 +6,45 @@ textattack.commands package
    :undoc-members:
    :show-inheritance:
 
-Subpackages
------------
-
-.. toctree::
-   :maxdepth: 6
-
-   textattack.commands.attack
-   textattack.commands.eval_model
-   textattack.commands.train_model
 
 
-
-.. automodule:: textattack.commands.augment
+.. automodule:: textattack.commands.attack_command
    :members:
    :undoc-members:
    :show-inheritance:
 
 
-.. automodule:: textattack.commands.benchmark_recipe
+.. automodule:: textattack.commands.attack_resume_command
    :members:
    :undoc-members:
    :show-inheritance:
 
 
-.. automodule:: textattack.commands.list_things
+.. automodule:: textattack.commands.augment_command
    :members:
    :undoc-members:
    :show-inheritance:
 
 
-.. automodule:: textattack.commands.peek_dataset
+.. automodule:: textattack.commands.benchmark_recipe_command
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.commands.eval_model_command
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.commands.list_things_command
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.commands.peek_dataset_command
    :members:
    :undoc-members:
    :show-inheritance:
@@ -52,3 +60,9 @@ Subpackages
    :members:
    :undoc-members:
    :show-inheritance:
+
+
+.. automodule:: textattack.commands.train_model_command
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/apidoc/textattack.commands.train_model.rst

@@ -1,26 +0,0 @@
-textattack.commands.train\_model package
-========================================
-
-.. automodule:: textattack.commands.train_model
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-
-.. automodule:: textattack.commands.train_model.run_training
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.commands.train_model.train_args_helpers
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.commands.train_model.train_model_command
-   :members:
-   :undoc-members:
-   :show-inheritance:

docs/apidoc/textattack.constraints.grammaticality.language_models.rst

@@ -6,8 +6,7 @@ textattack.constraints.grammaticality.language\_models package
    :undoc-members:
    :show-inheritance:
 
-Subpackages
------------
+
 
 .. toctree::
    :maxdepth: 6

docs/apidoc/textattack.constraints.grammaticality.rst

@@ -6,8 +6,7 @@ textattack.constraints.grammaticality package
    :undoc-members:
    :show-inheritance:
 
-Subpackages
------------
+
 
 .. toctree::
    :maxdepth: 6

docs/apidoc/textattack.constraints.pre_transformation.rst

@@ -14,6 +14,12 @@ textattack.constraints.pre\_transformation package
    :show-inheritance:
 
 
+.. automodule:: textattack.constraints.pre_transformation.max_modification_rate
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
 .. automodule:: textattack.constraints.pre_transformation.max_word_index_modification
    :members:
    :undoc-members:

docs/apidoc/textattack.constraints.rst

@@ -6,8 +6,7 @@ textattack.constraints package
    :undoc-members:
    :show-inheritance:
 
-Subpackages
------------
+
 
 .. toctree::
    :maxdepth: 6

docs/apidoc/textattack.constraints.semantics.rst

@@ -6,8 +6,7 @@ textattack.constraints.semantics package
    :undoc-members:
    :show-inheritance:
 
-Subpackages
------------
+
 
 .. toctree::
    :maxdepth: 6

docs/apidoc/textattack.constraints.semantics.sentence_encoders.rst

@@ -6,8 +6,7 @@ textattack.constraints.semantics.sentence\_encoders package
    :undoc-members:
    :show-inheritance:
 
-Subpackages
------------
+
 
 .. toctree::
    :maxdepth: 6

docs/apidoc/textattack.datasets.helpers.rst

@@ -0,0 +1,14 @@
+textattack.datasets.helpers package
+===================================
+
+.. automodule:: textattack.datasets.helpers
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+
+.. automodule:: textattack.datasets.helpers.ted_multi
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/apidoc/textattack.datasets.rst

@@ -6,13 +6,12 @@ textattack.datasets package
    :undoc-members:
    :show-inheritance:
 
-Subpackages
------------
+
 
 .. toctree::
    :maxdepth: 6
 
-   textattack.datasets.translation
+   textattack.datasets.helpers
 
 
 

docs/apidoc/textattack.datasets.translation.rst

@@ -1,14 +0,0 @@
-textattack.datasets.translation package
-=======================================
-
-.. automodule:: textattack.datasets.translation
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-
-.. automodule:: textattack.datasets.translation.ted_multi
-   :members:
-   :undoc-members:
-   :show-inheritance:

docs/apidoc/textattack.goal_functions.rst

@@ -6,8 +6,7 @@ textattack.goal\_functions package
    :undoc-members:
    :show-inheritance:
 
-Subpackages
------------
+
 
 .. toctree::
    :maxdepth: 6

docs/apidoc/textattack.metrics.attack_metrics.rst

@@ -0,0 +1,26 @@
+textattack.metrics.attack\_metrics package
+==========================================
+
+.. automodule:: textattack.metrics.attack_metrics
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+
+.. automodule:: textattack.metrics.attack_metrics.attack_queries
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.metrics.attack_metrics.attack_success_rate
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.metrics.attack_metrics.words_perturbed
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/apidoc/textattack.metrics.quality_metrics.rst

@@ -0,0 +1,20 @@
+textattack.metrics.quality\_metrics package
+===========================================
+
+.. automodule:: textattack.metrics.quality_metrics
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+
+.. automodule:: textattack.metrics.quality_metrics.perplexity
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.metrics.quality_metrics.use
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/apidoc/textattack.metrics.rst

@@ -0,0 +1,22 @@
+textattack.metrics package
+==========================
+
+.. automodule:: textattack.metrics
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+
+.. toctree::
+   :maxdepth: 6
+
+   textattack.metrics.attack_metrics
+   textattack.metrics.quality_metrics
+
+
+
+.. automodule:: textattack.metrics.metric
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/apidoc/textattack.models.helpers.rst

@@ -7,11 +7,6 @@ textattack.models.helpers package
    :show-inheritance:
 
 
-.. automodule:: textattack.models.helpers.bert_for_classification
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
 
 .. automodule:: textattack.models.helpers.glove_embedding_layer
    :members:

docs/apidoc/textattack.models.rst

@@ -6,8 +6,7 @@ textattack.models package
    :undoc-members:
    :show-inheritance:
 
-Subpackages
------------
+
 
 .. toctree::
    :maxdepth: 6

docs/apidoc/textattack.models.tokenizers.rst

@@ -8,12 +8,6 @@ textattack.models.tokenizers package
 
 
 
-.. automodule:: textattack.models.tokenizers.auto_tokenizer
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
 .. automodule:: textattack.models.tokenizers.glove_tokenizer
    :members:
    :undoc-members:

docs/apidoc/textattack.rst

@@ -1,5 +1,5 @@
-Complete API Reference
-========================
+textattack package
+==================
 
 .. automodule:: textattack
    :members:
@@ -19,7 +19,57 @@ Complete API Reference
    textattack.goal_function_results
    textattack.goal_functions
    textattack.loggers
+   textattack.metrics
    textattack.models
    textattack.search_methods
    textattack.shared
    textattack.transformations
+
+
+
+.. automodule:: textattack.attack
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.attack_args
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.attacker
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.augment_args
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.dataset_args
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.model_args
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.trainer
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.training_args
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/apidoc/textattack.shared.rst

@@ -6,8 +6,7 @@ textattack.shared package
    :undoc-members:
    :show-inheritance:
 
-Subpackages
------------
+
 
 .. toctree::
    :maxdepth: 6
@@ -16,12 +15,6 @@ Subpackages
 
 
 
-.. automodule:: textattack.shared.attack
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
 .. automodule:: textattack.shared.attacked_text
    :members:
    :undoc-members:
@@ -46,7 +39,7 @@ Subpackages
    :show-inheritance:
 
 
-.. automodule:: textattack.shared.word_embedding
+.. automodule:: textattack.shared.word_embeddings
    :members:
    :undoc-members:
    :show-inheritance:

docs/apidoc/textattack.shared.utils.rst

@@ -8,6 +8,12 @@ textattack.shared.utils package
 
 
 
+.. automodule:: textattack.shared.utils.importing
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
 .. automodule:: textattack.shared.utils.install
    :members:
    :undoc-members:

docs/apidoc/textattack.transformations.rst

@@ -8,18 +8,21 @@ textattack.transformations package
 
 
 
+.. toctree::
+   :maxdepth: 6
+
+   textattack.transformations.word_insertions
+   textattack.transformations.word_merges
+   textattack.transformations.word_swaps
+
+
+
 .. automodule:: textattack.transformations.composite_transformation
    :members:
    :undoc-members:
    :show-inheritance:
 
 
-.. automodule:: textattack.transformations.random_synonym_insertion
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
 .. automodule:: textattack.transformations.transformation
    :members:
    :undoc-members:
@@ -32,115 +35,7 @@ textattack.transformations package
    :show-inheritance:
 
 
-.. automodule:: textattack.transformations.word_swap
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.transformations.word_swap_change_location
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.transformations.word_swap_change_name
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.transformations.word_swap_change_number
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.transformations.word_swap_contract
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.transformations.word_swap_embedding
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.transformations.word_swap_extend
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.transformations.word_swap_gradient_based
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.transformations.word_swap_homoglyph_swap
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.transformations.word_swap_hownet
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.transformations.word_swap_inflections
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.transformations.word_swap_masked_lm
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.transformations.word_swap_neighboring_character_swap
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.transformations.word_swap_qwerty
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.transformations.word_swap_random_character_deletion
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.transformations.word_swap_random_character_insertion
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.transformations.word_swap_random_character_substitution
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.transformations.word_swap_random_word
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-
-.. automodule:: textattack.transformations.word_swap_wordnet
+.. automodule:: textattack.transformations.word_innerswap_random
    :members:
    :undoc-members:
    :show-inheritance:

docs/apidoc/textattack.transformations.word_insertions.rst

@@ -0,0 +1,26 @@
+textattack.transformations.word\_insertions package
+===================================================
+
+.. automodule:: textattack.transformations.word_insertions
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+
+.. automodule:: textattack.transformations.word_insertions.word_insertion
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.transformations.word_insertions.word_insertion_masked_lm
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.transformations.word_insertions.word_insertion_random_synonym
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/apidoc/textattack.transformations.word_merges.rst

@@ -0,0 +1,20 @@
+textattack.transformations.word\_merges package
+===============================================
+
+.. automodule:: textattack.transformations.word_merges
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+
+.. automodule:: textattack.transformations.word_merges.word_merge
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.transformations.word_merges.word_merge_masked_lm
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/apidoc/textattack.transformations.word_swaps.rst

@@ -0,0 +1,116 @@
+textattack.transformations.word\_swaps package
+==============================================
+
+.. automodule:: textattack.transformations.word_swaps
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+
+.. automodule:: textattack.transformations.word_swaps.word_swap
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.transformations.word_swaps.word_swap_change_location
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.transformations.word_swaps.word_swap_change_name
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.transformations.word_swaps.word_swap_change_number
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.transformations.word_swaps.word_swap_contract
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.transformations.word_swaps.word_swap_embedding
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.transformations.word_swaps.word_swap_extend
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.transformations.word_swaps.word_swap_gradient_based
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.transformations.word_swaps.word_swap_homoglyph_swap
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.transformations.word_swaps.word_swap_hownet
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.transformations.word_swaps.word_swap_inflections
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.transformations.word_swaps.word_swap_masked_lm
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.transformations.word_swaps.word_swap_neighboring_character_swap
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.transformations.word_swaps.word_swap_qwerty
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.transformations.word_swaps.word_swap_random_character_deletion
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.transformations.word_swaps.word_swap_random_character_insertion
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.transformations.word_swaps.word_swap_random_character_substitution
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+.. automodule:: textattack.transformations.word_swaps.word_swap_wordnet
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/conf.py

@@ -21,7 +21,7 @@ copyright = "2020, UVA QData Lab"
 author = "UVA QData Lab"
 
 # The full version, including alpha/beta/rc tags
-release = "0.2.12"
+release = "0.3.3"
 
 # Set master doc to `index.rst`.
 master_doc = "index"
@@ -43,6 +43,8 @@ extensions = [
     "nbsphinx",
     # Enable .md doc files
     "recommonmark",
+    "sphinx_markdown_tables",
+    "IPython.sphinxext.ipython_console_highlighting",
 ]
 autosummary_generate = True
 
@@ -80,6 +82,10 @@ html_css_files = [
     "css/custom.css",
 ]
 
+html_sidebars = {
+    "**": ["globaltoc.html", "relations.html", "sourcelink.html", "searchbox.html"]
+}
+
 # Path to favicon.
 html_favicon = "favicon.png"
 

docs/environment.yml

@@ -0,0 +1,5 @@
+name: textattackenv
+channels:
+  - defaults
+dependencies:
+  - python=3.7

docs/index.rst

@@ -4,20 +4,37 @@ TextAttack Documentation
 
 .. toctree::
    :maxdepth: 6
-   :caption: About
+   :caption: Get Started
 
-   1start/basic-Intro.rst
-   1start/what_is_an_adversarial_attack.md
-   1start/references.md
-   1start/benchmark-search.md
+   Basic-Introduction <0_get_started/basic-Intro.rst>
+   Installation <0_get_started/installation.md>
+   Command-Line Usage <0_get_started/command_line_usage.md>
+   Quick API Usage  <0_get_started/quick_api_tour.rst>
+   FAQ <1start/FAQ.md>
 
 .. toctree::
    :maxdepth: 6
-   :caption: Get Started
+   :caption: Recipes
+   
+   3recipes/attack_recipes_cmd.md
+   3recipes/attack_recipes.rst
+   3recipes/augmenter_recipes_cmd.md
+   3recipes/augmenter_recipes.rst
+   3recipes/models.md
 
-   Installation <1start/installation>
-   Command-Line Usage <1start/command_line_usage.md>
+.. toctree::
+   :maxdepth: 6
+   :caption: Using TextAttack
 
+   1start/what_is_an_adversarial_attack.md
+   1start/references.md
+   1start/attacks4Components.md
+   1start/benchmark-search.md
+   1start/quality-SOTA-recipes.md
+   1start/api-design-tips.md
+   1start/multilingual-visualization.md
+   1start/support.md
+   
  
 .. toctree::
    :maxdepth: 6
@@ -26,19 +43,32 @@ TextAttack Documentation
    Tutorial 0: TextAttack End-To-End (Train, Eval, Attack) <2notebook/0_End_to_End.ipynb>
    Tutorial 1: Transformations <2notebook/1_Introduction_and_Transformations.ipynb>
    Tutorial 2: Constraints <2notebook/2_Constraints.ipynb>
-   Tutorial 3: Attacking TensorFlow models <2notebook/Example_0_tensorflow.ipynb>
-   Tutorial 4: Attacking scikit-learn models <2notebook/Example_1_sklearn.ipynb>
-   Tutorial 5: Attacking AllenNLP models <2notebook/Example_2_allennlp.ipynb>
-   Tutorial 6: Attacking multilingual models <2notebook/Example_4_CamemBERT.ipynb>
+   Tutorial 3: Augmentation <2notebook/3_Augmentations.ipynb>
+   Tutorial 4: Custom Word Embeddings <2notebook/4_Custom_Datasets_Word_Embedding.ipynb>
+   Tutorial 5: Attacking TensorFlow models <2notebook/Example_0_tensorflow.ipynb>
+   Tutorial 6: Attacking scikit-learn models <2notebook/Example_1_sklearn.ipynb>
+   Tutorial 7: Attacking AllenNLP models <2notebook/Example_2_allennlp.ipynb>
+   Tutorial 8: Attacking Keras models <2notebook/Example_3_Keras.ipynb>
+   Tutorial 9: Attacking multilingual models <2notebook/Example_4_CamemBERT.ipynb>
+   Tutorial10: Explaining Attacking BERT model using Captum <2notebook/Example_5_Explain_BERT.ipynb>
+
+.. toctree::
+   :maxdepth: 6
+   :caption: API User Guide
+
+   Attack <api/attack.rst>
+   Attacker <api/attacker.rst>
+   AttackResult <api/attack_results.rst>
+   Trainer <api/trainer.rst>
+   Datasets <api/datasets.rst>
+   GoalFunction <api/goal_functions.rst>
+   Constraints <api/constraints.rst>
+   SearchMethod <api/search_methods.rst>
    
 
 .. toctree::
    :maxdepth: 6
    :glob:
-   :caption: Developer Guide
-   
-   1start/support.md
-   1start/api-design-tips.md
-   3recipes/attack_recipes
-   3recipes/augmenter_recipes
-   apidoc/textattack
+   :caption: Full Reference
+
+   apidoc/textattack

docs/requirements.txt

@@ -1,4 +1,35 @@
-recommonmark
-nbsphinx
-sphinx - autobuild
-sphinx - rtd - theme
+recommonmark==0.7.1
+Sphinx==4.1.2
+sphinx-autobuild==2021.3.14
+sphinx-markdown-tables==0.0.15
+sphinx-rtd-theme==0.5.2
+sphinxcontrib-applehelp==1.0.2
+sphinxcontrib-devhelp==1.0.2
+sphinxcontrib-htmlhelp==2.0.0
+sphinxcontrib-jsmath==1.0.1
+sphinxcontrib-qthelp==1.0.3
+sphinxcontrib-serializinghtml==1.1.5
+nbclient==0.5.4
+nbconvert==6.1.0
+nbformat==5.1.3
+nbsphinx==0.8.7
+widgetsnbextension==3.5.1
+ipykernel==6.4.1
+ipython==7.27.0
+ipython-genutils==0.2.0
+ipywidgets==7.6.4
+scipy==1.7.1
+tensorboard==2.6.0
+tensorboard-data-server==0.6.1
+tensorboard-plugin-wit==1.8.0
+tensorboardX==2.4
+tensorflow==2.6.0
+tensorflow-estimator==2.6.0
+tensorflow-hub==0.12.0
+tensorflow-text==2.6.0
+sentence-transformers==2.0.0
+transformers==4.10.1
+textattack==0.3.3
+sqlitedict==1.7.0
+stanza==1.2.3
+Cython==0.29.24

examples/attack/attack_camembert.py

@@ -4,6 +4,7 @@ import os
 import numpy as np
 from transformers import AutoTokenizer, TFAutoModelForSequenceClassification, pipeline
 
+from textattack import Attacker
 from textattack.attack_recipes import PWWSRen2019
 from textattack.datasets import HuggingFaceDataset
 from textattack.models.wrappers import ModelWrapper
@@ -20,11 +21,11 @@ class HuggingFaceSentimentAnalysisPipelineWrapper(ModelWrapper):
         [[0.218262017, 0.7817379832267761]
     """
 
-    def __init__(self, pipeline):
-        self.pipeline = pipeline
+    def __init__(self, model):
+        self.model = model
 
     def __call__(self, text_inputs):
-        raw_outputs = self.pipeline(text_inputs)
+        raw_outputs = self.model(text_inputs)
         outputs = []
         for output in raw_outputs:
             score = output["score"]
@@ -55,7 +56,6 @@ recipe = PWWSRen2019.build(model_wrapper)
 recipe.transformation.language = "fra"
 
 dataset = HuggingFaceDataset("allocine", split="test")
-for idx, result in enumerate(recipe.attack_dataset(dataset)):
-    print(("-" * 20), f"Result {idx+1}", ("-" * 20))
-    print(result.__str__(color_method="ansi"))
-    print()
+
+attacker = Attacker(recipe, dataset)
+results = attacker.attack_dataset()

examples/attack/attack_from_components.sh

@@ -3,5 +3,5 @@
 # model on the Yelp dataset.
 textattack attack --attack-n --goal-function untargeted-classification \
     --model bert-base-uncased-yelp --num-examples 8 --transformation word-swap-wordnet \
-    --constraints edit-distance^12 max-words-perturbed:max_percent=0.75 repeat stopword \
+    --constraints edit-distance^12 max-words-perturbed^max_percent=0.75 repeat stopword \
     --search greedy

examples/attack/attack_keras_parallel.py

@@ -0,0 +1,142 @@
+"""
+Recent upgrade of keras versions in TF 2.5+, keras has been moved to tf.keras
+This has resulted in certain exceptions when keras models are attacked in parallel
+This script fixes this behavior by adding an official hotfix for this situation detailed here:
+https://github.com/tensorflow/tensorflow/issues/34697
+All models/dataset are similar to keras attack tutorial at :
+https://textattack.readthedocs.io/en/latest/2notebook/Example_3_Keras.html#
+NOTE: This fix might be deprecated in future TF releases
+NOTE: This script is not designed to run in a Jupyter notebook due to conflicting namespace issues
+We recommend running it as a script only
+"""
+
+import numpy as np
+import tensorflow as tf
+from tensorflow.keras.layers import Dense, Dropout
+from tensorflow.keras.models import Model, Sequential
+from tensorflow.keras.utils import to_categorical
+from tensorflow.python.keras.layers import deserialize, serialize
+from tensorflow.python.keras.saving import saving_utils
+import torch
+
+from textattack import AttackArgs, Attacker
+from textattack.attack_recipes import PWWSRen2019
+from textattack.datasets import HuggingFaceDataset
+from textattack.models.wrappers import ModelWrapper
+
+NUM_WORDS = 1000
+
+
+def unpack(model, training_config, weights):
+    restored_model = deserialize(model)
+    if training_config is not None:
+        restored_model.compile(
+            **saving_utils.compile_args_from_training_config(training_config)
+        )
+    restored_model.set_weights(weights)
+    return restored_model
+
+
+# Hotfix function
+def make_keras_picklable():
+    def __reduce__(self):
+        model_metadata = saving_utils.model_metadata(self)
+        training_config = model_metadata.get("training_config", None)
+        model = serialize(self)
+        weights = self.get_weights()
+        return (unpack, (model, training_config, weights))
+
+    cls = Model
+    cls.__reduce__ = __reduce__
+
+
+# Run the function
+make_keras_picklable()
+
+
+def transform(x):
+    x_transform = []
+    for i, word_indices in enumerate(x):
+        BoW_array = np.zeros((NUM_WORDS,))
+        for index in word_indices:
+            if index < len(BoW_array):
+                BoW_array[index] += 1
+        x_transform.append(BoW_array)
+    return np.array(x_transform)
+
+
+class CustomKerasModelWrapper(ModelWrapper):
+    def __init__(self, model):
+        self.model = model
+
+    def __call__(self, text_input_list):
+
+        x_transform = []
+        for i, review in enumerate(text_input_list):
+            tokens = [x.strip(",") for x in review.split()]
+            BoW_array = np.zeros((NUM_WORDS,))
+            for word in tokens:
+                if word in vocabulary:
+                    if vocabulary[word] < len(BoW_array):
+                        BoW_array[vocabulary[word]] += 1
+            x_transform.append(BoW_array)
+        x_transform = np.array(x_transform)
+        prediction = self.model.predict(x_transform)
+        return prediction
+
+
+model = Sequential()
+model.add(Dense(512, activation="relu", input_dim=NUM_WORDS))
+model.add(Dropout(0.3))
+model.add(Dense(100, activation="relu"))
+model.add(Dense(2, activation="sigmoid"))
+opt = tf.keras.optimizers.Adam(learning_rate=0.00001)
+
+model.compile(optimizer=opt, loss="binary_crossentropy", metrics=["accuracy"])
+
+
+(x_train_tokens, y_train), (x_test_tokens, y_test) = tf.keras.datasets.imdb.load_data(
+    path="imdb.npz",
+    num_words=NUM_WORDS,
+    skip_top=0,
+    maxlen=None,
+    seed=113,
+    start_char=1,
+    oov_char=2,
+    index_from=3,
+)
+
+
+index = int(0.9 * len(x_train_tokens))
+x_train = transform(x_train_tokens)[:index]
+x_test = transform(x_test_tokens)[index:]
+y_train = np.array(y_train[:index])
+y_test = np.array(y_test[index:])
+y_train = to_categorical(y_train)
+y_test = to_categorical(y_test)
+
+vocabulary = tf.keras.datasets.imdb.get_word_index(path="imdb_word_index.json")
+
+results = model.fit(
+    x_train, y_train, epochs=1, batch_size=512, validation_data=(x_test, y_test)
+)
+
+
+if __name__ == "__main__":
+    torch.multiprocessing.freeze_support()
+
+    model_wrapper = CustomKerasModelWrapper(model)
+    dataset = HuggingFaceDataset("rotten_tomatoes", None, "test", shuffle=True)
+
+    attack = PWWSRen2019.build(model_wrapper)
+
+    attack_args = AttackArgs(
+        num_examples=10,
+        checkpoint_dir="checkpoints",
+        parallel=True,
+        num_workers_per_device=2,
+    )
+
+    attacker = Attacker(attack, dataset, attack_args)
+
+    attacker.attack_dataset()

examples/augmentation/augment.csv

@@ -1,11 +1,11 @@
 text,label
-"the rock is destined to be the 21st century's novel conan and that he's go to make a splash yet greater than arnold schwarzenegger , jean- claud van damme or steven segal.",1
-"the rock is destined to be the 21st century's novo conan and that he's going to make a splash yet greater than arnold schwarzenegger , jean- claud van damme or stephens segal.",1
-the gorgeously elaborate continuation of 'the lord of the rings' triad is so massive that a column of words cannot adequately describe co-writer/director pete jackson's expanded vision of j . r . r . tolkien's middle-earth .,1
-the gorgeously elaborate continuation of 'the lordy of the rings' trilogy is so huge that a column of words cannot adequately describe co-writer/superintendent peter jackson's enlargements vision of j . r . r . tolkien's middle-earth .,1
-take care of my cat offers a cheerfully different slice of asian cinema .,1
-take care of my cat offers a refreshingly different slice of asian cinemas .,1
-a technically well-made suspenser . . . but its abrupt fall in iq points as it races to the finish line demonstrating simply too discouraging to let slide .,0 
-a technologically well-made suspenser . . . but its abrupt dip in iq points as it races to the finish line proves simply too discouraging to let slide .,0 
-it's a mystery how the cinematography could be released in this condition .,0
-it's a mystery how the movies could be released in this condition .,0
+"the rock is destined to be the new conan and that he's going to make a splash even greater than arnold , jean- claud van damme or steven segal.",1
+"the rock is destined to be the 21st century's new conan and that he's going to caravan make a splash even greater than arnold schwarzenegger , jean- claud van damme or steven segal.",1
+the gorgeously rarify continuation of 'the lord of the rings' trilogy is so huge that a column of give-and-take cannot adequately describe co-writer/director shaft jackson's expanded vision of j . r . r . tolkien's middle-earth .,1
+the gorgeously elaborate of 'the of the rings' trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded of j . r . r . tolkien's middle-earth .,1
+take care different my cat offers a refreshingly of slice of asian cinema .,1
+take care of my cat offers a different slice of asian cinema .,1
+a technically well-made suspenser . . . but its abrupt drop in iq points as it races to the finish IT line proves simply too discouraging to let slide .,0 
+a technically well-made suspenser . . . but its abrupt drop in iq points as it races to the finish demarcation proves plainly too discouraging to let slide .,0 
+it's pic a mystery how the movie could be released in this condition .,0
+it's a mystery how the movie could in released be this condition .,0

examples/augmentation/augment.sh

@@ -1,2 +1,2 @@
 #!/bin/bash
-textattack augment --csv examples.csv --input-column text --recipe eda --pct-words-to-swap .1 --transformations-per-example 2 --exclude-original --overwrite
+textattack augment --input-csv examples.csv --output-csv output.csv --input-column text --recipe eda --pct-words-to-swap .1 --transformations-per-example 2 --exclude-original --overwrite

examples/augmentation/example.csv

@@ -1,2 +0,0 @@
-"text",label
-"it's a mystery how the movie could be released in this condition .", 0

examples/train/train_albert_snli_entailment.sh

@@ -2,4 +2,4 @@
 # Trains `bert-base-cased` on the STS-B task for 3 epochs. This is a 
 # demonstration of how our training script can handle different `transformers`
 # models and customize for different datasets.
-textattack train --model albert-base-v2 --dataset snli --batch-size 128 --epochs 5 --max-length 128 --learning-rate 1e-5 --allowed-labels 0 1 2
+textattack train --model-name-or-path albert-base-v2 --dataset snli --per-device-train-batch-size 8 --epochs 5 --learning-rate 1e-5 

examples/train/train_bert_stsb_similarity.sh

@@ -1,4 +1,4 @@
 #!/bin/bash
 # Trains `bert-base-cased` on the STS-B task for 3 epochs. This is a demonstration
 # of how our training script handles regression.
-textattack train --model bert-base-cased --dataset glue^stsb --batch-size 128 --epochs 3 --max-length 128 --learning-rate 1e-5
+textattack train --model-name-or-path bert-base-cased --dataset glue^stsb  --epochs 3 --learning-rate 1e-5

examples/train/train_lstm_imdb_sentiment_classification.sh

@@ -0,0 +1,4 @@
+#!/bin/bash
+# Trains `bert-base-cased` on the STS-B task for 3 epochs. This is a basic
+# demonstration of our training script and `datasets` integration.
+textattack train --model-name-or-path lstm --dataset imdb  --epochs 50 --learning-rate 1e-5