Merge pull request #331 from QData/new_docs

add a single doc page expaining our benchmarking results on search methods used in the literature

docs/1start/benchmark-search.md

@@ -0,0 +1,41 @@
+
+Benchmarking Search Algorithms for Generating NLP Adversarial Examples
+=========================================================================
+
+
+*This documentation page was adapted from Our Paper in [EMNLP BlackNLP](https://arxiv.org/abs/2009.06368).*
+
+
+### Title: Searching for a Search Method: Benchmarking Search Algorithms for Generating NLP Adversarial Examples
+
+
+- Abstract:  We study the behavior of several black-box search algorithms used for generating adversarial examples for natural language processing (NLP) tasks. We perform a fine-grained analysis of three elements relevant to search: search algorithm, search space, and search budget. When new search methods are proposed in past work, the attack search space is often modified alongside the search method. Without ablation studies benchmarking the search algorithm change with the search space held constant, an increase in attack success rate could from an improved search method or a less restrictive search space. Additionally, many previous studies fail to properly consider the search algorithms' run-time cost, which is essential for downstream tasks like adversarial training. Our experiments provide a reproducible benchmark of search algorithms across a variety of search spaces and query budgets to guide future research in adversarial NLP. Based on our experiments, we recommend greedy attacks with word importance ranking when under a time constraint or attacking long inputs, and either beam search or particle swarm optimization otherwise. 
+
+
+###  Citations: 
+```
+@misc{yoo2020searching,
+      title={Searching for a Search Method: Benchmarking Search Algorithms for Generating NLP Adversarial Examples}, 
+      author={Jin Yong Yoo and John X. Morris and Eli Lifland and Yanjun Qi},
+      year={2020},
+      eprint={2009.06368},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```
+
+
+
+### Our benchmarking results on comparing search methods used in the past attacks. 
+
+
+![Table1](/_static/imgs/benchmark/search-example.pdf)
+
+![Table1](/_static/imgs/benchmark/search-table2.png)
+![Table1](/_static/imgs/benchmark/search-table31.png)
+![Table1](/_static/imgs/benchmark/search-table32.png)
+![Table1](/_static/imgs/benchmark/search-fig1.png)
+![Table1](/_static/imgs/benchmark/search-fig2.png)
+
+![Table1](/_static/imgs/benchmark/search-table1.png)
+

docs/1start/references.md

@@ -59,3 +59,4 @@ How to Cite TextAttack
       primaryClass={cs.CL}
 }
 ```
+

docs/index.rst

@@ -1,3 +1,6 @@
+TextAttack Documentation
+=======================================
+
 
 .. toctree::
    :maxdepth: 6
@@ -6,6 +9,7 @@
    1start/basic-Intro.rst
    1start/what_is_an_adversarial_attack.md
    1start/references.md
+   1start/benchmark-search.md
 
 .. toctree::
    :maxdepth: 6

docs/requirements.txt

@@ -1,5 +1,4 @@
 recommonmark
 nbsphinx
-sphinx-autobuild
-sphinx-rtd-theme
-
+sphinx - autobuild
+sphinx - rtd - theme

textattack/commands/attack/attack_args.py

@@ -1,7 +1,7 @@
 """
 
-TextAttack Command Args for Attack
-------------------------------------------
+TextAttack Command Args Designed for Attack
+----------------------------------------------
 
 """
 

textattack/commands/textattack_cli.py

@@ -1,7 +1,7 @@
 """
 
-TextAttack Command Arg Parsing
-=====================================
+TextAttack Command Arg Parsing Main Function
+=============================================
 """
 
 # !/usr/bin/env python
@@ -17,6 +17,16 @@ from textattack.commands.train_model import TrainModelCommand
 
 
 def main():
+
+    """This is the main command line parer and entry function to use TextAttack via command lines
+
+    texattack <command> [<args>]
+
+    Args:
+        command (string): augment, attack, train, eval-model, attack-resume, list, peek-dataset
+        [<args>] (string): depending on the command string
+    """
+
     parser = argparse.ArgumentParser(
         "TextAttack CLI",
         usage="[python -m] texattack <command> [<args>]",