alihan/textattack-nlp-transformer
1
0
Fork 0
mirror of https://github.com/QData/TextAttack.git synced 2021-10-13 00:05:06 +03:00
Code Issues Projects Releases Wiki Activity

improve augmentation; merge in fix-docs

Browse Source
This commit is contained in:
Jack Morris
2020-05-20 16:32:25 -04:00
parent 0230d0b7dd 93741a7b8b
commit f030a16c71
137 changed files with 2760 additions and 1587 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
.gitignore vendored
View File

@@ -34,3 +34,9 @@ dist/
# Weights & Biases outputs
wandb/
# checkpoints
checkpoints/
# vim
*.swp

16
.readthedocs.yml Normal file
View File

@@ -0,0 +1,16 @@
# .readthedocs.yml
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
# Required
version: 2
# Build documentation in the docs/ directory with Sphinx
sphinx:
configuration: docs/conf.py
# Optionally set the version of Python and requirements required to build your docs
python:
version: 3.7
install:
- requirements: docs/requirements.txt

35
README.md
View File

@@ -13,12 +13,15 @@
<a target="_blank" href="https://travis-ci.org/QData/TextAttack">
<img src="https://travis-ci.org/QData/TextAttack.svg?branch=master" alt="Coverage Status">
</a>
<a href="https://badge.fury.io/py/textattack">
<img src="https://badge.fury.io/py/textattack.svg" alt="PyPI version" height="18">
</a>
</p>
## About
TextAttack is a Python framework for running adversarial attacks against NLP models. TextAttack builds attacks from four components: a serach method, goal function, transformation, and set of constraints. TextAttack's modular design makes it easily extensible to new NLP tasks, models, and attack strategies. TextAttack currently supports attacks on models trained for classification, entailment, and translation.
TextAttack is a Python framework for running adversarial attacks against NLP models. TextAttack builds attacks from four components: a search method, goal function, transformation, and set of constraints. TextAttack's modular design makes it easily extensible to new NLP tasks, models, and attack strategies. TextAttack currently supports attacks on models trained for classification, entailment, and translation.
## Setup
@@ -39,26 +42,25 @@ environment variable `TA_CACHE_DIR`.
### Running Attacks
The [`examples/`](examples/) folder contains notebooks walking through examples of basic usage of TextAttack, including building a custom transformation and a custom constraint.
The [`examples/`](docs/examples/) folder contains notebooks walking through examples of basic usage of TextAttack, including building a custom transformation and a custom constraint. These examples can also be viewed through the [documentation website](https://textattack.readthedocs.io/en/latest).
We also have a command-line interface for running attacks. See help info and list of arguments with `python -m textattack --help`.
### Attack Recipes
We include attack recipes which build an attack such that only one command line argument has to be passed. To run an attack recipes, run `python -m textattack --recipe [recipe_name]`
Currently, we include six recipes, all synonym substitution-based.
The first are for classification and entailment attacks:
- **textfooler**: Greedy attack with word importance ranking (["Is Bert Really Robust?" (Jin et al., 2019)](https://arxiv.org/abs/1907.11932)).
- **alzantot**: Genetic algorithm attack from (["Generating Natural Language Adversarial Examples" (Alzantot et al., 2018)](https://arxiv.org/abs/1804.07998)).
- **tf-adjusted**: TextFooler attack with constraint thresholds adjusted based on human evaluation and grammaticality enforced.
- **alz-adjusted**: Alzantot's attack adjusted to follow the same constraints as tf-adjusted such that the only difference is the search method.
- **deepwordbug**: Replace-1 scoring and multi-transformation character-swap attack (["Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers"](https://arxiv.org/abs/1801.04354)).
- **hotflip**: Beam search and gradient-based word swap (["HotFlip: White-Box Adversarial Examples for Text Classification"](https://arxiv.org/abs/1712.06751)
- **kuleshov**: Greedy search and counterfitted embedding swap (["Adversarial Examples for Natural Language Classification Problems"](https://openreview.net/pdf?id=r1QZ3zbAZ)
- **deepwordbug**: Replace-1 scoring and multi-transformation character-swap attack (["Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers" (Gao et al., 2018)](https://arxiv.org/abs/1801.04354)).
- **hotflip**: Beam search and gradient-based word swap (["HotFlip: White-Box Adversarial Examples for Text Classification" (Ebrahimi et al., 2017)](https://arxiv.org/abs/1712.06751)).
- **kuleshov**: Greedy search and counterfitted embedding swap (["Adversarial Examples for Natural Language Classification Problems" (Kuleshov et al., 2018)](https://openreview.net/pdf?id=r1QZ3zbAZ)).
The final is for translation attacks:
- **seq2sick**: Greedy attack with goal of changing every word in the output translation. Currently implemented as black-box with plans to change to white-box as done in paper (["Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples"](https://arxiv.org/abs/1803.01128)).
- **seq2sick**: Greedy attack with goal of changing every word in the output translation. Currently implemented as black-box with plans to change to white-box as done in paper (["Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples" (Cheng et al., 2018)](https://arxiv.org/abs/1803.01128)).
### Augmenting Text
@@ -109,7 +111,7 @@ The `attack_one` method in an `Attack` takes as input a `TokenizedText`, and out
### Goal Functions
A `GoalFunction` takes as input a `TokenizedText` object and the ground truth output, and determines whether the attack has succeeded.
A `GoalFunction` takes as input a `TokenizedText` object and the ground truth output, and determines whether the attack has succeeded, returning a `GoalFunctionResult`.
### Constraints
@@ -121,8 +123,23 @@ A `Transformation` takes as input a `TokenizedText` and returns a list of possib
### Search Methods
A search method is currently implemented in an extension of the `Attack` class, through implementing the `attack_one` method. The `get_transformations` function takes as input a `TokenizedText` object and outputs a list of possible transformations filtered by meeting all of the attack’s constraints. A search consists of successive calls to `get_transformations` until the search succeeds or is exhausted.
A `SearchMethod` takes as input an initial `GoalFunctionResult` and returns a final `GoalFunctionResult` The search is given access to the `get_transformations` function, which takes as input a `TokenizedText` object and outputs a list of possible transformations filtered by meeting all of the attack’s constraints. A search consists of successive calls to `get_transformations` until the search succeeds (determined using `get_goal_results`) or is exhausted.
## Contributing to TextAttack
We welcome contributions and suggestions! Submit a pull request or issue and we will do our best to respond in a timely manner.
## Citing TextAttack
If you use TextAttack for your research, please cite [TextAttack: A Framework for Adversarial Attacks in Natural Language Processing](https://arxiv.org/abs/2005.05909).
```bibtex
@misc{Morris2020TextAttack,
Author = {John X. Morris and Eli Lifland and Jin Yong Yoo and Yanjun Qi},
Title = {TextAttack: A Framework for Adversarial Attacks in Natural Language Processing},
Year = {2020},
Eprint = {arXiv:2005.05909},
}
```

BIN
dist/textattack-0.0.1.7-py3-none-any.whl vendored
View File

Binary file not shown.

BIN
dist/textattack-0.0.1.7.tar.gz vendored
View File

Binary file not shown.

6
docs/Makefile
View File

@@ -12,9 +12,15 @@ BUILDDIR = _build
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
# For autobuild
# https://pypi.org/project/sphinx-autobuild/
livehtml:
sphinx-autobuild -b html $(SPHINXOPTS) "$(BUILDDIR)/html"
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

6
docs/attack_methods/attack.rst
View File

@@ -1,6 +0,0 @@
=====================
Attack Documentation
=====================
.. automodule:: textattack.attack_methods.attack
:members:

7
docs/attack_methods/beam_search.rst
View File

@@ -1,7 +0,0 @@
=====================
Beam Search
=====================
.. automodule:: textattack.attack_methods.beam_search
:members:

6
docs/attack_methods/genetic_algorithm.rst
View File

@@ -1,6 +0,0 @@
===================
Genetic Algorithm
===================
.. automodule:: textattack.attack_methods.genetic_algorithm
:members:

9
docs/attack_methods/greedy_word_swap.rst
View File

@@ -1,9 +0,0 @@
===================
Greedy Word Swap
===================
.. automodule:: textattack.attack_methods.greedy_word_swap
:members:
.. automodule:: textattack.attack_methods.greedy_word_swap_wir
:members:

9
docs/attack_recipes/alzantot_genetic.rst
View File

@@ -1,9 +0,0 @@
=====================================
Genetic Algorithm by Alzantot (2018)
=====================================
.. automodule:: textattack.attack_recipes.alzantot_2018_genetic_algorithm_adjusted
:members:
.. automodule:: textattack.attack_recipes.alzantot_2018_genetic_algorithm
:members:

6
docs/attack_recipes/gao_deepwordbug.rst
View File

@@ -1,6 +0,0 @@
===========================
DeepWordBug by Gao (2018)
===========================
.. automodule:: textattack.attack_recipes.gao_2018_deepwordbug
:members:

10
docs/attack_recipes/jin_textfooler.rst
View File

@@ -1,10 +0,0 @@
===========================
TextFooler by Jin (2019)
===========================
.. automodule:: textattack.attack_recipes.jin_2019_textfooler_adjusted
:members:
.. automodule:: textattack.attack_recipes.jin_2019_textfooler
:members:

7
docs/attack_results/attack_result.rst
View File

@@ -1,7 +0,0 @@
===================
Attack Result
===================
.. automodule:: textattack.attack_results.attack_result
:members:

8
docs/attacks/attack.rst Normal file
View File

@@ -0,0 +1,8 @@
========
Attack
========
The ``Attack`` class represents an adversarial attack composed of a goal function, search method, transformation, and constraints.
.. automodule:: textattack.shared.attack
:members:

54
docs/attacks/attack_recipes.rst Normal file
View File

@@ -0,0 +1,54 @@
Attack Recipes
===============
We provide a number of pre-built attack recipes. To run an attack recipe, run::
python -m textattack --recipe [recipe_name]
TextFooler
###########
.. automodule:: textattack.attack_recipes.textfooler_jin_2019
:members:
TextFooler-adjusted
#####################
.. automodule:: textattack.attack_recipes.textfooler_jin_2019_adjusted
:members:
Alzantot
###########
.. automodule:: textattack.attack_recipes.alzantot_2018
:members:
Alzantot-adjusted
###################
.. automodule:: textattack.attack_recipes.alzantot_2018_adjusted
:members:
DeepWordBug
############
.. automodule:: textattack.attack_recipes.deepwordbug_gao_2018
:members:
Hotflip
###########
.. automodule:: textattack.attack_recipes.hotflip_ebrahimi_2017
:members:
Kuleshov
###########
.. automodule:: textattack.attack_recipes.kuleshov_2017
:members:
Seq2Sick
###########
.. automodule:: textattack.attack_recipes.seq2sick_cheng_2018_blackbox
:members:

23
docs/attacks/attack_result.rst Normal file
View File

@@ -0,0 +1,23 @@
===================
Attack Result
===================
The result of an attack's attempt to find a successful adversarial perturbation.
.. automodule:: textattack.attack_results.attack_result
:members:
.. automodule:: textattack.attack_results.successful_attack_result
:members:
.. automodule:: textattack.attack_results.failed_attack_result
:members:
.. automodule:: textattack.attack_results.skipped_attack_result
:members:

162
docs/attacks/constraint.rst Normal file
View File

@@ -0,0 +1,162 @@
.. _constraint:
=============
Constraint
=============
Constraints determine whether a given transformation is valid. Since transformations do not perfectly preserve semantics semantics or grammaticality, constraints can increase the likelihood that the resulting transformation preserves these qualities. All constraints are subclasses of the ``Constraint`` abstract class, and must implement at least one of ``__call__`` or ``call_many``.
We split constraints into three main categories.
:ref:`Semantics`: Based on the meaning of the input and perturbation.
:ref:`Grammaticality`: Based on syntactic properties like part-of-speech and grammar.
:ref:`Overlap`: Based on character-based properties, like edit distance.
A fourth type of constraint restricts the search method from exploring certain parts of the search space:
:ref:`pre_transformation`: Based on the input and index of word replacement.
.. automodule:: textattack.constraints.constraint
:special-members: __call__
:private-members:
:members:
.. _semantics:
Semantics
----------
Semantic constraints determine if a transformation is valid based on similarity
of the semantics of the orignal input and the transformed input.
Word Embedding Distance
########################
.. automodule:: textattack.constraints.semantics.word_embedding_distance
:members:
Sentence Encoders
##################
.. automodule:: textattack.constraints.semantics.sentence_encoders.sentence_encoder
:members:
Thought Vectors
****************
.. automodule:: textattack.constraints.semantics.sentence_encoders.thought_vector
:members:
BERT
*****
.. automodule:: textattack.constraints.semantics.sentence_encoders.bert.bert
:members:
InferSent
***********
.. automodule:: textattack.constraints.semantics.sentence_encoders.infer_sent.infer_sent_model
:members:
.. automodule:: textattack.constraints.semantics.sentence_encoders.infer_sent.infer_sent
:members:
Universal Sentence Encoder
***************************
.. automodule:: textattack.constraints.semantics.sentence_encoders.universal_sentence_encoder.universal_sentence_encoder
:members:
.. _grammaticality:
Grammaticality
-----------------
Grammaticality constraints determine if a transformation is valid based on
syntactic properties of the perturbation.
Language Models
################
.. automodule:: textattack.constraints.grammaticality.language_models.language_model_constraint
:members:
GPT-2
*******
.. automodule:: textattack.constraints.grammaticality.language_models.gpt2
:members:
Google 1-Billion Words Language Model
**************************************
.. automodule:: textattack.constraints.grammaticality.language_models.google_language_model.google_language_model
:members:
LanguageTool Grammar Checker
##############################
.. automodule:: textattack.constraints.grammaticality.language_tool
:members:
Part of Speech
###############
.. automodule:: textattack.constraints.grammaticality.part_of_speech
:members:
.. _overlap:
Overlap
-----------
Overlap constraints determine if a transformation is valid based on character-level analysis.
BLEU Score
############
.. automodule:: textattack.constraints.overlap.bleu_score
:members:
chrF Score
###########
.. automodule:: textattack.constraints.overlap.chrf_score
:members:
Lenvenshtein Edit Distance
############################
.. automodule:: textattack.constraints.overlap.levenshtein_edit_distance
:members:
METEOR Score
#############
.. automodule:: textattack.constraints.overlap.meteor_score
:members:
Maximum Words Perturbed
###########################
.. automodule:: textattack.constraints.overlap.max_words_perturbed
:members:
.. _pre_transformation:
Pre-Transformation
----------
Pre-transformation constraints determine if a transformation is valid based on
only the original input and the position of the replacement. These constraints
are applied before the transformation is even called. For example, these
constraints can prevent search methods from swapping words at the same index
twice, or from replacing stopwords.
Pre-Transformation Constraint
########################
.. automodule:: textattack.constraints.pre_transformation.pre_transformation_constraint
:special-members: __call__
:private-members:
:members:
Stopword Modification
########################
.. automodule:: textattack.constraints.pre_transformation.stopword_modification
:members:
Repeat Modification
########################
.. automodule:: textattack.constraints.pre_transformation.repeat_modification
:members:

30
docs/attacks/goal_function.rst Normal file
View File

@@ -0,0 +1,30 @@
.. _goal_function:
================
Goal Function
================
Goal functions determine if an attack has been successful.
.. automodule:: textattack.goal_functions.goal_function
:members:
:private-members:
Classification
################
.. automodule:: textattack.goal_functions.classification.classification_goal_function
:members:
.. automodule:: textattack.goal_functions.classification.targeted_classification
:members:
.. automodule:: textattack.goal_functions.classification.untargeted_classification
:members:
Text to Text
##############
.. automodule:: textattack.goal_functions.text.text_to_text_goal_function
:members:
.. automodule:: textattack.goal_functions.text.non_overlapping_output
:members:

14
docs/attacks/goal_function_result.rst Normal file
View File

@@ -0,0 +1,14 @@
=======================
Goal Function Result
=======================
Goal function results report the result of a goal function evaluation, indicating whether an attack succeeded for a given example.
.. automodule:: textattack.goal_function_results.goal_function_result
:members:
.. automodule:: textattack.goal_function_results.classification_goal_function_result
:members:
.. automodule:: textattack.goal_function_results.text_to_text_goal_function_result
:members:

36
docs/attacks/search_method.rst Normal file
View File

@@ -0,0 +1,36 @@
=====================
Search Method
=====================
Search methods explore the transformation space in an attempt to find a successful attack as determined by a :ref:`goal_function` and list of :ref:`constraint`\s.
.. automodule:: textattack.search_methods.search_method
:special-members: __call__
:private-members:
:members:
Greedy Search
####################
.. automodule:: textattack.search_methods.greedy_search
:members:
Beam Search
############
.. automodule:: textattack.search_methods.beam_search
:members:
Greedy Word Swap with Word Importance Ranking
##############################################
.. automodule:: textattack.search_methods.greedy_word_swap_wir
:members:
Genetic Algorithm Word Swap
###########################
.. automodule:: textattack.search_methods.genetic_algorithm
:members:

75
docs/attacks/transformation.rst Normal file
View File

@@ -0,0 +1,75 @@
==========================
Transformation
==========================
A transformation is a method which perturbs a text input through the insertion, deletion and substiution of words, characters, and phrases. All transformations take a ``TokenizedText`` as input and return a list of ``TokenizedText``\s that contains possible transformations. Every transformation is a subclass of the abstract ``Transformation`` class.
.. automodule:: textattack.transformations.transformation
:special-members: __call__
:private-members:
:members:
Composite Transformation
--------------------------
Multiple transformations can be used by providing a list of ``Transformation``\s to ``CompositeTransformation``
.. automodule:: textattack.transformations.composite_transformation
:members:
Word Swap
-----------------
Word swap transformations act by replacing some words in the input. Subclasses can implement the abstract ``WordSwap`` class by overriding ``self._get_replacement_words``
.. automodule:: textattack.transformations.word_swap
:private-members:
:members:
Word Swap by Embedding
----------------------
.. automodule:: textattack.transformations.word_swap_embedding
:members:
Word Swap by WordNet Word Replacement
---------------------------------------
.. automodule:: textattack.transformations.word_swap_wordnet
:members:
Word Swap by Gradient
---------------------------------------
.. automodule:: textattack.transformations.word_swap_gradient_based
:members:
Word Swap by Homoglyph
----------------------
.. automodule:: textattack.transformations.word_swap_homoglyph
:members:
Word Swap by Neighboring Character Swap
---------------------------------------
.. automodule:: textattack.transformations.word_swap_neighboring_character_swap
:members:
Word Swap by Random Character Deletion
---------------------------------------
.. automodule:: textattack.transformations.word_swap_random_character_deletion
:members:
Word Swap by Random Character Insertion
---------------------------------------
.. automodule:: textattack.transformations.word_swap_random_character_insertion
:members:
Word Swap by Random Character Substitution
---------------------------------------
.. automodule:: textattack.transformations.word_swap_random_character_substitution
:members:

13
docs/augmentation/augmenter.rst Normal file
View File

@@ -0,0 +1,13 @@
======================
Augmenter
======================
Transformations and constraints can be used outside of an attack for simple NLP data augmentation with the ``Augmenter`` module.
.. automodule:: textattack.augmentation.augmenter
:members:
:exclude-members: DummyTokenizer
.. automodule:: textattack.augmentation.recipes
:members:

27
docs/conf.py
View File

@@ -17,11 +17,11 @@ sys.path.insert(0, os.path.abspath('../'))
# -- Project information -----------------------------------------------------
project = 'TextAttack'
copyright = '2019, UVA QData Lab'
copyright = '2020, UVA QData Lab'
author = 'UVA QData Lab'
# The full version, including alpha/beta/rc tags
release = '0.0.1'
release = '0.0.1.9'
# Set master doc to `index.rst`.
master_doc = 'index'
@@ -35,22 +35,20 @@ extensions = [
'sphinx.ext.viewcode',
'sphinx.ext.autodoc',
'sphinx.ext.napoleon',
"sphinx_rtd_theme"
'sphinx_rtd_theme',
'nbsphinx'
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
templates_path = []
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
exclude_patterns = ['_build', '**.ipynb_checkpoints']
# Mock language_check to stop issues with Sphinx not loading it
autodoc_mock_imports = ["language_check"]
autodoc_mock_imports = []
# -- Options for HTML output -------------------------------------------------
@@ -62,4 +60,13 @@ html_theme = 'sphinx_rtd_theme'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
html_static_path = []
# Path to favicon.
html_favicon = 'favicon.png'
# Don't show module names in front of class names.
add_module_names = False
# Sort members by group
autodoc_member_order = 'groupwise'

14
docs/contraints/constraint.rst
View File

@@ -1,14 +0,0 @@
=============
Contraints
=============
.. automodule:: textattack.constraints.constraint
:members:
We split constraints into three categories:
:ref:`semantics`
:ref:`syntax`
:ref:`overlap`

20
docs/contraints/overlap.rst
View File

@@ -1,20 +0,0 @@
.. _overlap:
==========================================
Constraints based on Syntax and Semantics
==========================================
.. automodule:: textattack.constraints.overlap.bleu_score
:members:
.. automodule:: textattack.constraints.overlap.chrf_score
:members:
.. automodule:: textattack.constraints.overlap.levenshtein_edit_distance
:members:
.. automodule:: textattack.constraints.overlap.meteor_score
:members:
.. automodule:: textattack.constraints.overlap.words_perturbed_percentage
:members:

36
docs/contraints/semantics.rst
View File

@@ -1,36 +0,0 @@
.. _semantics:
================================
Constraints based on Semantics
================================
.. automodule:: textattack.constraints.semantics.word_embedding_distance
:members:
Sentence Encoders
##################
.. automodule:: textattack.constraints.semantics.sentence_encoders.sentence_encoder
:members:
.. automodule:: textattack.constraints.semantics.sentence_encoders.bert.bert
:members:
.. automodule:: textattack.constraints.semantics.sentence_encoders.infer_sent.infer_sent_model
:members:
.. automodule:: textattack.constraints.semantics.sentence_encoders.infer_sent.infer_sent
:members:
.. automodule:: textattack.constraints.semantics.sentence_encoders.universal_sentence_encoder.universal_sentence_encoder
:members:
Language Models
################
.. automodule:: textattack.constraints.semantics.language_models.google_language_model.google_language_model
:members:
.. automodule:: textattack.constraints.semantics.language_models.google_language_model.alzantot_goog_lm
:members:

11
docs/contraints/syntax.rst
View File

@@ -1,11 +0,0 @@
.. _syntax:
==============================
Constraints based on Syntax
==============================
.. automodule:: textattack.constraints.syntax.language_tool
:members:
.. automodule:: textattack.constraints.syntax.part_of_speech
:members:

19
docs/datasets/classification.rst
View File

@@ -1,19 +0,0 @@
==================================
Bulit-in Classification Datasets
==================================
.. automodule:: textattack.datasets.classification.ag_news
:members:
.. automodule:: textattack.datasets.classification.imdb_sentiment
:members:
.. automodule:: textattack.datasets.classification.kaggle_fake_news
:members:
.. automodule:: textattack.datasets.classification.movie_review_sentiment
:members:
.. automodule:: textattack.datasets.classification.yelp_sentiment
:members:

12
docs/datasets/entailment.rst
View File

@@ -1,12 +0,0 @@
=============================
Built-in Entailment Datasets
=============================
.. automodule:: textattack.datasets.entailment.mnli
:members:
.. automodule:: textattack.datasets.entailment.snli
:members:

6
docs/datasets/generic.rst
View File

@@ -1,6 +0,0 @@
=================
Generic Dataset
=================
.. automodule:: textattack.datasets.dataset
:members:

44
docs/datasets_models/datasets.rst Normal file
View File

@@ -0,0 +1,44 @@
=================
Datasets
=================
.. automodule:: textattack.datasets.dataset
:members:
Classification
###############
.. automodule:: textattack.datasets.classification.classification_dataset
:members:
.. automodule:: textattack.datasets.classification.ag_news
:members:
.. automodule:: textattack.datasets.classification.imdb_sentiment
:members:
.. automodule:: textattack.datasets.classification.kaggle_fake_news
:members:
.. automodule:: textattack.datasets.classification.movie_review_sentiment
:members:
.. automodule:: textattack.datasets.classification.yelp_sentiment
:members:
Entailment
############
.. automodule:: textattack.datasets.entailment.entailment_dataset
:members:
.. automodule:: textattack.datasets.entailment.mnli
:members:
.. automodule:: textattack.datasets.entailment.snli
:members:
Translation
#############
.. automodule:: textattack.datasets.translation.translation_datasets
:members:

126
docs/datasets_models/models.rst Normal file
View File

@@ -0,0 +1,126 @@
Models
===============
TextAttack provides different pre-trained models for testing NLP attacks.
We split models up into two broad categories:
- **Classification**: models that output probability scores for some number of classes. These include models for sentiment classification, topic classification, and entailment.
- **Text-to-text**: models that output a sequence of text. These include models that do translation and summarization.
**Classification models:**
:ref:`BERT`: ``bert-base-uncased`` fine-tuned on various datasets using transformers_.
:ref:`LSTM`: a standard LSTM fine-tuned on various datasets.
:ref:`CNN`: a word-CNN fine-tuned on various datasets.
**Text-to-text models:**
:ref:`T5`: ``T5`` fine-tuned on various datasets using transformers_.
BERT
********
.. _BERT:
.. automodule:: textattack.models.helpers.bert_for_classification
:members:
We provide pre-trained BERT models on the following datasets:
.. automodule:: textattack.models.classification.bert.bert_for_ag_news_classification
:members:
.. automodule:: textattack.models.classification.bert.bert_for_imdb_sentiment_classification
:members:
.. automodule:: textattack.models.classification.bert.bert_for_mr_sentiment_classification
:members:
.. automodule:: textattack.models.classification.bert.bert_for_yelp_sentiment_classification
:members:
.. automodule:: textattack.models.entailment.bert.bert_for_mnli
:members:
.. automodule:: textattack.models.entailment.bert.bert_for_snli
:members:
LSTM
*******
.. _LSTM:
.. automodule:: textattack.models.helpers.lstm_for_classification
:members:
We provide pre-trained LSTM models on the following datasets:
.. automodule:: textattack.models.classification.lstm.lstm_for_ag_news_classification
:members:
.. automodule:: textattack.models.classification.lstm.lstm_for_imdb_sentiment_classification
:members:
.. automodule:: textattack.models.classification.lstm.lstm_for_mr_sentiment_classification
:members:
.. automodule:: textattack.models.classification.lstm.lstm_for_yelp_sentiment_classification
:members:
word-CNN
************
.. _CNN:
.. automodule:: textattack.models.helpers.word_cnn_for_classification
:members:
We provide pre-trained CNN models on the following datasets:
.. automodule:: textattack.models.classification.cnn.word_cnn_for_ag_news_classification
:members:
.. automodule:: textattack.models.classification.cnn.word_cnn_for_imdb_sentiment_classification
:members:
.. automodule:: textattack.models.classification.cnn.word_cnn_for_mr_sentiment_classification
:members:
.. automodule:: textattack.models.classification.cnn.word_cnn_for_yelp_sentiment_classification
:members:
.. _T5:
T5
*****************
.. automodule:: textattack.models.helpers.t5_for_text_to_text
:members:
We provide pre-trained T5 models on the following tasks & datasets:
Translation
##############
.. automodule:: textattack.models.translation.t5.t5_models
:members:
Summarization
##############
.. automodule:: textattack.models.summarization.t5_summarization
:members:
.. _transformers: https://github.com/huggingface/transformers

21
docs/datasets_models/tokenizers.rst Normal file
View File

@@ -0,0 +1,21 @@
===========
Tokenizers
===========
.. automodule:: textattack.tokenizers.tokenizer
:members:
.. automodule:: textattack.tokenizers.auto_tokenizer
:members:
.. automodule:: textattack.tokenizers.spacy_tokenizer
:members:
.. automodule:: textattack.tokenizers.t5_tokenizer
:members:
.. automodule:: textattack.tokenizers.bert_tokenizer
:members:
.. automodule:: textattack.tokenizers.bert_entailment_tokenizer
:members:

0
examples/.gitignore → docs/examples/.gitignore vendored
View File

334
docs/examples/1_Introduction_and_Transformations.ipynb Normal file
View File

@@ -0,0 +1,334 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# The TextAttack🐙 ecosystem: search, transformations, and constraints\n",
"\n",
"An attack in TextAttack consists of four parts.\n",
"\n",
"### Goal function\n",
"\n",
"The **goal function** determines if the attack is successful or not. One common goal function is **untargeted classification**, where the attack tries to perturb an input to change its classification. \n",
"\n",
"### Search method\n",
"The **search method** explores the space of potential transformations and tries to locate a successful perturbation. Greedy search, beam search, and brute-force search are all examples of search methods.\n",
"\n",
"### Transformation\n",
"A **transformation** takes a text input and transforms it, replacing words or phrases with similar ones, while trying not to change the meaning. Paraphrase and synonym substitution are two broad classes of transformations.\n",
"\n",
"### Constraints\n",
"Finally, **constraints** determine whether or not a given transformation is valid. Transformations don't perfectly preserve syntax or semantics, so additional constraints can increase the probability that these qualities are preserved from the source to adversarial example. There are many types of constraints: overlap constraints that measure edit distance, syntactical constraints check part-of-speech and grammar errors, and semantic constraints like language models and sentence encoders."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### A custom transformation\n",
"\n",
"This lesson explains how to create a custom transformation. In TextAttack, many transformations involve *word swaps*: they take a word and try and find suitable substitutes. Some attacks focus on replacing characters with neighboring characters to create \"typos\" (these don't intend to preserve the grammaticality of inputs). Other attacks rely on semantics: they take a word and try to replace it with semantic equivalents.\n",
"\n",
"\n",
"### Banana word swap 🍌\n",
"\n",
"As an introduction to writing transformations for TextAttack, we're going to try a very simple transformation: one that replaces any given word with the word 'banana'. In TextAttack, there's an abstract `WordSwap` class that handles the heavy lifting of breaking sentences into words and avoiding replacement of stopwords. We can extend `WordSwap` and implement a single method, `_get_replacement_words`, to indicate to replace each word with 'banana'."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"from textattack.transformations import WordSwap\n",
"\n",
"class BananaWordSwap(WordSwap):\n",
" \"\"\" Transforms an input by replacing any word with 'banana'.\n",
" \"\"\"\n",
" \n",
" # We don't need a constructor, since our class doesn't require any parameters.\n",
"\n",
" def _get_replacement_words(self, word):\n",
" \"\"\" Returns 'banana', no matter what 'word' was originally.\n",
" \n",
" Returns a list with one item, since `_get_replacement_words` is intended to\n",
" return a list of candidate replacement words.\n",
" \"\"\"\n",
" return ['banana']"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"### Using our transformation\n",
"\n",
"Now we have the transformation chosen, but we're missing a few other things. To complete the attack, we need to choose the **search method** and **constraints**. And to use the attack, we need a **goal function**, a **model** and a **dataset**. (The goal function indicates the task our model performs – in this case, classification – and the type of attack – in this case, we'll perform an untargeted attack.)\n",
"\n",
"### Creating the goal function, model, and dataset\n",
"We are performing an untargeted attack on a classification model, so we'll use the `UntargetedClassification` class. For the model, let's use an LSTM trained for news classification on the AG News dataset. Luckily, TextAttack comes with 1000 text samples from some popular datasets, as well as pretrained models for those datasets. So we don't have to train our own model, or procure someone else's. We can just use the built-in datasets and models for this."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[34;1mtextattack\u001b[0m: Goal function <class 'textattack.goal_functions.classification.untargeted_classification.UntargetedClassification'> matches model LSTMForAGNewsClassification.\n"
]
}
],
"source": [
"# Import the dataset.\n",
"from textattack.datasets.classification import AGNews\n",
"# Create the model.\n",
"from textattack.models.classification.lstm import LSTMForAGNewsClassification\n",
"model = LSTMForAGNewsClassification()\n",
"# Create the goal function using the model.\n",
"from textattack.goal_functions import UntargetedClassification\n",
"goal_function = UntargetedClassification(model)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Creating the attack\n",
"Let's keep it simple: let's use a greedy search method, and let's not use any constraints for now. "
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"from textattack.search_methods import GreedySearch\n",
"from textattack.constraints.pre_transformation import RepeatModification, StopwordModification\n",
"from textattack.shared import Attack\n",
"\n",
"# We're going to use our Banana word swap class as the attack transformation.\n",
"transformation = BananaWordSwap() \n",
"# We'll constrain modification of already modified indices and stopwords\n",
"constraints = [RepeatModification(),\n",
" StopwordModification()]\n",
"# We'll use the Greedy search method\n",
"search_method = GreedySearch()\n",
"# Now, let's make the attack from the 4 components:\n",
"attack = Attack(goal_function, constraints, transformation, search_method)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's print our attack to see all the parameters:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Attack(\n",
" (search_method): GreedySearch\n",
" (goal_function): UntargetedClassification\n",
" (transformation): BananaWordSwap\n",
" (constraints): \n",
" (0): RepeatModification\n",
" (1): StopwordModification\n",
" (is_black_box): True\n",
")\n"
]
}
],
"source": [
"print(attack)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Using the attack\n",
"\n",
"Let's use our attack to attack 10 samples (by setting `num_examples` to 10). Additionally, we set `attack_n` to `True`, which indicates that we should attack 10 samples, no matter what. If the model mispredicts a sample already, it isn't attacked; since `attack_n` is `True`, if a sample is mispredicted, we'll take try the next thing in the dataset, and continue until `num_examples` attacks have been completed."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"12it [00:00, 19.61it/s] \n"
]
}
],
"source": [
"from tqdm import tqdm # tqdm provides us a nice progress bar.\n",
"from textattack.loggers import CSVLogger # tracks a dataframe for us.\n",
"\n",
"results_iterable = attack.attack_dataset(AGNews(), num_examples=10, attack_n=True)\n",
"results = []\n",
"\n",
"logger = CSVLogger(color_method='html')\n",
"\n",
"for result in tqdm(results_iterable, total=10):\n",
" logger.log_attack_result(result)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Visualizing attack results\n",
"\n",
"We are logging `AttackResult` objects using a `CSVLogger`. This logger stores all attack results in a dataframe, which we can easily access and display. Since we set `color_method` to `'html'`, the attack results will display their differences, in color, in HTML. Using `IPython` utilities and `pandas`"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>original_text</th>\n",
" <th>perturbed_text</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Thirst, Fear and Bribes on <font color = red>Desert</font> Escape from Africa <font color = red>AGADEZ</font>, <font color = red>Niger</font> (Reuters) - Customs officers in this dusty Saharan town turned a blind eye as yet another creaking truck piled with grain, smuggled cigarettes and dozens of migrants heading for Europe rumbled off into the desert.</td>\n",
" <td>Thirst, Fear and Bribes on <font color = blue>banana</font> Escape from Africa <font color = blue>banana</font>, <font color = blue>banana</font> (Reuters) - Customs officers in this dusty Saharan town turned a blind eye as yet another creaking truck piled with grain, smuggled cigarettes and dozens of migrants heading for Europe rumbled off into the desert.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Toshiba 20 TV freaks out, sends distress signal See what happens when your warranty runs out?. In this case, a 20 Toshiba owned by Chris van Rossman started sending out the international distress signal at 121.</td>\n",
" <td>Toshiba 20 TV freaks out, sends distress signal See what happens when your warranty runs out?. In this case, a 20 Toshiba owned by Chris van Rossman started sending out the international distress signal at 121.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>British hostage tried fleeing before death: report The portrait of Ken Bigley, who was murdered in Iraq October 7, stands in front of the congregation during a service at Liverpool #39;s Roman Catholic Cathedral on October 10.</td>\n",
" <td>British hostage tried fleeing before death: report The portrait of Ken Bigley, who was murdered in Iraq October 7, stands in front of the congregation during a service at Liverpool #39;s Roman Catholic Cathedral on October 10.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td><font color = purple>Keychain</font> <font color = purple>clicker</font> kills TVs Discrete device turns off televisions, creating a little peace and quiet. Until the yelling starts.</td>\n",
" <td><font color = blue>banana</font> <font color = blue>banana</font> kills TVs Discrete device turns off televisions, creating a little peace and quiet. Until the yelling starts.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td><font color = red>Cleric</font> returns to <font color = red>Iraq</font>, orders march on Najaf <font color = red>Powerful</font> Shiite leader says he plans to <font color = red>lead</font> a mass demonstration today to end fighting. <font color = red>Iraqi</font> <font color = red>hostage</font>: <font color = red>Militants</font> <font color = red>said</font> Wednesday they had <font color = red>kidnapped</font> the brother-in-law of <font color = red>Iraqi</font> Defense <font color = red>Minister</font> Hazem Shaalan</td>\n",
" <td><font color = blue>banana</font> returns to <font color = blue>banana</font>, orders march on Najaf <font color = blue>banana</font> Shiite leader says he plans to <font color = blue>banana</font> a mass demonstration today to end fighting. <font color = blue>banana</font> <font color = blue>banana</font>: <font color = blue>banana</font> <font color = blue>banana</font> Wednesday they had <font color = blue>banana</font> the brother-in-law of <font color = blue>banana</font> Defense <font color = blue>banana</font> Hazem Shaalan</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td><font color = green>Hewitt</font> Beats Roddick to <font color = green>Reach</font> Masters Final HOUSTON (Reuters) - <font color = green>A</font> fired-up Lleyton Hewitt defused hard-hitting Andy Roddick 6-3, 6-2 on Saturday, scurrying into the final of the <font color = green>Masters</font> Cup for the third time in four years.</td>\n",
" <td><font color = blue>banana</font> Beats Roddick to <font color = blue>banana</font> Masters Final HOUSTON (Reuters) - <font color = blue>banana</font> fired-up Lleyton Hewitt defused hard-hitting Andy Roddick 6-3, 6-2 on Saturday, scurrying into the final of the <font color = blue>banana</font> Cup for the third time in four years.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td><font color = blue>Despite</font> <font color = blue>booming</font> economy, no concrete move on <font color = blue>debt</font> relief (AFP) AFP - Senior finance officials have hailed a robust global <font color = blue>economic</font> recovery, albeit one threatened by surging <font color = blue>oil</font> prices, but made little headway pushing China toward currency reform and took no firm <font color = blue>steps</font> to ease the debt of the world's poorest nations.</td>\n",
" <td><font color = red>banana</font> <font color = red>banana</font> economy, no concrete move on <font color = red>banana</font> relief (AFP) AFP - Senior finance officials have hailed a robust global <font color = red>banana</font> recovery, albeit one threatened by surging <font color = red>banana</font> prices, but made little headway pushing China toward currency reform and took no firm <font color = red>banana</font> to ease the debt of the world's poorest nations.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td><font color = red>Ethiopian</font> court sentences 3 former <font color = red>rebels</font> to death for mass murders (Canadian <font color = red>Press</font>) Canadian <font color = red>Press</font> - ADDIS ABABA, <font color = red>Ethiopia</font> (AP) - A court has sentenced three former <font color = red>rebels</font> to death for <font color = red>killing</font> dozens of people while rebel factions jockeyed for power more than a decade ago, a government spokesman said Thursday.</td>\n",
" <td><font color = blue>banana</font> court sentences 3 former <font color = blue>banana</font> to death for mass murders (Canadian <font color = blue>banana</font>) Canadian <font color = blue>banana</font> - ADDIS ABABA, <font color = blue>banana</font> (AP) - A court has sentenced three former <font color = blue>banana</font> to death for <font color = blue>banana</font> dozens of people while rebel factions jockeyed for power more than a decade ago, a government spokesman said Thursday.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>Just a close call for closer No need to check with your primary care <font color = green>provider</font> about the origin of that <font color = green>nervous</font> <font color = green>tic</font> you woke up with this morning, on the first full day of autumn. Ninth-inning home runs allowed in three consecutive <font color = green>games</font> by Sox closer <font color = green>Keith</font> Foulke, who also was tagged with blown saves in each of the last two <font color = green>games</font>, were enough to leave ...</td>\n",
" <td>Just a close call for closer No need to check with your primary care <font color = blue>banana</font> about the origin of that <font color = blue>banana</font> <font color = blue>banana</font> you woke up with this morning, on the first full day of autumn. Ninth-inning home runs allowed in three consecutive <font color = blue>banana</font> by Sox closer <font color = blue>banana</font> Foulke, who also was tagged with blown saves in each of the last two <font color = blue>banana</font>, were enough to leave ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td><font color = purple>Study</font>: Wild <font color = purple>Monkeys</font> Resort to Use of Tools WASHINGTON - Wild South American monkeys routinely use fist-sized rocks to crack open seeds and to dig in dry Brazilian soil for grubs and edible tubers, researchers report in the journal Science.</td>\n",
" <td><font color = blue>banana</font>: Wild <font color = blue>banana</font> Resort to Use of Tools WASHINGTON - Wild South American monkeys routinely use fist-sized rocks to crack open seeds and to dig in dry Brazilian soil for grubs and edible tubers, researchers report in the journal Science.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td>Bryant's Request to Seal Evidence Opposed (AP) AP - <font color = green>The</font> prosecutor who charged Kobe Bryant with felony sexual assault has joined news organizations in opposing an attempt by the <font color = green>NBA</font> star's attorney to permanently seal evidence and documents in the case.</td>\n",
" <td>Bryant's Request to Seal Evidence Opposed (AP) AP - <font color = blue>banana</font> prosecutor who charged Kobe Bryant with felony sexual assault has joined news organizations in opposing an attempt by the <font color = blue>banana</font> star's attorney to permanently seal evidence and documents in the case.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>11</th>\n",
" <td>Eagles have lift off Crystal Palace were triumphant last night (Oct 4) over 10-<font color = green>man</font> Fulham at Selhurst Park, lifting themselves off the <font color = green>bottom</font> of the <font color = green>Premiership</font>.</td>\n",
" <td>Eagles have lift off Crystal Palace were triumphant last night (Oct 4) over 10-<font color = purple>banana</font> Fulham at Selhurst Park, lifting themselves off the <font color = purple>banana</font> of the <font color = purple>banana</font>.</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import pandas as pd\n",
"pd.options.display.max_colwidth = 480 # increase colum width so we can actually read the examples\n",
"\n",
"from IPython.core.display import display, HTML\n",
"display(HTML(logger.df[['original_text', 'perturbed_text']].to_html(escape=False)))"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"### Conclusion 🍌\n",
"\n",
"We can examine these examples for a good idea of how many words had to be changed to \"banana\" to change the prediction score from the correct class to another class. The examples without perturbed words were originally misclassified, so they were skipped by the attack. Looks like some examples needed only a single \"banana\", while others needed up to 17 \"banana\" substitutions to change the class score. Wow!"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
}
},
"nbformat": 4,
"nbformat_minor": 2
}

131
examples/[2] Constraints.ipynb → docs/examples/2_Constraints.ipynb
View File

@@ -14,7 +14,7 @@
"\n",
"- **Overlap constraints** determine if a perturbation is valid based on character-level analysis. For example, some attacks are constrained by edit distance: a perturbation is only valid if it perturbs some small number of characters (or fewer).\n",
"\n",
"- **Syntactic constraints** filter inputs based on their syntax. For example, an attack may require that adversarial perturbations do not introduce grammatical errors.\n",
"- **Grammaticality constraints** filter inputs based on syntactical information. For example, an attack may require that adversarial perturbations do not introduce grammatical errors.\n",
"\n",
"- **Semantic constraints** try to ensure that the perturbation is semantically similar to the original input. For example, we may design a constraint that uses a sentence encoder to encode the original and perturbed inputs, and enforce that the sentence encodings be within some fixed distance of one another. (This is what happens in subclasses of `textattack.constraints.semantics.sentence_encoders`.)"
]
@@ -35,7 +35,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## A custom constraint\n",
"### A custom constraint\n",
"\n",
"\n",
"For fun, we're going to see what happens when we constrain an attack to only allow perturbations that substitute out a named entity for another. In linguistics, a **named entity** is a proper noun, the name of a person, organization, location, product, etc. Named Entity Recognition is a popular NLP task (and one that state-of-the-art models can perform quite well). \n",
@@ -45,21 +45,59 @@
"\n",
"**NLTK**, the Natural Language Toolkit, is a Python package that helps developers write programs that process natural language. NLTK comes with predefined algorithms for lots of linguistic tasks– including Named Entity Recognition.\n",
"\n",
"First, we're going to write a constraint class. In the `__call__` method, we're going to use NLTK to find the named entities in both `x` and `x_adv`. We will only return `True` (that is, our constraint is met) if `x_adv` has substituted one named entity in `x` for another."
"First, we're going to write a constraint class. In the `__call__` method, we're going to use NLTK to find the named entities in both `x` and `x_adv`. We will only return `True` (that is, our constraint is met) if `x_adv` has substituted one named entity in `x` for another.\n",
"\n",
"Let's import NLTK and download the required modules:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"[nltk_data] Downloading package punkt to /u/edl9cy/nltk_data...\n",
"[nltk_data] Package punkt is already up-to-date!\n",
"[nltk_data] Downloading package maxent_ne_chunker to\n",
"[nltk_data] /u/edl9cy/nltk_data...\n",
"[nltk_data] Package maxent_ne_chunker is already up-to-date!\n",
"[nltk_data] Downloading package words to /u/edl9cy/nltk_data...\n",
"[nltk_data] Package words is already up-to-date!\n"
]
},
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import nltk\n",
"nltk.download('punkt') # The NLTK tokenizer\n",
"nltk.download('maxent_ne_chunker') # NLTK named-entity chunker\n",
"nltk.download('words') # NLTK list of words"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## NLTK NER Example\n",
"### NLTK NER Example\n",
"\n",
"Here's an example of using NLTK to find the named entities in a sentence:"
]
},
{
"cell_type": "code",
"execution_count": 34,
"execution_count": 2,
"metadata": {},
"outputs": [
{
@@ -90,8 +128,6 @@
}
],
"source": [
"import nltk\n",
"\n",
"sentence = ('In 2017, star quarterback Tom Brady led the Patriots to the Super Bowl, '\n",
" 'but lost to the Philadelphia Eagles.')\n",
"\n",
@@ -115,7 +151,7 @@
},
{
"cell_type": "code",
"execution_count": 51,
"execution_count": 3,
"metadata": {},
"outputs": [
{
@@ -145,14 +181,14 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Putting it all together: getting a list of Named Entity Labels from a sentence\n",
"### Putting it all together: getting a list of Named Entity Labels from a sentence\n",
"\n",
"Now that we know how to tokenize, parse, and detect named entities using NLTK, let's put it all together into a single helper function. Later, when we implement our constraint, we can query this function to easily get the entity labels from a sentence. We can even use `@functools.lru_cache` to try and speed this process up."
]
},
{
"cell_type": "code",
"execution_count": 36,
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
@@ -178,7 +214,7 @@
},
{
"cell_type": "code",
"execution_count": 37,
"execution_count": 5,
"metadata": {},
"outputs": [
{
@@ -200,7 +236,7 @@
" ('.', '.')]"
]
},
"execution_count": 37,
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
@@ -221,14 +257,14 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Creating our NamedEntityConstraint\n",
"### Creating our NamedEntityConstraint\n",
"\n",
"Now that we know how to detect named entities using NLTK, let's create our custom constraint."
]
},
{
"cell_type": "code",
"execution_count": 38,
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
@@ -237,7 +273,7 @@
"class NamedEntityConstraint(Constraint):\n",
" \"\"\" A constraint that ensures `x_adv` only substitutes named entities from `x` with other named entities.\n",
" \"\"\"\n",
" def __call__(self, x, x_adv, original_text=None):\n",
" def _check_constraint(self, x, x_adv, original_text=None):\n",
" x_entities = get_entities(x.text)\n",
" x_adv_entities = get_entities(x_adv.text)\n",
" # If there aren't named entities, let's return False (the attack\n",
@@ -273,16 +309,24 @@
"collapsed": true
},
"source": [
"## Testing our constraint\n",
"### Testing our constraint\n",
"\n",
"We need to create an attack and a dataset to test our constraint on. We went over all of this in the first tutorial, so let's gloss over this part for now."
]
},
{
"cell_type": "code",
"execution_count": 39,
"execution_count": 7,
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[34;1mtextattack\u001b[0m: Goal function <class 'textattack.goal_functions.classification.untargeted_classification.UntargetedClassification'> matches model LSTMForYelpSentimentClassification.\n"
]
}
],
"source": [
"# Import the dataset.\n",
"from textattack.datasets.classification import YelpSentiment\n",
@@ -296,22 +340,24 @@
},
{
"cell_type": "code",
"execution_count": 40,
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"GreedyWordSwap(\n",
"Attack(\n",
" (search_method): GreedySearch\n",
" (goal_function): UntargetedClassification\n",
" (transformation): WordSwapEmbedding(\n",
" (max_candidates): 15\n",
" (embedding_type): paragramcf\n",
" (replace_stopwords): False\n",
" )\n",
" (constraints): \n",
" (0): NamedEntityConstraint\n",
" (1): RepeatModification\n",
" (2): StopwordModification\n",
" (is_black_box): True\n",
")\n"
]
@@ -319,20 +365,31 @@
],
"source": [
"from textattack.transformations import WordSwapEmbedding\n",
"from textattack.attack_methods import GreedyWordSwap\n",
"from textattack.search_methods import GreedySearch\n",
"from textattack.constraints.pre_transformation import RepeatModification, StopwordModification\n",
"from textattack.shared import Attack\n",
"\n",
"# We're going to the `WordSwapEmbedding` transformation. Using the default settings, this\n",
"# will try substituting words with their neighbors in the counter-fitted embedding space. \n",
"transformation = WordSwapEmbedding(max_candidates=15) \n",
"# Now, let's make the attack using these parameters. And add one constraint: our \n",
"# custom NamedEntityConstraint.\n",
"attack = GreedyWordSwap(goal_function, transformation, constraints=[NamedEntityConstraint()])\n",
"\n",
"# We'll use the greedy search method again\n",
"search_method = GreedySearch()\n",
"\n",
"# Our constraints will be the same as Tutorial 1, plus the named entity constraint\n",
"constraints = [RepeatModification(),\n",
" StopwordModification(),\n",
" NamedEntityConstraint()]\n",
"\n",
"# Now, let's make the attack using these parameters. \n",
"attack = Attack(goal_function, constraints, transformation, search_method)\n",
"\n",
"print(attack)"
]
},
{
"cell_type": "code",
"execution_count": 41,
"execution_count": 9,
"metadata": {},
"outputs": [
{
@@ -341,7 +398,7 @@
"True"
]
},
"execution_count": 41,
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
@@ -361,12 +418,13 @@
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"from tqdm import tqdm # tqdm provides us a nice progress bar.\n",
"from textattack.loggers import CSVLogger # tracks a dataframe for us.\n",
"from textattack.attack_results import FailedAttackResult, SkippedAttackResult\n",
"from textattack.attack_results import SuccessfulAttackResult\n",
"\n",
"results_iterable = attack.attack_dataset(YelpSentiment(), attack_n=True)\n",
"logger = CSVLogger(color_method='html')\n",
@@ -374,9 +432,10 @@
"num_successes = 0\n",
"while num_successes < 10:\n",
" result = next(results_iterable)\n",
" if not (isinstance(result, FailedAttackResult) or isinstance(result, SkippedAttackResult)):\n",
" if isinstance(result, SuccessfulAttackResult):\n",
" logger.log_attack_result(result)\n",
" num_successes += 1"
" num_successes += 1\n",
" print(num_successes)"
]
},
{
@@ -398,8 +457,8 @@
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>passage_1</th>\n",
" <th>passage_2</th>\n",
" <th>original_text</th>\n",
" <th>perturbed_text</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
@@ -486,9 +545,9 @@
],
"metadata": {
"kernelspec": {
"display_name": "torch",
"display_name": "Python 3",
"language": "python",
"name": "build_central"
"name": "python3"
},
"language_info": {
"codemirror_mode": {

BIN
docs/favicon.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 187 KiB

15
docs/goal_functions/goal_functions.rst
View File

@@ -1,15 +0,0 @@
================
Goal Functions
================
.. automodule:: textattack.goal_functions.goal_function
:members:
.. automodule:: textattack.goal_functions.goal_function_result
:members:
..automodule:: textattack.goal_functions.targeted_classification
:members:
..automodule:: textattack.goal_functions.untargeted_classification
:members:

147
docs/index.rst
View File

@@ -1,101 +1,78 @@
.. TextAttack documentation master file, created by
sphinx-quickstart on Sat Oct 19 20:54:30 2019.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to TextAttack's documentation!
TextAttack
======================================
.. toctree::
:maxdepth: 2
:caption: User Documentation
`TextAttack <https://github.com/QData/TextAttack>`__ is a Python framework for adversarial attacks and data augmentation in NLP.
users/introduction
users/installation
users/examples
NLP Attacks
-----------
TextAttack provides a framework for constructing and thinking about attacks via perturbation in NLP. TextAttack builds attacks from four components:
- `Goal Functions <attacks/goal_function.html>`__ stipulate the goal of the attack, like to change the prediction score of a classification model, or to change all of the words in a translation output.
- `Constraints <attacks/constraint.html>`__ determine if a potential perturbation is valid with respect to the original input.
- `Transformations <attacks/transformation.html>`__ take a text input and transform it by inserting and deleting characters, words, and/or phrases.
- `Search Methods <attacks/search_method.html>`__ explore the space of possible **transformations** within the defined **constraints** and attempt to find a successful perturbation which satisfies the **goal function**.
TextAttack provides a set of `Attack Recipes <attacks/attack_recipes.html>`__ that assemble attacks from the literature from these four components.
Data Augmentation
-------------
Data augmentation is easy and extremely common in computer vision but harder and less common in NLP. We provide a `Data Augmentation <augmentation/augmenter.html>`__ module using transformations and constraints.
Features
------------
TextAttack has some other features that make it a pleasure to use:
- `Built-in Datasets <datasets_models/datasets.html>`__ for running attacks without supplying your own data
- `Pre-trained Models <datasets_models/models.html>`__ for testing attacks and evaluating constraints
- `Built-in Tokenizers <datasets_models/tokenizers.html>`__ so you don't have to worry about tokenizing the inputs
- `Visualization options <misc/loggers.html>`__ like Weights & Biases and Visdom
.. toctree::
:maxdepth: 2
:caption: Attack Methods:
attack_methods/attack
attack_methods/beam_search
attack_methods/greedy_word_swap
attack_methods/genetic_algorithm
:maxdepth: 1
:hidden:
:caption: Quickstart
quickstart/installation
quickstart/overview
Example 1: Transformations <examples/1_Introduction_and_Transformations.ipynb>
Example 2: Constraints <examples/2_Constraints.ipynb>
.. toctree::
:maxdepth: 2
:caption: Attack Recipes:
:maxdepth: 3
:hidden:
:caption: NLP Attacks
attack_recipes/alzantot_genetic
attack_recipes/gao_deepwordbug
attack_recipes/jin_textfooler
attacks/attack
attacks/attack_result
attacks/goal_function
attacks/goal_function_result
attacks/constraint
attacks/transformation
attacks/search_method
attacks/attack_recipes
.. toctree::
:maxdepth: 3
:hidden:
:caption: Data Augmentation
augmentation/augmenter
.. toctree::
:maxdepth: 2
:caption: Attack Results:
attack_results/attack_result
:maxdepth: 3
:hidden:
:caption: Models, Datasets and Tokenizers
datasets_models/models
datasets_models/datasets
datasets_models/tokenizers
.. toctree::
:maxdepth: 2
:caption: Models:
models/bert
models/lstm
models/cnn
.. toctree::
:maxdepth: 2
:caption: Goal Functions:
goal_functions/goal_functions
.. toctree::
:maxdepth: 2
:caption: Transformations:
transformations/transformation
transformations/composite_transformation
transformations/word_swap
.. toctree::
:maxdepth: 2
:caption: Constraints:
contraints/constraint
contraints/semantics
contraints/syntax
contraints/overlap
.. toctree::
:maxdepth: 2
:caption: Tokenizers:
tokenizers/tokenizer
tokenizers/bert_tokenizer
tokenizers/spacy_tokenizer
.. toctree::
:maxdepth: 2
:caption: Datasets:
datasets/generic
datasets/classification
datasets/entailment
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
:maxdepth: 3
:hidden:
:caption: Miscellaneous
misc/loggers
misc/validators
misc/tokenized_text

29
docs/misc/loggers.rst Normal file
View File

@@ -0,0 +1,29 @@
======================
Loggers
======================
Loggers track, visualize, and export attack results.
.. automodule:: textattack.loggers.logger
:members:
Loggers
########
.. automodule:: textattack.loggers.file_logger
:members:
.. automodule:: textattack.loggers.csv_logger
:members:
.. automodule:: textattack.loggers.visdom_logger
:members:
.. automodule:: textattack.loggers.weights_and_biases_logger
:members:
Log Manager
############
.. automodule:: textattack.loggers.attack_log_manager
:members:

6
docs/misc/tokenized_text.rst Normal file
View File

@@ -0,0 +1,6 @@
===================
Tokenized Text
===================
.. automodule:: textattack.shared.tokenized_text
:members:

8
docs/misc/validators.rst Normal file
View File

@@ -0,0 +1,8 @@
======================
Validators
======================
Validators ensure compatibility between search methods, transformations, constraints, and goal functions.
.. automodule:: textattack.shared.validators
:members:

34
docs/models/bert.rst
View File

@@ -1,34 +0,0 @@
=============
BERT
=============
.. automodule:: textattack.models.helpers.bert_for_classification
:members:
We provide pre-trained BERT models on the following datasets:
Classification
##############
.. automodule:: textattack.models.classification.bert.bert_for_ag_news_classification
:members:
.. automodule:: textattack.models.classification.bert.bert_for_imdb_sentiment_classification
:members:
.. automodule:: textattack.models.classification.bert.bert_for_mr_sentiment_classification
:members:
.. automodule:: textattack.models.classification.bert.bert_for_yelp_sentiment_classification
:members:
Entailment
##########
.. automodule:: textattack.models.entailment.bert.bert_for_mnli
:members:
.. automodule:: textattack.models.entailment.bert.bert_for_snli
:members:

25
docs/models/cnn.rst
View File

@@ -1,25 +0,0 @@
=========
CNN
=========
.. automodule:: textattack.models.helpers.word_cnn_for_classification
:members:
We provide pre-trained CNN models on the following datasets:
Classification
###############
.. automodule:: textattack.models.classification.cnn.word_cnn_for_ag_news_classification
:members:
.. automodule:: textattack.models.classification.cnn.word_cnn_for_imdb_sentiment_classification
:members:
.. automodule:: textattack.models.classification.cnn.word_cnn_for_mr_sentiment_classification
:members:
.. automodule:: textattack.models.classification.cnn.word_cnn_for_yelp_sentiment_classification
:members:

25
docs/models/lstm.rst
View File

@@ -1,25 +0,0 @@
=========
LSTM
=========
.. automodule:: textattack.models.helpers.lstm_for_classification
:members:
We provide pre-trained LSTM models on the following datasets:
Classification
###############
.. automodule:: textattack.models.classification.lstm.lstm_for_ag_news_classification
:members:
.. automodule:: textattack.models.classification.lstm.lstm_for_imdb_sentiment_classification
:members:
.. automodule:: textattack.models.classification.lstm.lstm_for_mr_sentiment_classification
:members:
.. automodule:: textattack.models.classification.lstm.lstm_for_yelp_sentiment_classification
:members:

17
docs/quickstart/installation.rst Normal file
View File

@@ -0,0 +1,17 @@
==============
Installation
==============
To use TextAttack, you must be running Python 3.6+. A CUDA-compatible GPU is optional but will greatly improve speed. To install, simply run::
pip install textattack
You're now all set to use TextAttack! Try running an attack from the command line::
python -m textattack --recipe textfooler --model bert-mr --num-examples 10
This will run an attack using the TextFooler_ recipe, attacking BERT fine-tuned on the MR dataset. It will attack the first 10 samples. Once everything downloads and starts running, you should see attack results print to ``stdout``.
Read on for more information on TextAttack, including how to use it from a Python script (``import textattack``).
.. _TextFooler: https://arxiv.org/abs/1907.11932

52
docs/quickstart/overview.rst Normal file
View File

@@ -0,0 +1,52 @@
===========
Overview
===========
TextAttack builds attacks from four components:
- `Goal Functions <attacks/goal_functions.html>`__ stipulate the goal of the attack, like to change the prediction score of a classification model, or to change all of the words in a translation output.
- `Constraints <attacks/constraints.html>`__ determine if a potential perturbation is valid with respect to the original input.
- `Transformations <attacks/transformations.html>`__ take a text input and transform it by inserting and deleting characters, words, and/or phrases.
- `Search Methods <attacks/search_methods.html>`__ explore the space of possible **transformations** within the defined **constraints** and attempt to find a successful perturbation which satisfies the **goal function**.
Any model that overrides ``__call__``, takes ``TokenizedText`` as input, and formats output correctly can be used with TextAttack. TextAttack also has built-in datasets and pre-trained models on these datasets. Below is an example of attacking a pre-trained model on the AGNews dataset::
from tqdm import tqdm
from textattack.loggers import FileLogger
from textattack.datasets.classification import AGNews
from textattack.models.classification.lstm import LSTMForAGNewsClassification
from textattack.goal_functions import UntargetedClassification
from textattack.shared import Attack
from textattack.search_methods import GreedySearch
from textattack.transformations import WordSwapEmbedding
from textattack.constraints.grammaticality import PartOfSpeech
from textattack.constraints.semantics import RepeatModification, StopwordModification
# Create the model and goal function
model = LSTMForAGNewsClassification()
goal_function = UntargetedClassification(model)
# Use the default WordSwapEmbedding transformation
transformation = WordSwapEmbedding()
# Add a constraint, note that an empty list can be used if no constraints are wanted
constraints = [
RepeatModification(),
StopwordModification(),
PartOfSpeech()
]
# Choose a search method
search = GreedySearch()
# Make an attack with the above parameters
attack = Attack(goal_function, constraints, transformation, search)
# Run the attack on 5 examples and see the results using a logger to output to stdout
results = attack.attack_dataset(AGNews(), num_examples=5, attack_n=True)
logger = FileLogger(stdout=True)
for result in tqdm(results, total=5):
logger.log_attack_result(result)

2
docs/requirements.txt Normal file
View File

@@ -0,0 +1,2 @@
nbsphinx
sphinx-rtd-theme

10
docs/tokenizers/bert_tokenizer.rst
View File

@@ -1,10 +0,0 @@
===============
Bert Tokenizer
===============
.. automodule:: textattack.tokenizers.bert_tokenizer
:members:
.. automodule:: textattack.tokenizers.bert_entailment_tokenizer
:members:

6
docs/tokenizers/spacy_tokenizer.rst
View File

@@ -1,6 +0,0 @@
================
Spacy Tokenizer
================
.. automodule:: textattack.tokenizers.spacy_tokenizer
:members:

9
docs/tokenizers/tokenizer.rst
View File

@@ -1,9 +0,0 @@
==========
Tokenizer
==========
.. automodule:: textattack.tokenizers.tokenizer
:members:
.. automodule:: textattack.shared.tokenized_text
:members:

6
docs/transformations/composite_transformation.rst
View File

@@ -1,6 +0,0 @@
==========================
Composite Transformation
==========================
.. automodule:: textattack.transformations.composite_transformation
:members:

6
docs/transformations/transformation.rst
View File

@@ -1,6 +0,0 @@
==========================
Transformation
==========================
.. automodule:: textattack.transformations.transformation
:members:

24
docs/transformations/word_swap.rst
View File

@@ -1,24 +0,0 @@
=============
Word Swap
=============
.. automodule:: textattack.transformations.word_swap
:members:
.. automodule:: textattack.transformations.word_swap_embedding
:members:
.. automodule:: textattack.transformations.word_swap_homoglyph
:members:
.. automodule:: textattack.transformations.word_swap_neighboring_character_swap
:members:
.. automodule:: textattack.transformations.word_swap_random_character_deletion
:members:
.. automodule:: textattack.transformations.word_swap_random_character_insertion
:members:
.. automodule:: textattack.transformations.word_swap_random_character_substitution
:members:

10
docs/users/examples.rst
View File

@@ -1,10 +0,0 @@
=========
Examples
=========
BERT Example
############
.. parsed-literal::
To come

44
docs/users/installation.rst
View File

@@ -1,44 +0,0 @@
==============
Installation
==============
### First download
```
git clone https://github.com/QData/TextAttack.git
```
### Installation
You should be running Python 3.6+ to use this package. A CUDA-compatible GPU is optional but will greatly improve code speed. After cloning this git repository, run the following commands to install the `textattack` page a `conda` environment:
```
conda create -n text-attack python=3.7
conda activate text-attack
pip install -e .
```
We use the NLTK package for its list of stopwords and access to the WordNet lexical database. To download them run in Python shell:
```
import nltk
nltk.download('stopwords')
nltk.download('wordnet')
```
We use spaCy's English model. To download it, after installing spaCy run:
```
python -m spacy download en
```
### Cache
TextAttack provides pretrained models and datasets for user convenience. By default, all this stuff is downloaded to `~/.cache`. You can change this location by editing the `CACHE_DIR` field in `config.json`.
### Common Errors
#### Errors regarding GCC
If you see an error that GCC is incompatible, make sure your system has an up-to-date version of the GCC compiler.
#### Errors regarding Java
Using the LanguageTool constraint relies on Java 8 internally (it's not ideal, we know). Please install Java 8 if you're interested in using the LanguageTool grammaticality constraint.

5
docs/users/introduction.rst
View File

@@ -1,5 +0,0 @@
=====================
What is TextAttack?
=====================
TextAttack is a library for running adversarial attacks against NLP models. These may be useful for evaluating attack methods and evaluating model robustness. TextAttack is designed in order to be easily extensible to new NLP tasks, models, attack methods, and attack constraints. The separation between these aspects of an adversarial attack and standardization of constraint evaluation allows for easier ablation studies. TextAttack supports attacks on models trained for classification and entailment.

312
examples/[1] Introduction & Transformations.ipynb
View File

@@ -1,312 +0,0 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# The TextAttack🐙 ecosystem: search, transformations, and constraints\n",
"\n",
"An attack in TextAttack consists of three basic parts.\n",
"\n",
"### Search method\n",
"The **search method** which explores the space of potential transformations and tries to locate a successful attack. Greedy search, beam search, and brute-force search are all examples of search methods. Since the search method is the backbone of the attack, the term \"search\" is often substituted with \"attack method\" or just \"attack\". In TextAttack, all three of those terms (search, attack, attack method) mean the same thing.\n",
"\n",
"### Transformation\n",
"A **transformation** takes a text input and transforms it, replacing words or phrases with similar ones, while trying not to change the meaning. Paraphrase and synonym substitution are two broad classes of transformations.\n",
"\n",
"### Constraints\n",
"Finally, **constraints** determine whether or not a given transformation is valid. Transformations don't perfectly preserve syntax or semantics, so additional constraints can increase the probability that these qualities are preserved from the source to adversarial example. There are many types of constraints: overlap constraints that measure edit distance, syntactical constraints check part-of-speech and grammar errors, and semantic constraints like language models and sentence encoders."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## A custom transformation\n",
"\n",
"This lesson explains how to create a custom transformation. In TextAttack, many transformations involve *word swaps*: they take a word and try and find suitable substitutes. Some attacks focus on replacing characters with neighboring characters to create \"typos\". (These don't intend to preserve the grammaticality of inputs.) Other attacks rely on semantics: they take a word and try to replace it with semantic equivalents.\n",
"\n",
"\n",
"### Banana word swap 🍌\n",
"\n",
"As an introduction to writing transformations for TextAttack, we're going to try a very simple transformation: one that replaces any given word with the word 'banana'. In TextAttack, there's an abstract `WordSwap` class that handles the heavy lifting of breaking sentences into words and avoiding replacement of stopwords. We can extend `WordSwap` and implement a single method, `_get_replacement_words`, to indicate to replace each word with 'banana'."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"from textattack.transformations.word_swap import WordSwap\n",
"\n",
"class BananaWordSwap(WordSwap):\n",
" \"\"\" Transforms an input by replacing any word with 'banana'.\n",
" \"\"\"\n",
" \n",
" # We don't need a constructor, since our class doesn't require any parameters.\n",
"\n",
" def _get_replacement_words(self, word):\n",
" \"\"\" Returns 'banana', no matter what 'word' was originally.\n",
" \n",
" Returns a list with one item, since `_get_replacement_words` is intended to\n",
" return a list of candidate replacement words.\n",
" \"\"\"\n",
" return ['banana']"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"## Using our transformation\n",
"\n",
"Now we have the transformation chosen, but we're missing a few other things. To complete the attack, we need to choose the **search method** and **constraints**. And to use the attack, we need a **goal function**, a **model** and a **dataset**. (The goal function indicates the task our model performs – in this case, classification – and the type of attack – in this case, we'll perform an untargeted attack.)\n",
"\n",
"### Creating the goal function, model, and dataset\n",
"We are performing an untargeted attack on a classification model, so we'll use the `UntargetedClassification` class. For the model, let's use an LSTM trained for news classification on the AG News dataset. Luckily, TextAttack comes with 1000 text samples from some popular datasets, as well as pretrained models for those datasets. So we don't have to train our own model, or procure someone else's. We can just use the built-in datasets and models for this."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# Import the dataset.\n",
"from textattack.datasets.classification import AGNews\n",
"# Create the model.\n",
"from textattack.models.classification.lstm import LSTMForAGNewsClassification\n",
"model = LSTMForAGNewsClassification()\n",
"# Create the goal function using the model.\n",
"from textattack.goal_functions import UntargetedClassification\n",
"goal_function = UntargetedClassification(model)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Creating the attack\n",
"Let's keep it simple: let's use a greedy search method, and let's not use any constraints for now. "
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"from textattack.attack_methods import GreedyWordSwap\n",
"\n",
"# We're going to use our Banana word swap class as the attack transformation.\n",
"transformation = BananaWordSwap() \n",
"# And, we don't want to use any constraints.\n",
"constraints = []\n",
"# Now, let's make the attack using these parameters:\n",
"attack = GreedyWordSwap(goal_function, transformation, constraints)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's print our attack to see all the parameters:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"GreedyWordSwap(\n",
" (goal_function): UntargetedClassification\n",
" (transformation): BananaWordSwap(\n",
" (replace_stopwords): False\n",
" )\n",
" (constraints): \n",
" \n",
" (is_black_box): True\n",
")\n"
]
}
],
"source": [
"print(attack)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Using the attack\n",
"\n",
"Let's use our attack to attack 10 samples (by setting `num_examples` to 10). Additionally, we set `attack_n` to `True`, which indicates that we should attack 10 samples, no matter what. If the model mispredicts a sample already, it isn't attacked; since `attack_n` is `True`, if a sample is mispredicted, we'll take try the next thing in the dataset, and continue until `num_examples` attacks have been completed."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"12it [00:03, 3.18it/s] \n"
]
}
],
"source": [
"from tqdm import tqdm # tqdm provides us a nice progress bar.\n",
"from textattack.loggers import CSVLogger # tracks a dataframe for us.\n",
"\n",
"results_iterable = attack.attack_dataset(AGNews(), num_examples=10, attack_n=True)\n",
"results = []\n",
"\n",
"logger = CSVLogger(color_method='html')\n",
"\n",
"for result in tqdm(results_iterable, total=10):\n",
" logger.log_attack_result(result)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Visualizing attack results\n",
"\n",
"We are logging `AttackResult` objects using a `CSVLogger`. This logger stores all attack results in a dataframe, which we can easily access and display. Since we set `color_method` to `'html'`, the attack results will display their differences, in color, in HTML. Using `IPython` utilities and `pandas`"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>passage_1</th>\n",
" <th>passage_2</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Thirst, Fear and Bribes on Desert Escape from Africa <font color = red>AGADEZ</font>, Niger (Reuters) - <font color = red>Customs</font> officers in this dusty Saharan town turned a <font color = red>blind</font> <font color = red>eye</font> as <font color = red>yet</font> another creaking truck piled with grain, smuggled cigarettes and dozens of migrants heading for Europe rumbled off into the desert.</td>\n",
" <td>Thirst, Fear and Bribes on Desert Escape from Africa <font color = blue>banana</font>, Niger (Reuters) - <font color = blue>banana</font> officers in this dusty Saharan town turned a <font color = blue>banana</font> <font color = blue>banana</font> as <font color = blue>banana</font> another creaking truck piled with grain, smuggled cigarettes and dozens of migrants heading for Europe rumbled off into the desert.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Toshiba 20 TV freaks out, sends distress signal See what <font color = purple>happens</font> when your warranty runs out?. In this case, a 20 Toshiba owned by Chris van Rossman started sending out the international distress signal at 121.</td>\n",
" <td>Toshiba 20 TV freaks out, sends distress signal See what <font color = blue>banana</font> when your warranty runs out?. In this case, a 20 Toshiba owned by Chris van Rossman started sending out the international distress signal at 121.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>British hostage tried fleeing before death: report The portrait of Ken Bigley, who was murdered in Iraq October 7, stands in front of the congregation during a service at Liverpool #39;s Roman Catholic Cathedral on October 10.</td>\n",
" <td>British hostage tried fleeing before death: report The portrait of Ken Bigley, who was murdered in Iraq October 7, stands in front of the congregation during a service at Liverpool #39;s Roman Catholic Cathedral on October 10.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td><font color = purple>Keychain</font> <font color = purple>clicker</font> kills <font color = purple>TVs</font> <font color = purple>Discrete</font> <font color = purple>device</font> <font color = purple>turns</font> off <font color = purple>televisions</font>, <font color = purple>creating</font> a <font color = purple>little</font> <font color = purple>peace</font> and <font color = purple>quiet</font>. Until the yelling starts.</td>\n",
" <td><font color = blue>banana</font> <font color = blue>banana</font> kills <font color = blue>banana</font> <font color = blue>banana</font> <font color = blue>banana</font> <font color = blue>banana</font> off <font color = blue>banana</font>, <font color = blue>banana</font> a <font color = blue>banana</font> <font color = blue>banana</font> and <font color = blue>banana</font>. Until the yelling starts.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td><font color = red>Cleric</font> <font color = red>returns</font> to <font color = red>Iraq</font>, <font color = red>orders</font> <font color = red>march</font> on <font color = red>Najaf</font> <font color = red>Powerful</font> <font color = red>Shiite</font> <font color = red>leader</font> <font color = red>says</font> he <font color = red>plans</font> to <font color = red>lead</font> a <font color = red>mass</font> <font color = red>demonstration</font> <font color = red>today</font> to <font color = red>end</font> <font color = red>fighting</font>. <font color = red>Iraqi</font> <font color = red>hostage</font>: Militants <font color = red>said</font> <font color = red>Wednesday</font> they had <font color = red>kidnapped</font> the <font color = red>brother</font>-in-<font color = red>law</font> of <font color = red>Iraqi</font> <font color = red>Defense</font> <font color = red>Minister</font> <font color = red>Hazem</font> <font color = red>Shaalan</font></td>\n",
" <td><font color = blue>banana</font> <font color = blue>banana</font> to <font color = blue>banana</font>, <font color = blue>banana</font> <font color = blue>banana</font> on <font color = blue>banana</font> <font color = blue>banana</font> <font color = blue>banana</font> <font color = blue>banana</font> <font color = blue>banana</font> he <font color = blue>banana</font> to <font color = blue>banana</font> a <font color = blue>banana</font> <font color = blue>banana</font> <font color = blue>banana</font> to <font color = blue>banana</font> <font color = blue>banana</font>. <font color = blue>banana</font> <font color = blue>banana</font>: Militants <font color = blue>banana</font> <font color = blue>banana</font> they had <font color = blue>banana</font> the <font color = blue>banana</font>-in-<font color = blue>banana</font> of <font color = blue>banana</font> <font color = blue>banana</font> <font color = blue>banana</font> <font color = blue>banana</font> <font color = blue>banana</font></td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>Hewitt Beats Roddick to <font color = green>Reach</font> <font color = green>Masters</font> Final HOUSTON (Reuters) - A fired-up <font color = green>Lleyton</font> <font color = green>Hewitt</font> <font color = green>defused</font> <font color = green>hard</font>-<font color = green>hitting</font> Andy Roddick 6-3, 6-2 on <font color = green>Saturday</font>, scurrying into the final of the Masters Cup for the third time in four years.</td>\n",
" <td>Hewitt Beats Roddick to <font color = blue>banana</font> <font color = blue>banana</font> Final HOUSTON (Reuters) - A fired-up <font color = blue>banana</font> <font color = blue>banana</font> <font color = blue>banana</font> <font color = blue>banana</font>-<font color = blue>banana</font> Andy Roddick 6-3, 6-2 on <font color = blue>banana</font>, scurrying into the final of the Masters Cup for the third time in four years.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td><font color = blue>Despite</font> <font color = blue>booming</font> <font color = blue>economy</font>, no <font color = blue>concrete</font> <font color = blue>move</font> on <font color = blue>debt</font> <font color = blue>relief</font> (AFP) AFP - Senior finance officials have hailed a robust global economic recovery, albeit one threatened by surging oil <font color = blue>prices</font>, but <font color = blue>made</font> <font color = blue>little</font> <font color = blue>headway</font> <font color = blue>pushing</font> <font color = blue>China</font> <font color = blue>toward</font> <font color = blue>currency</font> <font color = blue>reform</font> and <font color = blue>took</font> no firm steps to ease the debt of the world's poorest nations.</td>\n",
" <td><font color = red>banana</font> <font color = red>banana</font> <font color = red>banana</font>, no <font color = red>banana</font> <font color = red>banana</font> on <font color = red>banana</font> <font color = red>banana</font> (AFP) AFP - Senior finance officials have hailed a robust global economic recovery, albeit one threatened by surging oil <font color = red>banana</font>, but <font color = red>banana</font> <font color = red>banana</font> <font color = red>banana</font> <font color = red>banana</font> <font color = red>banana</font> <font color = red>banana</font> <font color = red>banana</font> <font color = red>banana</font> and <font color = red>banana</font> no firm steps to ease the debt of the world's poorest nations.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>Just a <font color = green>close</font> call for closer No <font color = green>need</font> to <font color = green>check</font> with your primary care provider about the <font color = green>origin</font> of that <font color = green>nervous</font> tic you woke up with this <font color = green>morning</font>, on the first full day of autumn. Ninth-inning home runs allowed in three consecutive games by Sox <font color = green>closer</font> Keith <font color = green>Foulke</font>, who also was tagged with blown saves in each of the last two <font color = green>games</font>, were <font color = green>enough</font> to <font color = green>leave</font> ...</td>\n",
" <td>Just a <font color = blue>banana</font> call for closer No <font color = blue>banana</font> to <font color = blue>banana</font> with your primary care provider about the <font color = blue>banana</font> of that <font color = blue>banana</font> tic you woke up with this <font color = blue>banana</font>, on the first full day of autumn. Ninth-inning home runs allowed in three consecutive games by Sox <font color = blue>banana</font> Keith <font color = blue>banana</font>, who also was tagged with blown saves in each of the last two <font color = blue>banana</font>, were <font color = blue>banana</font> to <font color = blue>banana</font> ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td><font color = purple>Study</font>: Wild Monkeys Resort to Use of Tools WASHINGTON - Wild South <font color = purple>American</font> <font color = purple>monkeys</font> <font color = purple>routinely</font> <font color = purple>use</font> <font color = purple>fist</font>-<font color = purple>sized</font> <font color = purple>rocks</font> to crack open <font color = purple>seeds</font> and to dig in dry Brazilian soil for grubs and <font color = purple>edible</font> <font color = purple>tubers</font>, <font color = purple>researchers</font> report in the journal Science.</td>\n",
" <td><font color = blue>banana</font>: Wild Monkeys Resort to Use of Tools WASHINGTON - Wild South <font color = blue>banana</font> <font color = blue>banana</font> <font color = blue>banana</font> <font color = blue>banana</font> <font color = blue>banana</font>-<font color = blue>banana</font> <font color = blue>banana</font> to crack open <font color = blue>banana</font> and to dig in dry Brazilian soil for grubs and <font color = blue>banana</font> <font color = blue>banana</font>, <font color = blue>banana</font> report in the journal Science.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>Bryant's Request to Seal Evidence <font color = green>Opposed</font> (AP) AP - The prosecutor who charged Kobe Bryant with <font color = green>felony</font> sexual assault has <font color = green>joined</font> news organizations in <font color = green>opposing</font> an attempt by the <font color = green>NBA</font> star's attorney to permanently seal evidence and documents in the case.</td>\n",
" <td>Bryant's Request to Seal Evidence <font color = blue>banana</font> (AP) AP - The prosecutor who charged Kobe Bryant with <font color = blue>banana</font> sexual assault has <font color = blue>banana</font> news organizations in <font color = blue>banana</font> an attempt by the <font color = blue>banana</font> star's attorney to permanently seal evidence and documents in the case.</td>\n",
" </tr>\n",
" <tr>\n",
" <th>10</th>\n",
" <td><font color = green>Eagles</font> have lift off Crystal Palace were triumphant last night (Oct 4) over 10-man Fulham at <font color = green>Selhurst</font> <font color = green>Park</font>, <font color = green>lifting</font> themselves off the bottom of the Premiership.</td>\n",
" <td><font color = blue>banana</font> have lift off Crystal Palace were triumphant last night (Oct 4) over 10-man Fulham at <font color = blue>banana</font> <font color = blue>banana</font>, <font color = blue>banana</font> themselves off the bottom of the Premiership.</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import pandas as pd\n",
"pd.options.display.max_colwidth = 480 # increase colum width so we can actually read the examples\n",
"\n",
"from IPython.core.display import display, HTML\n",
"display(HTML(logger.df[['passage_1', 'passage_2']].to_html(escape=False)))"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"## Conclusion 🍌\n",
"\n",
"We can examine these examples for a good idea of how many words had to be changed to \"banana\" to change the prediction score from the correct class to another class. Looks like some examples needed only a single \"banana\", while others needed up to 17 \"banana\" substitutions to change the class score. Wow!"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "torch",
"language": "python",
"name": "build_central"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
}
},
"nbformat": 4,
"nbformat_minor": 2
}

23
local_tests/command_line_tests.py
View File

@@ -16,6 +16,16 @@ def register_test(command, name=None, output_file=None, desc=None):
## BEGIN TESTS ##
#######################################
#
# test: run_attack --interactive
#
register_test(('printf "All that glitters is not gold\nq\n"',
'python -m textattack --recipe textfooler --model bert-imdb --interactive'),
name='interactive_mode',
output_file='local_tests/sample_outputs/interactive_mode.txt',
desc='Runs textfooler attack on BERT trained on IMDB using interactive mode')
#
# test: run_attack_parallel textfooler attack on 10 samples from BERT MR
# (takes about 81s)
@@ -29,10 +39,10 @@ register_test('python -m textattack --model bert-mr --recipe textfooler --num-ex
# test: run_attack_parallel textfooler attack on 10 samples from BERT SNLI
# (takes about 51s)
#
register_test('python -m textattack --model bert-snli --recipe textfooler --num-examples 10',
register_test('python -m textattack --model bert-snli --recipe deepwordbug --num-examples 10',
name='run_attack_textfooler_bert_snli_10',
output_file='local_tests/sample_outputs/run_attack_textfooler_bert_snli_10.txt',
desc='Runs attack using TextFooler recipe on BERT using 10 examples from the SNLI dataset')
output_file='local_tests/sample_outputs/run_attack_deepwordbug_bert_snli_10.txt',
desc='Runs attack using DeepWordBug recipe on BERT using 10 examples from the SNLI dataset')
#
# test: run_attack deepwordbug attack on 10 samples from LSTM MR
@@ -51,7 +61,7 @@ register_test('python -m textattack --model lstm-mr --recipe deepwordbug --num-e
#
register_test(('python -m textattack --attack-n --goal-function targeted-classification:target_class=2 '
'--enable-csv --model bert-mnli --num-examples 4 --transformation word-swap-wordnet '
'--constraints lang-tool --attack beam-search:beam_width=2'),
'--constraints lang-tool repeat stopword --search beam-search:beam_width=2'),
name='run_attack_targeted2_bertmnli_wordnet_beamwidth_2_enablecsv_attackn',
output_file='local_tests/sample_outputs/run_attack_targetedclassification2_wordnet_langtool_enable_csv_beamsearch2_attack_n_4.txt',
desc=('Runs attack using targeted classification on class 2 on BERT MNLI with'
@@ -67,10 +77,11 @@ register_test(('python -m textattack --attack-n --goal-function targeted-classif
#
register_test(('python -m textattack --attack-n --goal-function non-overlapping-output '
'--model t5-en2de --num-examples 6 --transformation word-swap-random-char-substitution '
'--constraints edit-distance:12 words-perturbed:max_percent=0.75 --attack greedy-word'),
'--constraints edit-distance:12 max-words-perturbed:max_percent=0.75 repeat stopword '
'--search greedy'),
name='run_attack_nonoverlapping_t5en2de_randomcharsub_editdistance_wordsperturbed_greedyword',
output_file='local_tests/sample_outputs/run_attack_nonoverlapping_t5ende_editdistance_bleu.txt',
desc=('Runs attack using targeted classification on class 2 on BERT MNLI with'
'enable_csv and attack_n set, using the WordNet transformation and beam '
'search with beam width 2, using language tool constraint, on 10 samples')
)
)

41
local_tests/sample_outputs/interactive_mode.txt Normal file
View File

@@ -0,0 +1,41 @@
Attack(
(search_method): GreedyWordSwapWIR(
(wir_method): unk
)
(goal_function): UntargetedClassification
(transformation): WordSwapEmbedding(
(max_candidates): 50
(embedding_type): paragramcf
)
(constraints):
(0): WordEmbeddingDistance(
(embedding_type): paragramcf
(min_cos_sim): 0.5
(cased): False
(include_unknown_words): True
)
(1): PartOfSpeech(
(tagset): universal
(allow_verb_noun_swap): True
)
(2): UniversalSentenceEncoder(
(metric): angular
(threshold): 0.904458599
(compare_with_original): False
(window_size): 15
(skip_text_shorter_than_window): True
)
(3): RepeatModification
(4): StopwordModification
(is_black_box): True
)
Load time: /.*/s
Running in interactive mode
----------------------------
Enter a sentence to attack or "q" to quit:
Attacking...
1-->0
All that glitters is not gold
All that glisten is not gold
Enter a sentence to attack or "q" to quit:

93
local_tests/sample_outputs/run_attack_textfooler_bert_snli_10.txt → local_tests/sample_outputs/run_attack_deepwordbug_bert_snli_10.txt
View File

@@ -1,32 +1,19 @@
/.*/GreedyWordSwapWIR(
(goal_function): UntargetedClassification
(transformation): WordSwapEmbedding(
(max_candidates): 50
(embedding_type): paragramcf
(replace_stopwords): False
Attack(
(search_method): GreedyWordSwapWIR(
(wir_method): unk
)
(goal_function): UntargetedClassification
(transformation): CompositeTransformation
(constraints):
(0): WordEmbeddingDistance(
(embedding_type): paragramcf
(min_cos_sim): 0.5
(cased): False
(include_unknown_words): True
)
(1): PartOfSpeech(
(tagset): universal
(allow_verb_noun_swap): True
)
(2): UniversalSentenceEncoder(
(metric): angular
(threshold): 0.904458599
(compare_with_original): False
(window_size): 15
(skip_text_shorter_than_window): True
(0): LevenshteinEditDistance(
(max_edit_distance): 30
)
(1): RepeatModification
(2): StopwordModification
(is_black_box): True
)
Load time: /.*/
Load time: /.*/s
--------------------------------------------- Result 1 ---------------------------------------------
0-->[SKIPPED]
A person in a black and green outfit is riding a bicycle .
@@ -41,37 +28,37 @@ A person rolls down a hill riding a wagon as another watches .
A child in a wagon rolls down a hill .
A person rolls down a hill riding a wagon as another watches .
A enfant in a wagon rolls down a hill .
A cihld in a wagon rolls down a hill .
--------------------------------------------- Result 3 ---------------------------------------------
2-->1
A man in a black tank top wearing a red plaid hat
A man in a black tank top wearing a red plaid hat
A man wearing football pads .
A man in a black container major wearing a red plaid hat
A man wearing football pads .
A man in a black tank top wearing a red plaid at
A man wears football pads .
A man wearing football pads .
--------------------------------------------- Result 4 ---------------------------------------------
2-->1
2-->0
Families with strollers waiting in front of a carousel .
Families have some dogs in front of a carousel
Families with pram waiting in front of a carousel .
Families with strlollers waiting in front of a carousel .
Families have some dogs in front of a carousel
--------------------------------------------- Result 5 ---------------------------------------------
1-->0
A person with dark hair in a white shirt is sitting in a chair in water with a `` swim at your own risk '' sign on the wall .
1-->2
A person with dark hair in a white shirt is sitting in a chair in water with a `` swim at your own risk '' sign on the wall .
a person is relaxing on his day off
A person with dark hair in a white shirt is sitting in a chair in water with a `` swum at your own risk '' sign on the wall .
a person is relaxing on his day off
A person with dark hair in a white shirt is sitting in a chair in water with a `` swim at your own risk '' sign on the wall .
a somebody is relaxed on his hoy off
a person is rllaxing on his Hay off
--------------------------------------------- Result 6 ---------------------------------------------
@@ -79,49 +66,49 @@ a somebody is relaxed on his hoy off
pedestrian walking on the street
A person walking inside a building .
pedestrian walking on the sant
pedestrian walking on the stret
A person walking inside a building .
--------------------------------------------- Result 7 ---------------------------------------------
1-->2
A man on a bicycle , wearing cycle gear , riding at a fast past down paved trail surrounded by tree 's and grass .
A man on a bicycle , wearing cycle gear , riding at a fast past down paved trail surrounded by tree 's and grass .
He is heading to the cabin down the trail .
A men on a bicycle , wearing cycle gear , riding at a fast past down emerged track surrounded by tree 's and sod .
He is heading to the cabin down the trail .
A man on a bicycle , wearing cycle gear , riding at a fast past down paved trail surrounded by tree 's and grass .
He is heading to the cabin down the chemin .
He is hezading to the cabin down the Wrail .
--------------------------------------------- Result 8 ---------------------------------------------
0-->1
A man in a bright yellow shirt juggles while riding a unicycle .
A man in a bright yellow shirt juggles while riding a unicycle .
Man performs a juggling act on a unicycle .
A man in a bright yellow shirt juggles while riding a unicycle .
Man performs a juggling act on a unicycle .
A man in a bright yellow shirt jdggles while riding a unicycle .
Man performs a juggling loi on a unicycle .
Man performs a juggling act on a unicycle .
--------------------------------------------- Result 9 ---------------------------------------------
0-->2
Some cars and many tents are set up at the foot of a mountain .
Cars and tents are at the foot of the mountain .
Some autos and many tents are set up at the foot of a mountain .
Cars and tents are at the foot of the mountain .
Some crars and many tents are set up at the foot of a mountain .
Cars and tents are at the foot of the montagne .
Cars and tents are at the foot of the mountain .
--------------------------------------------- Result 10 ---------------------------------------------
0-->1
0-->2
A young child climbing a stack of logs even though the sign warns against it .
A kid climbing some logs
A young child escalade a stack of logs even though the sign warns against it .
A young child Ulimbing a stack of logs even though the sign warns against it .
A kid climbing some register
A kid climbing some lyogs
@@ -134,8 +121,8 @@ A kid climbing some register
(0x(B Original accuracy: (0x(B 90.0% (0x(B
(0x(B Accuracy under attack: (0x(B 0.0% (0x(B
(0x(B Attack success rate: (0x(B 100.0% (0x(B
(0x(B Average perturbed word %: (0x(B 9.77% (0x(B
(0x(B Average perturbed word %: (0x(B 6.29% (0x(B
(0x(B Average num. words per input: (0x(B 21.6 (0x(B
(0x(B Avg num queries: (0x(B 43.33 (0x(B
(0x(B Avg num queries: (0x(B 28.22 (0x(B
(0mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqj(B
Attack time: /.*/
Attack time: /.*/s

13
local_tests/sample_outputs/run_attack_deepwordbug_lstm_mr_10.txt
View File

@@ -1,14 +1,19 @@
GreedyWordSwapWIR(
Attack(
(search_method): GreedyWordSwapWIR(
(wir_method): unk
)
(goal_function): UntargetedClassification
(transformation): CompositeTransformation
(constraints):
(0): LevenshteinEditDistance(
(max_edit_distance): 30
)
(1): RepeatModification
(2): StopwordModification
(is_black_box): True
)
Load time: /.*/
Load time: /.*/s
--------------------------------------------- Result 1 ---------------------------------------------
0-->1
possibly the most irresponsible picture ever released by a major film studio .
@@ -79,6 +84,6 @@ disappointingly , the characters are too strange and dysfunctional , tom include
(0x(B Attack success rate: (0x(B 80.0% (0x(B
(0x(B Average perturbed word %: (0x(B 20.05% (0x(B
(0x(B Average num. words per input: (0x(B 20.7 (0x(B
(0x(B Avg num queries: (0x(B 39.9 (0x(B
(0x(B Avg num queries: (0x(B 38.9 (0x(B
(0mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqj(B
Attack time: /.*/
Attack time: /.*/s

37
local_tests/sample_outputs/run_attack_nonoverlapping_t5ende_editdistance_bleu.txt
View File

@@ -1,15 +1,16 @@
GreedyWordSwap(
Attack(
(search_method): GreedySearch
(goal_function): NonOverlappingOutput
(transformation): WordSwapRandomCharacterSubstitution(
(replace_stopwords): False
)
(transformation): WordSwapRandomCharacterSubstitution
(constraints):
(0): LevenshteinEditDistance(
(max_edit_distance): 12
)
(1): WordsPerturbed(
(1): MaxWordsPerturbed(
(max_percent): 0.75
)
(2): RepeatModification
(3): StopwordModification
(is_black_box): True
)
@@ -20,26 +21,26 @@ A Republican strategy to counter the re-election of Obama
--------------------------------------------- Result 2 ---------------------------------------------
Die republikanischen Führer rechtfertigten ihre Politik durch die Not-->[FAILED]
Republican leaders justified their policy by the need to combat electoral fraud.
Die republikanischen Führer rechtfertigten ihre Politik durch die Not-->Repuzlican leaders justifZed their policy by the need to coq
Republican leaders justified their policy by the need to combat electoral fraud.
Repuzlican leaders justifZed their policy by the need to coqbat electoral fraud.
--------------------------------------------- Result 3 ---------------------------------------------
Das Brennan-Zentrum betrachtet dies jedoch als Mythos und behaupt-->Allerdings hält das Brennan Centre dies für einen Mythos, indem e
However, the Brennan Centre considers this a myth, stating that electoral fraud is rarer in the United States than the number of people killed by lightning.
However, the Brennan Centre cTnsiders this a myth, stating that electoral fraud is rarer in the United States than the number of people killed by lightning.
Das Brennan-Zentrum betrachtet dies jedoch als Mythos und behaupt-->HTwo die Brennan-Zentren halten dies jedoch für einen My
However, the Brennan Centre considers this a myth, stating that electoral fraud is rarer in the United States than the number of people killed by lightning.
HTwever, the Brennan Centre considers this a myth, stating that electoral frauP is rarer in the United States than the number of people killed by lightning.
--------------------------------------------- Result 4 ---------------------------------------------
Tatsächlich identifizierten republikanische Anwälte-->In einer DecOde identifizierten republikanische Anwält
Indeed, Republican lawyers identified only 300 cases of electoral fraud in the United States in a decade.
Indedd, Republican lawyers identified only 300 cases of electoral fraud in the United Ttates in a decOde.
Tatsächlich identifizierten republikanische Anwälte-->In einem Jahrzehnt hat der republikanische Rechtsanwalt
Indeed, Republican lawyers identified only 300 cases of electoral fraud in the United States in a decade.
IndeGd, Republican lawyerf identified only 300 cases of electoral fraud in the United States in a decade.
--------------------------------------------- Result 5 ---------------------------------------------
Eines ist sicher: Diese neuen Bestimmungen werden sich negativ auf die Wahlbeteiligung aus-->Ein Hhing ist sicher: Diese neuen Bestimmungen werden sich negativ auf die Wahlbeteil
One thing is certain: these new provisions will have a negative impact on voter turn-out.
One Hhing is certain: these new provisions will have a negative impact on voter turn-out.
Eines ist sicher: Diese neuen Bestimmungen werden sich negativ auf die Wahlbeteiligung aus-->[FAILED]
One thing is certain: these new provisions will have a negative impact on voter turn-out.
--------------------------------------------- Result 6 ---------------------------------------------
@@ -57,8 +58,8 @@ In this sense, the measures will partially undermine the American democratic sys
(0x(B Original accuracy: (0x(B 100.0% (0x(B
(0x(B Accuracy under attack: (0x(B 50.0% (0x(B
(0x(B Attack success rate: (0x(B 50.0% (0x(B
(0x(B Average perturbed word %: (0x(B 9.62% (0x(B
(0x(B Average perturbed word %: (0x(B 15.06% (0x(B
(0x(B Average num. words per input: (0x(B 15.33 (0x(B
(0x(B Avg num queries: (0x(B 23.67 (0x(B
(0x(B Avg num queries: (0x(B 28.5 (0x(B
(0mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqj(B
Attack time: /.*/s

34
local_tests/sample_outputs/run_attack_targetedclassification2_wordnet_langtool_enable_csv_beamsearch2_attack_n_4.txt
View File

@@ -1,18 +1,21 @@
BeamSearch(
Attack(
(search_method): BeamSearch(
(beam_width): 2
)
(goal_function): TargetedClassification(
(target_class): 2
)
(transformation): WordSwapWordNet(
(replace_stopwords): False
)
(transformation): WordSwapWordNet
(constraints):
(0): LanguageTool(
(grammar_error_threshold): 0
)
(1): RepeatModification
(2): StopwordModification
(is_black_box): True
)
Logging to CSV at path /.*/csv.
Logging to CSV at path /.*/.csv.
Load time: /.*/s
--------------------------------------------- Result 1 ---------------------------------------------
0-->2
@@ -25,10 +28,13 @@ There is a bookshop at the gallery .
--------------------------------------------- Result 2 ---------------------------------------------
0-->[FAILED]
On Naxos , you can walk through the pretty villages of the Tragea Valley and the foothills of Mount Zas , admiring Byzantine churches and exploring olive groves at your leisure .
0-->2
On Naxos , you can walk through the pretty villages of the Tragea Valley and the foothills of Mount Zas , admiring Byzantine churches and exploring olive groves at your leisure .
Naxos is a place with beautiful scenery for leisure .
Naxos is a place with beautiful scenery for leisure .
On Naxos , you can walk through the pretty villages of the Tragea Valley and the foothills of Mount Zas , admiring convoluted churches and exploring olive groves at your leisure .
Naxos is a piazza with beautiful scenery for leisure .
--------------------------------------------- Result 3 ---------------------------------------------
@@ -52,14 +58,14 @@ Net cost for education programs can be calculated as a way to increment
(0lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqwqqqqqqqqk(B
(0x(B Attack Results (0x(B (0x(B
(0tqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqnqqqqqqqqu(B
(0x(B Number of successful attacks: (0x(B 2 (0x(B
(0x(B Number of failed attacks: (0x(B 2 (0x(B
(0x(B Number of successful attacks: (0x(B 3 (0x(B
(0x(B Number of failed attacks: (0x(B 1 (0x(B
(0x(B Number of skipped attacks: (0x(B 0 (0x(B
(0x(B Original accuracy: (0x(B 100.0% (0x(B
(0x(B Accuracy under attack: (0x(B 50.0% (0x(B
(0x(B Attack success rate: (0x(B 50.0% (0x(B
(0x(B Average perturbed word %: (0x(B 2.38% (0x(B
(0x(B Accuracy under attack: (0x(B 25.0% (0x(B
(0x(B Attack success rate: (0x(B 75.0% (0x(B
(0x(B Average perturbed word %: (0x(B 3.34% (0x(B
(0x(B Average num. words per input: (0x(B 34.25 (0x(B
(0x(B Avg num queries: (0x(B 278.5 (0x(B
(0x(B Avg num queries: (0x(B 144.25 (0x(B
(0mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqj(B
Attack time: /.*/s

12
local_tests/sample_outputs/run_attack_textfooler_bert_mr_10.txt
View File

@@ -1,9 +1,11 @@
/.*/GreedyWordSwapWIR(
Attack(
(search_method): GreedyWordSwapWIR(
(wir_method): unk
)
(goal_function): UntargetedClassification
(transformation): WordSwapEmbedding(
(max_candidates): 50
(embedding_type): paragramcf
(replace_stopwords): False
)
(constraints):
(0): WordEmbeddingDistance(
@@ -23,6 +25,8 @@
(window_size): 15
(skip_text_shorter_than_window): True
)
(3): RepeatModification
(4): StopwordModification
(is_black_box): True
)
@@ -96,6 +100,6 @@ disappointingly , the character are too loopy and dysfunctiona
(0x(B Attack success rate: (0x(B 100.0% (0x(B
(0x(B Average perturbed word %: (0x(B 19.15% (0x(B
(0x(B Average num. words per input: (0x(B 20.7 (0x(B
(0x(B Avg num queries: (0x(B 99.86 (0x(B
(0x(B Avg num queries: (0x(B 98.86 (0x(B
(0mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqj(B
Attack time: /.*/s
Attack time: /.*/s

29
local_tests/test_models.py
View File

@@ -2,6 +2,7 @@ import colored
import io
import os
import re
import shlex
import signal
import sys
import subprocess
@@ -87,11 +88,29 @@ class CommandLineTest(TextAttackTest):
def execute(self):
stderr_file = open(stderr_file_name, 'w+')
result = subprocess.run(
self.command.split(),
stdout=subprocess.PIPE,
stderr=stderr_file
)
if isinstance(self.command, tuple):
# Support pipes via tuple of commands
procs = []
for i in range(len(self.command) - 1):
if i == 0:
proc = subprocess.Popen(shlex.split(self.command[i]), stdout=subprocess.PIPE)
else:
proc = subprocess.Popen(shlex.split(self.command[i]), stdout=subprocess.PIPE, stdin=proc.stdout)
procs.append(proc)
# Run last commmand
result = subprocess.run(
shlex.split(self.command[-1]), stdin=procs[-1].stdout,
stdout=subprocess.PIPE, stderr=stderr_file
)
# Wait for all intermittent processes
for proc in procs:
proc.wait()
else:
result = subprocess.run(
shlex.split(self.command),
stdout=subprocess.PIPE,
stderr=stderr_file
)
stderr_file.seek(0) # go back to beginning of file so we can read the whole thing
stderr_str = stderr_file.read()
# Remove temp file.

4
requirements.txt
View File

@@ -5,7 +5,7 @@ lru-dict
nltk
numpy
pandas
pyyaml
pyyaml>=5.1
scikit-learn
scipy
sentence_transformers
@@ -17,4 +17,4 @@ tensorflow_hub
terminaltables
tqdm
visdom
wandb
wandb

2
setup.py
View File

@@ -5,7 +5,7 @@ with open("README.md", "r") as fh:
setuptools.setup(
name="textattack",
version="0.0.1.8",
version="0.0.1.9",
author="QData Lab at the University of Virginia",
author_email="jm8wx@virginia.edu",
description="A library for generating text adversarial examples",

39
textattack/attack_recipes/alzantot_2018.py
View File

@@ -1,22 +1,20 @@
"""
Alzantot, M., Sharma, Y., Elgohary, A., Ho, B., Srivastava, M.B., & Chang,
K. (2018).
Generating Natural Language Adversarial Examples.
EMNLP.
ArXiv, abs/1801.00554.
"""
from textattack.constraints.overlap import WordsPerturbed
from textattack.shared.attack import Attack
from textattack.constraints.overlap import MaxWordsPerturbed
from textattack.constraints.grammaticality.language_models import Google1BillionWordsLanguageModel
from textattack.constraints.semantics import WordEmbeddingDistance
from textattack.constraints.pre_transformation import RepeatModification, StopwordModification
from textattack.goal_functions import UntargetedClassification
from textattack.search_methods import GeneticAlgorithm
from textattack.transformations import WordSwapEmbedding
def Alzantot2018(model):
"""
Alzantot, M., Sharma, Y., Elgohary, A., Ho, B., Srivastava, M.B., & Chang, K. (2018).
Generating Natural Language Adversarial Examples.
https://arxiv.org/abs/1801.00554
"""
#
# Swap words with their embedding nearest-neighbors.
#
@@ -25,12 +23,18 @@ def Alzantot2018(model):
# "[We] fix the hyperparameter values to S = 60, N = 8, K = 4, and δ = 0.5"
#
transformation = WordSwapEmbedding(max_candidates=8)
constraints = []
#
# Don't modify the same word twice or stopwords
#
constraints = [
RepeatModification(),
StopwordModification()
]
#
# Maximum words perturbed percentage of 20%
#
constraints.append(
WordsPerturbed(max_percent=0.2)
MaxWordsPerturbed(max_percent=0.2)
)
#
# Maximum word embedding euclidean distance of 0.5.
@@ -51,7 +55,6 @@ def Alzantot2018(model):
#
# Perform word substitution with a genetic algorithm.
#
attack = GeneticAlgorithm(goal_function, constraints=constraints,
transformation=transformation, pop_size=60, max_iters=20)
return attack
search_method = GeneticAlgorithm(pop_size=60, max_iters=20)
return Attack(goal_function, constraints, transformation, search_method)

40
textattack/attack_recipes/alzantot_2018_adjusted.py
View File

@@ -1,22 +1,23 @@
"""
Alzantot, M., Sharma, Y., Elgohary, A., Ho, B., Srivastava, M.B., & Chang,
K. (2018).
Generating Natural Language Adversarial Examples.
EMNLP.
ArXiv, abs/1801.00554.
"""
from textattack.shared.attack import Attack
from textattack.constraints.grammaticality import PartOfSpeech, LanguageTool
from textattack.constraints.semantics import WordEmbeddingDistance
from textattack.constraints.pre_transformation import RepeatModification, StopwordModification
from textattack.constraints.semantics.sentence_encoders import UniversalSentenceEncoder, BERT
from textattack.goal_functions import UntargetedClassification
from textattack.search_methods import GeneticAlgorithm
from textattack.transformations import WordSwapEmbedding
def Alzantot2018Adjusted(model, SE_thresh=0.98, sentence_encoder='bert'):
"""
Alzantot, M., Sharma, Y., Elgohary, A., Ho, B., Srivastava, M.B., & Chang,
K. (2018).
Generating Natural Language Adversarial Examples.
https://arxiv.org/abs/1801.00554
Constraints adjusted from paper to align with human evaluation.
"""
#
# Swap words with their embedding nearest-neighbors.
#
@@ -24,7 +25,14 @@ def Alzantot2018Adjusted(model, SE_thresh=0.98, sentence_encoder='bert'):
#
# "[We] fix the hyperparameter values to S = 60, N = 8, K = 4, and δ = 0.5"
#
transformation = WordSwapEmbedding(max_candidates=50, textfooler_stopwords=True)
transformation = WordSwapEmbedding(max_candidates=50)
#
# Don't modify the same word twice or stopwords
#
constraints = [
RepeatModification(),
StopwordModification()
]
#
# Minimum word embedding cosine similarity of 0.9.
#
@@ -55,8 +63,8 @@ def Alzantot2018Adjusted(model, SE_thresh=0.98, sentence_encoder='bert'):
#
goal_function = UntargetedClassification(model)
#
# Greedily swap words with "Word Importance Ranking".
# Perform word substitution with a genetic algorithm.
#
attack = GeneticAlgorithm(goal_function, transformation=transformation,
constraints=constraints, pop_size=60, max_iters=20)
return attack
search_method = GeneticAlgorithm(pop_size=60, max_iters=20)
return Attack(goal_function, constraint, transformation, search_method)

36
textattack/attack_recipes/deepwordbug_gao_2018.py
View File

@@ -1,13 +1,5 @@
"""
Gao, Lanchantin, Soffa, Qi.
Black-box Generation of Adversarial Text Sequences to Evade Deep Learning
Classifiers.
ArXiv, abs/1801.04354.
"""
from textattack.shared.attack import Attack
from textattack.constraints.pre_transformation import RepeatModification, StopwordModification
from textattack.constraints.overlap import LevenshteinEditDistance
from textattack.goal_functions import UntargetedClassification
from textattack.search_methods import GreedyWordSwapWIR
@@ -18,6 +10,14 @@ from textattack.transformations import \
WordSwapRandomCharacterSubstitution, WordSwapNeighboringCharacterSwap
def DeepWordBugGao2018(model, use_all_transformations=True):
"""
Gao, Lanchantin, Soffa, Qi.
Black-box Generation of Adversarial Text Sequences to Evade Deep Learning
Classifiers.
https://arxiv.org/abs/1801.04354
"""
#
# Swap characters out from words. Choose the best of four potential transformations.
#
@@ -39,12 +39,19 @@ def DeepWordBugGao2018(model, use_all_transformations=True):
# (ϵ = 30).
transformation = WordSwapRandomCharacterSubstitution()
#
# Don't modify the same word twice or stopwords
#
constraints = [
RepeatModification(),
StopwordModification()
]
#
# In these experiments, we hold the maximum difference
# on edit distance (ϵ) to a constant 30 for each sample.
#
constraints = [
constraints.append(
LevenshteinEditDistance(30)
]
)
#
# Goal is untargeted classification
#
@@ -52,7 +59,6 @@ def DeepWordBugGao2018(model, use_all_transformations=True):
#
# Greedily swap words with "Word Importance Ranking".
#
attack = GreedyWordSwapWIR(goal_function, transformation=transformation,
constraints=constraints, max_depth=None)
search_method = GreedyWordSwapWIR()
return attack
return Attack(goal_function, constraints, transformation, search_method)

48
textattack/attack_recipes/hotflip_ebrahimi_2017.py
View File

@@ -1,36 +1,41 @@
"""
Ebrahimi, J. et al. (2017)
HotFlip: White-Box Adversarial Examples for Text Classification
EMNLP.
ArXiv, abs/1801.00554.
This is a reproduction of the HotFlip word-level attack (section 5 of the
paper).
"""
from textattack.shared.attack import Attack
from textattack.goal_functions import UntargetedClassification
from textattack.constraints.grammaticality import PartOfSpeech
from textattack.constraints.overlap import WordsPerturbed
from textattack.constraints.overlap import MaxWordsPerturbed
from textattack.constraints.semantics import WordEmbeddingDistance
from textattack.constraints.pre_transformation import RepeatModification, StopwordModification
from textattack.search_methods import BeamSearch
from textattack.transformations import GradientBasedWordSwap
from textattack.transformations import WordSwapGradientBased
def HotFlipEbrahimi2017(model):
"""
Ebrahimi, J. et al. (2017)
HotFlip: White-Box Adversarial Examples for Text Classification
https://arxiv.org/abs/1712.06751
This is a reproduction of the HotFlip word-level attack (section 5 of the
paper).
"""
#
# "HotFlip ... uses the gradient with respect to a one-hot input
# representation to efficiently estimate which individual change has the
# highest estimated loss."
transformation = GradientBasedWordSwap(model, top_n=1, replace_stopwords=False)
constraints = []
transformation = WordSwapGradientBased(model, top_n=1)
#
# Don't modify the same word twice or stopwords
#
constraints = [
RepeatModification(),
StopwordModification()
]
#
# 0. "We were able to create only 41 examples (2% of the correctly-
# classified instances of the SST test set) with one or two flips."
#
constraints.append(
WordsPerturbed(max_num_words=2)
MaxWordsPerturbed(max_num_words=2)
)
#
# 1. "The cosine similarity between the embedding of words is bigger than a
@@ -52,7 +57,6 @@ def HotFlipEbrahimi2017(model):
# well together to confuse a classifier ... The adversary uses a beam size
# of 10."
#
attack = BeamSearch(goal_function, constraints=constraints,
transformation=transformation, beam_width=10)
return attack
search_method = BeamSearch(beam_width=10)
return Attack(goal_function, constraints, transformation, search_method)

40
textattack/attack_recipes/kuleshov_2017.py
View File

@@ -1,20 +1,20 @@
"""
Kuleshov, V. et al.
Generating Natural Language Adversarial Examples.
https://openreview.net/pdf?id=r1QZ3zbAZ.
"""
from textattack.constraints.overlap import WordsPerturbed
from textattack.shared.attack import Attack
from textattack.constraints.overlap import MaxWordsPerturbed
from textattack.constraints.grammaticality.language_models import GPT2
from textattack.constraints.semantics.sentence_encoders import ThoughtVector
from textattack.constraints.pre_transformation import RepeatModification, StopwordModification
from textattack.goal_functions import UntargetedClassification
from textattack.search_methods import GreedyWordSwap
from textattack.search_methods import GreedySearch
from textattack.transformations import WordSwapEmbedding
def Kuleshov2017(model):
"""
Kuleshov, V. et al.
Generating Natural Language Adversarial Examples.
https://openreview.net/pdf?id=r1QZ3zbAZ.
"""
#
# "Specifically, in all experiments, we used a target of τ = 0.7,
# a neighborhood size of N = 15, and parameters λ_1 = 0.2 and δ = 0.5; we set
@@ -25,11 +25,17 @@ def Kuleshov2017(model):
#
transformation = WordSwapEmbedding(max_candidates=15)
#
# Don't modify the same word twice or stopwords
#
constraints = [
RepeatModification(),
StopwordModification()
]
#
# Maximum of 50% of words perturbed (δ in the paper).
#
constraints = []
constraints.append(
WordsPerturbed(max_percent=0.5)
MaxWordsPerturbed(max_percent=0.5)
)
#
# Maximum thought vector Euclidean distance of λ_1 = 0.2. (eq. 4)
@@ -52,10 +58,6 @@ def Kuleshov2017(model):
#
# Perform word substitution with a genetic algorithm.
#
attack = GreedyWordSwap(goal_function, constraints=constraints,
transformation=transformation)
search_method = GreedySearch()
return attack
# GPT2(max_log_prob_diff=2)
return Attack(goal_function, constraints, transformation, search_method)

44
textattack/attack_recipes/seq2sick_cheng_2018_blackbox.py
View File

@@ -1,23 +1,24 @@
"""
Cheng, Minhao, et al.
Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with
Adversarial Examples
ArXiv, abs/1803.01128.
This is a greedy re-implementation of the seq2sick attack method. It does
not use gradient descent.
"""
from textattack.shared.attack import Attack
from textattack.constraints.overlap import LevenshteinEditDistance
from textattack.constraints.semantics import WordEmbeddingDistance
from textattack.constraints.pre_transformation import RepeatModification, StopwordModification
from textattack.goal_functions import NonOverlappingOutput
from textattack.search_methods import GreedyWordSwapWIR
from textattack.transformations import WordSwapEmbedding
def Seq2SickCheng2018BlackBox(model, goal_function='non_overlapping'):
"""
Cheng, Minhao, et al.
Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with
Adversarial Examples
https://arxiv.org/abs/1803.01128
This is a greedy re-implementation of the seq2sick attack method. It does
not use gradient descent.
"""
#
# Goal is non-overlapping output.
#
@@ -26,13 +27,22 @@ def Seq2SickCheng2018BlackBox(model, goal_function='non_overlapping'):
# seq2sick.
transformation = WordSwapEmbedding(max_candidates=50)
#
# Don't modify the same word twice or stopwords
#
constraints = [
RepeatModification(),
StopwordModification()
]
#
# In these experiments, we hold the maximum difference
# on edit distance (ϵ) to a constant 30 for each sample.
#
constraints.append(
LevenshteinEditDistance(30)
)
#
# Greedily swap words with "Word Importance Ranking".
#
attack = GreedyWordSwapWIR(goal_function, transformation=transformation,
constraints=[], max_depth=10)
search_method = GreedyWordSwapWIR()
return attack
return Attack(goal_function, constraints, transformation, search_method)

42
textattack/attack_recipes/textfooler_jin_2019.py
View File

@@ -1,21 +1,20 @@
"""
Jin, D., Jin, Z., Zhou, J.T., & Szolovits, P. (2019).
Is BERT Really Robust? Natural Language Attack on Text Classification and
Entailment.
ArXiv, abs/1907.11932.
"""
from textattack.shared.attack import Attack
from textattack.goal_functions import UntargetedClassification
from textattack.constraints.grammaticality import PartOfSpeech
from textattack.constraints.semantics import WordEmbeddingDistance
from textattack.constraints.pre_transformation import RepeatModification, StopwordModification
from textattack.constraints.semantics.sentence_encoders import UniversalSentenceEncoder
from textattack.search_methods import GreedyWordSwapWIR
from textattack.transformations import WordSwapEmbedding
def TextFoolerJin2019(model):
"""
Jin, D., Jin, Z., Zhou, J.T., & Szolovits, P. (2019).
Is BERT Really Robust? Natural Language Attack on Text Classification and Entailment.
https://arxiv.org/abs/1907.11932
"""
#
# Swap words with their embedding nearest-neighbors.
#
@@ -25,19 +24,27 @@ def TextFoolerJin2019(model):
# (The paper claims 0.7, but analysis of the code and some empirical
# results show that it's definitely 0.5.)
#
transformation = WordSwapEmbedding(max_candidates=50, textfooler_stopwords=True)
transformation = WordSwapEmbedding(max_candidates=50)
#
# Don't modify the same word twice or the stopwords defined
# in the TextFooler public implementation.
#
stopwords = set(['a', 'about', 'above', 'across', 'after', 'afterwards', 'again', 'against', 'ain', 'all', 'almost', 'alone', 'along', 'already', 'also', 'although', 'am', 'among', 'amongst', 'an', 'and', 'another', 'any', 'anyhow', 'anyone', 'anything', 'anyway', 'anywhere', 'are', 'aren', "aren't", 'around', 'as', 'at', 'back', 'been', 'before', 'beforehand', 'behind', 'being', 'below', 'beside', 'besides', 'between', 'beyond', 'both', 'but', 'by', 'can', 'cannot', 'could', 'couldn', "couldn't", 'd', 'didn', "didn't", 'doesn', "doesn't", 'don', "don't", 'down', 'due', 'during', 'either', 'else', 'elsewhere', 'empty', 'enough', 'even', 'ever', 'everyone', 'everything', 'everywhere', 'except', 'first', 'for', 'former', 'formerly', 'from', 'hadn', "hadn't", 'hasn', "hasn't", 'haven', "haven't", 'he', 'hence', 'her', 'here', 'hereafter', 'hereby', 'herein', 'hereupon', 'hers', 'herself', 'him', 'himself', 'his', 'how', 'however', 'hundred', 'i', 'if', 'in', 'indeed', 'into', 'is', 'isn', "isn't", 'it', "it's", 'its', 'itself', 'just', 'latter', 'latterly', 'least', 'll', 'may', 'me', 'meanwhile', 'mightn', "mightn't", 'mine', 'more', 'moreover', 'most', 'mostly', 'must', 'mustn', "mustn't", 'my', 'myself', 'namely', 'needn', "needn't", 'neither', 'never', 'nevertheless', 'next', 'no', 'nobody', 'none', 'noone', 'nor', 'not', 'nothing', 'now', 'nowhere', 'o', 'of', 'off', 'on', 'once', 'one', 'only', 'onto', 'or', 'other', 'others', 'otherwise', 'our', 'ours', 'ourselves', 'out', 'over', 'per', 'please','s', 'same', 'shan', "shan't", 'she', "she's", "should've", 'shouldn', "shouldn't", 'somehow', 'something', 'sometime', 'somewhere', 'such', 't', 'than', 'that', "that'll", 'the', 'their', 'theirs', 'them', 'themselves', 'then', 'thence', 'there', 'thereafter', 'thereby', 'therefore', 'therein', 'thereupon', 'these', 'they','this', 'those', 'through', 'throughout', 'thru', 'thus', 'to', 'too','toward', 'towards', 'under', 'unless', 'until', 'up', 'upon', 'used', 've', 'was', 'wasn', "wasn't", 'we', 'were', 'weren', "weren't", 'what', 'whatever', 'when', 'whence', 'whenever', 'where', 'whereafter', 'whereas', 'whereby', 'wherein', 'whereupon', 'wherever', 'whether', 'which', 'while', 'whither', 'who', 'whoever', 'whole', 'whom', 'whose', 'why', 'with', 'within', 'without', 'won', "won't", 'would', 'wouldn', "wouldn't", 'y', 'yet', 'you', "you'd", "you'll", "you're", "you've", 'your', 'yours', 'yourself', 'yourselves'])
constraints = [
RepeatModification(),
StopwordModification(stopwords=stopwords)
]
#
# Minimum word embedding cosine similarity of 0.5.
#
constraints = []
constraints.append(
WordEmbeddingDistance(min_cos_sim=0.5)
WordEmbeddingDistance(min_cos_sim=0.5)
)
#
# Only replace words with the same part of speech (or nouns with verbs)
#
constraints.append(
PartOfSpeech(allow_verb_noun_swap=True)
PartOfSpeech(allow_verb_noun_swap=True)
)
#
# Universal Sentence Encoder with a minimum angular similarity of ε = 0.7.
@@ -57,7 +64,6 @@ def TextFoolerJin2019(model):
#
# Greedily swap words with "Word Importance Ranking".
#
attack = GreedyWordSwapWIR(goal_function, transformation=transformation,
constraints=constraints, max_depth=None)
return attack
search_method = GreedyWordSwapWIR()
return Attack(goal_function, constraints, transformation, search_method)

40
textattack/attack_recipes/textfooler_jin_2019_adjusted.py
View File

@@ -1,14 +1,6 @@
"""
Jin, D., Jin, Z., Zhou, J.T., & Szolovits, P. (2019).
Is BERT Really Robust? Natural Language Attack on Text Classification and
Entailment.
ArXiv, abs/1907.11932.
"""
from textattack.shared.attack import Attack
from textattack.constraints.semantics import WordEmbeddingDistance
from textattack.constraints.pre_transformation import RepeatModification, StopwordModification
from textattack.constraints.semantics.sentence_encoders import UniversalSentenceEncoder, BERT
from textattack.constraints.grammaticality import PartOfSpeech, LanguageTool
from textattack.goal_functions import UntargetedClassification
@@ -16,6 +8,15 @@ from textattack.search_methods import GreedyWordSwapWIR
from textattack.transformations import WordSwapEmbedding
def TextFoolerJin2019Adjusted(model, SE_thresh=0.98, sentence_encoder='bert'):
"""
Jin, D., Jin, Z., Zhou, J.T., & Szolovits, P. (2019).
Is BERT Really Robust? Natural Language Attack on Text Classification and Entailment.
https://arxiv.org/abs/1907.11932
Constraints adjusted from paper to align with human evaluation.
"""
#
# Swap words with their embedding nearest-neighbors.
#
@@ -25,13 +26,19 @@ def TextFoolerJin2019Adjusted(model, SE_thresh=0.98, sentence_encoder='bert'):
# (The paper claims 0.7, but analysis of the code and some empirical
# results show that it's definitely 0.5.)
#
transformation = WordSwapEmbedding(max_candidates=50, textfooler_stopwords=True)
transformation = WordSwapEmbedding(max_candidates=50)
#
# Don't modify the same word twice or stopwords
#
constraints = [
RepeatModification(),
StopwordModification()
]
#
# Minimum word embedding cosine similarity of 0.9.
#
constraints = []
constraints.append(
WordEmbeddingDistance(min_cos_sim=0.9)
WordEmbeddingDistance(min_cos_sim=0.9)
)
#
# Universal Sentence Encoder with a minimum angular similarity of ε = 0.7.
@@ -49,7 +56,7 @@ def TextFoolerJin2019Adjusted(model, SE_thresh=0.98, sentence_encoder='bert'):
# Do grammar checking
#
constraints.append(
LanguageTool(0)
LanguageTool(0)
)
#
@@ -60,7 +67,6 @@ def TextFoolerJin2019Adjusted(model, SE_thresh=0.98, sentence_encoder='bert'):
#
# Greedily swap words with "Word Importance Ranking".
#
attack = GreedyWordSwapWIR(goal_function, transformation=transformation,
constraints=constraints, max_depth=None)
search_method = GreedyWordSwapWIR()
return attack
return Attack(goal_function, constraints, transformation, search_method)

3
textattack/attack_results/attack_result.py
View File

@@ -52,6 +52,9 @@ class AttackResult:
return '\n'.join(self.str_lines(color_method=color_method))
def goal_function_result_str(self, color_method=None):
"""
Returns a string illustrating the results of the goal function.
"""
orig_colored = self.original_result.get_colored_output(color_method) # @TODO add this method to goal function results
# @TODO also display confidence
pert_colored = self.perturbed_result.get_colored_output(color_method)

2
textattack/attack_results/failed_attack_result.py
View File

@@ -2,6 +2,8 @@ from .attack_result import AttackResult
from textattack.shared import utils
class FailedAttackResult(AttackResult):
'''The result of a failed attack.'''
def __init__(self, original_result, perturbed_result=None):
perturbed_result = perturbed_result or original_result
super().__init__(original_result, perturbed_result)

2
textattack/attack_results/skipped_attack_result.py
View File

@@ -2,6 +2,8 @@ from .attack_result import AttackResult
from textattack.shared import utils
class SkippedAttackResult(AttackResult):
'''The result of a skipped attack.'''
def __init__(self, original_result):
super().__init__(original_result, original_result)

87
textattack/augmentation/augmenter.py
View File

@@ -1,28 +1,36 @@
import random
import tqdm
from textattack.constraints.pre_transformation import PreTransformationConstraint
from textattack.shared.tokenized_text import TokenizedText
class Augmenter:
""" A class for performing data augmentation using TextAttack.
"""
A class for performing data augmentation using TextAttack.
Returns all possible transformations for a given string.
Args:
transformation (textattack.Transformation): the transformation
that suggests new texts from an input.
constraints: (list(textattack.Constraint)): constraints
that each transformation must meet
words_to_swap (int): number of words to swap in each augmented
example
transformations_per_example (int): number of transformations
to return for each training example
Returns all possible transformations for a given string.
Args:
transformation (textattack.Transformation): the transformation
that suggests new texts from an input.
constraints: (list(textattack.Constraint)): constraints
that each transformation must meet
words_to_swap: (int): Number of words to swap per augmented example
transformations_per_example: (int): Number of words to swap per augmented example
"""
def __init__(self, transformation, constraints=[], words_to_swap=1,
transformations_per_example=1):
self.transformation = transformation
self.constraints = constraints
self.words_to_swap = words_to_swap
self.transformations_per_example = transformations_per_example
self.constraints = []
self.pre_transformation_constraints = []
for constraint in constraints:
if isinstance(constraint, PreTransformationConstraint):
self.pre_transformation_constraints.append(constraint)
else:
self.constraints.append(constraint)
def _filter_transformations(self, tokenized_text, transformations):
""" Filters a list of `TokenizedText` objects to include only the ones
@@ -40,21 +48,24 @@ class Augmenter:
tokenized_text = TokenizedText(text, DummyTokenizer())
all_transformations = set()
for _ in range(self.transformations_per_example):
indices_to_replace = list(range(len(tokenized_text.words)))
indices_to_modify = set(range(len(tokenized_text.words)))
next_tokenized_text = tokenized_text
for __ in range(self.words_to_swap):
transformations = []
for constraint in self.pre_transformation_constraints:
indices_to_modify = list(set(indices_to_modify) & constraint(tokenized_text, self))
while not len(transformations):
if not len(indices_to_replace):
# Loop until we find a valid transformation.
if not len(indices_to_modify):
# This occurs when we couldn't find valid transformations
# – either the constraints were too strict, or the number
# of words to swap was too high, or we were just plain
# unlucky. In any event, don't throw an error, and just
# don't return anything.
break
replacement_index = random.choice(indices_to_replace)
indices_to_replace.remove(replacement_index)
transformations = self.transformation(next_tokenized_text, indices_to_replace=[replacement_index])
replacement_index = random.choice(indices_to_modify)
indices_to_modify.remove(replacement_index)
transformations = self.transformation(next_tokenized_text, indices_to_modify=[replacement_index])
# Get rid of transformations we already have
transformations = [t for t in transformations if t not in all_transformations]
# Filter out transformations that don't match the constraints.
@@ -65,15 +76,43 @@ class Augmenter:
return [t.clean_text() for t in all_transformations]
def augment_many(self, text_list):
""" Returns all possible augmentations of a list of strings according to
`self.transformation`.
"""
Returns all possible augmentations of a list of strings according to
`self.transformation`.
Args:
text_list (list(string)): a list of strings for data augmentation
Returns a list(string) of augmented texts.
"""
return [self.augment(text) for text in text_list]
def augment_text_and_ids(self, text_list, id_list, show_progress=True):
""" Supplements a list of text with more text data. Returns the augmented
text along with the corresponding IDs for each augmented example.
"""
if len(text_list) != len(id_list):
raise ValueError('List of text must be same length as list of IDs')
if self.transformations_per_example == 0:
return text_list, id_list
all_text_list = []
all_id_list = []
if show_progress:
text_list = tqdm.tqdm(text_list, desc='Augmenting data...')
for text, ids in zip(text_list, id_list):
all_text_list.append(text)
all_id_list.append(ids)
augmented_texts = self.augment(text)
all_text_list.extend
all_text_list.extend([text] + augmented_texts)
all_id_list.extend([ids] * (1 + len(augmented_texts)))
return all_text_list, all_id_list
class DummyTokenizer:
""" A dummy tokenizer class. Data augmentation applies a transformation
without querying a model, which means that tokenization is unnecessary.
In this case, we pass a dummy tokenizer to `TokenizedText`.
"""
A dummy tokenizer class. Data augmentation applies a transformation
without querying a model, which means that tokenization is unnecessary.
In this case, we pass a dummy tokenizer to `TokenizedText`.
"""
def encode(self, _):
return []
return []

11
textattack/augmentation/recipes.py
View File

@@ -2,12 +2,17 @@ from . import Augmenter
import textattack
DEFAULT_CONSTRAINTS = [
textattack.constraints.pre_transformation.RepeatModification(),
textattack.constraints.pre_transformation.StopwordModification()
]
class WordNetAugmenter(Augmenter):
""" Augments text by replacing with synonyms from the WordNet thesaurus. """
def __init__(self, **kwargs):
from textattack.transformations import WordSwapWordNet
transformation = WordSwapWordNet()
super().__init__(transformation, constraints=[], **kwargs)
super().__init__(transformation, constraints=DEFAULT_CONSTRAINTS, **kwargs)
class EmbeddingAugmenter(Augmenter):
@@ -18,7 +23,7 @@ class EmbeddingAugmenter(Augmenter):
max_candidates=50, embedding_type='paragramcf'
)
from textattack.constraints.semantics import WordEmbeddingDistance
constraints = [
constraints = DEFAULT_CONSTRAINTS + [
WordEmbeddingDistance(min_cos_sim=0.8)
]
super().__init__(transformation, constraints=constraints, **kwargs)
@@ -42,4 +47,4 @@ class CharSwapAugmenter(Augmenter):
# (4) Insertion: Insert a random letter in the word.
WordSwapRandomCharacterInsertion()
])
super().__init__(transformation, constraints=[], **kwargs)
super().__init__(transformation, constraints=DEFAULT_CONSTRAINTS, **kwargs)

2
textattack/constraints/__init__.py
View File

@@ -2,3 +2,5 @@ from .constraint import Constraint
from . import grammaticality
from . import semantics
from . import overlap
from . import pre_transformation

90
textattack/constraints/constraint.py
View File

@@ -1,6 +1,3 @@
""" Abstract classes represent constraints on text adversarial examples.
"""
from textattack.shared.utils import default_class_repr
class Constraint:
@@ -9,24 +6,91 @@ class Constraint:
A constraint evaluates if (x,x_adv) meets a certain constraint.
"""
def call_many(self, x, x_adv_list, original_text=None, **kwargs):
def call_many(self, x, x_adv_list, original_text=None):
"""
Filters x_adv_list to x_adv where C(x,x_adv) is true.
Filters ``x_adv_list`` to ``x_adv`` where ``x_adv`` fulfills the constraint.
First checks compatibility with latest ``Transformation``, then calls
``_check_constraint_many``\.
Args:
x:
x_adv_list:
original_text(:obj:`type`, optional): Defaults to None.
x: The current ``TokenizedText``.
x_adv_list: The potential altered ``TokenizedText``\s.
original_text: The original ``TokenizedText`` from which the attack began.
"""
incompatible_x_advs = []
compatible_x_advs = []
for x_adv in x_adv_list:
try:
if self.check_compatibility(x_adv.attack_attrs['last_transformation']):
compatible_x_advs.append(x_adv)
else:
incompatible_x_advs.append(x_adv)
except KeyError:
raise KeyError('x_adv must have `last_transformation` attack_attr to apply constraint')
filtered_x_advs = self._check_constraint_many(x, compatible_x_advs, original_text=original_text)
return list(filtered_x_advs) + incompatible_x_advs
def _check_constraint_many(self, x, x_adv_list, original_text=None):
"""
Filters ``x_adv_list`` to ``x_adv`` where ``x_adv`` fulfills the constraint.
Calls ``check_constraint``\.
Args:
x: The current ``TokenizedText``.
x_adv_list: The potential altered ``TokenizedText``\s.
original_text: The original ``TokenizedText`` from which the attack began.
"""
return [x_adv for x_adv in x_adv_list
if self.__call__(x, x_adv, original_text=original_text)]
if self._check_constraint(x, x_adv, original_text=original_text)]
def __call__(self, x, x_adv, original_text=None):
""" Returns True if C(x,x_adv) is true. """
"""
Returns True if the constraint is fulfilled, False otherwise. First checks
compatibility with latest ``Transformation``, then calls ``_check_constraint``\.
Args:
x: The current ``TokenizedText``.
x_adv: The potential altered ``TokenizedText``.
original_text: The original ``TokenizedText`` from which the attack began.
"""
if not isinstance(x, TokenizedText):
raise TypeError('x must be of type TokenizedText')
if not isinstance(x_adv, TokenizedText):
raise TypeError('x_adv must be of type TokenizedText')
try:
if not self.check_compatibility(x_adv.attack_attrs['last_transformation']):
return True
except KeyError:
raise KeyError('x_adv must have `last_transformation` attack_attr to apply constraint.')
return self._check_constraint(x_adv, original_text=original_text)
def _check_constraint(self, x, x_adv, original_text=None):
"""
Returns True if the constraint is fulfilled, False otherwise. Must be implemented
by the specific constraint.
Args:
x: The current ``TokenizedText``.
x_adv: The potential altered ``TokenizedText``.
original_text: The original ``TokenizedText`` from which the attack began.
"""
raise NotImplementedError()
def check_compatibility(self, transformation):
"""
Checks if this constraint is compatible with the given transformation.
For example, the ``WordEmbeddingDistance`` constraint compares the embedding of
the word inserted with that of the word deleted. Therefore it can only be
applied in the case of word swaps, and not for transformations which involve
only one of insertion or deletion.
Args:
transformation: The ``Transformation`` to check compatibility with.
"""
return True
def extra_repr_keys(self):
"""Set the extra representation of the constraint using these keys.

19
textattack/constraints/grammaticality/language_models/google_language_model/google_language_model.py
View File

@@ -3,6 +3,7 @@ import time
from collections import defaultdict
from textattack.transformations import WordSwap
from textattack.constraints import Constraint
from .alzantot_goog_lm import GoogLMHelper
@@ -21,12 +22,6 @@ class GoogleLanguageModel(Constraint):
Raises:
ValueError: If :obj:`top_n` or :obj:`top_n_per_index` are not provided.
@TODO allow user to set perplexity threshold; implement __call__.
@TODO this use of the language model only really makes sense for
adversarial examples based on word swaps
"""
def __init__(self, top_n=None, top_n_per_index=None, print_step=False):
if not (top_n or top_n_per_index):
@@ -35,18 +30,22 @@ class GoogleLanguageModel(Constraint):
self.top_n = top_n
self.top_n_per_index = top_n_per_index
self.print_step = print_step
def call_many(self, x, x_adv_list, original_text=None):
def check_compatibility(self, transformation):
return isinstance(transformation, WordSwap)
def _check_constraint_many(self, x, x_adv_list, original_text=None):
"""
Returns the `top_n` of x_adv_list, as evaluated by the language
model.
Args:
x:
X_adv_list:
x_adv_list:
original_text (:obj:`type`, optional): Defaults to None.
"""
# @TODO Allow user to implement perplexity threshold
if not len(x_adv_list): return []
def get_probs(x, x_adv_list):
@@ -96,7 +95,7 @@ class GoogleLanguageModel(Constraint):
# same order they were passed in.
max_el_indices.sort()
return np.array(x_adv_list)[max_el_indices]
return [x_adv_list[i] for i in max_el_indices]
def __call__(self, x, x_adv):
raise NotImplementedError()

38
textattack/constraints/grammaticality/language_models/language_model_constraint.py
View File

@@ -5,12 +5,12 @@ from textattack.constraints import Constraint
class LanguageModelConstraint(Constraint):
"""
Determines if two sentences have a swapped word that has a similar
probability according to a language model.
Determines if two sentences have a swapped word that has a similar
probability according to a language model.
Args:
max_log_prob_diff (float): the maximum difference in log-probability
between x and x_adv
Args:
max_log_prob_diff (float): the maximum difference in log-probability
between x and x_adv
"""
def __init__(self, max_log_prob_diff=None):
@@ -24,19 +24,23 @@ class LanguageModelConstraint(Constraint):
"""
raise NotImplementedError()
def __call__(self, x, x_adv, original_text=None):
def _check_constraint(self, x, x_adv, original_text=None):
try:
i = x_adv.attack_attrs['modified_word_index']
except AttributeError:
raise AttributeError('Cannot apply language model constraint without `modified_word_index`')
probs = self.get_log_probs_at_index((x, x_adv), i)
if len(probs) != 2:
raise ValueError(f'Error: get_log_probs_at_index returned {len(probs)} values for 2 inputs')
x_prob, x_adv_prob = probs
if self.max_log_prob_diff is None:
x_prob, x_adv_prob = math.log(p1), math.log(p2)
return abs(x_prob - x_adv_prob) <= self.max_log_prob_diff
indices = x_adv.attack_attrs['newly_modified_indices']
except KeyError:
raise KeyError('Cannot apply language model constraint without `newly_modified_indices`')
for i in indices:
probs = self.get_log_probs_at_index((x, x_adv), i)
if len(probs) != 2:
raise ValueError(f'Error: get_log_probs_at_index returned {len(probs)} values for 2 inputs')
x_prob, x_adv_prob = probs
if self.max_log_prob_diff is None:
x_prob, x_adv_prob = math.log(p1), math.log(p2)
if abs(x_prob - x_adv_prob) > self.max_log_prob_diff:
return False
return True
def extra_repr_keys(self):
return ['max_log_prob_diff']

2
textattack/constraints/grammaticality/language_tool.py
View File

@@ -27,7 +27,7 @@ class LanguageTool(Constraint):
else:
return len(self.lang_tool.check(text))
def __call__(self, x, x_adv, original_text=None):
def _check_constraint(self, x, x_adv, original_text=None):
original_num_errors = self.get_errors(original_text, use_cache=True)
errors_added = self.get_errors(x_adv) - original_num_errors
return errors_added <= self.grammar_error_threshold

32
textattack/constraints/grammaticality/part_of_speech.py
View File

@@ -3,6 +3,7 @@ import nltk
from textattack.constraints import Constraint
from textattack.shared import TokenizedText
from textattack.shared.validators import transformation_consists_of_word_swaps
class PartOfSpeech(Constraint):
""" Constraints word swaps to only swap words with the same part of speech.
@@ -14,7 +15,7 @@ class PartOfSpeech(Constraint):
self.tagset = tagset
self.allow_verb_noun_swap = allow_verb_noun_swap
self._pos_tag_cache = lru.LRU(2**14)
def _can_replace_pos(self, pos_a, pos_b):
return (pos_a == pos_b) or (self.allow_verb_noun_swap and set([pos_a,pos_b]) <= set(['NOUN','VERB']))
@@ -28,24 +29,31 @@ class PartOfSpeech(Constraint):
self._pos_tag_cache[context_key] = pos_list
return pos_list
def __call__(self, x, x_adv, original_text=None):
def _check_constraint(self, x, x_adv, original_text=None):
if not isinstance(x, TokenizedText):
raise TypeError('x must be of type TokenizedText')
if not isinstance(x_adv, TokenizedText):
raise TypeError('x_adv must be of type TokenizedText')
try:
i = x_adv.attack_attrs['modified_word_index']
indices = x_adv.attack_attrs['newly_modified_indices']
except KeyError:
raise KeyError('Cannot apply part-of-speech constraint without `newly_modified_indices`')
for i in indices:
x_word = x.words[i]
x_adv_word = x_adv.words[i]
except AttributeError:
raise AttributeError('Cannot apply part-of-speech constraint without `modified_word_index`')
before_ctx = x.words[max(i-4,0):i]
after_ctx = x.words[i+1:min(i+5,len(x.words))]
cur_pos = self._get_pos(before_ctx, x_word, after_ctx)
replace_pos = self._get_pos(before_ctx, x_adv_word, after_ctx)
return self._can_replace_pos(cur_pos, replace_pos)
before_ctx = x.words[max(i-4,0):i]
after_ctx = x.words[i+1:min(i+5,len(x.words))]
cur_pos = self._get_pos(before_ctx, x_word, after_ctx)
replace_pos = self._get_pos(before_ctx, x_adv_word, after_ctx)
if not self._can_replace_pos(cur_pos, replace_pos):
return False
return True
def check_compatibility(self, transformation):
return transformation_consists_of_word_swaps(transformation)
def extra_repr_keys(self):
return ['tagset', 'allow_verb_noun_swap']

2
textattack/constraints/overlap/__init__.py
View File

@@ -2,4 +2,4 @@ from .bleu_score import BLEU
from .chrf_score import chrF
from .levenshtein_edit_distance import LevenshteinEditDistance
from .meteor_score import METEOR
from .words_perturbed import WordsPerturbed
from .max_words_perturbed import MaxWordsPerturbed

2
textattack/constraints/overlap/bleu_score.py
View File

@@ -11,7 +11,7 @@ class BLEU(Constraint):
self.max_bleu_score = max_bleu_score
def __call__(self, x, x_adv, original_text=None):
def _check_constraint(self, x, x_adv, original_text=None):
if not original_text:
return True
ref = original_text.words

2
textattack/constraints/overlap/chrf_score.py
View File

@@ -9,7 +9,7 @@ class chrF(Constraint):
raise TypeError('max_chrf must be an int')
self.max_chrf = max_chrf
def __call__(self, x, x_adv, original_text=None):
def _check_constraint(self, x, x_adv, original_text=None):
if not original_text:
return True
ref = original_text.words

2
textattack/constraints/overlap/levenshtein_edit_distance.py
View File

@@ -12,7 +12,7 @@ class LevenshteinEditDistance(Constraint):
self.max_edit_distance = max_edit_distance
def __call__(self, x, x_adv, original_text=None):
def _check_constraint(self, x, x_adv, original_text=None):
if not original_text:
return True
edit_distance = editdistance.eval(original_text.text, x_adv.text)

7
textattack/constraints/overlap/words_perturbed.py → textattack/constraints/overlap/max_words_perturbed.py
View File

@@ -1,7 +1,7 @@
import math
from textattack.constraints import Constraint
class WordsPerturbed(Constraint):
class MaxWordsPerturbed(Constraint):
""" A constraint representing a maximum allowed perturbed words. """
def __init__(self, max_num_words=None, max_percent=None):
@@ -12,9 +12,9 @@ class WordsPerturbed(Constraint):
self.max_num_words = max_num_words
self.max_percent = max_percent
def __call__(self, x, x_adv, original_text=None):
def _check_constraint(self, x, x_adv, original_text=None):
if not original_text:
raise ValueError('Cannot constraint WordsPerturbed without original_text')
raise ValueError('Cannot apply constraint MaxWordsPerturbed without original_text')
num_words_diff = len(x_adv.all_words_diff(original_text))
if self.max_percent:
@@ -38,4 +38,3 @@ class WordsPerturbed(Constraint):
metric.append('max_num_words')
return metric

2
textattack/constraints/overlap/meteor_score.py
View File

@@ -11,7 +11,7 @@ class METEOR(Constraint):
self.max_meteor = max_meteor
def __call__(self, x, x_adv, original_text=None):
def _check_constraint(self, x, x_adv, original_text=None):
if not original_text:
return True
meteor = nltk.translate.meteor([original_text], x_adv)

3
textattack/constraints/pre_transformation/__init__.py Normal file
View File

@@ -0,0 +1,3 @@
from .pre_transformation_constraint import PreTransformationConstraint
from .stopword_modification import StopwordModification
from .repeat_modification import RepeatModification

33
textattack/constraints/pre_transformation/pre_transformation_constraint.py Normal file
View File

@@ -0,0 +1,33 @@
from textattack.shared.utils import default_class_repr
from textattack.constraints import Constraint
class PreTransformationConstraint(Constraint):
"""
An abstract class that represents constraints which are applied before
the transformation. These restrict which words are allowed to be modified
during the transformation. For example, we might not allow stopwords to be
modified.
"""
def __call__(self, x, transformation):
"""
Returns the word indices in ``x`` which are able to be modified. First checks compatibility
with ``transformation`` then calls ``_get_modifiable_indices``\.
Args:
x: The ``TokenizedText`` input to consider.
transformation: The ``Transformation`` which will be applied.
"""
if not self.check_compatibility(transformation):
return set(range(len(x.words)))
return self._get_modifiable_indices(x)
def _get_modifiable_indices(x):
"""
Returns the word indices in x which are able to be modified. Must be overridden by
specific pre-transformation constraints.
Args:
x: The ``TokenizedText`` input to consider.
"""
raise NotImplementedError()

15
textattack/constraints/pre_transformation/repeat_modification.py Normal file
View File

@@ -0,0 +1,15 @@
from textattack.shared.utils import default_class_repr
from textattack.constraints.pre_transformation import PreTransformationConstraint
class RepeatModification(PreTransformationConstraint):
"""
A constraint disallowing the modification of words which have already been modified.
"""
def _get_modifiable_indices(self, tokenized_text):
""" Returns the word indices in x which are able to be deleted """
try:
return set(range(len(tokenized_text.words))) - tokenized_text.attack_attrs['modified_indices']
except KeyError:
raise KeyError('`modified_indices` in attack_attrs required for RepeatModification constraint.')

33
textattack/constraints/pre_transformation/stopword_modification.py Normal file
View File

@@ -0,0 +1,33 @@
from textattack.shared.utils import default_class_repr
from textattack.constraints.pre_transformation import PreTransformationConstraint
from textattack.shared.validators import transformation_consists_of_word_swaps
import nltk
class StopwordModification(PreTransformationConstraint):
"""
A constraint disallowing the modification of stopwords
"""
def __init__(self, stopwords=None):
if stopwords is not None:
self.stopwords = set(stopwords)
else:
self.stopwords = set(nltk.corpus.stopwords.words('english'))
def _get_modifiable_indices(self, tokenized_text):
""" Returns the word indices in x which are able to be deleted """
non_stopword_indices = set()
for i, word in enumerate(tokenized_text.words):
if word not in self.stopwords:
non_stopword_indices.add(i)
return non_stopword_indices
def check_compatibility(self, transformation):
"""
The stopword constraint only is concerned with word swaps since paraphrasing phrases
containing stopwords is OK.
Args:
transformation: The ``Transformation`` to check compatibility with.
"""
return transformation_consists_of_word_swaps(transformation)

2
textattack/constraints/semantics/__init__.py
View File

@@ -1,3 +1,3 @@
from . import sentence_encoders
from .word_embedding_distance import WordEmbeddingDistance
from .word_embedding_distance import WordEmbeddingDistance

12
textattack/constraints/semantics/sentence_encoders/sentence_encoder.py
View File

@@ -93,7 +93,11 @@ class SentenceEncoder(Constraint):
x_list_text = []
x_adv_list_text = []
for x_adv in x_adv_list:
modified_index = x_adv.attack_attrs['modified_word_index']
#@TODO make this work when multiple indices have been modified
try:
modified_index = next(iter(x_adv.attack_attrs['newly_modified_indices']))
except KeyError:
raise KeyError('Cannot apply sentence encoder constraint without `newly_modified_indices`')
x_list_text.append(x.text_window_around_index(modified_index, self.window_size))
x_adv_list_text.append(x_adv.text_window_around_index(modified_index, self.window_size))
embeddings = self.encode(x_list_text + x_adv_list_text)
@@ -121,7 +125,7 @@ class SentenceEncoder(Constraint):
return self.sim_metric(original_embeddings, perturbed_embeddings)
def call_many(self, x, x_adv_list, original_text=None):
def _check_constraint_many(self, x, x_adv_list, original_text=None):
"""
Filters the list of perturbed texts so that the similarity between the original text
and the perturbed text is greater than the :obj:`threshold`.
@@ -151,7 +155,7 @@ class SentenceEncoder(Constraint):
mask = (scores >= self.threshold).cpu().numpy().nonzero()
return np.array(x_adv_list)[mask]
def __call__(self, x, x_adv):
def _check_constraint(self, x, x_adv, original_text=None):
return self.sim_score(x.text, x_adv.text) >= self.threshold
def extra_repr_keys(self):
@@ -169,4 +173,4 @@ def get_neg_euclidean_dist(emb1, emb2):
""" Returns the Euclidean distance between a batch of vectors and a batch of
vectors.
"""
return -torch.sum((emb1 - emb2) ** 2, dim=1)
return -torch.sum((emb1 - emb2) ** 2, dim=1)

Some files were not shown because too many files have changed in this diff Show More