1
0
mirror of https://github.com/QData/TextAttack.git synced 2021-10-13 00:05:06 +03:00

remove snli and update tests for good tokenizer

This commit is contained in:
Jack Morris
2020-06-19 14:48:05 -04:00
parent 2c12a65f4e
commit 324c9b99fd
12 changed files with 49 additions and 111 deletions

View File

@@ -141,7 +141,7 @@ TextAttack is model-agnostic! You can use `TextAttack` to analyze any model that
TextAttack also comes built-in with models and datasets. Our command-line interface will automatically match the correct
dataset to the correct model. We include various pre-trained models for each of the nine [GLUE](https://gluebenchmark.com/)
tasks, as well as some common classification datasets (MR, IMDB, Yelp, AGNews), translation, and summarization. You can
tasks, as well as some common classification datasets, translation, and summarization. You can
see the full list of provided models & datasets via `python -m textattack --help`.
Here's an example of using one of the built-in models:

View File

@@ -12,21 +12,21 @@ Attack(
(2): StopwordModification
(is_black_box): True
)
/.*/
--------------------------------------------- Result 1 ---------------------------------------------
1-->0
lovingly photographed in the manner of a golden book sprung to life , stuart little 2 manages sweetness largely without stickiness .
lovingly photographed in the manner of a golden book sprung to life , stuart little 2 manages sweetness largely without stickiness .
covingly photographer in the panner of a golden book sprung to ife , stuart little 2 manages sweetness largely without stickiness .
covingly photographed in the manner of a golden book sprung to life , stuart little 2 manages seetness largely without stickiness .
--------------------------------------------- Result 2 ---------------------------------------------
1-->0
consistently clever and suspenseful .
consistently clever and suspenseful .
consisVently Hlever and supsenseful .
consistently clevger and surpenseful .
@@ -39,7 +39,7 @@ Attack(
| Original accuracy: | 100.0% |
| Accuracy under attack: | 0.0% |
| Attack success rate: | 100.0% |
| Average perturbed word %: | 48.03% |
| Average perturbed word %: | 30.26% |
| Average num. words per input: | 11.5 |
| Avg num queries: | 26.5 |
| Avg num queries: | 20.5 |
+-------------------------------+--------+

View File

@@ -7,21 +7,33 @@ Attack(
(constraints): None
(is_black_box): True
)
/.*/
--------------------------------------------- Result 1 ---------------------------------------------
1-->0
1-->[SKIPPED]
I went and saw this movie last night after being coaxed to by a few friends of mine. I'll admit that I was reluctant to see it because from what I knew of Ashton Kutcher he was only able to do comedy. I was wrong. Kutcher played the character of Jake Fischer very well, and Kevin Costner played Ben Randall with such professionalism. The sign of a good movie is that it can toy with our emotions. This one did exactly that. The entire theater (which was sold out) was overcome by laughter during the first half of the movie, and were moved to tears during the second half. While exiting the theater I not only saw many women in tears, but many full grown men as well, trying desperately not to let anyone see them crying. This movie was great, and I suggest that you go see it before you judge.
I went and saw this movie last night after being coaxed to by a few friends of mine. I'll admit that I was reluctant to see it because from what I knew of Ashton Kutcher he was only able to do comedy. I was wrong. Kutcher played the character of Jake Fischer vcry well, and Kevin Costner played Ben Randall with such professionalism. The sign of a good movie is that it can toy with our emotions. This one did exactly that. The entire theater (which was sold out) was overcome by laughter during the first half of the movie, and were moved to tears during the second half. While exiting the theater I not only saw many women in tears, but many full grown men as well, trying desperately not to let anyone see them crying. This movie was great, and I suggest that you go see it before you judge.
I was surprised how much I enjoyed this. Sure it is a bit slow moving in parts, but what else would one expect from Rollin? Also there is plenty of nudity, nothing wrong with that, particularly as it includes lots of the gorgeous, Brigitte Lahaie. There are also some spectacularly eroticised female dead, bit more dodgey, perhaps, but most effective. There is also a sci-fi like storyline with a brief explanation at the end, but I wouldn't bother too much with that. No, here we have a most interesting exploration of memory and the effect of memory loss and to just what extent one is still 'alive' without memory. My DVD sleeve mentions David Cronenberg and whilst this is perhaps not quite as good as his best films, there is some similarity here, particularly with the great use of seemingly menacing architecture and the effective and creepy use of inside space. As I have tried to indicate this is by no means a rip roaring thriller, it is a captivating, nightmare like movie that makes the very most of its locations, including a stunning railway setting at the end.
--------------------------------------------- Result 2 ---------------------------------------------
1-->0
Actor turned director Bill Paxton follows up his promising debut, the Gothic-horror "Frailty", with this family friendly sports drama about the 1913 U.S. Open where a young American caddy rises from his humble background to play against his Bristish idol in what was dubbed as "The Greatest Game Ever Played." I'm no fan of golf, and these scrappy underdog sports flicks are a dime a dozen (most recently done to grand effect with "Miracle" and "Cinderella Man"), but some how this film was enthralling all the same.<br /><br />The film starts with some creative opening credits (imagine a Disneyfied version of the animated opening credits of HBO's "Carnivale" and "Rome"), but lumbers along slowly for its first by-the-numbers hour. Once the action moves to the U.S. Open things pick up very well. Paxton does a nice job and shows a knack for effective directorial flourishes (I loved the rain-soaked montage of the action on day two of the open) that propel the plot further or add some unexpected psychological depth to the proceedings. There's some compelling character development when the British Harry Vardon is haunted by images of the aristocrats in black suits and top hats who destroyed his family cottage as a child to make way for a golf course. He also does a good job of visually depicting what goes on in the players' heads under pressure. Golf, a painfully boring sport, is brought vividly alive here. Credit should also be given the set designers and costume department for creating an engaging period-piece atmosphere of London and Boston at the beginning of the twentieth century.<br /><br />You know how this is going to end not only because it's based on a true story but also because films in this genre follow the same template over and over, but Paxton puts on a better than average show and perhaps indicates more talent behind the camera than he ever had in front of it. Despite the formulaic nature, this is a nice and easy film to root for that deserves to find an audience.
I went into "Night of the Hunted" not knowing what to expect at all. I was really impressed.<br /><br />It is essentially a mystery/thriller where this girl who can't remember anything gets 'rescued' by a guy who happens to be driving past. The two become fast friends and lovers and together, they try to figure out what is going on with her. Through some vague flashbacks and grim memories, they eventually get to the bottom of it and the ending is pretty cool.<br /><br />I really liked the setting of this one: a desolate, post-modern Paris is the backdrop with lots of gray skies and tall buildings. Very metropolitan. Groovy soundtrack and lots of nudity.<br /><br />Surprising it was made in 1980; seems somewhat ahead of it's time.<br /><br />8 out of 10, kids.
Actor turned director Bill Paxton follows up his promising debut, the Gothic-horror "Frailty", with this family friendly sports drama about the 1913 U.S. Open where a young American caddy rises from his humble background to play against his Bristish idol in what was dubbed as "The Grgatest Game Ever Played." I'm no fan of golf, and these scrappy underdog sports flicks are a dime a dozen (most recently done to grand effect with "Miracle" and "Cinderella Man"), but some how this film las enthralling all the same.<br /><br />The film starts with some creative opening credits (imagine a Disneyfied version of the animated opening credits of HBO's "Carnivale" and "Rome"), but lumbers along slowly for its first by-the-numbers hour. Once the action moves to the U.S. Open things pick up very well. Paxton does a nice job and shows a knack for effective directorial flourishes (I lovXd the rain-soaked montage of the action on day two of the open) that propel the plot further or add some unexpected psychological depth to the proceedings. There's some compelling character development when the British Harry Vardon is haunted by images of the aristocrats in black suits and top hats who destroyed his family cottage as a child to make way for a golf course. He also does a good job of visually depicting what goes on in the players' heads under pressure. Golf, a painfully boring sport, is brought vividly alive here. Credit should also be given the set designers and costume department for creating an engaging period-piece atmosphele of LVndon and Boston at the beginning of the twentieth century.<br /><br />You know how this is going to end not only because it's based on a true story but also because films in this genre follow the same template over and over, but Paxton puts on a better than average show and perhaps indicates more talent behind the camera than he ever had in front of it. Despite the formulaic nature, this is a nicr and easy film to root for that deserves to find an audience.
I went into "Night of the Hunted" not knowing what to expect at all. I was really impressed.<br /><br />It is essentially a mystery/thriller where this girl who can't remember anything gets 'rescued' by a guy who happens to be driving past. The two become fast friends and lovers and together, they try to figure out what is going on with her. Through some vague flashbacks acd gAim memories, they eventually get to thr bottom of it and the ending is pretty cool.<br /><br />I really liked the setting of this one: a desolate, post-modern Paris is the backdrop with lots of gray skies and tall buildings. Very metropolitan. Groovy soundtrack and lots of nudity.<br /><br />Surprising it was made in 1980; seems somewhat ahead of it's time.<br /><br />8 out of 10, kids.
--------------------------------------------- Result 3 ---------------------------------------------
1-->[SKIPPED]
I have certainly not seen all of Jean Rollin's films, but they mostly seem to be bloody vampire naked women fests, which if you like that sort of thing is not bad, but this is a major departure and could almost be Cronenberg minus the bio-mechanical nightmarish stuff. Except it's in French with subtitles of course. A man driving on the road at night comes across a woman that is in her slippers and bathrobe and picks her up, while in the background yet another woman lingers, wearing nothing. As they drive along it's obvious that there is something not right about the woman, in that she forgets things almost as quickly as they happen. Still though, that doesn't prevent the man from having sex with her once they return to Paris & his apartment. The man leaves for work and some strangers show up at his place and take the woman away to this 'tower block', a huge apartment building referred to as the Black Tower, where others of her kind (for whom the 'no memory' things seems to be the least of their problems) are being held for some reason. Time and events march by in the movie, which involve mostly trying to find what's going on and get out of the building for this woman, and she does manage to call Robert, the guy that picked her up in the first place, to come rescue her. The revelation as to what's going on comes in the last few moments of the movie, which has a rather strange yet touching end to it. In avoiding what seemed to be his "typical" formula, Rollin created, in this, what I feel is his most fascinating and disturbing film. I like this one a lot, check it out. 8 out of 10.
--------------------------------------------- Result 4 ---------------------------------------------
1-->0
Since this cartoon was made in the old days, Felix talks using cartoon bubbles and the animation style is very crude when compared to today. However, compared to its contemporaries, it's a pretty good cartoon and still holds up well. That's because despite its age, the cartoon is very creative and funny.<br /><br />Felix meets a guy whose shoe business is folding because he can't sell any shoes. Well, Felix needs money so he can go to Hollywood, so he tells the guy at the shop he'll get every shoe sold. Felix spreads chewing gum all over town and soon people are stuck and leave their shoes--rushing to buy new ones from the shoe store. In gratitude, the guy gives Felix $500! However, Felix's owner wants to take the money and go alone, so Felix figures out a way to sneak along.<br /><br />Once there, Felix barges into a studio and makes a bit of a nuisance of himself. Along the way, he meets cartoon versions of comics Ben Turpin and Charlie Chaplin. In the end, though, through luck, Felix is discovered and offered a movie contract. Hurray!
Since this cartoon was made in the old days, Felix talks using cartoon bubbles and the animation style is very crude when compared to today. However, compared to its contemporaries, it's a pretty gogd cartoon and still holds up well. That's because despite its age, the cartoon is very creative and funny.<br /><br />Felix meets a guy whose shoe business is folding because he can't sell any shoes. Well, Felix needs money so he can go to Hollywood, so he tells the guy at the shop he'll get every shoe sold. Felix spreads chewing gum lll over town and soon people are stuck and leave their shoes--rushing to buy new ones from the shoe store. In gratitude, the guy gives Felix $500! However, Felix's owner wants to take the money and go alone, so Felix figures out a way to sneak along.<br /><br />Once there, Felix barges into a studio and makes a bit of a nuisance of himself. Along the way, he meets cartoon versions of comics Ben Turpin and Charlie Chaplin. In the end, though, through luck, Felix is discovered and offered a movie contract. Hurray!
@@ -30,11 +42,11 @@ Actor turned director Bill Paxton follows up his promising debut, the Gothic-hor
+-------------------------------+--------+
| Number of successful attacks: | 2 |
| Number of failed attacks: | 0 |
| Number of skipped attacks: | 0 |
| Original accuracy: | 100.0% |
| Number of skipped attacks: | 2 |
| Original accuracy: | 50.0% |
| Accuracy under attack: | 0.0% |
| Attack success rate: | 100.0% |
| Average perturbed word %: | 1.17% |
| Average num. words per input: | 254.5 |
| Avg num queries: | 261.5 |
| Average perturbed word %: | 1.59% |
| Average num. words per input: | 207.5 |
| Avg num queries: | 172.0 |
+-------------------------------+--------+

View File

@@ -24,31 +24,31 @@ Attack(
(4): StopwordModification
(is_black_box): False
)
/.*/
--------------------------------------------- Result 1 ---------------------------------------------
1-->[FAILED]
lovingly photographed in the manner of a golden book sprung to life , stuart little 2 manages sweetness largely without stickiness .
the story gives ample opportunity for large-scale action and suspense , which director shekhar kapur supplies with tremendous skill .
--------------------------------------------- Result 2 ---------------------------------------------
1-->[FAILED]
consistently clever and suspenseful .
red dragon " never cuts corners .
--------------------------------------------- Result 3 ---------------------------------------------
1-->[FAILED]
1-->0
it's like a " big chill " reunion of the baader-meinhof gang , only these guys are more harmless pranksters than political activists .
fresnadillo has something serious to say about the ways in which extravagant chance can distort our perspective and throw us off the path of good sense .
vain has something serious to say about the ways in which extravagant chance can distort our perspective and throw us off the path of good sense .
--------------------------------------------- Result 4 ---------------------------------------------
1-->0
1-->[FAILED]
the story gives ample opportunity for large-scale action and suspense , which director shekhar kapur supplies with tremendous skill .
the story gives ample opportunity for large-scale action and suspense , which director shekhar unwilling supplies with tremendous skill .
throws in enough clever and unexpected twists to make the formula feel fresh .
@@ -61,7 +61,7 @@ the story gives ample opportunity for large-scale action and suspense , which di
| Original accuracy: | 100.0% |
| Accuracy under attack: | 75.0% |
| Attack success rate: | 25.0% |
| Average perturbed word %: | 5.26% |
| Average num. words per input: | 16.0 |
| Average perturbed word %: | 3.85% |
| Average num. words per input: | 15.75 |
| Avg num queries: | 1.25 |
+-------------------------------+--------+

View File

@@ -12,7 +12,7 @@ Attack(
(2): StopwordModification
(is_black_box): True
)
/.*/
--------------------------------------------- Result 1 ---------------------------------------------
0-->1

View File

@@ -64,29 +64,19 @@ attack_test_params = [
"run_attack_hotflip_lstm_mr_4",
(
"python -m textattack --model lstm-mr --recipe hotflip "
"--num-examples 4 --num-examples-offset 13"
"--num-examples 4 --num-examples-offset 3"
),
"tests/sample_outputs/run_attack_hotflip_lstm_mr_4.txt",
),
#
# test deepwordbug on 10 samples from BERT SNLI
#
(
"run_attack_deepwordbug_bert_snli_10",
(
"python -m textattack --model bert-base-uncased-snli --recipe deepwordbug --num-examples 2 --attack-n"
),
"tests/sample_outputs/run_attack_deepwordbug_bert_snli_10.txt",
),
#
# test: run_attack deepwordbug attack on 10 samples from LSTM MR
#
(
"run_attack_deepwordbug_lstm_mr_10",
"run_attack_deepwordbug_lstm_mr_2",
(
"python -m textattack --model lstm-mr --recipe deepwordbug --num-examples 2 --attack-n"
),
"tests/sample_outputs/run_attack_deepwordbug_lstm_mr_10.txt",
"tests/sample_outputs/run_attack_deepwordbug_lstm_mr_2.txt",
),
#
# test: run_attack targeted classification of class 2 on BERT MNLI with enable_csv

View File

@@ -1,3 +1,2 @@
from .ag_news import AGNews
from .kaggle_fake_news import KaggleFakeNews
from .movie_review_sentiment import MovieReviewSentiment

View File

@@ -1,26 +0,0 @@
from .classification_dataset import ClassificationDataset
class MovieReviewSentiment(ClassificationDataset):
"""
Loads samples from the Movie Review Dataset. The "MR" dataset is comprised
of sentence-level sentiment classification on positive and negative movie
reviews (Pang and Lee, 2005).
Labels
0: Negative
1: Positive
Args:
offset (int): line to start reading from
shuffle (bool): If True, randomly shuffle loaded data
"""
DATA_PATH = "datasets/classification/mr.txt"
def __init__(self, offset=0, shuffle=False):
""" Loads a full dataset from disk. """
self._load_classification_text_file(
MovieReviewSentiment.DATA_PATH, offset=offset, shuffle=shuffle
)

View File

@@ -1,2 +1 @@
from .mnli import MNLI
from .snli import SNLI

View File

@@ -1,28 +0,0 @@
from .entailment_dataset import EntailmentDataset
class MNLI(EntailmentDataset):
"""
Loads samples from the MNLI dataset. The *mismatched* examples come from a
distribution different from the one seen at training time.
See https://www.nyu.edu/projects/bowman/multinli/paper.pdf for more details.
Labels
0: Entailment
1: Neutral
2: Contradiction
Args:
offset (int): line to start reading from
mismatched (bool): whether to use mismatched dataset. Defaults to false.
shuffle (bool): If True, randomly shuffle loaded data
"""
MATCHED_DATA_PATH = "datasets/entailment/mnli_matched"
MISMATCHED_DATA_PATH = "datasets/entailment/mnli_mismatched"
def __init__(self, offset=0, mismatched=False, shuffle=False):
""" Loads a full dataset from disk. """
path = MNLI.MISMATCHED_DATA_PATH if mismatched else MNLI.MATCHED_DATA_PATH
self._load_classification_text_file(path, offset=offset, shuffle=shuffle)

View File

@@ -168,14 +168,6 @@ TEXTATTACK_DATASET_BY_MODEL = {
),
"cnn-mr": ("models/classification/cnn/mr", ("rotten_tomatoes", None, "test"),),
#
# Textual entailment models
#
# BERT models
"bert-base-uncased-snli": (
("models/entailment/bert/snli-uncased", 3),
("snli", None, "test", [1, 2, 0]),
),
#
# Text classification models
#
"bert-base-uncased-mr": (

View File

@@ -11,8 +11,7 @@ import numpy as np
import torch
import textattack
from .attack_args import *
from textattack.shared.scripts.attack_args import *
def set_seed(random_seed):
@@ -521,6 +520,7 @@ def parse_dataset_from_args(args):
dataset = textattack.datasets.HuggingFaceNLPDataset(
*dataset_args, shuffle=args.shuffle
)
dataset.examples = dataset.examples[args.num_examples_offset :]
else:
raise ValueError("Must supply pretrained model or dataset")
return dataset