alihan/textattack-nlp-transformer
1
0
Fork 0
mirror of https://github.com/QData/TextAttack.git synced 2021-10-13 00:05:06 +03:00
Code Issues Projects Releases Wiki Activity

formatting update and new test

Browse Source
This commit is contained in:
Jin Yong Yoo
2020-06-30 04:08:43 -04:00
parent 73c0fda293
commit ec5598baed
4 changed files with 81 additions and 86 deletions
Download Patch File Download Diff File Expand all files Collapse all files

74
tests/sample_outputs/run_attack_faster_alzantot_recipe.txt Normal file
View File

@@ -0,0 +1,74 @@
The current process just got forked. Disabling parallelism to avoid deadlocks...
To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)
The current process just got forked. Disabling parallelism to avoid deadlocks...
To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)
The current process just got forked. Disabling parallelism to avoid deadlocks...
To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)
Attack(
(search_method): GeneticAlgorithm(
(pop_size): 60
(max_iters): 20
(temp): 0.3
(give_up_if_no_improvement): False
)
(goal_function): UntargetedClassification
(transformation): WordSwapEmbedding(
(max_candidates): 8
(embedding_type): paragramcf
)
(constraints):
(0): MaxWordsPerturbed(
(max_percent): 0.2
)
(1): WordEmbeddingDistance(
(embedding_type): paragramcf
(max_mse_dist): 0.5
(cased): False
(include_unknown_words): True
)
(2): LearningToWriteLanguageModel(
(max_log_prob_diff): 5.0
)
(3): RepeatModification
(4): StopwordModification
(is_black_box): True
)
--------------------------------------------- Result 1 ---------------------------------------------
Positive (100%) --> Negative (72%)
this kind of hands-on storytelling is ultimately what makes shanghai ghetto move beyond a good , dry , reliable textbook and what allows it to rank with its worthy predecessors .
this type of hands-on tale is ultimately what makes shanghai ghetto move beyond a good , dry , reliable manuals and what allowed it to rank with its worthy ancestors .
--------------------------------------------- Result 2 ---------------------------------------------
Positive (80%) --> Negative (99%)
making such a tragedy the backdrop to a love story risks trivializing it , though chouraqui no doubt intended the film to affirm love's power to help people endure almost unimaginable horror .
making such a dramas the backdrop to a love story risks trivializing it , notwithstanding chouraqui no doubt intended the film to affirm love's power to help people endure almost unimaginable horror .
--------------------------------------------- Result 3 ---------------------------------------------
Positive (92%) --> Negative (76%)
grown-up quibbles are beside the point here . the little girls understand , and mccracken knows that's all that matters .
increased-up quibbles are beside the point here . the little girls understands , and mccracken knows that's all that matters .
+-------------------------------+--------+
| Attack Results | |
+-------------------------------+--------+
| Number of successful attacks: | 3 |
| Number of failed attacks: | 0 |
| Number of skipped attacks: | 0 |
| Original accuracy: | 100.0% |
| Accuracy under attack: | 0.0% |
| Attack success rate: | 100.0% |
| Average perturbed word %: | 11.16% |
| Average num. words per input: | 27.0 |
| Avg num queries: | 766.0 |
+-------------------------------+--------+

77
tests/sample_outputs/run_attack_genetic_lstm_imdb.txt
View File

@@ -1,77 +0,0 @@
Attack(
(search_method): GeneticAlgorithm(
(pop_size): 20
(max_iters): 50
(temp): 0.3
(give_up_if_no_improvement): False
)
(goal_function): UntargetedClassification
(transformation): WordSwapEmbedding(
(max_candidates): 15
(embedding_type): paragramcf
)
(constraints):
(0): WordEmbeddingDistance(
(embedding_type): paragramcf
(min_cos_sim): 0.7
(cased): False
(include_unknown_words): True
)
(1): BERT(
(metric): cosine
(threshold): 0.7
(compare_with_original): False
(window_size): None
(skip_text_shorter_than_window): False
)
(2): RepeatModification
(3): StopwordModification
(is_black_box): True
)
--------------------------------------------- Result 1 ---------------------------------------------
Positive (98%) --> Negative (54%)
Since this cartoon was made in the old days, Felix talks using cartoon bubbles and the animation style is very crude when compared to today. However, compared to its contemporaries, it's a pretty good cartoon and still holds up well. That's because despite its age, the cartoon is very creative and funny.<br /><br />Felix meets a guy whose shoe business is folding because he can't sell any shoes. Well, Felix needs money so he can go to Hollywood, so he tells the guy at the shop he'll get every shoe sold. Felix spreads chewing gum all over town and soon people are stuck and leave their shoes--rushing to buy new ones from the shoe store. In gratitude, the guy gives Felix $500! However, Felix's owner wants to take the money and go alone, so Felix figures out a way to sneak along.<br /><br />Once there, Felix barges into a studio and makes a bit of a nuisance of himself. Along the way, he meets cartoon versions of comics Ben Turpin and Charlie Chaplin. In the end, though, through luck, Felix is discovered and offered a movie contract. Hurray!
Since this cartoon was made in the antiquity days, Felix talks exploit toon bubbles and the animation style is very crude when compared to today. However, compared to its contemporaries, it's a pretty alright cartoon and nevertheless holds up well. That's because despite its age, the cartoon is very creative and funny.<br /><br />Felix meets a guy whose shoe business is folding because he can't sell any shoes. Well, Felix needs money so he can go to Hollywood, so he tells the guy at the shop he'll get every sneakers sold. Felix spreads chewed gum all over town and soon people are stuck and exiting their shoes--rushing to buy new ones from the shoe store. In gratitude, the guy gives Felix $500! However, Felix's owner wants to take the money and go alone, so Felix figures out a way to sneak along.<br /><br />Once there, Felix barges into a studio and makes a bit of a nuisance of himself. Along the way, he meets cartoon versions of comics Ben Turpin and Charlie Chaplin. In the end, though, through luck, Felix is discovered and offered a movie contract. Hurray!
--------------------------------------------- Result 2 ---------------------------------------------
Positive (98%) --> Negative (59%)
Despite the title and unlike some other stories about love and war, this film isn't too sticky and pink, because love is as a rose: With thorns, that is. The four leading actors set their characters realistic and with a good sense and balance between the tragic and the down-to-earth. <br /><br />The music and lyrics of the cabaret/chanson-esquire songs (sung b Keira Knightley herself) drag the viewer deeper and deeper in the film, from one place to another, between the brutal war and amongst the peaceful love. Some people may find it too much a biopic, but it ís mostly a romantic story, even though it consequently follows the life of Dylan Thomas and the triangular relationship which is steeped by joy and jealousy. <br /><br />London gets visualized from another angle for once, the bohemian life of Dylan during the bombings of the Germans is set in a floating atmosphere of small bedrooms, pubs and bars. The independent women, the soldier and the charismatic poet are constantly swept in both feelings of love and anger. <br /><br />Maybe the end is too twisted and hangs somewhat loosely to the rest of the film, but all in all this is a great romantic story.
Despite the title and unlike some other stories about love and war, this film isn't too sticky and pink, because love is as a rose: With thorns, that is. The four leading actors set their characters realistic and with a good meaning and balance between the terrible and the down-to-earth. <br /><br />The music and lyrics of the cabaret/chanson-esquire songs (sung b Keira Knightley herself) drag the viewer deeper and deeper in the film, from one place to another, between the brutal war and amongst the peaceful love. Some people may find it too much a biopic, but it ís basically a romantic story, even though it consequently follows the life of Dylan Thomas and the triangular relationship which is steeped by joy and jealousy. <br /><br />London gets visualized from another angle for once, the bohemian life of Dylan during the bombings of the Germans is set in a floating atmosphere of small bedrooms, pubs and bars. The independent women, the soldier and the charismatic poet are constantly swept in both feelings of love and anger. <br /><br />Maybe the end is too twisted and hangs somewhat loosely to the rest of the film, but all in all this is a great romantic story.
--------------------------------------------- Result 3 ---------------------------------------------
Positive (97%) --> Negative (89%)
Felix in Hollywood is a great film. The version I viewed was very well restored, which is sometimes a problem with these silent era animated films. It has some of Hollywood's most famous stars making cameo animated appearances. A must for any silent film or animation enthusiast.
Felix in Hollywood is a whopping film. The version I construed was very well restored, which is ordinarily a troubles with these silent era animated films. It has some of Hollywood's most proverbial stars making cameo animated appearances. A must for any voiceless film or animation enthusiast.
--------------------------------------------- Result 4 ---------------------------------------------
Positive (97%) --> Negative (69%)
A gem of a cartoon from the silent era---it was re-discovered by CARTOON NETWORK, and was broadcast for likely the first time in decades, if ever.<br /><br />What makes this so enjoyable are the varied cameos...Douglas Fairbanks is attacked by giant mosquitos; Will Hays pays a visit as 'boss' of Static Studios; as well as appearances by Chaplin, Keaton, and William S. Hart. The image of chewing gum decimating the shoes of the populace (a money-making idea for Felix's near-bankrupt shoe-=salesman boss) cannot be described--it must be viewed. A terrific cultural gem.
una gem of a cartoon from the silencers era---it was re-discovered by TRAVESTY NETWORK, and was broadcast for likely the first time in decades, if ever.<br /><br />What makes this so pleasurable are the assorted cameos...Douglas Fairbanks is attacked by giant mosquitos; Will Hays pays a visit as 'boss' of Static Studios; as well as appearances by Chaplin, Keaton, and William S. Hart. The image of chewing erasers decimating the shoes of the populace (a money-making idea for Felix's near-bankrupt shoe-=salesman boss) cannot be described--it must be viewed. A awesome cultural gem.
+-------------------------------+--------+
| Attack Results | |
+-------------------------------+--------+
| Number of successful attacks: | 4 |
| Number of failed attacks: | 0 |
| Number of skipped attacks: | 0 |
| Original accuracy: | 100.0% |
| Accuracy under attack: | 0.0% |
| Attack success rate: | 100.0% |
| Average perturbed word %: | 6.24% |
| Average num. words per input: | 138.5 |
| Avg num queries: | 2318.5 |
+-------------------------------+--------+

9
tests/test_command_line/test_attack.py
View File

@@ -112,14 +112,13 @@ attack_test_params = [
),
# fmt: on
#
# test: run_attack on LSTM IMDB using word embedding transformation and genetic algorithm. Simulate alzantot recipe without using expensive LM
# test: run_attack on LSTM MR using word embedding transformation and genetic algorithm. Simulate alzantot recipe without using expensive LM
(
"run_attack_genetic_lstm_imdb",
"run_attack_faster_alzantot_recipe",
(
"textattack attack --model lstm-imdb --search ga-word --transformation word-swap-embedding "
"--constraints repeat stopword embedding:min_cos_sim=0.7 bert --num-examples 4 --num-examples-offset 21"
"textattack attack --model lstm-mr --recipe faster-alzantot --num-examples 3 --num-examples-offset 20"
),
"tests/sample_outputs/run_attack_genetic_lstm_imdb.txt",
"tests/sample_outputs/run_attack_faster_alzantot_recipe.txt",
),
]

7
textattack/search_methods/genetic_algorithm.py
View File

@@ -35,7 +35,6 @@ class GeneticAlgorithm(SearchMethod):
temp=0.3,
give_up_if_no_improvement=False,
max_crossover_retries=20,
compare_against_original=True,
):
self.max_iters = max_iters
self.pop_size = pop_size
@@ -68,7 +67,7 @@ class GeneticAlgorithm(SearchMethod):
transformations = self.get_transformations(
pop_member.attacked_text,
original_text=original_result.attacked_text,
indices_to_modify=[rand_idx]
indices_to_modify=[rand_idx],
)
if not len(transformations):
@@ -175,8 +174,8 @@ class GeneticAlgorithm(SearchMethod):
# Therefore, we give small non-zero probability for words with no candidates
# Epsilon is some small number to approximately assign 1% probability
total_candidates = np.sum(num_candidates_per_word)
num_zero_elements = len(words) - np.count_zero(num_candidates_per_word)
epsilon = min(1, int(total_candidates / (100 - num_zero_elements)))
num_zero_elements = len(words) - np.count_nonzero(num_candidates_per_word)
epsilon = max(1, int(total_candidates / (100 - num_zero_elements)))
for i in range(len(num_candidates_per_word)):
if num_candidates_per_word[i] == 0:
num_candidates_per_word[i] = epsilon