{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "word_embeddings_visualize.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [
        "TbSNxUn7kTIM",
        "lv6XVM22kTId",
        "oaorFoE3kTIk",
        "Esjuaq3jkTI9",
        "M-sq8TFakTJK",
        "6eUXtEMfkTJx",
        "VGoQb8tgkTJ-",
        "A1DV4ciIkTKP",
        "wk6n6UM7kTLO"
      ]
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "[View in Colaboratory](https://colab.research.google.com/github/gmihaila/deep_learning_toolbox/blob/master/word_embeddings_visualize.ipynb)"
      ]
    },
    {
      "metadata": {
        "id": "nMD5ET5YkTHJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Homework 2: Word Embeddings \n",
        "\n",
        "### George Mihaila\n",
        "\n",
        "<p> This homework is heavily based on <a href=https://github.com/oxford-cs-deepnlp-2017/practical-1>practical1</a> from \n",
        "the <a href=https://www.cs.ox.ac.uk/teaching/courses/2016-2017/dl/>Oxford CS - Deep NLP 2017</a> by Yannis Assael, Brendan Shillingford and Chris Dyer.\""
      ]
    },
    {
      "metadata": {
        "id": "6VPmZTC4kTHL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Steps to run the homework:\n",
        "\n",
        "<ol>\n",
        "<li> Download and install Anaconda from <a href=\\\"https://conda.io/docs/user-guide/install/index.html#regular-installation\\\">here</a>. In Linux, you have to download a file (choose Python 2.7 or 3.6, whichever you have in your machine) and run a simple command.\n",
        "    \n",
        "<li> The file <code>homework2.env</code> lists all the packages required to run the homework (and some that you don't really need). Create an conda environment with the right packages and versions: <code> conda create --name hw2 --file homework2.env </code>\n",
        "\n",
        "<li> Activate the environment: <code> source activate hw2 </code>\n",
        "\n",
        "<li> Run the homework (it will open in your browser): <code> jupyter notebook word_embeddings.ipynb </code>\n",
        "    \n",
        "<li> You can now run each cell using <code> Shift + Enter </code>\n",
        "</ol>\n",
        "\n",
        "Conda is extremely useful if you need to share your current installation (packages + versions) with others, keep different versions in different environments (for example, if you have old and new code), etc. You can find the documentation <a href=\"https://conda.io/docs/user-guide/tasks/manage-environments.html\">here</a>."
      ]
    },
    {
      "metadata": {
        "id": "jPIPNJPhkTHM",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import os\n",
        "from random import shuffle\n",
        "import re"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "P3UIzEcvkTHQ",
        "colab_type": "code",
        "colab": {},
        "outputId": "d304759e-a0f6-4dbc-cd6f-f4f083d5df00"
      },
      "cell_type": "code",
      "source": [
        "from bokeh.models import ColumnDataSource, LabelSet\n",
        "from bokeh.plotting import figure, show, output_file\n",
        "from bokeh.io import output_notebook\n",
        "output_notebook()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "    <div class=\"bk-root\">\n",
              "        <a href=\"https://bokeh.pydata.org\" target=\"_blank\" class=\"bk-logo bk-logo-small bk-logo-notebook\"></a>\n",
              "        <span id=\"cd4fd9fb-9e11-4927-9288-3aa2e926ed76\">Loading BokehJS ...</span>\n",
              "    </div>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "\n",
              "(function(root) {\n",
              "  function now() {\n",
              "    return new Date();\n",
              "  }\n",
              "\n",
              "  var force = true;\n",
              "\n",
              "  if (typeof (root._bokeh_onload_callbacks) === \"undefined\" || force === true) {\n",
              "    root._bokeh_onload_callbacks = [];\n",
              "    root._bokeh_is_loading = undefined;\n",
              "  }\n",
              "\n",
              "\n",
              "  \n",
              "  if (typeof (root._bokeh_timeout) === \"undefined\" || force === true) {\n",
              "    root._bokeh_timeout = Date.now() + 5000;\n",
              "    root._bokeh_failed_load = false;\n",
              "  }\n",
              "\n",
              "  var NB_LOAD_WARNING = {'data': {'text/html':\n",
              "     \"<div style='background-color: #fdd'>\\n\"+\n",
              "     \"<p>\\n\"+\n",
              "     \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n",
              "     \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n",
              "     \"</p>\\n\"+\n",
              "     \"<ul>\\n\"+\n",
              "     \"<li>re-rerun `output_notebook()` to attempt to load from CDN again, or</li>\\n\"+\n",
              "     \"<li>use INLINE resources instead, as so:</li>\\n\"+\n",
              "     \"</ul>\\n\"+\n",
              "     \"<code>\\n\"+\n",
              "     \"from bokeh.resources import INLINE\\n\"+\n",
              "     \"output_notebook(resources=INLINE)\\n\"+\n",
              "     \"</code>\\n\"+\n",
              "     \"</div>\"}};\n",
              "\n",
              "  function display_loaded() {\n",
              "    if (root.Bokeh !== undefined) {\n",
              "      var el = document.getElementById(\"cd4fd9fb-9e11-4927-9288-3aa2e926ed76\");\n",
              "      if (el != null) {\n",
              "        el.textContent = \"BokehJS \" + Bokeh.version + \" successfully loaded.\";\n",
              "      }\n",
              "    } else if (Date.now() < root._bokeh_timeout) {\n",
              "      setTimeout(display_loaded, 100)\n",
              "    }\n",
              "  }\n",
              "\n",
              "\n",
              "  function run_callbacks() {\n",
              "    try {\n",
              "      root._bokeh_onload_callbacks.forEach(function(callback) { callback() });\n",
              "    }\n",
              "    finally {\n",
              "      delete root._bokeh_onload_callbacks\n",
              "    }\n",
              "    console.info(\"Bokeh: all callbacks have finished\");\n",
              "  }\n",
              "\n",
              "  function load_libs(js_urls, callback) {\n",
              "    root._bokeh_onload_callbacks.push(callback);\n",
              "    if (root._bokeh_is_loading > 0) {\n",
              "      console.log(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n",
              "      return null;\n",
              "    }\n",
              "    if (js_urls == null || js_urls.length === 0) {\n",
              "      run_callbacks();\n",
              "      return null;\n",
              "    }\n",
              "    console.log(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n",
              "    root._bokeh_is_loading = js_urls.length;\n",
              "    for (var i = 0; i < js_urls.length; i++) {\n",
              "      var url = js_urls[i];\n",
              "      var s = document.createElement('script');\n",
              "      s.src = url;\n",
              "      s.async = false;\n",
              "      s.onreadystatechange = s.onload = function() {\n",
              "        root._bokeh_is_loading--;\n",
              "        if (root._bokeh_is_loading === 0) {\n",
              "          console.log(\"Bokeh: all BokehJS libraries loaded\");\n",
              "          run_callbacks()\n",
              "        }\n",
              "      };\n",
              "      s.onerror = function() {\n",
              "        console.warn(\"failed to load library \" + url);\n",
              "      };\n",
              "      console.log(\"Bokeh: injecting script tag for BokehJS library: \", url);\n",
              "      document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
              "    }\n",
              "  };var element = document.getElementById(\"cd4fd9fb-9e11-4927-9288-3aa2e926ed76\");\n",
              "  if (element == null) {\n",
              "    console.log(\"Bokeh: ERROR: autoload.js configured with elementid 'cd4fd9fb-9e11-4927-9288-3aa2e926ed76' but no matching script tag was found. \")\n",
              "    return false;\n",
              "  }\n",
              "\n",
              "  var js_urls = [\"https://cdn.pydata.org/bokeh/release/bokeh-0.12.7.min.js\", \"https://cdn.pydata.org/bokeh/release/bokeh-widgets-0.12.7.min.js\", \"https://cdn.pydata.org/bokeh/release/bokeh-tables-0.12.7.min.js\", \"https://cdn.pydata.org/bokeh/release/bokeh-gl-0.12.7.min.js\"];\n",
              "\n",
              "  var inline_js = [\n",
              "    function(Bokeh) {\n",
              "      Bokeh.set_log_level(\"info\");\n",
              "    },\n",
              "    \n",
              "    function(Bokeh) {\n",
              "      \n",
              "    },\n",
              "    \n",
              "    function(Bokeh) {\n",
              "      \n",
              "      document.getElementById(\"cd4fd9fb-9e11-4927-9288-3aa2e926ed76\").textContent = \"BokehJS is loading...\";\n",
              "    },\n",
              "    function(Bokeh) {\n",
              "      console.log(\"Bokeh: injecting CSS: https://cdn.pydata.org/bokeh/release/bokeh-0.12.7.min.css\");\n",
              "      Bokeh.embed.inject_css(\"https://cdn.pydata.org/bokeh/release/bokeh-0.12.7.min.css\");\n",
              "      console.log(\"Bokeh: injecting CSS: https://cdn.pydata.org/bokeh/release/bokeh-widgets-0.12.7.min.css\");\n",
              "      Bokeh.embed.inject_css(\"https://cdn.pydata.org/bokeh/release/bokeh-widgets-0.12.7.min.css\");\n",
              "      console.log(\"Bokeh: injecting CSS: https://cdn.pydata.org/bokeh/release/bokeh-tables-0.12.7.min.css\");\n",
              "      Bokeh.embed.inject_css(\"https://cdn.pydata.org/bokeh/release/bokeh-tables-0.12.7.min.css\");\n",
              "    }\n",
              "  ];\n",
              "\n",
              "  function run_inline_js() {\n",
              "    \n",
              "    if ((root.Bokeh !== undefined) || (force === true)) {\n",
              "      for (var i = 0; i < inline_js.length; i++) {\n",
              "        inline_js[i].call(root, root.Bokeh);\n",
              "      }if (force === true) {\n",
              "        display_loaded();\n",
              "      }} else if (Date.now() < root._bokeh_timeout) {\n",
              "      setTimeout(run_inline_js, 100);\n",
              "    } else if (!root._bokeh_failed_load) {\n",
              "      console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n",
              "      root._bokeh_failed_load = true;\n",
              "    } else if (force !== true) {\n",
              "      var cell = $(document.getElementById(\"cd4fd9fb-9e11-4927-9288-3aa2e926ed76\")).parents('.cell').data().cell;\n",
              "      cell.output_area.append_execute_result(NB_LOAD_WARNING)\n",
              "    }\n",
              "\n",
              "  }\n",
              "\n",
              "  if (root._bokeh_is_loading === 0) {\n",
              "    console.log(\"Bokeh: BokehJS loaded, going straight to plotting\");\n",
              "    run_inline_js();\n",
              "  } else {\n",
              "    load_libs(js_urls, function() {\n",
              "      console.log(\"Bokeh: BokehJS plotting callback run at\", now());\n",
              "      run_inline_js();\n",
              "    });\n",
              "  }\n",
              "}(window));"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "VUdgSiEIkTHX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Part 0: Download the TED dataset"
      ]
    },
    {
      "metadata": {
        "id": "YUcQzgPdkTHY",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import urllib.request\n",
        "import zipfile\n",
        "import lxml.etree"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "yIs0U1mykTHd",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Download the dataset if it's not already there: this may take a minute as it is 75MB\n",
        "if not os.path.isfile('ted_en-20160408.zip'):\n",
        "    urllib.request.urlretrieve(\"https://wit3.fbk.eu/get.php?path=XML_releases/xml/ted_en-20160408.zip&filename=ted_en-20160408.zip\", filename=\"ted_en-20160408.zip\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "7pjuO8e7kTHj",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# For now, we're only interested in the subtitle text, so let's extract that from the XML:\n",
        "with zipfile.ZipFile('ted_en-20160408.zip', 'r') as z:\n",
        "    doc = lxml.etree.parse(z.open('ted_en-20160408.xml', 'r'))\n",
        "input_text = '\\n'.join(doc.xpath('//content/text()'))\n",
        "del doc"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "GKtqNHWQkTHn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Part 1: Preprocessing\n",
        "\n",
        "In this part, we attempt to clean up the raw subtitles a bit, so that we get only sentences. The following substring shows examples of what we're trying to get rid of. Since it's hard to define precisely what we want to get rid of, we'll just use some simple heuristics."
      ]
    },
    {
      "metadata": {
        "id": "R5CGNVn3kTHo",
        "colab_type": "code",
        "colab": {},
        "outputId": "b9626207-a356-4bb5-d396-ff9140f186fd"
      },
      "cell_type": "code",
      "source": [
        "i = input_text.find(\"Hyowon Gweon: See this?\")\n",
        "input_text[i-20:i+150]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "' baby does.\\n(Video) Hyowon Gweon: See this? (Ball squeaks) Did you see that? (Ball squeaks) Cool. See this one? (Ball squeaks) Wow.\\nLaura Schulz: Told you. (Laughs)\\n(Vide'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "metadata": {
        "id": "hyhC11XRkTHt",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Let's start by removing all parenthesized strings using a regex:"
      ]
    },
    {
      "metadata": {
        "id": "jioT-rKbkTHu",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "input_text_noparens = re.sub(r'\\([^)]*\\)', '', input_text)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "j8hv5GmgkTHz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We can verify the same location in the text is now clean as follows. We won't worry about the irregular spaces since we'll later split the text into sentences and tokenize it anyway."
      ]
    },
    {
      "metadata": {
        "id": "to15OsfDkTH0",
        "colab_type": "code",
        "colab": {},
        "outputId": "2bfbb489-f832-4d9e-bc59-e5a24151ad8d"
      },
      "cell_type": "code",
      "source": [
        "i = input_text_noparens.find(\"Hyowon Gweon: See this?\")\n",
        "input_text_noparens[i-20:i+150]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"hat the baby does.\\n Hyowon Gweon: See this?  Did you see that?  Cool. See this one?  Wow.\\nLaura Schulz: Told you. \\n HG: See this one?  Hey Clara, this one's for you. You \""
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "metadata": {
        "id": "87aJ_TjQkTH5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now, let's attempt to remove speakers' names that occur at the beginning of a line, by deleting pieces of the form \"`<up to 20 characters>:`\", as shown in this example. Of course, this is an imperfect heuristic. "
      ]
    },
    {
      "metadata": {
        "id": "7NzjpJe_kTH5",
        "colab_type": "code",
        "colab": {},
        "outputId": "3bd8bf4e-ec05-4c7f-c1f4-d52070d21bf4"
      },
      "cell_type": "code",
      "source": [
        "sentences_strings_ted = []\n",
        "for line in input_text_noparens.split('\\n'):\n",
        "    m = re.match(r'^(?:(?P<precolon>[^:]{,20}):)?(?P<postcolon>.*)$', line)\n",
        "    sentences_strings_ted.extend(sent for sent in m.groupdict()['postcolon'].split('.') if sent)\n",
        "\n",
        "# Uncomment if you need to save some RAM: these strings are about 50MB.\n",
        "# del input_text, input_text_noparens\n",
        "\n",
        "# Let's view the first few:\n",
        "sentences_strings_ted[:5]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[\"Here are two reasons companies fail: they only do more of the same, or they only do what's new\",\n",
              " 'To me the real, real solution to quality growth is figuring out the balance between two activities: exploration and exploitation',\n",
              " ' Both are necessary, but it can be too much of a good thing',\n",
              " 'Consider Facit',\n",
              " \" I'm actually old enough to remember them\"]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "metadata": {
        "id": "GOIXXhG3kTIA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now that we have sentences, we're ready to tokenize each of them into words. This tokenization is imperfect, of course. For instance, how many tokens is \"can't\", and where/how do we split it? We'll take the simplest naive approach of splitting on spaces. Before splitting, we remove non-alphanumeric characters, such as punctuation. You may want to consider the following question: why do we replace these characters with spaces rather than deleting them? Think of a case where this yields a different answer."
      ]
    },
    {
      "metadata": {
        "id": "D_jq7UCbkTIB",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "sentences_ted = []\n",
        "for sent_str in sentences_strings_ted:\n",
        "    tokens = re.sub(r\"[^a-z0-9]+\", \" \", sent_str.lower()).split()\n",
        "    sentences_ted.append(tokens)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "BCG4MpSpkTIF",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Two sample processed sentences:"
      ]
    },
    {
      "metadata": {
        "id": "Sfun50r_kTIH",
        "colab_type": "code",
        "colab": {},
        "outputId": "fc74b34c-4f0a-4b9e-ba37-765a0da2aecc"
      },
      "cell_type": "code",
      "source": [
        "print(sentences_ted[0])\n",
        "print(sentences_ted[1])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['here', 'are', 'two', 'reasons', 'companies', 'fail', 'they', 'only', 'do', 'more', 'of', 'the', 'same', 'or', 'they', 'only', 'do', 'what', 's', 'new']\n",
            "['to', 'me', 'the', 'real', 'real', 'solution', 'to', 'quality', 'growth', 'is', 'figuring', 'out', 'the', 'balance', 'between', 'two', 'activities', 'exploration', 'and', 'exploitation']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "TbSNxUn7kTIM",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Part 2: Word Frequencies"
      ]
    },
    {
      "metadata": {
        "id": "BJdVQGAbkTIN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "If you store (1) the counts of the top 1000 words in a list called `counts_ted_top1000` and (2) the actual words in `words_top_ted`, the code below will plot the histogram requested in the writeup."
      ]
    },
    {
      "metadata": {
        "id": "g58QCip7kTIP",
        "colab_type": "code",
        "colab": {},
        "outputId": "ac625d6f-4da0-42b7-d289-e994a32685eb"
      },
      "cell_type": "code",
      "source": [
        "import collections\n",
        "ted_counts = collections.defaultdict(int)\n",
        "counts_ted_top1000 = []\n",
        "words_top_ted = []\n",
        "\n",
        "for sentence in sentences_ted:\n",
        "    for word in sentence:\n",
        "        if ted_counts.keys().__contains__(word):\n",
        "            ted_counts[word] += 1\n",
        "        else:\n",
        "            ted_counts[word] = 1\n",
        "#sort words in decenting order\n",
        "sorted_words_ted = sorted(ted_counts, key=ted_counts.__getitem__, reverse=True)\n",
        "#get first 1000 counts from sorted words\n",
        "{counts_ted_top1000.append(ted_counts[word]) for word in sorted_words_ted[:1000]}\n",
        "#get first 1000 words from sorted words\n",
        "{words_top_ted.append(word) for word in sorted_words_ted[:1000]}\n",
        "print (\"Succesfully sorted Ted Dataset!\")\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Succesfully sorted Ted Dataset!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "y6woBIFOkTIW",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Plot distribution of top-1000 words"
      ]
    },
    {
      "metadata": {
        "id": "0-6Ha3htkTIX",
        "colab_type": "code",
        "colab": {},
        "outputId": "66a94986-8a98-4215-8934-8bf1390561d3"
      },
      "cell_type": "code",
      "source": [
        "hist, edges = np.histogram(counts_ted_top1000, density=True, bins=100, normed=True)\n",
        "\n",
        "p = figure(tools=\"pan,wheel_zoom,reset,save\",\n",
        "           toolbar_location=\"above\",\n",
        "           title=\"Top-1000 words distribution\")\n",
        "p.quad(top=hist, bottom=0, left=edges[:-1], right=edges[1:], line_color=\"#555555\")\n",
        "show(p)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "\n",
              "    <div class=\"bk-root\">\n",
              "        <div class=\"bk-plotdiv\" id=\"7f32199b-ff01-4506-b2af-5fcd148edd67\"></div>\n",
              "    </div>\n",
              "<script type=\"text/javascript\">\n",
              "  \n",
              "  (function(root) {\n",
              "    function now() {\n",
              "      return new Date();\n",
              "    }\n",
              "  \n",
              "    var force = false;\n",
              "  \n",
              "    if (typeof (root._bokeh_onload_callbacks) === \"undefined\" || force === true) {\n",
              "      root._bokeh_onload_callbacks = [];\n",
              "      root._bokeh_is_loading = undefined;\n",
              "    }\n",
              "  \n",
              "  \n",
              "    \n",
              "    if (typeof (root._bokeh_timeout) === \"undefined\" || force === true) {\n",
              "      root._bokeh_timeout = Date.now() + 0;\n",
              "      root._bokeh_failed_load = false;\n",
              "    }\n",
              "  \n",
              "    var NB_LOAD_WARNING = {'data': {'text/html':\n",
              "       \"<div style='background-color: #fdd'>\\n\"+\n",
              "       \"<p>\\n\"+\n",
              "       \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n",
              "       \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n",
              "       \"</p>\\n\"+\n",
              "       \"<ul>\\n\"+\n",
              "       \"<li>re-rerun `output_notebook()` to attempt to load from CDN again, or</li>\\n\"+\n",
              "       \"<li>use INLINE resources instead, as so:</li>\\n\"+\n",
              "       \"</ul>\\n\"+\n",
              "       \"<code>\\n\"+\n",
              "       \"from bokeh.resources import INLINE\\n\"+\n",
              "       \"output_notebook(resources=INLINE)\\n\"+\n",
              "       \"</code>\\n\"+\n",
              "       \"</div>\"}};\n",
              "  \n",
              "    function display_loaded() {\n",
              "      if (root.Bokeh !== undefined) {\n",
              "        var el = document.getElementById(\"7f32199b-ff01-4506-b2af-5fcd148edd67\");\n",
              "        if (el != null) {\n",
              "          el.textContent = \"BokehJS \" + Bokeh.version + \" successfully loaded.\";\n",
              "        }\n",
              "      } else if (Date.now() < root._bokeh_timeout) {\n",
              "        setTimeout(display_loaded, 100)\n",
              "      }\n",
              "    }\n",
              "  \n",
              "  \n",
              "    function run_callbacks() {\n",
              "      try {\n",
              "        root._bokeh_onload_callbacks.forEach(function(callback) { callback() });\n",
              "      }\n",
              "      finally {\n",
              "        delete root._bokeh_onload_callbacks\n",
              "      }\n",
              "      console.info(\"Bokeh: all callbacks have finished\");\n",
              "    }\n",
              "  \n",
              "    function load_libs(js_urls, callback) {\n",
              "      root._bokeh_onload_callbacks.push(callback);\n",
              "      if (root._bokeh_is_loading > 0) {\n",
              "        console.log(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n",
              "        return null;\n",
              "      }\n",
              "      if (js_urls == null || js_urls.length === 0) {\n",
              "        run_callbacks();\n",
              "        return null;\n",
              "      }\n",
              "      console.log(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n",
              "      root._bokeh_is_loading = js_urls.length;\n",
              "      for (var i = 0; i < js_urls.length; i++) {\n",
              "        var url = js_urls[i];\n",
              "        var s = document.createElement('script');\n",
              "        s.src = url;\n",
              "        s.async = false;\n",
              "        s.onreadystatechange = s.onload = function() {\n",
              "          root._bokeh_is_loading--;\n",
              "          if (root._bokeh_is_loading === 0) {\n",
              "            console.log(\"Bokeh: all BokehJS libraries loaded\");\n",
              "            run_callbacks()\n",
              "          }\n",
              "        };\n",
              "        s.onerror = function() {\n",
              "          console.warn(\"failed to load library \" + url);\n",
              "        };\n",
              "        console.log(\"Bokeh: injecting script tag for BokehJS library: \", url);\n",
              "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
              "      }\n",
              "    };var element = document.getElementById(\"7f32199b-ff01-4506-b2af-5fcd148edd67\");\n",
              "    if (element == null) {\n",
              "      console.log(\"Bokeh: ERROR: autoload.js configured with elementid '7f32199b-ff01-4506-b2af-5fcd148edd67' but no matching script tag was found. \")\n",
              "      return false;\n",
              "    }\n",
              "  \n",
              "    var js_urls = [];\n",
              "  \n",
              "    var inline_js = [\n",
              "      function(Bokeh) {\n",
              "        (function() {\n",
              "          var fn = function() {\n",
              "            var docs_json = {\"7ca617d7-7d6e-4a7b-b97f-056da97badf7\":{\"roots\":{\"references\":[{\"attributes\":{},\"id\":\"39708965-8390-4a62-8dc7-51f555382ed3\",\"type\":\"SaveTool\"},{\"attributes\":{},\"id\":\"86b80ce2-d4b3-4a1f-8620-f922f5efce17\",\"type\":\"ResetTool\"},{\"attributes\":{},\"id\":\"bf6128f7-00d8-41c2-a353-e3d7fb1ff30b\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{\"callback\":null},\"id\":\"eef286d7-065c-4389-87db-1d0855ef573c\",\"type\":\"DataRange1d\"},{\"attributes\":{\"data_source\":{\"id\":\"2867c197-46ba-4c1a-ae06-9bc07d928374\",\"type\":\"ColumnDataSource\"},\"glyph\":{\"id\":\"05bd1788-8d96-4d89-bd92-904d39e6b98f\",\"type\":\"Quad\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"ab987e01-6866-49be-ac7c-90799ac18a8b\",\"type\":\"Quad\"},\"selection_glyph\":null,\"view\":{\"id\":\"f1cfbfff-c0e2-49dd-b91e-cf7f6ea72418\",\"type\":\"CDSView\"}},\"id\":\"00dfd0be-bd2d-4dde-a0d2-0750f59c350a\",\"type\":\"GlyphRenderer\"},{\"attributes\":{\"plot\":{\"id\":\"cc9bc963-7f4b-4ae6-abf2-8ef213366009\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"b4d3cef7-2783-4f77-9589-88ef9ba9a0a7\",\"type\":\"BasicTicker\"}},\"id\":\"11739d59-a759-4763-8c99-739acffc2e5d\",\"type\":\"Grid\"},{\"attributes\":{},\"id\":\"756cb35e-d04d-437e-9242-b08abeee1706\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{\"callback\":null,\"column_names\":[\"left\",\"right\",\"top\"],\"data\":{\"left\":{\"__ndarray__\":\"AAAAAABAeUB7FK5H4VqjQHsUrkfhxrFAuB6F61HguUB7FK5H4fzAQJqZmZmZCcVAuB6F61EWyUDXo3A9CiPNQHsUrkfhl9BACtejcD2e0kCamZmZmaTUQClcj8L1qtZAuB6F61Gx2EBI4XoUrrfaQNejcD0KvtxAZ2ZmZmbE3kB7FK5HYWXgQMP1KFyPaOFACtejcL1r4kBSuB6F627jQJqZmZkZcuRA4XoUrkd15UApXI/CdXjmQHE9Cteje+dAuB6F69F+6EAAAAAAAILpQEjhehQuhepAkML1KFyI60DXo3A9iovsQB+F61G4ju1AZ2ZmZuaR7kCuR+F6FJXvQHsUrkchTPBAH4XrUbjN8EDD9ShcT0/xQGdmZmbm0PFACtejcH1S8kCuR+F6FNTyQFK4HoWrVfNA9ihcj0LX80CamZmZ2Vj0QD4K16Nw2vRA4XoUrgdc9UCF61G4nt31QClcj8I1X/ZAzczMzMzg9kBxPQrXY2L3QBWuR+H64/dAuB6F65Fl+EBcj8L1KOf4QAAAAADAaPlApHA9Clfq+UBI4XoU7mv6QOxRuB6F7fpAkML1KBxv+0AzMzMzs/D7QNejcD1KcvxAexSuR+Hz/EAfhetReHX9QMP1KFwP9/1AZ2ZmZqZ4/kAK16NwPfr+QK5H4XrUe/9AUrgehWv9/0B7FK5HgT8AQc3MzMxMgABBH4XrURjBAEFxPQrX4wEBQcP1KFyvQgFBFa5H4XqDAUFnZmZmRsQBQbgehesRBQJBCtejcN1FAkFcj8L1qIYCQa5H4Xp0xwJBAAAAAEAIA0FSuB6FC0kDQaRwPQrXiQNB9ihcj6LKA0FI4XoUbgsEQZqZmZk5TARB7FG4HgWNBEE+Ctej0M0EQZDC9SicDgVB4XoUrmdPBUEzMzMzM5AFQYXrUbj+0AVB16NwPcoRBkEpXI/ClVIGQXsUrkdhkwZBzczMzCzUBkEfhetR+BQHQXE9CtfDVQdBw/UoXI+WB0EVrkfhWtcHQWdmZmYmGAhBuB6F6/FYCEEK16NwvZkIQVyPwvWI2ghBrkfhelQbCUE=\",\"dtype\":\"float64\",\"shape\":[100]},\"right\":{\"__ndarray__\":\"exSuR+Fao0B7FK5H4caxQLgehetR4LlAexSuR+H8wECamZmZmQnFQLgehetRFslA16NwPQojzUB7FK5H4ZfQQArXo3A9ntJAmpmZmZmk1EApXI/C9arWQLgehetRsdhASOF6FK632kDXo3A9Cr7cQGdmZmZmxN5AexSuR2Fl4EDD9Shcj2jhQArXo3C9a+JAUrgehetu40CamZmZGXLkQOF6FK5HdeVAKVyPwnV45kBxPQrXo3vnQLgehevRfuhAAAAAAACC6UBI4XoULoXqQJDC9ShciOtA16NwPYqL7EAfhetRuI7tQGdmZmbmke5ArkfhehSV70B7FK5HIUzwQB+F61G4zfBAw/UoXE9P8UBnZmZm5tDxQArXo3B9UvJArkfhehTU8kBSuB6Fq1XzQPYoXI9C1/NAmpmZmdlY9EA+CtejcNr0QOF6FK4HXPVAhetRuJ7d9UApXI/CNV/2QM3MzMzM4PZAcT0K12Ni90AVrkfh+uP3QLgeheuRZfhAXI/C9Sjn+EAAAAAAwGj5QKRwPQpX6vlASOF6FO5r+kDsUbgehe36QJDC9Sgcb/tAMzMzM7Pw+0DXo3A9SnL8QHsUrkfh8/xAH4XrUXh1/UDD9ShcD/f9QGdmZmameP5ACtejcD36/kCuR+F61Hv/QFK4HoVr/f9AexSuR4E/AEHNzMzMTIAAQR+F61EYwQBBcT0K1+MBAUHD9Shcr0IBQRWuR+F6gwFBZ2ZmZkbEAUG4HoXrEQUCQQrXo3DdRQJBXI/C9aiGAkGuR+F6dMcCQQAAAABACANBUrgehQtJA0GkcD0K14kDQfYoXI+iygNBSOF6FG4LBEGamZmZOUwEQexRuB4FjQRBPgrXo9DNBEGQwvUonA4FQeF6FK5nTwVBMzMzMzOQBUGF61G4/tAFQdejcD3KEQZBKVyPwpVSBkF7FK5HYZMGQc3MzMws1AZBH4XrUfgUB0FxPQrXw1UHQcP1KFyPlgdBFa5H4VrXB0FnZmZmJhgIQbgehevxWAhBCtejcL2ZCEFcj8L1iNoIQa5H4XpUGwlBAAAAACBcCUE=\",\"dtype\":\"float64\",\"shape\":[100]},\"top\":{\"__ndarray__\":\"50H1zNSBOT/ypb1tgHIBPwsriUb5Uew+zuTFiLI04j78kGUPxTHhPh40R25oQNY+zuTFiLI00j4pPQWW1y7APsTbB2FDRsg+JT0FltcuwD4tPQWW1y6wPi09BZbXLrA+JT0FltcuwD4tPQWW1y6gPiU9BZbXLrA+LT0FltcuoD4AAAAAAAAAADU9BZbXLqA+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADU9BZbXLqA+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAANT0FltcuoD4AAAAAAAAAAAAAAAAAAAAANT0FltcuoD4AAAAAAAAAACU9BZbXLqA+AAAAAAAAAAAlPQWW1y6gPkU9BZbXLqA+AAAAAAAAAAAlPQWW1y6gPgAAAAAAAAAAJT0FltcuoD4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJT0FltcuoD4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACU9BZbXLqA+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAlPQWW1y6gPgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAART0FltcuoD4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACU9BZbXLqA+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJT0FltcuoD4=\",\"dtype\":\"float64\",\"shape\":[100]}}},\"id\":\"2867c197-46ba-4c1a-ae06-9bc07d928374\",\"type\":\"ColumnDataSource\"},{\"attributes\":{},\"id\":\"b4d3cef7-2783-4f77-9589-88ef9ba9a0a7\",\"type\":\"BasicTicker\"},{\"attributes\":{},\"id\":\"456866bb-722a-4903-8442-5d11d3a75059\",\"type\":\"PanTool\"},{\"attributes\":{},\"id\":\"cd472512-1e73-4aad-b76e-5ae1ba9ceade\",\"type\":\"LinearScale\"},{\"attributes\":{},\"id\":\"46c2b6ec-7a08-498b-9e32-d73660496ba9\",\"type\":\"WheelZoomTool\"},{\"attributes\":{\"source\":{\"id\":\"2867c197-46ba-4c1a-ae06-9bc07d928374\",\"type\":\"ColumnDataSource\"}},\"id\":\"f1cfbfff-c0e2-49dd-b91e-cf7f6ea72418\",\"type\":\"CDSView\"},{\"attributes\":{\"bottom\":{\"value\":0},\"fill_color\":{\"value\":\"#1f77b4\"},\"left\":{\"field\":\"left\"},\"line_color\":{\"value\":\"#555555\"},\"right\":{\"field\":\"right\"},\"top\":{\"field\":\"top\"}},\"id\":\"05bd1788-8d96-4d89-bd92-904d39e6b98f\",\"type\":\"Quad\"},{\"attributes\":{},\"id\":\"b57c7a96-0265-47fb-bacc-468157f0dc47\",\"type\":\"LinearScale\"},{\"attributes\":{\"callback\":null},\"id\":\"eb55a82d-cebd-4217-bf8b-cadeff0c2964\",\"type\":\"DataRange1d\"},{\"attributes\":{\"dimension\":1,\"plot\":{\"id\":\"cc9bc963-7f4b-4ae6-abf2-8ef213366009\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"001986da-e474-418f-a15f-0f6b718e5bfc\",\"type\":\"BasicTicker\"}},\"id\":\"95f920d2-1948-4a45-a697-897f7b5e47e6\",\"type\":\"Grid\"},{\"attributes\":{\"active_drag\":\"auto\",\"active_inspect\":\"auto\",\"active_scroll\":\"auto\",\"active_tap\":\"auto\",\"tools\":[{\"id\":\"456866bb-722a-4903-8442-5d11d3a75059\",\"type\":\"PanTool\"},{\"id\":\"46c2b6ec-7a08-498b-9e32-d73660496ba9\",\"type\":\"WheelZoomTool\"},{\"id\":\"86b80ce2-d4b3-4a1f-8620-f922f5efce17\",\"type\":\"ResetTool\"},{\"id\":\"39708965-8390-4a62-8dc7-51f555382ed3\",\"type\":\"SaveTool\"}]},\"id\":\"8ad842d0-a391-4dad-8713-6ef33fd68d23\",\"type\":\"Toolbar\"},{\"attributes\":{},\"id\":\"001986da-e474-418f-a15f-0f6b718e5bfc\",\"type\":\"BasicTicker\"},{\"attributes\":{\"below\":[{\"id\":\"3b0bca9c-8d94-4cb0-b74b-b638308514f6\",\"type\":\"LinearAxis\"}],\"left\":[{\"id\":\"11e1119a-7c9b-4f25-96c3-ff5d119f49d4\",\"type\":\"LinearAxis\"}],\"renderers\":[{\"id\":\"3b0bca9c-8d94-4cb0-b74b-b638308514f6\",\"type\":\"LinearAxis\"},{\"id\":\"11739d59-a759-4763-8c99-739acffc2e5d\",\"type\":\"Grid\"},{\"id\":\"11e1119a-7c9b-4f25-96c3-ff5d119f49d4\",\"type\":\"LinearAxis\"},{\"id\":\"95f920d2-1948-4a45-a697-897f7b5e47e6\",\"type\":\"Grid\"},{\"id\":\"00dfd0be-bd2d-4dde-a0d2-0750f59c350a\",\"type\":\"GlyphRenderer\"}],\"title\":{\"id\":\"49ed9727-3afc-479c-8509-77fa5fb42940\",\"type\":\"Title\"},\"toolbar\":{\"id\":\"8ad842d0-a391-4dad-8713-6ef33fd68d23\",\"type\":\"Toolbar\"},\"toolbar_location\":\"above\",\"x_range\":{\"id\":\"eb55a82d-cebd-4217-bf8b-cadeff0c2964\",\"type\":\"DataRange1d\"},\"x_scale\":{\"id\":\"b57c7a96-0265-47fb-bacc-468157f0dc47\",\"type\":\"LinearScale\"},\"y_range\":{\"id\":\"eef286d7-065c-4389-87db-1d0855ef573c\",\"type\":\"DataRange1d\"},\"y_scale\":{\"id\":\"cd472512-1e73-4aad-b76e-5ae1ba9ceade\",\"type\":\"LinearScale\"}},\"id\":\"cc9bc963-7f4b-4ae6-abf2-8ef213366009\",\"subtype\":\"Figure\",\"type\":\"Plot\"},{\"attributes\":{\"formatter\":{\"id\":\"bf6128f7-00d8-41c2-a353-e3d7fb1ff30b\",\"type\":\"BasicTickFormatter\"},\"plot\":{\"id\":\"cc9bc963-7f4b-4ae6-abf2-8ef213366009\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"001986da-e474-418f-a15f-0f6b718e5bfc\",\"type\":\"BasicTicker\"}},\"id\":\"11e1119a-7c9b-4f25-96c3-ff5d119f49d4\",\"type\":\"LinearAxis\"},{\"attributes\":{\"bottom\":{\"value\":0},\"fill_alpha\":{\"value\":0.1},\"fill_color\":{\"value\":\"#1f77b4\"},\"left\":{\"field\":\"left\"},\"line_alpha\":{\"value\":0.1},\"line_color\":{\"value\":\"#1f77b4\"},\"right\":{\"field\":\"right\"},\"top\":{\"field\":\"top\"}},\"id\":\"ab987e01-6866-49be-ac7c-90799ac18a8b\",\"type\":\"Quad\"},{\"attributes\":{\"plot\":null,\"text\":\"Top-1000 words distribution\"},\"id\":\"49ed9727-3afc-479c-8509-77fa5fb42940\",\"type\":\"Title\"},{\"attributes\":{\"formatter\":{\"id\":\"756cb35e-d04d-437e-9242-b08abeee1706\",\"type\":\"BasicTickFormatter\"},\"plot\":{\"id\":\"cc9bc963-7f4b-4ae6-abf2-8ef213366009\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"b4d3cef7-2783-4f77-9589-88ef9ba9a0a7\",\"type\":\"BasicTicker\"}},\"id\":\"3b0bca9c-8d94-4cb0-b74b-b638308514f6\",\"type\":\"LinearAxis\"}],\"root_ids\":[\"cc9bc963-7f4b-4ae6-abf2-8ef213366009\"]},\"title\":\"Bokeh Application\",\"version\":\"0.12.7\"}};\n",
              "            var render_items = [{\"docid\":\"7ca617d7-7d6e-4a7b-b97f-056da97badf7\",\"elementid\":\"7f32199b-ff01-4506-b2af-5fcd148edd67\",\"modelid\":\"cc9bc963-7f4b-4ae6-abf2-8ef213366009\"}];\n",
              "            \n",
              "            Bokeh.embed.embed_items(docs_json, render_items);\n",
              "          };\n",
              "          if (document.readyState != \"loading\") fn();\n",
              "          else document.addEventListener(\"DOMContentLoaded\", fn);\n",
              "        })();\n",
              "      },\n",
              "      function(Bokeh) {\n",
              "      }\n",
              "    ];\n",
              "  \n",
              "    function run_inline_js() {\n",
              "      \n",
              "      if ((root.Bokeh !== undefined) || (force === true)) {\n",
              "        for (var i = 0; i < inline_js.length; i++) {\n",
              "          inline_js[i].call(root, root.Bokeh);\n",
              "        }if (force === true) {\n",
              "          display_loaded();\n",
              "        }} else if (Date.now() < root._bokeh_timeout) {\n",
              "        setTimeout(run_inline_js, 100);\n",
              "      } else if (!root._bokeh_failed_load) {\n",
              "        console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n",
              "        root._bokeh_failed_load = true;\n",
              "      } else if (force !== true) {\n",
              "        var cell = $(document.getElementById(\"7f32199b-ff01-4506-b2af-5fcd148edd67\")).parents('.cell').data().cell;\n",
              "        cell.output_area.append_execute_result(NB_LOAD_WARNING)\n",
              "      }\n",
              "  \n",
              "    }\n",
              "  \n",
              "    if (root._bokeh_is_loading === 0) {\n",
              "      console.log(\"Bokeh: BokehJS loaded, going straight to plotting\");\n",
              "      run_inline_js();\n",
              "    } else {\n",
              "      load_libs(js_urls, function() {\n",
              "        console.log(\"Bokeh: BokehJS plotting callback run at\", now());\n",
              "        run_inline_js();\n",
              "      });\n",
              "    }\n",
              "  }(window));\n",
              "</script>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "lv6XVM22kTId",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Part 3: Train Word2Vec"
      ]
    },
    {
      "metadata": {
        "id": "BGfOEUpHkTId",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Full documentation: https://radimrehurek.com/gensim/models/word2vec.html\n",
        "from gensim.models import Word2Vec"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "j8ejvFaqkTIh",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "model_ted = Word2Vec(sentences_ted, min_count=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "oaorFoE3kTIk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Part 4: Ted Learnt Representations"
      ]
    },
    {
      "metadata": {
        "id": "gJuVo4XWkTIl",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Finding similar words: (see gensim docs for more functionality of `most_similar`)"
      ]
    },
    {
      "metadata": {
        "id": "xkl9erlmkTIl",
        "colab_type": "code",
        "colab": {},
        "outputId": "08cebfa9-0322-4c3d-cfc4-c0301cc184e9"
      },
      "cell_type": "code",
      "source": [
        "model_ted.most_similar(\"man\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('woman', 0.8682868480682373),\n",
              " ('guy', 0.8158002495765686),\n",
              " ('lady', 0.7774063348770142),\n",
              " ('girl', 0.74993896484375),\n",
              " ('boy', 0.7380883693695068),\n",
              " ('soldier', 0.7181751728057861),\n",
              " ('gentleman', 0.7085753083229065),\n",
              " ('kid', 0.7070672512054443),\n",
              " ('philosopher', 0.6738376021385193),\n",
              " ('doctor', 0.6634228825569153)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "metadata": {
        "id": "_vUtUYddkTIq",
        "colab_type": "code",
        "colab": {},
        "outputId": "061970c5-a944-40a8-9e5e-aa0bce09c773"
      },
      "cell_type": "code",
      "source": [
        "model_ted.most_similar(\"computer\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('software', 0.7724950909614563),\n",
              " ('machine', 0.7566120028495789),\n",
              " ('robot', 0.6976919174194336),\n",
              " ('3d', 0.6860084533691406),\n",
              " ('device', 0.6844151616096497),\n",
              " ('game', 0.6713730096817017),\n",
              " ('camera', 0.6707618236541748),\n",
              " ('video', 0.6641008257865906),\n",
              " ('program', 0.6588221788406372),\n",
              " ('interface', 0.6544047594070435)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "metadata": {
        "id": "9tuDZ4igkTIx",
        "colab_type": "code",
        "colab": {},
        "outputId": "4ee7e144-9287-4d44-fe30-288fabdad595"
      },
      "cell_type": "code",
      "source": [
        "model_ted.most_similar(\"language\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('culture', 0.7175629734992981),\n",
              " ('mathematics', 0.7056708931922913),\n",
              " ('emotion', 0.6953589916229248),\n",
              " ('narrative', 0.6929461359977722),\n",
              " ('logic', 0.682327389717102),\n",
              " ('nature', 0.6549948453903198),\n",
              " ('behavior', 0.6513528227806091),\n",
              " ('english', 0.6415160894393921),\n",
              " ('meaning', 0.6356196999549866),\n",
              " ('beauty', 0.6317970752716064)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "metadata": {
        "id": "ZxMY-MjLkTI1",
        "colab_type": "code",
        "colab": {},
        "outputId": "a5e81579-cd60-4caf-97d6-97bed00ae0e7"
      },
      "cell_type": "code",
      "source": [
        "model_ted.wv.most_similar(positive=['nice', 'man'])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('guy', 0.7498212456703186),\n",
              " ('woman', 0.6990991234779358),\n",
              " ('boy', 0.6976661682128906),\n",
              " ('gentleman', 0.695278525352478),\n",
              " ('kid', 0.6902039051055908),\n",
              " ('lady', 0.6837806701660156),\n",
              " ('girl', 0.6695646047592163),\n",
              " ('dog', 0.6527926921844482),\n",
              " ('physicist', 0.6447546482086182),\n",
              " ('hoodie', 0.6394599676132202)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "metadata": {
        "id": "Icwg7AAdkTI6",
        "colab_type": "code",
        "colab": {},
        "outputId": "b01cd801-22e4-4a2a-ec15-681bce7b9d88"
      },
      "cell_type": "code",
      "source": [
        "model_ted.wv.most_similar(positive=['nice', 'woman'])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('girl', 0.71077561378479),\n",
              " ('boy', 0.7074770927429199),\n",
              " ('kid', 0.7037371397018433),\n",
              " ('gentleman', 0.6774724721908569),\n",
              " ('lady', 0.6747616529464722),\n",
              " ('poster', 0.67110276222229),\n",
              " ('guy', 0.665742039680481),\n",
              " ('baby', 0.6600582599639893),\n",
              " ('hoodie', 0.6593862175941467),\n",
              " ('man', 0.6558953523635864)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "metadata": {
        "id": "Esjuaq3jkTI9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### t-SNE visualization\n",
        "To use the t-SNE code below, first put a list of the top 1000 words (as strings) into a variable `words_top_ted`. The following code gets the corresponding vectors from the model, assuming it's called `model_ted`:"
      ]
    },
    {
      "metadata": {
        "id": "nXIGfs-QkTI-",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# This assumes words_top_ted is a list of strings, the top 1000 words\n",
        "words_top_vec_ted = model_ted[words_top_ted]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "AEvrRUxKkTJC",
        "colab_type": "code",
        "colab": {},
        "outputId": "0b76e15f-0a08-45b8-c49f-09ba5f351f85"
      },
      "cell_type": "code",
      "source": [
        "from sklearn.manifold import TSNE\n",
        "tsne = TSNE(n_components=2, random_state=0)\n",
        "words_top_ted_tsne = tsne.fit_transform(words_top_vec_ted)\n",
        "print (\"Model fit finished!\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model fit finished!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "WnZUy8BDkTJG",
        "colab_type": "code",
        "colab": {},
        "outputId": "3e31cd37-9558-46b4-a293-7c7c3c6ea376"
      },
      "cell_type": "code",
      "source": [
        "p = figure(tools=\"pan,wheel_zoom,reset,save\",\n",
        "           toolbar_location=\"above\",\n",
        "           title=\"word2vec T-SNE for most common words\")\n",
        "\n",
        "source = ColumnDataSource(data=dict(x1=words_top_ted_tsne[:,0],\n",
        "                                    x2=words_top_ted_tsne[:,1],\n",
        "                                    names=words_top_ted))\n",
        "\n",
        "p.scatter(x=\"x1\", y=\"x2\", size=8, source=source)\n",
        "\n",
        "labels = LabelSet(x=\"x1\", y=\"x2\", text=\"names\", y_offset=6,\n",
        "                  text_font_size=\"8pt\", text_color=\"#555555\",\n",
        "                  source=source, text_align='center')\n",
        "p.add_layout(labels)\n",
        "\n",
        "show(p)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "\n",
              "    <div class=\"bk-root\">\n",
              "        <div class=\"bk-plotdiv\" id=\"0e2218e8-4803-48fa-8546-744587c5b40b\"></div>\n",
              "    </div>\n",
              "<script type=\"text/javascript\">\n",
              "  \n",
              "  (function(root) {\n",
              "    function now() {\n",
              "      return new Date();\n",
              "    }\n",
              "  \n",
              "    var force = false;\n",
              "  \n",
              "    if (typeof (root._bokeh_onload_callbacks) === \"undefined\" || force === true) {\n",
              "      root._bokeh_onload_callbacks = [];\n",
              "      root._bokeh_is_loading = undefined;\n",
              "    }\n",
              "  \n",
              "  \n",
              "    \n",
              "    if (typeof (root._bokeh_timeout) === \"undefined\" || force === true) {\n",
              "      root._bokeh_timeout = Date.now() + 0;\n",
              "      root._bokeh_failed_load = false;\n",
              "    }\n",
              "  \n",
              "    var NB_LOAD_WARNING = {'data': {'text/html':\n",
              "       \"<div style='background-color: #fdd'>\\n\"+\n",
              "       \"<p>\\n\"+\n",
              "       \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n",
              "       \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n",
              "       \"</p>\\n\"+\n",
              "       \"<ul>\\n\"+\n",
              "       \"<li>re-rerun `output_notebook()` to attempt to load from CDN again, or</li>\\n\"+\n",
              "       \"<li>use INLINE resources instead, as so:</li>\\n\"+\n",
              "       \"</ul>\\n\"+\n",
              "       \"<code>\\n\"+\n",
              "       \"from bokeh.resources import INLINE\\n\"+\n",
              "       \"output_notebook(resources=INLINE)\\n\"+\n",
              "       \"</code>\\n\"+\n",
              "       \"</div>\"}};\n",
              "  \n",
              "    function display_loaded() {\n",
              "      if (root.Bokeh !== undefined) {\n",
              "        var el = document.getElementById(\"0e2218e8-4803-48fa-8546-744587c5b40b\");\n",
              "        if (el != null) {\n",
              "          el.textContent = \"BokehJS \" + Bokeh.version + \" successfully loaded.\";\n",
              "        }\n",
              "      } else if (Date.now() < root._bokeh_timeout) {\n",
              "        setTimeout(display_loaded, 100)\n",
              "      }\n",
              "    }\n",
              "  \n",
              "  \n",
              "    function run_callbacks() {\n",
              "      try {\n",
              "        root._bokeh_onload_callbacks.forEach(function(callback) { callback() });\n",
              "      }\n",
              "      finally {\n",
              "        delete root._bokeh_onload_callbacks\n",
              "      }\n",
              "      console.info(\"Bokeh: all callbacks have finished\");\n",
              "    }\n",
              "  \n",
              "    function load_libs(js_urls, callback) {\n",
              "      root._bokeh_onload_callbacks.push(callback);\n",
              "      if (root._bokeh_is_loading > 0) {\n",
              "        console.log(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n",
              "        return null;\n",
              "      }\n",
              "      if (js_urls == null || js_urls.length === 0) {\n",
              "        run_callbacks();\n",
              "        return null;\n",
              "      }\n",
              "      console.log(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n",
              "      root._bokeh_is_loading = js_urls.length;\n",
              "      for (var i = 0; i < js_urls.length; i++) {\n",
              "        var url = js_urls[i];\n",
              "        var s = document.createElement('script');\n",
              "        s.src = url;\n",
              "        s.async = false;\n",
              "        s.onreadystatechange = s.onload = function() {\n",
              "          root._bokeh_is_loading--;\n",
              "          if (root._bokeh_is_loading === 0) {\n",
              "            console.log(\"Bokeh: all BokehJS libraries loaded\");\n",
              "            run_callbacks()\n",
              "          }\n",
              "        };\n",
              "        s.onerror = function() {\n",
              "          console.warn(\"failed to load library \" + url);\n",
              "        };\n",
              "        console.log(\"Bokeh: injecting script tag for BokehJS library: \", url);\n",
              "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
              "      }\n",
              "    };var element = document.getElementById(\"0e2218e8-4803-48fa-8546-744587c5b40b\");\n",
              "    if (element == null) {\n",
              "      console.log(\"Bokeh: ERROR: autoload.js configured with elementid '0e2218e8-4803-48fa-8546-744587c5b40b' but no matching script tag was found. \")\n",
              "      return false;\n",
              "    }\n",
              "  \n",
              "    var js_urls = [];\n",
              "  \n",
              "    var inline_js = [\n",
              "      function(Bokeh) {\n",
              "        (function() {\n",
              "          var fn = function() {\n",
              "            var docs_json = {\"a8d65160-0d9c-4fd4-80d0-0672d5539895\":{\"roots\":{\"references\":[{\"attributes\":{\"callback\":null},\"id\":\"4ecc1b41-5d33-4b67-bd65-1e56942fe84a\",\"type\":\"DataRange1d\"},{\"attributes\":{},\"id\":\"91a3c21b-af9e-4054-aa7d-12a0b5269d50\",\"type\":\"BasicTicker\"},{\"attributes\":{},\"id\":\"091eddcd-05f7-497a-85c7-b2552fdce9a8\",\"type\":\"PanTool\"},{\"attributes\":{\"formatter\":{\"id\":\"204525e2-33ed-480f-8316-df7d601d4cfc\",\"type\":\"BasicTickFormatter\"},\"plot\":{\"id\":\"e9b9ef3f-d609-4074-a96b-d5ac76b5b9f5\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"768ba5c4-b527-4059-93e4-2ab696137bf6\",\"type\":\"BasicTicker\"}},\"id\":\"0e6ebc51-9a7d-41b7-8eff-c683f8a6bca2\",\"type\":\"LinearAxis\"},{\"attributes\":{\"active_drag\":\"auto\",\"active_inspect\":\"auto\",\"active_scroll\":\"auto\",\"active_tap\":\"auto\",\"tools\":[{\"id\":\"091eddcd-05f7-497a-85c7-b2552fdce9a8\",\"type\":\"PanTool\"},{\"id\":\"17aa2852-7089-4d73-a004-4d25cff4651d\",\"type\":\"WheelZoomTool\"},{\"id\":\"99478021-e3e7-4a26-9705-669613735e17\",\"type\":\"ResetTool\"},{\"id\":\"df5b139e-feb6-4cc9-a75c-d2c796f1b44a\",\"type\":\"SaveTool\"}]},\"id\":\"b219fc22-3d4c-4fbb-ac7e-eebfd1658da7\",\"type\":\"Toolbar\"},{\"attributes\":{\"fill_color\":{\"value\":\"#1f77b4\"},\"line_color\":{\"value\":\"#1f77b4\"},\"size\":{\"units\":\"screen\",\"value\":8},\"x\":{\"field\":\"x1\"},\"y\":{\"field\":\"x2\"}},\"id\":\"dbb89759-103b-4165-a033-b968693ccf15\",\"type\":\"Circle\"},{\"attributes\":{},\"id\":\"17aa2852-7089-4d73-a004-4d25cff4651d\",\"type\":\"WheelZoomTool\"},{\"attributes\":{\"plot\":{\"id\":\"e9b9ef3f-d609-4074-a96b-d5ac76b5b9f5\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"source\":{\"id\":\"6b82f790-c17c-4cae-aa56-351aa63cd296\",\"type\":\"ColumnDataSource\"},\"text\":{\"field\":\"names\"},\"text_align\":\"center\",\"text_color\":{\"value\":\"#555555\"},\"text_font_size\":{\"value\":\"8pt\"},\"x\":{\"field\":\"x1\"},\"y\":{\"field\":\"x2\"},\"y_offset\":{\"value\":6}},\"id\":\"a6c48224-5f1c-4a1c-8086-31285d4b01e6\",\"type\":\"LabelSet\"},{\"attributes\":{\"below\":[{\"id\":\"d54d621a-6057-417e-a416-30b373598d36\",\"type\":\"LinearAxis\"}],\"left\":[{\"id\":\"0e6ebc51-9a7d-41b7-8eff-c683f8a6bca2\",\"type\":\"LinearAxis\"}],\"renderers\":[{\"id\":\"d54d621a-6057-417e-a416-30b373598d36\",\"type\":\"LinearAxis\"},{\"id\":\"f224387f-f14b-444d-a583-5aeb7d6aa79a\",\"type\":\"Grid\"},{\"id\":\"0e6ebc51-9a7d-41b7-8eff-c683f8a6bca2\",\"type\":\"LinearAxis\"},{\"id\":\"6bbce135-40d9-4ae0-911a-1315e1427f99\",\"type\":\"Grid\"},{\"id\":\"44718454-e18b-4d31-ae92-2fb629bf9e43\",\"type\":\"GlyphRenderer\"},{\"id\":\"a6c48224-5f1c-4a1c-8086-31285d4b01e6\",\"type\":\"LabelSet\"}],\"title\":{\"id\":\"f69cee42-9c0b-4dc9-925c-f17df66bfbfa\",\"type\":\"Title\"},\"toolbar\":{\"id\":\"b219fc22-3d4c-4fbb-ac7e-eebfd1658da7\",\"type\":\"Toolbar\"},\"toolbar_location\":\"above\",\"x_range\":{\"id\":\"4ecc1b41-5d33-4b67-bd65-1e56942fe84a\",\"type\":\"DataRange1d\"},\"x_scale\":{\"id\":\"32a2659d-6a75-45c9-8505-7aba1ed8b816\",\"type\":\"LinearScale\"},\"y_range\":{\"id\":\"b69e5a4e-ebeb-4a3d-8676-c0c657ec3aff\",\"type\":\"DataRange1d\"},\"y_scale\":{\"id\":\"17144848-5937-4c83-abe7-838bdcbedf6a\",\"type\":\"LinearScale\"}},\"id\":\"e9b9ef3f-d609-4074-a96b-d5ac76b5b9f5\",\"subtype\":\"Figure\",\"type\":\"Plot\"},{\"attributes\":{},\"id\":\"99478021-e3e7-4a26-9705-669613735e17\",\"type\":\"ResetTool\"},{\"attributes\":{},\"id\":\"df5b139e-feb6-4cc9-a75c-d2c796f1b44a\",\"type\":\"SaveTool\"},{\"attributes\":{\"callback\":null,\"column_names\":[\"x1\",\"x2\",\"names\"],\"data\":{\"names\":[\"the\",\"and\",\"to\",\"of\",\"a\",\"that\",\"i\",\"in\",\"it\",\"you\",\"we\",\"is\",\"s\",\"this\",\"so\",\"they\",\"was\",\"for\",\"are\",\"have\",\"but\",\"what\",\"on\",\"with\",\"can\",\"t\",\"about\",\"there\",\"be\",\"as\",\"at\",\"all\",\"not\",\"do\",\"my\",\"one\",\"re\",\"people\",\"like\",\"if\",\"from\",\"now\",\"our\",\"he\",\"an\",\"just\",\"these\",\"or\",\"when\",\"because\",\"very\",\"me\",\"out\",\"by\",\"them\",\"how\",\"know\",\"up\",\"going\",\"had\",\"more\",\"think\",\"who\",\"were\",\"see\",\"your\",\"their\",\"which\",\"would\",\"here\",\"really\",\"get\",\"ve\",\"then\",\"m\",\"world\",\"us\",\"time\",\"some\",\"has\",\"don\",\"actually\",\"into\",\"way\",\"where\",\"will\",\"years\",\"things\",\"other\",\"no\",\"could\",\"go\",\"well\",\"want\",\"been\",\"make\",\"right\",\"she\",\"said\",\"something\",\"those\",\"first\",\"two\",\"than\",\"much\",\"also\",\"look\",\"new\",\"thing\",\"little\",\"got\",\"back\",\"over\",\"most\",\"say\",\"even\",\"his\",\"life\",\"only\",\"work\",\"many\",\"take\",\"need\",\"did\",\"lot\",\"kind\",\"why\",\"good\",\"around\",\"every\",\"different\",\"down\",\"ll\",\"let\",\"her\",\"through\",\"same\",\"being\",\"come\",\"d\",\"day\",\"year\",\"three\",\"use\",\"doing\",\"put\",\"called\",\"any\",\"today\",\"percent\",\"made\",\"after\",\"thank\",\"tell\",\"great\",\"human\",\"find\",\"didn\",\"fact\",\"talk\",\"change\",\"started\",\"another\",\"idea\",\"big\",\"last\",\"own\",\"before\",\"its\",\"never\",\"should\",\"better\",\"give\",\"thought\",\"went\",\"might\",\"important\",\"000\",\"again\",\"able\",\"together\",\"still\",\"problem\",\"off\",\"next\",\"part\",\"course\",\"system\",\"him\",\"does\",\"each\",\"start\",\"show\",\"long\",\"ago\",\"story\",\"came\",\"brain\",\"few\",\"bit\",\"between\",\"used\",\"place\",\"technology\",\"women\",\"too\",\"old\",\"mean\",\"data\",\"water\",\"looking\",\"question\",\"maybe\",\"found\",\"love\",\"doesn\",\"end\",\"example\",\"10\",\"done\",\"point\",\"four\",\"real\",\"wanted\",\"ever\",\"school\",\"understand\",\"sort\",\"live\",\"call\",\"whole\",\"children\",\"always\",\"trying\",\"may\",\"person\",\"away\",\"believe\",\"feel\",\"try\",\"million\",\"working\",\"help\",\"everything\",\"five\",\"country\",\"thinking\",\"second\",\"using\",\"information\",\"money\",\"means\",\"power\",\"took\",\"times\",\"high\",\"space\",\"number\",\"kids\",\"home\",\"become\",\"create\",\"small\",\"design\",\"making\",\"best\",\"left\",\"getting\",\"future\",\"enough\",\"man\",\"quite\",\"city\",\"without\",\"sense\",\"happened\",\"comes\",\"social\",\"probably\",\"less\",\"light\",\"talking\",\"energy\",\"am\",\"building\",\"science\",\"food\",\"body\",\"told\",\"interesting\",\"ask\",\"half\",\"pretty\",\"hard\",\"play\",\"anything\",\"lives\",\"countries\",\"coming\",\"such\",\"family\",\"stuff\",\"dollars\",\"earth\",\"moment\",\"20\",\"imagine\",\"across\",\"side\",\"while\",\"saw\",\"happen\",\"once\",\"okay\",\"build\",\"having\",\"men\",\"later\",\"experience\",\"makes\",\"asked\",\"living\",\"seen\",\"says\",\"room\",\"hand\",\"simple\",\"health\",\"ways\",\"else\",\"case\",\"yet\",\"almost\",\"young\",\"days\",\"nothing\",\"care\",\"bad\",\"happens\",\"goes\",\"move\",\"states\",\"reason\",\"computer\",\"open\",\"learn\",\"africa\",\"process\",\"inside\",\"someone\",\"six\",\"far\",\"mind\",\"project\",\"remember\",\"single\",\"picture\",\"both\",\"whether\",\"problems\",\"saying\",\"community\",\"basically\",\"already\",\"within\",\"looked\",\"myself\",\"billion\",\"often\",\"possible\",\"business\",\"planet\",\"global\",\"everybody\",\"top\",\"sure\",\"public\",\"set\",\"wrong\",\"car\",\"book\",\"keep\",\"yes\",\"answer\",\"oh\",\"hope\",\"sometimes\",\"true\",\"history\",\"war\",\"guy\",\"child\",\"instead\",\"months\",\"ideas\",\"looks\",\"matter\",\"government\",\"amazing\",\"united\",\"since\",\"bring\",\"age\",\"cells\",\"job\",\"heard\",\"until\",\"face\",\"wasn\",\"100\",\"read\",\"control\",\"isn\",\"research\",\"30\",\"words\",\"u\",\"group\",\"under\",\"self\",\"somebody\",\"built\",\"state\",\"woman\",\"turn\",\"friends\",\"beautiful\",\"line\",\"knew\",\"couple\",\"order\",\"form\",\"yeah\",\"internet\",\"middle\",\"music\",\"piece\",\"nature\",\"head\",\"though\",\"stop\",\"everyone\",\"50\",\"places\",\"video\",\"learned\",\"language\",\"run\",\"night\",\"decided\",\"study\",\"word\",\"cancer\",\"taking\",\"works\",\"became\",\"exactly\",\"species\",\"society\",\"completely\",\"education\",\"against\",\"stories\",\"large\",\"share\",\"level\",\"heart\",\"america\",\"model\",\"questions\",\"gets\",\"mother\",\"god\",\"company\",\"turns\",\"ourselves\",\"happening\",\"hear\",\"art\",\"themselves\",\"must\",\"itself\",\"rather\",\"kinds\",\"students\",\"name\",\"hours\",\"disease\",\"front\",\"house\",\"huge\",\"couldn\",\"created\",\"universe\",\"ok\",\"animals\",\"american\",\"environment\",\"worked\",\"minutes\",\"ones\",\"black\",\"perhaps\",\"1\",\"past\",\"third\",\"along\",\"others\",\"finally\",\"sound\",\"early\",\"game\",\"thousands\",\"century\",\"based\",\"least\",\"per\",\"ted\",\"lots\",\"figure\",\"free\",\"guys\",\"particular\",\"happy\",\"news\",\"learning\",\"entire\",\"won\",\"gave\",\"machine\",\"india\",\"during\",\"systems\",\"air\",\"difference\",\"outside\",\"natural\",\"taken\",\"seven\",\"given\",\"changed\",\"leave\",\"cell\",\"15\",\"close\",\"behind\",\"cities\",\"full\",\"scale\",\"difficult\",\"china\",\"takes\",\"companies\",\"yourself\",\"area\",\"reality\",\"seeing\",\"easy\",\"turned\",\"cost\",\"eyes\",\"team\",\"moving\",\"population\",\"culture\",\"york\",\"hands\",\"whatever\",\"began\",\"needs\",\"terms\",\"image\",\"simply\",\"needed\",\"beginning\",\"father\",\"local\",\"realized\",\"media\",\"death\",\"parents\",\"view\",\"walk\",\"white\",\"market\",\"parts\",\"ocean\",\"economic\",\"eight\",\"powerful\",\"known\",\"week\",\"size\",\"felt\",\"humans\",\"certain\",\"spend\",\"longer\",\"phone\",\"wonderful\",\"cannot\",\"grow\",\"common\",\"center\",\"tried\",\"fish\",\"land\",\"oil\",\"political\",\"interested\",\"deal\",\"red\",\"gone\",\"amount\",\"weeks\",\"spent\",\"opportunity\",\"paper\",\"lost\",\"national\",\"quickly\",\"blue\",\"step\",\"poor\",\"green\",\"buy\",\"growth\",\"wouldn\",\"either\",\"patients\",\"sitting\",\"ability\",\"changes\",\"write\",\"challenge\",\"south\",\"low\",\"growing\",\"friend\",\"field\",\"shows\",\"2\",\"born\",\"rest\",\"climate\",\"40\",\"test\",\"street\",\"surface\",\"incredible\",\"average\",\"physical\",\"morning\",\"scientists\",\"program\",\"pay\",\"value\",\"feeling\",\"hundreds\",\"girl\",\"met\",\"behavior\",\"economy\",\"dna\",\"complex\",\"access\",\"risk\",\"animal\",\"structure\",\"feet\",\"attention\",\"areas\",\"anyone\",\"deep\",\"watch\",\"short\",\"absolutely\",\"speak\",\"brought\",\"die\",\"bottom\",\"numbers\",\"audience\",\"stage\",\"realize\",\"law\",\"images\",\"wrote\",\"understanding\",\"knowledge\",\"literally\",\"books\",\"movement\",\"ground\",\"giving\",\"eat\",\"force\",\"seems\",\"alone\",\"telling\",\"starting\",\"hold\",\"nice\",\"forward\",\"sea\",\"developed\",\"individual\",\"kid\",\"millions\",\"support\",\"miles\",\"tools\",\"running\",\"online\",\"result\",\"act\",\"medical\",\"technologies\",\"north\",\"lab\",\"development\",\"blood\",\"fear\",\"nobody\",\"map\",\"personal\",\"issue\",\"voice\",\"material\",\"key\",\"12\",\"recently\",\"theory\",\"cut\",\"cars\",\"sun\",\"fast\",\"changing\",\"playing\",\"clear\",\"especially\",\"soon\",\"girls\",\"patient\",\"choice\",\"fly\",\"creating\",\"normal\",\"discovered\",\"europe\",\"talked\",\"relationship\",\"gives\",\"generation\",\"dark\",\"type\",\"showed\",\"chance\",\"seem\",\"industry\",\"asking\",\"rate\",\"fun\",\"issues\",\"color\",\"designed\",\"hour\",\"university\",\"innovation\",\"several\",\"class\",\"computers\",\"english\",\"tiny\",\"allow\",\"save\",\"focus\",\"digital\",\"solve\",\"developing\",\"network\",\"film\",\"special\",\"wall\",\"situation\",\"baby\",\"solution\",\"reasons\",\"box\",\"knows\",\"meet\",\"dead\",\"stand\",\"begin\",\"haven\",\"impact\",\"anybody\",\"term\",\"beyond\",\"pictures\",\"please\",\"ice\",\"produce\",\"shape\",\"non\",\"resources\",\"robot\",\"cool\",\"groups\",\"likely\",\"stay\",\"available\",\"truth\",\"rights\",\"major\",\"cause\",\"evidence\",\"obviously\",\"writing\",\"experiment\",\"modern\",\"drug\",\"becomes\",\"incredibly\",\"aren\",\"bigger\",\"guess\",\"nine\",\"product\",\"drugs\",\"lived\",\"involved\",\"perfect\",\"google\",\"month\",\"effect\",\"similar\",\"putting\",\"eye\",\"pick\",\"message\",\"quality\",\"violence\",\"web\",\"evolution\",\"security\",\"office\",\"basic\",\"solar\",\"indeed\",\"general\",\"towards\",\"present\",\"drive\",\"send\",\"listen\",\"worth\",\"revolution\",\"explain\",\"hundred\",\"certainly\",\"died\",\"ready\",\"teach\",\"journey\",\"25\",\"hit\",\"walking\",\"led\",\"approach\",\"international\",\"200\",\"scientific\",\"games\",\"60\",\"potential\",\"crazy\",\"communities\",\"chinese\",\"device\",\"source\",\"camera\",\"rules\",\"sex\",\"carbon\",\"sounds\",\"boy\",\"reach\",\"starts\",\"examples\",\"west\",\"3\",\"success\",\"measure\",\"particularly\",\"software\",\"totally\",\"suddenly\",\"action\",\"college\",\"higher\",\"code\",\"eventually\",\"democracy\",\"among\",\"minute\",\"sit\",\"largest\",\"showing\",\"hospital\",\"notice\",\"period\",\"mass\",\"develop\",\"dream\",\"onto\",\"plants\",\"break\",\"add\",\"memory\",\"everywhere\",\"favorite\",\"11\",\"zero\",\"speed\",\"schools\",\"moved\",\"response\",\"wants\",\"extremely\",\"movie\",\"individuals\",\"table\",\"wait\",\"medicine\",\"follow\",\"anyway\",\"trust\",\"plant\",\"biggest\",\"watching\",\"organization\",\"creative\",\"road\",\"choose\",\"worse\",\"lead\",\"results\",\"500\",\"exciting\",\"grew\",\"materials\",\"jobs\",\"plan\",\"east\",\"poverty\",\"positive\",\"safe\",\"strong\",\"fall\",\"essentially\",\"object\",\"door\",\"son\",\"vision\",\"further\",\"happiness\",\"continue\",\"student\",\"doctor\",\"including\",\"named\",\"role\",\"objects\",\"extraordinary\",\"standing\",\"skin\",\"leaders\",\"usually\",\"african\",\"conversation\",\"projects\",\"allowed\",\"models\",\"supposed\",\"faster\",\"interest\",\"families\",\"police\",\"fight\",\"tree\",\"connected\",\"screen\",\"5\",\"cases\",\"goal\",\"somewhere\",\"finding\",\"buildings\",\"shown\"],\"x1\":{\"__ndarray__\":\"[encoded data, 5336 characters]\",\"dtype\":\"float32\",\"shape\":[1000]},\"x2\":{\"__ndarray__\":\"[encoded data, 5336 characters]\",\"dtype\":\"float32\",\"shape\":[1000]}}},\"id\":\"6b82f790-c17c-4cae-aa56-351aa63cd296\",\"type\":\"ColumnDataSource\"},{\"attributes\":{},\"id\":\"4427ad57-11de-405f-b415-1ef3364125f7\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{\"source\":{\"id\":\"6b82f790-c17c-4cae-aa56-351aa63cd296\",\"type\":\"ColumnDataSource\"}},\"id\":\"63f14c7d-29ff-4d45-8597-89e2e8f6ec00\",\"type\":\"CDSView\"},{\"attributes\":{\"plot\":null,\"text\":\"word2vec T-SNE for most common words\"},\"id\":\"f69cee42-9c0b-4dc9-925c-f17df66bfbfa\",\"type\":\"Title\"},{\"attributes\":{},\"id\":\"204525e2-33ed-480f-8316-df7d601d4cfc\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{\"fill_alpha\":{\"value\":0.1},\"fill_color\":{\"value\":\"#1f77b4\"},\"line_alpha\":{\"value\":0.1},\"line_color\":{\"value\":\"#1f77b4\"},\"size\":{\"units\":\"screen\",\"value\":8},\"x\":{\"field\":\"x1\"},\"y\":{\"field\":\"x2\"}},\"id\":\"ced11694-98c8-433b-b5f7-e3bf5134ac41\",\"type\":\"Circle\"},{\"attributes\":{},\"id\":\"768ba5c4-b527-4059-93e4-2ab696137bf6\",\"type\":\"BasicTicker\"},{\"attributes\":{\"dimension\":1,\"plot\":{\"id\":\"e9b9ef3f-d609-4074-a96b-d5ac76b5b9f5\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"768ba5c4-b527-4059-93e4-2ab696137bf6\",\"type\":\"BasicTicker\"}},\"id\":\"6bbce135-40d9-4ae0-911a-1315e1427f99\",\"type\":\"Grid\"},{\"attributes\":{},\"id\":\"32a2659d-6a75-45c9-8505-7aba1ed8b816\",\"type\":\"LinearScale\"},{\"attributes\":{\"data_source\":{\"id\":\"6b82f790-c17c-4cae-aa56-351aa63cd296\",\"type\":\"ColumnDataSource\"},\"glyph\":{\"id\":\"dbb89759-103b-4165-a033-b968693ccf15\",\"type\":\"Circle\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"ced11694-98c8-433b-b5f7-e3bf5134ac41\",\"type\":\"Circle\"},\"selection_glyph\":null,\"view\":{\"id\":\"63f14c7d-29ff-4d45-8597-89e2e8f6ec00\",\"type\":\"CDSView\"}},\"id\":\"44718454-e18b-4d31-ae92-2fb629bf9e43\",\"type\":\"GlyphRenderer\"},{\"attributes\":{\"callback\":null},\"id\":\"b69e5a4e-ebeb-4a3d-8676-c0c657ec3aff\",\"type\":\"DataRange1d\"},{\"attributes\":{},\"id\":\"17144848-5937-4c83-abe7-838bdcbedf6a\",\"type\":\"LinearScale\"},{\"attributes\":{\"plot\":{\"id\":\"e9b9ef3f-d609-4074-a96b-d5ac76b5b9f5\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"91a3c21b-af9e-4054-aa7d-12a0b5269d50\",\"type\":\"BasicTicker\"}},\"id\":\"f224387f-f14b-444d-a583-5aeb7d6aa79a\",\"type\":\"Grid\"},{\"attributes\":{\"formatter\":{\"id\":\"4427ad57-11de-405f-b415-1ef3364125f7\",\"type\":\"BasicTickFormatter\"},\"plot\":{\"id\":\"e9b9ef3f-d609-4074-a96b-d5ac76b5b9f5\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"91a3c21b-af9e-4054-aa7d-12a0b5269d50\",\"type\":\"BasicTicker\"}},\"id\":\"d54d621a-6057-417e-a416-30b373598d36\",\"type\":\"LinearAxis\"}],\"root_ids\":[\"e9b9ef3f-d609-4074-a96b-d5ac76b5b9f5\"]},\"title\":\"Bokeh Application\",\"version\":\"0.12.7\"}};\n",
              "            var render_items = [{\"docid\":\"a8d65160-0d9c-4fd4-80d0-0672d5539895\",\"elementid\":\"0e2218e8-4803-48fa-8546-744587c5b40b\",\"modelid\":\"e9b9ef3f-d609-4074-a96b-d5ac76b5b9f5\"}];\n",
              "            \n",
              "            Bokeh.embed.embed_items(docs_json, render_items);\n",
              "          };\n",
              "          if (document.readyState != \"loading\") fn();\n",
              "          else document.addEventListener(\"DOMContentLoaded\", fn);\n",
              "        })();\n",
              "      },\n",
              "      function(Bokeh) {\n",
              "      }\n",
              "    ];\n",
              "  \n",
              "    function run_inline_js() {\n",
              "      \n",
              "      if ((root.Bokeh !== undefined) || (force === true)) {\n",
              "        for (var i = 0; i < inline_js.length; i++) {\n",
              "          inline_js[i].call(root, root.Bokeh);\n",
              "        }if (force === true) {\n",
              "          display_loaded();\n",
              "        }} else if (Date.now() < root._bokeh_timeout) {\n",
              "        setTimeout(run_inline_js, 100);\n",
              "      } else if (!root._bokeh_failed_load) {\n",
              "        console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n",
              "        root._bokeh_failed_load = true;\n",
              "      } else if (force !== true) {\n",
              "        var cell = $(document.getElementById(\"0e2218e8-4803-48fa-8546-744587c5b40b\")).parents('.cell').data().cell;\n",
              "        cell.output_area.append_execute_result(NB_LOAD_WARNING)\n",
              "      }\n",
              "  \n",
              "    }\n",
              "  \n",
              "    if (root._bokeh_is_loading === 0) {\n",
              "      console.log(\"Bokeh: BokehJS loaded, going straight to plotting\");\n",
              "      run_inline_js();\n",
              "    } else {\n",
              "      load_libs(js_urls, function() {\n",
              "        console.log(\"Bokeh: BokehJS plotting callback run at\", now());\n",
              "        run_inline_js();\n",
              "      });\n",
              "    }\n",
              "  }(window));\n",
              "</script>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "M-sq8TFakTJK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Part 5: Wiki Learnt Representations"
      ]
    },
    {
      "metadata": {
        "id": "pa3YwaNLkTJK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Download dataset"
      ]
    },
    {
      "metadata": {
        "id": "ou5u7wMVkTJL",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "if not os.path.isfile('wikitext-103-raw-v1.zip'):\n",
        "    urllib.request.urlretrieve(\"https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-raw-v1.zip\", filename=\"wikitext-103-raw-v1.zip\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "7PbPEZXokTJP",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "with zipfile.ZipFile('wikitext-103-raw-v1.zip', 'r') as z:\n",
        "    input_text = str(z.open('wikitext-103-raw/wiki.train.raw', 'r').read(), encoding='utf-8') # Thanks Robert Bastian"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "uZblURakkTJS",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Preprocess sentences (note that it's important to remove small sentences for performance)"
      ]
    },
    {
      "metadata": {
        "id": "Q7YfRaXykTJT",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "sentences_wiki = []\n",
        "for line in input_text.split('\\n'):\n",
        "    s = [x for x in line.split('.') if x and len(x.split()) >= 5]\n",
        "    sentences_wiki.extend(s)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ThkPTGkvkTJX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# sample 1/5 of the data\n",
        "shuffle(sentences_wiki)\n",
        "sentences_wiki = sentences_wiki[:int(len(sentences_wiki)/5)]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "U-gGO0VVkTJb",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#clean sentences from wiki\n",
        "for s_i in range(len(sentences_wiki)):\n",
        "    sentences_wiki[s_i] = re.sub(\"[^a-z]\", \" \", sentences_wiki[s_i].lower())\n",
        "    sentences_wiki[s_i] = re.sub(r'\\([^)]*\\)', '', sentences_wiki[s_i]).split()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4T4kvRRskTJg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now, repeat all the same steps that you performed above. You should be able to reuse essentially all the code."
      ]
    },
    {
      "metadata": {
        "id": "a8RJR1s1kTJh",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "model_wiki = Word2Vec(sentences_wiki, min_count=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "lhRHTHfMkTJk",
        "colab_type": "code",
        "colab": {},
        "outputId": "bc60db8b-b8ac-4208-b205-21df913f7f03"
      },
      "cell_type": "code",
      "source": [
        "model_wiki.most_similar('man')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('woman', 0.7413420677185059),\n",
              " ('dog', 0.6260066032409668),\n",
              " ('boy', 0.6195032596588135),\n",
              " ('girl', 0.6146693229675293),\n",
              " ('person', 0.6144819259643555),\n",
              " ('men', 0.5713915228843689),\n",
              " ('soldier', 0.5568812489509583),\n",
              " ('joke', 0.5567601919174194),\n",
              " ('creature', 0.5542395710945129),\n",
              " ('mask', 0.5220149755477905)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "metadata": {
        "id": "lYiGgm60kTJo",
        "colab_type": "code",
        "colab": {},
        "outputId": "af564f2c-ef20-4777-bee0-acb2db50c066"
      },
      "cell_type": "code",
      "source": [
        "# Calculate the most frequent words, store the results in words_top_wiki\n",
        "import collections\n",
        "wiki_counts = collections.defaultdict(int)\n",
        "words_top_wiki1000 = []\n",
        "words_top_wiki = []\n",
        "\n",
        "for sentence in sentences_wiki:\n",
        "    for word in sentence:\n",
        "        if wiki_counts.keys().__contains__(word):\n",
        "            wiki_counts[word] += 1\n",
        "        else:\n",
        "            wiki_counts[word] = 1\n",
        "#sort words in decenting order\n",
        "sorted_words_wiki = sorted(wiki_counts, key=wiki_counts.__getitem__, reverse=True)\n",
        "#get first 1000 counts from sorted words\n",
        "{words_top_wiki1000.append(wiki_counts[word]) for word in sorted_words_wiki[:1000]}\n",
        "#get first 1000 words from sorted words\n",
        "{words_top_wiki.append(word) for word in sorted_words_wiki[:1000]}\n",
        "print (\"Succesfully sorted!\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Succesfully sorted!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "6eUXtEMfkTJx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### t-SNE visualization"
      ]
    },
    {
      "metadata": {
        "id": "3XFxs0FYkTJx",
        "colab_type": "code",
        "colab": {},
        "collapsed": true
      },
      "cell_type": "code",
      "source": [
        "# This assumes words_top_wiki is a list of strings, the top 1000 words\n",
        "words_top_vec_wiki = model_wiki[words_top_wiki]\n",
        "\n",
        "tsne = TSNE(n_components=2, random_state=0)\n",
        "words_top_wiki_tsne = tsne.fit_transform(words_top_vec_wiki)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9UxH5t38kTJ2",
        "colab_type": "code",
        "colab": {},
        "outputId": "f9562936-e4aa-49bc-d7d6-ac9d49d5331c"
      },
      "cell_type": "code",
      "source": [
        "p = figure(tools=\"pan,wheel_zoom,reset,save\",\n",
        "           toolbar_location=\"above\",\n",
        "           title=\"word2vec T-SNE for most common words\")\n",
        "\n",
        "source = ColumnDataSource(data=dict(x1=words_top_wiki_tsne[:,0],\n",
        "                                    x2=words_top_wiki_tsne[:,1],\n",
        "                                    names=words_top_wiki))\n",
        "\n",
        "p.scatter(x=\"x1\", y=\"x2\", size=8, source=source)\n",
        "\n",
        "labels = LabelSet(x=\"x1\", y=\"x2\", text=\"names\", y_offset=6,\n",
        "                  text_font_size=\"8pt\", text_color=\"#555555\",\n",
        "                  source=source, text_align='center')\n",
        "p.add_layout(labels)\n",
        "\n",
        "show(p)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "\n",
              "    <div class=\"bk-root\">\n",
              "        <div class=\"bk-plotdiv\" id=\"d7d0cfc8-28a5-4058-8882-b3a5d7b3c25b\"></div>\n",
              "    </div>\n",
              "<script type=\"text/javascript\">\n",
              "  \n",
              "  (function(root) {\n",
              "    function now() {\n",
              "      return new Date();\n",
              "    }\n",
              "  \n",
              "    var force = false;\n",
              "  \n",
              "    if (typeof (root._bokeh_onload_callbacks) === \"undefined\" || force === true) {\n",
              "      root._bokeh_onload_callbacks = [];\n",
              "      root._bokeh_is_loading = undefined;\n",
              "    }\n",
              "  \n",
              "  \n",
              "    \n",
              "    if (typeof (root._bokeh_timeout) === \"undefined\" || force === true) {\n",
              "      root._bokeh_timeout = Date.now() + 0;\n",
              "      root._bokeh_failed_load = false;\n",
              "    }\n",
              "  \n",
              "    var NB_LOAD_WARNING = {'data': {'text/html':\n",
              "       \"<div style='background-color: #fdd'>\\n\"+\n",
              "       \"<p>\\n\"+\n",
              "       \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n",
              "       \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n",
              "       \"</p>\\n\"+\n",
              "       \"<ul>\\n\"+\n",
              "       \"<li>re-rerun `output_notebook()` to attempt to load from CDN again, or</li>\\n\"+\n",
              "       \"<li>use INLINE resources instead, as so:</li>\\n\"+\n",
              "       \"</ul>\\n\"+\n",
              "       \"<code>\\n\"+\n",
              "       \"from bokeh.resources import INLINE\\n\"+\n",
              "       \"output_notebook(resources=INLINE)\\n\"+\n",
              "       \"</code>\\n\"+\n",
              "       \"</div>\"}};\n",
              "  \n",
              "    function display_loaded() {\n",
              "      if (root.Bokeh !== undefined) {\n",
              "        var el = document.getElementById(\"d7d0cfc8-28a5-4058-8882-b3a5d7b3c25b\");\n",
              "        if (el != null) {\n",
              "          el.textContent = \"BokehJS \" + Bokeh.version + \" successfully loaded.\";\n",
              "        }\n",
              "      } else if (Date.now() < root._bokeh_timeout) {\n",
              "        setTimeout(display_loaded, 100)\n",
              "      }\n",
              "    }\n",
              "  \n",
              "  \n",
              "    function run_callbacks() {\n",
              "      try {\n",
              "        root._bokeh_onload_callbacks.forEach(function(callback) { callback() });\n",
              "      }\n",
              "      finally {\n",
              "        delete root._bokeh_onload_callbacks\n",
              "      }\n",
              "      console.info(\"Bokeh: all callbacks have finished\");\n",
              "    }\n",
              "  \n",
              "    function load_libs(js_urls, callback) {\n",
              "      root._bokeh_onload_callbacks.push(callback);\n",
              "      if (root._bokeh_is_loading > 0) {\n",
              "        console.log(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n",
              "        return null;\n",
              "      }\n",
              "      if (js_urls == null || js_urls.length === 0) {\n",
              "        run_callbacks();\n",
              "        return null;\n",
              "      }\n",
              "      console.log(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n",
              "      root._bokeh_is_loading = js_urls.length;\n",
              "      for (var i = 0; i < js_urls.length; i++) {\n",
              "        var url = js_urls[i];\n",
              "        var s = document.createElement('script');\n",
              "        s.src = url;\n",
              "        s.async = false;\n",
              "        s.onreadystatechange = s.onload = function() {\n",
              "          root._bokeh_is_loading--;\n",
              "          if (root._bokeh_is_loading === 0) {\n",
              "            console.log(\"Bokeh: all BokehJS libraries loaded\");\n",
              "            run_callbacks()\n",
              "          }\n",
              "        };\n",
              "        s.onerror = function() {\n",
              "          console.warn(\"failed to load library \" + url);\n",
              "        };\n",
              "        console.log(\"Bokeh: injecting script tag for BokehJS library: \", url);\n",
              "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
              "      }\n",
              "    };var element = document.getElementById(\"d7d0cfc8-28a5-4058-8882-b3a5d7b3c25b\");\n",
              "    if (element == null) {\n",
              "      console.log(\"Bokeh: ERROR: autoload.js configured with elementid 'd7d0cfc8-28a5-4058-8882-b3a5d7b3c25b' but no matching script tag was found. \")\n",
              "      return false;\n",
              "    }\n",
              "  \n",
              "    var js_urls = [];\n",
              "  \n",
              "    var inline_js = [\n",
              "      function(Bokeh) {\n",
              "        (function() {\n",
              "          var fn = function() {\n",
              "            var docs_json = {\"7d37c85a-c75a-450c-9c4e-4e6457026ff1\":{\"roots\":{\"references\":[{\"attributes\":{\"plot\":null,\"text\":\"word2vec T-SNE for most common words\"},\"id\":\"5ef935d3-3caa-4702-867b-09a4b452178b\",\"type\":\"Title\"},{\"attributes\":{\"formatter\":{\"id\":\"fbba4a8c-bceb-4349-9ec3-b0ecabfc0123\",\"type\":\"BasicTickFormatter\"},\"plot\":{\"id\":\"da612c08-2c8d-4640-9802-d363e04ad3a0\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"8e1a61c3-f6b6-4746-8a7c-b9ba3e38f86e\",\"type\":\"BasicTicker\"}},\"id\":\"b6a8bf72-41e9-4a6a-a98b-87e43d429642\",\"type\":\"LinearAxis\"},{\"attributes\":{},\"id\":\"6a1bffb1-ff25-469c-b9b8-6a4b7d3e7d6e\",\"type\":\"ResetTool\"},{\"attributes\":{},\"id\":\"8e1a61c3-f6b6-4746-8a7c-b9ba3e38f86e\",\"type\":\"BasicTicker\"},{\"attributes\":{},\"id\":\"93b18cff-0e29-49f1-89b5-817879a956a3\",\"type\":\"SaveTool\"},{\"attributes\":{\"callback\":null},\"id\":\"d0432a82-8a32-4274-b772-417fc6c43df8\",\"type\":\"DataRange1d\"},{\"attributes\":{\"source\":{\"id\":\"b6ba8617-d644-4f27-b21c-3255be54ea65\",\"type\":\"ColumnDataSource\"}},\"id\":\"45b61e09-c89f-4fa2-bd9d-c4ee025fb32e\",\"type\":\"CDSView\"},{\"attributes\":{\"active_drag\":\"auto\",\"active_inspect\":\"auto\",\"active_scroll\":\"auto\",\"active_tap\":\"auto\",\"tools\":[{\"id\":\"10497a06-3cc6-436b-aa56-3985e1de7f29\",\"type\":\"PanTool\"},{\"id\":\"01dcc880-e462-4d33-968d-be7cef6385f9\",\"type\":\"WheelZoomTool\"},{\"id\":\"6a1bffb1-ff25-469c-b9b8-6a4b7d3e7d6e\",\"type\":\"ResetTool\"},{\"id\":\"93b18cff-0e29-49f1-89b5-817879a956a3\",\"type\":\"SaveTool\"}]},\"id\":\"0426bfae-53ce-4ff5-9408-0e77f8920d17\",\"type\":\"Toolbar\"},{\"attributes\":{},\"id\":\"b2792fee-447a-48de-ada0-5ec96a78383f\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{},\"id\":\"fbba4a8c-bceb-4349-9ec3-b0ecabfc0123\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{},\"id\":\"68f393e7-103f-4af7-b66c-8263e0778e23\",\"type\":\"LinearScale\"},{\"attributes\":{\"callback\":null},\"id\":\"abfa9ffe-b7e1-47f7-a787-d023b9567672\",\"type\":\"DataRange1d\"},{\"attributes\":{},\"id\":\"0fa9f3b6-9b36-44be-a8a5-357befab0c42\",\"type\":\"LinearScale\"},{\"attributes\":{\"plot\":{\"id\":\"da612c08-2c8d-4640-9802-d363e04ad3a0\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"8e1a61c3-f6b6-4746-8a7c-b9ba3e38f86e\",\"type\":\"BasicTicker\"}},\"id\":\"6fa64afc-9061-4d24-8bd3-93a02605a891\",\"type\":\"Grid\"},{\"attributes\":{\"formatter\":{\"id\":\"b2792fee-447a-48de-ada0-5ec96a78383f\",\"type\":\"BasicTickFormatter\"},\"plot\":{\"id\":\"da612c08-2c8d-4640-9802-d363e04ad3a0\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"f94b8193-9ad2-413e-b740-8d272b33972b\",\"type\":\"BasicTicker\"}},\"id\":\"e11b35dc-a37a-46f6-9cee-2c83a438b02a\",\"type\":\"LinearAxis\"},{\"attributes\":{},\"id\":\"f94b8193-9ad2-413e-b740-8d272b33972b\",\"type\":\"BasicTicker\"},{\"attributes\":{\"dimension\":1,\"plot\":{\"id\":\"da612c08-2c8d-4640-9802-d363e04ad3a0\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"f94b8193-9ad2-413e-b740-8d272b33972b\",\"type\":\"BasicTicker\"}},\"id\":\"4d2083e6-dbd0-46da-9657-bad0ae398c89\",\"type\":\"Grid\"},{\"attributes\":{\"data_source\":{\"id\":\"b6ba8617-d644-4f27-b21c-3255be54ea65\",\"type\":\"ColumnDataSource\"},\"glyph\":{\"id\":\"e5cce034-1aa9-407d-aced-d079115beca4\",\"type\":\"Circle\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"759d0b61-62ab-40d7-a602-f544935673cd\",\"type\":\"Circle\"},\"selection_glyph\":null,\"view\":{\"id\":\"45b61e09-c89f-4fa2-bd9d-c4ee025fb32e\",\"type\":\"CDSView\"}},\"id\":\"aad7c2b4-4ad0-43a7-8c59-a90ec466dc58\",\"type\":\"GlyphRenderer\"},{\"attributes\":{\"plot\":{\"id\":\"da612c08-2c8d-4640-9802-d363e04ad3a0\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"source\":{\"id\":\"b6ba8617-d644-4f27-b21c-3255be54ea65\",\"type\":\"ColumnDataSource\"},\"text\":{\"field\":\"names\"},\"text_align\":\"center\",\"text_color\":{\"value\":\"#555555\"},\"text_font_size\":{\"value\":\"8pt\"},\"x\":{\"field\":\"x1\"},\"y\":{\"field\":\"x2\"},\"y_offset\":{\"value\":6}},\"id\":\"f1a08680-0632-4bff-b720-4d2bce424c78\",\"type\":\"LabelSet\"},{\"attributes\":{\"fill_alpha\":{\"value\":0.1},\"fill_color\":{\"value\":\"#1f77b4\"},\"line_alpha\":{\"value\":0.1},\"line_color\":{\"value\":\"#1f77b4\"},\"size\":{\"units\":\"screen\",\"value\":8},\"x\":{\"field\":\"x1\"},\"y\":{\"field\":\"x2\"}},\"id\":\"759d0b61-62ab-40d7-a602-f544935673cd\",\"type\":\"Circle\"},{\"attributes\":{\"below\":[{\"id\":\"b6a8bf72-41e9-4a6a-a98b-87e43d429642\",\"type\":\"LinearAxis\"}],\"left\":[{\"id\":\"e11b35dc-a37a-46f6-9cee-2c83a438b02a\",\"type\":\"LinearAxis\"}],\"renderers\":[{\"id\":\"b6a8bf72-41e9-4a6a-a98b-87e43d429642\",\"type\":\"LinearAxis\"},{\"id\":\"6fa64afc-9061-4d24-8bd3-93a02605a891\",\"type\":\"Grid\"},{\"id\":\"e11b35dc-a37a-46f6-9cee-2c83a438b02a\",\"type\":\"LinearAxis\"},{\"id\":\"4d2083e6-dbd0-46da-9657-bad0ae398c89\",\"type\":\"Grid\"},{\"id\":\"aad7c2b4-4ad0-43a7-8c59-a90ec466dc58\",\"type\":\"GlyphRenderer\"},{\"id\":\"f1a08680-0632-4bff-b720-4d2bce424c78\",\"type\":\"LabelSet\"}],\"title\":{\"id\":\"5ef935d3-3caa-4702-867b-09a4b452178b\",\"type\":\"Title\"},\"toolbar\":{\"id\":\"0426bfae-53ce-4ff5-9408-0e77f8920d17\",\"type\":\"Toolbar\"},\"toolbar_location\":\"above\",\"x_range\":{\"id\":\"d0432a82-8a32-4274-b772-417fc6c43df8\",\"type\":\"DataRange1d\"},\"x_scale\":{\"id\":\"68f393e7-103f-4af7-b66c-8263e0778e23\",\"type\":\"LinearScale\"},\"y_range\":{\"id\":\"abfa9ffe-b7e1-47f7-a787-d023b9567672\",\"type\":\"DataRange1d\"},\"y_scale\":{\"id\":\"0fa9f3b6-9b36-44be-a8a5-357befab0c42\",\"type\":\"LinearScale\"}},\"id\":\"da612c08-2c8d-4640-9802-d363e04ad3a0\",\"subtype\":\"Figure\",\"type\":\"Plot\"},{\"attributes\":{},\"id\":\"10497a06-3cc6-436b-aa56-3985e1de7f29\",\"type\":\"PanTool\"},{\"attributes\":{\"callback\":null,\"column_names\":[\"x1\",\"x2\",\"names\"],\"data\":{\"names\":[\"the\",\"of\",\"and\",\"in\",\"to\",\"a\",\"was\",\"s\",\"on\",\"as\",\"for\",\"that\",\"with\",\"by\",\"is\",\"his\",\"at\",\"he\",\"from\",\"it\",\"were\",\"an\",\"had\",\"which\",\"be\",\"this\",\"but\",\"first\",\"are\",\"not\",\"their\",\"after\",\"one\",\"her\",\"also\",\"its\",\"they\",\"two\",\"or\",\"have\",\"who\",\"new\",\"been\",\"has\",\"she\",\"when\",\"time\",\"during\",\"other\",\"would\",\"into\",\"all\",\"i\",\"more\",\"while\",\"over\",\"game\",\"him\",\"only\",\"later\",\"three\",\"out\",\"most\",\"up\",\"about\",\"th\",\"between\",\"than\",\"there\",\"some\",\"film\",\"may\",\"before\",\"year\",\"made\",\"no\",\"such\",\"second\",\"season\",\"where\",\"years\",\"war\",\"world\",\"both\",\"then\",\"used\",\"however\",\"through\",\"being\",\"song\",\"them\",\"these\",\"against\",\"series\",\"m\",\"many\",\"album\",\"number\",\"team\",\"several\",\"city\",\"north\",\"can\",\"including\",\"part\",\"united\",\"well\",\"became\",\"early\",\"four\",\"under\",\"state\",\"south\",\"music\",\"day\",\"so\",\"episode\",\"said\",\"following\",\"states\",\"known\",\"american\",\"could\",\"although\",\"did\",\"work\",\"began\",\"until\",\"like\",\"called\",\"because\",\"released\",\"t\",\"high\",\"people\",\"end\",\"british\",\"since\",\"each\",\"around\",\"long\",\"national\",\"best\",\"if\",\"west\",\"september\",\"life\",\"same\",\"million\",\"found\",\"us\",\"back\",\"five\",\"along\",\"another\",\"show\",\"took\",\"any\",\"use\",\"area\",\"you\",\"final\",\"received\",\"km\",\"group\",\"century\",\"august\",\"off\",\"october\",\"what\",\"government\",\"school\",\"will\",\"line\",\"due\",\"system\",\"june\",\"john\",\"november\",\"single\",\"east\",\"home\",\"july\",\"march\",\"third\",\"games\",\"large\",\"april\",\"much\",\"general\",\"set\",\"down\",\"york\",\"left\",\"family\",\"based\",\"major\",\"just\",\"included\",\"name\",\"history\",\"house\",\"road\",\"place\",\"character\",\"december\",\"we\",\"company\",\"army\",\"king\",\"band\",\"played\",\"six\",\"wrote\",\"now\",\"january\",\"won\",\"men\",\"according\",\"river\",\"described\",\"own\",\"times\",\"within\",\"death\",\"last\",\"those\",\"main\",\"near\",\"next\",\"air\",\"d\",\"video\",\"de\",\"named\",\"record\",\"late\",\"led\",\"battle\",\"ship\",\"release\",\"league\",\"make\",\"st\",\"still\",\"way\",\"club\",\"very\",\"original\",\"man\",\"great\",\"even\",\"production\",\"u\",\"top\",\"small\",\"force\",\"species\",\"public\",\"storm\",\"though\",\"old\",\"february\",\"days\",\"again\",\"route\",\"held\",\"book\",\"members\",\"division\",\"further\",\"university\",\"among\",\"do\",\"german\",\"built\",\"side\",\"service\",\"french\",\"play\",\"b\",\"role\",\"match\",\"having\",\"player\",\"point\",\"story\",\"power\",\"water\",\"career\",\"white\",\"often\",\"h\",\"order\",\"continued\",\"despite\",\"black\",\"without\",\"town\",\"local\",\"former\",\"park\",\"considered\",\"party\",\"came\",\"take\",\"songs\",\"half\",\"become\",\"form\",\"county\",\"support\",\"development\",\"england\",\"island\",\"period\",\"different\",\"military\",\"few\",\"written\",\"moved\",\"version\",\"c\",\"court\",\"london\",\"building\",\"given\",\"international\",\"lost\",\"english\",\"forces\",\"president\",\"tropical\",\"re\",\"my\",\"country\",\"little\",\"tour\",\"gave\",\"never\",\"street\",\"lead\",\"published\",\"ii\",\"attack\",\"performance\",\"returned\",\"royal\",\"similar\",\"n\",\"ships\",\"using\",\"went\",\"aircraft\",\"position\",\"short\",\"church\",\"produced\",\"track\",\"women\",\"children\",\"fire\",\"total\",\"television\",\"class\",\"recorded\",\"western\",\"land\",\"live\",\"highway\",\"how\",\"per\",\"stated\",\"making\",\"week\",\"right\",\"throughout\",\"run\",\"control\",\"father\",\"include\",\"characters\",\"good\",\"australia\",\"star\",\"upon\",\"head\",\"once\",\"love\",\"players\",\"less\",\"me\",\"himself\",\"ten\",\"design\",\"seven\",\"writing\",\"age\",\"central\",\"light\",\"remained\",\"previous\",\"instead\",\"return\",\"young\",\"away\",\"ft\",\"political\",\"developed\",\"law\",\"months\",\"result\",\"title\",\"rock\",\"james\",\"sea\",\"race\",\"r\",\"center\",\"red\",\"station\",\"college\",\"died\",\"eight\",\"night\",\"across\",\"southern\",\"reached\",\"hurricane\",\"too\",\"e\",\"felt\",\"america\",\"win\",\"low\",\"noted\",\"field\",\"reported\",\"various\",\"created\",\"style\",\"office\",\"areas\",\"william\",\"act\",\"japanese\",\"announced\",\"award\",\"eventually\",\"performed\",\"should\",\"construction\",\"miles\",\"son\",\"seen\",\"together\",\"level\",\"served\",\"member\",\"killed\",\"full\",\"followed\",\"northern\",\"sent\",\"common\",\"range\",\"chart\",\"campaign\",\"able\",\"action\",\"taken\",\"mm\",\"modern\",\"points\",\"works\",\"george\",\"playing\",\"body\",\"originally\",\"thought\",\"special\",\"stage\",\"front\",\"others\",\"success\",\"located\",\"born\",\"established\",\"appeared\",\"features\",\"bridge\",\"critics\",\"formed\",\"post\",\"rather\",\"get\",\"started\",\"caused\",\"australian\",\"project\",\"site\",\"addition\",\"heavy\",\"behind\",\"present\",\"football\",\"added\",\"decided\",\"every\",\"championship\",\"guns\",\"fleet\",\"feet\",\"go\",\"popular\",\"fourth\",\"f\",\"eastern\",\"almost\",\"sold\",\"saw\",\"close\",\"soon\",\"open\",\"kingdom\",\"awards\",\"director\",\"see\",\"command\",\"damage\",\"case\",\"completed\",\"o\",\"worked\",\"france\",\"strong\",\"david\",\"free\",\"opened\",\"population\",\"important\",\"non\",\"hall\",\"art\",\"initially\",\"weeks\",\"recording\",\"leading\",\"working\",\"ground\",\"coast\",\"mother\",\"region\",\"training\",\"scene\",\"l\",\"sound\",\"michael\",\"towards\",\"wanted\",\"mi\",\"event\",\"average\",\"generally\",\"ever\",\"navy\",\"allowed\",\"least\",\"wife\",\"scored\",\"featured\",\"union\",\"cup\",\"brought\",\"records\",\"help\",\"provided\",\"does\",\"henry\",\"washington\",\"runs\",\"council\",\"either\",\"victory\",\"put\",\"placed\",\"events\",\"magazine\",\"studio\",\"example\",\"summer\",\"troops\",\"mid\",\"opening\",\"joined\",\"hit\",\"co\",\"hours\",\"europe\",\"far\",\"minutes\",\"media\",\"european\",\"start\",\"society\",\"month\",\"list\",\"designed\",\"significant\",\"met\",\"possible\",\"involved\",\"operation\",\"x\",\"squadron\",\"believed\",\"police\",\"japan\",\"replaced\",\"forced\",\"robert\",\"outside\",\"base\",\"finished\",\"human\",\"saying\",\"particularly\",\"big\",\"al\",\"taking\",\"earlier\",\"radio\",\"films\",\"official\",\"turned\",\"test\",\"nine\",\"process\",\"mph\",\"review\",\"rest\",\"brown\",\"association\",\"crew\",\"beginning\",\"largest\",\"successful\",\"shot\",\"better\",\"social\",\"research\",\"highest\",\"change\",\"real\",\"stone\",\"goal\",\"available\",\"increased\",\"ended\",\"chief\",\"section\",\"appearance\",\"must\",\"praised\",\"come\",\"captain\",\"community\",\"going\",\"future\",\"canada\",\"reviews\",\"move\",\"lower\",\"staff\",\"business\",\"middle\",\"thus\",\"thomas\",\"relationship\",\"enough\",\"san\",\"signed\",\"previously\",\"cast\",\"positive\",\"gold\",\"above\",\"units\",\"additional\",\"castle\",\"charles\",\"space\",\"hill\",\"g\",\"winds\",\"cross\",\"rights\",\"passed\",\"parts\",\"find\",\"directed\",\"california\",\"elements\",\"female\",\"lake\",\"interest\",\"attempt\",\"required\",\"program\",\"ordered\",\"itself\",\"spent\",\"give\",\"infantry\",\"don\",\"board\",\"failed\",\"critical\",\"plan\",\"told\",\"brother\",\"whose\",\"round\",\"port\",\"plot\",\"news\",\"paul\",\"evidence\",\"committee\",\"groups\",\"musical\",\"grand\",\"district\",\"prior\",\"claimed\",\"turn\",\"sometimes\",\"india\",\"overall\",\"election\",\"hand\",\"type\",\"entire\",\"gun\",\"term\",\"introduced\",\"asked\",\"reception\",\"issue\",\"novel\",\"personal\",\"money\",\"usually\",\"lines\",\"students\",\"services\",\"our\",\"green\",\"complete\",\"person\",\"already\",\"governor\",\"v\",\"square\",\"woman\",\"cover\",\"anti\",\"germany\",\"effects\",\"room\",\"press\",\"p\",\"background\",\"uk\",\"study\",\"education\",\"past\",\"idea\",\"surface\",\"flight\",\"movement\",\"whom\",\"bill\",\"k\",\"structure\",\"stars\",\"village\",\"estimated\",\"score\",\"fifth\",\"indian\",\"decision\",\"loss\",\"industry\",\"shows\",\"battalion\",\"speed\",\"limited\",\"arrived\",\"voice\",\"la\",\"ny\",\"food\",\"operations\",\"quickly\",\"response\",\"science\",\"britain\",\"carried\",\"commercial\",\"earth\",\"pressure\",\"mark\",\"teams\",\"episodes\",\"suggested\",\"bay\",\"debut\",\"richard\",\"islands\",\"peter\",\"length\",\"writer\",\"blue\",\"smith\",\"appointed\",\"depression\",\"los\",\"minister\",\"department\",\"centre\",\"virginia\",\"hot\",\"experience\",\"especially\",\"entered\",\"might\",\"empire\",\"queen\",\"child\",\"defeated\",\"proposed\",\"car\",\"cut\",\"current\",\"remaining\",\"material\",\"latter\",\"winning\",\"intended\",\"running\",\"nearly\",\"leaving\",\"changes\",\"commander\",\"scenes\",\"course\",\"becoming\",\"date\",\"view\",\"larger\",\"wide\",\"living\",\"destroyed\",\"god\",\"agreed\",\"network\",\"daughter\",\"self\",\"shortly\",\"civil\",\"provide\",\"awarded\",\"professional\",\"language\",\"natural\",\"tracks\",\"married\",\"naval\",\"finally\",\"higher\",\"nature\",\"influence\",\"spanish\",\"pop\",\"lack\",\"approximately\",\"remains\",\"producer\",\"leader\",\"today\",\"subsequently\",\"dark\",\"supported\",\"chicago\",\"percent\",\"nd\",\"face\",\"regular\",\"health\",\"immediately\",\"soviet\",\"size\",\"trade\",\"planned\",\"contract\",\"railway\",\"fans\",\"brigade\",\"manager\",\"fact\",\"whether\",\"report\",\"soldiers\",\"create\",\"pacific\",\"feature\",\"billboard\",\"mostly\",\"related\",\"chinese\",\"ball\",\"museum\",\"compared\",\"officer\",\"longer\",\"appear\",\"associated\",\"plans\",\"cost\",\"singles\",\"corps\",\"helped\",\"themselves\",\"captured\",\"yet\",\"difficult\",\"changed\",\"fort\",\"upper\",\"your\",\"private\",\"edward\",\"guitar\",\"independent\",\"officers\",\"jackson\",\"countries\",\"friends\",\"key\",\"regiment\",\"cm\",\"extended\",\"really\",\"federal\",\"status\",\"probably\",\"something\",\"initial\",\"removed\",\"via\",\"forward\",\"cause\",\"got\",\"copies\",\"china\",\"goals\",\"increase\",\"effect\",\"location\",\"issued\",\"always\",\"books\",\"lyrics\",\"dance\",\"problems\",\"mixed\",\"numerous\",\"lord\",\"done\",\"know\",\"atlantic\",\"ran\",\"johnson\",\"moving\",\"pass\",\"christian\",\"inspired\",\"friend\",\"minor\",\"buildings\",\"information\",\"j\",\"energy\",\"suffered\",\"entertainment\",\"leave\",\"active\",\"conference\",\"artist\"],\"x1\":{\"__ndarray__\":\"[encoded data, 5336 characters]\",\"dtype\":\"float32\",\"shape\":[1000]},\"x2\":{\"__ndarray__\":\"[encoded data, 5336 characters]\",\"dtype\":\"float32\",\"shape\":[1000]}}},\"id\":\"b6ba8617-d644-4f27-b21c-3255be54ea65\",\"type\":\"ColumnDataSource\"},{\"attributes\":{},\"id\":\"01dcc880-e462-4d33-968d-be7cef6385f9\",\"type\":\"WheelZoomTool\"},{\"attributes\":{\"fill_color\":{\"value\":\"#1f77b4\"},\"line_color\":{\"value\":\"#1f77b4\"},\"size\":{\"units\":\"screen\",\"value\":8},\"x\":{\"field\":\"x1\"},\"y\":{\"field\":\"x2\"}},\"id\":\"e5cce034-1aa9-407d-aced-d079115beca4\",\"type\":\"Circle\"}],\"root_ids\":[\"da612c08-2c8d-4640-9802-d363e04ad3a0\"]},\"title\":\"Bokeh Application\",\"version\":\"0.12.7\"}};\n",
              "            var render_items = [{\"docid\":\"7d37c85a-c75a-450c-9c4e-4e6457026ff1\",\"elementid\":\"d7d0cfc8-28a5-4058-8882-b3a5d7b3c25b\",\"modelid\":\"da612c08-2c8d-4640-9802-d363e04ad3a0\"}];\n",
              "            \n",
              "            Bokeh.embed.embed_items(docs_json, render_items);\n",
              "          };\n",
              "          if (document.readyState != \"loading\") fn();\n",
              "          else document.addEventListener(\"DOMContentLoaded\", fn);\n",
              "        })();\n",
              "      },\n",
              "      function(Bokeh) {\n",
              "      }\n",
              "    ];\n",
              "  \n",
              "    function run_inline_js() {\n",
              "      \n",
              "      if ((root.Bokeh !== undefined) || (force === true)) {\n",
              "        for (var i = 0; i < inline_js.length; i++) {\n",
              "          inline_js[i].call(root, root.Bokeh);\n",
              "        }if (force === true) {\n",
              "          display_loaded();\n",
              "        }} else if (Date.now() < root._bokeh_timeout) {\n",
              "        setTimeout(run_inline_js, 100);\n",
              "      } else if (!root._bokeh_failed_load) {\n",
              "        console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n",
              "        root._bokeh_failed_load = true;\n",
              "      } else if (force !== true) {\n",
              "        var cell = $(document.getElementById(\"d7d0cfc8-28a5-4058-8882-b3a5d7b3c25b\")).parents('.cell').data().cell;\n",
              "        cell.output_area.append_execute_result(NB_LOAD_WARNING)\n",
              "      }\n",
              "  \n",
              "    }\n",
              "  \n",
              "    if (root._bokeh_is_loading === 0) {\n",
              "      console.log(\"Bokeh: BokehJS loaded, going straight to plotting\");\n",
              "      run_inline_js();\n",
              "    } else {\n",
              "      load_libs(js_urls, function() {\n",
              "        console.log(\"Bokeh: BokehJS plotting callback run at\", now());\n",
              "        run_inline_js();\n",
              "      });\n",
              "    }\n",
              "  }(window));\n",
              "</script>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "VGoQb8tgkTJ-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Homework:\n",
        "\n",
        "#### Question 1 [100pt] \n",
        "#### 1.\tRegarding the TED and Wiki datasets: \n",
        "\n",
        "    (a)\tWhat are the most frequent words in each dataset? List the top 20 most frequent words and draw the histograms.\n"
      ]
    },
    {
      "metadata": {
        "id": "ajjz_dD4kTJ_",
        "colab_type": "code",
        "colab": {},
        "outputId": "cbc15f11-e3be-429b-9be0-3faa4a0969f3"
      },
      "cell_type": "code",
      "source": [
        "print(\"Top 20 words Ted: %s\"%words_top_ted[:20])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Top 20 words Ted: ['the', 'and', 'to', 'of', 'a', 'that', 'i', 'in', 'it', 'you', 'we', 'is', 's', 'this', 'so', 'they', 'was', 'for', 'are', 'have']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "7nlDVbhUkTKD",
        "colab_type": "code",
        "colab": {},
        "outputId": "8b29cf0d-4c35-4fc4-9b78-7c70cc99319a"
      },
      "cell_type": "code",
      "source": [
        "#histogram\n",
        "hist, edges = np.histogram(counts_ted_top1000[:20], density=True, bins=100, normed=True)\n",
        "\n",
        "p = figure(tools=\"pan,wheel_zoom,reset,save\",\n",
        "           toolbar_location=\"above\",\n",
        "           title=\"Top-1000 words distribution\")\n",
        "p.quad(top=hist, bottom=0, left=edges[:-1], right=edges[1:], line_color=\"#555555\")\n",
        "show(p)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "\n",
              "    <div class=\"bk-root\">\n",
              "        <div class=\"bk-plotdiv\" id=\"673a024e-126c-4f71-ae4c-e6022c917247\"></div>\n",
              "    </div>\n",
              "<script type=\"text/javascript\">\n",
              "  \n",
              "  (function(root) {\n",
              "    function now() {\n",
              "      return new Date();\n",
              "    }\n",
              "  \n",
              "    var force = false;\n",
              "  \n",
              "    if (typeof (root._bokeh_onload_callbacks) === \"undefined\" || force === true) {\n",
              "      root._bokeh_onload_callbacks = [];\n",
              "      root._bokeh_is_loading = undefined;\n",
              "    }\n",
              "  \n",
              "  \n",
              "    \n",
              "    if (typeof (root._bokeh_timeout) === \"undefined\" || force === true) {\n",
              "      root._bokeh_timeout = Date.now() + 0;\n",
              "      root._bokeh_failed_load = false;\n",
              "    }\n",
              "  \n",
              "    var NB_LOAD_WARNING = {'data': {'text/html':\n",
              "       \"<div style='background-color: #fdd'>\\n\"+\n",
              "       \"<p>\\n\"+\n",
              "       \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n",
              "       \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n",
              "       \"</p>\\n\"+\n",
              "       \"<ul>\\n\"+\n",
              "       \"<li>re-rerun `output_notebook()` to attempt to load from CDN again, or</li>\\n\"+\n",
              "       \"<li>use INLINE resources instead, as so:</li>\\n\"+\n",
              "       \"</ul>\\n\"+\n",
              "       \"<code>\\n\"+\n",
              "       \"from bokeh.resources import INLINE\\n\"+\n",
              "       \"output_notebook(resources=INLINE)\\n\"+\n",
              "       \"</code>\\n\"+\n",
              "       \"</div>\"}};\n",
              "  \n",
              "    function display_loaded() {\n",
              "      if (root.Bokeh !== undefined) {\n",
              "        var el = document.getElementById(\"673a024e-126c-4f71-ae4c-e6022c917247\");\n",
              "        if (el != null) {\n",
              "          el.textContent = \"BokehJS \" + Bokeh.version + \" successfully loaded.\";\n",
              "        }\n",
              "      } else if (Date.now() < root._bokeh_timeout) {\n",
              "        setTimeout(display_loaded, 100)\n",
              "      }\n",
              "    }\n",
              "  \n",
              "  \n",
              "    function run_callbacks() {\n",
              "      try {\n",
              "        root._bokeh_onload_callbacks.forEach(function(callback) { callback() });\n",
              "      }\n",
              "      finally {\n",
              "        delete root._bokeh_onload_callbacks\n",
              "      }\n",
              "      console.info(\"Bokeh: all callbacks have finished\");\n",
              "    }\n",
              "  \n",
              "    function load_libs(js_urls, callback) {\n",
              "      root._bokeh_onload_callbacks.push(callback);\n",
              "      if (root._bokeh_is_loading > 0) {\n",
              "        console.log(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n",
              "        return null;\n",
              "      }\n",
              "      if (js_urls == null || js_urls.length === 0) {\n",
              "        run_callbacks();\n",
              "        return null;\n",
              "      }\n",
              "      console.log(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n",
              "      root._bokeh_is_loading = js_urls.length;\n",
              "      for (var i = 0; i < js_urls.length; i++) {\n",
              "        var url = js_urls[i];\n",
              "        var s = document.createElement('script');\n",
              "        s.src = url;\n",
              "        s.async = false;\n",
              "        s.onreadystatechange = s.onload = function() {\n",
              "          root._bokeh_is_loading--;\n",
              "          if (root._bokeh_is_loading === 0) {\n",
              "            console.log(\"Bokeh: all BokehJS libraries loaded\");\n",
              "            run_callbacks()\n",
              "          }\n",
              "        };\n",
              "        s.onerror = function() {\n",
              "          console.warn(\"failed to load library \" + url);\n",
              "        };\n",
              "        console.log(\"Bokeh: injecting script tag for BokehJS library: \", url);\n",
              "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
              "      }\n",
              "    };var element = document.getElementById(\"673a024e-126c-4f71-ae4c-e6022c917247\");\n",
              "    if (element == null) {\n",
              "      console.log(\"Bokeh: ERROR: autoload.js configured with elementid '673a024e-126c-4f71-ae4c-e6022c917247' but no matching script tag was found. \")\n",
              "      return false;\n",
              "    }\n",
              "  \n",
              "    var js_urls = [];\n",
              "  \n",
              "    var inline_js = [\n",
              "      function(Bokeh) {\n",
              "        (function() {\n",
              "          var fn = function() {\n",
              "            var docs_json = {\"01e413e0-4c33-4706-8489-6ef9892527a4\":{\"roots\":{\"references\":[{\"attributes\":{\"callback\":null},\"id\":\"a1583217-1e39-49b3-9315-2a8bd41cc5aa\",\"type\":\"DataRange1d\"},{\"attributes\":{},\"id\":\"681a19f8-8a0b-493b-bb63-77479cfc8f90\",\"type\":\"WheelZoomTool\"},{\"attributes\":{\"plot\":null,\"text\":\"Top-1000 words distribution\"},\"id\":\"48b3c76c-30f5-4bd8-8731-8c80cb624faf\",\"type\":\"Title\"},{\"attributes\":{},\"id\":\"785d3002-c468-4179-8dce-e0dd6d8a5f63\",\"type\":\"SaveTool\"},{\"attributes\":{\"formatter\":{\"id\":\"932212c9-1597-4668-a8e5-32b76bedcb12\",\"type\":\"BasicTickFormatter\"},\"plot\":{\"id\":\"d5216934-09e7-471f-a5e1-8c924fec64ec\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"d819dee7-8803-4d9a-bbc6-2739690454bd\",\"type\":\"BasicTicker\"}},\"id\":\"4d7a06a0-001c-4520-886c-2cea83aabdfd\",\"type\":\"LinearAxis\"},{\"attributes\":{},\"id\":\"d819dee7-8803-4d9a-bbc6-2739690454bd\",\"type\":\"BasicTicker\"},{\"attributes\":{},\"id\":\"07889723-f8db-4df6-aa3a-a9d60454de7a\",\"type\":\"LinearScale\"},{\"attributes\":{\"callback\":null},\"id\":\"afd6d1e4-c386-4dab-8351-013a76a11cb0\",\"type\":\"DataRange1d\"},{\"attributes\":{},\"id\":\"1cec7807-c292-42bd-8ddf-6adcb9b9f66f\",\"type\":\"LinearScale\"},{\"attributes\":{},\"id\":\"932212c9-1597-4668-a8e5-32b76bedcb12\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{},\"id\":\"f38780ec-7f55-4fe4-b135-6c60d3e7b79d\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{\"active_drag\":\"auto\",\"active_inspect\":\"auto\",\"active_scroll\":\"auto\",\"active_tap\":\"auto\",\"tools\":[{\"id\":\"f0775f9f-d852-4319-8b03-79c564d1399a\",\"type\":\"PanTool\"},{\"id\":\"681a19f8-8a0b-493b-bb63-77479cfc8f90\",\"type\":\"WheelZoomTool\"},{\"id\":\"40d975f2-bc88-492e-955a-993a0fc610da\",\"type\":\"ResetTool\"},{\"id\":\"785d3002-c468-4179-8dce-e0dd6d8a5f63\",\"type\":\"SaveTool\"}]},\"id\":\"a9fe4ae9-84f3-4574-866a-f8c617aa150d\",\"type\":\"Toolbar\"},{\"attributes\":{\"callback\":null,\"column_names\":[\"left\",\"right\",\"top\"],\"data\":{\"left\":{\"__ndarray__\":\"AAAAAAC02kD2KFyPAnfcQOxRuB4FOt5A4XoUrgf930DsUbgeBeDgQGZmZmaGweFA4XoUrgej4kBcj8L1iITjQNejcD0KZuRAUrgehYtH5UDNzMzMDCnmQEjhehSOCudAwvUoXA/s50A+CtejkM3oQLgehesRr+lAMzMzM5OQ6kCuR+F6FHLrQClcj8KVU+xApHA9Chc17UAfhetRmBbuQJqZmZkZ+O5AFK5H4ZrZ70BI4XoUjl3wQIXrUbhOzvBAwvUoXA8/8UAAAAAA0K/xQD4K16OQIPJAexSuR1GR8kC4HoXrEQLzQPYoXI/ScvNAMzMzM5Pj80BwPQrXU1T0QK5H4XoUxfRA7FG4HtU19UApXI/Clab1QGZmZmZWF/ZApHA9CheI9kDhehSu1/j2QB+F61GYafdAXI/C9Vja90CamZmZGUv4QNejcD3au/hAFK5H4Zos+UBSuB6FW535QI/C9SgcDvpAzczMzNx++kAK16Nwne/6QEjhehReYPtAhetRuB7R+0DC9Shc30H8QAAAAACgsvxAPQrXo2Aj/UB7FK5HIZT9QLgehevhBP5A9ihcj6J1/kAzMzMzY+b+QHA9CtcjV/9ArkfheuTH/0D2KFyPUhwAQRSuR+GyVABBMzMzMxONAEFSuB6Fc8UAQXA9CtfT/QBBkML1KDQ2AUGuR+F6lG4BQczMzMz0pgFB7FG4HlXfAUEK16NwtRcCQSlcj8IVUAJBSOF6FHaIAkFmZmZm1sACQYXrUbg2+QJBpHA9CpcxA0HC9Shc92kDQeF6FK5XogNBAAAAALjaA0EfhetRGBMEQT0K16N4SwRBXI/C9diDBEF7FK5HObwEQZqZmZmZ9ARBuB6F6/ksBUHXo3A9WmUFQfYoXI+6nQVBFK5H4RrWBUEzMzMzew4GQVK4HoXbRgZBcT0K1zt/BkGPwvUonLcGQa5H4Xr87wZBzczMzFwoB0HrUbgevWAHQQrXo3AdmQdBKVyPwn3RB0FI4XoU3gkIQWZmZmY+QghBhetRuJ56CEGkcD0K/7IIQcL1KFxf6whB4XoUrr8jCUE=\",\"dtype\":\"float64\",\"shape\":[100]},\"right\":{\"__ndarray__\":\"9ihcjwJ33EDsUbgeBTreQOF6FK4H/d9A7FG4HgXg4EBmZmZmhsHhQOF6FK4Ho+JAXI/C9YiE40DXo3A9CmbkQFK4HoWLR+VAzczMzAwp5kBI4XoUjgrnQML1KFwP7OdAPgrXo5DN6EC4HoXrEa/pQDMzMzOTkOpArkfhehRy60ApXI/ClVPsQKRwPQoXNe1AH4XrUZgW7kCamZmZGfjuQBSuR+Ga2e9ASOF6FI5d8ECF61G4Ts7wQML1KFwPP/FAAAAAANCv8UA+CtejkCDyQHsUrkdRkfJAuB6F6xEC80D2KFyP0nLzQDMzMzOT4/NAcD0K11NU9ECuR+F6FMX0QOxRuB7VNfVAKVyPwpWm9UBmZmZmVhf2QKRwPQoXiPZA4XoUrtf49kAfhetRmGn3QFyPwvVY2vdAmpmZmRlL+EDXo3A92rv4QBSuR+GaLPlAUrgehVud+UCPwvUoHA76QM3MzMzcfvpACtejcJ3v+kBI4XoUXmD7QIXrUbge0ftAwvUoXN9B/EAAAAAAoLL8QD0K16NgI/1AexSuRyGU/UC4HoXr4QT+QPYoXI+idf5AMzMzM2Pm/kBwPQrXI1f/QK5H4Xrkx/9A9ihcj1IcAEEUrkfhslQAQTMzMzMTjQBBUrgehXPFAEFwPQrX0/0AQZDC9Sg0NgFBrkfhepRuAUHMzMzM9KYBQexRuB5V3wFBCtejcLUXAkEpXI/CFVACQUjhehR2iAJBZmZmZtbAAkGF61G4NvkCQaRwPQqXMQNBwvUoXPdpA0HhehSuV6IDQQAAAAC42gNBH4XrURgTBEE9CtejeEsEQVyPwvXYgwRBexSuRzm8BEGamZmZmfQEQbgehev5LAVB16NwPVplBUH2KFyPup0FQRSuR+Ea1gVBMzMzM3sOBkFSuB6F20YGQXE9Ctc7fwZBj8L1KJy3BkGuR+F6/O8GQc3MzMxcKAdB61G4Hr1gB0EK16NwHZkHQSlcj8J90QdBSOF6FN4JCEFmZmZmPkIIQYXrUbieeghBpHA9Cv+yCEHC9ShcX+sIQeF6FK6/IwlBAAAAACBcCUE=\",\"dtype\":\"float64\",\"shape\":[100]},\"top\":{\"__ndarray__\":\"bjP4ldcPDT9uM/iV1w8NPwAAAAAAAAAAXjP4ldcP/T4AAAAAAAAAAG4z+JXXD/0+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAATjP4ldcP/T4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABuM/iV1w/9PgAAAAAAAAAAAAAAAAAAAABuM/iV1w/9PgAAAAAAAAAAAAAAAAAAAACQM/iV1w/9PgAAAAAAAAAATjP4ldcP/T4AAAAAAAAAAJAz+JXXD/0+AAAAAAAAAABOM/iV1w/9PgAAAAAAAAAAkDP4ldcP/T4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABOM/iV1w/9PgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACQM/iV1w/9PgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAkDP4ldcP/T4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAkDP4ldcP/T4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABOM/iV1w/9PgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAATjP4ldcP/T4=\",\"dtype\":\"float64\",\"shape\":[100]}}},\"id\":\"ee0fc240-4faa-466d-a60e-1df25f6b1768\",\"type\":\"ColumnDataSource\"},{\"attributes\":{},\"id\":\"92447c4c-cd67-4d05-832d-9dfd042673b5\",\"type\":\"BasicTicker\"},{\"attributes\":{\"formatter\":{\"id\":\"f38780ec-7f55-4fe4-b135-6c60d3e7b79d\",\"type\":\"BasicTickFormatter\"},\"plot\":{\"id\":\"d5216934-09e7-471f-a5e1-8c924fec64ec\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"92447c4c-cd67-4d05-832d-9dfd042673b5\",\"type\":\"BasicTicker\"}},\"id\":\"743cd3dd-26c9-47b4-857a-44b01857f482\",\"type\":\"LinearAxis\"},{\"attributes\":{\"plot\":{\"id\":\"d5216934-09e7-471f-a5e1-8c924fec64ec\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"d819dee7-8803-4d9a-bbc6-2739690454bd\",\"type\":\"BasicTicker\"}},\"id\":\"a85d1bba-7a35-4dc5-b40f-0afe65ab1949\",\"type\":\"Grid\"},{\"attributes\":{\"bottom\":{\"value\":0},\"fill_color\":{\"value\":\"#1f77b4\"},\"left\":{\"field\":\"left\"},\"line_color\":{\"value\":\"#555555\"},\"right\":{\"field\":\"right\"},\"top\":{\"field\":\"top\"}},\"id\":\"bd39e8bc-dc29-490f-8b8a-f241c1285dda\",\"type\":\"Quad\"},{\"attributes\":{\"data_source\":{\"id\":\"ee0fc240-4faa-466d-a60e-1df25f6b1768\",\"type\":\"ColumnDataSource\"},\"glyph\":{\"id\":\"bd39e8bc-dc29-490f-8b8a-f241c1285dda\",\"type\":\"Quad\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"88dfbeaa-7b79-4ba8-95d1-b2e48d9af528\",\"type\":\"Quad\"},\"selection_glyph\":null,\"view\":{\"id\":\"f2b1b302-10ea-48c9-864d-603e73219755\",\"type\":\"CDSView\"}},\"id\":\"f0020af0-a3cb-4cde-aa74-5452542eecc9\",\"type\":\"GlyphRenderer\"},{\"attributes\":{\"bottom\":{\"value\":0},\"fill_alpha\":{\"value\":0.1},\"fill_color\":{\"value\":\"#1f77b4\"},\"left\":{\"field\":\"left\"},\"line_alpha\":{\"value\":0.1},\"line_color\":{\"value\":\"#1f77b4\"},\"right\":{\"field\":\"right\"},\"top\":{\"field\":\"top\"}},\"id\":\"88dfbeaa-7b79-4ba8-95d1-b2e48d9af528\",\"type\":\"Quad\"},{\"attributes\":{\"dimension\":1,\"plot\":{\"id\":\"d5216934-09e7-471f-a5e1-8c924fec64ec\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"92447c4c-cd67-4d05-832d-9dfd042673b5\",\"type\":\"BasicTicker\"}},\"id\":\"960944e3-87bf-44b0-9bd2-bd51a7ddf42d\",\"type\":\"Grid\"},{\"attributes\":{},\"id\":\"f0775f9f-d852-4319-8b03-79c564d1399a\",\"type\":\"PanTool\"},{\"attributes\":{\"source\":{\"id\":\"ee0fc240-4faa-466d-a60e-1df25f6b1768\",\"type\":\"ColumnDataSource\"}},\"id\":\"f2b1b302-10ea-48c9-864d-603e73219755\",\"type\":\"CDSView\"},{\"attributes\":{\"below\":[{\"id\":\"4d7a06a0-001c-4520-886c-2cea83aabdfd\",\"type\":\"LinearAxis\"}],\"left\":[{\"id\":\"743cd3dd-26c9-47b4-857a-44b01857f482\",\"type\":\"LinearAxis\"}],\"renderers\":[{\"id\":\"4d7a06a0-001c-4520-886c-2cea83aabdfd\",\"type\":\"LinearAxis\"},{\"id\":\"a85d1bba-7a35-4dc5-b40f-0afe65ab1949\",\"type\":\"Grid\"},{\"id\":\"743cd3dd-26c9-47b4-857a-44b01857f482\",\"type\":\"LinearAxis\"},{\"id\":\"960944e3-87bf-44b0-9bd2-bd51a7ddf42d\",\"type\":\"Grid\"},{\"id\":\"f0020af0-a3cb-4cde-aa74-5452542eecc9\",\"type\":\"GlyphRenderer\"}],\"title\":{\"id\":\"48b3c76c-30f5-4bd8-8731-8c80cb624faf\",\"type\":\"Title\"},\"toolbar\":{\"id\":\"a9fe4ae9-84f3-4574-866a-f8c617aa150d\",\"type\":\"Toolbar\"},\"toolbar_location\":\"above\",\"x_range\":{\"id\":\"a1583217-1e39-49b3-9315-2a8bd41cc5aa\",\"type\":\"DataRange1d\"},\"x_scale\":{\"id\":\"1cec7807-c292-42bd-8ddf-6adcb9b9f66f\",\"type\":\"LinearScale\"},\"y_range\":{\"id\":\"afd6d1e4-c386-4dab-8351-013a76a11cb0\",\"type\":\"DataRange1d\"},\"y_scale\":{\"id\":\"07889723-f8db-4df6-aa3a-a9d60454de7a\",\"type\":\"LinearScale\"}},\"id\":\"d5216934-09e7-471f-a5e1-8c924fec64ec\",\"subtype\":\"Figure\",\"type\":\"Plot\"},{\"attributes\":{},\"id\":\"40d975f2-bc88-492e-955a-993a0fc610da\",\"type\":\"ResetTool\"}],\"root_ids\":[\"d5216934-09e7-471f-a5e1-8c924fec64ec\"]},\"title\":\"Bokeh Application\",\"version\":\"0.12.7\"}};\n",
              "            var render_items = [{\"docid\":\"01e413e0-4c33-4706-8489-6ef9892527a4\",\"elementid\":\"673a024e-126c-4f71-ae4c-e6022c917247\",\"modelid\":\"d5216934-09e7-471f-a5e1-8c924fec64ec\"}];\n",
              "            \n",
              "            Bokeh.embed.embed_items(docs_json, render_items);\n",
              "          };\n",
              "          if (document.readyState != \"loading\") fn();\n",
              "          else document.addEventListener(\"DOMContentLoaded\", fn);\n",
              "        })();\n",
              "      },\n",
              "      function(Bokeh) {\n",
              "      }\n",
              "    ];\n",
              "  \n",
              "    function run_inline_js() {\n",
              "      \n",
              "      if ((root.Bokeh !== undefined) || (force === true)) {\n",
              "        for (var i = 0; i < inline_js.length; i++) {\n",
              "          inline_js[i].call(root, root.Bokeh);\n",
              "        }if (force === true) {\n",
              "          display_loaded();\n",
              "        }} else if (Date.now() < root._bokeh_timeout) {\n",
              "        setTimeout(run_inline_js, 100);\n",
              "      } else if (!root._bokeh_failed_load) {\n",
              "        console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n",
              "        root._bokeh_failed_load = true;\n",
              "      } else if (force !== true) {\n",
              "        var cell = $(document.getElementById(\"673a024e-126c-4f71-ae4c-e6022c917247\")).parents('.cell').data().cell;\n",
              "        cell.output_area.append_execute_result(NB_LOAD_WARNING)\n",
              "      }\n",
              "  \n",
              "    }\n",
              "  \n",
              "    if (root._bokeh_is_loading === 0) {\n",
              "      console.log(\"Bokeh: BokehJS loaded, going straight to plotting\");\n",
              "      run_inline_js();\n",
              "    } else {\n",
              "      load_libs(js_urls, function() {\n",
              "        console.log(\"Bokeh: BokehJS plotting callback run at\", now());\n",
              "        run_inline_js();\n",
              "      });\n",
              "    }\n",
              "  }(window));\n",
              "</script>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "wm52dC23kTKH",
        "colab_type": "code",
        "colab": {},
        "outputId": "420c82cf-dc84-4a98-e505-c61f02cb25ae"
      },
      "cell_type": "code",
      "source": [
        "print(\"Top 20 words Wiki: %s\"%words_top_wiki[:20])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Top 20 words Wiki: ['the', 'of', 'and', 'in', 'to', 'a', 'was', 's', 'on', 'as', 'for', 'that', 'with', 'by', 'is', 'his', 'at', 'he', 'from', 'it']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "qZ_j-KWtkTKL",
        "colab_type": "code",
        "colab": {},
        "outputId": "356460e1-80d3-45a3-ecff-6de97c65e54f"
      },
      "cell_type": "code",
      "source": [
        "#histogram\n",
        "hist, edges = np.histogram(words_top_wiki1000[:20], density=True, bins=100, normed=True)\n",
        "\n",
        "p = figure(tools=\"pan,wheel_zoom,reset,save\",\n",
        "           toolbar_location=\"above\",\n",
        "           title=\"Top-1000 words distribution\")\n",
        "p.quad(top=hist, bottom=0, left=edges[:-1], right=edges[1:], line_color=\"#555555\")\n",
        "show(p)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "\n",
              "    <div class=\"bk-root\">\n",
              "        <div class=\"bk-plotdiv\" id=\"24ec8360-1659-4a62-8cdf-0a7fbe456cea\"></div>\n",
              "    </div>\n",
              "<script type=\"text/javascript\">\n",
              "  \n",
              "  (function(root) {\n",
              "    function now() {\n",
              "      return new Date();\n",
              "    }\n",
              "  \n",
              "    var force = false;\n",
              "  \n",
              "    if (typeof (root._bokeh_onload_callbacks) === \"undefined\" || force === true) {\n",
              "      root._bokeh_onload_callbacks = [];\n",
              "      root._bokeh_is_loading = undefined;\n",
              "    }\n",
              "  \n",
              "  \n",
              "    \n",
              "    if (typeof (root._bokeh_timeout) === \"undefined\" || force === true) {\n",
              "      root._bokeh_timeout = Date.now() + 0;\n",
              "      root._bokeh_failed_load = false;\n",
              "    }\n",
              "  \n",
              "    var NB_LOAD_WARNING = {'data': {'text/html':\n",
              "       \"<div style='background-color: #fdd'>\\n\"+\n",
              "       \"<p>\\n\"+\n",
              "       \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n",
              "       \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n",
              "       \"</p>\\n\"+\n",
              "       \"<ul>\\n\"+\n",
              "       \"<li>re-rerun `output_notebook()` to attempt to load from CDN again, or</li>\\n\"+\n",
              "       \"<li>use INLINE resources instead, as so:</li>\\n\"+\n",
              "       \"</ul>\\n\"+\n",
              "       \"<code>\\n\"+\n",
              "       \"from bokeh.resources import INLINE\\n\"+\n",
              "       \"output_notebook(resources=INLINE)\\n\"+\n",
              "       \"</code>\\n\"+\n",
              "       \"</div>\"}};\n",
              "  \n",
              "    function display_loaded() {\n",
              "      if (root.Bokeh !== undefined) {\n",
              "        var el = document.getElementById(\"24ec8360-1659-4a62-8cdf-0a7fbe456cea\");\n",
              "        if (el != null) {\n",
              "          el.textContent = \"BokehJS \" + Bokeh.version + \" successfully loaded.\";\n",
              "        }\n",
              "      } else if (Date.now() < root._bokeh_timeout) {\n",
              "        setTimeout(display_loaded, 100)\n",
              "      }\n",
              "    }\n",
              "  \n",
              "  \n",
              "    function run_callbacks() {\n",
              "      try {\n",
              "        root._bokeh_onload_callbacks.forEach(function(callback) { callback() });\n",
              "      }\n",
              "      finally {\n",
              "        delete root._bokeh_onload_callbacks\n",
              "      }\n",
              "      console.info(\"Bokeh: all callbacks have finished\");\n",
              "    }\n",
              "  \n",
              "    function load_libs(js_urls, callback) {\n",
              "      root._bokeh_onload_callbacks.push(callback);\n",
              "      if (root._bokeh_is_loading > 0) {\n",
              "        console.log(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n",
              "        return null;\n",
              "      }\n",
              "      if (js_urls == null || js_urls.length === 0) {\n",
              "        run_callbacks();\n",
              "        return null;\n",
              "      }\n",
              "      console.log(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n",
              "      root._bokeh_is_loading = js_urls.length;\n",
              "      for (var i = 0; i < js_urls.length; i++) {\n",
              "        var url = js_urls[i];\n",
              "        var s = document.createElement('script');\n",
              "        s.src = url;\n",
              "        s.async = false;\n",
              "        s.onreadystatechange = s.onload = function() {\n",
              "          root._bokeh_is_loading--;\n",
              "          if (root._bokeh_is_loading === 0) {\n",
              "            console.log(\"Bokeh: all BokehJS libraries loaded\");\n",
              "            run_callbacks()\n",
              "          }\n",
              "        };\n",
              "        s.onerror = function() {\n",
              "          console.warn(\"failed to load library \" + url);\n",
              "        };\n",
              "        console.log(\"Bokeh: injecting script tag for BokehJS library: \", url);\n",
              "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
              "      }\n",
              "    };var element = document.getElementById(\"24ec8360-1659-4a62-8cdf-0a7fbe456cea\");\n",
              "    if (element == null) {\n",
              "      console.log(\"Bokeh: ERROR: autoload.js configured with elementid '24ec8360-1659-4a62-8cdf-0a7fbe456cea' but no matching script tag was found. \")\n",
              "      return false;\n",
              "    }\n",
              "  \n",
              "    var js_urls = [];\n",
              "  \n",
              "    var inline_js = [\n",
              "      function(Bokeh) {\n",
              "        (function() {\n",
              "          var fn = function() {\n",
              "            var docs_json = {\"9fff811e-4bc7-4619-9860-1791adecaecb\":{\"roots\":{\"references\":[{\"attributes\":{\"formatter\":{\"id\":\"63d2c72b-621b-4910-a637-b76e41406db7\",\"type\":\"BasicTickFormatter\"},\"plot\":{\"id\":\"7babd966-a5b2-4fb5-98cc-a7b9663d7191\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"f0f00029-5607-496f-aa14-686c2a99627a\",\"type\":\"BasicTicker\"}},\"id\":\"30c21fd2-7f03-456c-ab79-4ff894b43672\",\"type\":\"LinearAxis\"},{\"attributes\":{\"data_source\":{\"id\":\"8b76c886-08f1-4ac5-bbfd-cff2975eba08\",\"type\":\"ColumnDataSource\"},\"glyph\":{\"id\":\"af6378da-17d5-4228-b86b-561fbed9a86d\",\"type\":\"Quad\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"af787c4a-b329-4f38-a00e-aae69576eb2c\",\"type\":\"Quad\"},\"selection_glyph\":null,\"view\":{\"id\":\"98b9a66d-ce5c-42c2-9469-65fd6e259823\",\"type\":\"CDSView\"}},\"id\":\"2ce37896-9688-4b25-8903-6ddd77d9695d\",\"type\":\"GlyphRenderer\"},{\"attributes\":{},\"id\":\"6147337a-26a9-4a1e-8cfa-a96db895a68c\",\"type\":\"LinearScale\"},{\"attributes\":{},\"id\":\"4da93136-b2f8-4efc-8c96-dc48d206a95d\",\"type\":\"WheelZoomTool\"},{\"attributes\":{},\"id\":\"dc46b041-817d-461e-8f6c-4a644d7924ec\",\"type\":\"BasicTicker\"},{\"attributes\":{\"plot\":{\"id\":\"7babd966-a5b2-4fb5-98cc-a7b9663d7191\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"dc46b041-817d-461e-8f6c-4a644d7924ec\",\"type\":\"BasicTicker\"}},\"id\":\"6285b6c4-9a68-4d7a-9641-6abbe609484f\",\"type\":\"Grid\"},{\"attributes\":{\"plot\":null,\"text\":\"Top-1000 words distribution\"},\"id\":\"9b12f159-9d3d-4a31-b66e-f7b036138cfe\",\"type\":\"Title\"},{\"attributes\":{\"callback\":null,\"column_names\":[\"left\",\"right\",\"top\"],\"data\":{\"left\":{\"__ndarray__\":\"AAAAAOCr9UCkcD0K55j4QEjhehTuhftA7FG4HvVy/kBI4XoU/q8AQZqZmZmBJgJB7FG4HgWdA0E+CtejiBMFQZDC9SgMigZB4noUro8ACEE0MzMzE3cJQYXrUbiW7QpB16NwPRpkDEEpXI/CndoNQXsUrkchUQ9BZmZmZtJjEEGQwvUoFB8RQbgehetV2hFB4noUrpeVEkEK16Nw2VATQTQzMzMbDBRBXI/C9VzHFEGF61G4noIVQa5H4XrgPRZB16NwPSL5FkEAAAAAZLQXQSlcj8KlbxhBUrgehecqGUF7FK5HKeYZQaRwPQproRpBzczMzKxcG0H2KFyP7hccQR+F61Ew0xxBSOF6FHKOHUFxPQrXs0keQZqZmZn1BB9Bw/UoXDfAH0H2KFyPvD0gQQrXo3BdmyBBH4XrUf74IEE0MzMzn1YhQUjhehRAtCFBXI/C9eARIkFxPQrXgW8iQYXrUbgizSJBmpmZmcMqI0GuR+F6ZIgjQcP1KFwF5iNB16NwPaZDJEHsUbgeR6EkQQAAAADo/iRBFa5H4YhcJUEpXI/CKbolQT4K16PKFyZBUrgehWt1JkFnZmZmDNMmQXsUrketMCdBkML1KE6OJ0GkcD0K7+snQbkeheuPSShBzczMzDCnKEHiehSu0QQpQfYoXI9yYilBC9ejcBPAKUEfhetRtB0qQTMzMzNVeypBSOF6FPbYKkFcj8L1ljYrQXE9Ctc3lCtBhetRuNjxK0GamZmZeU8sQa5H4XoarSxBw/UoXLsKLUHXo3A9XGgtQexRuB79xS1BAAAAAJ4jLkEVrkfhPoEuQSlcj8Lf3i5BPgrXo4A8L0FSuB6FIZovQWdmZmbC9y9BPgrXo7EqMEFI4XoUglkwQVK4HoVSiDBBXI/C9SK3MEFmZmZm8+UwQXE9CtfDFDFBexSuR5RDMUGF61G4ZHIxQZDC9Sg1oTFBmpmZmQXQMUGkcD0K1v4xQa5H4XqmLTJBuR6F63ZcMkHD9ShcR4syQc3MzMwXujJB16NwPejoMkHhehSuuBczQexRuB6JRjNB9ihcj1l1M0E=\",\"dtype\":\"float64\",\"shape\":[100]},\"right\":{\"__ndarray__\":\"pHA9CueY+EBI4XoU7oX7QOxRuB71cv5ASOF6FP6vAEGamZmZgSYCQexRuB4FnQNBPgrXo4gTBUGQwvUoDIoGQeJ6FK6PAAhBNDMzMxN3CUGF61G4lu0KQdejcD0aZAxBKVyPwp3aDUF7FK5HIVEPQWZmZmbSYxBBkML1KBQfEUG4HoXrVdoRQeJ6FK6XlRJBCtejcNlQE0E0MzMzGwwUQVyPwvVcxxRBhetRuJ6CFUGuR+F64D0WQdejcD0i+RZBAAAAAGS0F0EpXI/CpW8YQVK4HoXnKhlBexSuRynmGUGkcD0Ka6EaQc3MzMysXBtB9ihcj+4XHEEfhetRMNMcQUjhehRyjh1BcT0K17NJHkGamZmZ9QQfQcP1KFw3wB9B9ihcj7w9IEEK16NwXZsgQR+F61H++CBBNDMzM59WIUFI4XoUQLQhQVyPwvXgESJBcT0K14FvIkGF61G4Is0iQZqZmZnDKiNBrkfhemSII0HD9ShcBeYjQdejcD2mQyRB7FG4HkehJEEAAAAA6P4kQRWuR+GIXCVBKVyPwim6JUE+CtejyhcmQVK4HoVrdSZBZ2ZmZgzTJkF7FK5HrTAnQZDC9ShOjidBpHA9Cu/rJ0G5HoXrj0koQc3MzMwwpyhB4noUrtEEKUH2KFyPcmIpQQvXo3ATwClBH4XrUbQdKkEzMzMzVXsqQUjhehT22CpBXI/C9ZY2K0FxPQrXN5QrQYXrUbjY8StBmpmZmXlPLEGuR+F6Gq0sQcP1KFy7Ci1B16NwPVxoLUHsUbge/cUtQQAAAACeIy5BFa5H4T6BLkEpXI/C394uQT4K16OAPC9BUrgehSGaL0FnZmZmwvcvQT4K16OxKjBBSOF6FIJZMEFSuB6FUogwQVyPwvUitzBBZmZmZvPlMEFxPQrXwxQxQXsUrkeUQzFBhetRuGRyMUGQwvUoNaExQZqZmZkF0DFBpHA9Ctb+MUGuR+F6pi0yQbkehet2XDJBw/UoXEeLMkHNzMzMF7oyQdejcD3o6DJB4XoUrrgXM0HsUbgeiUYzQfYoXI9ZdTNBAAAAACqkM0E=\",\"dtype\":\"float64\",\"shape\":[100]},\"top\":{\"__ndarray__\":\"H8RNjqrf9T5MAwulu3/RPgAAAAAAAAAATAMLpbt/4T7yhJB3mT/qPkwDC6W7f9E+TAMLpbt/0T4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABYAwulu3/RPgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABMAwulu3/RPgAAAAAAAAAAAAAAAAAAAABMAwulu3/RPgAAAAAAAAAAAAAAAAAAAABMAwulu3/RPgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABMAwulu3/RPgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADQDC6W7f9E+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAZAMLpbt/0T4=\",\"dtype\":\"float64\",\"shape\":[100]}}},\"id\":\"8b76c886-08f1-4ac5-bbfd-cff2975eba08\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"callback\":null},\"id\":\"a3520653-1b8b-4734-9828-dd8b81ab1d28\",\"type\":\"DataRange1d\"},{\"attributes\":{\"below\":[{\"id\":\"136b8734-c975-4c6d-9679-3a3727f9bef1\",\"type\":\"LinearAxis\"}],\"left\":[{\"id\":\"30c21fd2-7f03-456c-ab79-4ff894b43672\",\"type\":\"LinearAxis\"}],\"renderers\":[{\"id\":\"136b8734-c975-4c6d-9679-3a3727f9bef1\",\"type\":\"LinearAxis\"},{\"id\":\"6285b6c4-9a68-4d7a-9641-6abbe609484f\",\"type\":\"Grid\"},{\"id\":\"30c21fd2-7f03-456c-ab79-4ff894b43672\",\"type\":\"LinearAxis\"},{\"id\":\"d387a5d3-e9de-4dd9-b73b-c4bfa32f2b7f\",\"type\":\"Grid\"},{\"id\":\"2ce37896-9688-4b25-8903-6ddd77d9695d\",\"type\":\"GlyphRenderer\"}],\"title\":{\"id\":\"9b12f159-9d3d-4a31-b66e-f7b036138cfe\",\"type\":\"Title\"},\"toolbar\":{\"id\":\"a14b6631-23a4-4cbe-ad2b-83f7809c870d\",\"type\":\"Toolbar\"},\"toolbar_location\":\"above\",\"x_range\":{\"id\":\"a3520653-1b8b-4734-9828-dd8b81ab1d28\",\"type\":\"DataRange1d\"},\"x_scale\":{\"id\":\"4a2d9d2f-21a7-45e6-aa3d-b1f9fa63dfad\",\"type\":\"LinearScale\"},\"y_range\":{\"id\":\"7669eb78-e29b-47d2-91f4-ace53f3a2eb6\",\"type\":\"DataRange1d\"},\"y_scale\":{\"id\":\"6147337a-26a9-4a1e-8cfa-a96db895a68c\",\"type\":\"LinearScale\"}},\"id\":\"7babd966-a5b2-4fb5-98cc-a7b9663d7191\",\"subtype\":\"Figure\",\"type\":\"Plot\"},{\"attributes\":{\"formatter\":{\"id\":\"aa36d23c-ebf9-4d10-b6a8-8cb205be65b1\",\"type\":\"BasicTickFormatter\"},\"plot\":{\"id\":\"7babd966-a5b2-4fb5-98cc-a7b9663d7191\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"dc46b041-817d-461e-8f6c-4a644d7924ec\",\"type\":\"BasicTicker\"}},\"id\":\"136b8734-c975-4c6d-9679-3a3727f9bef1\",\"type\":\"LinearAxis\"},{\"attributes\":{},\"id\":\"e67658da-303a-4beb-8afd-898a7c32beea\",\"type\":\"SaveTool\"},{\"attributes\":{\"bottom\":{\"value\":0},\"fill_color\":{\"value\":\"#1f77b4\"},\"left\":{\"field\":\"left\"},\"line_color\":{\"value\":\"#555555\"},\"right\":{\"field\":\"right\"},\"top\":{\"field\":\"top\"}},\"id\":\"af6378da-17d5-4228-b86b-561fbed9a86d\",\"type\":\"Quad\"},{\"attributes\":{\"dimension\":1,\"plot\":{\"id\":\"7babd966-a5b2-4fb5-98cc-a7b9663d7191\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"f0f00029-5607-496f-aa14-686c2a99627a\",\"type\":\"BasicTicker\"}},\"id\":\"d387a5d3-e9de-4dd9-b73b-c4bfa32f2b7f\",\"type\":\"Grid\"},{\"attributes\":{},\"id\":\"4a2d9d2f-21a7-45e6-aa3d-b1f9fa63dfad\",\"type\":\"LinearScale\"},{\"attributes\":{\"source\":{\"id\":\"8b76c886-08f1-4ac5-bbfd-cff2975eba08\",\"type\":\"ColumnDataSource\"}},\"id\":\"98b9a66d-ce5c-42c2-9469-65fd6e259823\",\"type\":\"CDSView\"},{\"attributes\":{},\"id\":\"f0f00029-5607-496f-aa14-686c2a99627a\",\"type\":\"BasicTicker\"},{\"attributes\":{},\"id\":\"ae3add5d-760e-41ca-8231-f90501039fb6\",\"type\":\"ResetTool\"},{\"attributes\":{\"bottom\":{\"value\":0},\"fill_alpha\":{\"value\":0.1},\"fill_color\":{\"value\":\"#1f77b4\"},\"left\":{\"field\":\"left\"},\"line_alpha\":{\"value\":0.1},\"line_color\":{\"value\":\"#1f77b4\"},\"right\":{\"field\":\"right\"},\"top\":{\"field\":\"top\"}},\"id\":\"af787c4a-b329-4f38-a00e-aae69576eb2c\",\"type\":\"Quad\"},{\"attributes\":{\"active_drag\":\"auto\",\"active_inspect\":\"auto\",\"active_scroll\":\"auto\",\"active_tap\":\"auto\",\"tools\":[{\"id\":\"258b2f01-93ba-48fe-aa15-59dc64aa627a\",\"type\":\"PanTool\"},{\"id\":\"4da93136-b2f8-4efc-8c96-dc48d206a95d\",\"type\":\"WheelZoomTool\"},{\"id\":\"ae3add5d-760e-41ca-8231-f90501039fb6\",\"type\":\"ResetTool\"},{\"id\":\"e67658da-303a-4beb-8afd-898a7c32beea\",\"type\":\"SaveTool\"}]},\"id\":\"a14b6631-23a4-4cbe-ad2b-83f7809c870d\",\"type\":\"Toolbar\"},{\"attributes\":{},\"id\":\"63d2c72b-621b-4910-a637-b76e41406db7\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{},\"id\":\"258b2f01-93ba-48fe-aa15-59dc64aa627a\",\"type\":\"PanTool\"},{\"attributes\":{\"callback\":null},\"id\":\"7669eb78-e29b-47d2-91f4-ace53f3a2eb6\",\"type\":\"DataRange1d\"},{\"attributes\":{},\"id\":\"aa36d23c-ebf9-4d10-b6a8-8cb205be65b1\",\"type\":\"BasicTickFormatter\"}],\"root_ids\":[\"7babd966-a5b2-4fb5-98cc-a7b9663d7191\"]},\"title\":\"Bokeh Application\",\"version\":\"0.12.7\"}};\n",
              "            var render_items = [{\"docid\":\"9fff811e-4bc7-4619-9860-1791adecaecb\",\"elementid\":\"24ec8360-1659-4a62-8cdf-0a7fbe456cea\",\"modelid\":\"7babd966-a5b2-4fb5-98cc-a7b9663d7191\"}];\n",
              "            \n",
              "            Bokeh.embed.embed_items(docs_json, render_items);\n",
              "          };\n",
              "          if (document.readyState != \"loading\") fn();\n",
              "          else document.addEventListener(\"DOMContentLoaded\", fn);\n",
              "        })();\n",
              "      },\n",
              "      function(Bokeh) {\n",
              "      }\n",
              "    ];\n",
              "  \n",
              "    function run_inline_js() {\n",
              "      \n",
              "      if ((root.Bokeh !== undefined) || (force === true)) {\n",
              "        for (var i = 0; i < inline_js.length; i++) {\n",
              "          inline_js[i].call(root, root.Bokeh);\n",
              "        }if (force === true) {\n",
              "          display_loaded();\n",
              "        }} else if (Date.now() < root._bokeh_timeout) {\n",
              "        setTimeout(run_inline_js, 100);\n",
              "      } else if (!root._bokeh_failed_load) {\n",
              "        console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n",
              "        root._bokeh_failed_load = true;\n",
              "      } else if (force !== true) {\n",
              "        var cell = $(document.getElementById(\"24ec8360-1659-4a62-8cdf-0a7fbe456cea\")).parents('.cell').data().cell;\n",
              "        cell.output_area.append_execute_result(NB_LOAD_WARNING)\n",
              "      }\n",
              "  \n",
              "    }\n",
              "  \n",
              "    if (root._bokeh_is_loading === 0) {\n",
              "      console.log(\"Bokeh: BokehJS loaded, going straight to plotting\");\n",
              "      run_inline_js();\n",
              "    } else {\n",
              "      load_libs(js_urls, function() {\n",
              "        console.log(\"Bokeh: BokehJS plotting callback run at\", now());\n",
              "        run_inline_js();\n",
              "      });\n",
              "    }\n",
              "  }(window));\n",
              "</script>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "A1DV4ciIkTKP",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Homework:\n",
        "\n",
        "#### Question 1 [100pt] \n",
        "#### 1.\tRegarding the TED and Wiki datasets: \n",
        "\n",
        "    (b)\tGet the most similar words to man using the embeddings you got using each corpus. \n"
      ]
    },
    {
      "metadata": {
        "id": "rdkgye7KkTKR",
        "colab_type": "code",
        "colab": {},
        "outputId": "52eb9c96-cac5-4b05-b617-e7a9b76356cf"
      },
      "cell_type": "code",
      "source": [
        "print(\"Most similar to man for TED: \")\n",
        "model_ted.most_similar(\"man\")\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Most similar to man for TED: \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('woman', 0.8682868480682373),\n",
              " ('guy', 0.8158002495765686),\n",
              " ('lady', 0.7774063348770142),\n",
              " ('girl', 0.74993896484375),\n",
              " ('boy', 0.7380883693695068),\n",
              " ('soldier', 0.7181751728057861),\n",
              " ('gentleman', 0.7085753083229065),\n",
              " ('kid', 0.7070672512054443),\n",
              " ('philosopher', 0.6738376021385193),\n",
              " ('doctor', 0.6634228825569153)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 61
        }
      ]
    },
    {
      "metadata": {
        "id": "vaY6iV1GkTKW",
        "colab_type": "code",
        "colab": {},
        "outputId": "44f3b6c0-3bc2-4b0c-b36c-96f6dfc0086c"
      },
      "cell_type": "code",
      "source": [
        "print(\"Most similar to man for Wiki: \")\n",
        "model_wiki.most_similar('man')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Most similar to man for Wiki: \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('woman', 0.7413420677185059),\n",
              " ('dog', 0.6260066032409668),\n",
              " ('boy', 0.6195032596588135),\n",
              " ('girl', 0.6146693229675293),\n",
              " ('person', 0.6144819259643555),\n",
              " ('men', 0.5713915228843689),\n",
              " ('soldier', 0.5568812489509583),\n",
              " ('joke', 0.5567601919174194),\n",
              " ('creature', 0.5542395710945129),\n",
              " ('mask', 0.5220149755477905)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 62
        }
      ]
    },
    {
      "metadata": {
        "id": "4wW8rFDekTKb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "i. Why are they different?\n",
        "    Because each word embedding is trained on different dataset.\n",
        "    \n",
        "ii. Can you identity any relationship(s) between man and its most similar words beyond “they are similar”?\n",
        "    The appear in same context\n",
        "    \n",
        "(c) Using model ted.wv.most similar(positive=[’nice’, ’man’]) you can get the most similar words to “nice plus man”. Does the result make sense to you?"
      ]
    },
    {
      "metadata": {
        "id": "yZQ_cb7YkTKc",
        "colab_type": "code",
        "colab": {},
        "outputId": "74796d72-6ef4-4595-afb3-96911ece16da"
      },
      "cell_type": "code",
      "source": [
        "model_ted.wv.most_similar(positive=['nice', 'man'])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('guy', 0.7498212456703186),\n",
              " ('woman', 0.6990991234779358),\n",
              " ('boy', 0.6976661682128906),\n",
              " ('gentleman', 0.695278525352478),\n",
              " ('kid', 0.6902039051055908),\n",
              " ('lady', 0.6837806701660156),\n",
              " ('girl', 0.6695646047592163),\n",
              " ('dog', 0.6527926921844482),\n",
              " ('physicist', 0.6447546482086182),\n",
              " ('hoodie', 0.6394599676132202)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 63
        }
      ]
    },
    {
      "metadata": {
        "id": "JljV_1qgkTKj",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Yes they make sense."
      ]
    },
    {
      "metadata": {
        "id": "vnXJvAl8kTKk",
        "colab_type": "code",
        "colab": {},
        "outputId": "a072fdf1-acef-40b4-b2de-ae7f62b0a493"
      },
      "cell_type": "code",
      "source": [
        "print(\"January: \", model_ted.most_similar(\"january\"))\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "January:  [('october', 0.9308812618255615), ('june', 0.9222588539123535), ('august', 0.910740852355957), ('december', 0.9084295034408569), ('september', 0.9003639221191406), ('april', 0.899808943271637), ('2011', 0.8972553014755249), ('2008', 0.8921166658401489), ('february', 0.8907141089439392), ('july', 0.8866482377052307)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "8VH0YmABkTKn",
        "colab_type": "code",
        "colab": {},
        "outputId": "ce90c982-9b78-4e13-d238-14b93fea5de5"
      },
      "cell_type": "code",
      "source": [
        "print(\"february: \",model_ted.most_similar(\"february\"))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "february:  [('2012', 0.9077060222625732), ('2005', 0.896077036857605), ('2002', 0.8909220099449158), ('january', 0.8907140493392944), ('2013', 0.8901843428611755), ('2001', 0.889124870300293), ('2007', 0.8888581395149231), ('2009', 0.8882791996002197), ('november', 0.8857223987579346), ('december', 0.8837077617645264)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "OvKiuI46kTKq",
        "colab_type": "code",
        "colab": {},
        "outputId": "4d6b667e-5013-4d47-8131-2be48fb28ee9"
      },
      "cell_type": "code",
      "source": [
        "print(\"minutes: \",model_ted.most_similar(\"minutes\"))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "minutes:  [('seconds', 0.9405599236488342), ('hours', 0.8603693842887878), ('months', 0.8312658667564392), ('weeks', 0.8265615701675415), ('days', 0.7821897864341736), ('bucks', 0.7633097171783447), ('cents', 0.7531062960624695), ('minute', 0.7507410049438477), ('years', 0.7418460249900818), ('month', 0.7325657606124878)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "oRhRwJagkTKt",
        "colab_type": "code",
        "colab": {},
        "outputId": "6c38bb75-0498-467a-8202-e62e91df80e7"
      },
      "cell_type": "code",
      "source": [
        "print(\"days: \",model_ted.most_similar(\"days\"))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "days:  [('months', 0.9083890914916992), ('weeks', 0.8966906070709229), ('hours', 0.789345383644104), ('minutes', 0.7821897864341736), ('decades', 0.780881404876709), ('seconds', 0.755976140499115), ('centuries', 0.736128032207489), ('years', 0.7320805191993713), ('month', 0.7242785692214966), ('nights', 0.7189063429832458)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "_hjiF8aXkTKw",
        "colab_type": "code",
        "colab": {},
        "outputId": "653642a5-903e-4865-e0c5-3fa5e0676bba"
      },
      "cell_type": "code",
      "source": [
        "print(\"hours: \",model_ted.most_similar(\"hours\"))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "hours:  [('seconds', 0.8678438663482666), ('minutes', 0.8603694438934326), ('weeks', 0.8548505306243896), ('months', 0.85135817527771), ('days', 0.7893454432487488), ('hour', 0.7662705183029175), ('cents', 0.7621139287948608), ('milliseconds', 0.7607388496398926), ('bucks', 0.759858250617981), ('month', 0.7552515864372253)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "JN2UtNvEkTK5",
        "colab_type": "code",
        "colab": {},
        "outputId": "f61f032b-e047-4f7c-d716-c959d63375fd"
      },
      "cell_type": "code",
      "source": [
        "print(\"minutes: \",model_ted.most_similar(\"minutes\"))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "minutes:  [('seconds', 0.9405599236488342), ('hours', 0.8603693842887878), ('months', 0.8312658667564392), ('weeks', 0.8265615701675415), ('days', 0.7821897864341736), ('bucks', 0.7633097171783447), ('cents', 0.7531062960624695), ('minute', 0.7507410049438477), ('years', 0.7418460249900818), ('month', 0.7325657606124878)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "hZKStimtkTK8",
        "colab_type": "code",
        "colab": {},
        "outputId": "9e5882f6-9208-4603-c753-ca3b58bd9427"
      },
      "cell_type": "code",
      "source": [
        "print(\"january: \",model_wiki.most_similar('january'))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "january:  [('december', 0.981354296207428), ('june', 0.9783721566200256), ('february', 0.9782575368881226), ('july', 0.9752212166786194), ('november', 0.9731165170669556), ('april', 0.9698600769042969), ('october', 0.9695407748222351), ('march', 0.9683542251586914), ('august', 0.9592667818069458), ('september', 0.9530515670776367)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "-5DB5WbykTK_",
        "colab_type": "code",
        "colab": {},
        "outputId": "9dde71de-1b64-4f44-d8b3-4915d95ac84a"
      },
      "cell_type": "code",
      "source": [
        "print(\"february: \",model_wiki.most_similar('february'))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "february:  [('december', 0.9809674024581909), ('july', 0.9795922040939331), ('june', 0.9786773920059204), ('january', 0.9782577157020569), ('november', 0.9776405096054077), ('october', 0.974767804145813), ('march', 0.9721773862838745), ('april', 0.9711993932723999), ('august', 0.9666255712509155), ('september', 0.9569708108901978)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "15CL06BSkTLF",
        "colab_type": "code",
        "colab": {},
        "outputId": "e977d676-2770-44a4-df8b-582b7f9d2928"
      },
      "cell_type": "code",
      "source": [
        "print(\"minutes: \",model_wiki.most_similar('minutes'))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "minutes:  [('seconds', 0.8670499324798584), ('hours', 0.8279467821121216), ('minute', 0.7537791728973389), ('days', 0.7382129430770874), ('weeks', 0.7073256373405457), ('laps', 0.6940144300460815), ('hour', 0.6737622022628784), ('rounds', 0.6610020399093628), ('months', 0.6564478874206543), ('overs', 0.6534638404846191)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "DXWAwlnpkTLI",
        "colab_type": "code",
        "colab": {},
        "outputId": "848e1d8c-9b5c-46ea-c992-9365afa280ee"
      },
      "cell_type": "code",
      "source": [
        "print(\"hours: \",model_wiki.most_similar('hours'))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "hours:  [('days', 0.884272575378418), ('minutes', 0.8279469013214111), ('weeks', 0.819317102432251), ('months', 0.8149807453155518), ('hour', 0.6774991750717163), ('years', 0.6743379831314087), ('seconds', 0.6711820363998413), ('month', 0.6431746482849121), ('nights', 0.6345077753067017), ('laps', 0.5993154048919678)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "_47vFFTckTLL",
        "colab_type": "code",
        "colab": {},
        "outputId": "b0bf8c60-7e73-4d38-a53a-d16f0ec5153d"
      },
      "cell_type": "code",
      "source": [
        "print(\"days: \",model_wiki.most_similar('days'))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "days:  [('months', 0.9211745262145996), ('weeks', 0.8980214595794678), ('hours', 0.884272575378418), ('years', 0.8103913068771362), ('minutes', 0.7382129430770874), ('decades', 0.7201712131500244), ('month', 0.7128969430923462), ('nights', 0.6599725484848022), ('week', 0.607032060623169), ('laps', 0.6009765863418579)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "wk6n6UM7kTLO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### 2. The leases.zip "
      ]
    },
    {
      "metadata": {
        "id": "TpNlMl8zkTLO",
        "colab_type": "code",
        "colab": {},
        "outputId": "63243fc5-62e8-4b35-b9a2-0a89ad3dc25e"
      },
      "cell_type": "code",
      "source": [
        "input_text = \"\"\n",
        "with zipfile.ZipFile('leases.zip', 'r') as z:\n",
        "    for filename in z.namelist():\n",
        "        input_text += str(z.open(\"%s\"%filename, 'r').read(), encoding='utf-8')+ \" \" # Thanks Robert Bastian\n",
        "print(\"Length text: \",len(input_text))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Length text:  295773821\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "eAMsZUNSkTLQ",
        "colab_type": "code",
        "colab": {},
        "collapsed": true
      },
      "cell_type": "code",
      "source": [
        "sentences_leases = []\n",
        "for line in input_text.split('\\n'):\n",
        "    s = [x for x in line.split('.') if x and len(x.split()) >= 5]\n",
        "    sentences_leases.extend(s)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "JZZ4S6nxkTLT",
        "colab_type": "code",
        "colab": {},
        "collapsed": true
      },
      "cell_type": "code",
      "source": [
        "# sample 1/5 of the data\n",
        "shuffle(sentences_leases)\n",
        "sentences_leases = sentences_leases[:int(len(sentences_wiki)/5)]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "61A0KeLxkTLV",
        "colab_type": "code",
        "colab": {},
        "collapsed": true
      },
      "cell_type": "code",
      "source": [
        "#clean sentences from wiki\n",
        "for s_i in range(len(sentences_leases)):\n",
        "    sentences_leases[s_i] = re.sub(\"[^a-z]\", \" \", sentences_leases[s_i].lower())\n",
        "    sentences_leases[s_i] = re.sub(r'\\([^)]*\\)', '', sentences_leases[s_i]).split()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "sc-yZC_1kTLW",
        "colab_type": "code",
        "colab": {},
        "collapsed": true
      },
      "cell_type": "code",
      "source": [
        "model_leases = Word2Vec(sentences_leases, min_count=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "RfZ4q9AckTLX",
        "colab_type": "code",
        "colab": {},
        "outputId": "c4c60986-22ce-4f0b-f449-2da986752545"
      },
      "cell_type": "code",
      "source": [
        "# Calculate the most frequent words, store the results in words_top_leases\n",
        "import collections\n",
        "leases_counts = collections.defaultdict(int)\n",
        "words_top_leases1000 = []\n",
        "words_top_leases = []\n",
        "\n",
        "for sentence in sentences_leases:\n",
        "    for word in sentence:\n",
        "        if leases_counts.keys().__contains__(word):\n",
        "            leases_counts[word] += 1\n",
        "        else:\n",
        "            leases_counts[word] = 1\n",
        "#sort words in decenting order\n",
        "sorted_words_leases = sorted(leases_counts, key=leases_counts.__getitem__, reverse=True)\n",
        "#get first 1000 counts from sorted words\n",
        "{words_top_leases1000.append(leases_counts[word]) for word in sorted_words_leases[:1000]}\n",
        "#get first 1000 words from sorted words\n",
        "{words_top_leases.append(word) for word in sorted_words_leases[:1000]}\n",
        "print (\"Succesfully sorted!\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Succesfully sorted!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "UkcBNw5lkTLb",
        "colab_type": "code",
        "colab": {},
        "collapsed": true
      },
      "cell_type": "code",
      "source": [
        "# This assumes words_top_leases is a list of strings, the top 1000 words\n",
        "words_top_vec_leases = model_leases[words_top_leases]\n",
        "\n",
        "tsne = TSNE(n_components=2, random_state=0)\n",
        "words_top_leases_tsne = tsne.fit_transform(words_top_vec_leases)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "orehHuRPkTLg",
        "colab_type": "code",
        "colab": {},
        "outputId": "90bdab85-c3b7-48a8-a6da-9f7ca0b3589a"
      },
      "cell_type": "code",
      "source": [
        "p = figure(tools=\"pan,wheel_zoom,reset,save\",\n",
        "           toolbar_location=\"above\",\n",
        "           title=\"word2vec T-SNE for most common words\")\n",
        "\n",
        "source = ColumnDataSource(data=dict(x1=words_top_leases_tsne[:,0],\n",
        "                                    x2=words_top_leases_tsne[:,1],\n",
        "                                    names=words_top_leases))\n",
        "\n",
        "p.scatter(x=\"x1\", y=\"x2\", size=8, source=source)\n",
        "\n",
        "labels = LabelSet(x=\"x1\", y=\"x2\", text=\"names\", y_offset=6,\n",
        "                  text_font_size=\"8pt\", text_color=\"#555555\",\n",
        "                  source=source, text_align='center')\n",
        "p.add_layout(labels)\n",
        "\n",
        "show(p)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "\n",
              "    <div class=\"bk-root\">\n",
              "        <div class=\"bk-plotdiv\" id=\"37e31b7f-346f-4d0c-81d6-c691a59cf365\"></div>\n",
              "    </div>\n",
              "<script type=\"text/javascript\">\n",
              "  \n",
              "  (function(root) {\n",
              "    function now() {\n",
              "      return new Date();\n",
              "    }\n",
              "  \n",
              "    var force = false;\n",
              "  \n",
              "    if (typeof (root._bokeh_onload_callbacks) === \"undefined\" || force === true) {\n",
              "      root._bokeh_onload_callbacks = [];\n",
              "      root._bokeh_is_loading = undefined;\n",
              "    }\n",
              "  \n",
              "  \n",
              "    \n",
              "    if (typeof (root._bokeh_timeout) === \"undefined\" || force === true) {\n",
              "      root._bokeh_timeout = Date.now() + 0;\n",
              "      root._bokeh_failed_load = false;\n",
              "    }\n",
              "  \n",
              "    var NB_LOAD_WARNING = {'data': {'text/html':\n",
              "       \"<div style='background-color: #fdd'>\\n\"+\n",
              "       \"<p>\\n\"+\n",
              "       \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n",
              "       \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n",
              "       \"</p>\\n\"+\n",
              "       \"<ul>\\n\"+\n",
              "       \"<li>re-rerun `output_notebook()` to attempt to load from CDN again, or</li>\\n\"+\n",
              "       \"<li>use INLINE resources instead, as so:</li>\\n\"+\n",
              "       \"</ul>\\n\"+\n",
              "       \"<code>\\n\"+\n",
              "       \"from bokeh.resources import INLINE\\n\"+\n",
              "       \"output_notebook(resources=INLINE)\\n\"+\n",
              "       \"</code>\\n\"+\n",
              "       \"</div>\"}};\n",
              "  \n",
              "    function display_loaded() {\n",
              "      if (root.Bokeh !== undefined) {\n",
              "        var el = document.getElementById(\"37e31b7f-346f-4d0c-81d6-c691a59cf365\");\n",
              "        if (el != null) {\n",
              "          el.textContent = \"BokehJS \" + Bokeh.version + \" successfully loaded.\";\n",
              "        }\n",
              "      } else if (Date.now() < root._bokeh_timeout) {\n",
              "        setTimeout(display_loaded, 100)\n",
              "      }\n",
              "    }\n",
              "  \n",
              "  \n",
              "    function run_callbacks() {\n",
              "      try {\n",
              "        root._bokeh_onload_callbacks.forEach(function(callback) { callback() });\n",
              "      }\n",
              "      finally {\n",
              "        delete root._bokeh_onload_callbacks\n",
              "      }\n",
              "      console.info(\"Bokeh: all callbacks have finished\");\n",
              "    }\n",
              "  \n",
              "    function load_libs(js_urls, callback) {\n",
              "      root._bokeh_onload_callbacks.push(callback);\n",
              "      if (root._bokeh_is_loading > 0) {\n",
              "        console.log(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n",
              "        return null;\n",
              "      }\n",
              "      if (js_urls == null || js_urls.length === 0) {\n",
              "        run_callbacks();\n",
              "        return null;\n",
              "      }\n",
              "      console.log(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n",
              "      root._bokeh_is_loading = js_urls.length;\n",
              "      for (var i = 0; i < js_urls.length; i++) {\n",
              "        var url = js_urls[i];\n",
              "        var s = document.createElement('script');\n",
              "        s.src = url;\n",
              "        s.async = false;\n",
              "        s.onreadystatechange = s.onload = function() {\n",
              "          root._bokeh_is_loading--;\n",
              "          if (root._bokeh_is_loading === 0) {\n",
              "            console.log(\"Bokeh: all BokehJS libraries loaded\");\n",
              "            run_callbacks()\n",
              "          }\n",
              "        };\n",
              "        s.onerror = function() {\n",
              "          console.warn(\"failed to load library \" + url);\n",
              "        };\n",
              "        console.log(\"Bokeh: injecting script tag for BokehJS library: \", url);\n",
              "        document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
              "      }\n",
              "    };var element = document.getElementById(\"37e31b7f-346f-4d0c-81d6-c691a59cf365\");\n",
              "    if (element == null) {\n",
              "      console.log(\"Bokeh: ERROR: autoload.js configured with elementid '37e31b7f-346f-4d0c-81d6-c691a59cf365' but no matching script tag was found. \")\n",
              "      return false;\n",
              "    }\n",
              "  \n",
              "    var js_urls = [];\n",
              "  \n",
              "    var inline_js = [\n",
              "      function(Bokeh) {\n",
              "        (function() {\n",
              "          var fn = function() {\n",
              "            var docs_json = {\"c80568a6-d64c-4b05-8a90-8cf210297be4\":{\"roots\":{\"references\":[{\"attributes\":{},\"id\":\"3e8aa621-90d5-412d-9c7c-2d001d92fad5\",\"type\":\"SaveTool\"},{\"attributes\":{},\"id\":\"287978ff-df1b-484f-b1b5-b5540516c639\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{\"callback\":null},\"id\":\"df27defe-d1e5-4fc5-9c31-071ec3ba4cd9\",\"type\":\"DataRange1d\"},{\"attributes\":{\"callback\":null,\"column_names\":[\"x1\",\"x2\",\"names\"],\"data\":{\"names\":[\"the\",\"or\",\"of\",\"to\",\"and\",\"in\",\"lessee\",\"shall\",\"any\",\"by\",\"a\",\"lease\",\"this\",\"be\",\"lessor\",\"such\",\"for\",\"on\",\"is\",\"as\",\"other\",\"production\",\"premises\",\"all\",\"not\",\"leased\",\"well\",\"s\",\"gas\",\"from\",\"which\",\"oil\",\"at\",\"that\",\"with\",\"hereunder\",\"if\",\"lands\",\"operations\",\"may\",\"are\",\"unit\",\"land\",\"interest\",\"pooled\",\"wells\",\"no\",\"shut\",\"an\",\"covered\",\"royalties\",\"more\",\"said\",\"therewith\",\"then\",\"its\",\"time\",\"royalty\",\"have\",\"part\",\"under\",\"term\",\"so\",\"after\",\"hereby\",\"rights\",\"drilling\",\"but\",\"being\",\"substances\",\"has\",\"producing\",\"including\",\"than\",\"herein\",\"right\",\"provided\",\"it\",\"paying\",\"days\",\"date\",\"price\",\"acreage\",\"quantities\",\"texas\",\"same\",\"thereafter\",\"one\",\"above\",\"before\",\"terms\",\"there\",\"pay\",\"period\",\"paid\",\"primary\",\"less\",\"produced\",\"payment\",\"pooling\",\"completion\",\"otherwise\",\"force\",\"within\",\"upon\",\"made\",\"authority\",\"written\",\"whether\",\"obligations\",\"ofthe\",\"day\",\"event\",\"acres\",\"record\",\"end\",\"either\",\"surface\",\"described\",\"instrument\",\"thereof\",\"provisions\",\"portion\",\"each\",\"notice\",\"option\",\"horizontal\",\"reworking\",\"depository\",\"field\",\"breach\",\"i\",\"market\",\"sold\",\"prevailing\",\"agrees\",\"additional\",\"use\",\"payments\",\"respect\",\"successors\",\"assigns\",\"mineral\",\"proportion\",\"governmental\",\"been\",\"county\",\"parties\",\"records\",\"long\",\"purpose\",\"address\",\"effective\",\"property\",\"hereof\",\"state\",\"necessary\",\"party\",\"cessation\",\"obligation\",\"conditions\",\"ownership\",\"feet\",\"reasonable\",\"reasonably\",\"order\",\"deemed\",\"facilities\",\"except\",\"interests\",\"until\",\"area\",\"drill\",\"credit\",\"means\",\"equipment\",\"paragraph\",\"full\",\"contained\",\"included\",\"without\",\"purchase\",\"similar\",\"tarrant\",\"maintained\",\"law\",\"addition\",\"water\",\"capable\",\"public\",\"estate\",\"executed\",\"describing\",\"gross\",\"prescribed\",\"only\",\"default\",\"release\",\"bonus\",\"consideration\",\"will\",\"per\",\"number\",\"tender\",\"were\",\"therein\",\"revision\",\"effect\",\"therefrom\",\"marketing\",\"b\",\"expiration\",\"following\",\"whole\",\"used\",\"notwithstanding\",\"pursuant\",\"now\",\"due\",\"change\",\"title\",\"remain\",\"their\",\"cause\",\"terminate\",\"obtain\",\"between\",\"when\",\"payable\",\"up\",\"provision\",\"percent\",\"develop\",\"offer\",\"was\",\"conform\",\"costs\",\"file\",\"failure\",\"amount\",\"transferred\",\"me\",\"declaration\",\"includes\",\"contrary\",\"rules\",\"formed\",\"net\",\"person\",\"another\",\"limited\",\"into\",\"regulations\",\"recorded\",\"undivided\",\"stating\",\"do\",\"below\",\"c\",\"consecutive\",\"nearest\",\"twenty\",\"laws\",\"jurisdiction\",\"purposes\",\"during\",\"applicable\",\"conducted\",\"sale\",\"t\",\"remove\",\"first\",\"total\",\"anything\",\"plus\",\"having\",\"arising\",\"heirs\",\"nevertheless\",\"minerals\",\"liable\",\"binding\",\"exceed\",\"termination\",\"proper\",\"based\",\"acknowledged\",\"your\",\"ofthis\",\"notary\",\"hereafter\",\"located\",\"prevented\",\"exploring\",\"five\",\"out\",\"covenants\",\"tolerance\",\"taxes\",\"implied\",\"filed\",\"prior\",\"existing\",\"permitted\",\"hundred\",\"constitute\",\"years\",\"two\",\"l\",\"depth\",\"you\",\"orders\",\"developing\",\"receive\",\"maximum\",\"depths\",\"transfers\",\"accordance\",\"however\",\"given\",\"proceeds\",\"writing\",\"owns\",\"unless\",\"expressly\",\"would\",\"form\",\"deliver\",\"easements\",\"entitled\",\"express\",\"original\",\"hole\",\"bears\",\"initial\",\"materials\",\"should\",\"delayed\",\"along\",\"ninety\",\"determination\",\"calculated\",\"strike\",\"dry\",\"specified\",\"subject\",\"spacing\",\"division\",\"commencement\",\"lessors\",\"separator\",\"named\",\"operate\",\"subsurface\",\"respective\",\"action\",\"result\",\"request\",\"wellhead\",\"cubic\",\"inability\",\"commences\",\"persons\",\"ratio\",\"executors\",\"relieved\",\"reduced\",\"signatory\",\"against\",\"saved\",\"continuing\",\"control\",\"delay\",\"extent\",\"hereinafter\",\"barrel\",\"density\",\"tenders\",\"hydrocarbons\",\"transport\",\"while\",\"administrators\",\"grants\",\"component\",\"entered\",\"page\",\"covers\",\"leases\",\"agreed\",\"whose\",\"hydrocarbon\",\"name\",\"drilled\",\"because\",\"reason\",\"liquid\",\"city\",\"products\",\"acre\",\"adjacent\",\"worth\",\"execute\",\"cash\",\"last\",\"material\",\"devisees\",\"prevention\",\"foregoing\",\"take\",\"processing\",\"zones\",\"hazardous\",\"d\",\"receipt\",\"vicinity\",\"thereon\",\"extend\",\"depending\",\"real\",\"covering\",\"commercial\",\"furnished\",\"owned\",\"granted\",\"through\",\"pool\",\"agent\",\"called\",\"operator\",\"consent\",\"plat\",\"ten\",\"hereinabove\",\"designated\",\"r\",\"entire\",\"requirements\",\"o\",\"separate\",\"fort\",\"e\",\"he\",\"geophysical\",\"execution\",\"environmental\",\"equivalent\",\"institution\",\"official\",\"remedy\",\"j\",\"invalid\",\"excluded\",\"acknowledges\",\"testing\",\"claim\",\"tracts\",\"services\",\"natural\",\"bore\",\"required\",\"purchaser\",\"expenses\",\"n\",\"satisfactory\",\"value\",\"caused\",\"certified\",\"incurred\",\"corporation\",\"obtaining\",\"pattern\",\"gases\",\"pipelines\",\"transportation\",\"both\",\"tract\",\"rental\",\"liability\",\"partial\",\"decedent\",\"filing\",\"transferee\",\"conduct\",\"vertical\",\"information\",\"construction\",\"description\",\"instruments\",\"restrictions\",\"printed\",\"according\",\"pipeline\",\"owners\",\"retained\",\"appropriate\",\"federal\",\"anniversary\",\"accept\",\"known\",\"engaged\",\"shalt\",\"lot\",\"extended\",\"unitized\",\"interval\",\"also\",\"damage\",\"jointly\",\"delivered\",\"security\",\"agents\",\"defend\",\"documents\",\"prepared\",\"affect\",\"check\",\"cost\",\"non\",\"section\",\"social\",\"volume\",\"those\",\"contracts\",\"fails\",\"fees\",\"agreement\",\"judicial\",\"units\",\"lines\",\"releases\",\"sixty\",\"death\",\"follows\",\"released\",\"cross\",\"final\",\"complete\",\"extension\",\"virtue\",\"covenant\",\"p\",\"realized\",\"assigned\",\"incapable\",\"location\",\"set\",\"neither\",\"exclusively\",\"hand\",\"three\",\"tease\",\"actually\",\"highest\",\"access\",\"accurate\",\"month\",\"draft\",\"whereof\",\"prudent\",\"whenever\",\"witness\",\"received\",\"satisfaction\",\"preceding\",\"employees\",\"copies\",\"make\",\"treated\",\"representations\",\"portions\",\"duly\",\"recordable\",\"year\",\"ii\",\"lien\",\"occurred\",\"reservoir\",\"considered\",\"requested\",\"prosecuted\",\"itself\",\"boundaries\",\"unto\",\"deems\",\"produce\",\"delivery\",\"intended\",\"labor\",\"dollar\",\"majeure\",\"confidentiality\",\"gravity\",\"claims\",\"casing\",\"drills\",\"forth\",\"satisfy\",\"certain\",\"notify\",\"six\",\"revised\",\"situated\",\"court\",\"treating\",\"locations\",\"attached\",\"regardless\",\"test\",\"fail\",\"receiving\",\"using\",\"free\",\"fully\",\"correct\",\"apply\",\"meanings\",\"stated\",\"circumstances\",\"making\",\"fuel\",\"normal\",\"ordinary\",\"instances\",\"lets\",\"supplemental\",\"grade\",\"permanent\",\"good\",\"absence\",\"standard\",\"quality\",\"exhaust\",\"containing\",\"zone\",\"expansion\",\"affiliate\",\"permits\",\"deed\",\"tendered\",\"riot\",\"thereupon\",\"carbon\",\"definition\",\"tx\",\"hereto\",\"include\",\"dioxide\",\"block\",\"proportionately\",\"contiguous\",\"helium\",\"exceeds\",\"war\",\"separated\",\"tanks\",\"faith\",\"development\",\"revise\",\"held\",\"formations\",\"establishing\",\"electric\",\"immediately\",\"disposal\",\"reversion\",\"strips\",\"exercising\",\"proportionate\",\"streets\",\"contraction\",\"ceases\",\"ofany\",\"restore\",\"power\",\"continue\",\"flood\",\"company\",\"damages\",\"electricity\",\"adverse\",\"weather\",\"protect\",\"exists\",\"conveyance\",\"nor\",\"anywhere\",\"disputes\",\"drainage\",\"whom\",\"restoring\",\"render\",\"rebellion\",\"unenforceable\",\"hour\",\"restricts\",\"enlarging\",\"authenticated\",\"comparable\",\"sabotage\",\"warning\",\"assignment\",\"maintaining\",\"race\",\"recovery\",\"site\",\"usual\",\"u\",\"recurring\",\"insurrection\",\"least\",\"larger\",\"next\",\"devised\",\"exercised\",\"v\",\"fire\",\"th\",\"hold\",\"thirty\",\"casinghead\",\"discharge\",\"satisfied\",\"productive\",\"notification\",\"understood\",\"ofsuch\",\"roads\",\"determining\",\"dated\",\"grant\",\"acquire\",\"reducing\",\"negotiated\",\"go\",\"prudently\",\"commission\",\"entity\",\"carriers\",\"fixtures\",\"owner\",\"arlington\",\"f\",\"severance\",\"thereunder\",\"purchasers\",\"comply\",\"us\",\"color\",\"govemmental\",\"exploratory\",\"treat\",\"ad\",\"telephone\",\"limitation\",\"easement\",\"joint\",\"properly\",\"receives\",\"thereto\",\"stations\",\"separately\",\"seismic\",\"partnership\",\"warrants\",\"exercise\",\"liens\",\"outstanding\",\"contractors\",\"nature\",\"deepest\",\"voting\",\"times\",\"subrogated\",\"added\",\"currency\",\"behalf\",\"house\",\"determined\",\"llc\",\"these\",\"dale\",\"act\",\"computed\",\"established\",\"sum\",\"requiring\",\"evidence\",\"purchases\",\"dollars\",\"rentals\",\"product\",\"canceled\",\"mortgages\",\"permanently\",\"office\",\"prescription\",\"particularly\",\"survive\",\"point\",\"ratification\",\"arise\",\"amended\",\"regulation\",\"does\",\"residential\",\"related\",\"excise\",\"values\",\"strata\",\"gaseous\",\"litigation\",\"association\",\"warranty\",\"uncompensated\",\"w\",\"my\",\"canals\",\"teased\",\"changes\",\"bear\",\"m\",\"rate\",\"initiated\",\"where\",\"bury\",\"matter\",\"alter\",\"succeeded\",\"timber\",\"harmless\",\"negotiation\",\"store\",\"valorem\",\"kind\",\"parcels\",\"mail\",\"venture\",\"ross\",\"expires\",\"crops\",\"divided\",\"place\",\"plow\",\"herewith\",\"growing\",\"liabilities\",\"small\",\"amendment\",\"differing\",\"injection\",\"offered\",\"associated\",\"regarding\",\"liquidate\",\"exhibit\",\"plant\",\"run\",\"delivering\",\"accretions\",\"pertinent\",\"ofsaid\",\"noise\",\"barn\",\"litigated\",\"levied\",\"spaces\",\"naming\",\"assessed\",\"selected\",\"addressed\",\"bores\",\"becoming\",\"counterparts\",\"pits\",\"way\",\"together\",\"stamped\",\"refuse\",\"transporting\",\"offering\",\"conveyed\",\"suspend\",\"reimburse\",\"manner\",\"discover\",\"deposit\",\"forfeited\",\"off\",\"routes\",\"exploration\",\"bank\",\"sites\",\"envelope\",\"whichever\",\"egress\",\"violation\",\"inconsistent\",\"willing\",\"down\",\"reference\",\"exercises\",\"benefit\",\"alteration\",\"suite\",\"aforementioned\",\"code\",\"resolved\",\"like\",\"operation\",\"surveys\",\"aware\",\"improvements\",\"offeror\",\"placement\",\"preferred\",\"become\",\"welt\",\"ancillary\",\"directional\",\"negotiations\",\"actions\",\"mails\",\"ste\",\"pipe\",\"recited\",\"conveys\",\"sales\",\"multiple\",\"am\",\"presence\",\"perpetual\",\"shell\",\"ingress\",\"equal\",\"h\",\"attorney\",\"buildings\",\"welihead\",\"survey\",\"lb\",\"blank\",\"vary\",\"allocated\",\"formation\",\"cultivated\",\"recognizes\",\"interrupted\",\"substance\",\"liii\",\"actual\",\"could\",\"notices\",\"allowed\",\"ot\",\"bona\",\"legal\",\"conducting\",\"though\",\"construed\",\"understands\",\"seek\",\"stimulation\",\"k\",\"enhanced\",\"fracture\",\"hydraulic\",\"rule\",\"effectuate\",\"determine\",\"must\",\"different\",\"re\",\"gathering\",\"get\",\"over\",\"negotiate\",\"waiting\",\"share\",\"g\",\"causes\",\"commenced\",\"factors\",\"fee\",\"indemnify\",\"ave\",\"opportunity\",\"transaction\",\"structure\",\"pm\",\"available\",\"disclaimer\",\"continued\",\"sensitive\",\"ponds\",\"third\",\"representatives\",\"referred\",\"st\",\"provide\",\"y\"],\"x1\":{\"__ndarray__\":\"[encoded data, 5336 characters]\",\"dtype\":\"float32\",\"shape\":[1000]},\"x2\":{\"__ndarray__\":\"[encoded data, 5336 characters]\",\"dtype\":\"float32\",\"shape\":[1000]}}},\"id\":\"2518459f-d929-4207-aa95-0a4dd3d84564\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"data_source\":{\"id\":\"2518459f-d929-4207-aa95-0a4dd3d84564\",\"type\":\"ColumnDataSource\"},\"glyph\":{\"id\":\"c7c6b58b-d539-4877-965c-dee252655890\",\"type\":\"Circle\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"1a79a4e1-4efd-45c4-a5fc-d53e35461698\",\"type\":\"Circle\"},\"selection_glyph\":null,\"view\":{\"id\":\"081893a0-93f6-4225-8861-12aa78ba306f\",\"type\":\"CDSView\"}},\"id\":\"fb72a482-ea05-49c3-9520-9dc4a54fc8e5\",\"type\":\"GlyphRenderer\"},{\"attributes\":{},\"id\":\"9bf88e87-66c1-4cc0-a4ba-cb2ae61e42b7\",\"type\":\"LinearScale\"},{\"attributes\":{},\"id\":\"8d81bff3-20db-4695-83b5-faa12cacb96c\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{\"fill_alpha\":{\"value\":0.1},\"fill_color\":{\"value\":\"#1f77b4\"},\"line_alpha\":{\"value\":0.1},\"line_color\":{\"value\":\"#1f77b4\"},\"size\":{\"units\":\"screen\",\"value\":8},\"x\":{\"field\":\"x1\"},\"y\":{\"field\":\"x2\"}},\"id\":\"1a79a4e1-4efd-45c4-a5fc-d53e35461698\",\"type\":\"Circle\"},{\"attributes\":{\"plot\":{\"id\":\"2c330f72-ecff-415a-8347-c133135581d4\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"73777af8-148c-4aa3-9ee2-c80339387379\",\"type\":\"BasicTicker\"}},\"id\":\"c9aa612b-b5bc-4ca7-b66f-bf297dc69416\",\"type\":\"Grid\"},{\"attributes\":{\"fill_color\":{\"value\":\"#1f77b4\"},\"line_color\":{\"value\":\"#1f77b4\"},\"size\":{\"units\":\"screen\",\"value\":8},\"x\":{\"field\":\"x1\"},\"y\":{\"field\":\"x2\"}},\"id\":\"c7c6b58b-d539-4877-965c-dee252655890\",\"type\":\"Circle\"},{\"attributes\":{},\"id\":\"73777af8-148c-4aa3-9ee2-c80339387379\",\"type\":\"BasicTicker\"},{\"attributes\":{\"plot\":null,\"text\":\"word2vec T-SNE for most common words\"},\"id\":\"1daf392a-b730-4e22-a96c-d3849df467b4\",\"type\":\"Title\"},{\"attributes\":{\"callback\":null},\"id\":\"1a4536ba-e14a-4577-bade-fbe8916860f4\",\"type\":\"DataRange1d\"},{\"attributes\":{},\"id\":\"7d1adbf4-49bb-412f-85db-c8171bfcd3f3\",\"type\":\"WheelZoomTool\"},{\"attributes\":{\"active_drag\":\"auto\",\"active_inspect\":\"auto\",\"active_scroll\":\"auto\",\"active_tap\":\"auto\",\"tools\":[{\"id\":\"bb56cc13-76a6-4846-a151-39cb1999018f\",\"type\":\"PanTool\"},{\"id\":\"7d1adbf4-49bb-412f-85db-c8171bfcd3f3\",\"type\":\"WheelZoomTool\"},{\"id\":\"63130cbd-e6b8-4434-bc49-ec7a7b4a1679\",\"type\":\"ResetTool\"},{\"id\":\"3e8aa621-90d5-412d-9c7c-2d001d92fad5\",\"type\":\"SaveTool\"}]},\"id\":\"8b38589d-3161-4517-94dd-daefd031c61a\",\"type\":\"Toolbar\"},{\"attributes\":{\"plot\":{\"id\":\"2c330f72-ecff-415a-8347-c133135581d4\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"source\":{\"id\":\"2518459f-d929-4207-aa95-0a4dd3d84564\",\"type\":\"ColumnDataSource\"},\"text\":{\"field\":\"names\"},\"text_align\":\"center\",\"text_color\":{\"value\":\"#555555\"},\"text_font_size\":{\"value\":\"8pt\"},\"x\":{\"field\":\"x1\"},\"y\":{\"field\":\"x2\"},\"y_offset\":{\"value\":6}},\"id\":\"62fa1bb0-dc86-4d16-b260-1886ee5c8832\",\"type\":\"LabelSet\"},{\"attributes\":{\"dimension\":1,\"plot\":{\"id\":\"2c330f72-ecff-415a-8347-c133135581d4\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"97c035d4-c9c8-4ee3-8ed7-2d7bd4f507ca\",\"type\":\"BasicTicker\"}},\"id\":\"388c898d-3bca-4bf7-b457-3db71cb51183\",\"type\":\"Grid\"},{\"attributes\":{\"formatter\":{\"id\":\"287978ff-df1b-484f-b1b5-b5540516c639\",\"type\":\"BasicTickFormatter\"},\"plot\":{\"id\":\"2c330f72-ecff-415a-8347-c133135581d4\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"73777af8-148c-4aa3-9ee2-c80339387379\",\"type\":\"BasicTicker\"}},\"id\":\"34293422-dae9-4452-b6e0-62ee9b8f1f6c\",\"type\":\"LinearAxis\"},{\"attributes\":{},\"id\":\"bb56cc13-76a6-4846-a151-39cb1999018f\",\"type\":\"PanTool\"},{\"attributes\":{},\"id\":\"97c035d4-c9c8-4ee3-8ed7-2d7bd4f507ca\",\"type\":\"BasicTicker\"},{\"attributes\":{},\"id\":\"96372a52-0148-42b9-bba6-796f4ea93bc9\",\"type\":\"LinearScale\"},{\"attributes\":{\"source\":{\"id\":\"2518459f-d929-4207-aa95-0a4dd3d84564\",\"type\":\"ColumnDataSource\"}},\"id\":\"081893a0-93f6-4225-8861-12aa78ba306f\",\"type\":\"CDSView\"},{\"attributes\":{},\"id\":\"63130cbd-e6b8-4434-bc49-ec7a7b4a1679\",\"type\":\"ResetTool\"},{\"attributes\":{\"formatter\":{\"id\":\"8d81bff3-20db-4695-83b5-faa12cacb96c\",\"type\":\"BasicTickFormatter\"},\"plot\":{\"id\":\"2c330f72-ecff-415a-8347-c133135581d4\",\"subtype\":\"Figure\",\"type\":\"Plot\"},\"ticker\":{\"id\":\"97c035d4-c9c8-4ee3-8ed7-2d7bd4f507ca\",\"type\":\"BasicTicker\"}},\"id\":\"c3a2b682-2b82-441e-a4f2-b8c0b09075f6\",\"type\":\"LinearAxis\"},{\"attributes\":{\"below\":[{\"id\":\"34293422-dae9-4452-b6e0-62ee9b8f1f6c\",\"type\":\"LinearAxis\"}],\"left\":[{\"id\":\"c3a2b682-2b82-441e-a4f2-b8c0b09075f6\",\"type\":\"LinearAxis\"}],\"renderers\":[{\"id\":\"34293422-dae9-4452-b6e0-62ee9b8f1f6c\",\"type\":\"LinearAxis\"},{\"id\":\"c9aa612b-b5bc-4ca7-b66f-bf297dc69416\",\"type\":\"Grid\"},{\"id\":\"c3a2b682-2b82-441e-a4f2-b8c0b09075f6\",\"type\":\"LinearAxis\"},{\"id\":\"388c898d-3bca-4bf7-b457-3db71cb51183\",\"type\":\"Grid\"},{\"id\":\"fb72a482-ea05-49c3-9520-9dc4a54fc8e5\",\"type\":\"GlyphRenderer\"},{\"id\":\"62fa1bb0-dc86-4d16-b260-1886ee5c8832\",\"type\":\"LabelSet\"}],\"title\":{\"id\":\"1daf392a-b730-4e22-a96c-d3849df467b4\",\"type\":\"Title\"},\"toolbar\":{\"id\":\"8b38589d-3161-4517-94dd-daefd031c61a\",\"type\":\"Toolbar\"},\"toolbar_location\":\"above\",\"x_range\":{\"id\":\"1a4536ba-e14a-4577-bade-fbe8916860f4\",\"type\":\"DataRange1d\"},\"x_scale\":{\"id\":\"9bf88e87-66c1-4cc0-a4ba-cb2ae61e42b7\",\"type\":\"LinearScale\"},\"y_range\":{\"id\":\"df27defe-d1e5-4fc5-9c31-071ec3ba4cd9\",\"type\":\"DataRange1d\"},\"y_scale\":{\"id\":\"96372a52-0148-42b9-bba6-796f4ea93bc9\",\"type\":\"LinearScale\"}},\"id\":\"2c330f72-ecff-415a-8347-c133135581d4\",\"subtype\":\"Figure\",\"type\":\"Plot\"}],\"root_ids\":[\"2c330f72-ecff-415a-8347-c133135581d4\"]},\"title\":\"Bokeh Application\",\"version\":\"0.12.7\"}};\n",
              "            var render_items = [{\"docid\":\"c80568a6-d64c-4b05-8a90-8cf210297be4\",\"elementid\":\"37e31b7f-346f-4d0c-81d6-c691a59cf365\",\"modelid\":\"2c330f72-ecff-415a-8347-c133135581d4\"}];\n",
              "            \n",
              "            Bokeh.embed.embed_items(docs_json, render_items);\n",
              "          };\n",
              "          if (document.readyState != \"loading\") fn();\n",
              "          else document.addEventListener(\"DOMContentLoaded\", fn);\n",
              "        })();\n",
              "      },\n",
              "      function(Bokeh) {\n",
              "      }\n",
              "    ];\n",
              "  \n",
              "    function run_inline_js() {\n",
              "      \n",
              "      if ((root.Bokeh !== undefined) || (force === true)) {\n",
              "        for (var i = 0; i < inline_js.length; i++) {\n",
              "          inline_js[i].call(root, root.Bokeh);\n",
              "        }if (force === true) {\n",
              "          display_loaded();\n",
              "        }} else if (Date.now() < root._bokeh_timeout) {\n",
              "        setTimeout(run_inline_js, 100);\n",
              "      } else if (!root._bokeh_failed_load) {\n",
              "        console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n",
              "        root._bokeh_failed_load = true;\n",
              "      } else if (force !== true) {\n",
              "        var cell = $(document.getElementById(\"37e31b7f-346f-4d0c-81d6-c691a59cf365\")).parents('.cell').data().cell;\n",
              "        cell.output_area.append_execute_result(NB_LOAD_WARNING)\n",
              "      }\n",
              "  \n",
              "    }\n",
              "  \n",
              "    if (root._bokeh_is_loading === 0) {\n",
              "      console.log(\"Bokeh: BokehJS loaded, going straight to plotting\");\n",
              "      run_inline_js();\n",
              "    } else {\n",
              "      load_libs(js_urls, function() {\n",
              "        console.log(\"Bokeh: BokehJS plotting callback run at\", now());\n",
              "        run_inline_js();\n",
              "      });\n",
              "    }\n",
              "  }(window));\n",
              "</script>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "JGQxj62SkTLj",
        "colab_type": "code",
        "colab": {},
        "outputId": "85128337-89e5-4a44-ec6f-94b6d14ee59a"
      },
      "cell_type": "code",
      "source": [
        "print(\"January: \", model_leases.most_similar(\"january\"))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "January:  [('november', 0.9086244106292725), ('june', 0.898212730884552), ('december', 0.8965006470680237), ('february', 0.8936348557472229), ('august', 0.8898540735244751), ('october', 0.8842123746871948), ('april', 0.8836274147033691), ('march', 0.8824495673179626), ('july', 0.8805142641067505), ('september', 0.8302119374275208)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "wSdozZNXkTLp",
        "colab_type": "code",
        "colab": {},
        "outputId": "bd733e55-4694-4d90-e047-1fe151316d22"
      },
      "cell_type": "code",
      "source": [
        "print(\"february: \", model_leases.most_similar(\"february\"))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "february:  [('june', 0.9295380711555481), ('august', 0.8984723091125488), ('january', 0.8936350345611572), ('march', 0.8927676677703857), ('july', 0.8921082615852356), ('december', 0.8764469027519226), ('october', 0.8713834285736084), ('april', 0.871055006980896), ('november', 0.8623570799827576), ('september', 0.8598427772521973)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "h4IX-OGgkTLs",
        "colab_type": "code",
        "colab": {},
        "outputId": "4221bd7a-77a2-450e-f935-f0edfb454e7f"
      },
      "cell_type": "code",
      "source": [
        "print(\"hours: \", model_leases.most_similar(\"hours\"))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "hours:  [('appeals', 0.4939519464969635), ('times', 0.49231672286987305), ('offices', 0.4722555875778198), ('pdnazy', 0.46289709210395813), ('normal', 0.4611148238182068), ('jltt', 0.4558117687702179), ('conforming', 0.4435980021953583), ('soon', 0.43607157468795776), ('view', 0.4327104091644287), ('horizorital', 0.4219496250152588)]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "BAXUw9q1kTLu",
        "colab_type": "code",
        "colab": {},
        "outputId": "1ca3a349-e4b7-4319-e694-c4d5ad53ac8c"
      },
      "cell_type": "code",
      "source": [
        "print(\"minutes: \", model_leases.most_similar(\"minutes\"))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "minutes:  [('seconds', 0.9467544555664062), ('degrees', 0.9440059661865234), ('west', 0.9167031049728394), ('thence', 0.910275936126709), ('east', 0.8943793773651123), ('south', 0.8640549778938293), ('north', 0.8484262824058533), ('weatherford', 0.7775397896766663), ('davis', 0.7715314030647278), ('inch', 0.7621535062789917)]\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}