{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Info\n",
    "\n",
    "This notebook is designed to perform portoflio optimization using **Polici Gradient** reinforcement learnign algorithm.\n",
    "\n",
    "Main paper used as guidance [A Deep Reinforcement Learning Framework for the\n",
    "Financial Portfolio Management Problem](https://arxiv.org/pdf/1706.10059.pdf)\n",
    "\n",
    "## Describe Finance Environment\n",
    "\n",
    "* Always start with the closing price `v` and calculating the **price relative vector** as `v[t]/v[t-1]` .\n",
    "* Define a `state` / `observation` as being the **vector price** as `yt = [1, v['stock_1'][t] / v['stock_2'][t-1], ..] ` along with the current assigned portfolio weights `w[t-1]`.\n",
    "* Define an action as the values assigned for `w[t]` of values between `0-1` that sum up to `1`.\n",
    "* Define the reward in two ways:\n",
    "    * Average Portfolio logarithmic rate of return `r[t] = np.log(y[t] * w[t-1])`.\n",
    "    * Sharpe ratio using portfolio logarithmic rate of return.\n",
    "\n",
    "\n",
    "## Notations and Formulas:\n",
    "\n",
    "`p0` - Initial investment amount.\n",
    "\n",
    "`m` - Number of assets in portfolio.\n",
    "\n",
    "`w` - Portfolio weights(vector of sum 1). Represents how much does each asset count.\n",
    "\n",
    "`v` - Asset closing prices (list).\n",
    "\n",
    "`v[t]` - Asset closing price at time `t`\n",
    "\n",
    "`y` - Price relative vector(matrix).\n",
    "\n",
    "`y[t]` - Price relative vector at time `t` that start with cash value of `1`.\n",
    "\n",
    "`p` - Portfolio value(list). Is calculated as the **dot product** between `y` and `w`. \n",
    "\n",
    "`r` - Logarithmic rate of return. \n",
    "\n",
    "`avg_r` -  Avergae logarithmic cumulated return. Is basically the average of the logarithmic return.\n",
    "\n",
    "`sharpe_ratio` - Average(`r`) / Standard Deviation(`r`).\n",
    "\n",
    "\n",
    "\n",
    "## Sharpe Ratio\n",
    "\n",
    "The Sharpe ratio is a commonly used indicator to measure the risk adjusted performance of an investment over time. Assuming a risk-free rate of 0, the formula for computing Sharpe ratio is simply the mean returns of the investment divided by the standard deviation of the returns. This can be written as:\n",
    "\n",
    "$S _T = {A \\over \\sqrt{B - A^2}}$\n",
    "\n",
    "where $A={1\\over T}\\sum\\limits _{t=1}^{T}R _t$, and $B={1\\over T}\\sum\\limits _{t=1}^{T}R _t^2$\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('1.4.0', device(type='cpu'))"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from tqdm.notebook import tqdm\n",
    "from IPython import display\n",
    "import torch\n",
    "import matplotlib.dates as mdates\n",
    "\n",
    "%matplotlib inline\n",
    "plt.style.use('ggplot')\n",
    "torch.manual_seed(123)\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "\n",
    "torch.__version__, device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Read Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read stock data\n",
    "stock_df = pd.read_csv(\"\")\n",
    "\n",
    "# convert 'Date' to datetime\n",
    "stock_df['Date'] = pd.to_datetime(stock_df['Date'])\n",
    "\n",
    "# set as index 'Date' columns\n",
    "stock_df = stock_df.set_index('Date')\n",
    "\n",
    "# show head\n",
    "stock_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sample Example\n",
    "\n",
    "Math formulas sanity checks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Same weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 14908 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15.799927600344615\n"
     ]
    }
   ],
   "source": [
    "# initial investment\n",
    "p0 = 1\n",
    "# intial weights - fixed for this example\n",
    "w = np.array([0.2, 0.3, 0.5])\n",
    "# start with fewer stocks\n",
    "ret_data = stock_df[[]].copy(deep=True)\n",
    "# length of data\n",
    "T = len(ret_data)\n",
    "# add cash values \n",
    "ret_data['CASH'] = np.ones(T)\n",
    "# convert to numpy for easy computation\n",
    "ret_data = ret_data.to_numpy()\n",
    "# calculate price vector\n",
    "y = ret_data[1:T] / ret_data[0:T-1]\n",
    "# weighted return\n",
    "r = np.dot(y, w)\n",
    "# plot cumulative weighted return\n",
    "plt.plot(np.cumprod(r))\n",
    "plt.show()\n",
    "# final portfolio value\n",
    "pf = p0*np.exp(np.sum(np.log(r)))\n",
    "print(pf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Equal Weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 14384 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "27.740783335359666\n"
     ]
    }
   ],
   "source": [
    "# initial investment\n",
    "p0 = 1\n",
    "# intial weights - fixed for this example\n",
    "# w = np.array([0.2, 0.3, 0.5])\n",
    "w = np.array([1/3]*3)\n",
    "# start with fewer stocks\n",
    "ret_data = stock_df[[]].copy(deep=True)\n",
    "# length of data\n",
    "T = len(ret_data)\n",
    "# add cash values \n",
    "ret_data['CASH'] = np.ones(T)\n",
    "# convert to numpy for easy computation\n",
    "ret_data = ret_data.to_numpy()\n",
    "# calculate price vector\n",
    "y = ret_data[1:T] / ret_data[0:T-1]\n",
    "# weighted return\n",
    "r = np.dot(y, w)\n",
    "# plot cumulative weighted return\n",
    "plt.plot(np.cumprod(r))\n",
    "plt.show()\n",
    "# final portfolio value\n",
    "pf = p0*np.exp(np.sum(np.log(r)))\n",
    "print(pf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Random Weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 15920 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "17.772327664198578\n"
     ]
    }
   ],
   "source": [
    "# initial investment\n",
    "p0 = 1\n",
    "# start with fewer stocks\n",
    "ret_data = stock_df[[]].copy(deep=True)\n",
    "# length of data\n",
    "T = len(ret_data)\n",
    "# add cash values \n",
    "ret_data['CASH'] = np.ones(T)\n",
    "# convert to numpy for easy computation\n",
    "ret_data = ret_data.to_numpy()\n",
    "# calculate price vector\n",
    "y = ret_data[1:T] / ret_data[0:T-1]\n",
    "# intial weights - fixed for this example\n",
    "w = np.random.dirichlet(np.ones(3),size=len(y))\n",
    "# weighted return\n",
    "r = np.sum((y*w), axis=1)\n",
    "# r = np.dot(y, w)\n",
    "# plot cumulative weighted return\n",
    "plt.plot(np.cumprod(r))\n",
    "plt.show()\n",
    "# final portfolio value\n",
    "pf = p0*np.exp(np.sum(np.log(r)))\n",
    "print(pf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Price Vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.        , 1.        , 1.        ],\n",
       "       [1.        , 1.02064896, 1.03251997],\n",
       "       [1.        , 1.02890191, 1.02362262],\n",
       "       [1.        , 1.00842742, 0.92692276],\n",
       "       [1.        , 0.99442846, 0.9875519 ],\n",
       "       [1.        , 0.99439808, 1.00420174],\n",
       "       [1.        , 1.00281664, 1.00418422],\n",
       "       [1.        , 1.01123559, 1.        ],\n",
       "       [1.        , 0.98333401, 0.99166649],\n",
       "       [1.        , 0.98022553, 1.00840354]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# portfolio assets\n",
    "assets = []\n",
    "\n",
    "# number of assets + cash\n",
    "m = len(assets) + 1\n",
    "\n",
    "# length of data\n",
    "t = len(stock_df)\n",
    "\n",
    "# numpy data\n",
    "stock_np = stock_df[assets].to_numpy()\n",
    "\n",
    "# cash column\n",
    "cash_column = np.ones((1,t)).T\n",
    "\n",
    "# add cash column\n",
    "stock_np = np.concatenate((cash_column, stock_np), axis=1)\n",
    "\n",
    "# calculate price vector\n",
    "y = stock_np[1:t] / stock_np[0:t-1]\n",
    "\n",
    "# set first row to 1\n",
    "y[0] = np.ones(m)\n",
    "\n",
    "y[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Split Price Vector in Train-Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train data: (5095, 3)\n",
      "Test data:  (1800, 3)\n"
     ]
    }
   ],
   "source": [
    "# use for traiding\n",
    "n_trade = 1800\n",
    "\n",
    "# used for training\n",
    "train_data = y[:-n_trade]\n",
    "# used for trading\n",
    "test_data = y[-n_trade:]\n",
    "\n",
    "# keep track of dates for plot purposes\n",
    "dates = [stock_df[1:-n_trade].index[0], stock_df[-n_trade:].index[0]]\n",
    "\n",
    "print(\"Train data:\", train_data.shape)\n",
    "print(\"Test data: \", test_data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyTorch Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PortfolioModel(torch.nn.Module):\n",
    "    def __init__(self, input_dim, n_assets, lr=0.3):\n",
    "        super(PortfolioModel, self).__init__()\n",
    "        self.theta = torch.nn.Linear(in_features=input_dim, out_features=n_assets, bias=False)\n",
    "        self.potfolio_weights = torch.nn.Softmax(dim=1)\n",
    "        \n",
    "        self.optimizer = torch.optim.SGD(self.parameters(), lr=lr, momentum=0.0)\n",
    "        self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "        self.to(self.device)\n",
    "\n",
    "    def forward(self, observation):\n",
    "        state = torch.Tensor(observation).to(self.device)\n",
    "        logits = self.theta(state)\n",
    "        weights = self.potfolio_weights(logits)\n",
    "        return weights\n",
    "    \n",
    "\n",
    "def portfolio_weights(torch_model, normalized_data, number_assets):\n",
    "    \"\"\"Get agents predicted portofolio weights for each time step\n",
    "    \"\"\"\n",
    "    # number of examples in data\n",
    "    n_examples = len(normalized_data)\n",
    "    # initialize portfolio weights\n",
    "    portfolio_weights = np.zeros((n_examples, number_assets))\n",
    "    # intialize first weights\n",
    "    portfolio_weights[0][0] = 1.\n",
    "    # get all predicted weights\n",
    "    with torch.no_grad():\n",
    "        for i in range(n_examples-1):\n",
    "            # get input\n",
    "            input_sequence = np.concatenate((normalized_data[i], portfolio_weights[i]), axis=0)\n",
    "            # update weights for next time-step\n",
    "            portfolio_weights[i+1] = model(np.expand_dims(input_sequence, axis=0))\n",
    "    \n",
    "    return portfolio_weights\n",
    "\n",
    "\n",
    "def mean_return_loss(model_output, train_data, device='cpu'):\n",
    "    \"\"\"Agent reward as calculated in the paper\n",
    "    \"\"\"\n",
    "    # move data to tensor\n",
    "    tensor_data = torch.tensor(train_data, dtype=torch.float32, device=device)\n",
    "    # perform dot product\n",
    "    # element wise multiplication\n",
    "    weighted_return = torch.mul(tensor_data, model_output)\n",
    "    # sum on row axis\n",
    "    weighted_return = torch.sum(weighted_return, dim=1)\n",
    "    # final reward is sum of all rewards\n",
    "    # negative sign is for gradient ascent training\n",
    "    loss = -1. * torch.sum(torch.log(weighted_return))\n",
    "    # return mean of weighted return with negative sign\n",
    "    # return list of weighted returns\n",
    "    return loss, weighted_return.detach().numpy()\n",
    "\n",
    "\n",
    "def train(torch_model, train_data, test_data, epochs=1, evaluate_step=2, device='cpu'):\n",
    "    number_assets = len(train_data[0])\n",
    "    train_loss = []\n",
    "    # baseline model - keep same weights\n",
    "    base_w = np.array([1/number_assets]*number_assets)\n",
    "    base_return = np.sum((test_data*base_w), axis=1)\n",
    "    for epoch in range(epochs):\n",
    "        # forward pass to get portfolio weights\n",
    "        w = portfolio_weights(torch_model, train_data, number_assets)\n",
    "        # get train data\n",
    "        forward_data = np.concatenate((train_data, w), axis=1)\n",
    "        # move tensor to device\n",
    "        forward_data = torch.tensor(forward_data, dtype=torch.float32, device=device)\n",
    "        # set model for training\n",
    "        torch_model.train()\n",
    "        # reset optimizer\n",
    "        torch_model.optimizer.zero_grad()\n",
    "        # forward pass\n",
    "        outputs = torch_model(forward_data)\n",
    "        # get loss as portfolio return\n",
    "        loss, weighted_returns = mean_return_loss(outputs, train_data, device=device)\n",
    "        # backward pass\n",
    "        loss.backward()\n",
    "        # optimize step\n",
    "        torch_model.optimizer.step()\n",
    "        # keep track of loss (sharpe ratio)\n",
    "        # remember this was negative so need to make it positive again\n",
    "        train_loss.append(-loss.item())\n",
    "        \n",
    "        # verbose\n",
    "        if epoch % evaluate_step == 0 or (epoch == (epochs-1)):\n",
    "            # forward pass to get portfolio weights\n",
    "            test_w = portfolio_weights(torch_model, test_data, number_assets)\n",
    "            # weighted return\n",
    "            test_weighted_returns = np.sum((test_data*test_w), axis=1)\n",
    "            # display evaluation steps\n",
    "            display.clear_output(wait=True)\n",
    "            plt.plot(np.cumprod(test_weighted_returns), label=\"RL Optimization\", linewidth=1)\n",
    "            plt.plot(np.cumprod(base_return), label=\"Same Weights\", linewidth=1)\n",
    "            plt.title(\"Epoch %d/%d\"%(epoch, epochs))\n",
    "            plt.legend()\n",
    "            print(\"weights: \", test_w)\n",
    "            display.display(plt.show())\n",
    "    \n",
    "    return weighted_returns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train\n",
    "\n",
    "Note: \n",
    "* No need to standardize data?\n",
    "* it looks like it's not training.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "weights:  [[1.         0.         0.        ]\n",
      " [0.15449794 0.21031621 0.63518578]\n",
      " [0.13433141 0.17144799 0.6942206 ]\n",
      " ...\n",
      " [0.13470101 0.16914947 0.69614947]\n",
      " [0.13587937 0.17022809 0.69389254]\n",
      " [0.13488737 0.169294   0.6958186 ]]\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 31920 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "None"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# standardize data\n",
    "std = np.std(train_data)\n",
    "mean = np.mean(train_data)\n",
    "\n",
    "y_train = (train_data - mean) / std\n",
    "y_test = (test_data - mean) / std\n",
    "\n",
    "\n",
    "input_dim = m*2\n",
    "\n",
    "# build portfolio optimization model\n",
    "model = PortfolioModel(input_dim=input_dim, n_assets=m, lr=0.01)\n",
    "\n",
    "rt = train(torch_model=model, train_data=train_data, test_data=test_data, epochs=50, evaluate_step=2)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "name": "Trading with Reinforcement Learning in Python Part II: Application"
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
