{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "tensorflow loss functions.ipynb",
"provenance": [],
"authorship_tag": "ABX9TyO2FwWROZRNnJTZM0VU++oV",
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Pjo2jozZ61qT"
},
"source": [
"# Tensorflow Keras Loss Functions\r\n",
"``tf.keras.losses\r\n",
"``\r\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bWnrTwy-2pjn"
},
"source": [
"# Probabilistic Losses\r\n",
"* BinaryCrossentropy \r\n",
"* CategoricalCrossentropy \r\n",
"* SparseCategoricalCrossentropy \r\n",
"* Poisson \r\n",
"* KLDivergence "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Pg20dnZYjxlp"
},
"source": [
"## BinaryCrossentropy loss\r\n",
"```\r\n",
"tf.keras.losses.BinaryCrossentropy\r\n",
"```\r\n",
"\r\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "9YS__A9kkepM"
},
"source": [
"import tensorflow as tf\r\n",
"import numpy as np"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "eW83N_wxjmgw",
"outputId": "60ff1293-05a3-48aa-a550-0a72e29e1bbc"
},
"source": [
"y_true = [[0., 1.], [0., 0.]]\r\n",
"y_pred = [[0.5, 0.4], [0.4, 0.5]]\r\n",
"\r\n",
"# Using 'auto'/'sum_over_batch_size' reduction type\r\n",
"bce_loss = tf.keras.losses.BinaryCrossentropy()\r\n",
"bce_loss(y_true, y_pred).numpy()"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.70335245"
]
},
"metadata": {
"tags": []
},
"execution_count": 57
}
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "dZL5sAiRlGXf",
"outputId": "7c7e2591-e92b-402b-8cc3-f9dae60e96e5"
},
"source": [
"# Calling with 'sample_weight'.\r\n",
"bce_loss(y_true, y_pred, sample_weight=[1, 0]).numpy()"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.40235937"
]
},
"metadata": {
"tags": []
},
"execution_count": 58
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ilNm02z9lBkh"
},
"source": [
"## CategoricalCrossentropy loss\r\n",
"`tf.keras.losses.CategoricalCrossentropy`"
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "yGwl2wyqkhtC",
"outputId": "9f9d058c-8264-4d73-bc78-c4808a748f59"
},
"source": [
"# inputs\r\n",
"y_true = [[0, 1, 0], [0, 0, 1]]\r\n",
"y_pred = [[0.05, 0.95, 0], [0.1, 0.8, 0.1]]\r\n",
"\r\n",
"# Using 'auto'/'sum_over_batch_size' reduction type.\r\n",
"cce_loss = tf.keras.losses.CategoricalCrossentropy()\r\n",
"cce_loss(y_true, y_pred).numpy()\r\n"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"1.1769392"
]
},
"metadata": {
"tags": []
},
"execution_count": 59
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sm8Bmm7Bt9Eo"
},
"source": [
"## sparse_categorical_crossentropy\r\n",
"```\r\n",
"tf.keras.losses.sparse_categorical_crossentropy\r\n",
"```"
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "gfPX_yNMtOAI",
"outputId": "7a0cbea5-3978-4f84-85d8-b550ce3d2f6f"
},
"source": [
"y_true = [1, 2]\r\n",
"y_pred = [[0.05, 0.95, 0], [0.1, 0.8, 0.1]]\r\n",
"loss = tf.keras.losses.sparse_categorical_crossentropy(y_true, y_pred)\r\n",
"assert loss.shape == (2,)\r\n",
"loss.numpy()\r\n"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"array([0.05129344, 2.3025851 ], dtype=float32)"
]
},
"metadata": {
"tags": []
},
"execution_count": 60
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6dlnf5G-s4G_"
},
"source": [
"## poisson loss"
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "0lQ4ijw6svri",
"outputId": "52cb35ef-20b8-4075-83df-3affcbb7064d"
},
"source": [
"y_true = [[0., 1.], [0., 0.]]\r\n",
"y_pred = [[1., 1.], [0., 0.]]\r\n",
"# Using 'auto'/'sum_over_batch_size' reduction type.\r\n",
"p = tf.keras.losses.Poisson()\r\n",
"p(y_true, y_pred).numpy()\r\n"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.49999997"
]
},
"metadata": {
"tags": []
},
"execution_count": 61
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "q5niJ1uJrdul"
},
"source": [
"## KLDivergence loss\r\n"
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "7iR4uKkmrb-5",
"outputId": "65f010e8-245e-4014-8093-b75de7bdb3b4"
},
"source": [
"y_true = [[0, 1], [0, 0]]\r\n",
"y_pred = [[0.6, 0.4], [0.4, 0.6]]\r\n",
"# Using 'auto'/'sum_over_batch_size' reduction type.\r\n",
"kl = tf.keras.losses.KLDivergence()\r\n",
"kl(y_true, y_pred).numpy()\r\n"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.45814306"
]
},
"metadata": {
"tags": []
},
"execution_count": 62
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yBpo_uNa3HCg"
},
"source": [
"\r\n",
"\r\n",
"---\r\n",
"\r\n",
"---\r\n",
"\r\n",
"\r\n",
"\r\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KmdqbWfr0vgv"
},
"source": [
"# Regression losses\r\n",
"\r\n",
"\r\n",
"* Mean Squared error\r\n",
"* Mean Absolute Error\r\n",
"* Mean Absolute percentage error\r\n",
"* MeanSquaredLogarithmicError \r\n",
"* CosineSimilarity\r\n",
"* Huber\r\n",
"* LogCosh\r\n",
"\r\n",
"\r\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "pR0kzNoJsomp"
},
"source": [
"## MeanSquaredError"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "u4bj32uX56An"
},
"source": [
"### a.) Mean Square error class"
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "kf8AwSF4smO6",
"outputId": "a4eb6c1a-ebb5-4986-cf43-5bcd66cc86ed"
},
"source": [
"y_true = [[0., 1.], [0., 0.]]\r\n",
"y_pred = [[1., 1.], [1., 0.]]\r\n",
"# Using 'auto'/'sum_over_batch_size' reduction type.\r\n",
"mse = tf.keras.losses.MeanSquaredError()\r\n",
"mse(y_true, y_pred).numpy()"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.5"
]
},
"metadata": {
"tags": []
},
"execution_count": 63
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SySj0geW5xTE"
},
"source": [
"### b.) Mean square function"
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "t7QdmC6V51FB",
"outputId": "928b4bb0-8f01-4f0d-e402-5172ed1d5f24"
},
"source": [
"tf.keras.losses.mean_squared_error"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
""
]
},
"metadata": {
"tags": []
},
"execution_count": 64
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uiNAvFCmsGx4"
},
"source": [
"## MeanAbsoluteError"
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Ns5T_b6RsTY5",
"outputId": "d653b9a8-b7b5-48d6-ef73-4f817f25d8ed"
},
"source": [
"y_true = [[0., 1.], [0., 0.]]\r\n",
"y_pred = [[1., 1.], [1., 0.]]\r\n",
"# Using 'auto'/'sum_over_batch_size' reduction type.\r\n",
"mae = tf.keras.losses.MeanAbsoluteError()\r\n",
"mae(y_true, y_pred).numpy()"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.5"
]
},
"metadata": {
"tags": []
},
"execution_count": 65
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zuGLdWj6skb6"
},
"source": [
"## MeanAbsolutePercentageError"
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "A0hYJEdhsT7E",
"outputId": "d6867e3b-fc06-4497-f585-0f6dc2972689"
},
"source": [
"y_true = [[2., 1.], [2., 3.]]\r\n",
"y_pred = [[1., 1.], [1., 0.]]\r\n",
"# Using 'auto'/'sum_over_batch_size' reduction type.\r\n",
"mape = tf.keras.losses.MeanAbsolutePercentageError()\r\n",
"mape(y_true, y_pred).numpy()\r\n"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"50.0"
]
},
"metadata": {
"tags": []
},
"execution_count": 66
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b4DxdB1Ust0m"
},
"source": [
"## MeanSquaredLogarithmicError"
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "ALRKo6c6sqgb",
"outputId": "4efd0cfb-7dbf-4a1d-883a-aebea89b45db"
},
"source": [
"y_true = [[0., 1.], [0., 0.]]\r\n",
"y_pred = [[1., 1.], [1., 0.]]\r\n",
"# Using 'auto'/'sum_over_batch_size' reduction type.\r\n",
"msle = tf.keras.losses.MeanSquaredLogarithmicError()\r\n",
"msle(y_true, y_pred).numpy()"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.24022643"
]
},
"metadata": {
"tags": []
},
"execution_count": 67
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "RjPmk_C4mnWW"
},
"source": [
"## CosineSimilarity loss\r\n",
"```\r\n",
"tf.keras.losses.CosineSimilarity\r\n",
"```\r\n",
"\r\n"
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "svq-ZnO4miIF",
"outputId": "dc3e4309-f993-4cb8-9b15-e74a4c56428a"
},
"source": [
"y_true = [[0., 1.], [1., 1.]]\r\n",
"y_pred = [[1., 0.], [1., 1.]]\r\n",
"\r\n",
"# Using 'auto'/'sum_over_batch_size' reduction type.\r\n",
"cosine_loss = tf.keras.losses.CosineSimilarity(axis=1)\r\n",
"cosine_loss(y_true, y_pred).numpy()"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"-0.49999997"
]
},
"metadata": {
"tags": []
},
"execution_count": 68
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cWtUns91nZm2"
},
"source": [
"## Huber loss\r\n",
"```\r\n",
"tf.keras.losses.Huber\r\n",
"```\r\n",
"\r\n"
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Pa_sP8BZnQ_E",
"outputId": "38176304-6de3-4e79-9e82-6f158afb24b4"
},
"source": [
"y_true = [[0, 1], [0, 0]]\r\n",
"y_pred = [[0.5, 0.4], [0.4, 0.5]]\r\n",
"\r\n",
"# Using 'auto'/'sum_over_batch_size' reduction type.\r\n",
"hub_loss = tf.keras.losses.Huber()\r\n",
"h_loss(y_true, y_pred).numpy()\r\n"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"1.25"
]
},
"metadata": {
"tags": []
},
"execution_count": 69
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "50cj6TdiryNK"
},
"source": [
"## LogCosh loss"
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "wPFg0Jpin42Y",
"outputId": "cd942ae8-c208-4e4d-869c-5451abbe24cb"
},
"source": [
"y_true = [[0., 1.], [0., 0.]]\r\n",
"y_pred = [[1., 1.], [0., 0.]]\r\n",
"\r\n",
"# Using 'auto'/'sum_over_batch_size' reduction type.\r\n",
"l = tf.keras.losses.LogCosh()\r\n",
"l(y_true, y_pred).numpy()\r\n"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"0.1084452"
]
},
"metadata": {
"tags": []
},
"execution_count": 70
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ygG8G4Xc3y9f"
},
"source": [
"\r\n",
"\r\n",
"---\r\n",
"\r\n",
"---\r\n",
"\r\n",
"---\r\n",
"\r\n",
"\r\n",
"\r\n",
"\r\n",
"\r\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xTn0RllY30c_"
},
"source": [
"# Hinge losses for \"maximum-margin\" classification\r\n",
"* Hinge \r\n",
"* SquaredHinge \r\n",
"* CategoricalHinge \r\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ix3NZRmfm5LJ"
},
"source": [
"## Hinge loss\r\n",
"```\r\n",
"tf.keras.losses.Hinge\r\n",
"```\r\n",
"\r\n"
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "qd3FprohmvXg",
"outputId": "26f82e29-bb3e-48e3-8c58-c5c6fc5ab021"
},
"source": [
"y_true = [[0., 1.], [0., 0.]]\r\n",
"y_pred = [[0.5, 0.4], [0.4, 0.5]]\r\n",
"\r\n",
"# Using 'auto'/'sum_over_batch_size' reduction type.\r\n",
"h_loss = tf.keras.losses.Hinge()\r\n",
"h_loss(y_true, y_pred).numpy()"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"1.25"
]
},
"metadata": {
"tags": []
},
"execution_count": 71
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LQU4RKFj4Euv"
},
"source": [
"## Squared Hinge loss\r\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "NtYYekEt5ABB"
},
"source": [
"\r\n",
"\r\n",
"### a.) squared hinge Class\r\n",
"```\r\n",
"tf.keras.losses.SquaredHinge\r\n",
"```"
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "KRRfX_yW5BXX",
"outputId": "7a762f57-edda-4a6b-ada5-844b8198a280"
},
"source": [
"y_true = [[0., 1.], [0., 0.]]\r\n",
"y_pred = [[0.6, 0.4], [0.4, 0.6]]\r\n",
"\r\n",
"# Using 'auto'/'sum_over_batch_size' reduction type. \r\n",
"h = tf.keras.losses.SquaredHinge()\r\n",
"h(y_true, y_pred).numpy()"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"1.86"
]
},
"metadata": {
"tags": []
},
"execution_count": 72
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "NNGw-p4448YZ"
},
"source": [
"\r\n",
"### b.) squared hinge Function\r\n",
"```\r\n",
"tf.keras.losses.squared_hinge\r\n",
"```"
]
},
{
"cell_type": "code",
"metadata": {
"id": "lXLHdOuA5DXa"
},
"source": [
"y_true = np.random.choice([-1, 1], size=(2, 3))\r\n",
"y_pred = np.random.random(size=(2, 3))\r\n",
"loss = tf.keras.losses.squared_hinge(y_true, y_pred)\r\n",
"assert loss.shape == (2,)\r\n",
"assert np.array_equal(\r\n",
" loss.numpy(),\r\n",
" np.mean(np.square(np.maximum(1. - y_true * y_pred, 0.)), axis=-1))"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "aADWZmvzmPfk"
},
"source": [
"## CategoricalHinge loss\r\n",
"```\r\n",
"tf.keras.losses.CategoricalHinge\r\n",
"```\r\n",
"\r\n"
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "0Q07huIFmD46",
"outputId": "6c3f4de2-b9d0-4bac-c630-78fd123540c9"
},
"source": [
"y_true = [[0, 1], [0, 0]]\r\n",
"y_pred = [[0.5, 0.4], [0.4, 0.5]]\r\n",
"\r\n",
"# Using 'auto'/'sum_over_batch_size' reduction type.\r\n",
"h = tf.keras.losses.CategoricalHinge()\r\n",
"h(y_true, y_pred).numpy()"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"1.3"
]
},
"metadata": {
"tags": []
},
"execution_count": 74
}
]
}
]
}