{
    "info": {
        "author": "Benjamin Paassen",
        "author_email": "bpaassen@techfak.uni-bielefeld.de",
        "bugtrack_url": null,
        "classifiers": [
            "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
            "Operating System :: OS Independent",
            "Programming Language :: Python :: 3"
        ],
        "description": "# Linear Supervised Transfer Learning  \n\nCopyright (C) 2019 - Benjamin Paassen  \nMachine Learning Research Group  \nCenter of Excellence Cognitive Interaction Technology (CITEC)  \nBielefeld University\n\nThis program is free software; you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation; either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program; if not, see <http://www.gnu.org/licenses/>.\n\n## Introduction\n\nThis Python3 library provides several algorithms to learn a linear mapping from\nan $`m`$-dimensional source space to an $`n`$-dimensional target space, such that\na classification model trained in the source space becomes applicable in the\ntarget space. The source space model is assumed to be a labelled mixture of\nGaussians. Note that this library assumes that the relation between the source\nand target space is (approximately) linear and will necessarily fail if the\nrelationship is highly nonlinear. Further note that this library requires a\nfew labelled target space data points to work (typically, even ~10 data points\nare enough). However, not all classes are required, since the learned linear\ntransformation generalizes across classes.\n\nIf you intend to use this library in academic work, please cite [our paper][1].\n\n## Installation\n\nThis package is available on `pypi` as `em_transfer_learning`. You can install\nit via\n\n```\npip install --user em_transfer_learning\n```\n\n## QuickStart Guide\n\nFor a quick start we recommend to take a look at the demo in the notebook\n`demo.ipynb`. In this file we demonstrate how to perform transfer learning\non example data. For the actual transfer learning, we recommend to initialize\none of the following models, depending on your source space model:\n\n1. `em_transfer_learning.transfer_learning.LGMM_transfer_model` : If you have a full labelled\n   Gaussian mixture model.\n2. `em_transfer_learning.transfer_learning.SLGMM_transfer_model` : If you have a labelled Gaussian\n   mixture model with shared precision matrices.\n3. `em_transfer_learning.transfer_learning.Local_LVQ_transfer_model` : If you have a learning vector\n   quantization model with individual metric learning matrices.\n4. `em_transfer_learning.transfer_learning.LVQ_transfer_model` : If you have a learning vector\n   quantization model with shared metric learning matrix or no metric learning\n   at all.\n\nNote that models 2 and 4 are _much_ faster to train compared to models 1 and 3\n(refer to the next section for more information on that).\n\nAll these models follow the [scikit-learn][2] convention, i.e. you need to call\nthe `fit` function with target space data first and then the `predict` function\nto map new target space data to the source space according to the learned\nmapping.\n\n## Background\n\nThe basic idea of our transfer learning approach is to maximize the likelihood\nof target space data according to the source space data distribution _after_ the\nlearned transfer function $`h`$ has been applied. More precisely, assume we have\na data set $`(\\vec x_1, y_1), \\ldots, (\\vec x_m, y_m)`$ of target data points\n$`\\vec x_j \\in \\mathbb{R}^n`$ and their labels $`y_j \\in \\{1, \\ldots, L\\}`$.\nThen, we wish to maximize the joint probability\n\n```math\n\\max_h \\prod_{j=1}^m p\\Big(h(\\vec x_j), y_j\\Big)\n```\n\nTo make this optimization problem feasible, we introduce two assumptions:\nFirst, that $`p(x, y)`$ can be modelled by a labelled Gaussian mixture\nmodel (lGMM) and, second, that $`h`$ can be approximated by a linear function.\nIn more detail, that means the following.\n\n### Labelled Gaussian Mixture Models\n\nA _labelled Gaussian mixture model_ assumes that data is generated by a mixture\nof $`K`$ Gaussians, each of which has a prior $`P(k)`$, a data generating\nGaussian density $`p(\\vec x|k)`$, and a label generating distribution\n$`P(y|k)`$. Using these distributions, we can derive the joint probability\ndensity $`p(\\vec x, y)`$ as follows.\n\n```math\np(\\vec x, y) = \\sum_{k=1}^K p(\\vec x, y, k) = \\sum_{k=1}^K p(\\vec x, y|k) \\cdot P(k)\n```\n\nOur model assumes that $`\\vec x`$ and $`y`$ are conditionally independent given\nthe component index $`k`$, such that we can re-write:\n\n```math\np(\\vec x, y) = \\sum_{k=1}^K p(\\vec x|k) \\cdot P(y|k) \\cdot P(k)\n```\n\nNote that $`p(\\vec x|k)`$ is a [multivariate Gaussian probability density][5]\nwith parameters for the mean $`\\vec \\mu_k`$ and the precision matrix\n$`\\Lambda_k`$. Also note that this model is a proper generalization over\nstandard [Gaussian mixture models][6] and that many of the GMM properties\ntranslate directly to lGMMs. More precisely, we obtain a standard GMM by setting\nthe label distribution $`P(y|k)`$ to a uniform distribution and leaving it\nunchanged during training. Alternatively, we also obtain a standard GMM by\nassigning the same label to all data points.\n\nAlso note that lGMMs generalize over learning vector quantization models if we\napply a scaling trick to the precision matrices (for more details on this,\nrefer to [our paper][1]).\n\n### Expectation Maximization transfer learning\n\nOur assumption that the transfer function $`h`$ is approximately linear implies\nthat $`h`$ can be re-written as $`h(\\vec x) \\approx H \\cdot \\vec x`$ for some\nmatrix $`H`$. Thus, our transfer learning problem becomes:\n\n```math\n\\max_H \\prod_{j=1}^m \\sum_{k=1}^K p(H \\cdot \\vec x|k) \\cdot P(y|k) \\cdot P(k)\n```\n\nDue to the product of sums, a direct optimization of this expression is\ninfeasible. However, we can apply an [expectation maximization][6] scheme.\nIn particular, we initialize $`H`$ with the identity matrix (padded with zeros\nwherever necessary) and then iteratively perform the following two steps:\n\n1. _Expectation_: We compute the posterior $`p(k|H \\cdot \\vec x_j, y_j)`$\n\tfor the current transfer matrix $`H`$, all data points $`j`$ and all\n    Gaussian components $`k`$, yielding a matrix $`\\Gamma \\in \\mathbb{R}^{K \\times m}`$\n    with entries $`\\gamma_{k,j} = p(k|H \\cdot \\vec x_j, y_j)`$. The full\n    expression for the posterior is given in [our paper][1].\n2. _Maximization_: We maximize the expected log likelihood under fixed posterior,\n    i.e.\n\n    ```math\n    \\max_H \\sum_{j=1}^m \\sum_{k=1}^K \\gamma_{k, j} \\cdot \\log\\big[p(H \\cdot \\vec x_j, y_j| k)\\big]\n    ```\n\n\tThis optimization problem can be shown to be convex und thus lends itself\n\tfor optimization techniques like l-BFGS. Even better, if the precision\n\tmatrix $`\\Lambda_k`$ is shared across all Gaussians $`k`$, the problem has\n\ta closed-form solution, namely\n\n\t```math\n\tH = W \\cdot \\Gamma \\cdot X^T \\cdot (X \\cdot X^T + \\lambda \\cdot I)^{-1}\n\t```\n\n\twhere $`W = (\\vec \\mu_1, \\ldots, \\vec \\mu_K)`$, $`X = (\\vec x_1, \\ldots, \\vec x_m)`$,\n\t$`\\lambda`$ is a (small) regularization constant, and $`I`$ is the identity\n\tmatrix. Due to this closed form solution, the `SLGMM_transfer_model` and\n\tthe `LVQ_model` are much faster to train compared to the\n\t`LGMM_transfer_model` and the `Local_LVQ_transfer_model`.\n\nFor more detailed background, please refer to [our paper][1].\n\n## Contents\n\nThis library contains the following files.\n\n* `demo.ipynb` : A demo script illustrating how to use this library.\n* `LICENSE` : A copy of the GPLv3 license.\n* `em_transfer_learning/lgmm.py` : A file to train labelled Gaussian mixture models with or without\n  shared precision matrices.\n* `em_transfer_learning/transfer_learning.py` : The actual transfer learning models.\n* `lgmm_test.py` : A set of unit tests for `lgmm.py`.\n* `README.md` : This file.\n* `transfer_learning_test.py` : A set of unit tests for `transfer_learning.py`.\n\n## Licensing\n\nThis library is licensed under the [GNU General Public License Version 3][7].\n\n## Dependencies\n\nThis library depends on [NumPy][3] for matrix operations, on [scikit-learn][2]\nfor the base interfaces and on [SciPy][4] for optimization.\n\n## Literature\n\n* Paassen, B., Schulz, A., Hahne, J., and Hammer, B (2018).\n  _Expectation maximization transfer learning and its application for bionic hand prostheses_.\n  Neurocomputing, 298, 122-133. doi:[10.1016/j.neucom.2017.11.072](https://doi.org/10.1016/j.neucom.2017.11.072). [Link][1]\n\n<!-- References -->\n\n[1]: https://arxiv.org/abs/1711.09256 \"Paassen, B., Schulz, A., Hahne, J., and Hammer, B (2018). Expectation maximization transfer learning and its application for  bionic hand prostheses. Neurocomputing. accepted\"\n[2]: https://scikit-learn.org/stable/ \"Scikit-learn homepage\"\n[3]: http://numpy.org/ \"Numpy homepage\"\n[4]: https://scipy.org/ \"SciPy homepage\"\n[5]: https://en.wikipedia.org/wiki/Multivariate_normal_distribution \"Wikipedia page to multivariate Gaussian distributions\"\n[6]: http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=Brml.HomePage \"Barber, D. (2012). _Bayesian Reasoning and Machine Learning_ Cambridge University Press.\"\n[7]: https://www.gnu.org/licenses/gpl-3.0.en.html \"The GNU General Public License Version 3\"\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://gitlab.ub.uni-bielefeld.de/bpaassen/transfer-learning",
        "keywords": "transfer-learning gaussian-mixture-models learning-vector-quantization gmm lvq",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "em-transfer-learning",
        "package_url": "https://pypi.org/project/em-transfer-learning/",
        "platform": "",
        "project_url": "https://pypi.org/project/em-transfer-learning/",
        "project_urls": {
            "Homepage": "https://gitlab.ub.uni-bielefeld.de/bpaassen/transfer-learning"
        },
        "release_url": "https://pypi.org/project/em-transfer-learning/1.0.0/",
        "requires_dist": [
            "numpy",
            "scikit-learn",
            "scipy"
        ],
        "requires_python": "",
        "summary": "Supervised linear transfer learning based on labelled Gaussian mixture models and expectation maximization in scikit-learn-compatible form.",
        "version": "1.0.0"
    },
    "last_serial": 5627671,
    "releases": {
        "1.0.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "e20ae784529336c1ca8df5550eb3094d",
                    "sha256": "f2b123b032f5538baed32e59e3ebf0ad86c92dab604a35abeea22b0f91204ce7"
                },
                "downloads": -1,
                "filename": "em_transfer_learning-1.0.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "e20ae784529336c1ca8df5550eb3094d",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 26491,
                "upload_time": "2019-08-03T09:26:43",
                "url": "https://files.pythonhosted.org/packages/6b/1d/10a6f4188440479befec2ca8952c7e388aa820c89aa71ebbf8d9af240ed8/em_transfer_learning-1.0.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "161c91a9b4d30ccd29cc44b9aa20c419",
                    "sha256": "1759936536eb72b63e528ee23241bba15841f72c04a3a677779ba39d193f22df"
                },
                "downloads": -1,
                "filename": "em_transfer_learning-1.0.0.tar.gz",
                "has_sig": false,
                "md5_digest": "161c91a9b4d30ccd29cc44b9aa20c419",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 15021,
                "upload_time": "2019-08-03T09:26:45",
                "url": "https://files.pythonhosted.org/packages/c7/9d/297b70d5b478e91ef50d72093d8853742710caf16606569ca088c18157b3/em_transfer_learning-1.0.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "e20ae784529336c1ca8df5550eb3094d",
                "sha256": "f2b123b032f5538baed32e59e3ebf0ad86c92dab604a35abeea22b0f91204ce7"
            },
            "downloads": -1,
            "filename": "em_transfer_learning-1.0.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "e20ae784529336c1ca8df5550eb3094d",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 26491,
            "upload_time": "2019-08-03T09:26:43",
            "url": "https://files.pythonhosted.org/packages/6b/1d/10a6f4188440479befec2ca8952c7e388aa820c89aa71ebbf8d9af240ed8/em_transfer_learning-1.0.0-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "161c91a9b4d30ccd29cc44b9aa20c419",
                "sha256": "1759936536eb72b63e528ee23241bba15841f72c04a3a677779ba39d193f22df"
            },
            "downloads": -1,
            "filename": "em_transfer_learning-1.0.0.tar.gz",
            "has_sig": false,
            "md5_digest": "161c91a9b4d30ccd29cc44b9aa20c419",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 15021,
            "upload_time": "2019-08-03T09:26:45",
            "url": "https://files.pythonhosted.org/packages/c7/9d/297b70d5b478e91ef50d72093d8853742710caf16606569ca088c18157b3/em_transfer_learning-1.0.0.tar.gz"
        }
    ]
}