{
    "info": {
        "author": "Richard Darst",
        "author_email": "rkd@zgib.net",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "Environment :: Console",
            "Framework :: Jupyter",
            "Intended Audience :: Developers",
            "Intended Audience :: Science/Research",
            "Intended Audience :: System Administrators",
            "License :: OSI Approved :: MIT License",
            "Operating System :: OS Independent",
            "Programming Language :: Python :: 3"
        ],
        "description": "# Switch environments before running Jupyter kernels\n\nSometimes, one needs to execute Jupyter kernels in a different\nenvironment.  Say you want to execute the kernel in a conda\nenvironment (that's easy, but actually misses setting certain\nenvironment variables).  Or run it inside a Docker container.  One\ncould manually adjust the kernelspec files to set environment\nvariables or run commands before starting the kernel, but envkernel\nautomates this process.\n\nenvkernel is equally usable for end users (on their own systems or\nclusters) to easily access environments in Jupyter, or sysadmins\ndeploying this access on systems they administer.\n\nIn general, there are two passes: First, install the kernel, e.g.:\n`envkernel virtualenv --name=my-venv /path/to/venv`.  This parses some\noptions and writes a kernelspec file with the the `--name` you\nspecify.  When Jupyter tries to start this kernel, it will execute the\nnext phase.  When Jupyter tries to run the kernel, the kernelspec file\nwill re-execute `envkernel` in the run mode, which does whatever is\nneeded to set up the environment (in this case, sets `PATH` to the\n`/path/to/venv/bin/` that is needed).  Then it starts the normal\nIPython kernel.\n\nAvailable modes:\n* `conda`: Activate a [conda environment](https://docs.conda.io/) first.\n* `virtualenv`: Activate a virtualenv first.\n* `docker`: Run the kernel in a Docker container.\n* `singularity`: Run the kernel in a [singularity container](https://www.sylabs.io/docs/).\n* `Lmod`: Activate [Lmod](https://lmod.readthedocs.io/) modules first.\n\n\n\n\n\n## Installation\n\n```\npip install https://github.com/NordicHPC/envkernel/archive/master.zip\n```\n\nNot currently distributed through other channels, but hopefully this\nwill change.  This is a single-file script and can be copied just like\nthis.  The script must be available both when a kernel is set up, and\neach time the kernel is started (and currently assumes they are in the\nsame location).\n\n\n\n\n\n## General usage and common arguments\n\nGeneral invocation:\n\n```shell\nenvkernel [mode] [envkernel options] [mode-specific-options]\n```\n\n\nGeneral arguments usable by *all* classes during the setup phase:\n\nThese options directly map to normal Jupyter kernel install options:\n\n* `mode`: `singularity`, `docker`, `lmod`, or whatever mode is desired.\n* `--name $name`: Name of kernel to install (**required**).\n* `--user`: Install kernel into user directory.\n* `--sys-prefix`: Install to the current Python's `sys.prefix` (the Python which is running envkernel).\n* `--prefix`: same as normal kernel install option.\n* `--display-name NAME`: Human-readable name.\n* `--replace`: Replace existing kernel (Jupyter option, unsure what this means).\n* `--language`: What language to tag this kernel (default `python`).\n\nThese are envkernel-specific options:\n\n* `--python`: Python interpreter to use when invoking inside the\n  environment. (Default `python`.  Unlike other kernels, this defaults\n  to a relative path because the point of envkernel is to set up PATH\n  properly.)  If this is the special value `SELF`, this will be replaced\n  with the value of `sys.executable` of the Python running envkernel.\n* `--kernel=NAME`: Auto-set `--language` and `--kernel-cmd` to\n  that needed for these well-known kernels.  Options include `ipykernel` (the default) or `ir`.\n* `--kernel-cmd`: a string which is the kernel to start - space\n  separated, no shell quoting, it will be split when saving.  The\n  default is `python -m ipykernel_launcher -f {connection_file}`,\n  which is suitable for IPython.  For example, to start an R kernel in\n  the environment use `R --slave -e IRkernel::main() --args\n  {connection_file}` as the value to this, being careful with quoting\n  the spaces only once.  To find what the strings should be, copy form\n  some existing kernels.  `--kernel=NAME` includes shortcut for some\n  popular kernels.\n* `--kernel-template`: An already-installed kernel name which is used\n  as a template for the new envkernel.  This is searched using the\n  normal Jupyter search paths.  This kernel json file is loaded and\n  used as a template for all kernel options (`--language`,\n  `--kernel-cmd`, etc).  Also, any other file in this directory (such\n  as logos) are copied to the new kernel (like kernel.js in irkernel).\n* `--env=NAME=VALUE`.  Set these environment variables when running\n  the kernel.  These are actually just saved in the `kernel.json` file\n  under the `env` key, which is used by Jupyter itself.  So, this is\n  just a shorthand for adding variables there, it is not used at the\n  envkernel stage at all.\n\nOrder of precedence of options (later in the list overrides earlier):\n`--kernel-template`, `--kernel`, `--kernel-cmd`, `--language`,\n`--python`, `--display-name`.\n\n\n\n\n## Conda\n\nThe Conda envkernel will activate Conda environments (set the `PATH`,\n`CPATH`, `LD_LIBRARY_PATH`, and `LIBRARY_PATH` environment variables).\nThis is done manually, if anyone knows a better way to do this, please\ninform us.\n\n### Conda example\n\nThis will load the `anaconda` environment before invoking an IPython\nkernel using the name `python`, which will presumably be the one\ninside the `anaconda3` environment.\n\n```shell\nenvkernel conda --name=conda-anaconda3 /path/to/anaconda3\n```\n\n### Conda mode arguments\n\nGeneral invocation:\n\n```shell\nenvkernel conda --name=NAME [envkernel options] conda-env-full-path\n```\n\n* `conda-env-full-path`: Full path to the conda environment to load.\n\n\n\n\n\n## Virtualenv\n\nThis operates identically to `conda` mode, but with name `virtualenv`\non virtualenvs.\n\n### Virtualenv example\n\n```shell\nenvkernel virtualenv --name=conda-anaconda3 /path/to/anaconda3\n```\n\n\n\n\n\n## Docker\n\nDocker is a containerization system that runs as a system service.\n\nNote: docker has not been fully tested, but has been reported to work.\n\n\n### Docker example\n\n```shell\nenvkernel singularity --name=NAME  --pwd --bind /m/jh/coursedata/:/coursedata /path/to/image.simg\n```\n\n### Docker mode arguments\n\nGeneral invocation:\n\n```shell\nenvkernel singularity --name=NAME [envkernel options] [singularity options] [image]\n```\n\n* `image`: Required positional argument: name of docker image to run.\n\n* `--pwd`: Bind-mount the current working directory and use it as the\n  current working directory inside the notebook.  This is usually\n  useful.\n\n* A few more yet-undocumented and untested arguments...\n\nAny unknown argument is passed directly to the `docker run` call, and\nthus can be any normal Docker argument.  If `,copy` is included in the\n`--mount` command options, the directory will be copied before\nmounting.  This may be useful if the directory is on a network mount\nwhich the root docker can't access.\n\n\n\n\n\n## Singularity\n\n[Singularity](https://www.sylabs.io/docs/) is a containerization\nsystem somewhat similar to Docker, but designed for user-mode usage\nwithout root, and with a mindset of using user software instead of\nsystem services.\n\n\n### Singularity example\n\n```shell\nenvkernel singularity --name=NAME --contain --bind /m/jh/coursedata/:/coursedata /path/to/image.simg\n```\n\n### Singularity mode arguments\n\nGeneral invocation:\n\n```shell\nenvkernel singularity --name=NAME [envkernel options] [singularity options] [image]\n```\n\n* `image`: Required positional argument: name of singularity image to\n  run.\n\n* `--pwd`: Bind-mount the current working directory and use it as the\n  current working directory inside the notebook.  This may happen by\n  default if you don't `--contain`.\n\nAny unknown argument is passed directly to the `singularity exec`\ncall, and thus can be any normal Singularity arguments.  The most\nuseful Singularity options are (nothing envkernel specific here):\n\n* `--contain` or `-c`: Don't share any filesystems by default.\n\n* `--bind src:dest[:ro]`: Bind mount `src` from the host to `dest` in\n  the container.  `:ro` is optional, and defaults to `rw`.\n\n* `--cleanenv`: Clean all environment before executing.\n\n* `--net` or `-n`: Run in new network namespace.  This does **NOT**\n  work with Jupyter kernels, because localhost must currently be\n  shared.  So don't use this unless we create proper net gateway.\n\n\n\n\n\n## Lmod\n\nThe Lmod envkernel will load/unload\n[Lmod](https://lmod.readthedocs.io/) modules before running a normal\nIPython kernel.\n\nUsing envkernel is better than the naive (but functional) method of\nmodifying a kernel to invoke a particular Python binary, because that\nwill invoke the right Python interpreter but not set relevant other\nenvironment variables (so, for example, subprocesses won't be in the\nright environment).\n\n### Lmod example\n\nThis will run `module purge` and then `module load anaconda3` before\ninvoking an IPython kernel using the name `python`, which will\npresumably be the one inside the `anaconda3` environment.\n\n```shell\nenvkernel lmod --name=anaconda3 --purge anaconda3\n```\n\n### Lmod mode arguments\n\nGeneral invocation:\n\n```shell\nenvkernel lmod --name=NAME [envkernel options] [module ...]\n```\n\n* `module ...`: Modules to load (positional argument).  Note that if\n   the module is prefixed with `-`, it is actually unloaded (this is a\n   Lmod feature).\n\n* `--purge`: Purge all modules before loading the new modules.  This\n  can be safer, because sometimes users may automatically load modules\n  from their `.bashrc` which will cause failures if you try to load\n  conflicting ones.\n\n\n\n\n\n## How it works\n\nWhen envkernel first runs, it sets up a kernelspec that will re-invoke\nenvkernel when it runs.  Some options are when firs run (kernelspec\nname and options), while usually most are passed through straight to\nthe kernelspec.  When the kernel is started, envkernel is re-invoked\n\nExample envkernel setup command.  This makes a new Jupyter kernel\n(`envkernel singularity` means singularity create mode) named\n`testcourse-0.5.9` out of the image `/l/simg/0.5.9.simg` with the\nSingularity options `--contain` (contain, on default mounts) and\n`--bind` (bind a dir).`\n\n```shell\nenvkernel singularity --sys-prefix --name=testcourse-0.5.9 /l/simg/0.5.9.simg --contain --bind /m/jh/coursedata/:/coursedata\n```\n\nThat will create this kernelspec.  Note that most of the arguments are passed through:\n\n```json\n{\n    \"argv\": [\n        \"/opt/conda-nbserver-0.5.9/bin/envkernel\",\n        \"singularity\",\n        \"run\",\n        \"--connection-file\",\n        \"{connection_file}\",\n        \"--contain\",\n        \"--bind\",\n        \"/m/jh/coursedata/:/coursedata\",\n        \"/l/simg/0.5.9.simg\",\n        \"--\",\n        \"python\",\n        \"-m\",\n        \"ipykernel_launcher\",\n        \"-f\",\n        \"{connection_file}\"\n    ],\n    \"display_name\": \"Singularity with /l/simg/0.5.9.simg\",\n    \"language\": \"python\"\n}\n```\n\nWhen this runs, it runs `singularity --contain --bind\n/m/jh/coursedata/:/coursedata /l/simg/0.5.9.simg`.  Inside the image,\nit runs `python -m ipykernel_launcher -f {connection_file}`.\nenvkernel parses and manipulates these arguments however is needed.\n\n\n\n\n\n## Use with nbgrader\n\nenvkernel was orginally inspired by the need for nbgrader to securely\ncontain student's code while autograding.  To do this, set up a\ncontained kernel as above - it's up to you to figure out how to do\nthis properly with your chosen method (docker or singularity).  Then\nautograde like normal, but add the `--ExecutePreprocessor.kernel_name`\noption.\n\nSet up a kernel:\n\n```shell\nenvkernel docker --user --name=testcourse-0.5.9 --pwd aaltoscienceit/notebook-server:0.5.9 --bind /mnt/jupyter/course/testcourse/data/:/coursedata\n```\n\nRun the autograding:\n\n```shell\nnbgrader autograde --ExecutePreprocessor.kernel_name=testcourse-0.5.9 R1_Introduction\n\n```\n\n\n\n\n\n## See also\n\n* General\n  * [a2km, \"Assistant to the kernel manager\"](https://github.com/minrk/a2km) is a command line tool for dealing with kernels, including making kernels which activate conda/venv kernels.  And some other handy kernel manipulations stuff.  Unfortunately written in Ruby.\n  * https://github.com/Anaconda-Platform/nb_conda_kernels - automatically create kernels from conda environments.  Uses a KernelSpecManager so possibly overrides everything at once, and also defaults to all kernels.\n  * The direct way to make a conda/virtualenv available in Jupyter is to activate the environment, then run `python -m ipykernel install [--user|--prefix=/path/to/other/env/]`.  But this does *not* set up `PATH`, so calling other executables doesn't work... thus the benefit of envkernel.\n  * [This thread](https://groups.google.com/forum/#!topic/jupyter/kQ9ZDX4rDEE) was the clue to getting a kernel inside Docker working.\n\n\n\n\n## Development and contributions\n\nDeveloped at Aalto University Science-IT.  Primary contact: Richard\nDarst.  Contributions welcome from anyone.  As of early 2019, it is\nmid 2019, it's usable but there may be bugs as it gets used in more\nsites.\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/NordicHPC/envkernel",
        "keywords": "jupyter kernelspec",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "envkernel",
        "package_url": "https://pypi.org/project/envkernel/",
        "platform": "",
        "project_url": "https://pypi.org/project/envkernel/",
        "project_urls": {
            "Homepage": "https://github.com/NordicHPC/envkernel"
        },
        "release_url": "https://pypi.org/project/envkernel/1.0.0/",
        "requires_dist": null,
        "requires_python": ">=3.5",
        "summary": "Jupyter kernels manipulation and in other environments (docker, Lmod, etc.)",
        "version": "1.0.0"
    },
    "last_serial": 5796051,
    "releases": {
        "1.0.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a6d0b403ed9c7c3bd6469bd1f95d1541",
                    "sha256": "955a40f34005dd26b21a57bd8d2e6943b57722a26c86e329d2eddaabef307ef3"
                },
                "downloads": -1,
                "filename": "envkernel-1.0.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "a6d0b403ed9c7c3bd6469bd1f95d1541",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.5",
                "size": 13384,
                "upload_time": "2019-09-07T12:52:38",
                "url": "https://files.pythonhosted.org/packages/09/71/b7f496ed9b8196cb206d6dc1231dc7ee7e54a0fd972aea24c88ba028d4b5/envkernel-1.0.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "2fc0be25cd6fa28424560c100c8ce79d",
                    "sha256": "271da1b63a1fdfc08177571c5e8e85ccd6e20274bf45f7f6571d47f5b95c58cf"
                },
                "downloads": -1,
                "filename": "envkernel-1.0.0.tar.gz",
                "has_sig": false,
                "md5_digest": "2fc0be25cd6fa28424560c100c8ce79d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.5",
                "size": 17435,
                "upload_time": "2019-09-07T12:52:41",
                "url": "https://files.pythonhosted.org/packages/29/12/c9afc7b8f7db99514804731857b2df8054c903b9b922f23150ee71d4d5a1/envkernel-1.0.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "a6d0b403ed9c7c3bd6469bd1f95d1541",
                "sha256": "955a40f34005dd26b21a57bd8d2e6943b57722a26c86e329d2eddaabef307ef3"
            },
            "downloads": -1,
            "filename": "envkernel-1.0.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "a6d0b403ed9c7c3bd6469bd1f95d1541",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.5",
            "size": 13384,
            "upload_time": "2019-09-07T12:52:38",
            "url": "https://files.pythonhosted.org/packages/09/71/b7f496ed9b8196cb206d6dc1231dc7ee7e54a0fd972aea24c88ba028d4b5/envkernel-1.0.0-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "2fc0be25cd6fa28424560c100c8ce79d",
                "sha256": "271da1b63a1fdfc08177571c5e8e85ccd6e20274bf45f7f6571d47f5b95c58cf"
            },
            "downloads": -1,
            "filename": "envkernel-1.0.0.tar.gz",
            "has_sig": false,
            "md5_digest": "2fc0be25cd6fa28424560c100c8ce79d",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.5",
            "size": 17435,
            "upload_time": "2019-09-07T12:52:41",
            "url": "https://files.pythonhosted.org/packages/29/12/c9afc7b8f7db99514804731857b2df8054c903b9b922f23150ee71d4d5a1/envkernel-1.0.0.tar.gz"
        }
    ]
}