{
    "info": {
        "author": "Twin Karmakharm",
        "author_email": "t.karmakharm@sheffield.ac.uk",
        "bugtrack_url": null,
        "classifiers": [
            "License :: OSI Approved :: MIT License",
            "Operating System :: OS Independent",
            "Programming Language :: Python :: 2"
        ],
        "description": "# Chaste Parameter Sweeper\n\nA tool to assist with parameter sweeping for [Chaste](https://github.com/Chaste/Chaste) on the HPC (SGE & Slurm). The package provides parameter expansion, generation of task array batch script, individual job runner and a c++ main() function generator. \n\nThe parameter sweeping code was brought in from the [pscan](https://github.com/brunobeltran/pscan) package and converted to python 2.x to ensure it works with the rest of Chaste's build dependencies.\n\nWhile originally created for Chaste, it can also be applied to any other application that correctly accepts the parameters in the correct format (i.e. your program should accept parameters in the format of  `--varname value`).\n\n## Installation\n\n### By pip\n\nInstall by using:\n\n```bash\npip install chastesweep\n```\n\n### From repository\n\n```bash\n# Clone the respository then install it\ngit clone https://github.com/twinkarma/chastesweep.git\ncd chastesweep\npython setup.py install\n```\n\n## General Usage\n\nAfter installing the `chastesweep` package, create a python script for defining your sweep parameters e.g. we create a `sweep.py` file:\n\n```python\nimport numpy as np\nfrom chastesweep import ParamSweeper\n\n# We'll use 3 parameters in this example, param0, param1 and param2\n# we create a dictionary and declare what each parameter values are\np = {}\np['param0'] = np.linspace(0, 10, 5)\np['param1'] = np.linspace(0.1, -0.5, 5)\np['param2'] = [10, 20, 30]\n\n# We must also declare the execuatable that will run with the paramters\nexec_cmd = \"my_executable.sh\"\n\n# And the output directory \noutput_dir = \"sweep_results\"\n\n# We then create the parameter sweeper and have it generate the batch file in the output directory\nsweeper = ParamSweeper()\nsweeper.generate_batch_output(output_dir=output_dir,\n                              exec_cmd=exec_cmd,\n                              parameters=p,\n                              scheduler=ParamSweeper.SGE,\n                              batch_params=[\"-M myemail@mydomain.com\", \"-m bes\"])\n```\n\nWe then run `sweep.py`:\n\n```bash\npython sweep.py\n```\n\nIn the output directory `sweep_results`, you will see the `params.json`, `runsimulation.py` and `batch.sge.sh` file. \n\n  * `params.json` Contains an expanded list of parameters that will be explored.\n  * `batch.sge.sh` Batch script containing a task array for running through all of the parameters to be explored.\n  * `runsimulation.py` Simulation runner script for running individual instances of the simulation.   \n\nYou can submit this file to the SGE scheduler to start your parameter sweeping task:\n\n```bash\nqsub sweep_results/batch.sge.sh\n```\n\nIf we're running a SLURM scheduler, simply change the `scheduler=ParamSweeper.SGE` to `scheduler=ParamSweeper.SLURM`. You'll also want to change your `batch_params` parameters as they'll likely be different from SGE. Your batch submit command will also change to:\n \n```bash\nsbatch sweep_results/batch.slurm.sh\n```\n\n### Expanding parameters\n\nThe following is a demonstration of how parameters can be expanded. All examples also apply to `generate_batch_output`.\n\nTo only get the parameter expansion, call the `expand_parameters` function:\n\n```python\nimport numpy as np\nfrom chastesweep import ParamSweeper\n\n# We'll use 3 parameters in this example, param0, param1 and param2\n# we create a dictionary and declare what each parameter values are\np = {}\np['param0'] = np.linspace(0, 10, 5)\np['param1'] = np.linspace(0.1, -0.5, 5)\np['param2'] = [10, 20, 30]\n# We can then call the exp\nsweeper = ParamSweeper()\nresults = sweeper.expand_parameters(parameters=p)\n```\n\n### Joining Parameters\n\nBy default, parameters are combinatorially expanded. To prevent this expansion between some parameters, the parameters can be joined together by providing a `join` list. For example if you'd like to join `param0` and `param1` in the above example:\n\n```python\n# We create a join list\njoin_lists = [['param0', 'param1']]\n# Add the join list to the call parameters\nresults = sweeper.expand_parameters(parameters=p, joint_lists=join_lists)\n```\n\nYou'll see that our `results` list has `15` items instead of `75` if we didn't include the join list.\n\n### Globally running each parameter set more than once\n\nYou can set the `default_repeats` to specify globally how many iterations of each parameter set will run, e.g. the value of `2`:\n\n```python\nresults = sweeper.expand_parameters(parameters=p, default_repeats=2)\n```\n\nResult in `150` simulation runs.\n\n### Variable number of iteration for each parameter set\n\nInstead of setting the number of iterations globally, the number of iteration per parameter set can vary by declaring count functions:\n\n```python\n# The default count function, runs 2 iterations of every parameter\ndefault_count = lambda p: 2\n\n# A big_count function that forces the iteration to 1 if param 2 is larger than 13\nbig_count = lambda p: 1 if p['param2'] >= 14 else None\n\n# The order of counters makes a difference, the next counter in the sequence can overwrite previous counts\ncount_funcs = [default_count, big_count]\n\nresults = sweeper.expand_parameters(parameters=p, count_funcs=count_funcs)\n```\n\n### Running the sweep locally \n\nSweep can also be run locally on your machine with the `perform_serial_sweep` function:\n\n```python\nimport numpy as np\nfrom chastesweep import ParamSweeper\np = {}\np['a'] = np.linspace(0, 10, 5)\np['b'] = np.linspace(0.1, -0.5, 5)\np['c'] = [10, 20, 30]\n\nsweeper = ParamSweeper()\nexec_cmd = \"my_executable.sh\"\noutput_dir = \"sweep_results\"\nsweeper.perform_serial_sweep(output_dir=output_dir, exec_cmd=exec_cmd, parameters=p)\n```\n\n\n## Parameter Sweeping Tutorial (Sheffield HPC)\n\nIn this tutorial we will go through how to setup chaste for parameter sweeping on the HPC cluster. Run this tutorial directly on the cluster to be able to follow all examples including job submission.\n\n### 1. Getting a compute node\n\nWhen logging in to ShARC or Bessemer, you'll start off on a login node. You'll want to get an interactive session on a worker node to start development:\n\n\n```bash\n# On ShARC\nqrshx\n```\n\n```bash\n# Bessemer\nsrun --pty bash -i\n```\n\nYou'll see the console change from `login` to `node` e.g. `[username@sharc-login#] $` to `[username@sharc-node###] $` in the case of ShARC. \n\n\n### 2. Install chastesweep python package\n\n \n#### On ShARC\n \nIt is recommended to create a virtual environment on ShARC:\n \n```bash\n# Load the python module\nmodule load apps/python/anaconda3-4.2.0\n\n# Create a new virtual python environment called my_chaste_env with python 2.7\nconda create -n my_chaste_env python=2.7\n\n# Activate the environment\nsource activate my_chaste_env\n\n# Install the chastesweep package\npip install chastesweep\n\n```\n\n#### On Bessemer\n\nEquivalent code on Bessemer\n\n```bash\n# Load the python module\nmodule load Anaconda3/5.3.0\n\n# Create a new virtual python environment called my_chaste_env with python 2.7\nconda create -n my_chaste_env2 python=2.7 chastesweep\n\n# Activate the environment\nsource activate my_chaste_env2\n\n# Install the chastesweep package\npip install chastesweep\n```\n \n\n\n### 3. Development Environment, pulling a a Singularity image\n\nShARC and Bessemer supports the [Singularity](https://sylabs.io/docs/) containerisation technology. The use of containers allow us to standardise the development environment and build dependencies on your local machine and your cluster. \n\nNote: On your local machine, if you're not developing on Linux, it is also possible to use docker instead. You will want to install either [Singularity](https://sylabs.io/docs/) or [docker](https://docker.com) on your local machine.\n\nTo get the latest Singularity image of Chaste, run the following:\n\n```bash\nsingularity pull docker://chaste/chaste-docker:latest\n```\n\nYou should see a file `chaste-docker_latest.sif`. \n\nRun the following to get into the image's bash shell:\n\n```bash\nsingularity exec chaste-docker_latest.sif /bin/bash\n```\n\nYou're now actually inside the image, try running the command `cat /etc/lsb-release` to see the current Ubuntu version e.g.:\n\n```bash\ncat /etc/lsb-release\nDISTRIB_ID=Ubuntu\nDISTRIB_RELEASE=19.04\nDISTRIB_CODENAME=disco\nDISTRIB_DESCRIPTION=\"Ubuntu 19.04\"\n```\n\nYou can run `exit` to exit from the image.\n\nSingularity's `exec` command can be used to run any program from inside the image without needing to go into the interactive session. This is useful when we're running our tasks in a batch script. For example, to get the Ubuntu version as above:\n\n```bash\nsingularity exec chaste-docker_latest.sif cat /etc/lsb-release\n```\n\nYou will be using the image you've just pulled to do the building and running of the code using the `exec` command as shown above.\n\nNote: As you will only be using the container for its build dependency, you will not need to modify the image during development and so should not need admin permission.  \n\n\n### 4. Preparing your code\n\nYou now need your own version of Chaste code. The Chaste repository can be found at [https://github.com/Chaste/Chaste](https://github.com/Chaste/Chaste) and this can be forked or cloned directly depending on your requirements. We will just clone directly in this example:\n\n```bash\n# Clone the Chaste source code into our home folder\ngit clone https://github.com/Chaste/Chaste.git ~/Chaste\n\n# Go inside the folder\ncd ~/Chaste\n```\n\nThe above command clones Chaste code into the `Chaste` folder to your home directory. You will now need create a custom Chaste app or project with a main() function to be able to accept parameters.\n\nThe the instructions on how to create  Chaste executable apps and user project can be followed below:\n  * Make executable apps: https://chaste.cs.ox.ac.uk/trac/wiki/ChasteGuides/BuildingExecutableApps\n  * User projects: https://chaste.cs.ox.ac.uk/trac/wiki/ChasteGuides/UserProjects\n  \nFor this example we'll go with the executable app route due to the simpler setup. We'll ask the tool to generate the `.cpp` file for us.\n\nWe'll call our app `ParamSweep` and will use three parameters for this example, named `param0`, `param1` and `param2`. To automatically generate a template using the parameter sweeper tool, call:\n\n```bash\nchastesweep_genmain param0,param1,param2 apps/src/ParamSweep.cpp\n```\n\nThe generated source code is printed on screen as well as created for you.\n\nWe can then try to build the project using `cmake`:\n\n```bash\n# Go into the build directory\ncd build\n\n# Generate makefiles using cmake from inside the Chaste image\nsingularity exec chaste-docker_latest.sif cmake ..\n\n# Build our ParamSweep app from inside the Chaste image\nsingularity exec chaste-docker_latest.sif make ParamSweep\n```\n\nAfter the build has finished, we can now run our app:\n\n```bash\nsingularity exec chaste-docker_latest.sif apps/ParamSweep\n```\n\nYour code should execute as is but you'll get errors stating that parameters are missing.\n\t\n\n### 5. Passing parameters to the main function\n\nThe parameter sweeper follows the boost::program_options format, where each parameter is passed in the format of `--param_name param_value` e.g. for our `ParamSweep` app: \n\n```bash\nsingularity exec chaste-docker_latest.sif apps/ParamSweep --output_dir \"/outtput/path/1\" --param0 0.2 --param1 3 --param2 -0.53 \n```\n\nNote: The `output_dir` parameter is always passed to your app and all outputs of the app should be enforced to only write to this directory.\n\n\n### 6. Declaring parameters and generating the batch script\n\nNow that we have an executable ready for accepting parameters, we'll move on to declaring the values of parameters to be explored and generating batch script for submitting a batch task. For this purpose, python API has been created, the class `ParamSweeper` has been created to facilitate this parameter sweeping process. \n\nWe create a python script named `sweep.py` for defining our parameter and creating a batch job:\n\n```python\nimport numpy as np\nfrom chastesweep import ParamSweeper\np = {}\np['param0'] = np.linspace(0, 10, 5)\np['param1'] = np.linspace(0.1, -0.5, 5)\np['param2'] = [10, 20, 30]\n\nexec_cmd = \"~/Chaste/build/apps/ParamSweep\"\noutput_dir = \"~/Chaste/build/sweep_results\"\n\n# Generate batch for SGE\nsweeper = ParamSweeper()\nsweeper.generate_batch_output(output_dir=output_dir,\n                              exec_cmd=exec_cmd,\n                              parameters=p,\n                              scheduler=ParamSweeper.SGE,\n                              batch_params=[\"-M myemail@mydomain.com\", \"-m bes\"])\n```\n\nNotice that we can additional parameters to the batch script by adding to `batch_params`. In our example we've set the scheduler to alert us of the job status `-m bes` and provided our e-mail `-M myemail@mydomain.com`.\n\nWe then run the `sweep.py` script:\n\n```bash\npython sweep.py\n```\n\nIn the output directory `~/Chaste/build/sweep_results`, you will see the `params.json`, `runsimulation.py` and `batch.sge.sh` file. \n\n  * `params.json` Contains an expanded list of parameters that will be explored.\n  * `batch.sge.sh` Batch script containing a task array for running through all of the parameters to be explored.\n  * `runsimulation.py` Simulation runner script for running individual instances of the simulation.   \n\n\nYou can submit this file to the SGE scheduler to start your parameter sweeping task:\n\n```bash\nqsub ~/Chaste/build/sweep_results/batch.sge.sh\n```\n\nIf we're running on Bessemer or other cluster that uses SLURM scheduler, simply change the `scheduler=ParamSweeper.SGE` to `scheduler=ParamSweeper.SLURM`. You'll also want to change your `batch_params` parameters as they'll likely be different from SGE. Your batch submit command will also change to:\n \n```bash\nsbatch ~/Chaste/build/sweep_results/batch.slurm.sh\n```",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/twinkarma/chastesweep",
        "keywords": "",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "chastesweep",
        "package_url": "https://pypi.org/project/chastesweep/",
        "platform": "",
        "project_url": "https://pypi.org/project/chastesweep/",
        "project_urls": {
            "Homepage": "https://github.com/twinkarma/chastesweep"
        },
        "release_url": "https://pypi.org/project/chastesweep/0.10/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "Parameter Sweeper for Chaste",
        "version": "0.10",
        "yanked": false,
        "yanked_reason": null
    },
    "last_serial": 6007722,
    "releases": {
        "0.10": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "0aef4422ed90df41f5d323ae975d5162",
                    "sha256": "93e0f0579480664f8ef7ef69f067ac1373e1dfbd247f295d515fedfd60edc6ad"
                },
                "downloads": -1,
                "filename": "chastesweep-0.10.tar.gz",
                "has_sig": false,
                "md5_digest": "0aef4422ed90df41f5d323ae975d5162",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 15135,
                "upload_time": "2019-10-21T15:00:08",
                "upload_time_iso_8601": "2019-10-21T15:00:08.206802Z",
                "url": "https://files.pythonhosted.org/packages/90/b8/44b18667090b378f60b11d0f16bb78700122b55b170e3e462a5af5ec61f7/chastesweep-0.10.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "37144cc829bb7b89dcc19c333fd61fa9",
                    "sha256": "07ac3cb7f82e56d91e4945e2b0a59ff1b89e4884bbf6c4f2f2bc6a55f89cc70e"
                },
                "downloads": -1,
                "filename": "chastesweep-0.3-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "37144cc829bb7b89dcc19c333fd61fa9",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 17474,
                "upload_time": "2019-10-15T15:32:37",
                "upload_time_iso_8601": "2019-10-15T15:32:37.457339Z",
                "url": "https://files.pythonhosted.org/packages/b9/11/03b97552ed52e80c3295de0ebbe2072c3a751d5b3a37a74f058cf2ad4626/chastesweep-0.3-py2-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "861c2afd831b8380c5c4794f9df002d4",
                    "sha256": "316281976f1535dfa5b98bda670d57a07cfc119aaa06a178a774262d418825cc"
                },
                "downloads": -1,
                "filename": "chastesweep-0.4-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "861c2afd831b8380c5c4794f9df002d4",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 19999,
                "upload_time": "2019-10-15T15:45:10",
                "upload_time_iso_8601": "2019-10-15T15:45:10.594804Z",
                "url": "https://files.pythonhosted.org/packages/6a/c8/39dba6e83481bb343abb87353c53c901be43733ddb09998b1da74ade0baa/chastesweep-0.4-py2-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "e1fe09101d14f99f195ca5904db1f28d",
                    "sha256": "8f57c06c16954d79031eaa815b85a0048e5f039638c718f4d9c848fb4a58413e"
                },
                "downloads": -1,
                "filename": "chastesweep-0.5-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "e1fe09101d14f99f195ca5904db1f28d",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 20025,
                "upload_time": "2019-10-15T16:12:25",
                "upload_time_iso_8601": "2019-10-15T16:12:25.147046Z",
                "url": "https://files.pythonhosted.org/packages/ef/33/7eaac15882b0d74f4111cb43ad836164062364b48dd2a67a10c991629949/chastesweep-0.5-py2-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f2552845a432132881a9e64ddce9175a",
                    "sha256": "9aaf07ebbff6d9449acfadff710c22fbeef9a678c56f00f0b2e856ba1ce7e2cd"
                },
                "downloads": -1,
                "filename": "chastesweep-0.6-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "f2552845a432132881a9e64ddce9175a",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 20194,
                "upload_time": "2019-10-16T10:25:52",
                "upload_time_iso_8601": "2019-10-16T10:25:52.254782Z",
                "url": "https://files.pythonhosted.org/packages/e8/59/da1df9746bd6847a408b6721a28a1d298682d9150b8b9dc10d400f95f45c/chastesweep-0.6-py2-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5ddf0f6f6933352e1d6665cdb9114d0a",
                    "sha256": "d5192c36513416b9a2ea874c68db64e0e0d3ccfaad05cc0276ef3d47edbd9276"
                },
                "downloads": -1,
                "filename": "chastesweep-0.7-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "5ddf0f6f6933352e1d6665cdb9114d0a",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 20189,
                "upload_time": "2019-10-16T10:30:42",
                "upload_time_iso_8601": "2019-10-16T10:30:42.662886Z",
                "url": "https://files.pythonhosted.org/packages/07/5b/1b1252f072a10fa4a3caa19e0156b98026e79089533ed3a906a94244e3f2/chastesweep-0.7-py2-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.8": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "fac5941afba53270bb47485cc018ce80",
                    "sha256": "5575a63c28354f19434cd61a233070cd902b8ecbe3786d6f1036a16eed21e373"
                },
                "downloads": -1,
                "filename": "chastesweep-0.8.tar.gz",
                "has_sig": false,
                "md5_digest": "fac5941afba53270bb47485cc018ce80",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13699,
                "upload_time": "2019-10-21T13:09:10",
                "upload_time_iso_8601": "2019-10-21T13:09:10.790437Z",
                "url": "https://files.pythonhosted.org/packages/d0/82/f770f583e1e95c68819d1d4126b628d5264cda5e92dba766cf569bced512/chastesweep-0.8.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.9": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "759a5846b6f691d529f77ac44fde347c",
                    "sha256": "841875a48d7b5ddd57b28352544dfa26bff2321121a64c3a204719737b624786"
                },
                "downloads": -1,
                "filename": "chastesweep-0.9.tar.gz",
                "has_sig": false,
                "md5_digest": "759a5846b6f691d529f77ac44fde347c",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 15115,
                "upload_time": "2019-10-21T14:38:38",
                "upload_time_iso_8601": "2019-10-21T14:38:38.836787Z",
                "url": "https://files.pythonhosted.org/packages/51/18/882c7054d987dcd995a0655323afa61464a3dc264d5c7e69e592a6a17de6/chastesweep-0.9.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "0aef4422ed90df41f5d323ae975d5162",
                "sha256": "93e0f0579480664f8ef7ef69f067ac1373e1dfbd247f295d515fedfd60edc6ad"
            },
            "downloads": -1,
            "filename": "chastesweep-0.10.tar.gz",
            "has_sig": false,
            "md5_digest": "0aef4422ed90df41f5d323ae975d5162",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 15135,
            "upload_time": "2019-10-21T15:00:08",
            "upload_time_iso_8601": "2019-10-21T15:00:08.206802Z",
            "url": "https://files.pythonhosted.org/packages/90/b8/44b18667090b378f60b11d0f16bb78700122b55b170e3e462a5af5ec61f7/chastesweep-0.10.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "vulnerabilities": []
}