{ "info": { "author": "D. Brown", "author_email": "dorianstuartbrown@gmail.com", "bugtrack_url": null, "classifiers": [], "description": "# Rank-BM25: A two line search engine\n\nA collection of algorithms for querying a set of documents and returning the ones most relevant to the query. The most common use case for these algorithms is, as you might have guessed, to create search engines.\n\nSo far the algorithms that have been implemented are:\n- Okapi BM25\n- BM25L\n- BM25+\n\nTodo:\n- BM25-Adpt\n- BM25T \n\nThese algorithms were taken from [this paper](http://www.cs.otago.ac.nz/homepages/andrew/papers/2014-2.pdf), which gives a nice overview of each method, and also benchmarks them against each other. A nice inclusion is that they compare different kinds of preprocessing like stemming vs no-stemming, stopword removal or not, etc. Great read if you're new to the topic. \n\n## Installation\nThe easiest way to install this package is through `pip`, using\n```bash\npip install rank_bm25\n```\nIf you want to be sure you're getting the newest version, you can install it directly from github with\n```bash\npip install git+ssh://git@github.com/dorianbrown/rank_bm25.git\n```\n\n## Usage\nFor this example we'll be using the `BM25Okapi` algorithm, but the others are used in pretty much the same way.\n\n### Initalizing\n\nFirst thing to do is create an instance of the BM25 class, which reads in a corpus of text and does some indexing on it:\n```python\nfrom rank_bm25 import BM25Okapi\n\ncorpus = [\n \"Hello there good man!\",\n \"It is quite windy in London\",\n \"How is the weather today?\"\n]\n\ntokenized_corpus = [doc.split(\" \") for doc in corpus]\n\nbm25 = BM25Okapi(tokenized_corpus)\n# \n```\nNote that this package doesn't do any text preprocessing. If you want to do things like lowercasing, stopword removal, stemming, etc, you need to do it yourself. \n\nThe only requirements is that the class receives a list of lists of strings, which are the document tokens.\n\n### Ranking of documents\n\nNow that we've created our document indexes, we can give it queries and see which documents are the most relevant:\n```python\nquery = \"windy London\"\ntokenized_query = query.split(\" \")\n\ndoc_scores = bm25.get_scores(tokenized_query)\n# array([0. , 0.93729472, 0. ])\n```\nGood to note that we also need to tokenize our query, and apply the same preprocessing steps we did to the documents in order to have an apples-to-apples comparison\n\nInstead of getting the document scores, you can also just retrieve the best documents with\n```python\nbm25.get_top_n(tokenized_query, corpus, n=1)\n# ['It is quite windy in London']\n```\nAnd that's pretty much it!", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/dorianbrown/rank_bm25", "keywords": "", "license": "Apache2.0", "maintainer": "", "maintainer_email": "", "name": "rank-bm25", "package_url": "https://pypi.org/project/rank-bm25/", "platform": "", "project_url": "https://pypi.org/project/rank-bm25/", "project_urls": { "Homepage": "https://github.com/dorianbrown/rank_bm25" }, "release_url": "https://pypi.org/project/rank-bm25/0.2/", "requires_dist": null, "requires_python": "", "summary": "Various BM25 algorithms for document ranking", "version": "0.2" }, "last_serial": 5124816, "releases": { "0.1": [ { "comment_text": "", "digests": { "md5": "c8d95a60640e3a7de9181c31729059a7", "sha256": "594a1d1f98a9818ddcc6b11210b868945dc9c034f7c8f0ce75097de8f84a506f" }, "downloads": -1, "filename": "rank_bm25-0.1-py3-none-any.whl", "has_sig": false, "md5_digest": "c8d95a60640e3a7de9181c31729059a7", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 8113, "upload_time": "2019-01-24T23:14:15", "url": "https://files.pythonhosted.org/packages/06/5f/23ac059dbc81f3e7a6ae7e25e4203407a9246bd12894f89d70efe408cd26/rank_bm25-0.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "53d19e7570566dbbdb3a63abb2ce0739", "sha256": "53c19803f6a0a52134b2e5dbf4c5252e41a90a30bac49ff15dc0285da05d6b2a" }, "downloads": -1, "filename": "rank_bm25-0.1.tar.gz", "has_sig": false, "md5_digest": "53d19e7570566dbbdb3a63abb2ce0739", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 3928, "upload_time": "2019-01-24T23:14:17", "url": "https://files.pythonhosted.org/packages/8f/b1/4d56d4ba2194e33a7fe81985fc0b7d8b07984596cefb9a88703c1fbe5a90/rank_bm25-0.1.tar.gz" } ], "0.2": [ { "comment_text": "", "digests": { "md5": "3e11e1cd83106b1a6ff3e466e01c93d1", "sha256": "70c2441444b1e319a97bec3df050fc6bb7ab749fe84f602d597cb8b71312ee71" }, "downloads": -1, "filename": "rank_bm25-0.2.tar.gz", "has_sig": false, "md5_digest": "3e11e1cd83106b1a6ff3e466e01c93d1", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 4151, "upload_time": "2019-04-10T15:59:39", "url": "https://files.pythonhosted.org/packages/d2/e4/38d03d6d5e2deae8d2838b81d6ba2742475ced42045f5c46aeb00c5fb79c/rank_bm25-0.2.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "3e11e1cd83106b1a6ff3e466e01c93d1", "sha256": "70c2441444b1e319a97bec3df050fc6bb7ab749fe84f602d597cb8b71312ee71" }, "downloads": -1, "filename": "rank_bm25-0.2.tar.gz", "has_sig": false, "md5_digest": "3e11e1cd83106b1a6ff3e466e01c93d1", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 4151, "upload_time": "2019-04-10T15:59:39", "url": "https://files.pythonhosted.org/packages/d2/e4/38d03d6d5e2deae8d2838b81d6ba2742475ced42045f5c46aeb00c5fb79c/rank_bm25-0.2.tar.gz" } ] }