{
    "info": {
        "author": "The Rekall team",
        "author_email": "rekall-discuss@googlegroups.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "Environment :: Console",
            "Operating System :: OS Independent",
            "Programming Language :: Python"
        ],
        "description": "# Rekall Layout Expert\n\nLive Memory analysis on the Linux platform has traditionally been difficult to\nperform. Memory analysis requires precise knowledge of struct layout information\nin memory, usually obtained through debugging symbols generated at compile\ntime. The Linux kernel is however, highly configurable, implying that debugging\ninformation is rarely applicable to systems other than the ones that generated\nit. For incident response applications, obtaining the relevant debugging\ninformation is currently a slow and manual process, limiting its usefulness in\nrapid triaging.\n\n## How do we analyze Linux systems right now?\n\nThe current process for generating a Rekall profile for a Linux system is\ntedious:\n\n1) You must find and install the kernel headers package for the same kernel as\nthe running kernel (for example `apt-get install\nlinux-headers-3.16.0-39-generic`).\n\n2) Then you need to build a kernel module (`rekall/tools/linux/module.c`) on\nthat system to generate the debug kernel module `module_dwarf.ko`.\n\n3) Finally on a system with Rekall installed, one needs to convert this to a\nRekall profile (using `rekall convert_profile 4.2.0-generic.zip\n4.2.0-generic.json` for example).\n\nThis is hard to do in an incident response situation. Sometimes servers do not\nhave the required compilers, kernel headers etc. This is especially hard if the\nkernel was cusom made. In that case it may be difficult to even find the\nrequired kernel headers package (it may not have even been built with the custom\nkernel). In all likelyhood you may need to copy the kernel config and System.map\noff the system you want to analyze to another system (with compiler tool chains\nand kernel headers installed) so you can build the profile.\n\nThis logistical issue make it difficult to do Linux live memory analysis in\npractice - so you end up taking a memory image of the system for later analysis\n(Then you have to deal with transferring huge images around, smear and lot of\nother fun problems :-).\n\nIf you really want to be prepared, you must build a huge library of kernel\nprofiles in advance. For each released kernel version, you need to have every\nvariation released by every distribution. For example in Ubuntu, there are\ngeneric and low latency variation (e.g. `linux-headers-3.16.0-39-generic`,\n`linux-headers-3.16.0-39-lowlatency`) for each minor version). You can just\nforget about having custom kernels in your library because you can not predict\nin advance what config parameters someone will change!\n\n## Is there a better way?\n\nHave you ever found yourself uttering: \"I will pay someone $1000 to find a way\nto do Linux Memory forensics without building a *$@#!% profile for every #@$#%@#\nkernel?\" - I know I have!\n\nIn a perferct world, we would just run Rekall on any Linux system, point it at\n`/proc/kcore` or `/dev/pmem` and just go without worrying about building\nprofiles! That would be nice.\n\nWe are not quite there, but almost :-). The Layout Expert is the small step\nforward. The process using the Layout Expect is much simpler:\n\n1) On the system you want to analyze, run the Layout Expert which will download\na single Pre-AST file for every kernel version (regardless of kernel\nconfiguration, distribution flavour etc.).\n\n2) Then launch the layout expert, providing it the local system's config file and\nSystem.map:\n\n```\n$ layout_tool make_profile --config_file_path boot/config-4.2.02.0.smp \\\n    --system boot/System.map-4.2.02.0.smp pre_ast_4.2.0-22.json profile.json\n\n2016-01-23 09:44:29,416 INFO     LOADING PREPROCESSOR AST FROM: pre_ast_4.2.0-22.json\n2016-01-23 09:44:34,494 INFO     DONE\n2016-01-23 09:44:34,495 INFO     LINKING INCLUDES\n2016-01-23 09:44:34,937 INFO     LINKED\n2016-01-23 09:44:34,937 INFO     EXTRACTING CONFIG FLAGS\n2016-01-23 09:44:34,994 INFO     EXTRACTED\n2016-01-23 09:44:35,108 INFO     PREPROCESSING\n2016-01-23 09:44:50,856 INFO     PREPROCESSED\n2016-01-23 09:44:50,856 INFO     Completed preprocessing pre-ast in 16 Seconds\n2016-01-23 09:44:50,856 INFO     GENERATING PURE C FILE\n2016-01-23 09:44:53,047 INFO     GENERATED\n2016-01-23 09:44:53,048 INFO     Completed generating pure C file in 2 Seconds\n2016-01-23 09:44:53,048 INFO     TRIMMING C FILE\n2016-01-23 09:45:14,340 INFO     Completed trimming C file in 21 Seconds\n2016-01-23 09:45:14,341 INFO     TRIMMED C FILE\n2016-01-23 09:45:14,354 INFO     PARSING STRUCTS\n2016-01-23 09:45:37,853 INFO     Completed parsing struct layouts in 23 Seconds\n2016-01-23 09:45:37,853 INFO     PARSED\n2016-01-23 09:45:37,853 INFO     GENERATING PROFILE\n2016-01-23 09:45:37,949 INFO     Exporting 627 structs\n2016-01-23 09:45:38,763 INFO     GENERATED\n```\n\nThe Layout Expert is able to calculate the memory layout of critical kernel\nstructures at runtime on the target system without requiring extra tools, such\nas the compiler tool-chain to be pre-installed.\n\n## How does it work?\n\nThe main problem with memory analysis on Linux is that the Linux kernel is so\nconfigurable and customizable. For example, in order to properly parse the\nmemory layout of `struct task_struct`, we can see the source:\n\n```\nstruct task_struct {\n        volatile long state;    /* -1 unrunnable, 0 runnable, >0 stopped */\n        void *stack;\n        atomic_t usage;\n        unsigned int flags;     /* per process flags, defined below */\n        unsigned int ptrace;\n\n#ifdef CONFIG_SMP\n        struct llist_node wake_entry;\n        int on_cpu;\n        struct task_struct *last_wakee;\n        unsigned long wakee_flips;\n        unsigned long wakee_flip_decay_ts;\n\n        int wake_cpu;\n#endif\n        int on_rq;\n\n        int prio, static_prio, normal_prio;\n        unsigned int rt_priority;\n        const struct sched_class *sched_class;\n        struct sched_entity se;\n        struct sched_rt_entity rt;\n#ifdef CONFIG_CGROUP_SCHED\n        struct task_group *sched_task_group;\n#endif\n\n#ifdef CONFIG_PREEMPT_NOTIFIERS\n        /* list of struct preempt_notifier: */\n        struct hlist_head preempt_notifiers;\n#endif\n\n....\n```\n\nDepending on kernel configuration options there will be different members\ninserted in the middle of the struct - even for the same kernel version. This is\nprimarily why you need to compile a debug kernel module for every single kernel\nconfiguration - even of the same version. Depending on various kernel config\noptions the layouts can change dramatically (sometimes if the profiles are very\nclose some fields will be parsed correctly by Rekall but others wont - the\nfamiliar missing data in plugin outputs).\n\n\nThe Layout Expert attempts to emulate the GCC compiler chains to the extent of\nbeing able to predict the struct layout that the compiler might decide\non. Essentially we simulate the compilation of the kernel debug module.\n\nThe GCC compiler, reads the kernel config file and then preprocesses the kernel\nheaders to add or remove code depending on these configuration options. In the\nLayout Expert we wish to have a data structure that can be re-used for different\nconfigurations without needing the kernel config.\n\nTherefore, the Layout Expert first parses all the kernel headers into a\nPreprocessor Abstract Syntax Tree (Pre-AST for short). The Pre-AST includes all\nthe possibilities of each `#ifdef` branch. This is the file which the Layout\nExpert operates on.\n\nAt runtime (i.e. during system analysis), the Layout Expert combines the system\nconfiguration with the Pre-AST to produced the Preprocessed C code. In essence,\nthe headers `#ifdef` directives are removed, and the different options are\ncombined to produce a final C file, free from preprocessing macros. In this\nphase, the Layout Expert acts as a C pre-processor. The result is a huge C file\nwith all the code in all the headers stuck together.\n\nNext, the Layout Expert applies trimming to this file. This is essentially a\nquick once over pass to identify only structs, unions, enums and typedef\ninstructions. This optimization step means that we have much less code to parse\nin the next step and that the code that we do need to parse is more consistent\nand so easier to parse.\n\nFinally the Layout Expert parses the structs that Rekall is actually interested\nin (i.e. those structs with plugins that look at them). This parsing phase\nemulates a C compiler. We then apply the GCC struct layout model to the parsed\nstructs in order to predict the precise memory layout of all fields in the\nstructs (considering attributes, e.g. ``__attribute__((packed))`,\n`__attribute__((aligned(8))`).\n\nThe last step is to generate a Rekall profile ready for use.\n\n\n## Preparation.\n\nBefore the profile generation can occur in the field, we need to build the\n`Pre-AST` file for the specific kernel version. This is easy since it does not\nrequire any specific configuration file or special tools (Remember that the\nPre-AST includes all branches of any `#ifdef` directives so we do not need to\nevaluate any macros at this stage.).\n\nYou can use the kernel headers package for the specific kernel, or the full\nkernel source. There is no need to actually compile the source (i.e. `make\ndepmod` etc). Note that the kernel header package does not include \"private\" or\nnon exported structs, so these will be missing from the profile, but current\nRekall does not need those.\n\n```\n$ layout_tool build_pre_ast --source_file_path ~/rekall/tools/linux/module.c \\\n  --linux_repository_path /usr/src/linux-headers-4.2.0-22-generic/ pre_ast_4.2.0-22.json\n\n2016-01-23 10:38:00,493 INFO     LOADING AND PARSING HEADERS\n2016-01-23 10:38:58,912 INFO     Completed built pre-ast forest in 58 Seconds\n2016-01-23 10:38:58,913 INFO     LOADED AND PARSED\n```\n\n## Bugs and support\n\nThe Layout Expert is brought to you by the same people who develop Rekall, but\nit is considered a separate project. It is available under an Apache license\n(Check the LICENSE file). However, there is no official support or warranty; not\neven for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.\n\nIf you think you've found a bug, please report it at:\n\n    https://github.com/google/rekall/issues\n\nYou can also mail to the list rekall-discuss@googlegroups.com",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://www.rekall-forensic.com/",
        "keywords": "",
        "license": "Apache",
        "maintainer": "",
        "maintainer_email": "",
        "name": "rekall-layout-expert",
        "package_url": "https://pypi.org/project/rekall-layout-expert/",
        "platform": "UNKNOWN",
        "project_url": "https://pypi.org/project/rekall-layout-expert/",
        "project_urls": {
            "Homepage": "https://www.rekall-forensic.com/"
        },
        "release_url": "https://pypi.org/project/rekall-layout-expert/1.5.0.post4/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "Rekall Layout Expert",
        "version": "1.5.0.post4"
    },
    "last_serial": 2690215,
    "releases": {
        "1.4.1.post0.dev128": [],
        "1.5.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "09eea5590f552a1fe97b6e44f7edc3f8",
                    "sha256": "f1967d50eb00b533b873f053e6bebb2d9806cbef6851564ebc8b6505923a062b"
                },
                "downloads": -1,
                "filename": "rekall-layout-expert-1.5.0.zip",
                "has_sig": false,
                "md5_digest": "09eea5590f552a1fe97b6e44f7edc3f8",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 146535,
                "upload_time": "2016-03-28T06:13:32",
                "url": "https://files.pythonhosted.org/packages/bc/9a/79fee3b2140fae165c18d6eb9fd84ac0103ff937835c1a53ac3d6924b741/rekall-layout-expert-1.5.0.zip"
            }
        ],
        "1.5.0.post4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "3b8d7667cf2f863e01024fffdb845640",
                    "sha256": "d599f56f732ea9de4d72f06dc8915a15ba40a46f8568059e9a5aa961f1fc8711"
                },
                "downloads": -1,
                "filename": "rekall-layout-expert-1.5.0.post4.zip",
                "has_sig": false,
                "md5_digest": "3b8d7667cf2f863e01024fffdb845640",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 147491,
                "upload_time": "2016-03-29T00:01:29",
                "url": "https://files.pythonhosted.org/packages/c8/73/148696164da6d7d813911a4462b9efb873b3dd5a5446586a1b5f93abb1a4/rekall-layout-expert-1.5.0.post4.zip"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "3b8d7667cf2f863e01024fffdb845640",
                "sha256": "d599f56f732ea9de4d72f06dc8915a15ba40a46f8568059e9a5aa961f1fc8711"
            },
            "downloads": -1,
            "filename": "rekall-layout-expert-1.5.0.post4.zip",
            "has_sig": false,
            "md5_digest": "3b8d7667cf2f863e01024fffdb845640",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 147491,
            "upload_time": "2016-03-29T00:01:29",
            "url": "https://files.pythonhosted.org/packages/c8/73/148696164da6d7d813911a4462b9efb873b3dd5a5446586a1b5f93abb1a4/rekall-layout-expert-1.5.0.post4.zip"
        }
    ]
}