The GRR Configuration system
============================
:toc:
:toc-placement: preamble
:icons:

The GRR configuration system is the main way users can control and customize the
behavior of GRR components. Previous versions of GRR used command line flags to
configure components, however that became unwieldy with a lot of confusing and
often contradictory command line options embedded in various shell scripts. We
decided to replace the configuration system with a new file based system.

Overview.
---------

When software is written with the GRR configuration system, it can define
parameters that it is going to use. Parameters have the general form
'Section.parameter_name' where 'Section' corresponds to the component which
defines related parameters. For example, the 'Logging' subsystem can define
several parameters: 'Logging.path', 'Logging.verbose' etc. The use of section
names ensures that different components can safely reuse the same parameter name
without clashes.

In addition to the parameter name, it is possible to declare a helpful message,
to assist the user in understanding the purpose of this parameter. Parameters
are also strongly typed. For example we can find the following code:

[source,python]
--------------------------------------------------------------------
config_lib.DEFINE_float("Client.rss_max", 500,
                        "Maximum memory footprint in MB.")

config_lib.DEFINE_option(type_info.PEMPrivateKey(
    name="Client.private_key",
    description="Client private key in pem format. If not provided this "
    "will be generated by the enrollment process.",
    ))
--------------------------------------------------------------------

The first definition declares the parameter 'Client.rss_max' to be a float, with
a default value of 500.

The second definition is more complex. It defines the 'Client.private_key'
parameter as a 'PEMPrivateKey' type info declaration. This declaration allows
the configuration system itself to verify that the parameter is a valid private
key in PEM format. There is now no need for the code to specifically test for
sanity of the private key. If the user provides an invalid or corrupt PEM
encoded key, it is caught immediately by the config system and the program
aborts. For example the following happens after removing a random character from
the 'CA.certificate' PEM string:

--------------------------------------------------------------------
Traceback (most recent call last):
  File "grr/client/client.py", line 48, in <module>
    conf.StartMain(main)
  File "grr/client/conf.py", line 93, in StartMain
    main([sys.argv[0]])
......
  File "grr/lib/config_lib.py", line 467, in _VerifyParameters
    descriptor.Validate(value)
  File "grr/lib/rdfvalues/crypto.py", line 173, in Validate
    return self.ParseFromString(value)
  File "grr/lib/rdfvalues/crypto.py", line 185, in ParseFromString
    raise type_info.TypeValueError("Certificate %s is invalid." % self.name)
grr.lib.type_info.TypeValueError: Certificate CA.certificate is invalid.
--------------------------------------------------------------------

Since every parameter is defined in advance and has a default value, the
configuration file does not need to specify every parameter - only those which
should be changed from the default value. In order to see all declared
parameters and their default values, we can issue the '--config_help' command
line flag. This is analogous to the '--help' flag:

--------------------------------------------------------------------
Client.control_urls = ["http://www.example.com/control"]
   List of URLs of the controlling server.
* Value = ['http://localhost:8080/control']

Client.max_out_queue = 10240000
   Maximum size of the output queue.
* Value = 10240000
--------------------------------------------------------------------

We can see that 'Client.control_urls' has a default value of
http://www.example.com/control, it is used to specify the URL of the controlling
server, and currently (after reading the configuration files), the value is set
to http://localhost:8080/control. We can also see that Client.max_out_queue is
not changed from its default value.


Configuration file organization.
--------------------------------

When GRR is installed on a system, a system wide, distribution specific,
configuration file is also installed (by default
'/etc/grr/grr_server.yaml'). This specifies the basic configuration of the GRR
service (i.e. various distribution specific locations). However, the
configuration typically needs to be updated with site specific parameters (for
example new crypto keys, the 'Client.control_urls' setting to allow the client
to connect to the server etc.)

In order to avoid having to trash the user customized configuration files when
GRR is updated in future, we write site specific configuration files into the
'Config.writeback' location (by default '/etc/grr/grr.local.yaml'). This local
file only contains those parameters which are changed from the defaults, or the
system configuration file.

[NOTE]
==============================================================================
The system configuration file is never modified, all updated configurations are
always written to the writeback location. Do not edit the system configuration
file as changes in this file will be lost when GRR is upgraded.
==============================================================================


Using the '--config_help' option will print all the supported options. This
allows users to figure out what parameters can be changed in the writeback
configuration file to override the system configuration, or the
defaults. Generally it is not required for users to edit many configuration
parameters. The most important parameters will be automatically set upon
installation.


Configuration Contexts.
-----------------------

The next important concept to understand is that of configuration contexts. GRR
consists of many components, which share more of their configuration. Often,
however, we want to specify slight differences between each component. If we
were to write a separate configuration file say for each client version
(Currently, 3 operating systems, and 2 architectures) as well as for each
component (e.g. worker, enroller, frontend etc), we would end up with an
unmaintainable mess of many configuration files. It would make more sense to put
most of the common configuration parameters in the same file, and only specify
the different ones, depending on the component which is running.

GRR introduces the concept of running context. The context is a list of
attributes about the currently running component. When the code accesses the
configuration system, the configuration system considers the currently running
context and returns the appropriate value of the required parameter.

For example, consider the parameter 'Logging.path' which specifies the location
of the log file. In a full GRR installation, we wish the admin UI to log to
'/var/log/grr/adminui.log' while the frontend should log to
'/var/log/grr/frontend.log'. We can therefore specify in the YAML config file:

--------------------------------------------------------------------
Logging.path: /var/log/grr/grr_server.log

AdminUI Context:
  Logging.path: /var/log/grr/adminui.log

HTTPServer Context:
  Logging.path: /var/log/grr/httpserver.log

Client Context:
  Logging.path: /var/log/grr/grr_client.log

  Platform:Windows:
    Logging.path: c:\\windows\\system32\\grr.log


--------------------------------------------------------------------

When the 'adminui.py' program starts up, it populates its context with 'AdminUI
Context'. This forces the configuration system to return the value specific to
this context. Note that other components, such as the worker will set a context
of 'Worker Context', which will not match any of the above overrides. Therefore
the worker will simply log to the default of '/var/log/grr/grr_server.log'.

Note that it is possible to have multiple contexts defined at the same time. So
for example, the client running under windows will have the contexts, 'Client
Context' and 'Platform:Windows', while a client running under Linux will have
the context 'Client Context' and 'Platform:Linux'. When several possibilities
can apply, the option which matches the most context parameters will be
selected. So in the above example, linux and osx clients will have a context
like 'Client Context' and will select '/var/log/grr/grr_client.log'. On the
other hand, a windows client will also match the 'Platform:Windows' context, and
will therefore select 'c:\windows\system32\grr.log'.

The following is a non-exhaustive list of available contexts:

- 'Client Context', 'Worker Context', 'Console Context' etc. Are defined when
  running one of the main programs.
- 'Test Context' is defined when running unit tests.
- 'Arch:i386' and 'Arch:amd64' are set when running a 32 or 64 bit binary.
- 'Installer Context' is defined when the windows installer is active
  (i.e. during installation)
- 'Platform:Windows', 'Platform:Linux', 'Platform:Darwin' are set when running
  under these platforms.


Parameter Expansions
--------------------

The GRR configuration file format allows for expansion of configuration
parameters inside other parameters. For example consider the following
configuration file:

[source,conf]
--------------------------------------------------------------------
Client.name: GRR

Nanny.service_name: "%(Client.name)service.exe"
Nanny.service_key_hive: HKEY_LOCAL_MACHINE
Nanny.service_key: Software\\%(Client.name)
Nanny.child_command_line: |
  %(child_binary) --config "%(Client.install_path)\\%(Client.binary_name).yaml"
--------------------------------------------------------------------

The expansion sequence '%(parameter_name)' substitutes or expands the parameter
into the string. This allows us to build values automatically based on other
values. For example above, the 'Nanny.service_name' will be
'GRRservice.exe' and the service_key will be set to 'Software\GRR'

Expansion is very useful, but sometimes it gets in the way. For example, if we
need to pass literal % escape sequences. Consider the Logging.format parameter
which is actually a python format string:

[source,conf]
--------------------------------------------------------------------
Logging.format: \%(levelname\)s \%(module\)s:\%(lineno\)s] \%(message\)s  # <1>
Logging.format: %{%(levelname)s %(module)s:%(lineno)s] %(message)s}       # <2>
--------------------------------------------------------------------

<1> This form escapes GRR's special escaping sequence by preceding both opening
and closing sequences with the backslash character.

<2> This variation uses the literal expansion sequence '%{}' to declare the
entire line as a literal string and prevent expansion.

Filtering
---------

The configuration system may be extended to provide additional functionality
accessible from the configuration file. For example consider the following
configuration file:

[source,conf]
--------------------------------------------------------------------
Logging.path: "%(HOME|env)/grr.log"                             # <1>
--------------------------------------------------------------------

<1> This expansion sequence uses the 'env' filter which expands a value from the
environment. In this case the environment variable 'HOME' will be expanded into
the 'Logging.path' parameter to place the log file in the user's home directory.

There are a number of additional interesting filters. The 'file' filter allows
including other files from the filesystem into the configuration file. For
example, some people prefer to keep their certificates in separate files rather
than paste them into the config file:

[source,conf]
--------------------------------------------------------------------
CA.certificate: "%(/etc/certificates/ca.pem|file)"
--------------------------------------------------------------------

It is even possible to nest expansion sequences. For example this retrieves the
client's private key from a location which depends on the client name:

[source,conf]
--------------------------------------------------------------------
Client.private_key: "%(/etc/grr/client/%(Client.name)/private_key.pem|file)"
--------------------------------------------------------------------


Configuration parsers
---------------------

The above describes the GRR configuration system as an yaml file and that is the
most commonly used format. However, configuration files are parsed using a
parser implementation which can be switched on demand. Currently there are a few
configuration parser implementations but additional ones can be added.

The windows client implements a registry based configuration parser. This allows
GRR to use the registry instead of a flat file when running on windows. The
registry base parser is selected by using a URL of the form:

[source,sh]
--------------------------------------------------------------------
client.py --config reg://HKEY_LOCAL_MACHINE/Software/GRR
--------------------------------------------------------------------

Additionally an ini based configuration parser can be selected using a .conf
extension:

[source,sh]
--------------------------------------------------------------------
client.py --config /etc/grr.conf
--------------------------------------------------------------------

Since ini files can not represent nested sections, it is impossible to specify
different parameters for different contexts. It is therefore not recommended to
use ini files for the full system config, but they are sufficient for use as a
write back location.
