= Troubleshooting =

The many different steps makes the GRR server installation process susceptible to errors. Here some common issues and solutions are described.

== ImportError: cannot import name jobs_pb2 or similar ==

If you see "ImportError: cannot import name jobs_pb2" or a similar error for any other _pb2 file, you need to regenerate the protobuf files.  Just run
-------------------------------------------------------
python setup.py build
sudo python setup.py install
-------------------------------------------------------

== The upstart/init.d scripts show no output ==

When I run an init.d script e.g. "/etc/init.d/grr-http-server start" it does not show me any output.

Make sure that the "START" parameter in the corresponding default file, e.g. "/etc/default/grr-http-server", has been changed to "yes".

== I cannot start any/some of the GRR services using the init.d scripts ==

When it run an init.d script e.g. "/etc/init.d/grr-http-server start" it indicates it started the service although when I check with "/etc/init.d/grr-http-server status" it says it is not running.

You can troubleshoot by running the services in the foreground, e.g. to run the HTTP Front-end server in the foreground:
-------------------------------------------------------
sudo grr_server --start_http_server --verbose
-------------------------------------------------------

== Any/some of the GRR services are not running correctly ==

Check if the logs contain an indication of what is going wrong.

Troubleshoot by running the services in the foreground, e.g. to run the UI in the foreground:
-------------------------------------------------------
sudo grr_server --verbose --start_ui
-------------------------------------------------------

== Cannot open libtsk3.so.3 ==

error while loading shared libraries: libtsk3.so.3: cannot open shared object file: No such file or directory

The libtsk3 library cannot be found in the ld cache. Check if the path to libtsk3.so.3 is in /etc/ld.so.conf (or equivalent) and update the cache:
-------------------------------------------------------
sudo ldconfig
-------------------------------------------------------

== Cron Job view reports an error ==

Delete and recreate all the cronjobs using GRR console:
-----------------------------------------------------------------------
aff4.FACTORY.Delete("aff4:/cron", token=data_store.default_token)
from grr.server.aff4_objects import cronjobs
cronjobs.ScheduleSystemCronFlows(token=data_store.default_token)
-----------------------------------------------------------------------

== Protobuf 2.5.0 ==

GRR only works with the Protobuf library v2.5.0 or newer. If your distribution
comes with an older version, follow the instructions below:

Remove any version of protobuf that is older than 2.5.0.

Note that this will also remove a couple of dependencies:

-------------------------------------------------------
sudo apt-get remove libprotobuf python-protobuf
-------------------------------------------------------

To build Protobuf from source, install the following packages on a Debian-based
system:

--------------------------------------------------------------------
sudo apt-get install build-essential uuid-dev zlib1g-dev python-dev
--------------------------------------------------------------------

Download protobuf version 2.5.0 or later from link:http://code.google.com/p/protobuf/[protobuf]

Extract the package:
-------------------------------------------------------
tar xfv protobuf-2.5.0.tar.gz
cd protobuf-2.5.0
-------------------------------------------------------

Build and install libprotobuf:
-------------------------------------------------------
./configure --prefix=/usr/
make
sudo make install
sudo ldconfig
-------------------------------------------------------

Build and install the Python bindings:
-------------------------------------------------------
cd python
python setup.py build
python setup.py test
sudo python setup.py install
-------------------------------------------------------

== Missing Rekall Profiles ==

If you get errors like: 
----
Error loading profile: Could not load profile nt/GUID/ABC123...
Needed profile nt/GUID/ABC123... not found!
----

when using rekall, you're missing a profile (see the link:http://www.rekall-forensic.com/faq.html[Rekall FAQ] and link:http://www.rekall-forensic.com/posts/2014-02-20-profile-selection.html[blogpost] for some background about what this means).

The simplest way to get this fixed is to add it into link:https://github.com/google/rekall-profiles/blob/gh-pages/v1.0/src/guids.txt[Rekall's list of GUIDs], which is of great benefit to the whole memory forensics community.  You can do this yourself via a pull request on link:https://github.com/google/rekall-profiles[rekall-profiles], or simply email the GUID to rekall-discuss@googlegroups.com.  Once it's in the public rekall server, the GRR server will download and use it automatically next time you run a rekall flow that requires that profile.  If your GRR server doesn't have internet access you'll need to run the GetMissingProfiles function from the GRR console on a machine that has internet access and can access the GRR database, like this:

```
from grr.server import rekall_profile_server
rekall_profile_server.GRRRekallProfileServer().GetMissingProfiles()
```

== I don't see my clients ==

Here are common client troubleshooting steps you can work through to diagnose client install and communications problems.

*Check that the client got installed*

You should have something like:

 - ``C:\Windows\System32\GRR\3.1.0.2\GRR.exe`` on windows;
 - ``/usr/lib/grr/grr_3.1.0.2_amd64/grrd`` on linux; and
 - ``/usr/local/lib/grr_3.1.0.2_amd64/grrd`` on OSX
 
with variations based on the GRR version and architecture you installed (32bit is i386). If you didn't get this far, then there is a problem with your installer binary.

*Check the client is running*

On linux and OS X you should see two processes, something like:
----
$ ps aux | grep grr
root       957  0.0  0.0  11792   944 ?        Ss   01:12   0:00 /usr/sbin/grrd --config=/usr/lib/grr/grr_3.1.0.2_amd64/grrd.yaml
root      1015  0.2  2.4 750532 93532 ?        Sl   01:12   0:01 /usr/sbin/grrd --config=/usr/lib/grr/grr_3.1.0.2_amd64/grrd.yaml
----

On windows you should see a ``GRR Monitor`` service and a ``GRR.exe`` process in taskmanager.

If it isn't running check the install logs and other logs in the same directory:

- Linux/OS X: ``/var/log/grr_installer.txt``
- Windows: ``C:\Windows\System32\LogFiles\GRR_installer.txt``

and then try running it interactively as below.

*Check the client machine can talk to the server*

The URL here should be the server address and port you picked when you set up the server and listed in ``Client.control_urls`` in the client's config.
----
wget http://yourgrrserver.yourcompany.com:8080/server.pem
# Check your config settings, note that clients earlier than 3.1.0.2 used Client.control_urls
$ sudo cat /usr/lib/grr/grr_3.1.0.2_amd64/grrd.yaml | grep Client.server_urls
Client.server_urls: http://yourgrrserver.yourcompany.com:8080/
----

If you can't download that server.pem, the common causes are that your server isn't running or there are firewalls in the way.

*Run the client in verbose mode*

Linux: Stop the daemon version of the service and run it in verbose mode:
----
$ sudo service grr stop
$ sudo /usr/sbin/grrd --config=/usr/lib/grr/grr_3.1.0.2_amd64/grrd.yaml --verbose
----

OS X: Unload the service and run it in verbose mode:
----
$ sudo launchctl unload /Library/LaunchDaemons/com.google.code.grr.plist
$ sudo /usr/local/lib/grr_3.1.0.2_amd64/grrd --config=/usr/local/lib/grr_3.1.0.2_amd64/grrd.yaml --verbose
----

Windows: The normal installer isn't a terminal app, so you don't get any output if you run it interactively. 

- Install the debug ``dbg_GRR_3.1.0.2_(amd64|i386).exe`` version to make it a terminal app.
- Stop the ``GRR Monitor`` service in task manager

Then run in a terminal as Administrator
----
cd C:\Windows\System32\GRR\3.1.0.2\
GRR.exe --config=GRR.exe.yaml --verbose
----

If this is a new client you should see some 406's as it enrols, then they stop and are replaced with sending message lines with increasing sleeps in between. The output should look similar to this:

```
Starting client aff4:/C.a2be2a27a8d69c61 
aff4:/C.a2be2a27a8d69c61: Could not connect to server at http://somehost:port/, status 406
Server PEM re-keyed.
sending enrollment request
aff4:/C.a2be2a27a8d69c61: Could not connect to server at http://somehost:port/, status 406
Server PEM re-keyed.
sending enrollment request
aff4:/C.a2be2a27a8d69c61: Could not connect to server at http://somehost:port/, status 406
Server PEM re-keyed.
aff4:/C.a2be2a27a8d69c61: Sending 3(1499), Received 0 messages in 0.771058797836 sec. Sleeping for 0.34980125
aff4:/C.a2be2a27a8d69c61: Sending 0(634), Received 0 messages in 0.370272874832 sec. Sleeping for 0.4022714375
aff4:/C.a2be2a27a8d69c61: Sending 0(634), Received 0 messages in 0.333703994751 sec. Sleeping for 0.462612153125
aff4:/C.a2be2a27a8d69c61: Sending 0(634), Received 0 messages in 0.345727920532 sec. Sleeping for 0.532003976094
aff4:/C.a2be2a27a8d69c61: Sending 0(634), Received 0 messages in 0.346176147461 sec. Sleeping for 0.611804572508
aff4:/C.a2be2a27a8d69c61: Sending 0(634), Received 8 messages in 0.348709106445 sec. Sleeping for 0.2
```

If enrolment isn't succeeding (406s never go away), make sure you're running a worker process and check logs in `/var/log/grr-worker.log`.
