PK ! h3 3 README.md# Shell History
[](https://gitlab.com/pawamoy/shell-history/commits/master)
Inspired by [bamos/zsh-history-analysis](https://github.com/bamos/zsh-history-analysis).
Visualize your usage of Bash/Zsh through a web app
thanks to [Flask](http://flask.pocoo.org/) and [Highcharts](https://www.highcharts.com/)!
Duration
Length
Type
Exit code
Hourly
Daily
Over time
Markov chain
Top commands
- [Requirements](#requirements)
- [Installation](#installation)
- [Usage](#usage)
- [How it works](#how-it-works)
- [History file format](#history-file-format)
- [Chart ideas](#chart-ideas)
- [License](#license)
## Requirements
`shellhistory` requires Python 3.6 or above.
To install Python 3.6, I recommend using pyenv.
```bash
# install pyenv
git clone https://github.com/pyenv/pyenv ~/.pyenv
# setup pyenv (you should also put these two lines in .bashrc or similar)
export PATH="${HOME}/.pyenv/bin:${PATH}"
eval "$(pyenv init -)"
# install Python 3.6
pyenv install 3.6.7
# make it available globally
pyenv global system 3.6.7
```
## Installation
With `pip`:
```bash
python3.6 -m pip install shellhistory
```
With [`pipx`](https://github.com/pipxproject/pipx):
```bash
python3 -m pip install --user pipx
pipx install --python python3.6 shellhistory
```
## Setup
`shellhistory` needs a lot of info to be able to display various charts.
The basic shell history is not enough. In order to generate the necessary
information, you have to enable the shell extension.
At shell startup, in `.bashrc` or `.zshrc`, put the following:
```bash
# only load it for interactive shells
if [[ $- == *i* ]] && command -v shellhistory-location &>/dev/null; then
. $(shellhistory-location)
shellhistory enable
fi
```
... and now use your shell normally!
If you want to stop `shellhistory`, simply run `shellhistory disable`.
*Note: for performance reasons, you can also use the static, absolute path to the source file. Indeed, calling `shellhistory-location` spawns a Python process which can slow down your shell startup. Get the path once with `shellhistory-location`, and use `. `. In my case it's `. ~/.local/pipx/venvs/shellhistory/lib/python3.6/site-packages/shellhistory/shellhistory.sh`.*
## Usage
Launch the web app with `shellhistory-web`.
Now go to [http://localhost:5000/](http://localhost:5000/) and enjoy!
You will need Internet connection since assets are not bundled.
## How it works
In order to append a line each time a command is entered, the `PROMPT_COMMAND`
variable and the `precmd` function are used, respectively for Bash and Zsh.
They allow us to execute arbitrary instructions just before command execution,
or before the command prompt is displayed, meaning, just after the last command
has returned.
This is where we compute the start and stop time, return code, working
directory and command type, and append the line into our history file.
Start and stop time are obtained with `$(date '+%s%N')`, return code is passed
directly with `$?`, working directory is obtained with `$PWD` and command
type with `$(type -t arg)` for Bash and `$(type -w arg)` for Zsh.
Values for UUID, parents, hostname, and TTY are computed only once, when
`shellhistory.sh` is sourced. Indeed they do not change during usage of the current
shell process. Hostname and TTY are obtained through commands `$(hostname)` and
`$(tty)`. UUID is generated with command `$(uuidgen)`. Also note that UUID
is exported in subshells so we know which shell is a subprocess of another, and
so we are able to group shell processes by "sessions", a session being an opened
terminal (be it a tab, window, pane or else). Parents are obtained with a
function that iteratively greps `ps` result with PIDs (see `shellhistory.sh`).
Values for user, shell, and level are simply obtained through environment
variables: `$USER`, `$SHELL`, and `$SHLVL`.
Start time is computed just before the entered command is run by the shell,
thanks to a trap on the DEBUG signal. From Bash manual about `trap`:
>If a sigspec is DEBUG, the command arg is executed before every simple command.
The last command is obtained with the command `fc`. It will feel like your
history configuration is mimic'd by the extended history. If the commands
beginning with a space are ignored, `shellhistory` will notice it and will not
append these commands. Same for ignored duplicates. If you enter an empty line,
or hit Control-C before enter, nothing will be appended either. The trick behind
this is to check the command number in the current history (see `shellhistory.sh`
for technical details). Note however that if you type the same command in an
other terminal, it will still be appended, unless you manage to synchronize your
history between terminals.
## History file format
Fields saved along commands are start and stop timestamps, hostname, username,
uuid (generated), tty, process' parents, shell, shell level, command type,
return code, and working directory (path), in the following format:
`:start:stop:uuid:parents:host:user:tty:path:shell:level:type:code:command`.
- multi-line commands are prepended with a semi-colon `;` instead of a colon `:`,
starting at second line
- start and stop timestamps are in microseconds since epoch
- process' parents and working directory are encoded in base64 to avoid
delimiter corruption
Example (multi-line command):
```
:1510588139930150:1510588139936608:40701d9b-1807-4a3e-994b-dde68692aa14:L2Jpbi9iYXNoCi91c3IvYmluL3B5dGhvbiAvdXNyL2Jpbi94LXRlcm1pbmFsLWVtdWxhdG9yCi91c3IvYmluL29wZW5ib3ggLS1zdGFydHVwIC91c3IvbGliL3g4Nl82NC1saW51eC1nbnUvb3BlbmJveC1hdXRvc3RhcnQgT1BFTkJPWApsaWdodGRtIC0tc2Vzc2lvbi1jaGlsZCAxMiAyMQovdXNyL3NiaW4vbGlnaHRkbQovc2Jpbi9pbml0Cg==:myhost:pawamoy:/dev/pts/1:L21lZGlhL3Bhd2Ftb3kvRGF0YS9naXQvc2hlbGxoaXN0Cg==:/bin/bash:1:builtin:0:echo 'a
;b
;c' | wc -c
```
## Chart ideas
You can post your ideas in this issue: https://github.com/pawamoy/shell-history/issues/9.
## License
Software licensed under the
[ISC](https://www.isc.org/downloads/software-support-policy/isc-license/)
license. See the [LICENSE](/LICENSE) file.
PK ! n pyproject.toml[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
[tool.poetry]
name = "shellhistory"
version = "0.2.2"
description = " Visualize your shell usage with Highcharts!"
license = "ISC"
authors = ["Timothée Mazzucotelli "]
readme = 'README.md'
repository = "https://github.com/pawamoy/shell-history"
homepage = "https://github.com/pawamoy/shell-history"
keywords = ['shell', 'bash', 'zsh', 'history', 'analysis', 'visualization']
packages = [ { include = "shellhistory", from = "src" } ]
include = [
"README.md",
"pyproject.toml"
]
[tool.poetry.dependencies]
python = "~3.6"
flask = "^1.0"
Flask-Admin = "^1.5"
sqlalchemy = "^1.3"
tqdm = "^4.28"
[tool.poetry.dev-dependencies]
pytest = "*"
pytest-cov = "*"
pytest-sugar = "*"
ipython = "^7.2"
[tool.poetry.scripts]
shellhistory-cli = 'shellhistory.cli:main'
shellhistory-location = 'shellhistory.cli:location'
shellhistory-web = 'shellhistory.cli:web'
[tool.black]
line-length = 120
PK ! shellhistory/__init__.pyPK ! B shellhistory/__main__.py"""
Entry-point module, in case you use `python -m shellhistory`.
Why does this file exist, and why __main__? For more info, read:
- https://www.python.org/dev/peps/pep-0338/
- https://docs.python.org/2/using/cmdline.html#cmdoption-m
- https://docs.python.org/3/using/cmdline.html#cmdoption-m
"""
import sys
from shellhistory.cli import main
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
PK ! '5 5 shellhistory/app.py# -*- coding: utf-8 -*-
import time
from collections import Counter, defaultdict
from datetime import datetime, time as dt_time
import statistics
from flask import Flask, jsonify, render_template
from flask_admin import Admin
from flask_admin.contrib.sqla import ModelView
from sqlalchemy import extract, func, desc
from sqlalchemy.sql import func as sqlfunc
from . import db
# Initialization and constants ------------------------------------------------
app = Flask(__name__)
app.secret_key = "2kQOLbr6NtfHV0wIItjHWzuwsgCUXA4CSSBWFE9yELqrkSZU"
db.create_tables()
# Flask Admin stuff -----------------------------------------------------------
class HistoryModelView(ModelView):
can_create = False
can_delete = True
can_view_details = True
create_modal = True
edit_modal = True
can_export = True
page_size = 50
list_template = "admin/history_list.html"
column_exclude_list = ["parents"]
column_searchable_list = [
"id",
"start",
"stop",
"duration",
"host",
"user",
"uuid",
"tty",
"parents",
"shell",
"level",
"type",
"code",
"path",
"cmd",
]
column_filters = ["host", "user", "uuid", "tty", "parents", "shell", "level", "type", "code", "path", "cmd"]
column_editable_list = ["host", "user", "uuid", "tty", "shell", "level", "type", "code", "path", "cmd"]
form_excluded_columns = ["start", "stop", "duration"]
# form_widget_args = {
# 'start': {'format': '%Y-%m-%d %H:%M:%S.%f'},
# 'stop': {'format': '%Y-%m-%d %H:%M:%S.%f'},
# 'duration': {'format': '%Y-%m-%d %H:%M:%S.%f'}
# }
admin = Admin(app, name="Shell History", template_mode="bootstrap3")
admin.add_view(HistoryModelView(db.History, db.get_session()))
# Utils -----------------------------------------------------------------------
def since_epoch(date):
return time.mktime(date.timetuple())
def fractional_year(start, end):
this_year = end.year
this_year_start = datetime(year=this_year, month=1, day=1)
next_year_start = datetime(year=this_year + 1, month=1, day=1)
time_elapsed = since_epoch(end) - since_epoch(start)
year_duration = since_epoch(next_year_start) - since_epoch(this_year_start)
return time_elapsed / year_duration
# Special views ---------------------------------------------------------------
@app.route("/")
def home_view():
return render_template("home.html")
@app.route("/update")
def update_call():
data = {"message": None, "class": None}
try:
report = db.update()
except Exception as e:
data["class"] = "danger"
data["message"] = "%s\n%s: %s" % (
"Failed to import current history. The following exception occurred:",
type(e),
e,
)
else:
if report.inserted:
data["class"] = "success"
data["message"] = (
"Database successfully updated (%s new items), refresh the page to see the change." % report.inserted
)
if report.duplicates:
data["class"] = "info"
data["message"] += "\n%s duplicates were not imported." % report.duplicates
else:
data["class"] = "default"
data["message"] = "Database already synchronized, nothing changed."
return jsonify(data)
# Simple views rendering templates --------------------------------------------
@app.route("/codes")
def codes_view():
return render_template("codes.html")
@app.route("/daily")
def daily_view():
return render_template("daily.html")
@app.route("/daily_average")
def daily_average_view():
return render_template("daily_average.html")
@app.route("/duration")
def duration_view():
return render_template("duration.html")
@app.route("/hourly")
def hourly_view():
return render_template("hourly.html")
@app.route("/hourly_average")
def hourly_average_view():
return render_template("hourly_average.html")
@app.route("/length")
def length_view():
return render_template("length.html")
@app.route("/markov")
def markov_view():
return render_template("markov.html")
@app.route("/markov_full")
def markov_full_view():
return render_template("markov_full.html")
@app.route("/monthly")
def monthly_view():
return render_template("monthly.html")
@app.route("/monthly_average")
def monthly_average_view():
return render_template("monthly_average.html")
@app.route("/over_time")
def over_time_view():
return render_template("over_time.html")
@app.route("/top_commands_full")
def top_commands_full_view():
return render_template("top_commands_full.html")
@app.route("/top_commands")
def top_commands_view():
return render_template("top_commands.html")
@app.route("/trending")
def trending_view():
return render_template("trending.html")
@app.route("/type")
def type_view():
return render_template("type.html")
@app.route("/yearly")
def yearly_view():
return render_template("yearly.html")
# Routes to return JSON contents ----------------------------------------------
@app.route("/codes_json")
def codes_json():
session = db.Session()
results = session.query(db.History.code, func.count(db.History.code)).group_by(db.History.code).all()
# total = sum(r[1] for r in results)
data = [{"name": r[0], "y": r[1]} for r in sorted(results, key=lambda x: x[1], reverse=True)]
return jsonify(data)
@app.route("/daily_json")
def daily_json():
session = db.Session()
results = defaultdict(int)
results.update(
session.query(func.strftime("%w", db.History.start).label("day"), func.count("day")).group_by("day").all()
)
data = [results[str(day)] for day in range(1, 7)]
# put sunday at the end
data.append(results["0"])
return jsonify(data)
@app.route("/daily_average_json")
def daily_average_json():
session = db.Session()
mintime = session.query(func.min(db.History.start)).first()[0]
maxtime = session.query(func.max(db.History.start)).first()[0]
number_of_weeks = (maxtime - mintime).days / 7 + 1
results = defaultdict(int)
results.update(
session.query(func.strftime("%w", db.History.start).label("day"), func.count("day")).group_by("day").all()
)
data = [float("%.2f" % (results[str(day)] / number_of_weeks)) for day in range(1, 7)]
# put sunday at the end
data.append(float("%.2f" % (results["0"] / number_of_weeks)))
return jsonify(data)
@app.route("/duration_json")
def duration_json():
session = db.Session()
results = session.query(db.History.duration).all()
flat_values = [r[0].seconds + round(r[0].microseconds / 1000) for r in results]
counter = Counter(flat_values)
data = {
"average": float("%.2f" % statistics.mean(flat_values)),
"median": statistics.median(flat_values),
"series": [counter[duration] for duration in range(1, max(counter.keys()) + 1)],
}
return jsonify(data)
@app.route("/hourly_json")
def hourly_json():
session = db.Session()
results = defaultdict(lambda: 0)
results.update(
session.query(extract("hour", db.History.start).label("hour"), func.count("hour")).group_by("hour").all()
)
data = [results[hour] for hour in range(0, 24)]
return jsonify(data)
@app.route("/hourly_average_json")
def hourly_average_json():
session = db.Session()
mintime = session.query(func.min(db.History.start)).first()[0]
maxtime = session.query(func.max(db.History.start)).first()[0]
number_of_days = (maxtime - mintime).days + 1
results = defaultdict(lambda: 0)
results.update(
session.query(extract("hour", db.History.start).label("hour"), func.count("hour")).group_by("hour").all()
)
data = [float("%.2f" % (results[hour] / number_of_days)) for hour in range(0, 24)]
return jsonify(data)
@app.route("/length_json")
def length_json():
session = db.Session()
results = defaultdict(lambda: 0)
results.update(
session.query(func.char_length(db.History.cmd).label("length"), func.count("length")).group_by("length").all()
)
if not results:
return jsonify({})
flat_values = []
for length, number in results.items():
flat_values.extend([length] * number)
data = {
"average": float("%.2f" % statistics.mean(flat_values)),
"median": statistics.median(flat_values),
"series": [results[length] for length in range(1, max(results.keys()) + 1)],
}
return jsonify(data)
@app.route("/markov_json")
def markov_json():
session = db.Session()
words_2 = []
w2 = None
words = session.query(db.History.cmd).order_by(db.History.start).all()
for word in words:
w1, w2 = w2, word[0].split(" ")[0]
words_2.append((w1, w2))
counter = Counter(words_2).most_common(40)
unique_words = set()
for (w1, w2), count in counter:
unique_words.add(w1)
unique_words.add(w2)
unique_words = list(unique_words)
data = {
"xCategories": unique_words,
"yCategories": unique_words,
"series": [[unique_words.index(w2), unique_words.index(w1), count] for (w1, w2), count in counter],
}
return jsonify(data)
@app.route("/markov_full_json")
def markov_full_json():
session = db.Session()
words_2 = []
w2 = None
words = session.query(db.History.cmd).order_by(db.History.start).all()
for word in words:
w1, w2 = w2, word[0]
words_2.append((w1, w2))
counter = Counter(words_2).most_common(40)
unique_words = set()
for (w1, w2), count in counter:
unique_words.add(w1)
unique_words.add(w2)
unique_words = list(unique_words)
data = {
"xCategories": unique_words,
"yCategories": unique_words,
"series": [[unique_words.index(w2), unique_words.index(w1), count] for (w1, w2), count in counter],
}
return jsonify(data)
@app.route("/monthly_json")
def monthly_json():
session = db.Session()
results = defaultdict(lambda: 0)
results.update(
session.query(extract("month", db.History.start).label("month"), func.count("month")).group_by("month").all()
)
data = [results[month] for month in range(1, 13)]
return jsonify(data)
@app.route("/monthly_average_json")
def monthly_average_json():
session = db.Session()
mintime = session.query(func.min(db.History.start)).first()[0]
maxtime = session.query(func.max(db.History.start)).first()[0]
number_of_years = fractional_year(mintime, maxtime) + 1
results = defaultdict(lambda: 0)
results.update(
session.query(extract("month", db.History.start).label("month"), func.count("month")).group_by("month").all()
)
data = [float("%.2f" % (results[month] / number_of_years)) for month in range(1, 13)]
return jsonify(data)
@app.route("/over_time_json")
def over_time_json():
session = db.Session()
results = session.query(db.History.start).order_by(db.History.start).all()
def datetime_to_milliseconds(dt):
return int(datetime(dt.year, dt.month, dt.day).timestamp() * 1000)
counter = Counter([datetime_to_milliseconds(r[0]) for r in results])
data = [(k, v) for k, v in counter.items()]
return jsonify(data)
@app.route("/top_commands_full_json")
def top_commands_full_json():
session = db.Session()
results = (
session.query(db.History.cmd, func.count(db.History.cmd).label("count"))
.group_by(db.History.cmd)
.order_by(desc("count"))
.limit(20)
.all()
)
data = {"categories": [r[0] for r in results], "series": [r[1] for r in results]}
return jsonify(data)
@app.route("/top_commands_json")
def top_commands_json():
session = db.Session()
# Tried to do this with SQL only. Failed. POSITION is not a function.
# results = (
# session.query(
# sqlfunc.substr(db.History.cmd, 1, sqlfunc.position(" ", db.History.cmd)),
# func.count(db.History.cmd).label("count"),
# )
# .group_by(db.History.cmd)
# .order_by(desc("count"))
# .limit(20)
# .all()
# )
results = session.query(db.History.cmd).all()
counter = Counter([r[0].split(" ")[0] for r in results]).most_common(20)
data = {"categories": [c[0] for c in counter], "series": [c[1] for c in counter]}
return jsonify(data)
@app.route("/trending_json")
def trending_json():
session = db.Session()
data = None
return jsonify(data)
@app.route("/type_json")
def type_json():
session = db.Session()
results = session.query(db.History.type, func.count(db.History.type)).group_by(db.History.type).all()
# total = sum(r[1] for r in results)
data = [{"name": r[0] or "none", "y": r[1]} for r in sorted(results, key=lambda x: x[1], reverse=True)]
return jsonify(data)
@app.route("/wordcloud_json")
def wordcloud_json():
session = db.Session()
results = session.query(db.History.cmd).order_by(func.random()).limit(100)
text = " ".join(r[0] for r in results.all())
return jsonify(text)
@app.route("/yearly_json")
def yearly_json():
session = db.Session()
minyear = session.query(extract("year", func.min(db.History.start))).first()[0]
maxyear = session.query(extract("year", func.max(db.History.start))).first()[0]
results = defaultdict(lambda: 0)
results.update(
session.query(extract("year", db.History.start).label("year"), func.count("year")).group_by("year").all()
)
data = [(year, results[year]) for year in range(minyear, maxyear + 1)]
return jsonify(data)
PK ! shellhistory/cli.py"""
Module that contains the command line application.
Why does this file exist, and why not put this in __main__?
You might be tempted to import things from __main__ later,
but that will cause problems: the code will get executed twice:
- When you run `python -m shellhistory` python will execute
``__main__.py`` as a script. That means there won't be any
``shellhistory.__main__`` in ``sys.modules``.
- When you import __main__ it will get executed again (as a module) because
there's no ``shellhistory.__main__`` in ``sys.modules``.
Also see (1) from http://click.pocoo.org/5/setuptools/#setuptools-integration
"""
import argparse
from pathlib import Path
from . import db
def get_parser():
parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group()
group.add_argument("--location", dest="location", action="store_true")
group.add_argument("--web", dest="web", action="store_true")
group.add_argument("--import", dest="import_file", action="store_true")
return parser
def main(args=None):
parser = get_parser()
args = parser.parse_args(args=args)
if args.location:
return location()
elif args.web:
return web()
elif args.import_file:
report = db.update()
return 0
def location():
print(Path(__file__).parent / "shellhistory.sh")
return 0
def web():
from .app import app
app.run()
PK ! ۳t shellhistory/db.pyimport os
from base64 import b64decode
from collections import namedtuple
from datetime import datetime
from pathlib import Path
from sqlalchemy import (
Column,
DateTime,
Integer,
Interval,
String,
Text,
UnicodeText,
UniqueConstraint,
create_engine,
exc,
)
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
from tqdm import tqdm
DEFAULT_DIR = Path.home() / ".shellhistory"
DB_PATH = os.getenv("SHELLHISTORY_DB")
HISTFILE_PATH = os.getenv("SHELLHISTORY_FILE")
if (DB_PATH is None or HISTFILE_PATH is None) and not DEFAULT_DIR.exists():
DEFAULT_DIR.mkdir()
if DB_PATH is None:
DB_PATH = DEFAULT_DIR / "db.sqlite3"
else:
DB_PATH = Path(DB_PATH)
if HISTFILE_PATH is None:
HISTFILE_PATH = DEFAULT_DIR / "history"
else:
HISTFILE_PATH = Path(HISTFILE_PATH)
Base = declarative_base()
engine = create_engine("sqlite:///%s?check_same_thread=False" % DB_PATH)
def create_tables():
Base.metadata.create_all(engine)
if not DB_PATH.exists():
create_tables()
Session = sessionmaker(bind=engine)
def get_session():
_engine = create_engine("sqlite:///%s" % DB_PATH)
session = sessionmaker(bind=_engine)
return session()
class History(Base):
__tablename__ = "history"
__table_args__ = (UniqueConstraint("start", "uuid"), {"useexisting": True})
Tuple = namedtuple("HT", "start stop uuid parents host user tty path shell level type code cmd")
id = Column(Integer, primary_key=True)
start = Column(DateTime)
stop = Column(DateTime)
duration = Column(Interval)
host = Column(String)
user = Column(String)
uuid = Column(String)
tty = Column(String)
parents = Column(Text)
shell = Column(String)
level = Column(Integer)
type = Column(String)
code = Column(Integer)
path = Column(String)
cmd = Column(UnicodeText)
def __repr__(self):
return "" % (self.path, self.cmd)
@staticmethod
def line_to_tuple(line):
return History.Tuple(*line.split(":", 12))
@staticmethod
def tuple_to_db_object(nt):
start = datetime.fromtimestamp(float(nt.start) / 1000000.0)
stop = datetime.fromtimestamp(float(nt.stop) / 1000000.0)
duration = stop - start
return History(
start=start,
stop=stop,
duration=duration,
host=nt.host,
user=nt.user,
path=b64decode(nt.path).decode().rstrip("\n"),
uuid=nt.uuid,
tty=nt.tty,
parents=b64decode(nt.parents).decode().rstrip("\n"),
shell=nt.shell,
level=nt.level,
type=nt.type,
code=nt.code,
cmd=nt.cmd,
)
@staticmethod
def from_line(line):
return History.tuple_to_db_object(History.line_to_tuple(line))
def flush():
session = Session()
session.query(History).delete()
session.commit()
def delete_table(table=History):
table.__table__.drop(engine)
def yield_db_object_blocks(path, size=512):
block = []
with codecs.open(path, encoding="utf-8", errors="ignore") as stream:
num_lines = sum(1 for _ in stream)
with codecs.open(path, encoding="utf-8", errors="ignore") as stream:
current_obj = None
for i, line in enumerate(tqdm(stream, total=num_lines, unit="lines"), 1):
first_char, line = line[0], line[1:].rstrip("\n")
if first_char == ":":
# new command
if current_obj is not None:
block.append(current_obj)
current_obj = History.from_line(line)
elif first_char == ";":
# multi-line command
if current_obj is None:
continue # orphan line
current_obj.cmd += "\n" + line
else:
# would only happen if file is corrupted
raise ValueError("invalid line %s starting with %s" % (i, first_char))
if len(block) == size:
yield block
block = []
if current_obj is not None:
block.append(current_obj)
if block:
yield block
InsertionReport = namedtuple("Report", "inserted duplicates")
def insert(obj_list, session, one_by_one=False):
if obj_list:
if one_by_one:
duplicates, inserted = 0, 0
for obj in obj_list:
try:
session.add(obj)
session.commit()
inserted += 1
except exc.IntegrityError:
session.rollback()
duplicates += 1
return InsertionReport(inserted, duplicates)
else:
session.add_all(obj_list)
session.commit()
return InsertionReport(len(obj_list), 0)
return InsertionReport(0, 0)
def import_file(path):
session = Session()
reports = []
for obj_list in yield_db_object_blocks(path):
try:
reports.append(insert(obj_list, session))
except exc.IntegrityError:
session.rollback()
reports.append(insert(obj_list, session, one_by_one=True))
final_report = InsertionReport(sum([r.inserted for r in reports]), sum([r.duplicates for r in reports]))
return final_report
def import_history():
if not HISTFILE_PATH.exists():
raise ValueError("%s: no such file" % HISTFILE_PATH)
return import_file(HISTFILE_PATH)
def update():
return import_history()
PK !
O shellhistory/shellhistory.sh# FUNCTIONS --------------------------------------------------------------------
if [ -n "${ZSH_VERSION}" ]; then
_shellhistory_command_type() {
whence -w "$1" | cut -d' ' -f2
}
_shellhistory_last_command() {
# multi-line commands have prepended ';' (starting at line 2)
# shellcheck disable=SC2154
echo "${history[$HISTCMD]}" | sed -e '2,$s/^/;/'
}
_shellhistory_last_command_number() {
# shellcheck disable=SC2086
echo $HISTCMD
}
elif [ -n "${BASH_VERSION}" ]; then
_shellhistory_command_type() {
type -t "$1"
}
_shellhistory_last_command() {
# multi-line commands have prepended ';' (starting at line 2)
fc -lnr -0 | sed -e '1s/^\t //;2,$s/^/;/'
}
_shellhistory_last_command_number() {
fc -lr -0 | head -n1 | cut -f1
}
fi
# shellcheck disable=SC2120
_shellhistory_parents() {
local list pid
list="$(ps -eo pid,ppid,command | tr -s ' ' | sed 's/^ //g')"
pid=$$
while [ "${pid}" -ne 0 ]; do
echo "${list}" | grep "^${pid} " | cut -d' ' -f3-
pid=$(echo "${list}" | grep "^${pid} " | cut -d' ' -f2)
done
}
_shellhistory_time_now() {
local now
now=$(date '+%s%N')
echo "${now:0:-3}"
}
_shellhistory_start_timer() {
_SHELLHISTORY_START_TIME=${_SHELLHISTORY_START_TIME:-$(_shellhistory_time_now)}
}
_shellhistory_stop_timer() {
_SHELLHISTORY_STOP_TIME=$(_shellhistory_time_now)
}
_shellhistory_set_command() {
_SHELLHISTORY_COMMAND="$(_shellhistory_last_command)"
}
_shellhistory_set_command_type() {
# FIXME: what about "VAR=value command do something"?
# See https://github.com/Pawamoy/shell-history/issues/13
_SHELLHISTORY_TYPE="$(_shellhistory_command_type "${_SHELLHISTORY_COMMAND%% *}")"
}
_shellhistory_set_code() {
_SHELLHISTORY_CODE=$?
}
_shellhistory_set_pwd() {
_SHELLHISTORY_PWD="${PWD}"
_SHELLHISTORY_PWD_B64="$(base64 -w0 <<<"${_SHELLHISTORY_PWD}")"
}
_shellhistory_can_append() {
local last_number
# shellcheck disable=SC2086
[ ${_SHELLHISTORY_BEFORE_DONE} -ne 1 ] && return 1
last_number=$(_shellhistory_last_command_number)
if [ -n "${_SHELLHISTORY_PREVCMD_NUM}" ]; then
# shellcheck disable=SC2086
[ "${last_number}" -eq ${_SHELLHISTORY_PREVCMD_NUM} ] && return 1
_SHELLHISTORY_PREVCMD_NUM=${last_number}
else
_SHELLHISTORY_PREVCMD_NUM=${last_number}
fi
}
_shellhistory_append() {
if _shellhistory_can_append; then
_shellhistory_append_to_file
fi
}
_shellhistory_append_to_file() {
printf ':%s:%s:%s:%s:%s:%s:%s:%s:%s:%s:%s:%s:%s\n' \
"${_SHELLHISTORY_START_TIME}" \
"${_SHELLHISTORY_STOP_TIME}" \
"${_SHELLHISTORY_UUID}" \
"${_SHELLHISTORY_PARENTS_B64}" \
"${_SHELLHISTORY_HOSTNAME}" \
"${USER}" \
"${_SHELLHISTORY_TTY}" \
"${_SHELLHISTORY_PWD_B64}" \
"${SHELL}" \
"${SHLVL}" \
"${_SHELLHISTORY_TYPE}" \
"${_SHELLHISTORY_CODE}" \
"${_SHELLHISTORY_COMMAND}" >> "${SHELLHISTORY_FILE}"
}
_shellhistory_before() {
# shellcheck disable=SC2086
[ ${_SHELLHISTORY_BEFORE_DONE} -gt 0 ] && return
_shellhistory_set_command
_shellhistory_set_command_type
_shellhistory_set_pwd
_shellhistory_start_timer
_SHELLHISTORY_AFTER_DONE=0
_SHELLHISTORY_BEFORE_DONE=1
}
_shellhistory_after() {
_shellhistory_set_code # must always be done first
_shellhistory_stop_timer
[ ${_SHELLHISTORY_BEFORE_DONE} -eq 2 ] && _SHELLHISTORY_BEFORE_DONE=0
[ ${_SHELLHISTORY_AFTER_DONE} -eq 1 ] && return
_shellhistory_append
unset _SHELLHISTORY_START_TIME
_SHELLHISTORY_BEFORE_DONE=0
_SHELLHISTORY_AFTER_DONE=1
return ${_SHELLHISTORY_CODE}
}
_shellhistory_get_debug_trap() {
local trap
trap="$(trap -p | grep ' DEBUG$')" || return 0
trap=${trap:9}
trap=${trap:0:-7}
case ${trap} in
*';') ;;
*) trap+=";" ;;
esac
echo "${trap}"
}
_shellhistory_enable() {
_SHELLHISTORY_BEFORE_DONE=2
_SHELLHISTORY_AFTER_DONE=1
if [ "${ZSH_VERSION}" ]; then
preexec_functions+=(_shellhistory_before)
precmd_functions=(_shellhistory_after "${precmd_functions[@]}")
elif [ "${BASH_VERSION}" ]; then
PROMPT_COMMAND="_shellhistory_after;${PROMPT_COMMAND}"
# shellcheck disable=SC2064
trap "$(_shellhistory_get_debug_trap)_shellhistory_before;" DEBUG
fi
}
_shellhistory_disable() {
local trap
_SHELLHISTORY_AFTER_DONE=1
if [ "${ZSH_VERSION}" ]; then
# shellcheck disable=SC2206
preexec_functions=(${preexec_functions:#_shellhistory_before})
# shellcheck disable=SC2206
precmd_functions=(${precmd_functions:#_shellhistory_after})
elif [ "${BASH_VERSION}" ]; then
trap="$(_shellhistory_get_debug_trap)"
trap=${trap//_shellhistory_before;}
if [ -n "${trap}" ]; then
# shellcheck disable=SC2064
trap "${trap}" DEBUG
else
trap - DEBUG
fi
PROMPT_COMMAND="${PROMPT_COMMAND//_shellhistory_after;}"
fi
}
_shellhistory_usage() {
echo "usage: shellhistory "
}
_shellhistory_help() {
_shellhistory_usage
echo
echo "Commands:"
echo " disable disable shellhistory"
echo " enable enable shellhistory"
echo " help print this help and exit"
}
# GLOBAL VARIABLES -------------------------------------------------------------
_SHELLHISTORY_CODE=
_SHELLHISTORY_COMMAND=
_SHELLHISTORY_HOSTNAME="$(hostname)"
_SHELLHISTORY_PARENTS="$(_shellhistory_parents)"
_SHELLHISTORY_PARENTS_B64="$(echo "${_SHELLHISTORY_PARENTS}" | base64 -w0)"
_SHELLHISTORY_PWD=
_SHELLHISTORY_PWD_B64=
_SHELLHISTORY_START_TIME=
_SHELLHISTORY_STOP_TIME=
_SHELLHISTORY_TTY="$(tty)"
_SHELLHISTORY_TYPE=
_SHELLHISTORY_UUID="${_SHELLHISTORY_UUID:-$(uuidgen)}"
_SHELLHISTORY_AFTER_DONE=
_SHELLHISTORY_BEFORE_DONE=
_SHELLHISTORY_PREVCMD_NUM=
SHELLHISTORY_FILE="${SHELLHISTORY_FILE:-$HOME/.shellhistory/history}"
export SHELLHISTORY_FILE
export _SHELLHISTORY_UUID
# MAIN COMMAND -----------------------------------------------------------------
shellhistory() {
case "$1" in
disable) _shellhistory_disable ;;
enable) _shellhistory_enable ;;
help) _shellhistory_help ;;
*) _shellhistory_usage >&2; return 1 ;;
esac
}
PK ! *p shellhistory/static/css/home.cssfooter {
margin-bottom: 20px;
}
#sync {
min-height: 50px;
}
#sync-close {
font-size: 21px;
font-weight: 700;
line-height: 1;
color: #000;
text-shadow: 0 1px 0 #fff;
opacity: .2;
cursor: pointer;
}
#sync-close:hover {
text-decoration: none;
}
#welcome-title {
/*position: relative;*/
/*left: 40px;*/
text-align: center;
}
#sync-alert {
padding: 15px;
margin-bottom: 20px;
border: 1px solid transparent;
border-radius: 4px;
}
#sync-alert.danger {
color: #a94442;
background-color: #f2dede;
border-color: #ebccd1;
}
#sync-alert.info {
color: #31708f;
background-color: #d9edf7;
border-color: #bce8f1;
}
#sync-alert.success {
color: #3c763d;
background-color: #dff0d8;
border-color: #d6e9c6;
}
#sync-alert.warning {
color: #8a6d3b;
background-color: #fcf8e3;
border-color: #faebcc;
}
#sync-alert.default {
color: grey;
background-color: #f5f5f5;
border-color: #e3e3e3;
}PK ! shellhistory/static/js/codes.js$(document).ready(function () {
$.getJSON('/codes_json', function (data) {
Highcharts.chart('container', {
chart: {
plotBackgroundColor: null,
plotBorderWidth: null,
plotShadow: false,
type: 'pie'
},
title: {
text: 'Commands by exit code'
},
tooltip: {
pointFormat: '{series.name}: {point.y}'
},
legend: {},
plotOptions: {
pie: {
allowPointSelect: true,
cursor: 'pointer',
dataLabels: {
enabled: true,
format: '{point.name}: {point.percentage:.1f} %',
style: {
color: (Highcharts.theme && Highcharts.theme.contrastTextColor) || 'black'
}
},
showInLegend: true
}
},
series: [{
name: 'Commands',
colorByPoint: true,
data: data
}]
});
});
});
PK ! һ.b b shellhistory/static/js/daily.js$(document).ready(function () {
$.getJSON('/daily_json', function (data) {
Highcharts.chart('container', {
chart: {
type: 'column'
},
title: {
text: 'Daily Commands'
},
xAxis: {
title: {
text: 'Day of the week'
},
categories: [
'Mon',
'Tue',
'Wed',
'Thu',
'Fri',
'Sat',
'Sun'
],
crosshair: true
},
yAxis: {
min: 0,
title: {
text: 'Number of commands'
}
},
tooltip: {
shared: true,
useHTML: true
},
plotOptions: {
column: {
pointPadding: 0.2,
borderWidth: 0
}
},
series: [{
name: 'Number of commands',
data: data
}]
});
});
});
PK ! ]j j ' shellhistory/static/js/daily_average.js$(document).ready(function () {
$.getJSON('/daily_average_json', function (data) {
Highcharts.chart('container', {
chart: {
type: 'column'
},
title: {
text: 'Daily Commands'
},
xAxis: {
title: {
text: 'Day of the week'
},
categories: [
'Mon',
'Tue',
'Wed',
'Thu',
'Fri',
'Sat',
'Sun'
],
crosshair: true
},
yAxis: {
min: 0,
title: {
text: 'Number of commands'
}
},
tooltip: {
shared: true,
useHTML: true
},
plotOptions: {
column: {
pointPadding: 0.2,
borderWidth: 0
}
},
series: [{
name: 'Number of commands',
data: data
}]
});
});
});
PK ! I " shellhistory/static/js/duration.js$(document).ready(function () {
$.getJSON('/duration_json', function (data) {
Highcharts.chart('container', {
title: {
text: 'Commands by duration'
},
xAxis: {
title: {
text: 'Command duration in milliseconds'
},
type: 'logarithmic',
minorTickInterval: 1,
plotLines: [{
value: data.average,
color: 'orange',
width: 2,
label: {
text: 'Average: ' + data.average + ' milliseconds',
align: 'left',
style: {
color: 'gray'
}
}
}, {
value: data.median,
color: 'magenta',
width: 2,
label: {
text: 'Median: ' + data.median + ' milliseconds',
align: 'left',
style: {
color: 'gray'
}
}
}]
},
yAxis: {
title: {
text: 'Number of commands'
}
},
plotOptions: {
series: {
pointStart: 1
}
},
series: [{
name: 'Commands',
data: data.series
}],
});
});
});
PK ! X(ua a shellhistory/static/js/home.js$(document).ready(function () {
$('#sync-close').click(function (e) {
$(this).parent().fadeOut(500);
});
$("#sync-button").click(function () {
$.getJSON('/update', function (data) {
$('#sync-message').text(data.message);
$('#sync-alert')
.removeClass()
.addClass(data.class)
.show();
});
});
});
PK ! P' + this.series.xAxis.categories[this.point.y] + ' to ' +
this.series.yAxis.categories[this.point.x] + ': ' + this.point.value;
}
},
series: [{
name: 'Occurrences',
borderWidth: 1,
data: data.series,
dataLabels: {
enabled: true,
color: '#000000'
}
}]
});
});
});
PK ! X % shellhistory/static/js/markov_full.js$(document).ready(function () {
$.getJSON('/markov_full_json', function (data) {
Highcharts.chart('container', {
chart: {
type: 'heatmap',
marginTop: 40,
marginBottom: 80,
plotBorderWidth: 1
},
title: {
text: 'Markov chain (full commands)'
},
xAxis: {
categories: data.xCategories
},
yAxis: {
categories: data.yCategories,
title: null
},
colorAxis: {
min: 0,
minColor: '#FFFFFF',
maxColor: Highcharts.getOptions().colors[0]
},
legend: {
align: 'right',
layout: 'vertical',
margin: 0,
verticalAlign: 'top',
y: 25,
symbolHeight: 530
},
tooltip: {
formatter: function () {
return '' + this.series.xAxis.categories[this.point.y] + ' to ' +
this.series.yAxis.categories[this.point.x] + ': ' + this.point.value;
}
},
series: [{
name: 'Occurrences',
borderWidth: 1,
data: data.series,
dataLabels: {
enabled: true,
color: '#000000'
}
}]
});
});
});
PK ! 2 ! shellhistory/static/js/monthly.js$(document).ready(function () {
$.getJSON('/monthly_json', function (data) {
Highcharts.chart('container', {
chart: {
type: 'column'
},
title: {
text: 'Monthly Commands'
},
xAxis: {
title: {
text: 'Month'
},
categories: [
'Jan',
'Feb',
'Mar',
'Apr',
'May',
'Jun',
'Jul',
'Aug',
'Sep',
'Oct',
'Nov',
'Dec'
],
crosshair: true
},
yAxis: {
min: 0,
title: {
text: 'Number of commands'
}
},
tooltip: {
shared: true,
useHTML: true
},
plotOptions: {
column: {
pointPadding: 0.2,
borderWidth: 0
}
},
series: [{
name: 'Number of commands',
data: data
}]
});
});
});
PK ! wq ) shellhistory/static/js/monthly_average.js$(document).ready(function () {
$.getJSON('/monthly_average_json', function (data) {
Highcharts.chart('container', {
chart: {
type: 'column'
},
title: {
text: 'Average Monthly Commands'
},
xAxis: {
title: {
text: 'Month'
},
categories: [
'Jan',
'Feb',
'Mar',
'Apr',
'May',
'Jun',
'Jul',
'Aug',
'Sep',
'Oct',
'Nov',
'Dec'
],
crosshair: true
},
yAxis: {
min: 0,
title: {
text: 'Number of commands'
}
},
tooltip: {
shared: true,
useHTML: true
},
plotOptions: {
column: {
pointPadding: 0.2,
borderWidth: 0
}
},
series: [{
name: 'Number of commands',
data: data
}]
});
});
});
PK ! kq # shellhistory/static/js/over_time.js$(document).ready(function() {
$.getJSON('/over_time_json', function(data) {
Highcharts.chart('container', {
chart: {
zoomType: 'x'
},
title: {
text: 'Commands over time'
},
subtitle: {
text: document.ontouchstart === undefined ?
'Click and drag in the plot area to zoom in' : 'Pinch the chart to zoom in'
},
xAxis: {
type: 'datetime'
},
yAxis: {
min: 0,
title: {
text: 'Number of commands'
}
},
legend: {
enabled: false
},
plotOptions: {
area: {
fillColor: {
linearGradient: {
x1: 0,
y1: 0,
x2: 0,
y2: 1
},
stops: [
[0, Highcharts.getOptions().colors[0]],
[1, Highcharts.Color(Highcharts.getOptions().colors[0]).setOpacity(0).get('rgba')]
]
},
marker: {
radius: 2
},
lineWidth: 1,
states: {
hover: {
lineWidth: 1
}
},
threshold: null
}
},
series: [{
type: 'area',
name: 'Commands',
data: data
}]
});
});
});
PK ! WAB & shellhistory/static/js/top_commands.js$(document).ready(function () {
$.getJSON('/top_commands_json', function (data) {
Highcharts.chart('container', {
chart: {
type: 'bar'
},
title: {
text: 'Top commands'
},
xAxis: {
categories: data.categories,
min: 0,
title: {
text: 'Commands'
}
},
yAxis: {
title: {
text: 'Number of commands'
}
},
tooltip: {
shared: true,
useHTML: true
},
plotOptions: {
column: {
pointPadding: 0.2,
borderWidth: 0
}
},
series: [{
name: 'Number',
data: data.series
}]
});
});
});
PK ! & + shellhistory/static/js/top_commands_full.js$(document).ready(function () {
$.getJSON('/top_commands_full_json', function (data) {
Highcharts.chart('container', {
chart: {
type: 'bar'
},
title: {
text: 'Top commands (full lines)'
},
xAxis: {
categories: data.categories,
min: 0,
title: {
text: 'Commands'
}
},
yAxis: {
title: {
text: 'Number of commands'
}
},
tooltip: {
shared: true,
useHTML: true
},
plotOptions: {
column: {
pointPadding: 0.2,
borderWidth: 0
}
},
series: [{
name: 'Number',
data: data.series
}]
});
});
});
PK ! " shellhistory/static/js/trending.jsPK ! qB shellhistory/static/js/type.js$(document).ready(function () {
$.getJSON('/type_json', function (data) {
Highcharts.chart('container', {
chart: {
plotBackgroundColor: null,
plotBorderWidth: null,
plotShadow: false,
type: 'pie'
},
title: {
text: 'Commands by type'
},
tooltip: {
pointFormat: '{series.name}: {point.y}'
},
legend: {},
plotOptions: {
pie: {
allowPointSelect: true,
cursor: 'pointer',
dataLabels: {
enabled: true,
format: '{point.name}: {point.percentage:.1f} %',
style: {
color: (Highcharts.theme && Highcharts.theme.contrastTextColor) || 'black'
}
},
showInLegend: true
}
},
series: [{
name: 'Commands',
colorByPoint: true,
data: data
}]
});
});
});
PK ! Q> > # shellhistory/static/js/wordcloud.js$(document).ready(function () {
$.getJSON('/wordcloud_json', function (text) {
var lines = text.split(/[,\. ]+/g),
data = Highcharts.reduce(lines, function (arr, word) {
var obj = Highcharts.find(arr, function (obj) {
return obj.name === word;
});
if (obj) {
obj.weight += 1;
} else {
obj = {
name: word,
weight: 1
};
arr.push(obj);
}
return arr;
}, []);
Highcharts.chart('container', {
title: {
text: null
},
series: [{
type: 'wordcloud',
data: data,
name: 'Occurrences',
enableMouseTracking: false
}],
exporting: {
enabled: false
},
credits: {
enabled: false
}
});
});
});
PK ! shellhistory/static/js/yearly.js$(document).ready(function () {
$.getJSON('/yearly_json', function (data) {
Highcharts.chart('container', {
chart: {
type: 'column'
},
title: {
text: 'Yearly Commands'
},
xAxis: {
title: {
text: 'Year'
},
crosshair: true
},
yAxis: {
min: 0,
title: {
text: 'Number of commands'
}
},
tooltip: {
shared: true,
useHTML: true
},
plotOptions: {
column: {
pointPadding: 0.2,
borderWidth: 0
}
},
series: [{
name: 'Number of commands',
data: data
}]
});
});
});
PK ! Wp . shellhistory/templates/admin/history_list.html{% extends 'admin/model/list.html' %}
{% block main_menu %}
