Design

Furem Cape is comprised of 2 PostgreSQL databases:

And 7 Python packages:

As well as several other miscellaneous components:

HitDB

The hitdb PostgreSQL database stores log data that has been processed by the Transformer component, principally in the hit table. Each row in the hit table represents an individual log entry.

The key elements of each hit are:

at

log entry timestamp

system

system name

session

user session ID or token

ip

user IP address

user

user name or ID (the actor)

resource

resource name, path, ID, etc (the thing being acted on)

action

action name, path, ID, etc

error

error/status code, ID, description, etc (the result of the action)

Old hits (those older than ~90 days) aren’t useful to Furem Cape, and should be pruned from the hitdb periodically.

IssueDB

The issuedb PostgreSQL database stores log data that has been processed by the Analyzer component, principally in the issue table. Each row in the issue table represents an individual issue found by Furem Cape, related to a specific hit from the HitDB. An individual hit may generate multiple issues. Hits that have generated an issue are copied into the issuedb for reporting.

Each issue is flagged with a danger integer value. Danger values range from 1-9 (if the software internally flags an issue with a danger of less than 1, the issue is discarded; and a danger of greater than 9 is reduced to 9). Generally, issues with a danger of 1 can be ignored (interesting only for aggregate reporting); issues with danger of 5 or greater should be reviewed by a person; and issues with danger of 9 should be acted on immediately.

Feeder

The furemcape.feeder Python package runs as the furemcape-feeder service, and feeds log entries to the Transformer. It is implemented with the Twisted framework.

The feeder comes with a number of feeder modules (“processors”) built-in, and can be extended with any number of additional custom Python processors. The following processors are built-in:

socket

Reads from a Unix socket, and feeds each line as a log entry.

stdio

Reads from standard input, and feeds each line as a log entry.

Transformer

The furemcape.transformer Python package runs in the furemcape-feeder service, and transforms each log entry fed to it by the Feeder into an entry in the HitDB (or discards the log entry).

The transformer comes with a number of modules (“processors”) for parsing log data built-in, and can be extended with any number of additional custom Python processors. Processors are composed together as steps in a pipeline, and pipelines are applied to a given log entry by matching the system from which the entry originated to the pipeline.

Analyzer

The furemcape.analyzer Python package is the heart of Furem Cape. It runs as the furemcape-analyzer service, and analyzes each hit in the HitDB. It adds the issues it finds to the IssueDB.

The analyzer comes with a number of analyzer modules (“processors”) built-in, and can be extended with any number of additional custom Python processors.

Responder

The furemcape.responder Python package runs as the furemcape-responder service, and responds to issues as they are added to the IssueDB.

The responder comes with a number of responder modules (“processors”) built-in, and can be extended with any number of additional custom Python processors. The following processors are built-in:

email

Sends an email.

Reporter

The furemcape.reporter Python package provides command-line tools for reporting on issues that Furem Cape has found. It includes the following scripts in /usr/local/bin/:

furemcape-reporter

Lists recent and historical issues, and counts groups of issues.

Administrator

The furemcape.administrator Python package provides command-line tools for administering the Furem Cape system. It includes the following scripts in /usr/local/bin/:

furemcape-administrator-install

Runs database migration scripts and installs Furem Cape services.

Shared

The furemcape.shared Python package contains miscellaneous library code that is shared among Furem Cape components.

Docs

The docs component contains the documentation for Furem Cape in the form of reStructuredText markup. The source markup is built into HTML via Sphinx.

E2E

The e2e component contains “end-to-end” (aka “functional”) tests for Furem Cape.

Installer

The installer component contains scripts and tests for installing Furem Cape.