furemcape.analyzer package

Analyzes log-entry data from hitdb and adds issues to issuedb.

class furemcape.analyzer.Analyzer(*args, **kwargs)

Bases: furemcape.shared.obj.BaseAttrObject

Analyzes each hitdb entry and adds resulting issues to issuedb.

cfg

Raw configuration data.

Type

dict

issue_defaults

Fallback issue configuration.

Type

dict

default_system

Fallback SystemDefinition object.

Type

SystemDefinition

system_prototypes

Fallback system prototype configuration.

Type

dict

systems

Loaded list of SystemDefinition objects.

Type

list

system_processors

Loaded SystemProcessor objects, keyed by system name.

Type

dict

analyze_as_of

Datetime to consider as ‘now’ (defaults to now actually).

Type

DateTime

analyze_within

Duration within which to analyze hits (defaults to 1 day).

Type

Duration

ready_partially_after

Duration after which to start recording issues with scaled danger levels (defaults to 10 days).

Type

Duration

ready_fully_after

Duration after which to start recording issues with full danger levels (defaults to 30 days).

Type

Duration

issue_to_log

If a Logger instance, sends issue to it at info level instead of to issuedb.

Type

Logger

sleep_for

Seconds to sleep (defaults to 10).

Type

int

running

True while running analyze loop.

Type

bool

ISSUE_DEFAULTS = {'not_seen_before': {'base': 'not_seen_before', 'processor': 'furemcape.analyzer.processor.not_seen_before.NotSeenBefore'}}
LOG = <Logger furemcape.analyzer.analyzer (WARNING)>
SYSTEM_PROTOTYPES = {}
analyze_loop()

Main analysis loop.

analyze_next(hit)

Analyzes the specified hit.

Parameters

hit (Hit) – Hit record from hitdb.

Returns

List of issue dicts to insert into issuedb.

Return type

list

calculate_analyze_after()

Calculates the datetime after which to analyze hits.

Returns

Datetime after which to analyze hits.

Return type

DateTime

load_config(path)

Replaces configuration with specified config file.

Parameters

path (str) – Path to config file.

record(issues, hit)

Inserts the specified list of issues into the issuedb.

Parameters
  • issues (list) – List of issue dicts.

  • hit (Hit) – Hit record.

record_to_db(issues, hit)

Inserts issues into the issuedb instead of logging them.

Parameters
  • issues (list) – List of issue dicts.

  • hit (Hit) – Hit record.

record_to_log(issues, hit)

Logs issues instead of inserting them into the issuedb.

Parameters
  • issues (list) – List of issue dicts.

  • hit (Hit) – Hit record.

sleep()

Sleeps for configured number of seconds.

validate_db()

Raises exception if db not available.

Raises
  • AttributeError – Database not initialized. # noqa: DAR402

  • DoesNotExist – Required migrations not run. # noqa: DAR402

Submodules

furemcape.analyzer.analyzer module

Analyzes each hitdb entry and adds resulting issues to issuedb.

class furemcape.analyzer.analyzer.Analyzer(*args, **kwargs)

Bases: furemcape.shared.obj.BaseAttrObject

Analyzes each hitdb entry and adds resulting issues to issuedb.

cfg

Raw configuration data.

Type

dict

issue_defaults

Fallback issue configuration.

Type

dict

default_system

Fallback SystemDefinition object.

Type

SystemDefinition

system_prototypes

Fallback system prototype configuration.

Type

dict

systems

Loaded list of SystemDefinition objects.

Type

list

system_processors

Loaded SystemProcessor objects, keyed by system name.

Type

dict

analyze_as_of

Datetime to consider as ‘now’ (defaults to now actually).

Type

DateTime

analyze_within

Duration within which to analyze hits (defaults to 1 day).

Type

Duration

ready_partially_after

Duration after which to start recording issues with scaled danger levels (defaults to 10 days).

Type

Duration

ready_fully_after

Duration after which to start recording issues with full danger levels (defaults to 30 days).

Type

Duration

issue_to_log

If a Logger instance, sends issue to it at info level instead of to issuedb.

Type

Logger

sleep_for

Seconds to sleep (defaults to 10).

Type

int

running

True while running analyze loop.

Type

bool

ISSUE_DEFAULTS = {'not_seen_before': {'base': 'not_seen_before', 'processor': 'furemcape.analyzer.processor.not_seen_before.NotSeenBefore'}}
LOG = <Logger furemcape.analyzer.analyzer (WARNING)>
SYSTEM_PROTOTYPES = {}
analyze_loop()

Main analysis loop.

analyze_next(hit)

Analyzes the specified hit.

Parameters

hit (Hit) – Hit record from hitdb.

Returns

List of issue dicts to insert into issuedb.

Return type

list

calculate_analyze_after()

Calculates the datetime after which to analyze hits.

Returns

Datetime after which to analyze hits.

Return type

DateTime

load_config(path)

Replaces configuration with specified config file.

Parameters

path (str) – Path to config file.

record(issues, hit)

Inserts the specified list of issues into the issuedb.

Parameters
  • issues (list) – List of issue dicts.

  • hit (Hit) – Hit record.

record_to_db(issues, hit)

Inserts issues into the issuedb instead of logging them.

Parameters
  • issues (list) – List of issue dicts.

  • hit (Hit) – Hit record.

record_to_log(issues, hit)

Logs issues instead of inserting them into the issuedb.

Parameters
  • issues (list) – List of issue dicts.

  • hit (Hit) – Hit record.

sleep()

Sleeps for configured number of seconds.

validate_db()

Raises exception if db not available.

Raises
  • AttributeError – Database not initialized. # noqa: DAR402

  • DoesNotExist – Required migrations not run. # noqa: DAR402

furemcape.analyzer.config module

Utilities for loading analyzer configuration.

furemcape.analyzer.config.load_config(analyzer, path)

Replaces analyzer configuration with specified config file.

Parameters
  • analyzer (Analyzer) – Analyzer object.

  • path (str) – Path to Furem Cape config file.

furemcape.analyzer.config.load_default_config(analyzer)

Replaces analyzer configuration with default config file.

Parameters

analyzer (Analyzer) – Analyzer object.

furemcape.analyzer.config.set_config(analyzer, cfg)

Replaces analyzer configuration with specified config dict.

Parameters
  • analyzer (Analyzer) – Analyzer object.

  • cfg (dict) – Furem Cape config dict.

furemcape.analyzer.issue_definition module

Definition of an issue analyzer.

class furemcape.analyzer.issue_definition.IssueDefinition(*args, **kwargs)

Bases: furemcape.shared.obj.BaseAttrObject

Definition of an issue analyzer.

elements

Element list (eg [‘ip’, ‘user’]).

Type

list

base

Base ID.

Type

str

variant

Variant ID.

Type

str

system

System ID.

Type

str

name

Variant display name.

Type

str

description

Variant description.

Type

str

danger

Danger level.

Type

int

processor

Issue processor class name.

Type

str

processor_instance

Issue processor instance.

Type

BaseProcessor

full_id

Read-only fully-qualified ID (eg ‘myapp.ip_user.not_seen_before.non_anon’).

Type

str

LOG = <Logger furemcape.analyzer.issue_definition (WARNING)>
annotate_issue(issue, hit)

Annotates issue dict with hit and definition data.

Parameters
  • issue (dict) – Issue dict.

  • hit (Hit) – Hit record.

Returns

Updated issue dict.

Return type

dict

property full_id
process(hit)

Processes the specified hit.

Parameters

hit (Hit) – Hit record.

Returns

List of issue dicts.

Return type

list

to_dict()

Returns definition as dict.

Returns

Issue definition as dict.

Return type

dict

furemcape.analyzer.system_definition module

Definition of issue analyzers for a particular system.

class furemcape.analyzer.system_definition.SystemDefinition(*args, **kwargs)

Bases: furemcape.shared.obj.BaseAttrObject

Definition of issue analyzers for a particular system.

id

System definition ID (not necessary the system ID).

Type

str

match

Regex string against which to match keys to determine when to use this definition.

Type

str

match_re

Regex against which to match keys to determine when to use this definition.

Type

re

issues

Ordered list of issue definitions.

Type

list

LOG = <Logger furemcape.analyzer.system_definition (WARNING)>
matches(hit)

True if this definition matches the specified hit.

Parameters

hit (Hit) – Hit record.

Returns

True if this definition matches the specified hit.

Return type

bool

furemcape.analyzer.system_processor module

Processor/cache of issue analyzers for a particular system.

class furemcape.analyzer.system_processor.SystemProcessor(*args, **kwargs)

Bases: furemcape.shared.obj.BaseAttrObject

Processor/cache of issue analyzers for a particular system.

id

System definition ID (not necessary the system ID).

Type

str

issues

Ordered list of issue definitions.

Type

list

scale_factor

Factor to scale issue danger by (defaults to 1).

Type

int

first_hit_at

Datetime of first hit for this system.

Type

DateTime

analyze_as_of

Datetime to consider as ‘now’ (defaults to now actually).

Type

DateTime

ready_partially_after

Duration after which to start recording issues with scaled danger levels (defaults to 10 days).

Type

Duration

ready_fully_after

Duration after which to start recording issues with full danger levels (defaults to 30 days).

Type

Duration

LOG = <Logger furemcape.analyzer.system_processor (WARNING)>
analyze(hit)

Analyzes the specified hit.

Parameters

hit (Hit) – Hit record from hitdb.

Returns

List of issue dicts to insert into issuedb.

Return type

list

is_ready()

True if this system is at least partially ready to start recording issues.

Returns

True if ready.

scale_issues(issues)

Scales the danger level in the specified list of issues.

Parameters

issues (list) – List of issue dicts to scale.

Returns

Scaled issue dicts.

Return type

list