ml::LogisticRegression Class Reference

Binomial logistic regression algorithm. More...

#include <LogisticRegression.hpp>

Inheritance diagram for ml::LogisticRegression:

Classes
struct	Result
	Result of binomial logistic regression. More...

Public Member Functions
virtual	~LogisticRegression ()
	Virtual destructor.
virtual Result	fit (Eigen::Ref< const Eigen::MatrixXd > X, Eigen::Ref< const Eigen::VectorXd > y) const =0
	Fits the model and returns the result. More...

Static Public Member Functions
static double	probability (Eigen::Ref< const Eigen::VectorXd > x, double y, Eigen::Ref< const Eigen::VectorXd > w)
	Calculates the probability of label given model weights and feature vector. More...
static double	log_likelihood (Eigen::Ref< const Eigen::MatrixXd > X, Eigen::Ref< const Eigen::VectorXd > y, Eigen::Ref< const Eigen::VectorXd > w, double lam)
	Calculates the posterior log-likelihood of data given model weights. More...
static void	grad_log_likelihood (Eigen::Ref< const Eigen::MatrixXd > X, Eigen::Ref< const Eigen::VectorXd > y, Eigen::Ref< const Eigen::VectorXd > w, double lam, Eigen::Ref< Eigen::VectorXd > g)
	Calculates the gradient of the posterior log-likelihood of data given model weights, over those weights. More...
static void	hessian_log_likelihood (Eigen::Ref< const Eigen::MatrixXd > X, Eigen::Ref< const Eigen::VectorXd > y, Eigen::Ref< const Eigen::VectorXd > w, double lam, Eigen::Ref< Eigen::MatrixXd > H)
	Calculates the Hessian of the posterior log-likelihood of data given model weights, over those weights. More...

Detailed Description

Binomial logistic regression algorithm.

Based on Thomas P. Minka, "A comparison of numerical optimizers for logistic regression".

The model is:

\( P(y = +/-1 | \vec{x}, \vec{w}) = \frac{1}{1 + \exp(-y \vec{w} \cdot \vec{x} )} \)

We perform a MAP estimation of \( \vec{w} \), using a prior \( p(\vec{w}) \sim N(0, \lambda^{-1} I) \).

Given a dataset \( [(\vec{x}_i, y_i)]_{i=1}^n \), the optimisation problem is maximising the function \( l(\vec{w}) = - \sum_{i=1}^n \ln(1 + \exp(- y_i \vec{w} \cdot \vec{x}_i)) - \frac{\lambda}{2} \vec{w} \cdot \vec{w} \).

Member Function Documentation

fit()

virtual Result ml::LogisticRegression::fit ( Eigen::Ref< const Eigen::MatrixXd >  X, Eigen::Ref< const Eigen::VectorXd >  y  ) const

pure virtual

Fits the model and returns the result.

If fitting with intercept is desired, include a row of 1's in the X values.

Parameters

X	D x N matrix of X values, with data points in columns.
y	Y vector with length N. Values should be -1 or 1.

Exceptions

std::invalid_argument

if N or D are zero, or if dimensions of X and y do not match.

Returns

Result object.

Implemented in ml::ConjugateGradientLogisticRegression.

probability()

static double ml::LogisticRegression::probability ( Eigen::Ref< const Eigen::VectorXd >  x, double  y, Eigen::Ref< const Eigen::VectorXd >  w  )

static

Calculates the probability of label given model weights and feature vector.

Parameters

x	Feature vector.
y	Label (-1 or 1).
w	Model weight vector, equal length to x.

Returns

P(y|x,w).

log_likelihood()

static double ml::LogisticRegression::log_likelihood ( Eigen::Ref< const Eigen::MatrixXd >  X, Eigen::Ref< const Eigen::VectorXd >  y, Eigen::Ref< const Eigen::VectorXd >  w, double  lam  )

static

Calculates the posterior log-likelihood of data given model weights.

Parameters

X	D x N matrix of X values, with data points in columns.
y	Y vector with length N. Values should be -1 or 1.
w	Model weight vector with length D.
lam	Inverse variance of the Gaussian prior for w. Cannot be negative. Set it to 0 if you want to perform maximum likelihood estimation of w.

Returns

Log-likelihood.

Exceptions

std::domain_error	If lam is negative.
std::invalid_argument	If matrix or vector dimensions do not match.

grad_log_likelihood()

static void ml::LogisticRegression::grad_log_likelihood ( Eigen::Ref< const Eigen::MatrixXd >  X, Eigen::Ref< const Eigen::VectorXd >  y, Eigen::Ref< const Eigen::VectorXd >  w, double  lam, Eigen::Ref< Eigen::VectorXd >  g  )

static

Calculates the gradient of the posterior log-likelihood of data given model weights, over those weights.

Parameters

	X	D x N matrix of X values, with data points in columns.
	y	Y vector with length N. Values should be -1 or 1.
	w	Model weight vector with length D.
	lam	Inverse variance of the Gaussian prior for w. Cannot be negative. Set it to 0 if you want to perform maximum likelihood estimation of w.
[out]	g	Vector with length D for the computed gradient of log-likelihood over weights w.

Exceptions

std::domain_error	If lam is negative.
std::invalid_argument	If matrix or vector dimensions do not match.

hessian_log_likelihood()

static void ml::LogisticRegression::hessian_log_likelihood ( Eigen::Ref< const Eigen::MatrixXd >  X, Eigen::Ref< const Eigen::VectorXd >  y, Eigen::Ref< const Eigen::VectorXd >  w, double  lam, Eigen::Ref< Eigen::MatrixXd >  H  )

static

Calculates the Hessian of the posterior log-likelihood of data given model weights, over those weights.

Parameters

	X	D x N matrix of X values, with data points in columns.
	y	Y vector with length N. Values should be -1 or 1.
	w	Model weight vector with length D.
	lam	Inverse variance of the Gaussian prior for w. Cannot be negative. Set it to 0 if you want to perform maximum likelihood estimation of w.
[out]	H	D x D matrix for the computed Hessian of log-likelihood over weights w.

Exceptions

std::domain_error	If lam is negative.
std::invalid_argument	If matrix or vector dimensions do not match.

---

The documentation for this class was generated from the following file:

• ML/LogisticRegression.hpp