Class: Rumale::LinearModel::Ridge

Inherits:
BaseSGD
  • Object
show all
Includes:
Base::Regressor
Defined in:
lib/rumale/linear_model/ridge.rb

Overview

Ridge is a class that implements Ridge Regression with stochastic gradient descent (SGD) optimization, singular value decomposition (SVD), or L-BFGS optimization.

Reference

  • Bottou, L., “Large-Scale Machine Learning with Stochastic Gradient Descent,” Proc. COMPSTAT'10, pp. 177–186, 2010.

Examples:

estimator =
  Rumale::LinearModel::Ridge.new(reg_param: 0.1, max_iter: 1000, batch_size: 20, random_seed: 1)
estimator.fit(training_samples, traininig_values)
results = estimator.predict(testing_samples)

# If Numo::Linalg is installed, you can specify 'svd' for the solver option.
require 'numo/linalg/autoloader'
estimator = Rumale::LinearModel::Ridge.new(reg_param: 0.1, solver: 'svd')
estimator.fit(training_samples, traininig_values)
results = estimator.predict(testing_samples)

Instance Attribute Summary collapse

Attributes included from Base::BaseEstimator

#params

Instance Method Summary collapse

Methods included from Base::Regressor

#score

Constructor Details

#initialize(learning_rate: 0.01, decay: nil, momentum: 0.9, reg_param: 1.0, fit_bias: true, bias_scale: 1.0, max_iter: 1000, batch_size: 50, tol: 1e-4, solver: 'auto', n_jobs: nil, verbose: false, random_seed: nil) ⇒ Ridge

Create a new Ridge regressor.

Parameters:

  • learning_rate (Float) (defaults to: 0.01)

    The initial value of learning rate. The learning rate decreases as the iteration proceeds according to the equation: learning_rate / (1 + decay * t). If solver is not 'sgd', this parameter is ignored.

  • decay (Float) (defaults to: nil)

    The smoothing parameter for decreasing learning rate as the iteration proceeds. If nil is given, the decay sets to 'reg_param * learning_rate'. If solver is not 'sgd', this parameter is ignored.

  • momentum (Float) (defaults to: 0.9)

    The momentum factor. If solver is not 'sgd', this parameter is ignored.

  • reg_param (Float) (defaults to: 1.0)

    The regularization parameter.

  • fit_bias (Boolean) (defaults to: true)

    The flag indicating whether to fit the bias term.

  • bias_scale (Float) (defaults to: 1.0)

    The scale of the bias term.

  • max_iter (Integer) (defaults to: 1000)

    The maximum number of epochs that indicates how many times the whole data is given to the training process. If solver is 'svd', this parameter is ignored.

  • batch_size (Integer) (defaults to: 50)

    The size of the mini batches. If solver is not 'sgd', this parameter is ignored.

  • tol (Float) (defaults to: 1e-4)

    The tolerance of loss for terminating optimization. If solver is 'svd', this parameter is ignored.

  • solver (String) (defaults to: 'auto')

    The algorithm to calculate weights. ('auto', 'sgd', 'svd', or 'lbfgs'). 'auto' chooses the 'svd' solver if Numo::Linalg is loaded. Otherwise, it chooses the 'lbfgs' solver. 'sgd' uses the stochastic gradient descent optimization. 'svd' performs singular value decomposition of samples. 'lbfgs' uses the L-BFGS method for optimization.

  • n_jobs (Integer) (defaults to: nil)

    The number of jobs for running the fit method in parallel. If nil is given, the method does not execute in parallel. If zero or less is given, it becomes equal to the number of processors. This parameter is ignored if the Parallel gem is not loaded or solver is not 'sgd'.

  • verbose (Boolean) (defaults to: false)

    The flag indicating whether to output loss during iteration. If solver is 'svd', this parameter is ignored.

  • random_seed (Integer) (defaults to: nil)

    The seed value using to initialize the random generator.



75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
# File 'lib/rumale/linear_model/ridge.rb', line 75

def initialize(learning_rate: 0.01, decay: nil, momentum: 0.9,
               reg_param: 1.0, fit_bias: true, bias_scale: 1.0,
               max_iter: 1000, batch_size: 50, tol: 1e-4,
               solver: 'auto',
               n_jobs: nil, verbose: false, random_seed: nil)
  check_params_numeric(learning_rate: learning_rate, momentum: momentum,
                       reg_param: reg_param, bias_scale: bias_scale,
                       max_iter: max_iter, batch_size: batch_size, tol: tol)
  check_params_boolean(fit_bias: fit_bias, verbose: verbose)
  check_params_string(solver: solver)
  check_params_numeric_or_nil(decay: decay, n_jobs: n_jobs, random_seed: random_seed)
  check_params_positive(learning_rate: learning_rate, reg_param: reg_param, max_iter: max_iter, batch_size: batch_size)
  super()
  @params.merge!(method(:initialize).parameters.map { |_t, arg| [arg, binding.local_variable_get(arg)] }.to_h)
  @params[:solver] = if solver == 'auto'
                       enable_linalg?(warning: false) ? 'svd' : 'lbfgs'
                     else
                       solver.match?(/^svd$|^sgd$|^lbfgs$/) ? solver : 'lbfgs'
                     end
  @params[:decay] ||= @params[:reg_param] * @params[:learning_rate]
  @params[:random_seed] ||= srand
  @rng = Random.new(@params[:random_seed])
  @penalty_type = L2_PENALTY
  @loss_func = LinearModel::Loss::MeanSquaredError.new
  @weight_vec = nil
  @bias_term = nil
end

Instance Attribute Details

#bias_termNumo::DFloat (readonly)

Return the bias term (a.k.a. intercept).

Returns:

  • (Numo::DFloat)

    (shape: [n_outputs])



37
38
39
# File 'lib/rumale/linear_model/ridge.rb', line 37

def bias_term
  @bias_term
end

#rngRandom (readonly)

Return the random generator for random sampling.

Returns:

  • (Random)


41
42
43
# File 'lib/rumale/linear_model/ridge.rb', line 41

def rng
  @rng
end

#weight_vecNumo::DFloat (readonly)

Return the weight vector.

Returns:

  • (Numo::DFloat)

    (shape: [n_outputs, n_features])



33
34
35
# File 'lib/rumale/linear_model/ridge.rb', line 33

def weight_vec
  @weight_vec
end

Instance Method Details

#fit(x, y) ⇒ Ridge

Fit the model with given training data.

Parameters:

  • x (Numo::DFloat)

    (shape: [n_samples, n_features]) The training data to be used for fitting the model.

  • y (Numo::DFloat)

    (shape: [n_samples, n_outputs]) The target values to be used for fitting the model.

Returns:

  • (Ridge)

    The learned regressor itself.



108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
# File 'lib/rumale/linear_model/ridge.rb', line 108

def fit(x, y)
  x = check_convert_sample_array(x)
  y = check_convert_tvalue_array(y)
  check_sample_tvalue_size(x, y)

  if @params[:solver] == 'svd' && enable_linalg?(warning: false)
    fit_svd(x, y)
  elsif @params[:solver] == 'lbfgs'
    fit_lbfgs(x, y)
  else
    fit_sgd(x, y)
  end

  self
end

#predict(x) ⇒ Numo::DFloat

Predict values for samples.

Parameters:

  • x (Numo::DFloat)

    (shape: [n_samples, n_features]) The samples to predict the values.

Returns:

  • (Numo::DFloat)

    (shape: [n_samples, n_outputs]) Predicted values per sample.



128
129
130
131
# File 'lib/rumale/linear_model/ridge.rb', line 128

def predict(x)
  x = check_convert_sample_array(x)
  x.dot(@weight_vec.transpose) + @bias_term
end