Class: Rumale::LinearModel::LinearRegression

Inherits:
BaseSGD
  • Object
show all
Includes:
Base::Regressor
Defined in:
lib/rumale/linear_model/linear_regression.rb

Overview

LinearRegression is a class that implements ordinary least square linear regression with stochastic gradient descent (SGD) optimization, singular value decomposition (SVD), or L-BFGS optimization.

Reference

  • Bottou, L., “Large-Scale Machine Learning with Stochastic Gradient Descent,” Proc. COMPSTAT'10, pp. 177–186, 2010.

Examples:

estimator =
  Rumale::LinearModel::LinearRegression.new(max_iter: 1000, batch_size: 20, random_seed: 1)
estimator.fit(training_samples, traininig_values)
results = estimator.predict(testing_samples)

# If Numo::Linalg is installed, you can specify 'svd' for the solver option.
require 'numo/linalg/autoloader'
estimator = Rumale::LinearModel::LinearRegression.new(solver: 'svd')
estimator.fit(training_samples, traininig_values)
results = estimator.predict(testing_samples)

Instance Attribute Summary collapse

Attributes included from Base::BaseEstimator

#params

Instance Method Summary collapse

Methods included from Base::Regressor

#score

Constructor Details

#initialize(learning_rate: 0.01, decay: nil, momentum: 0.9, fit_bias: true, bias_scale: 1.0, max_iter: 1000, batch_size: 50, tol: 1e-4, solver: 'auto', n_jobs: nil, verbose: false, random_seed: nil) ⇒ LinearRegression

Create a new ordinary least square linear regressor.

Parameters:

  • learning_rate (Float) (defaults to: 0.01)

    The initial value of learning rate. The learning rate decreases as the iteration proceeds according to the equation: learning_rate / (1 + decay * t). If solver is not 'sgd', this parameter is ignored.

  • decay (Float) (defaults to: nil)

    The smoothing parameter for decreasing learning rate as the iteration proceeds. If nil is given, the decay sets to 'learning_rate'. If solver is not 'sgd', this parameter is ignored.

  • momentum (Float) (defaults to: 0.9)

    The momentum factor. If solver is not 'sgd', this parameter is ignored.

  • fit_bias (Boolean) (defaults to: true)

    The flag indicating whether to fit the bias term.

  • bias_scale (Float) (defaults to: 1.0)

    The scale of the bias term.

  • max_iter (Integer) (defaults to: 1000)

    The maximum number of epochs that indicates how many times the whole data is given to the training process. If solver is 'svd', this parameter is ignored.

  • batch_size (Integer) (defaults to: 50)

    The size of the mini batches. If solver is not 'sgd', this parameter is ignored.

  • tol (Float) (defaults to: 1e-4)

    The tolerance of loss for terminating optimization. If solver is 'svd', this parameter is ignored.

  • solver (String) (defaults to: 'auto')

    The algorithm to calculate weights. ('auto', 'sgd', 'svd' or 'lbfgs'). 'auto' chooses the 'svd' solver if Numo::Linalg is loaded. Otherwise, it chooses the 'lbfgs' solver. 'sgd' uses the stochastic gradient descent optimization. 'svd' performs singular value decomposition of samples. 'lbfgs' uses the L-BFGS method for optimization.

  • n_jobs (Integer) (defaults to: nil)

    The number of jobs for running the fit method in parallel. If nil is given, the method does not execute in parallel. If zero or less is given, it becomes equal to the number of processors. This parameter is ignored if the Parallel gem is not loaded or solver is not 'sgd'.

  • verbose (Boolean) (defaults to: false)

    The flag indicating whether to output loss during iteration. If solver is 'svd', this parameter is ignored.

  • random_seed (Integer) (defaults to: nil)

    The seed value using to initialize the random generator.



74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
# File 'lib/rumale/linear_model/linear_regression.rb', line 74

def initialize(learning_rate: 0.01, decay: nil, momentum: 0.9,
               fit_bias: true, bias_scale: 1.0, max_iter: 1000, batch_size: 50, tol: 1e-4,
               solver: 'auto',
               n_jobs: nil, verbose: false, random_seed: nil)
  check_params_numeric(learning_rate: learning_rate, momentum: momentum,
                       bias_scale: bias_scale, max_iter: max_iter, batch_size: batch_size)
  check_params_boolean(fit_bias: fit_bias, verbose: verbose)
  check_params_string(solver: solver)
  check_params_numeric_or_nil(decay: decay, n_jobs: n_jobs, random_seed: random_seed)
  check_params_positive(learning_rate: learning_rate, max_iter: max_iter, batch_size: batch_size)
  super()
  @params.merge!(method(:initialize).parameters.map { |_t, arg| [arg, binding.local_variable_get(arg)] }.to_h)
  @params[:solver] = if solver == 'auto'
                       enable_linalg?(warning: false) ? 'svd' : 'lbfgs'
                     else
                       solver.match?(/^svd$|^sgd$|^lbfgs$/) ? solver : 'lbfgs'
                     end
  @params[:decay] ||= @params[:learning_rate]
  @params[:random_seed] ||= srand
  @rng = Random.new(@params[:random_seed])
  @loss_func = LinearModel::Loss::MeanSquaredError.new
  @weight_vec = nil
  @bias_term = nil
end

Instance Attribute Details

#bias_termNumo::DFloat (readonly)

Return the bias term (a.k.a. intercept).

Returns:

  • (Numo::DFloat)

    (shape: [n_outputs])



37
38
39
# File 'lib/rumale/linear_model/linear_regression.rb', line 37

def bias_term
  @bias_term
end

#rngRandom (readonly)

Return the random generator for random sampling.

Returns:

  • (Random)


41
42
43
# File 'lib/rumale/linear_model/linear_regression.rb', line 41

def rng
  @rng
end

#weight_vecNumo::DFloat (readonly)

Return the weight vector.

Returns:

  • (Numo::DFloat)

    (shape: [n_outputs, n_features])



33
34
35
# File 'lib/rumale/linear_model/linear_regression.rb', line 33

def weight_vec
  @weight_vec
end

Instance Method Details

#fit(x, y) ⇒ LinearRegression

Fit the model with given training data.

Parameters:

  • x (Numo::DFloat)

    (shape: [n_samples, n_features]) The training data to be used for fitting the model.

  • y (Numo::DFloat)

    (shape: [n_samples, n_outputs]) The target values to be used for fitting the model.

Returns:



104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
# File 'lib/rumale/linear_model/linear_regression.rb', line 104

def fit(x, y)
  x = check_convert_sample_array(x)
  y = check_convert_tvalue_array(y)
  check_sample_tvalue_size(x, y)

  if @params[:solver] == 'svd' && enable_linalg?(warning: false)
    fit_svd(x, y)
  elsif @params[:solver] == 'lbfgs'
    fit_lbfgs(x, y)
  else
    fit_sgd(x, y)
  end

  self
end

#predict(x) ⇒ Numo::DFloat

Predict values for samples.

Parameters:

  • x (Numo::DFloat)

    (shape: [n_samples, n_features]) The samples to predict the values.

Returns:

  • (Numo::DFloat)

    (shape: [n_samples, n_outputs]) Predicted values per sample.



124
125
126
127
# File 'lib/rumale/linear_model/linear_regression.rb', line 124

def predict(x)
  x = check_convert_sample_array(x)
  x.dot(@weight_vec.transpose) + @bias_term
end