Class: Rumale::LinearModel::SGDRegressor

Inherits:
SGDEstimator show all
Includes:
Base::Regressor
Defined in:
rumale-linear_model/lib/rumale/linear_model/sgd_regressor.rb

Overview

SGDRegressor is a class that implements linear regressor with stochastic gradient descent optimization.

Reference

  • Shalev-Shwartz, S., and Singer, Y., “Pegasos: Primal Estimated sub-GrAdient SOlver for SVM,” Proc. ICML’07, pp. 807–814, 2007.

  • Tsuruoka, Y., Tsujii, J., and Ananiadou, S., “Stochastic Gradient Descent Training for L1-regularized Log-linear Models with Cumulative Penalty,” Proc. ACL’09, pp. 477–485, 2009.

  • Bottou, L., “Large-Scale Machine Learning with Stochastic Gradient Descent,” Proc. COMPSTAT’10, pp. 177–186, 2010.

Examples:

require 'rumale/linear_model/sgd_regressor'

estimator =
  Rumale::LinearModel::SGDRegressor.new(loss: 'squared_error', reg_param: 1.0, max_iter: 1000, batch_size: 50, random_seed: 1)
estimator.fit(training_samples, traininig_target_values)
results = estimator.predict(testing_samples)

Instance Attribute Summary collapse

Attributes inherited from BaseEstimator

#bias_term, #weight_vec

Attributes inherited from Base::Estimator

#params

Instance Method Summary collapse

Methods included from Base::Regressor

#score

Constructor Details

#initialize(loss: 'squared_error', learning_rate: 0.01, decay: nil, momentum: 0.9, penalty: 'l2', reg_param: 1.0, l1_ratio: 0.5, fit_bias: true, bias_scale: 1.0, epsilon: 0.1, max_iter: 1000, batch_size: 50, tol: 1e-4, n_jobs: nil, verbose: false, random_seed: nil) ⇒ SGDRegressor

Create a new linear regressor with stochastic gradient descent optimization.

Parameters:

  • loss (String) (defaults to: 'squared_error') —

    The loss function to be used (‘squared_error’ and ‘epsilon_insensitive’).

  • learning_rate (Float) (defaults to: 0.01) —

    The initial value of learning rate. The learning rate decreases as the iteration proceeds according to the equation: learning_rate / (1 + decay * t).

  • decay (Float) (defaults to: nil) —

    The smoothing parameter for decreasing learning rate as the iteration proceeds. If nil is given, the decay sets to ‘reg_param * learning_rate’.

  • momentum (Float) (defaults to: 0.9) —

    The momentum factor.

  • penalty (String) (defaults to: 'l2') —

    The regularization type to be used (‘l1’, ‘l2’, and ‘elasticnet’).

  • l1_ratio (Float) (defaults to: 0.5) —

    The elastic-net type regularization mixing parameter. If penalty set to ‘l2’ or ‘l1’, this parameter is ignored. If l1_ratio = 1, the regularization is similar to Lasso. If l1_ratio = 0, the regularization is similar to Ridge. If 0 < l1_ratio < 1, the regularization is a combination of L1 and L2.

  • reg_param (Float) (defaults to: 1.0) —

    The regularization parameter.

  • fit_bias (Boolean) (defaults to: true) —

    The flag indicating whether to fit the bias term.

  • bias_scale (Float) (defaults to: 1.0) —

    The scale of the bias term.

  • epsilon (Float) (defaults to: 0.1) —

    The margin of tolerance. If loss set to ‘squared_error’, this parameter is ignored.

  • max_iter (Integer) (defaults to: 1000) —

    The maximum number of epochs that indicates how many times the whole data is given to the training process.

  • batch_size (Integer) (defaults to: 50) —

    The size of the mini batches.

  • tol (Float) (defaults to: 1e-4) —

    The tolerance of loss for terminating optimization.

  • n_jobs (Integer) (defaults to: nil) —

    The number of jobs for running the fit method in parallel. If nil is given, the method does not execute in parallel. If zero or less is given, it becomes equal to the number of processors. This parameter is ignored if the Parallel gem is not loaded.

  • verbose (Boolean) (defaults to: false) —

    The flag indicating whether to output loss during iteration.

  • random_seed (Integer) (defaults to: nil) —

    The seed value using to initialize the random generator.



59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
 
# File 'rumale-linear_model/lib/rumale/linear_model/sgd_regressor.rb', line 59

def initialize(loss: 'squared_error', learning_rate: 0.01, decay: nil, momentum: 0.9,
               penalty: 'l2', reg_param: 1.0, l1_ratio: 0.5,
               fit_bias: true, bias_scale: 1.0,
               epsilon: 0.1,
               max_iter: 1000, batch_size: 50, tol: 1e-4,
               n_jobs: nil, verbose: false, random_seed: nil)
  super()
  @params.merge!(
    loss: loss,
    learning_rate: learning_rate,
    decay: decay,
    momentum: momentum,
    penalty: penalty,
    reg_param: reg_param,
    l1_ratio: l1_ratio,
    fit_bias: fit_bias,
    bias_scale: bias_scale,
    epsilon: epsilon,
    max_iter: max_iter,
    batch_size: batch_size,
    tol: tol,
    n_jobs: n_jobs,
    verbose: verbose,
    random_seed: random_seed
  )
  @params[:decay] ||= @params[:reg_param] * @params[:learning_rate]
  @params[:random_seed] ||= srand
  @rng = Random.new(@params[:random_seed])
  @penalty_type = @params[:penalty]
  @loss_func = case @params[:loss]
               when Rumale::LinearModel::Loss::MeanSquaredError::NAME
                 Rumale::LinearModel::Loss::MeanSquaredError.new
               when Rumale::LinearModel::Loss::EpsilonInsensitive::NAME
                 Rumale::LinearModel::Loss::EpsilonInsensitive.new(epsilon: @params[:epsilon])
               else
                 raise ArgumentError, "given loss '#{loss}' is not supported."
               end
end

Instance Attribute Details

#rng ⇒ Random (readonly)

Return the random generator for performing random sampling.

Returns:

  • (Random) 


29
30
31
 
# File 'rumale-linear_model/lib/rumale/linear_model/sgd_regressor.rb', line 29

def rng
  @rng
end

Instance Method Details

#fit(x, y) ⇒ Object

Fit the model with given training data.

@retu:rn [SGDRegressor] The learned regressor itself.

Parameters:

  • x (Numo::DFloat) —

    (shape: [n_samples, n_features]) The training data to be used for fitting the model.

  • y (Numo::DFloat) —

    (shape: [n_samples, n_outputs]) The target values to be used for fitting the model.



103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
 
# File 'rumale-linear_model/lib/rumale/linear_model/sgd_regressor.rb', line 103

def fit(x, y)
  x = Rumale::Validation.check_convert_sample_array(x)
  y = Rumale::Validation.check_convert_target_value_array(y)
  Rumale::Validation.check_sample_size(x, y)

  n_outputs = y.shape[1].nil? ? 1 : y.shape[1]
  n_features = x.shape[1]

  if n_outputs > 1
    @weight_vec = Numo::DFloat.zeros(n_outputs, n_features)
    @bias_term = Numo::DFloat.zeros(n_outputs)
    if enable_parallel?
      models = parallel_map(n_outputs) { |n| partial_fit_(x, y[true, n]) }
      n_outputs.times { |n| @weight_vec[n, true], @bias_term[n] = models[n] }
    else
      n_outputs.times { |n| @weight_vec[n, true], @bias_term[n] = partial_fit_(x, y[true, n]) }
    end
  else
    @weight_vec, @bias_term = partial_fit_(x, y)
  end

  self
end

#partial_fit(x, y) ⇒ SGDRegressor

Perform 1-epoch of stochastic gradient descent optimization with given training data.

Parameters:

  • x (Numo::DFloat) —

    (shape: [n_samples, n_features]) The training data to be used for fitting the model.

  • y (Numo::DFloat) —

    (shape: [n_samples]) The single target variables to be used for fitting the model.

Returns:



132
133
134
135
136
137
138
139
140
141
142
143
144
 
# File 'rumale-linear_model/lib/rumale/linear_model/sgd_regressor.rb', line 132

def partial_fit(x, y)
  x = Rumale::Validation.check_convert_sample_array(x)
  y = Rumale::Validation.check_convert_target_value_array(y)
  Rumale::Validation.check_sample_size(x, y)

  n_features = x.shape[1]
  n_features += 1 if fit_bias?
  need_init = @weight.nil? || @weight.shape[0] != n_features

  @weight_vec, @bias_term = partial_fit_(x, y, max_iter: 1, init: need_init)

  self
end

#predict(x) ⇒ Numo::DFloat

Predict values for samples.

Parameters:

  • x (Numo::DFloat) —

    (shape: [n_samples, n_features]) The samples to predict the values.

Returns:

  • (Numo::DFloat) —

    (shape: [n_samples, n_outputs]) Predicted values per sample.



150
151
152
153
154
 
# File 'rumale-linear_model/lib/rumale/linear_model/sgd_regressor.rb', line 150

def predict(x)
  x = Rumale::Validation.check_convert_sample_array(x)

  x.dot(@weight_vec.transpose) + @bias_term
end