Machine Learning Weight Optimization#

Classes for defining neural network weight optimization problems.

Neural Network#

Class for defining neural network classifier weights optimization problem.

class NeuralNetwork(
    hidden_nodes=None,
    activation='relu',
    algorithm='random_hill_climb',
    max_iters=100,
    bias=True,
    is_classifier=True,
    learning_rate=0.1,
    early_stopping=False,
    clip_max=1e10,
    restarts=0,
    schedule=<mlrose_ky.decay.GeomDecay object>,
    pop_size=200,
    mutation_prob=0.1,
    max_attempts=10,
    random_state=None,
    curve=False
)

Parameters:

hidden_nodes (list of ints) – List giving the number of nodes in each hidden layer.
activation (string, default: ‘relu’) – Activation function for each of the hidden layers. Must be one of: ‘identity’, ‘relu’, ‘sigmoid’ or ‘tanh’.
algorithm (string, default: ‘random_hill_climb’) – Algorithm used to find optimal network weights. Must be one of:’random_hill_climb’, ‘simulated_annealing’, ‘genetic_alg’ or ‘gradient_descent’.
max_iters (int, default: 100) – Maximum number of iterations used to fit the weights.
bias (bool, default: True) – Whether to include a bias term.
is_classifer (bool, default: True) – Whether the network is for classification or regression. Set True for classification and False for regression.
learning_rate (float, default: 0.1) – Learning rate for gradient descent or step size for randomized optimization algorithms.
early_stopping (bool, default: False) – Whether to terminate algorithm early if the loss is not improving. If True, then stop after max_attempts iters with no improvement.
clip_max (float, default: 1e+10) – Used to limit weights to the range [-1*clip_max, clip_max].
restarts (int, default: 0) – Number of random restarts. Only required if algorithm = 'random_hill_climb'.
schedule (schedule object, default = mlrose_ky.GeomDecay()) – Schedule used to determine the value of the temperature parameter. Only required if algorithm = 'simulated_annealing'.
pop_size (int, default: 200) – Size of population. Only required if algorithm = 'genetic_alg'.
mutation_prob (float, default: 0.1) – Probability of a mutation at each element of the state vector during reproduction, expressed as a value between 0 and 1. Only required if algorithm = 'genetic_alg'.
max_attempts (int, default: 10) – Maximum number of attempts to find a better state. Only required if early_stopping = True.
random_state (int, default: None) – If random_state is a positive integer, random_state is the seed used by np.random.seed(); otherwise, the random seed is not set.
curve (bool, default: False) – If bool is True, fitness_curve containing the fitness at each training iteration is returned.

Variables:

fitted_weights (array) – Numpy array giving the fitted weights when fit is performed.
loss (float) – Value of loss function for fitted weights when fit is performed.
predicted_probs (array) – Numpy array giving the predicted probabilities for each class when predict is performed for multi-class classification data; or the predicted probability for class 1 when predict is performed for binary classification data.
fitness_curve (array) – Numpy array giving the fitness at each training iteration.

Linear Regression#

Class for defining linear regression weights optimization problem. Inherits fit and predict methods from NeuralNetwork() class.

class LinearRegression(
    algorithm='random_hill_climb',
    max_iters=100,
    bias=True,
    learning_rate=0.1,
    early_stopping=False,
    clip_max=1e10,
    restarts=0,
    schedule=<mlrose_ky.decay.GeomDecay object>,
    pop_size=200,
    mutation_prob=0.1,
    max_attempts=10,
    random_state=None,
    curve=False
)

Parameters:

algorithm (string, default: ‘random_hill_climb’) – Algorithm used to find optimal network weights. Must be one of:’random_hill_climb’, ‘simulated_annealing’, ‘genetic_alg’ or ‘gradient_descent’.
max_iters (int, default: 100) – Maximum number of iterations used to fit the weights.
bias (bool, default: True) – Whether to include a bias term.
learning_rate (float, default: 0.1) – Learning rate for gradient descent or step size for randomized optimization algorithms.
early_stopping (bool, default: False) – Whether to terminate algorithm early if the loss is not improving. If True, then stop after max_attempts iters with no improvement.
clip_max (float, default: 1e+10) – Used to limit weights to the range [-1*clip_max, clip_max].
restarts (int, default: 0) – Number of random restarts. Only required if algorithm = 'random_hill_climb'.
schedule (schedule object, default = mlrose_ky.GeomDecay()) – Schedule used to determine the value of the temperature parameter. Only required if algorithm = 'simulated_annealing'.
pop_size (int, default: 200) – Size of population. Only required if algorithm = 'genetic_alg'.
mutation_prob (float, default: 0.1) – Probability of a mutation at each element of the state vector during reproduction, expressed as a value between 0 and 1. Only required if algorithm = 'genetic_alg'.
max_attempts (int, default: 10) – Maximum number of attempts to find a better state. Only required if early_stopping = True.
random_state (int, default: None) – If random_state is a positive integer, random_state is the seed used by np.random.seed(); otherwise, the random seed is not set.
curve (bool, default: False) – If bool is true, curve containing the fitness at each training iteration is returned.

Variables:

fitted_weights (array) – Numpy array giving the fitted weights when fit is performed.
loss (float) – Value of loss function for fitted weights when fit is performed.
fitness_curve (array) – Numpy array giving the fitness at each training iteration.

Logistic Regression#

Class for defining logistic regression weights optimization problem. Inherits fit and predict methods from NeuralNetwork() class.

class LogisticRegression(
    algorithm='random_hill_climb',
    max_iters=100,
    bias=True,
    learning_rate=0.1,
    early_stopping=False,
    clip_max=1e10,
    restarts=0,
    schedule=<mlrose_ky.decay.GeomDecay object>,
    pop_size=200,
    mutation_prob=0.1,
    max_attempts=10,
    random_state=None,
    curve=False
)

Parameters:

algorithm (string, default: ‘random_hill_climb’) – Algorithm used to find optimal network weights. Must be one of:’random_hill_climb’, ‘simulated_annealing’, ‘genetic_alg’ or ‘gradient_descent’.
max_iters (int, default: 100) – Maximum number of iterations used to fit the weights.
bias (bool, default: True) – Whether to include a bias term.
learning_rate (float, default: 0.1) – Learning rate for gradient descent or step size for randomized optimization algorithms.
early_stopping (bool, default: False) – Whether to terminate algorithm early if the loss is not improving. If True, then stop after max_attempts iters with no improvement.
clip_max (float, default: 1e+10) – Used to limit weights to the range [-1*clip_max, clip_max].
restarts (int, default: 0) – Number of random restarts. Only required if algorithm = 'random_hill_climb'.
schedule (schedule object, default = mlrose_ky.GeomDecay()) – Schedule used to determine the value of the temperature parameter. Only required if algorithm = 'simulated_annealing'.
pop_size (int, default: 200) – Size of population. Only required if algorithm = 'genetic_alg'.
mutation_prob (float, default: 0.1) – Probability of a mutation at each element of the state vector during reproduction, expressed as a value between 0 and 1. Only required if algorithm = 'genetic_alg'.
max_attempts (int, default: 10) – Maximum number of attempts to find a better state. Only required if early_stopping = True.
random_state (int, default: None) – If random_state is a positive integer, random_state is the seed used by np.random.seed(); otherwise, the random seed is not set.
curve (bool, default: False) – If bool is true, curve containing the fitness at each training iteration is returned.

Variables:

fitted_weights (array) – Numpy array giving the fitted weights when fit is performed.
loss (float) – Value of loss function for fitted weights when fit is performed.
fitness_curve (array) – Numpy array giving the fitness at each training iteration.