[🔷 Feature request ] Batch Back-Propogation

[solid-droid]

Feature

Batch Back-Propagation / Training

Type

• Dann
• Matrix
• Layer
• Activation functions
• Loss functions
• Pool functions
• Datasets
• Documentation
• tests & examples
• Other

Description

In batch back-propagation, we split the full input set into smaller batches
and during training the model feed-forward through all the input / target pair of a batch
without changing the weight / bias but will be updating the gradient (I am not sure on the math behind it).
once the batch is complete it updates the weights and biases based on a overall picture of batch rather than using one input.
this could help the model find the right weights and biases much faster than the single input back-propagation.

Examples

How I would expect the final method to look like.

//say arch = 1 -> 2 - > 2

input = [1] 
target = [1 , 0]
nn.backpropagate(input , target)
// for one input, target pair

input =[ [ 1 ] , [-1 ] , [-5 ], [ 5 ] , [ 2 ] ]
target =[ [ 1 ,  0 ] , [ 0 , 1 ] , [ 0 , 1 ] , [ 1 , 0 ] , [ 1 , 0 ] ]

nn.backpropagate(input, target)
// expression remains same, internally it just need to check if Array.isArray(input[0]) => then batch train.

Additional context

I think this is available in all major ML libraries due to its great efficiency,
also will help in creating distributed training capabilities.

[matiasvlevi]

would you like to be assigned to this issue?

[solid-droid]

Yeah, I can look into it.
But it might take some time, as I really don't know the math behind it.

[matiasvlevi]

Perfect! If you ever need help, you can contact me on discord.

[solid-droid]

I was going through the source dann.js and got confused with the backpropagate method.
Even though you have loss functions implemented, you are not using it to optimize the network ??
usually output layer activation is combination of loss and activation.
like combination of softmax and mse

error = target - predicted ✔️
loss = lossFunc(error) ❌ //lossFunc = activation_and_lossFunc_combination
update w and b of output layer based on d/dx of loss ❌
update w and b based on d/dx of activation functions ✔️

Dann/build/dann.js

Lines 2006 to 2018 in fdc7918

	this.outs = this.feedForward(inputs, { log: false, mode: mode });
	this.errors[this.errors.length - 1] = Matrix.sub(
	targets,
	this.Layers[this.Layers.length - 1].layer
	);
	this.gradients[this.gradients.length - 1] = Matrix.map(
	this.Layers[this.Layers.length - 1].layer,
	this.Layers[this.Layers.length - 1].actfunc_d
	);
	this.gradients[this.gradients.length - 1].mult(
	this.errors[this.errors.length - 1]
	);
	this.gradients[this.gradients.length - 1].mult(this.lr);