Neural Network

Simple model

In this model we have some input $x_1 … x_n$ as our dendrites and output is the result of our hypothesis function (axons).

Or we can say our input nodes (input layer) go into another node (hidden layer), and are output as the hypothesis function (output layer).

For a full connect linear propagation, we need understand BackPropagation(反向传播法):

If we have some input data $x$, target $y$, output $o$ and initial weight $w$. Our aim of Neural Network is to let the estiamte output close to the true output.

Input Layer to Hidden Layer

$result = w \cdot x + b$

Note that the weight is between input layer and hidden layer$(w_1, w_2, w_3, w_4)$.

The output of hidden layer with sigmoid function is

$out_{hidden} = \frac{1}{1+e^{-result}}$

Hidden Layer to Output Layer

Similar to the above and we can get $out_{output}$

By using the initial weight the $out_{output}$ always far away from the true output result then we need to use BackPropagation to renew the weight.

BackPropagation

Find the Total variance of output:

$Var_{total} = E[(target - out_{output})]^2$

For example in the figure above, the $Var_{total}$ can also write like $Var_o1 + Var_o2$

To renew the a weight, we want to know how much of influence of this weight to the total variance. Then this in Math is partial derivative:

$\frac{\delta Var_{total}}{\delta w} = \frac{\delta Var_{total}}{\delta out_{output}} \cdot \frac{out_{output}}{out_{hidden}} \cdot$