神经网络正向传播和反向传播算法示例：Sigmoid 激活函数

(1) 第一层隐层计算： $h_{1,1} = \sigma(w_{1,1}^{(1)}x_1 + w_{1,2}^{(1)}x_2 + b_{1,1}^{(1)}) = \sigma(0.5 \times 1 + 0.3 \times (-1) + 0.2) = 0.731$ $h_{1,2} = \sigma(w_{2,1}^{(1)}x_1 + w_{2,2}^{(1)}x_2 + b_{2,1}^{(1)}) = \sigma(0.8 \times 1 + 0.2 \times (-1) + 0.1) = 0.689$

第二层隐层计算： $h_{2,1} = \sigma(w_{1,1}^{(2)}h_{1,1} + w_{1,2}^{(2)}h_{1,2} + b_{1,1}^{(2)}) = \sigma(0.4 \times 0.731 + 0.9 \times 0.689 + 0.4) = 0.785$ $h_{2,2} = \sigma(w_{2,1}^{(2)}h_{1,1} + w_{2,2}^{(2)}h_{1,2} + b_{2,1}^{(2)}) = \sigma(0.3 \times 0.731 + 0.5 \times 0.689 + 0.3) = 0.709$

输出层计算： $y_1 = \sigma(w_{1,1}^{(3)}h_{2,1} + w_{1,2}^{(3)}h_{2,2} + b_{1,1}^{(3)}) = \sigma(0.6 \times 0.785 + 0.4 \times 0.709 + 0.1) = 0.751$ $y_2 = \sigma(w_{2,1}^{(3)}h_{2,1} + w_{2,2}^{(3)}h_{2,2} + b_{2,1}^{(3)}) = \sigma(0.2 \times 0.785 + 0.8 \times 0.709 + 0.2) = 0.811$

(2) 误差计算： $E = \frac{1}{2}\sum_{i=1}^{2}(y_i - t_i)^2 = \frac{1}{2}[(0.751 - 0.5)^2 + (0.811 - 1)^2] = 0.060$

调整参数： 可以使用反向传播算法来调整参数。首先计算输出层的误差项： $\delta_{1}^{(3)} = y_1(1 - y_1)(y_1 - t_1) = 0.751(1 - 0.751)(0.251) = 0.046$ $\delta_{2}^{(3)} = y_2(1 - y_2)(y_2 - t_2) = 0.811(1 - 0.811)(0.189) = 0.029$

然后计算第二层隐层的误差项： $\delta_{1}^{(2)} = h_{2,1}(1 - h_{2,1})[w_{1,1}^{(3)}\delta_{1}^{(3)} + w_{2,1}^{(3)}\delta_{2}^{(3)}] = 0.785(1 - 0.785)[0.6 \times 0.046 + 0.2 \times 0.029] = 0.019$ $\delta_{2}^{(2)} = h_{2,2}(1 - h_{2,2})[w_{1,2}^{(3)}\delta_{1}^{(3)} + w_{2,2}^{(3)}\delta_{2}^{(3)}] = 0.709(1 - 0.709)[0.4 \times 0.046 + 0.8 \times 0.029] = 0.013$

最后计算第一层隐层的误差项： $\delta_{1}^{(1)} = h_{1,1}(1 - h_{1,1})[w_{1,1}^{(2)}\delta_{1}^{(2)} + w_{2,1}^{(2)}\delta_{2}^{(2)}] = 0.731(1 - 0.731)[0.4 \times 0.019 + 0.9 \times 0.013] = 0.003$ $\delta_{2}^{(1)} = h_{1,2}(1 - h_{1,2})[w_{1,2}^{(2)}\delta_{1}^{(2)} + w_{2,2}^{(2)}\delta_{2}^{(2)}] = 0.689(1 - 0.689)[0.3 \times 0.019 + 0.5 \times 0.013] = 0.002$

根据误差项和权重更新公式，可以得到参数的更新值： $w_{1,1}^{(3)} \leftarrow w_{1,1}^{(3)} - \eta\delta_{1}^{(3)}h_{2,1} = 0.6 - 0.1 \times 0.046 \times 0.785 = 0.595$ $w_{1,2}^{(3)} \leftarrow w_{1,2}^{(3)} - \eta\delta_{2}^{(3)}h_{2,1} = 0.4 - 0.1 \times 0.029 \times 0.785 = 0.396$ $w_{2,1}^{(3)} \leftarrow w_{2,1}^{(3)} - \eta\delta_{1}^{(3)}h_{2,2} = 0.2 - 0.1 \times 0.046 \times 0.709 = 0.195$ $w_{2,2}^{(3)} \leftarrow w_{2,2}^{(3)} - \eta\delta_{2}^{(3)}h_{2,2} = 0.8 - 0.1 \times 0.029 \times 0.709 = 0.794$ $b_{1,1}^{(3)} \leftarrow b_{1,1}^{(3)} - \eta\delta_{1}^{(3)} = 0.1 - 0.1 \times 0.046 = 0.096$ $b_{2,1}^{(3)} \leftarrow b_{2,1}^{(3)} - \eta\delta_{2}^{(3)} = 0.2 - 0.1 \times 0.029 = 0.197$ $w_{1,1}^{(2)} \leftarrow w_{1,1}^{(2)} - \eta\delta_{1}^{(2)}h_{1,1} = 0.4 - 0.1 \times 0.019 \times 0.731 = 0.397$ $w_{1,2}^{(2)} \leftarrow w_{1,2}^{(2)} - \eta\delta_{2}^{(2)}h_{1,1} = 0.9 - 0.1 \times 0.013 \times 0.731 = 0.899$ $w_{2,1}^{(2)} \leftarrow w_{2,1}^{(2)} - \eta\delta_{1}^{(2)}h_{1,2} = 0.2 - 0.1 \times 0.019 \times 0.689 = 0.199$ $w_{2,2}^{(2)} \leftarrow w_{2,2}^{(2)} - \eta\delta_{2}^{(2)}h_{1,2} = 0.5 - 0.1 \times 0.013 \times 0.689 = 0.499$ $b_{1,1}^{(2)} \leftarrow b_{1,1}^{(2)} - \eta\delta_{1}^{(2)} = 0.4 - 0.1 \times 0.019 = 0.398$ $b_{2,1}^{(2)} \leftarrow b_{2,1}^{(2)} - \eta\delta_{2}^{(2)} = 0.3 - 0.1 \times 0.013 = 0.299$ $w_{1,1}^{(1)} \leftarrow w_{1,1}^{(1)} - \eta\delta_{1}^{(1)}x_1 = 0.5 - 0.1 \times 0.003 \times 1 = 0.4997$ $w_{1,2}^{(1)} \leftarrow w_{1,2}^{(1)} - \eta\delta_{2}^{(1)}x_1 = 0.3 - 0.1 \times 0.002 \times 1 = 0.2998$ $w_{2,1}^{(1)} \leftarrow w_{2,1}^{(1)} - \eta\delta_{1}^{(1)}x_2 = 0.2 - 0.1 \times 0.003 \times (-1) = 0.2003$ $w_{2,2}^{(1)} \leftarrow w_{2,2}^{(1)} - \eta\delta_{2}^{(1)}x_2 = 0.8 - 0.1 \times 0.002 \times (-1) = 0.8002$ $b_{1,1}^{(1)} \leftarrow b_{1,1}^{(1)} - \eta\delta_{1}^{(1)} = 0.2 - 0.1 \times 0.003 = 0.1997$ $b_{2,1}^{(1)} \leftarrow b_{2,1}^{(1)} - \eta\delta_{2}^{(1)} = 0.1 - 0.1 \times 0.002 = 0.0998$