Deep NN(深度神经网络)---deeplearning.ai---笔记及Python源码（15）
深度神经网络指的是多隐层的神经网络。
本文采用一个两输入，3个隐藏层，1个输出层来进行说明深度神经网络的前向和后向传播，以及具体的公式推导和代码撰写。
一、符号定义
参看笔记13和笔记14
二、基本模型
三、模型计算
（1） 前向传播：在样本数为1的情况
$$\begin{array}{l}{a^{[0]}} = {\left( {{x_1},{x_2}} \right)^T}\\{z^{[1]}} = {w^{[1]}}{a^{[0]}} + {b^{[1]}}\\{a^{[1]}} = {g^{[1]}}({z^{[1]}})\\{z^{[2]}} = {w^{[2]}}{a^{[1]}} + {b^{[2]}}\\{a^{[2]}} = {g^{[2]}}({z^{[2]}})\\{z^{[3]}} = {w^{[3]}}{a^{[2]}} + {b^{[3]}}\\{a^{[3]}} = {g^{[3]}}({z^{[3]}})\\{z^{[4]}} = {w^{[4]}}{a^{[3]}} + {b^{[4]}}\\{a^{[4]}} = {g^{[4]}}({z^{[4]}}) = \widehat y\end{array}$$
可以得出：
$$\begin{array}{l}{z^{[L]}} = {w^{[L]}}{a^{[L - 1]}} + {b^{[L]}}\\{a^{[L]}} = {g^{[L]}}({z^{[L]}})\end{array}$$
（2） 后向传播：求导数采用链式法则。
$$\begin{array}{l}d{a^{[4]}} = ( - \frac{y}{{{a^{[4]}}}} + \frac{{(1 - y)}}{{(1 - {a^{[4]}})}})\\d{z^{[4]}} = {a^{[4]}} - y\\d{w^{[4]}} = d{z^{[4]}}{a^{[3]}}\\d{b^{[4]}} = d{z^{[4]}}\\d{a^{[3]}} = d{z^{[4]}}{w^{[4]}}\\d{z^{[3]}} = d{a^{[3]}}{a^{[3]}}(1 - {a^{[3]}})\\d{w^{[3]}} = d{z^{[3]}}{a^{[2]}}\\d{b^{[3]}} = d{z^{[3]}}\\....\end{array}$$
隐层的权重和偏置的导数可以表示为：
$$\begin{array}{l}d{a^{[L - 1]}} = {w^{[L]T}}d{z^{[L]}}\\d{z^{[L]}} = d{a^{[L]}}{a^{[L]}}(1 - {a^{[L]}})\\d{w^{[L]}} = d{z^{[L]}}{a^{[L - 1]}}\\d{b^{[L]}} = d{z^{[L]}}\\....\end{array}$$
四、Python 代码实现
输入的样本和输出样本的格式同笔记14一样。来实现一中的深度神经网络。采取梯度下降的方法进行。为了更加清晰的展示深度神经网络中隐层的每个权值和偏置，代码中没有使用数组的形式存储，而是直接采取变量的形式进行展现，也没有使用for循环进行隐层的计算，而是将推导的每一步都编入代码。
# -*- coding: utf-8 -*-
"""
Created on Wed Apr 25 16:06:58 2018
@author: HGaviN
"""
import numpy as np
m = 10
n_0 = 2
n_1 = 4
n_2 = 3
n_3 = 2
n_4 = 1 #these can be save as an array
X = np.array([[1,1],[1.5,1.5],[2,2],[2.5,2.5],[2.75,2.75],[3.15,3.15],[3.5,3.5],[3.75,3.75],[4,4],[4.5,4.5]])#create some examples
X = X.T # transposition
Y = np.array([[0],[0],[0],[0],[0],[1],[1],[1],[1],[1]])
#initialization
alpha = 0.01
# Forward Propagation
W1 = 0.01*np.random.rand(n_0,n_1)
W2 = 0.01*np.random.rand(n_1,n_2)
W3 = 0.01*np.random.rand(n_2,n_3)
W4 = 0.01*np.random.rand(n_3,n_4)
b1 = np.zeros([n_1,1])
b2 = np.zeros([n_2,1])
b3 = np.zeros([n_3,1])
b4 = np.zeros([n_4,1])
# backward propagation
dW1 = np.zeros([n_0,n_1])
dZ1 = np.zeros([m,n_1])
db1 = np.zeros([n_1,1])
dW2 = np.zeros([n_1,n_2])
dZ2 = np.zeros([m,n_2])
db2 = np.zeros([n_2,1])
dW3 = np.zeros([n_2,n_3])
dZ3 = np.zeros([m,n_3])
db3 = np.zeros([n_3,1])
dW4 = np.zeros([n_3,n_4])
dZ4 = np.zeros([m,n_4])
db4 = np.zeros([n_4,1])
j = 0
for iter in range(50):
#Forward Propagation
Z1 = np.dot(W1.T,X)+b1 # n_1 X m
A1 = 1/(1+np.exp(-Z1)) # n_1 X m
Z2 = np.dot(W2.T,A1)+b2 # n_2 X m
A2 = 1/(1+np.exp(-Z2)) # n_2 X m
Z3 = np.dot(W3.T,A2)+b3# n_3 X m
A3 = 1/(1+np.exp(-Z3))# n_3 X m
Z4 = np.dot(W4.T,A3)+b4# n_4 X m
A4 = 1/(1+np.exp(-Z4))# n_4 X m
#backward propagation
dZ4 = A4.T - Y# m X n_4
dW4 = 1/m*np.dot(A3,dZ4)#n_3 X n_4
db4 = 1/m*np.sum(dZ4,axis=0).T # n_4X 1
dA3 = np.dot(dZ4,W4.T) # m X n_3
dZ3 = np.multiply(dA3,np.multiply(A3,1-A3).T) #m X n_3
dW3 = 1/m*np.dot(A2,dZ3)#n_2 X n_3
db3 = 1/m*np.sum(dZ3,axis=0,keepdims=True).T #n_3 X 1
dA2 = np.dot(dZ3,W3.T) # m X n_2
dZ2 = np.multiply(dA2,np.multiply(A2,1-A2).T) #m X n_2
dW2 = 1/m*np.dot(A1,dZ2)#n_1 X n_2
db2 = 1/m*np.sum(dZ2,axis=0,keepdims=True).T #n_2 X 1
dA1 = np.dot(dZ2,W2.T) # m X n_1
dZ1 = np.multiply(dA1,np.multiply(A1,1-A1).T) #m X n_1
dW1 = 1/m*np.dot(X,dZ1)#n_0 X n_1
db1 = 1/m*np.sum(dZ1,axis=0,keepdims=True).T #n_1 X 1
W4 = W4 - alpha*dW4
W3 = W3 - alpha*dW3
W2 = W2 - alpha*dW2
W1 = W1 - alpha*dW1
b4 = b4 - alpha*db4
b3 = b3 - alpha*db3
b2 = b2 - alpha*db2
b1 = b1 - alpha*db1
#np.multiply is like .* in matlab
j = -1/m*np.sum(np.multiply(Y.T,np.log(A2))+np.multiply((1-Y).T,np.log(1-A2)))#
print (j)
print ('\n')
xp = np.array([[4],[3.5]])
z1p = np.dot(W1.T,xp)+b1
a1p = 1/(1+np.exp(-z1p))
z2p = np.dot(W2.T,a1p)+b2
a2p = 1/(1+np.exp(-z2p))
z3p = np.dot(W3.T,a2p)+b3
a3p = 1/(1+np.exp(-z3p))
z4p=np.dot(W4.T,a3p)+b4
a4p = 1/(1+np.exp(-z4p))
print (a4p) 五、参数与超参数
一般而言，神经网络的参数指的是 权重和偏置，而超参数指的是隐层的数量、每个隐层隐元的数量、激活函数类型、学习参数、迭代次数等影响神经网络参数的参数。