神经网络和深度学习吴恩达coursera笔记

Deep Learning

Basic

• 神经网络：

algorithm1

input1

output

input2

input3

input4

algorithm2

• 监督学习：1个x对应1个y；

• Sigmoid : 激活函数
  $$sigmoid=\frac{1}{1+e^{-x}}$$

• ReLU : 线性整流函数；

Logistic Regression

–>binary classification / x–>y 0 1

some sign

$$\begin{gathered} (x,y),x\in {\mathbb{R}}^{n_x},y\in 0,1 \\ M=m_{train}{m}_{test}=test \\ M:(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)})...,(x^{(m)},y^{(m)}) \\ X=\left[\begin{array}{cccc}{x}^{(1)}& x^{(2)}& \cdots & x^{(m)}\end{array}\right]\leftarrow n^x\times m \\ \hat{y}=P(y=1\mid x)\hat{y}=\sigma (w^{t}x+b)w\in {\mathbb{R}}^{n_x}b\in \mathbb{R} \\ \sigma (z)=\frac{1}{1+e^{-z}} \end{gathered}$$

Loss function

单个样本
$$\begin{gathered} Lossfunction:\mathcal{L}(\hat{y},y)=\frac{1}{2}(\hat{y}-y{)}^2 \\ p(y\mid x)={\hat{y}}^y(1-\hat{y}{)}^{(1-y)} \\ mincost\to max\log (y\mid x) \\ \mathcal{L}(\hat{y},y)=-(y\log (\hat{y})+(1-y)\log (1-\hat{y})) \\ y=1:\mathcal{L}(\hat{y},y)=-\log \hat{y}\log \hat{y}\leftarrow larger\hat{y}\leftarrow larger \\ y=0:\mathcal{L}(\hat{y},y)=-\log (1-\hat{y})\log (1-\hat{y})\leftarrow larger\hat{y}\leftarrow smaller \end{gathered}$$

cost function

$$\mathcal{J}(w,b)=\frac{1}{m}\sum _{i=1}^m\mathcal{L}({\hat{y}}^{(i)},y^{(i)})$$

Gradient Descent

find w,b that minimiaze J(w,b) ;

Repeat:
$$\begin{gathered} w:=w-\alpha \frac{\partial \mathcal{J}(w,b)}{\partial w}(dw) \\ b:=b-\alpha \frac{\partial \mathcal{J}(w,b)}{\partial b}(db) \end{gathered}$$

Computation Grapha

example:
$$J=3(a+bc)$$

v=a+u

a

b

c

u=bc

J=3v

one example gradient descent computer grapha:

recap:
$$\begin{gathered} z=w^{T}x+b \\ \hat{y}=a=\sigma (z)=\frac{1}{1+e^{-z}} \\ \mathcal{L}(a,y)=-(y\log (a)+(1-y)\log (1-a)) \end{gathered}$$
The grapha:

$$\begin{gathered} {}^{\prime }d{a}^{\prime }=\frac{d\mathcal{L}(a,y)}{da}=-\frac{y}{a}+\frac{1-y}{1-a}{ \\ }^{\prime }d{z}^{\prime }=\frac{d\mathcal{L}(a,y)}{dz}=\frac{d\mathcal{L}}{da}\cdot \frac{da}{dz}=a-y{ \\ }^{\prime }d{w}_1^{\prime }=x_1\cdot dz... \\ {w}_1:=w_1-\alpha d{w}_1... \end{gathered}$$
m example gradient descent computer grapha:

recap:
$$\mathcal{J}(w,b)=\frac{1}{m}\sum _{i=1}^m\mathcal{L}(a^{(i)},y^{(1)})$$
The grapha: (two iterate)
$$\begin{gathered} \frac{\partial }{\partial w_1}\mathcal{J}(w,b)=\frac{1}{m}\sum _{i=1}^m\frac{\partial }{\partial w_1}\mathcal{L}(a^{(i)},y^{(1)}) \\ Fori=1tom:\{ \\ {a}^{(i)}=\sigma (w^T{x}^{(i)}+b) \\ \mathcal{J}+=-[y^{(i)}\log a^i+(1-y^{(i)}\log (1-a^{(i)}))] \\ d{z}^{(i)}=a^{(i)}-y^{(i)} \\ d{w}_1+=x_1^{(i)}d{z}^{(i)} \\ d{w}_2+=x_2^{(i)}d{z}^{(i)} \\ db+=d{z}^{(i)}\} \\ \mathcal{J}/=m;d{w}_1/=m;d{w}_2/=m;db/=m \\ d{w}_1=\frac{\partial \mathcal{J}}{\partial w_1} \\ {w}_1=w_1-\alpha d{w}_1 \end{gathered}$$

Vectorization

vectorized

$$z=np.dot(w,x)+b$$
logistic regression derivatives:

change:
$$\begin{gathered} d{w}_1=0,d{w}_2=0\to dw=np.zeros((n_x,1)) \\ \{\begin{array}{l}d{w}_1+=x_1^{(i)}d{z}^{(i)}\\ d{w}_2+=x_2^{(i)}d{z}^{(i)}\end{array}\to dw+=x^{(i)}d{z}^{(i)} \\ Z=\left(\begin{array}{cccc}{z}^{(1)}& z^{(2)}& ...& z^{(m)}\end{array}\right)=w^{T}X+b \\ A=\sigma (Z) \\ dz=A-Y=\left(\begin{array}{cccc}{a}^{(1)}-y^{(1)}& z^{(2)}-y^{(2)}& ...& z^{(m)}-y^{(m)}\end{array}\right) \\ db=\frac{1}{m}\sum _{i=1}^{m}d{z}^{(i)}=\frac{1}{m}np.sum(dz) \\ dw=\frac{1}{m}Xd{z}^T=\frac{1}{m}\left(\begin{array}{cccc}{x}^{(1)}\cdot d{z}^{(1)}& x^{(2)}\cdot d{z}^{(2)}& ...& x^{(m)}\cdot d{z}^{(m)}\end{array}\right) \end{gathered}$$

Implementing:

$$\begin{gathered} Z=w^{T}X+b=np.dot(w^T,X)+b \\ A=\sigma (Z) \\ J=-\frac{1}{m}\sum _{i=1}^m(y^{(i)}\log (a^{(i)})+(1-y^{(i)})\log (1-a^{(i)})) \\ dZ=A-Y \\ dw=\frac{1}{m}Xd{Z}^T \\ db=\frac{1}{m}np.sum(dZ) \\ w:=w-\alpha dw \\ b:=b-\alpha db \end{gathered}$$

broadcasting

$$np.dot(w^T,X)+b$$
A note on numpy
$$\begin{gathered} a=np.random.randn(5)//wrong\to a=a.reshape(5,1) \\ assert(a.shape==(5,1)) \\ a=np.random.randn(5,1)\to columvector \end{gathered}$$

Shallow Neural Network

Representation

2 layer NN:
$$\begin{gathered} Inputlayer\to hidden\to layer\to outlayer \\ {a}^{[0]}\to a^{[1]}\to a^{[2]} \\ {z}^{[1]}=W^{[1]}{a}^{[0]}+b^{[1]} \\ {a}^{[1]}=\sigma (z^{[1]}) \\ {z}^{[2]}=W^{[2]}{a}^{[1]}+b^{[2]} \\ {a}^{[2]}=\sigma (z^{[2]})=\hat{y} \end{gathered}$$

computing:

z i [ 1 ] = w i [ 1 ] T x + b i [ 1 ] a i [ 1 ] = σ ( z i [ 1 ] ) [ w 1 [ 1 ] T w 2 [ 1 ] T w 3 [ 1 ] T w 4 [ 1 ] T ] ⋅ [ x 1 x 2 x 3 ] + [ b 1 [ 1 ] b 2 [ 1 ] b 3 [ 1 ] b 4 [ 1 ] ] = [ z 1 [ 1 ] z 2 [ 1 ] z 3 [ 1 ] z 4 [ 1 ] ] z_i^{[1]}=w_i^{[1]T}x+b_i^{[1]}\\ a_i^{[1]}=\sigma(z_i^{[1]})\\ \left[ \begin{matrix} w_1^{[1]T}\\w_2^{[1]T}\\w_3^{[1]T}\\w_4^{[1]T} \end{matrix} \right] \cdot \left[ \begin{matrix} x_1\\x_2\\x_3 \end{matrix} \right]+\left[ \begin{matrix} b_1^{[1]}\\b_2^{[1]}\\b_3^{[1]}\\b_4^{[1]} \end{matrix} \right]=\left[ \begin{matrix} z_1^{[1]}\\z_2^{[1]}\\z_3^{[1]}\\z_4^{[1]} \end{matrix} \right] zi[1]​=wi[1]T​x+bi[1]​ai[1]​=σ(zi[1]​) ​w1[1]T​w2[1]T​w3[1]T​w4[1]T​​ ​⋅ ​x1​x2​x3​​ ​+ ​b1[1]​b2[1]​b3[1]​b4[1]​​ ​= ​z1[1]​z2[1]​z3[1]​z4[1]​​ ​

Vectorize:

$$\begin{gathered} x^{(i)}\to a^{[2](i)}={\hat{y}}^{(i)} \\ {Z}^{[1]}=W^{[1]}X+b^{[1]} \\ {A}^{[1]}=\sigma (Z^{[1]}) \\ {Z}^{[2]}=W^{[2]}{A}^{[1]}+b^{[2]} \\ {A}^{[2]}=\sigma (Z^{[2]}) \\ {W}^{[1]}\cdot \left[\begin{array}{cccc}{x}^{(1)}& x^{(2)}& \cdots & x^{(m)}\end{array}\right]+b=\left[\begin{array}{cccc}{z}^{[1](1)}& z^{[1](2)}& \cdots & z^{[1](m)}\end{array}\right]=Z^{[1]} \end{gathered}$$

Activation functions

$$\begin{gathered} a=\frac{1}{1+e^{-z}},a^{\prime }=a(1-a) \\ a=\tanh (z)=\frac{e^z-e^{-z}}{e^z+e^{-z}},a\in (-1,1),a^{\prime }=1-a^2 \\ a=max(0,z) \\ a=max(0.01z,z) \end{gathered}$$

Gradient descent

computation

$$\begin{gathered} z^{[1]}=W^{[1]}x+b^{[1]}\to \\ {a}^{[1]}=\sigma (z^{[1]})\to \\ {z}^{[2]}=W^{[2]}{a}^{[1]}+b^{[2]}\to \\ {a}^{[2]}=\sigma (z^{[2]})\to \\ \mathcal{L}(a^{[2]},y) \\ d{z}^{[2]}=a^{[2]}-y \\ d{w}^{[2]}=d{z}^{[2]}{a}^{[1]T} \\ d{b}^{[2]}=d{z}^{[2]} \\ d{z}^{[1]}=w^{[2]T}d{z}^{[2]}\ast a^{{}^{\prime }[1]} \\ d{w}^{[1]}=d{z}^{[1]}\cdot x^T \\ d{b}^{[1]}=d{z}^{[1]} \end{gathered}$$

dz^[1]的推导涉及到了矩阵求导

the dimension

$$\begin{gathered} x:(n_0,m)W^{[1]}:(n_1,n_0)\to \\ {a}^{[1]}:(n_1,m)W^{[2]:}:(n_2,n_1)\to \\ {a}^{[2]}:(n_2,m) \end{gathered}$$

vectorize

$$\begin{gathered} d{Z}^{[2]}=A^{[2]}-Y \\ d{W}^{[2]}=\frac{1}{m}d{Z}^{[2]}{A}^{[1]T} \\ d{b}^{[2]}=np.sum(d{Z}^{[2]},axis=1,keepdims=True) \\ d{Z}^{[1]}=W^{[2]T}d{Z}^{[2]}\ast A^{{}^{\prime }[1]} \\ d{W}^{[1]}=\frac{1}{m}d{Z}^{[1]}{X}^T \\ d{b}^{[1]}=\frac{1}{m}np.sum(dZ[1],axis=1,keepdims=True) \end{gathered}$$

Random Initialization

$$\begin{gathered} w^{[1]}=np.random.randn((2,2))\ast 0.01 \\ {b}^{[1]}=np.zero((2,1)) \end{gathered}$$

Deep neural network

notation

$$\begin{gathered} example:LlayerNN \\ {a}^{[l]}\to activationfunction \\ {w}^{[l]}\to weightsfor{z}^{[l]} \\ \hat{y}=a^{[L]} \end{gathered}$$

Forward propagation

$$\begin{gathered} forl=1,2,\mathrm{3..} \\ {z}^{[l]}=w^{[l]}{a}^{[l-1]}+b^{[l]} \\ cache{z}^{[l]},w^{[l]},b^{[l]} \\ {a}^{[l]}=g^{[l]}(z^{[l]}) \end{gathered}$$

Backward propagation

$$\begin{gathered} d{a}^{[l]}\to d{a}^{[l-1]}(d{z}^{[l]},d{w}^{[l]},d{b}^{[l]}) \\ d{z}^{[l]}=d{a}^{[l]}\ast g^{[l{]}^{\prime }}(z^{[l]})=w^{[l+1]}d{z}^{[l+1]}\ast g^{[l{]}^{\prime }}(z^{[l]}) \\ d{w}^{[l]}=d{z}^{[l]}\cdot a^{[l-1]T} \\ d{b}^{[l]}=d{z}^{[l]} \\ d{a}^{[l-1]}=w^{[l]T}\cdot d{z}^{[l]} \end{gathered}$$

matrix dimensions

$$\begin{gathered} dw,w^{[l]}:(n^{[l]},n^{[l-1]}) \\ db,b^{[l]}:(n^{[l]},1) \\ {Z}^{[l]},A^{[l]}:(n^{[l]},m) \end{gathered}$$