### Discover more content...

Enter some keywords in the search box above, we will do our best to offer you relevant results.

### We're sorry!

We couldn't find any results for your search. Please try again with another keywords.

## 一. Normal Equation

### 1. 正规方程

$\vec{x^{(i)}} = \begin{bmatrix} x^{(i)}{0}\ x^{(i)}{1}\ \vdots \ x^{(i)}_{n}\ \end{bmatrix}$

$\vec{x^{(i)}}$ 是这样一个 $(n+1)*1$ 维向量。每行都对应着 i 行 0-n 个变量。

$X = \begin{bmatrix} (\vec{x^{(1)}})^{T}\ \vdots \ (\vec{x^{(m)}})^{T} \end{bmatrix} ;;;; \Theta = \begin{bmatrix} \theta_{0}\ \theta_{1}\ \vdots \ \theta_{n}\ \end{bmatrix} ;;;; Y = \begin{bmatrix} y^{(1)}\ y^{(2)}\ \vdots \ y^{(m)}\ \end{bmatrix}$

X 是一个 $m * (n+1)$ 的矩阵，$\Theta$ 是一个 $(n+1) * 1$ 的向量，Y 是一个 $m * 1$的矩阵。

\begin{align*} X \cdot \Theta - Y = \begin{bmatrix} (\vec{x^{(1)}})^{T}\ \vdots \ (\vec{x^{(m)}})^{T} \end{bmatrix} \cdot \begin{bmatrix} \theta_{0}\ \theta_{1}\ \vdots \ \theta_{n}\ \end{bmatrix} - \begin{bmatrix} y^{(1)}\ y^{(2)}\ \vdots \ y^{(m)}\ \end{bmatrix} = \begin{bmatrix} h_{\theta}(x^{(1)})-y^{(1)}\ h_{\theta}(x^{(2)})-y^{(2)}\ \vdots \ h_{\theta}(x^{(m)})-y^{(m)}\ \end{bmatrix} \end{align*}

\begin{align*} \rm{CostFunction} = \rm{F}({\theta_{0}},{\theta_{1}}) &= \frac{1}{2m}\sum_{i = 1}^{m} (h_{\theta}(x^{(i)})-y^{(i)})^2\ & = \frac{1}{2m} (X \cdot \Theta - Y)^{T}(X \cdot \Theta - Y)\ \end{align*}

### 2. 矩阵的微分和矩阵的迹

$\rm{tr} A = \sum_{i=1}^{n}A_{ii}$

1. $\rm{tr};a = a$ ， a 是标量 ( $a \in \mathbb{R}$)

2. $\rm{tr};AB = \rm{tr};BA$ 更近一步 $\rm{tr};ABC = \rm{tr};CAB = \rm{tr};BCA$
证明：假设 A 是 $n * m$ 矩阵， B 是 $m * n$ 矩阵，则有
$\rm{tr};AB = \sum_{i=1}^{n}\sum_{j=1}^{m}A_{ij}B_{ji} = \sum_{j=1}^{n} \sum_{i=1}^{m}B_{ji}A_{ij}= \rm{tr};BA$
同理：$\rm{tr};ABC = \rm{tr};(AB)C = \rm{tr};C(AB) = \rm{tr};CAB$
$\rm{tr};ABC = \rm{tr};A(BC) = \rm{tr};(BC)A = \rm{tr};BCA$
连起来，即 $\rm{tr};ABC = \rm{tr};CAB = \rm{tr};BCA$

3. $\triangledown_{A}\rm{tr};AB = \triangledown_{A}\rm{tr};BA = B^{T}$
证明：按照矩阵梯度的定义：
$\triangledown_{X}f(X) = \begin{bmatrix} \frac{\partial f(X) }{\partial x_{11}} & \cdots & \frac{\partial f(X) }{\partial x_{1n}}\ \vdots & \ddots & \vdots \ \frac{\partial f(X) }{\partial x_{m1}} & \cdots & \frac{\partial f(X) }{\partial x_{mn}} \end{bmatrix} = \frac{\partial f(X) }{\partial X}$
假设 A 是 $n * m$ 矩阵， B 是 $m * n$ 矩阵，则有
\begin{align*}\triangledown_{A}\rm{tr};AB &= \triangledown_{A} \sum_{i=1}^{n}\sum_{j=1}^{m}A_{ij}B_{ji} = \frac{\partial}{\partial A}(\sum_{i=1}^{n}\sum_{j=1}^{m}A_{ij}B_{ji})\ & = \begin{bmatrix} \frac{\partial}{\partial A_{11}}(\sum_{i=1}^{n}\sum_{j=1}^{m}A_{ij}B_{ji}) & \cdots & \frac{\partial}{\partial A_{1m}}(\sum_{i=1}^{n}\sum_{j=1}^{m}A_{ij}B_{ji})\ \vdots & \ddots & \vdots \ \frac{\partial}{\partial A_{n1}}(\sum_{i=1}^{n}\sum_{j=1}^{m}A_{ij}B_{ji}) & \cdots & \frac{\partial}{\partial A_{nm}}(\sum_{i=1}^{n}\sum_{j=1}^{m}A_{ij}B_{ji}) \end{bmatrix} \ & = \begin{bmatrix} B_{11} & \cdots & B_{m1} \ \vdots & \ddots & \vdots \ B_{1n} & \cdots & B_{mn} \end{bmatrix} = B^{T}\ \end{align*}

\begin{align*}\triangledown_{A}\rm{tr};BA &= \triangledown_{A} \sum_{i=1}^{m}\sum_{j=1}^{n}B_{ij}A_{ji} = \frac{\partial}{\partial A}(\sum_{i=1}^{m}\sum_{j=1}^{n}B_{ij}A_{ji})\ & = \begin{bmatrix} \frac{\partial}{\partial A_{11}}(\sum_{i=1}^{m}\sum_{j=1}^{n}B_{ij}A_{ji}) & \cdots & \frac{\partial}{\partial A_{1m}}(\sum_{i=1}^{m}\sum_{j=1}^{n}B_{ij}A_{ji})\ \vdots & \ddots & \vdots \ \frac{\partial}{\partial A_{n1}}(\sum_{i=1}^{m}\sum_{j=1}^{n}B_{ij}A_{ji}) & \cdots & \frac{\partial}{\partial A_{nm}}(\sum_{i=1}^{m}\sum_{j=1}^{n}B_{ij}A_{ji}) \end{bmatrix} \ & = \begin{bmatrix} B_{11} & \cdots & B_{m1} \ \vdots & \ddots & \vdots \ B_{1n} & \cdots & B_{mn} \end{bmatrix} = B^{T}\ \end{align*}

所以有 $\triangledown_{A}\rm{tr};AB = \triangledown_{A}\rm{tr};BA = B^{T}$

4. $\triangledown_{A^{T}}a = (\triangledown_{A}a)^{T};;;; (a \in \mathbb{R})$
证明：假设 A 是 $n * m$ 矩阵
\begin{align*}\triangledown_{A^{T}}a & = \begin{bmatrix} \frac{\partial}{\partial A_{11}}a & \cdots & \frac{\partial}{\partial A_{1n}}a\ \vdots & \ddots & \vdots \ \frac{\partial}{\partial A_{m1}}a & \cdots & \frac{\partial}{\partial A_{mn}}a \end{bmatrix} = (\begin{bmatrix} \frac{\partial}{\partial A_{11}}a & \cdots & \frac{\partial}{\partial A_{1m}}a\ \vdots & \ddots & \vdots \ \frac{\partial}{\partial A_{n1}}a & \cdots & \frac{\partial}{\partial A_{nm}}a \end{bmatrix})^{T} \ & = (\triangledown_{A}a)^{T}\ \end{align*}

5. $\mathrm{d}(\rm{tr};A) = \rm{tr}(\mathrm{d}A)$
证明：
$\mathrm{d}(\rm{tr};A) = \mathrm{d}(\sum_{i=1}^{n}a_{ii}) = \sum_{i=1}^{n}\mathrm{d}a_{ii} = \rm{tr}(\mathrm{d}A)$
矩阵的迹的微分等于矩阵的微分的迹。

6. $\triangledown_{A}\rm{tr};ABA^{T}C = CAB + C^{T}AB^{T}$
证明：
根据实标量函数梯度的乘法法则：
若 f(A)、g(A)、h(A) 分别是矩阵 A 的实标量函数，则有
\begin{align*}\frac{\partial f(A)g(A)}{\partial A} &= g(A)\frac{\partial f(A)}{\partial A} + f(A)\frac{\partial g(A)}{\partial A}\ \frac{\partial f(A)g(A)h(A)}{\partial A} &= g(A)h(A)\frac{\partial f(A)}{\partial A} + f(A)h(A)\frac{\partial g(A)}{\partial A}+ f(A)g(A)\frac{\partial h(A)}{\partial A}\ \end{align*}
令 $f(A) = AB,g(A) = A^{T}C$，由性质5，矩阵的迹的微分等于矩阵的微分的迹，那么则有：
\begin{align*} \triangledown_{A}\rm{tr};ABA^{T}C & = \rm{tr}(\triangledown_{A}ABA^{T}C) = \rm{tr}(\triangledown_{A}f(A)g(A)) = \rm{tr}\triangledown_{A_{1}}(A_{1}BA^{T}C) + \rm{tr}\triangledown_{A_{2}}(ABA_{2}^{T}C) \ & = (BA^{T}C)^{T} + \rm{tr}\triangledown_{A_{2}}(ABA_{2}^{T}C) = C^{T}AB^{T} + \triangledown_{A_{2}}\rm{tr}(ABA_{2}^{T}C)\ & = C^{T}AB^{T} + \triangledown_{A_{2}}\rm{tr}(A_{2}^{T}CAB) = C^{T}AB^{T} + (\triangledown_{{A_{2}}^{T}};\rm{tr};A_{2}^{T}CAB)^{T} \ & = C^{T}AB^{T} + ((CAB)^{T})^{T} \ & = C^{T}AB^{T} + CAB \ \end{align*}

### 3. 推导

$\rm{CostFunction} = \rm{F}({\theta_{0}},{\theta_{1}}) = \frac{1}{2m} (X \cdot \Theta - Y)^{T}(X \cdot \Theta - Y)$

\begin{align*} \triangledown_{\theta}\rm{F}(\theta) & = \frac{1}{2m} \triangledown_{\theta}(X \cdot \Theta - Y)^{T}(X \cdot \Theta - Y) = \frac{1}{2m}\triangledown_{\theta}(\Theta^{T}X^{T}-Y^{T})(X\Theta-Y)\ & = \frac{1}{2m}\triangledown_{\theta}(\Theta^{T}X^{T}X\Theta-Y^{T}X\Theta-\Theta^{T}X^{T}Y+Y^{T}Y) \ \end{align*}

\begin{align*} \triangledown_{\theta}\rm{F}(\theta) & = \frac{1}{2m}\triangledown_{\theta}(\Theta^{T}X^{T}X\Theta-2Y^{T}X\Theta) \ & = \frac{1}{2m}\triangledown_{\theta}\rm{tr};(\Theta^{T}X^{T}X\Theta-2Y^{T}X\Theta) \ & = \frac{1}{2m}\triangledown_{\theta}\rm{tr};(\Theta\Theta^{T}X^{T}X-2Y^{T}X\Theta) \ & = \frac{1}{m}(\frac{1}{2}\triangledown_{\theta}\rm{tr};\Theta\Theta^{T}X^{T}X -\triangledown_{\theta}\rm{tr};Y^{T}X\Theta) \ & = \frac{1}{m}(\frac{1}{2}\triangledown_{\theta}\rm{tr};\Theta\Theta^{T}X^{T}X -(Y^{T}X)^{T}) = \frac{1}{m}(\frac{1}{2}\triangledown_{\theta}\rm{tr};\Theta\Theta^{T}X^{T}X -X^{T}Y)\ \end{align*}

\begin{align*} \triangledown_{\theta}\rm{F}(\theta) & = \frac{1}{m}(\frac{1}{2}\triangledown_{\theta}\rm{tr};\Theta\Theta^{T}X^{T}X -X^{T}Y) \ &= \frac{1}{m}(\frac{1}{2}\triangledown_{\theta}\rm{tr};\Theta I \Theta^{T}X^{T}X -X^{T}Y) \end{align*}

\begin{align*} \triangledown_{\theta}\rm{F}(\theta) &= \frac{1}{m}(\frac{1}{2}\triangledown_{\theta}\rm{tr};\Theta I \Theta^{T}X^{T}X -X^{T}Y) \ & = \frac{1}{m}(\frac{1}{2}(X^{T}X\Theta I + (X^{T}X)^{T}\Theta I^{T}) -X^{T}Y) \ & = \frac{1}{m}(\frac{1}{2}(X^{T}X\Theta I + (X^{T}X)^{T}\Theta I^{T}) -X^{T}Y) \ & = \frac{1}{m}(\frac{1}{2}(X^{T}X\Theta + X^{T}X\Theta) -X^{T}Y) = \frac{1}{m}(X^{T}X\Theta -X^{T}Y) \ \end{align*}

## 二. Normal Equation Noninvertibility

1. 多余的特征。特征之间呈倍数关系，线性依赖。
2. 过多的特征。当 $m \leqslant n$ 的时候，会导致过多的特征。解决办法是删除一些特征，或者进行正则化。

1. 删掉多余的特征，线性相关的，倍数关系的。直到没有多余的特征
2. 再删除一些不影响结果的特征，或者进行正则化。

## 三. Linear Regression with Multiple Variables 测试

### 1. Question 1

Suppose m=4 students have taken some class, and the class had a midterm exam and a final exam. You have collected a dataset of their scores on the two exams, which is as follows:

midterm exam (midterm exam)2 final exam
89 7921 96
72 5184 74
94 8836 87
69 4761 78
You'd like to use polynomial regression to predict a student's final exam score from their midterm exam score. Concretely, suppose you want to fit a model of the form hθ(x)=θ0+θ1x1+θ2x2, where x1 is the midterm score and x2 is (midterm score)2. Further, you plan to use both feature scaling (dividing by the "max-min", or range, of a feature) and mean normalization.

What is the normalized feature x(2)2? (Hint: midterm = 72, final = 74 is training example 2.) Please round off your answer to two decimal places and enter in the text box below.

### 2. Question 2

You run gradient descent for 15 iterations

with α=0.3 and compute J(θ) after each

iteration. You find that the value of J(θ) increases over

time. Based on this, which of the following conclusions seems

most plausible?

A. Rather than use the current value of α, it'd be more promising to try a smaller value of α (say α=0.1).

B. α=0.3 is an effective choice of learning rate.

C. Rather than use the current value of α, it'd be more promising to try a larger value of α (say α=1.0).

### 3. Question 3

Suppose you have m=28 training examples with n=4 features (excluding the additional all-ones feature for the intercept term, which you should add). The normal equation is θ=(XTX)−1XTy. For the given values of m and n, what are the dimensions of θ, X, and y in this equation?

A. X is 28×4, y is 28×1, θ is 4×4

B. X is 28×5, y is 28×5, θ is 5×5

C. X is 28×5, y is 28×1, θ is 5×1

D. X is 28×4, y is 28×1, θ is4×1

### 4. Question 4

Suppose you have a dataset with m=50 examples and n=15 features for each example. You want to use multivariate linear regression to fit the parameters θ to our data. Should you prefer gradient descent or the normal equation?

A. Gradient descent, since it will always converge to the optimal θ.

B. Gradient descent, since (XTX)−1 will be very slow to compute in the normal equation.

C. The normal equation, since it provides an efficient way to directly find the solution.

D. The normal equation, since gradient descent might be unable to find the optimal θ.

### 5. Question 5

Which of the following are reasons for using feature scaling?

A. It prevents the matrix XTX (used in the normal equation) from being non-invertable (singular/degenerate).

B. It is necessary to prevent the normal equation from getting stuck in local optima.

C. It speeds up gradient descent by making it require fewer iterations to get to a good solution.

D. It speeds up gradient descent by making each iteration of gradient descent less expensive to compute.

normal equation 不需要 Feature Scaling，排除AB， 特征缩放减少迭代数量，加快梯度下降，然而不能防止梯度下降陷入局部最优。

GitHub Repo：Halfrost-Field

Follow: halfrost · GitHub

Previous Post

Next Post