《数理统计》课程教学资源（参考资料）Large sample properties of MLE 04

63 Section 8.4.Example Let {(Ri,Xi):i=1,...,n}be an i.i.d.sample of n random vectors (R,X).Here R is a response indicator and X is a covariate.We assume that logitP[R 1X]=a+BX and assume that X is normally distributed with mean u and variance o2.So,our probability model has four parameters, 0=(a,B,u,o2).Let 0o =(ao,Bo,uo,denote the true value of 0

63 Section 8.4. Example Let {(Ri, Xi) : i = 1,...,n} be an i.i.d. sample of n random vectors (R, X). Here R is a response indicator and X is a covariate. We assume that logitP[R = 1|X] = α + βX and assume that X is normally distributed with mean µ and variance σ2. So, our probability model has four parameters, θ = (α, β, µ, σ2). Let θ0 = (α0, β0, µ0, σ20) denote the true value of θ

64 a.For a given realization of the data,write out the likelihood function of 0. 42-p2点g-An+

64 a. For a given realization of the data, write out the likelihood function of θ. L(θ; x, r) =
n i=1 1 σ√2π exp(− 12σ2 (xi − µ)2) exp(ri(α + βxi)) 1 + exp(α + βxi)

65 b.Find the maximum likelihood estimator of 0,0(If there exists a closed form solution,then present it.If not,indicate how a solution can be found). To find the MLE,solve the score equations.The log-likelihood for an individual is 1e:x,r）x-l1oga)-2ox-m2+r(a+m）-log(1+exp(a+x》 1 The score vector for an individual is al(0;x,r） r exp(a+Bx) 8a 1+exp(a+Bx) al(0;x,r) x(r- exp(a+Bx) b(x,T;0)= 83 1+exp(a+Bx)) ∂l(0;c,r x一业 ∂μ 8l(0;x,r） 8o2 动 2G4

65 b. Find the maximum likelihood estimator of θ, ˆ θn (If there exists a closed form solution, then present it. If not, indicate how a solution can be found). To find the MLE, solve the score equations. The log-likelihood for an individual is l(θ; x, r) ∝ − log(σ)− 12σ2 (x−µ)2 +r(α+βx)−log(1+ exp(α+βx)) The score vector for an individual is ψ(x, r; θ) = ⎡⎢ ⎢⎢⎢ ⎢⎣ ∂l(θ;x,r) ∂α ∂l(θ;x,r) ∂β ∂l(θ;x,r) ∂µ ∂l(θ;x,r) ∂σ2 ⎤⎥ ⎥⎥⎥ ⎥⎦ = ⎡⎢ ⎢⎢⎢ ⎢⎣ r − exp(α+βx) 1+exp(α+βx) x(r − exp(α+βx) 1+exp(α+βx) ) x−µ σ2 − 1 2σ2 + (x−µ)2 2σ4 ⎤⎥ ⎥⎥⎥ ⎥⎦

66 The score equations for the full sample are: m exp(a+i) ∑ exp(a+Bxi) =0 (1) i=1 -+=0 (2) i=1 x一业 =0 02 (3) i=1 n +0 2o4 (4) i=1 Note that Equations (3)and (4)can be solved explicitly to get solutions for a and 62,i.e., 在= 一 Ti ∑a:-创2 i=1 i=

66 The score equations for the full sample are:  n i=1 ri − exp(α + βxi) 1 + exp(α + βxi) = 0 (1)  n i=1 xi(ri − exp(α + βxi) 1 + exp(α + βxi)) = 0 (2)  n i=1 xi − µ σ2 = 0 (3)  n i=1 − 1 2σ2 + (xi − µ)2 2σ4 = 0 (4) Note that Equations (3) and (4) can be solved explicitly to get solutions for ˆµ and ˆσ2, i.e., µ ˆ = 1 n  n i=1 xi and ˆσ2 = 1 n  n i=1 (xi − µˆ)2

67 The solutions for a and B are obtained by solving Equations(1) and(2).This does not yield simple closed form solutions. Therefore,we use the Newton-Raphson algorithm.This entails computing the observed information matrix.Some of these computations will be needed later so let's compute the entire matrix of second partial derivatives.Let p() exp(a+Bxi) and qi(a,B)=1-pi(a,B),Now, 以物 where Jn1(a,)- ∑1p(a,3)g(a,3) ∑1cp(a,3)q(a,3) ∑1xp:(a,8)qa(a,3)∑=1p(a,3)q(a,)

67 The solutions for ˆα and β ˆ are obtained by solving Equations (1) and (2). This does not yield simple closed form solutions. Therefore, we use the Newton-Raphson algorithm. This entails computing the observed information matrix. Some of these computations will be needed later so let’s compute the entire matrix of second partial derivatives. Let pi(α, β) = exp(α+βxi) 1+exp(α+βxi) and qi(α, β)=1 − pi(α, β), Now, nJn(θ) = ⎡⎣ Jn1(α, β) 0 0 Jn2(µ, σ2) ⎤⎦ where Jn1(α, β) = ⎡⎣ ni=1 pi(α, β)qi(α, β) ni=1 xipi(α, β)qi(α, β) ni=1 xipi(α, β)qi(α, β) ni=1 x2i pi(α, β)qi(α, β) ⎤⎦

68 and 六∑1(r-川） To compute thethe solution to (1)and(2),we start with initial values for a and B,say a(0)and B(0),then update as follows: -）aw ∑=1(ri-p(a),gG) ∑1x(r:-p(a6),3) We iterate until convergence

68 and Jn2(µ, σ2) = ⎡⎣ nσ2 1σ4 ni=1(xi − µ) 1σ4 ni=1(xi − µ) − n2σ4 + 1σ6 ni=1(xi − µ)2 ⎤⎦ To compute the the solution to (1) and (2), we start with initial values for α and β, say α(0) and β(0), then update as follows: ￾
α(j+1) β(j+1) = ￾
α(j) β(j) +{Jn1(α(j), β(j))}−1·￾
ni=1(ri − pi(α(j), β(j)) ni=1 xi(ri − pi(α(j), β(j)) We iterate until convergence.

69 Assume that la≤K1,ll≤K2,luW≤K3,andK4≤o2≤K5, where K1,...,K5 are finite positive quantities.Is the parameter space,e,compact?YES!Assume that 0o (truth)belongs to the interior of c.Verify all of the regularity conditions that are necessary to show that the MLE is both consistent and asymptotically normal. Refer to Section 8c.Condition i)is satisfied by assumption. Condition ii)is satisfied since logp(x,r;0)is continuous at each for all -oolog(2z)+llog(o)l+lzc2(-p)2|+lr(@+Bz)l+llog(1+exp(@+Bz)l

69 Assume that |α| ≤ K1, |β| ≤ K2, |µ| ≤ K3, and K4 ≤ σ2 ≤ K5, where K1,...,K5 are finite positive quantities. Is the parameter space, Θ, compact? YES! Assume that θ0 (truth) belongs to the interior of Θ. c. Verify all of the regularity conditions that are necessary to show that the MLE is both consistent and asymptotically normal. Refer to Section 8c. Condition i) is satisfied by assumption. Condition ii) is satisfied since log p(x, r; θ) is continuous at each θ for all −∞ <x< ∞ and r = 0, 1. Now, we need to show condition iii) that | log p(x, r; θ)| ≤ d(x, r) for all θ ∈ Θ and Eθ0 [d(X, R)] < ∞. log p(x, r; θ) = −12 log(2π)−log(σ)− 12σ2 (x−µ)2+r(α+βx)−log(1+exp(α+βx)) By the triangle inequality, we know that | log p(x, r; θ)| ≤ 12 log(2π)+| log(σ)|+| 12σ2 (x−µ)2|+|r(α+βx)|+| log(1+exp(α+βx))|

70

70

71 Some inequalities that we will need. |zl≤1+x2 log(1+exp(a+Bx)≥0 log(1+exp(a+Bx))<log(1+exp(la+)) ≤1+log(exp(Ia+lxl) 1+a+8xl ≤1+K1+K2(1+x2) log(ol≤号max(log(Kal,|log(KlD

71 Some inequalities that we will need. |x| ≤ 1 + x2 log(1 + exp(α + βx)) ≥ 0 log(1 + exp(α + βx)) ≤ log(1 + exp(|α| + |β||x|)) ≤ 1 + log(exp(|α| + |β||x|)) = 1+ |α| + |β||x| ≤ 1 + K1 + K2(1 + x2) | log(σ)| ≤ 12 max(| log(K4)|, | log(K5)|)

72 -≤2++ 202 云+++ x2, ≤ lr(a+Bx川≤la+l3lxl≤K1+K2(1+x2) 1log(1+exp(a+Bx)川≤1+K1+K2(1+x2) Let d(,)log()m(log()o 宏+++器++n++ 2K4+E 1+K1+K2(1+x2) Now,E0o[X2]=u+o<oo.This implies that E0o[d(X,R)]<oo. For condition iv),we have to show that p(x,r;0)is

72 | 12σ2 (x − µ)2| ≤ 12σ2 x2 + 1σ2 |µ||x| + µ2 2σ2 ≤ x2 2K4 + K3 K4 (1 + x2) + K23 2K4 |r(α + βx)|≤|α| + |β||x| ≤ K1 + K2(1 + x2) | log(1 + exp(α + βx))| ≤ 1 + K1 + K2(1 + x2) Let d(x, r) = 12 log(2π) + 12 max(| log(K4)|, | log(K5)|) + x2 2K4 + K3 K4 (1 + x2) + K23 2K4 + K1 + K2(1 + x2) + 1 + K1 + K2(1 + x2) Now, Eθ0 [X2] = µ20 + σ20 < ∞. This implies that Eθ0 [d(X, R)] < ∞. For condition iv), we have to show that p(x, r; θ) is