《图像处理与计算机视觉 Image Processing and Computer Vision》课程教学资源（PPT课件讲稿）Chapter 11 Bundle adjustment Structure reconstruction SFM from N-frames

Image processing and Computer Vision Chapter 11: Bundle adjustment Structure reconstruction sfm from N-frames Pose estimation vo.a

Image Processing and Computer Vision Chapter 11: Bundle adjustment Structure reconstruction SFM from N-frames Pose estimation V0.a 1

Reconstruction from n-frames Factorization(inear, fast, not too accurate) Bundle adjustment (slower but more accurate) can use factorization results as the first guess Non linear iterative methods are more accurate han linear method require first guess (e.g. from factorization) Many different implementations but the concept is the same Pose estimation vo.a

Reconstruction from N-frames • Factorization (linear, fast, not too accurate) • Bundle adjustment (slower but more accurate), can use factorization results as the first guess. – Non linear iterative methods are more accurate than linear method, require first guess (e.g. From factorization). – Many different implementations, but the concept is the same. Pose estimation V0.a 2

Problem definition There are n features in the 3d object We takei pictures of the object at different VIews Input · Image sequence I2…lr Each image has n image feature points Output (structure=model, and motion=pose) 3-d coordinates of all 3-d model points x,x.X N Camera pose for each image taken [r()t(t)t=l, .T Pose estimation vo.a

Problem definition • There are N features in the 3D object . • We take  pictures of the object at different views. • Input : • Image sequence I1 ,I2 ,…I  . • Each image has n image feature points • Output (structure=model, and motion=pose) • 3-D coordinates of all 3-D model points X1 ,X2 ,..,XN . • Camera pose for each image taken [R(t),T(t)] t=1,…  Pose estimation V0.a 3

Example: Bundle adjustment 3D reconstruction (see also http://www.cse.cuhk.edu.hk/khwong/demo/index.html) grand canyon demo lask · robot http://www.youtube.com/watch?v=2klfrillOjc http://www.youtube.com/watch?v=xgcnv--wf2k http://www.youtube.com/watch?v=onx4cyyyyri http://www.youtube.com/watch?v=4h1pn2dis6g Pose estimation vo.a

Example: Bundle adjustment 3D reconstruction (see also http://www.cse.cuhk.edu.hk/khwong/demo/index.html) • Grand Canyon Demo • Flask • Robot Pose estimation V0.a 4 http://www.youtube.com/watch?v=2KLFRILlOjc http://www.youtube.com/watch?v=4h1pN2DIs6g http://www.youtube.com/watch?v=ONx4cyYYyrI http://www.youtube.com/watch?v=xgCnV--wf2k

The iterative Sfm alternating bundle adjustment method Break down the system into two phases SFM1: find pose phase SFM2: find model phase Initialize first guess of model The first guess is a flat model perpendicular to the image and is Zinit away e.g. Zinit =0.5 meters or any reasonable guess Iterative while( Err is not small SFM1: find pose phase SFM2: find model phase Measurement error(Err)or(model and pose stabilized) Pose estimation vo.a

The iterative SFM alternating bundle adjustment method • Break down the system into two phases: --SFM1: find pose phase --SFM2: find model phase • Initialize first guess of model – The first guess is a flat model perpendicular to the image and is Zinit away (e.g. Zinit = 0.5 meters or any reasonable guess) • Iterative while ( Err is not small ) • { – SFM1: find pose phase – SFM2: find model phase – Measurement error(Err) or(model and pose stabilized) • } Pose estimation V0.a 5

SFM1: find pose phase Pose estimation discussed in the last chapter Pose estimation vo.a 6

SFM1 : find pose phase Pose estimation discussed in the last chapter Pose estimation V0.a 6

SFM2: Model finding by the iterative method Similar to pose estimation In pose estimation model is known, pose is unknown Here model finding by the iterative method) Assume pose is known model is unknown The algorithms are similar Pose estimation vo.a

SFM2: Model finding by the iterative method • Similar to pose estimation. – In pose estimation: model is known, pose is unknown. – Here (Model finding by the iterative method) Assume pose is known, model is unknown. – The algorithms are similar. Pose estimation V0.a 7

From the pose estimation slides (chapter iv 10-pose estimation) Problem f= focal length. Assume for pose{RT}θ=[1a2@TT写 Formulation T, T,, T, are translations along,Y, Z axes repectively (@i, a )are rotation angles around X, Y, Z axes respectively, M,=x, y zI is the i model point(i =1, 2, N), *=u, v] the image point of P, rewrite(la),1(b) F1X1+n121+1321+T1 与X,+12+h32+F,(,M) -(2a) F21X1+2+r2321+72 v;h+r2y+n32+r8,(O,M) -(2b In this chapter Problem definition: There are t=1, 2, Iimage frames, and the object i=1, 2,.N features Asume the pose 0,=1.2.r=/91,92,B3, T,T2,T]i and image measurements xi. v,I are known SFM2 task: From the given image frames and pose 8, 12r Find Ml2N=x, Y ZP Pose estimation vo.a

Problem Formulation • Pose estimation V0.a 8     ( )         i N T i N i i i t , ,..,Γ T i,t i i t T t , ,..,Γ t v i i i i i i i i i u i i i i i i i i i i T i i i T th i i i i T M X Y Z θ x u v θ [ , , ,T ,T ,T ] g M b r X r Y r Z T r X r Y r Z T v f g M a r X r Y r Z T r X r Y r Z T u f P M X Y Z i x u v T T T f R,T θ T T T 1,2,.. 1,2,.. 1 2 1 2 1 2 3 1 2 3 , 31 32 33 3 21 22 23 2 , 31 32 33 3 11 12 13 1 1 2 3 1 2 3 1 2 3 1 2 3 Find SFM2 task :From the given image frames and pose and image measurements are known. Asume the pose , the object i 1,2,..., N features Problem definition :There are t 1,2,, image frames, and ---------------In this chapter -------------------------- ( , ) (2 ) ( , ) (2 ) the image point of , rewrite (1a),1(b) is the model point (i 1,2,, N), is , , are rotation angles around X,Y,Z axesrespectively, , , are translations along X,Y,Z axesrepectively, focal length.Assume for pose From the pose estimation slides(chapter iv10 - pose estimation): = = = = = = = = =  = − − − − − − − − + + + + + + = = − − − − − − − − + + + + + + = = = = = =           

Derivation for the model partial derivatives(note f= focal length en=g,(9,,,,T,T,x,,2)= Given u=81(,,,7,7,T,Xx,F,2)= D2z +T x+1+2+=2/-x++乙++n=若=厂 X1-3Y1+中21+7 r31 r321+73 n2X1+1+21+73 112 1 1≈1,2≈-g3,1≈2 R=r 1 31≈-n2,32≈内, Find I Find aX F1X1+F21+132+7 F1X1+F2x1+13z1+T r31X1+ry21+32z1+73 r,X+PY+、Z+T Y aX aX aX (xA)1 X ar aX aX o(x2) a(rr )f az ay ar Z aZ. aX OX Z X+r (z X (z) ZR ation vo.a

Derivation for the model partial derivatives (Note: f= focal length) • Pose estimation V0.a 9 ( ) ( ) ( ) ( ) ( )         = − +         = − +         = −   +           =   +         =         =        + + + + + +  =               − − −            = − + + + − + + =  + + + + + + = = = 2 1 1 3 1 1 1 2 3 1 2 3 1 1 1 3 1 3 2 3 3 3 1 1 1 2 1 3 1 2 1 3 1 3 2 3 1 3 2 3 3 2 1 2 2 2 3 1 1 1 2 1 3 2 1 3 3 2 1 3 1 3 2 3 3 3 1 1 1 2 1 3 1 1 2 3 1 2 3 1 1 Find . 1 1 1 R Given ( , , , , , , , , ) R i i R i R i R i R i R i R i R i R i R R i i R i R i R i R i R R i i R i R i R i i i i i i i i i i i i i i i R i R i i i i i i i i u i i i Z r X Z r f Z r f Z r X f Z f r X r Z f Z f X X X X Z Z Z f X Z f X X X Z f X Z X f X r X r Y r Z T r X r Y r Z T f X u X u r r r r r r r r r X Y Z T X Y Z T f Z X f r X r Y r Z T r X r Y r Z T f u g T T T X Y Z              ( ) ( ) ( ) ( ) ( )         = − +         = − +         = −   +           =   +         =         =        + + + + + +  =      −     −  − + + + − + + =  + + + + + + = = = 2 1 2 3 2 1 2 2 3 2 2 3 2 1 2 3 1 3 2 3 3 3 1 1 1 2 1 3 1 3 1 2 3 2 1 3 3 1 1 1 2 3 1 3 2 2 1 3 3 2 1 3 1 3 2 3 3 3 1 1 1 2 1 3 1 1 2 3 1 2 3 1 1 Find . , , 1 so 1, , Given ( , , , , , , , , ) R i i R i R i R i R i R i R i R i R i R R i i R i R i R i R i R R i i R i R i R i i i i i i i i i i i i i i i R i R i i i i i i i i u i i i Z r X Z r f Z r f Z r X f Z f r X r Z f Z f Y X X Y Z Z Z f Y Z f X X Y Z f Y Z X f Y r X r Y r Z T r X r Y r Z T f Y u Y u r r r r r r X Y Z T X Y Z T f Z X f r X r Y r Z T r X r Y r Z T f u g T T T X Y Z           

CMSC5711: Exercise 11. 1: (Note: f= focal length) Given u;=gu(@1, P2,23,Ti, T2, T3,Xi,Y, Zi) F1X1+121+F1321+7 R f≈f B3Y+o2zi+ rX.tr.t +t 33 2X1+ny1+Z1+ Son1≈12≈-的,73 r31≈-n22≈,3 ≈1.Or 2 m的 R 33 21 Find Xtr.trz+t X R Xtr.trz R 33 +T, aZ aZ aZ Pose estimation vo.a

CMSC5711: Exercise11.1: (Note: f= focal length) • Pose estimation V0.a 10 ? Find 1 1 1 R , , 1, so 1, , Given ( , , , , , , , , ) 3 1 3 2 3 3 3 1 1 1 2 1 3 1 2 1 3 1 3 2 3 1 3 2 3 3 2 1 2 2 2 3 1 1 1 2 1 3 3 1 2 3 2 1 3 3 1 1 1 2 3 1 3 2 2 1 3 3 2 1 3 1 3 2 3 3 3 1 1 1 2 1 3 1 1 2 3 1 2 3 =           =          + + + + + +  =             − − −            =  −     −  − + + + − + + =  + + + + + + = = = Z Z X f Z r X r Y r Z T r X r Y r Z T f Z u r r r r r r r r r r r r or r r r X Y Z T X Y Z T f Z X f r X r Y r Z T r X r Y r Z T f u g T T T X Y Z R i R i i i i i i i i i i i i i i R i R i i i i i i i i u i i i                 