We demonstrate how by combining the notion of modular eigenspaces with multiple keyframes, an image-alignment based tracking algorithm can be made to accommodate large variations in 3D pose. Since tracking without automatic pose initialization is practically useless, a second main contribution of our work is to demonstrate how local appearance based feature matching can be effectively used to augment the tracking algorithm for pose initialization. The initialization algorithm kicks in automatically whenever the tracking algorithm fails because there is a large and sudden variation in object pose. Our method for 3D pose refinement during object tracking extends the work of Cobzas et al. For tracking, we represent an object model by a set of 3D planar patches that approximate the boundary surface of the object. Associated with each patch is its appearance model that consists of a collection of eigenimage sets. Each eigenimage set is obtained from a keyframe image and the images recorded from the nearby viewpoints. Our work includes an automated procedure for keyframe identification from a training video sequence. With 640x480 images, the tracking algorithm takes roughly 0.3 to 1.0 second/frame on a 3 GHz Pentium-IV machine and pose initialization takes less than 1 second on the same processor.