Abstract: A novel method of tracking a moving object in video sequences was presented. Adaptive background updating based on dynamic information window was used to solve the problem of complicated background, and a novel color model was used to dispel motion shadow. A given moving object with accurate edge was obtained, The centroid of the moving object was calculated and the coordinates were established. The method of 2-stage least squares was adopted to implement tracking the moving object. The results showed that the presented method was promising to track and forecast the motion of a moving object in video sequences.