It is very essential to extract representative frames in the process of generating video summary. A method is proposed which analyses the color features of the video frames, sets the connectivity threshold values automatically according to the contents, extracts the color coherence vectors (CCV) and then performs adaptive clustering based on equivalent relation. After global partition, local partition was revised with time sequential features. The whole process does not need to set any threshold values. The experiments with diverse videos yields effective results.