There exist potential problems in current region-based image retrievals. This paper proposes a novel approach to object semanteme based image segmentation and classification. First, a hierarchical region growing image segmentation is established using HRGSeg algorithm, which can effectively get rid of weak object semantemes and play down the side effect of over-segmentation. Based on it, low-level features like color, edge and texture are extracted mapped into high-level object semantics hierarchically by using SVM. A fairly good experiment result is achieved and shows the feasibility of our approach.