Motion estimation is the most complex part in H.264. Intra prediction is an important part of motion estimation. There are a number of intra prediction modes in H.264, which no doubt greatly increase encoding complexity. In intra prediction, the way of choosing prediction method is rate distortion optimization or using correlation properties. For complexity problem, the paper provides an improved intra prediction mode selection algorithm. This algorithm uses a new detection method to detect the pixel correlation in the direction of mode selection. Test results show that the algorithm has reduced the computational complexity and increased the encoding speed highly in the premise of ensuring video quality.