Patch prior model has achieved great success in image restoration,but due to the enforced locality of the overall model, it tends to exhibit local artifacts and poor visual perception. A new image restoration method integrating multi-scale patch prior and histogram prior is proposed in this paper. The original image was filtered and down-sampled to maintain the scale invariance, and the same patch local model was applied on multiple-scale images,by which the simplicity of the low-dimensional patch model was maintained and the non-locality implemented in a large area of the image. The histogram global statistical prior was added to the regular constraint, and the similarity between the restored image and the reference histogram was measured by the Wasserstein distance. And the proposed model was solved by the theory of half quadratic splitting and optimal transfer. The effectiveness of the proposed model is verified by experiments of image denoising and deblurring in that our model has advantage over traditional methods in terms of both objective quality assessments and visual perception.