Isotropic and anisotropic porous medium with the same porosity, different pore size distributions and pore surface fractal dimensions were constructed by random growth method, and the impact of microscopic pore structure on moisture migration process in pores of porous medium was theoretically studied. The results show that the internal transport properties of porous medium are not only determinded by porosity, but also greatly influenced by pore size distribution and pore surface fractal dimension. The biggest water vapor diffusion concentration differences of three kinds of isotropic and three kinds of anisotropic porous medium with the same porosity, different pore size distributions and pore surface fractal dimensions are 30.6% and 23.3%, respectively. Better water vapor diffusion properties in pores of isotropic porous medium demands for bigger tiny pore size distribution rate and larger pore surface fractal dimension, which means the pore connectivity is better. While better water vapor diffusion properties in pores of anisotropic porous medium(water vapor diffusion direction is perpendicular to the direction of larger growth rate of anisotropic porous medium) demands for more big pore size distribution and smaller pore surface fractal dimension. The method lays a theoretical foundation for preparing indoor humidity control of building materials.