To explore the importance of different input features in load forecasting, effectively handle the linear and nonlinear components in load data, and improve the accuracy of load prediction, a combined load prediction model based on TabNet and LSTNet (long and short-term temporal networks) is proposed in this paper. First, the prediction accuracy of TabNet is improved by introducing self-supervised pre-training, and then the global importance of the input features and the prediction results are obtained by training. Next, the features with high importance are input to LSTNet, which is trained to obtain the prediction results. Finally, the prediction results of the combined model are derived using the variance-covariance combination method. Simulation analysis shows that the proposed combined model has higher accuracy compared with traditional LSTM (long and short-term memeory), Xgboost (extreme gradient boost), Lightgbm (lignt gradient boosting machine) and other combined models.