To address the challenge of autonomous vehicle decision-making and control at unsignalized intersections, this study investigates the merging behavior of two vehicles at a two-way single-lane intersection. Reinforcement learning is used to establish a mapping between the vehicle state space and action space for autonomous decision-making. To overcome the limitations of overly simplified speed settings in existing studies, real-world trajectory data of surrounding vehicles are used to construct an environmental traffic model. The autoregressive moving average (ARMA) model is applied to predict the speeds of surrounding vehicles. By integrating the predicted speed profiles with the autonomous vehicle’s motion parameters, a forward decision-making model is established to calculate reference speeds. These reference speeds are incorporated into the reinforcement learning reward function to accelerate training convergence. Experimental results show that the proposed model achieves rapid convergence, and the trained agent can safely navigate the intersection while interacting with surrounding vehicles exhibiting diverse driving behaviors. This work provides a reference framework for improving the safety and efficiency of autonomous vehicle decision-making at unsignalized intersections.