Taking the Haerwusu open-pit coal mine as a case study, this research addresses the challenge of achieving a high design capacity of 35 Mt/a under the constraints of shortened working line length and reduced mining district width, following the retraction of the first mining district’s working line into the mining license boundary. Based on principles of technical feasibility and economic rationality, and with considering the influence of working line length on production stripping ratio, stripping haulage distance, and raw coal production capacity, a mathematical model was established to minimize the annual total stripping cost. The economically optimal working line length was determined to range from 1 620 m to 2 315 m. Building upon this, and with integrating the production capacity and equipment configuration of existing mining equipment at Haerwusu, it was determined that the average annual advancing speed of the working line should be maintained between 400 m to 515.25 m. A working line length of 1 820 m was found to maximize the operational efficiency of the mining equipment. Based on the feasible range of advancing speeds, the Monte Carlo method was innovatively used to optimize the working line shape. Three working line layout and development schemes, all satisfying the constraints of the working line length and annual advancing speed, were proposed. To evaluate these schemes, a CRITIC-TOPSIS comprehensive evaluation model based on objective weighting was constructed, incorporating six key indicators: average production stripping ratio, weighted average haulage distance and lifting height for both stripping and raw coal, and the average maximum advancing speed of the working line. Evaluation results show that scheme 2 exhibits the closest proximity to the ideal solution. This scheme enables Haerwusu open-pit coal mine to achieve its 35 Mt/a raw coal production target within a narrow mining district, while delivering optimal technical and economic outcomes.