In recent years, the rapid development of artificial intelligence (AI) technology has been driving profound transformations in higher education. As a fundamental course in science and engineering disciplines, Advanced Mathematics plays a crucial role in cultivating students’ logical thinking and innovative capabilities. However, the current teaching models exhibit significant shortcomings in fostering students’ ability to identify and solve problems, primarily reflected in the monotony of teaching content, the limitations of students’ thinking, and the constraints of instructional methods. In response to these issues, this paper proposes an AI-based teaching approach for Advanced Mathematics. By utilizing scenario simulations to guide students in discovering problems and employing modeling tools to assist them in solving problems in real time, the study constructs a comprehensive teaching model that spans the entire process from problem identification to problem resolution. Research findings indicate that the application of AI technology can effectively enhance students’ abilities in problem awareness, logical reasoning, and creative thinking. This study provides both theoretical support and practical reference for the reform of Advanced Mathematics education and the innovation of higher-level talent cultivation models.