Traditional user interface layout design heavily relies on designers' subjective judgment and lacks effective quantitative assessment methods. This study aims to overcome this shortcoming. Therefore, this paper proposes a multi-objective optimization (MOO) system designed for the low-fidelity prototyping phase to enable faster production of high-quality interface layouts. A hybrid algorithm (SNSGA-II) is designed in the system, which combines the Non-Dominated Sorting Genetic Algorithm II (NSGA-II) and the simulated annealing (SA) algorithm, along with a scoring system including both aesthetics and usability. During the high-temperature phase, the SA algorithm is responsible for global exploration, while the genetic algorithm is introduced during the low-temperature phase for refined optimization, combined with an improved sigmoid function to dynamically balance the weights of each evaluation dimension. The experimental results show that this system achieved significant optimization effects across three UI design tasks with varying complexity levels. Compared with traditional NSGA-II algorithm, the SNSGA-II algorithm demonstrated approximately 50% improvement in convergence speed. Evaluation by 20 professional UI designers showed that system-optimized designs exhibited consistent improvements across all complexity levels: overall design quality enhanced by 3.0%, 6.7%, and 6.5% for simple, moderate, and complex tasks respectively; aesthetic scores correspondingly increased by 5.6%, 5.7%, and 5.2%; learnability indicators improved by 4.1%, 6.1%, and 7.2% respectively. This study confirmed the effectiveness of this approach in balancing aesthetics and usability, demonstrating its algorithmic performance and practical optimization potential.