A model for the quantum Brayton refrigerator that takes the harmonic oscillator system as the working substance is established. Expressions of cooling load, coefficient of performance (COP), and ecological function are derived. With numerical illustrations, the optimal ecological performance is investigated. At the same time, effects of heat leakage and quantum friction are also studied. For the case with the classical approximation, the optimal ecological performance, and effects of heat leakage and quantum friction are also investigated. For both general cases and the case with classical approximation, the results indicate that the ecological function has a maximum. The irreversible losses decrease the ecological performance, while having different effects on the optimal ecological performance. For the case with classical approximation, numerical calculation with friction coefficient μ = 0.02 and heat leakage coefficient Ce = 0.01 shows that the cooling load (RE) at the maximum ecological function is 6.23% smaller than the maximum cooling load (Rmax). The COP is also increased by 12.1%, and the exergy loss rate is decreased by 27.6%. Compared with the maximum COP state, the COP (ɛE) at the maximum ecological function is 0.55% smaller than the maximum COP (ɛmax) and that makes 7.63% increase in exergy loss rate, but also makes 6.17% increase in cooling load and 6.20% increase in exergy output rate.