This research deals with configuration design and layout optimization of communication satellite. First, an approach is proposed to design the configuration of GEO satellite. Since propulsion subsystem in GEO satellite is the massive item, it has a significant impact on satellite configuration. Consequently, it is necessary to consider the propulsion subsystem influence on satellite configuration. Then layout design process of the satellite components which is one of the complex problems in engineering is performed. In this paper, in order to optimize the layout design of satellite components, the algorithm which consists of two stages, primary and detail layout, is proposed. In order to express geometric constraints mathematically, the Finite Circle Method (FCM) is used. For The mathematical expression of performance constraints, the distance constrains related to distance relationships between components have been developed. The hybrid optimization method is proposed to optimize layout design which is a combination of Simulated Annealing optimization and Quasi Newton methods. The optimization method validation is applied on simple test problem. Finally, the proposed algorithm for configuration and optimal layout design is implemented on communication satellite. The results show that product of inertial (objective function) are minimized and considered constrains of communication satellite are satisfied.