In this paper, 3-D simulation of heat transfer in a power electronic device and its cooling system is performed. The device is a high voltage three-phase inverter manufactured by Semikron Company which its main application is in electric and hybrid vehicles. Cooling system is a forced-air plate-fin heat sink. Limitation factor of designing heat transfer is maximum temperature of the inverter’s chips, heat sources, called IGBT. Maximum temperature of IGBTs should be below 125 ᵒC in order to avoidance of both the thermal and the mechanical failures. One of the primary objectives is the reduction of the maximum temperature by designing layout of chips. Also, the heatsink geometry design is accomplished with the consideration of the maximum temperature and tradeoff between both the usage material volume and the heatsink efficiency. Geometries are the number of fins, the fin height, the fin thickness and the base thickness of the heatsink. The power dissipation is estimated using datasheet information and output waveforms obtained from simulation in MATLAB. A thermal model of the inverter and its cooling system are simulated by using finite-element method (FEM). The accuracy of the thermal model and power dissipation estimation are verified by Semisel software. The maximum temperature is significantly reduced about 20 ᵒC by designing layout precisely. Also, the heatsink efficiency is increased 10.35%, 16.67% and 27.51% with the increase of the material volume about 22.52%, 13.51% and 0% for the heat transfer coefficient, 50, 75 and 100 (W/m2.K) by good design of the heatsink geometry ,respectively.