The flow induced vibration in transonic turbomachines is an important and challenging issue in this field. Blades aeroelastic behavior, in addition to the aeroelastic instability, can leads to blades failure, flow instability and reduce efficiency of the system. Aerodynamic behavior of the system should be investigated prior to aeroelastic study. The purpose of this article is an investigation of aeroelastic instability and behavior of a selected turbomachine. For this purpose, transonic flow in Nasa 37 rotor is simulated and verified using CFX software. Then, rotor blade aeroelastic stability is investigated in three operating points; design, near stall and stall using blade forced vibration in the specified inter blade phase angle (IBPA). In order to reduce grid points and consequently, computational time, phase-lagged boundary condition and fourier transformation method is used. Also, in this research, the algorithm of simultaneous structure-fluid grid generation and the solution algorithm of force vibration structure-fluid interaction of turbomachines is codified and introduced in detail. Employment of fourier transformation method in CFX software for aeroelastic simulation is another innovation of this article. The value of the critical inter blade phase angle which is independent from rotor operational conditions, is obtained in the present research. Aeroelastic simulations show aeroelastic instability of Nasa 37 rotor in the stall condition. In this condition, flow entropy is increased rapidly relative to the design and near stall condition. The blade pressure side has more important role in stall aeroelastic instability and needs further attention in re-design phase.