In order to detect damage in a large-scale and complicated structure, there is need to exact nonlinear numerical modeling that its results have been analyzed using a method of system identification. In this way, the extended finite element model (XFEM) based on cohesive crack model (XFEM Based Cohesive Crack Segments) for concrete material as a reliable model is used for investigating real responses of Karun 3 concrete dam against applied loads and damages. In this model, whole of the structure is potentially under damage risk, while there is no initial crack. The dam is numerically modeled and analyzed using the finite element method (FEM) and XFEM Based Cohesive Crack Segments respectively, and the dam is analyzed under the seismic excitation. Then, for specification of crack effects and nonlinear behavior, the structural modal parameters and their variation should be investigated based on structure response for obtaining damage initiation time and its location by using system identification based on continuous Wavelet (CWT) transform. Results show that the dam natural frequencies decrease after the crack is formed, where decrease in longitudinal and vertical responses are more than the transversal response decrease. Moreover, crack width and its exact location are specified precisely from comparing the intact and damaged crest and central cantilever vibration modes. Therefore, the combination of XFEM Based Cohesive Crack Segments and CWT is useful procedure for structural health monitoring of concrete arch dams.