In this study, the viscous fingering instability in miscible displacement of Newtonian fluid by Viscoelastic fluid is investigated. The Criminale–Eriksen–Filbey (CEF) model has been used as the constitutive equation. Simplicity and dependence of rheological functions to shear rate are the advantages of this model. Also, the Carreau-Yasuda model was used to show this dependency. In nonlinear simulation, using spectral method based on Hartly transforms, the effect of rheological functions of displacing viscoelastic fluid on this instability has been studied. The results are included concentration contours, transversely averaged concentration profiles, mixing length and sweep efficiency. The results show that, by changing the parameters in order to increases of viscosity of displacing viscoelastic fluid, flow becomes more stable. In other word, sweep efficiency is increased and mixing length is decreased. Also, at first, the sweep efficiency increases with changing the parameters in order to increase the first normal stress difference in this type of fluid and then decreases with evolution of fingering. However, this factor will have little effect on mixing length. In addition, as well as viscous fingering, several nonlinear finger interactions such as Spreading, coalescence and tip splitting were observed in simulation of viscoelastic fingering instability.