In this study, the nonlinear vibration behavior of a dynamic atomic force microscope (DAFM) in the tapping mode is investigated. First, the governing differential equation of motion and boundary conditions for dynamic analysis are obtained by a combination of the basic equations of the modified couple stress theory and Hamilton principle. Regarding the nonlinear dynamics of the probe, perturbation technique has been used to solve the nonlinear equations. Afterwards, closed-form expressions for nonlinear frequency and effective nonlinear damping factor are derived. The effect of connection position of the tip on the vibration behavior of the microcantilever are also analyzed. The results obtained by couple stress theory are compared with those of classical beam theory. The results show that the nonlinear frequency and effective nonlinear damping factor are size dependant. According to the results, an increase in the equilibrium separation between the tip and the surface sample reduces the overall effect of van der Waals forces on the nonlinear frequency, but its effect on the effective nonlinear damping factor is negligible. The results also indicate that the change in the distance between tip and cantilever free end has a significant effect on the accuracy of the DAFM.