In this study, nonlinear behavior of an atomic force microscopes (AFM) immersed in acetone, water, carbon tetrachloride (CCl4), and 1-butanol is investigated using non-classical strain gradient theory. In this theory, the size effect of system is taking into account by means of material length scale parameter. The nonlinear behavior of the AFM is due to the nonlinearity of the AFM tip–sample interaction caused by the Van der Waals attraction/repulsion force. Behavior of micro beam immersed in liquid is completely different with its behavior in air and vacuum due to the existence of hydrodynamic force. The Resonant frequencies, mode shapes, governing nonlinear partial and ordinary differential equations (PDE and ODE) of motion, stability analysis, boundary conditions, potential function and phase-plane of the system are obtained analytically in the present study. Furthermore, the results are compared with the one obtained by the modified couple stress theory. For this purpose, the AFM and the probe at the free end of micro beam are modeled as a lumped mass. The fixed end of micro beam is excited by piezoelectric element. The nonlinear PDE of motion is derived based on Euler-Bernoulli theory by employing the Hamilton principle. The Galerkin method is utilized to gain the governing nonlinear ODE of motion and the obtained ODE is analytically solved by means of perturbation techniques.