Hybrid joints (Metal–Composite) is being used more and more in aerospace industry due to their low weight and high strength. Dynamic study of this joint, owing to limitation of increase in screw’s preload in composite substructure, has certain importance. Effective factors on nonlinear behavior of the joint are low preload of the screw and high excitation force amplitude on the structure. Layer Element Model has been used to better the description of joint’s behavior in recent years. In this study effects of nonlinear behavior of joint on the structure has been investigated using 2D layer element theory in two divisions: increase of damping and decrease of stiffness which result in nonlinearity. Stiffness characteristics of the joint was modeled with normal stiffness and damping characteristics of the joint with structural damping in shear direction. Nonlinear frequency response function for two preload and two excitation force was extracted and nonlinear finite element model for stiffness and damping of the joint is suggested by High-order polynomial approximation in terms of response amplitude. Effects of increase of excitation force amplitude and decrease of screw’s preload on increase of nonlinearity was extracted by this finite element model. Results indicate that presented nonlinear finite element model corresponds closely to nonlinear vibration tests.