A new 3-D nonlinear model of chatter vibration in turning process is presented in this paper. Workpiece and cutting tool are modeled as rotational shaft and cantilever beam, respectively which are excited by cutting forces. Equations of motion of workpiece are derived using Hamilton’s principal. Also, Timoshenko beam theory is used to simulate tool’s vibration. Then π-Buckingham theory is used to extract dimensionless equations of motion for transverse and torsional vibrations of workpiece and transverse and longitudinal vibrations of cutting tool. Using the mode summation method, a numerical solution is presented for this nonlinear problem. Effect of cutting parameters such as longitudinal cutting position, cutting width, cutting depth and radius of workpiece for machining with and without tailstock are investigated. Using these results turning velocity intervals for stable and unstable cuts are determined. Simulation results show that using tailstock in turning process for machining near the end of the workpiece increases process stability. Also in the case of using tailstock, machining near the end of the workpiece is more stable than the machining in the middle region of the workpiece.