Optimization has found a widespread application in many branches of science. In recent years, different methods and theories have been developed to find optimal solutions. Optimization algorithms inspired by nature as heuristics solutions to complex problems. Reverse engineering is one of the applications of optimization methods. In reverse engineering a set of scan points are defined relative to a particular coordination. In data registration process the scanned data sets separated and combined to a single coordinate system are called the process of registration. In this research, applications part has been digitized by coordinate measuring machine(CMM) and the process of point clouds registration in experimental on two pieces in position (without translation and with translation case) has been implemented. Using gravitational search algorithm (GSA), particle swarm optimization (PSO) and genetic algorithm (GA) optimization process is optimized and the registration parameters (rotation and displacement) are obtained. The algorithms mentioned, GSA the accuracy displacement, rotational accuracy and better convergence rate and the run time is less. Finally, a hybrid algorithm is proposed which is a combination of GSA, and Nelder-Mead algorithms (GSA-NM). In the proposed algorithm, the initial guess values obtained by GSA and Nelder-Mead algorithm is provided to ensure an accurate response. The proposed hybrid algorithm is superior to GSA and Nelder-Mead, in terms of the number of iterations and the amount of convergence.