Cooperation and autonomy are among the most important aspects of unmanned systems through which greater use of these system is possible. Most applications in civil market is related to government organizations requiring surveillance and inspection, such as coast guards, border patrol, emergency services and police. A cooperation algorithm is developed and simulated in this research for autonomous UAVs to track a dynamic target in an adversarial environment. First, a mathematical formulation is developed to represent the area of operation that contains various types of threats in a single framework. Then a search point guidance algorithm is developed by using a rule-based approach to guide every UAV to the way points created by the cooperation algorithm, with the requirements of completing mission, avoiding restricted areas, minimizing threat exposure level, considering the dynamic constraints of the UAVs and avoiding collision. The cooperation algorithm is designed based on a variable formation which depends on a cost function. The efficiency of the team is improved in the terms of increasing the area of coverage of the sensors, flexibility of the UAVs to search for better trajectories in terms of restricted area avoidance and threat exposure minimization, and improving the estimation. Finally, the performance of the algorithm is evaluated in a MATLAB environment, which includes the dynamics of vehicles, the models of sensor measurement and data communication and the discrete execution of the algorithms. The simulation results demonstrate that the proposed algorithms successfully generated the trajectories that satisfy the given mission objectives.