Signal processing has a key role in signal based fault diagnosis in rotating machinery for finding beneficial discriminating features. Task of Signal processing is conversion of the raw data into beneficial features to facilitate the diagnostic operations. the features should be robust regarding noise and working condition of the machine and simultaneously sensitive to the machine defects. Therefore, assignment of more efficient analyzing techniques in order to achieve more beneficial features of the signal and faster and more accurate fault detection is taken into consideration by researchers. In order to finding such features, the current research applies at first wavelet packet denoising and then applies wavelet packet based Hilbert transform as well as improved Hilbert-Huang transform separately to decompose vibration signal into narrow frequency bands in order to extracting instantaneous frequencies. The findings show that the wavelet packet based Hilbert transform generates better results in comparison to the improved Hilbert-Huang transform in detecting frequencies of the broken rotor bar fault.