<p>The observation of the nocturnal boundary layer height (NBLH) plays an important role in air pollution and monitoring. Through 39 days heavily polluted observation experiment over Beijing (China) and exhaustive evaluation of gradient method (GM), wavelet covariance transform method (WCT) and cubic roots gradient method (CRGM), a novel algorithm based on cluster analysis of gradient method (CA-GM) of lidar signal is developed to capture the multilayer structure and achieve stability in the nighttime. The CA-GM highlights its performance in comparison with radiosonde data, the best correlation (0.85), the weakest root mean square error (203 m), and the improved 25 % correlation coefficient by the GM. In comparison with long-term experiments with other algorithms, a reasonable parameter selection can distinguish layers with different properties, such as the cloud layer, elevated aerosol layers, and random noise. Consequently, the CA-GM can automatically deal with the uncertainty of the multiple structures and obtain a stable NBLH with a high time resolution, which expected to contribute to air pollution monitoring and climatologies, as well as model verification.</p>