An automatic cloud classification method on whole-sky images is presented. We first extract multiple pixel-level features to form region covariance descriptors (RCovDs) and then encode RCovDs by the Riemannian bag-of-feature (BoF) method to output the histogram representation. Reults show that a very high prediction accuracy can be obtained with a small number of training samples, which validate the proposed method and exhibit the competitive performance against state-of-the-art methods.