Effective conservation of global species diversity requires a clear understanding of spatial scales that support overall diversity across broad scales. Abandonment of semi-natural grasslands has increased their fragmentation and decreased their areal extent. We quantified diversity patterns of plant communities in Japan across hierarchical scales to facilitate the development of an effective nationwide strategy for conserving species diversity in remnant semi-natural grasslands. We applied additive partitioning of plant species diversity, using a nested hierarchical design at three spatial scales (quadrat, grassland, and western and eastern regions of Japan) for three groups of plant species (all species, grassland species and national Red Listed species). We consistently found lower proportions of among-quadrats diversity, and higher proportions of among-grasslands diversity and between-regions diversity in the overall diversity of the entire species complement than would be expected by chance. The high contribution of among-grasslands diversity to overall diversity suggests that each grassland had a unique species content. The second-ranking contributor to overall diversity differed between grassland species and Red Listed species: the second-ranking contributor for grassland species was diversity at the among-quadrats scale but the second-ranking contributor for all species and for Red Listed species was diversity at the between-regions scale. Thus, effective conservation of diversity of the entire species complement in remnant semi-natural grasslands requires preservation of beta diversity in individual grasslands. Our findings highlight the importance of strengthening local preservation and restoration activities within each grassland, and of nationwide strategies for conserving Red Listed species in remnant semi-natural grassland communities.