In this paper, we investigate the ideal structure of uniform Roe algebras for general metric spaces beyond the scope of Yu’s Property A. Inspired by the ideal of ghost operators coming from expander graphs and in contrast to the notion of geometric ideal, we introduce a notion of ghostly ideal in a uniform Roe algebra, whose elements are locally invisible in certain directions at infinity. We show that the geometric ideal and the ghostly ideal are, respectively, the smallest and the largest element in the lattice of ideals with a common invariant open subset of the unit space of the coarse groupoid by Skandalis–Tu–Yu, and hence the study of ideal structure can be reduced to classifying ideals between the geometric and the ghostly ones. We also provide a criterion to ensure that the geometric and the ghostly ideals have the same $K$-theory, which helps to recover counterexamples to the coarse Baum–Connes conjectures. Moreover, we introduce a notion of partial Property A for a metric space to characterize the situation in which the geometric ideal coincides with the ghostly ideal. As an application, we provide a concrete description for the maximal ideals in a uniform Roe algebra in terms of the minimal points in the Stone–Čech boundary of the space.