Nuclear stellar clusters (NSCs) are dense stellar systems known to exist at the center of most of the galaxies. Some of them host a central massive black hole (MBH). They are though to form through in-situ star formation following the infall of gas to the galactic center and/or because of the infall and merger of several stellar clusters. Here we explore the latter scenario by means of detailed self-consistent N-body simulations, proving that a NSC built by the infall and following merger of stellar clusters shows many of the observed features of the Milky Way NSC. We also explore the possibility that the infalling clusters host intermediate mass black holes (IMBHs). Once decayed to the center, the IMBHs act as massive-perturbers accelerating the relaxation of the NSC, filling the loss-cone and boosting the tidal disruption rate of stars up to a value larger than the observational estimates, therefore providing a cumulative constraint on the existence of IMBHs in NSCs. Studying how the properties of the infalling clusters map to the properties of the resulting NSC, we find that, in the IMBHs-free case, the infall mechanism is able to produce many different observational signatures in the form of age segregation.