In this paper, we study how the topology of the International Trade Network (ITN) changes in geographical space, and along time. We employ geographical distance between countries in the world to filter the links in the ITN, building a sequence of subnetworks, each one featuring trade links occurring at similar distance. We then test if the assortativity and clustering of ITN subnetworks changes as distance increases, and we find that this is indeed the case: distance strongly impacts, in a non-linear way, the topology of the ITN. We show that the ITN is disassortative at long distances, while it is assortative at short ones. Similarly, the main determinant of the overall high-ITN clustering level are triangular trade triples between geographically close countries. This means that trade partnership choices and trade patterns are highly differentiated over different distance ranges, even after controlling for the economic size and income per capita of trading partners, and it is persistent over time. This evidence has relevant implications for the non-linear evolution of globalization.