Drawing on qualitative research with refugees in and outside formal settlements, this article challenges characterisations of Uganda's UNHCR-supported refugee settlement system as un-problematically successful. It shows that by denying refugees freedom of movement, the settlement system undermines their socio-economic and other rights. Refugees who remain outside the formal system of refugee registration and settlement are deprived of the refugee status to which they are entitled under international law. The article questions the conventional opposition between refugees living in and out of refugee settlements in the Ugandan context, revealing a more complex and interconnected dynamic than is often assumed. It suggests that those refugees with some external support may be able to escape the confines of remote rural settlements, where refugee agricultural livelihoods are seriously compromised by distance from markets, unfavourable climatic conditions, exhausted soil and inadequate inputs. It argues that refugee livelihoods face more rather than fewer challenges as exile becomes protracted, and concludes that the government and UNHCR's Self Reliance Strategy (SRS) has not yet managed to overcome the contradiction inherent in denying people freedom of movement, without supporting them effectively to meet their needs in the places to which they are restricted.