Participants

The pooled data-set included participants from two studies, where participants underwent conventional imaging measures, reported previously^{Reference Menzies, Achard, Chamberlain, Fineberg, Chen and del Campo11,Reference Vaghi, Vertes, Kitzbichler, Apergis-Schoute, van der Flier and Fineberg12} (technical details of the scans are provided in Supplementary File 1 available at https://doi.org/10.1192/bjp.2021.45). The first sample comprised 32 pairs of unaffected first-degree relatives and OCD probands, and 32 age- and gender-matched healthy controls.^{Reference Macpherson and Hikida7} The second sample comprised 44 patients with OCD and 43 age- and gender-matched healthy controls.^{Reference Chamberlain, Blackwell, Fineberg, Robbins and Sahakian8}

Participants were screened for mental disorders by extended clinical interview conducted by a psychiatrist, supplemented with the Mini International Neuropsychiatric Inventory (MINI).^{Reference Sheehan, Lecrubier, Sheehan, Amorim, Janavs and Weiller13} OCD severity was quantified with the Yale-Brown Obsessive Compulsive Scale (Y-BOCS),^{Reference Goodman, Price, Rasmussen, Mazure, Fleischmann and Hill14} and IQ was measured by the National Adult Reading Test.^{Reference Nelson15, Reference Nelson and Willison16} Patients with OCD were excluded if they had any comorbidities, including depression. Controls and first-degree relatives of patients were excluded if they had any history of mental disorders. Controls were excluded if they reported any known history of OCD in first-degree family members. History of serious head injury, substance misuse, epilepsy or magnetic resonance imaging contraindications was exclusionary across all groups.

The study was approved by the Addenbrooke's NHS Trust Local Research Ethics Committee (Cambridge, UK) and Local Research Ethical Committee at the University of Cambridge (Cambridge, UK). The research complied with relevant ethical standards including the Declaration of Helsinki. All participants provided written informed consent after reading information sheets about the study and having the opportunity to ask questions.

Behavioural testing

Before scanning, all participants undertook the computerised IDED set-shift task from the Cambridge Neuropsychological Test Automated Battery (Cambridge Cognition, Cambridge, UK; see https://www.cambridgecognition.com/cantab/). Two stimuli were displayed on-screen for each trial, and participants attempted to learn underlying rules governing which stimulus was correct, based on computerised feedback. Set-shifting performance on the task was indexed by errors on the crucial extradimensional attentional shift stage. Of the different stages of the task (i.e. learning, intradimensional set-shifting, etc.) extradimensional attentional shift stage was previously found to be selectively impaired in OCD, in meta-analysis.^{Reference Chamberlain, Solly, Hook, Vaghi and Robbins10} Extradimensional shifting is widely regarded as the archetypal form of set-shifting on such tasks. A more detailed task description is provided in Supplementary File 1.

Data analysis

To address the two study hypotheses, our overall a priori analytic approach was first, to identify morphological regions of basal ganglia structures related to set-shifting performance; and second, to examine whether morphology in these set-shifting-related areas was abnormal in patients with OCD and their relatives versus controls.

Following standard pre-processing pipelines (for details, see Supplementary File 1), vertex analysis was deployed to investigate the shapes of subcortical structures, using FIRST from the FMRIB Software Library (FSL).^{Reference Patenaude, Smith, Kennedy and Jenkinson17} The vertex locations of each participant were projected onto the surface normal of the average shape template of the 336 training subjects provided by FSL, and the perpendicular distance from the average surface was calculated. A positive value of the vertex indicated deformation in the outward direction, and a negative value of a vertex indicated deformation in the inward direction. ‘Randomise’, a permutation-based, non-parametric testing method implemented in FSL (5000 iterations), was used to identify subcortical structures whose shape was significantly associated (i.e. correlated with) with extradimensional errors, across all participants.^{Reference Winkler, Ridgway, Webster, Smith and Nichols18} Based on extant models of OCD and structures implicated in flexibility, we focused on the caudate, putamen, globus pallidus and nucleus accumbens.^{Reference Fineberg, Apergis-Schoute, Vaghi, Banca, Gillan and Voon2,Reference Graybiel and Rauch3} We performed multiple comparison correction at P < 0.05, using threshold-free cluster enhancement.

Group differences in demographic data were explored by ANOVA or chi-squared test, as appropriate. Group differences in subcortical clusters associated with extradimensional performance were explored by non-parametric Kruskal–Wallis test, to identify overall group differences; and Wilcoxon test, for pairwise comparison of groups if overall effects of groups were found. Correlations (Spearman's rho) were examined between patient–relative pairs in terms of curvature in the identified clusters, and between randomly permuted pairs of participants, to identify whether the extradimensional-related structural changes were significantly familial. Statistical significance was defined as P < 0.05 (uncorrected).