The linear stability of nanofluid boundary-layer flow over a flat plate is investigated using a two-phase formulation that incorporates the Brinkman (1952 J. Chem. Phys., vol. 20, pp. 571–581) model for viscosity along with Brownian motion (BM) and thermophoresis (TP), building upon the earlier work of Buongiorno (2006 J. Heat Transfer, vol. 128, pp. 240–250). Solutions to the steady boundary-layer equations reveal a thin nanoparticle concentration layer near the plate surface, with a characteristic thickness of
$O({\textit{Re}}^{-1/2}{\textit{Sc}}^{-1/3})$, for a Reynolds number
${\textit{Re}}$ and Schmidt number
${\textit{Sc}}$. When BM and TP are neglected, the governing equations reduce to the standard Blasius formulation for a single-phase fluid, and the nanoparticle concentration layer disappears, resulting in a uniform concentration across the boundary layer. Neutral stability curves and critical conditions for the onset of the Tollmien–Schlichting (TS) wave are computed for a range of nanoparticle materials and volume concentrations. Results indicate that while the effects of BM and TP are negligible, the impact of nanoparticle density is significant. Denser nanoparticles, such as silver and copper, destabilise the TS wave, whereas lighter nanoparticles, like aluminium and silicon, establish a small stabilising effect. Additionally, the viscosity model plays a crucial role, with alternative formulations leading to different stability behaviour. Finally, a high Reynolds number asymptotic analysis is undertaken for the lower branch of the neutral stability curve.