We use a dynamical systems approach to identify coherent structures from often chaotic motions of inertial particles in open flows. We show that particle Lagrangian coherent structures (pLCS) act as boundaries between regions in which particles have different kinematics. They provide direct geometric information about the motion of ensembles of inertial particles, which is helpful to understand their transport. As an application, we apply the methodology to a planktonic predator–prey system in which moon jellyfish Aurelia aurita uses its body motion to generate a flow that transports small plankton such as copepods to its vicinity for feeding. With the flow field generated by the jellyfish measured experimentally and the dynamics of plankton described by a modified Maxey–Riley equation, we use the pLCS to identify a capture region in which prey can be captured by the jellyfish. The properties of the pLCS and the capture region enable analysis of the effect of several physiological and mechanical parameters on the predator–prey interaction, such as prey size, escape force, predator perception, etc. The methods developed here are equally applicable to multiphase and granular flows, and can be generalized to any other particle equation of motion, e.g. equations governing the motion of reacting particles or charged particles.