We show that for any integer $k\ge 1$ there exists an integer $t_0(k)$ such that, for integers $t, k_1, \ldots , k_{t+1}, n$ with $t\gt t_0(k)$, $\max \{k_1, \ldots , k_{t+1}\}\le k$, and $n \gt 2k(t+1)$, the following holds: If $F_i$ is a $k_i$-uniform hypergraph with vertex set $[n]$ and more than $ \binom{n}{k_i}-\binom{n-t}{k_i} - \binom{n-t-k}{k_i-1} + 1$ edges for all $i \in [t+1]$, then either $\{F_1,\ldots , F_{t+1}\}$ admits a rainbow matching of size $t+1$ or there exists $W\in \binom{[n]}{t}$ such that $W$ intersects $F_i$ for all $i\in [t+1]$. This may be viewed as a rainbow non-uniform extension of the classical Hilton-Milner theorem. We also show that the same holds for every $t$ and $n \gt 2k^3t$, generalizing a recent stability result of Frankl and Kupavskii on matchings to rainbow matchings.