The rational homology of the IA-automorphism group $\operatorname {IA}_n$ of the free group $F_n$ is still mysterious. We study the quotient of the rational homology of $\operatorname {IA}_n$ that is obtained as the image of the map induced by the abelianization map, which we call the Albanese homology of $\operatorname {IA}_n$. We obtain a representation-stable $\operatorname {GL}(n,\mathbb {Q})$-subquotient of the Albanese homology of $\operatorname {IA}_n$, which conjecturally coincides with the entire Albanese homology of $\operatorname {IA}_n$. In particular, we obtain a lower bound of the dimension of the Albanese homology of $\operatorname {IA}_n$ for each homological degree in a stable range. Moreover, we determine the entire third Albanese homology of $\operatorname {IA}_n$ for $n\ge 9$. We also study the Albanese homology of an analogue of $\operatorname {IA}_n$ for the outer automorphism group of $F_n$ and the Albanese homology of the Torelli groups of surfaces. Moreover, we study the relation between the Albanese homology of $\operatorname {IA}_n$ and the cohomology of $\operatorname {Aut}(F_n)$ with twisted coefficients.