Li Fu had one courtesy name, Chü-lai, and two pen names, Mu-t'ang and Hsiao-shan. He was known to his contemporaries as Master Mu-t'ang and has been commonly regarded as the major proponent of the Lu-Wang school in the Ch'ing dynasty. He was born on April 5, 1675, in Linch'uan, Kiangsi. His father was a native of She-hsien of Anhwei. His grandfather died when his father was still young. This probably caused his father to wander about central and southern China in search of a living. His father finally settled in Lin-ch'uan after marrying into a family named Wu.

He married the only child of the Wu family, and so the parents asked him to stay in their family. From Li Fu's later description, it seems that the marriage did not cause his father to stay home very long; Li Fu's father was by nature fond of traveling and left home often. Li Fu and his four brothers were brought up and educated mainly by their mother, who embodied the traditional virtues of Chinese women – frugality, diligence, and perseverance. All of these virtues were especially cherished in a poor family like Li Fu's.

Since his childhood, Li Fu was known for his gift in writing poetry. He began composing poems at the age of ten, writing prose at the age of twelve, and practicing “current essays” (the “eight-legged essays”) to prepare for the civil service examinations at fourteen.