On 27 January 1806, having been informed by the remnants of William Pitt's last ministry that they could not carry on after the death of their chief, the King sent for Lord Grenville. In partnership with the long-excluded Charles Fox, Grenville formed the administration known as ‘The Ministry of All the Talents’. For Eldon, the prospect of opposition – perhaps of a very lengthy duration – or of retirement, lay before him. In fact, he would be out of office for just over one year, for in May 1807 he returned to the Woolsack in the Pittite ministry of the Duke of Portland. In the years between 1806 and 1810 Eldon became more closely linked with overt political action, as evidenced by his parliamentary conduct, his involvement in Cabinet intrigues, and his relations with the royal family. The King continued to rely on him, and Eldon was not averse to using such political weapons as came his way. Opposition leaders came to see his hand in every scheme, every manoeuvre to thwart their aspirations. This, however, led them both to over-estimate the extent of his power and to misunderstand the direction of his interests. While a fighter of considerable and growing skill, Eldon's experience of political warfare was far from uniformly happy.

In the spring of 1806, with his son's death still fresh in his mind, Eldon professed to have little interest in current public affairs.