] Do we know that he intended to invade Denmark? ... I can't remember if there is anything in the text itself that says so.

The play begins with Denmark making preparations for an expected war:

Now, sir, young Fortinbras,
Of unimproved mettle hot and full, (...)
But to recover of us, by strong hand
And terms compulsatory, those foresaid lands
So by his father lost: and this, I take it,
Is the main motive of our preparations,

Claudius sends the ambassadors to Old Norway telling him what Fortinbras is doing, asking him to reign him in, which he does:

Upon our first, he sent out to suppress
His nephew's levies; which to him appear'd
To be a preparation 'gainst the Polack;
But, better look'd into, he truly found
It was against your highness: whereat grieved,
That so his sickness, age and impotence
Was falsely borne in hand, sends out arrests
On Fortinbras; which he, in brief, obeys;
Receives rebuke from Norway, and in fine
Makes vow before his uncle never more
To give the assay of arms against your majesty.
Whereon old Norway, overcome with joy,
Gives him three thousand crowns in annual fee,
And his commission to employ those soldiers,
So levied as before, against the Polack:
With an entreaty, herein further shown,
That it might please you to give quiet pass
Through your dominions for this enterprise,
On such regards of safety and allowance
As therein are set down.

So this isn't a figment of the imagination - Fortinbras' aim was to invade Denmark. But Norway is so "impotent" that he accepts Fortinbras' promise not to invade Denmark and rewards him with an even greater army to use against the Polack.

Now, it just seems obvious to me that Fortinbras is going to use that army against Denmark. After all, the land in Poland is nothing, everyone acknowledges that. But the lands in Denmark belonged to his father, and he has shown how eager he is to recover them.

At the end Osric says "Young Fortinbras, with conquest come from Poland," and Horatio addresses him with "You from the Polack wars," so it sounds as if he did go fight the Polack, but he still has to cross through Denmark, and why not use his battle-hardened, victorious troops to fight Denmark, as long as he's there? ;-)