During the Crimean War (1853-56), Britain and its allies came into conflict with Russia. Victoria, Australia, an undefended British colony on the other side of the world, was flush with wealth from the gold rush and vulnerable to attack by sea.

Following concerns in the mid-1870s about a possible attack by Russia, Swan Island was fortified to defend Port Phillip. A Fort was completed in 1884 to provide a forward artillery defence system of forts, supplemented by submarine mining of shipping channels. It was of impressive size and complex design.

By 1900, the Swan Island fortifications were inadequate for engaging modern warships and the Fort was used primarily for training. After Federation in 1901, the Commonwealth Government bought Swan Island from Victoria for £30,000 and it was then used by the army. In 1921 Swan Island was transformed to a naval facility.