Throughout that time, it was at the forefront of the growth of Horsham from a frontier town into a bustling commercial hub.

On July 7, 1864 John Poholke opened a wine shanty on the south-west corner of Firebrace and Pynsent streets. On September 15, he obtained a beer licence. Poholke owned allotment 9, section 2 in Horsham, consisting of half an acre. The hotel was initially just part of his own house until 1867, when he expanded the premises. He called his hotel the Carriers Arms.

In 1876, Poholke leased the hotel to Nicholas Rauert, who changed the name to Rauert's Victoria Hotel, then to Ferdinand Rissmann from 1879. Rissmann was an immigrant from Brandenburg, Germany. He was a blacksmith by trade but had turned his hand to gold digging in NSW, Ararat and Stawell, where he'd met with some success and invested in land. He married Anna Elizabeth Buettner in 1859 at Waurn Ponds, near Geelong where he had a small farm.

In 1874, Rissmann moved to Donald, where he farmed for five years, then travelled to Horsham in 1879 where he took over the lease of the hotel. He renovated and enlarged the hotel, renaming it the Railway Hotel; the railway having come to Horsham in February 1879. This was considered a marketing tactic, since the hotel was more than a kilometre from the station. Rissmann purchased the freehold from Poholke in 1883 for £3000 (about $2.7 million today).

Rissmann's probate document, dated 1897, lists his bar stock as: two cases and an eight-gallon barrel of brandy; one case and six bottles of rum; two bottles of schnapps; an eight-gallon barrel of whiskey; two x 36-gallon barrels of beer.

Rissmann died in July 1896, aged 68. Although he had several land holdings, the bank and land crash of 1895 severely reduced the net worth of his assets to £1211. His widow carried on as licensee until December 1902 when, having sold the hotel to her daughter and son-in-law, she transferred the licence to her daughter, Martha Gilliam. Martha was licensee from December 1902 until February 1903 when she sold the property to Cornelius "Con" Curran. The Rissmann family ran the hotel for a total of 24 years.

Curran had recently sold his Albion Hotel when he purchased the Railway. One of his first acts was to change its name to the Exchange Hotel, possibly because it was opposite the Bank of Victoria.

In 1908, Curran sold the freehold of the entire block to two developers, AW Leitner (the hotel and stables) and Joseph Puls (the southern portion, on which a row of shops sat). The price was £5500. A Horsham Times report of April 3, 1908 stated "the purchasers are to be complimented on their enterprise, as the present structures in the centre of the town are very unsightly".

John Joseph Hennessy purchased the freehold from Leitner for a reported £3000 in May 1908. Hennessy demolished the original buildings in November 1909 and constructed the present two-storey brick hotel, which he opened in February 1910. It had 24 bedrooms (14 in the main building with an additional 10 in an annexe), a wide entrance hall, billiards room, two bar parlours, a spacious dining room and large sitting room. Estimated cost of construction was £3000. Accommodation was at a premium in rapidly expanding Horsham. Hotels were often used by long-term residents and as premises for various professionals who wished to have a central location for their practices. A dentist and a solicitor set up rooms in the 1920s and 30s.

In July 1935, the hotel was sold to Ballarat Brewing Company for £16,300 (about $6.8 million today). The company renovated and extended west along Pynsent Street in early 1936. In late 1962, they removed the ornate lacework verandah. The company sold the Exchange to Michael and Julia Marra in 2000. Nick and Mel Murray bought the hotel in 2004, building up the business before leasing it to Jeff and Joanne Guy in 2008. Trade fell away until in June 2013, when the Guys abandoned their lease. The Murrays have since refitted the hotel and are the current licensees.