Moving the Horsham and Grampians Visitor Information Centre from O'Callaghans Parade to the Town Hall would make it difficult for "carvanners" to access information, a recreational vehicle (RV) user said.
Resident Neville McIntyre said the proposed location was geared towards "providers rather than users" .
In a question to Horsham Rural City Council at Monday's meeting Mr McIntyre said moving the tourist centre into the "congested" CBD would "further damage" Horsham's reputation among RV users.
"I as a tourist would not be dragging my motorhome and tow vehicle to the area (at Horsham Town Hall) nor would a 'caravanner' take their six metre caravan there," he said.
"That location would require a 'caravanner' to unhook and leave their van outside town. This will not happen. They will still keep travelling through."
The relocation was part of a series of recommendations from the Grampians Tourism Visitor Servicing Review Report, which council deferred a vote on after Cr John Robinson, Cr David Grimble and Cr Alethea Gulvin called for more information about the suggested changes.
Mr McIntyre said while the current site was difficult for some visitors to access it was better than what council had suggested.
"Even though it was off the main road (it) suited the purpose as it had easy access, long vehicle park (sic) and a toilet," he said.
Mr McIntyre asked council to consider two alternative sites - where McGennisken Hall is, which is set to demolished, on the Western and Henty Highways, or the former Red Rooster building on Dimboola Road.
HRCC director of development services defended the proposal and said council would create additional long vehicle parking in close proximity to the new site.
"Horsham Town Hall is a destination for visitors and the location provides a better opportunity for positive economic flow on to retail businesses," she said.
In the year ending December 2018, Horsham's Visitor Information Centres had 16,294 visitors.