Victorian Climbing Club President Kevin Lindorff said Mr Borg's suggestion of reaching out to Traditional Owners was not straightforward.
"Because Parks has besmirched our name with false accusations (of damaging cultural heritage) last year, they don't want to talk to us," he said.
"We have been trying, but they see us as cultural vandals."
Kerry Walker, of the Arapiles District Community Group set up earlier this year, said she was concerned for the future of Natimuk.
"We get something like 80,000 people come through the town in a year, largely for the rock," she said. "If we lose the pub and the climbing shop, the town will go.
"They are saying the survey will take about six months. After the survey, they will get a plan, and that could take 12 months, and after they've got the plan, then there will be some action.
"I put it to (Parks) at the roundtable meeting (on Thursday) that potentially it could all be off-limits for the next two years, and there was no disagreement."
Mr Lindorff, who moved from Melbourne to climb in Natimuk, said while climbers could do damage, he believed they had a good record of being responsible users of the mountain and the Grampians.
"If we have been climbing at Mount Arapiles for 60 years and the scatter is still there, what damage have we done to it while being there? None, otherwise they wouldn't be rediscovering it," he said.
The Mail-Times has contacted Barengi Gadjin Land Council for comment.
Parks Victoria's western regional director Jason Borg has suggested the new restrictions covering climbing spots Tiger Wall, Castle Crag and Plaque Wall, on the mountain's eastern edge will be in place until at least April, while a long-term plan for rockclimbing is devised.
The mountain is regarded as Australia's best rock climbing area, with over 2000 climbs according to Visit Victoria.
Parks and Barengi Gadjin Land Council plan to start six months worth of studies on the rediscovered stone tool quarry at the mountain immediately.
Mr Borg rejected the notion Parks was not working on reasonable management solutions.
"We work with BGLC as co-managers of the Park, that is our first and foremost partnership," he said.
"What I would encourage climbers to do, as I did at the roundtable, is ensuring rock climbing groups are fostering their own relationships with Barengi Gadjin, so that they are comfortable in having clibers as part of the discussion with Parks Victoria. At the moment, in some cases, they are not.
"We have been trying to facilitate these relationships with Traditional Owners.
Mr Borg said the new protections would be in place "indefinitely", until Parks and Traditional Owners had completed a full park survey at Mount Arapiles and determined the way forward.
"This will be a much smaller-scale plan, and focus mostly on climbing," he said.
He also said Parks would seek to continue preventing people entering another area of Mount Arapiles, Taylor's Rock/Declaration Crag, after the interim protection order expired a few weeks ago.
"The cultural heritage values are still there, and they need protection, and we will protect them by using set asides under the National Parks Act (1975)," he said.
Asked for a commitment that climbing in the restricted areas would eventually return, Mr Borg did not answer directly.
"We would like to think climbing can co-exist with cultural heritage where it is appropraite and sustainable, and that is why we are focused on doing this survey of the park," he said.
"These will determine what the plan is around access to Mount Arapiles. We know how much climbing means to Natimuk, and we've heard the uncertainty is challenging, so we are hoping this long-term approach gives us the opportunity to put together a long-term plan for rockclimbing."
Simon Carter, who now lives in New South Wales' Blue Mountains, labelled the changes "an absolute disaster for climbing in Victoria". As a committee member of the Australian Climbing Association of Victoria, he attended Thursday's roundtable meeting where Parks informed stakeholders of the changes.
"The point I keep repeating is that I see these bans as overreaching and counterproductive, in terms of management of these areas and reconciliation," he said.
"It would be better if a far more nuanced management approach is taken: I'm sure they could be talking to Traditional Owners that it is not necessary to be so extensive.
"Parks has made it hard for climbers to have dialogue (with Traditional Owners). The Gariwerd Wimmera Reconciliation Work has sort of filled that space, but they are not informing the representative climbing groups what is going on."
Mr Carter said he felt Parks Victoria has not shown any intent to reopen areas that had no cultural heritage.
"The climbing community has been incredibly generous and helpful in abiding by the bans put in pace, but this goodwill has not been reciprocated," he said.