There are concerns a planned rally outside its premises on Bridge Road could end in the same ugly scenes witnessed in Federation Square last month when members of anti-Islamic movement Reclaim Australia faced off against supporters of No Room For Racism, a coalition of community and left-wing groups.

The clash ended in arrests, assaults and people being pushed and spat on as mounted police tried to keep the groups apart.

Sunday's Bridge Road rally, organised by Reclaim Australia splinter group United Patriots Front, is being promoted as a demonstration against people with a "Communist dream of destroying this nation".

Also planned for Sunday is an Islamophobia community forum run by No Room For Racism.

This event will be chaired by Socialist Party councillor Stephen Jolly and attended by the president of the Islamic Council of Victoria, Ghaith Krayem , criminal defence lawyer Robert Stary, who defended Julian Assange, and Jack Thomas will also be there.

A counter-rally by the Campaign Against Racism and Fascism has been organised to disrupt the UPF demonstration on Sunday. According to the group's event page, "Stop the Fascists in Richmond", more than 570 supporters are planning to attend.

UPF is headed by former Reclaim Australia member Shermon Burgess, who runs a popular anti-Islam facebook page the Great Aussie Patriot. The group accepts anyone who is "against extremist Islam" and "extremist left wing groups", the page says.

In a video posted on Thursday afternoon calling for supporters to attend the rally, Burgess said: "We're going to take it to radical Islam. We're going to take it to these Communist traitors, who keep telling us we cannot be proud of our Australian flag, we cannot be proud of our nationality, we cannot be proud of anything pro-Australian."

Cr Jolly said the UPF propaganda had an undercurrent of violence, but that he would not be deterred from attending the Islamophobia forum on Sunday.

"These neo-nazis spend their lives trying to find a minority to blame for society's problems," he said.

"We're not going to get diverted, we're going to continue our work to build connections with Australians who are being victimised."

A Yarra Council spokeswoman confirmed staff who had planned to work at the town hall on Sunday had been instructed to stay away.

In a statement the council said neither rally had been endorsed by council and police would attend to ensure public safety, protect the building and minimise disruption to residents and traders.

"Some of the people likely to be involved were also involved in the rallies that occurred in Federation Square in April, during which there were public clashes," the statement said. "The rallies may cause some disruption to parking, trams and traders in the direct vicinity of the demonstration."

Victoria Police said it was aware of the demonstrations planned for Sunday.