Update Wednesday 8.53am:
A Beulah man has reportedly removed a flag with Nazi symbolism on it from his backyard, after condemnation on Tuesday including from Victorian Premier Daniel Andrews.
Another resident of the town said the flag was taken down last night.
Victoria's Equal Opportunity and Human Rights Commission said in a statement on Tuesday night the symbols on the flag caused harm to the Jewish community.
"(They) are more than just lines and colours - they represent hate and trauma for so many people around the world, including the members of Victoria's Jewish community," it said.
"To display that flag publicly is reprehensible - it runs counter to community values and the importance Victorians place on inclusion and diversity. It has no place in this state."
Tuesday 3.30pm: A Beulah man who made statewide headlines after raising a flag with Nazi symbolism has said he will not be taking it down.
Bill, who wished not to have his full name printed, said he had had the flag in his garden for a month, having received it as a gift.
He said he had not received any complaints from neighbours, the council or police prior to a story running in a Melbourne newspaper on Tuesday morning.
"I'm not taking it down. If the flag was a banned article I could understand, but you can buy (Nazi flags) on the internet, it's not illegal," he said.
"No one came and asked about the flag. I might have considered taking it down if someone came to my door and asked nicely."
Bill said he had been flying the flag to represent his German ancestry.
On Tuesday afternoon it remained up, but had been lowered.
Read more: Neo Nazis hold training camp in Grampians
His carer Cheryl, who also declined her surname being used, said she was considering legal action against the Melbourne newspaper, which she says incorrectly printed that the flag was hers.
"It has nothing to do with me, but the damage is done," she said.
Premier Daniel Andrews today called for the flag to be taken down.
"The people who are displaying that despicable flag, this is disgusting, it is absolutely disgusting behaviour," he told a press conference.
"And if there's any decency in that household they will take that flag down immediately."
Mr Andrews said that anti-semitism as on the rise and that the Jewish community had "made that point to me many, many times".
"The stats show that. And it is completely unacceptable for anybody to be flying what is perhaps the ultimate symbol of hate. And that is that flag," he said.
"That is just disgusting behaviour. Nothing justifies that whatsoever and it should come down."
Yarriambiack Shire chief executive Jessie Holmes said the council was contacted by a Beulah police officer late last week after the officer received complaints about the flag.
"He wanted to know what legislative mechanisms we had to remove it, and we don't have any," she said.
"We are working with Victoria Police. Obviously best scenario is the owners take the flag down.
"From council's perspective this doesn't represent Beulah at all or the wider community. We will work with any statutory agency that has the authority or mechanism to remove it, as we consider it offensive."
"Beulah is a great town and doesn't deserve the negative publicity. The flag does."
A Beulah resident who did not wish to be named said she thought the flag was "disgusting".
"It's not a good look for the town. I don't think most people in town knew the flag was up," she said.
Beulah resident Michelle said she thought flying the flag was "not right" and it wasn't a good look for the town.
"I don't think it (the attention the flag is receiving) will have a long-term impact on Beulah," she said.
The Mail-Times has contacted Beulah police for comment.
While you're with us, you can now receive updates straight to your inbox twice weekly from the Wimmera Mail-Times. To make sure you're up-to-date with all the news from across the Wimmera, sign up below.