Federal Social Services Minister Kevin Andrews and Victorian Attorney-General Robert Clark will no longer attend the World Congress of Families because the new venue is Catch the Fire Ministries in Hallam.

The pair were due to give opening speeches and had claimed their attendance was part of their ministerial duties.

There has been public pressure over the pair's attendence. In July The Age revealed prominent Liberals would be attending the forum which included a promoter of Russia's "crusade" against homosexuality and a doctor who spruiks a theory that abortions cause breast cancer.

Catch the Fire president Danny Nalliah, and his political branch the Rise Up Australia Party, have courted controversy. Mr Nalliah infamously blamed the Black Saturday bushfires on Victoria's abortion laws. He is also a critic of Islam.

In statements, both Mr Clark and Mr Andrews criticised opponents of the conference as intolerant, and that people had the right to free speech.

But the reason cited for their withdrawal has been questioned given Catch The Fire had been listed as a sponsor on promotional material, which has circulated for months.

The event has been targeted by protesters and has been forced to change locations several times. Event organiser Babette Francis played down Catch The Fire's involvement, saying she was just renting the venue and the group did not have any role in organising the congress.

Mr Nalliah confirmed the organisation had not been involved in organising the conference, but had offered its premises on Thursday afternoon after learning they had not been able to secure another venue.

Premier Denis Napthine also cited security concerns for Mr Clark's decision.

"We believe in free speech in Victoria … but what we have been advised is the location changed, there are security issues involved and also the new location is totally inappropriate for a senior government minister to attend," Dr Napthine said.

Victorian Liberal backbenchers Bernie Finn and Jan Kronberg, who were due to speak at the conference, have also withdrawn from the event.

The attendance of government MPs at the far right wing congress agitated moderates within the Coalition and angered many in the public who saw MPs' attendance as endorsement of the policies.

In a statement Mr Andrews said he had planned to attend because "tolerance was a critical value in a Western liberal democracy like Australia."

"The calls for me not to attend demonstrate the intolerance of the Greens and the left - instead of arguing their case in the public arena they seek to shut down debate," Mr Andrews said. "Equally, I cannot support intolerance from other quarters."

His office said he would not elaborate on what was so offensive about Catch the Fire.

Mr Nalliah told Fairfax Media the withdrawal from the conference on the grounds that it was being hosted by his organisation was a "cop-out".

"I think you just look for a scapegoat to try and get out of the situation. I'm not surprised in that sense because our politicians are gutless," he said.

Mr Nalliah said several senior Coalition figures had addressed Catch The Fire in the past, including former prime minister John Howard.

A Napthine government spokesman said because the event was now being hosted by Catch the Fire, Mr Clark would no longer attend the conference.

"In a democracy such as Victoria, all people are entitled to express their views within the law, and those who disagree should respond with argument and debate rather than by trying to prevent others from gathering to express their views," Mr Clark said.

Shadow Attorney-General Martin Pakula said it was no secret Catch the Fire was involved in the conference.

"Are we supposed to believe that suddenly this conference is beyond the pale because their premises are being used? It's a weak and bogus excuse by the Attorney-General," he said.