"A big, strong, good-looking man," says Paul Tatchell​, a former mayor of Moorabool shire and himself a victim of rape by a notorious Catholic priest in Ballarat – Edward Dowlan​ – in the bad old days. Pell was on Richmond's reserves list to play footy 20 years before. He gave up sport for the church. But he had kept in shape.

"He was never short of admirers," says Tatchell. Pell, now the Vatican's head of finance, would have been 38, 39 years old. "He was in very good nick. Everything he did was done with a grandeur – and he reckoned he went all right."

Many of the Catholic schools nearby used Eureka as their pool, but the best time to be there was summer school holidays and Pell was always there then, too.

But Tatchell, who estimates he has helped 60 survivors of sexual abuse by Ballarat clergy since the 1990s, never saw or heard anything untoward. "I've never heard his name mentioned once with this kind of thing," he says. "I hate the bloke, but this caught me by surprise."

Maureen Hatcher used to go to the pool as a child in the late 1970s too. Most of Ballarat East did. She recently founded the Loud Fence movement in Ballarat, which has hung coloured ribbons on the fences of 15 schools and Catholic churches where abuse occurred. Now there are ribbons on the pool fence too. She grew up as a close family friend of one of the men, Lyndon Monument, who alleged to the ABC that Pell abused him by putting his hand down the front of his bathers during a game in the water.

She says it was very unusual to see a priest at the pool. It was unusual to see a priest out of his collar and robes, let alone getting around in bathers. She first started to hear word of the new allegations a year ago. "You hear whispers," she says.

Both she and Tatchell say that when the royal commission into sexual abuse by clergy was sitting in Ballarat, and when Pell won a claim to not return to Australia to give evidence, anger and outrage grew, and the indications of new claims – against him and others – grew too.

"There was a sense he was avoiding Ballarat," Hatcher says. At the same time, Victoria Police called for anyone who had information about alleged sex crimes at the Eureka pool from 1977 until 1980 to come forward. Monument and his co-accuser, Damian Dignan, were both pupils at St Alipius primary school.

The ABC also broadcast allegations that Cardinal Pell stood in front of a group of boys while naked in the Torquay Life Saving Club's changing rooms in 1986-87.

Cardinal Pell has denied all the allegations and accused the ABC and Victoria Police of collusion. He has not been charged. Allegations about him sexually abusing boys first emerged in the late 1990s when he was Archbishop of Sydney.

Of the latest claims from the pool, Ms Hatcher says: "I am not surprised. These days I am not shocked by anything to do with child sexual abuse. This is Ballarat now."

She says she has "no reason to disbelieve" her friend Lyndon Monument's story. But she says there is a danger is pursuing the "big scalp" of Cardinal Pell, one of the most senior Catholics in Rome and a divisive figure due to his perceived arrogance.

"Behind all the noise over this," she says, "are two guys with their own stories and their own lives. I prefer to concentrate on them rather than anything else."