The driver, in his 50s, was so terrified he quit his job.

It was 5.40am on Sunday, March 9 during the long weekend.

The man, who asked to remain anonymous, said it was his first weekend night shift and his fourth day on the job.

He picked up two passengers from the taxi rank on Horsham's Wilson Street.

One of the men asked to be driven to McDonalds, the other to Dimboola.

"We went around the roundabout near Trev's, at the intersection of Wilson and Darlot streets, and the man sitting next to me started to get a bit mouthy," the alleged victim said.

"I told him to settle down, he was getting a bit wild."

He said the man was verbally aggressive throughout the journey, including towards McDonalds staff, despite repeated promises he would behave himself.

He also claimed the front seat passenger hit him in the face and pinned him to his seat.

The driver dropped off the two men in Dimboola, but did not wait around to find out if either man would pay the $72.30 fare.

"I just drove off and left them there," he said. "I was shaking like a leaf."

The driver quit his job two days later.

"I actually liked the job it was a good job," he said.

Horsham Taxi Service owner Max Hood said he was also disappointed.

"One bad egg has spoiled it for this particular driver," he said.

He hoped CCTV footage from the taxi rank would help police find the persons allegedly responsible.

Horsham Sergeant Michael Salter said police were investigating the alleged assault.

"We are also asking for people who are witnesses or with any information to call Horsham Police Station on 5382 9200," he said.