A cool change has swept through Melbourne, with the temperature dropping more than 10 degrees in 90 seconds.

After a scorching and gusty day, the temperature fell to 23 degrees at 3.50pm after sitting at 34 degrees just minutes earlier.

The Bureau of Meteorology advised the cool change sweeping through the city in a north-easterly direction was expected to reach the outer suburbs by 5pm.

Senior forecaster Andrea Peace said the strong gusts in the CBD had calmed, ahead of a cool evening.

Temperatures will drop to 11 degrees on Tuesday evening, in stark contrast to Monday night's minimum of 22 degrees.

High winds in the city on Tuesday caused a three-metre glass panel to become dislodged at a high-rise construction site on La Trobe Street in the CBD.

Police blocked off the road between Queen and Elizabeth streets as the balustrade dangled precariously from the 19th storey and threatened to fall off the edge.

Hutchinson Builders closed the site earlier on Tuesday due to the dangerous weather, before the newly installed railing became dislodged in the strong wind about 1.30pm.

A fire brigade spokesman said crews had raced to the scene and managed to secure the panel. Roads were reopened shortly after 5.15pm, he said.

WorkSafe Victoria confirmed it was making inquiries into the incident.

Construction, Forestry, Mining and Energy Union secretary John Setka said the incident was "another reminder of the high-risk work" undertaken by construction workers on a daily basis.

"The dangers ... can extend to the general public when things go wrong," he said.

Meanwhile, 15 children were rescued from cars statewide in Monday and Tuesday's hot weather.

Melbourne reached 30 degrees at 11am on Tuesday - two hours earlier than it did on Monday.

By 1pm, the mercury had topped 35 degrees.

But it was the high winds whipping up dust that most affected those in the city.

A 121km/h wind gust was recorded at Mount Buller, while blasts up to 100km/h were expected in the south of the state throughout the day.

"The dust is unbelievable," said Bureau of Meteorology forecaster Andrea Peace.

"It's really not pleasant outside."

Fire warnings

Total fire bans have been enacted through most of the state, although the squally conditions are set to create headaches for firefighters, who are battling two separate blazes near Geelong and Lancefield.

The Bureau of Meteorology has issued a severe weather warning for damaging winds for most of the state.

"People need to be mindful today that this is the first really bad fire day for the year," she said.

"Winds like this, particularly with a strong change moving through does create big challenges for the fire agencies."

Winds will continue for most of the day, accompanied by a possible temperature surge just before the cool change, due in Geelong at 2pm and Melbourne at 3pm.