Conditions could not have been much worse for football when Swifts made the trip up to Jeparit on Saturday.
The Baggies had plenty to play for, with a golden opportunity to consolidate their spot in the league's top two.
They took on the Storm in more ways than one, met with heavy rain and bog-like conditions at the ground.
Despite the weather making play difficult, the visitors made the better start to the game, with Marty Carey kicking the only goal of the quarter to as Swifts lef 9-6 at the first break.
Baggies coach Scott Carey said his side should have done more in the first term.
"We had a bit of a breeze in the first quarter which was an advantage. We probably should have done more on the scoreboard than what we did though," he said.
"They were able to work defensively off the line so I think we let the advantage slip.
READ MORE: Full round 11 HDFNL scores
"We would have liked to come away with a three goal start so we went into quarter-time disappointed."
It was all downhill from there for the Baggies as Jeparit-Rainbow managed five unanswered second-quarter goals with the wind.
Swifts could only manage four more points after the first break for the remainder of the game, going down 7.4 (46) to 1.7 (13).
"When they had the wind advantage they made the most of it," Carey said.
"We were still able to get the ball inside 50 but just couldn't convert. In the end they were too strong defensively for us."
Swifts' score was their lowest in more than 10 years.
"Frustrating is probably the best way to describe the day. When you can only kick the ball 30 metres, you are missing handballs left and right and the ball spends most of the time on the ground, it isn't conducive to playing a good style of football," Carey said.
"Even quick kicks out of packs weren't working for either team."
READ MORE: Netball runs in family for Ralph
Carey said failing to take opportunities early on let his side down in the end.
"I really do put it down to that first quarter and taking opportunities. After half time only two goals were kicked so it could have been anyone's game if we made the most of those chances early," he said.
The Baggies dropped from second to third following the loss, leapfrogged by Edenhope-Apsley.
The Saints, Swifts and Natimuk United are now all level on 32 points each. The Rams and Baggies play an "eight-point match" at Natimuk next weekend.
"We had a setback on Saturday but we are still hoping to cement a top two spot moving forward," Carey said.
"They don't come much harder than Nati at Nati this week. Hopefully we can refocus and respond when called upon."
While you're with us, you can now receive updates straight to your inbox each Friday morning from the Stawell Times-News. To make sure you're up-to-date with all the news from across the Northern Grampians shire, sign up here.