Vegeta was so obsessed with surpassing Goku in strength
that, even after seven years of peace on Earth and settling into fatherhood, he allowed Babidi to infiltrate his mind in exchange for more power. Despite Supreme Kai's repeated warnings that Majin Buu had to be stopped from hatching at all costs, Vegeta chose to ignore everything else going on just so he could fight Goku on a supposedly even footing.
Not only did Goku cede to his rival's demand, but he also chose to keep his recently discovered Super Saiyan 3 transformation a secret and fought Vegeta as a Super Saiyan 2 — knowing full well any damage he took would go straight to Majin Buu. In hindsight, this was an extremely odd decision. But Goku had his reasons for hiding Super Saiyan 3 at the time, some of which were more sensible than others.
For much of his time throughout Dragon Ball Z
, Vegeta is driven solely by his pride and desire for revenge against those who wronged him. His family was murdered, his home planet was destroyed and he had no known genuine friendships. Getting stronger in order to survive and become the best is what kept him motivated at all times. When a common-born Saiyan defeated him on Earth, this one thing that defined Vegeta had left was severely wounded. Even knowing how badly Vegeta desires to surpass him — going so far as to kill tournament spectators and get a powerup from an evil wizard — Goku still strove to protect Vegeta's pride.
Also, as Goku mentions later, using Super Saiyan 3 consumes the user's energy very quickly, and Goku's limited time on Earth would have run out that much faster. He didn't deem the battle against Majin Vegeta as being worth sacrificing the valuable time he had left on Earth, and he correctly considered — though underestimated — the possibility that the Babidi/Buu situation would grow more serious.
Why He Was Wrong
By Goku's own admission, his and Vegeta's most critical mistake was underestimating how strong Buu would be upon hatching. Rather arrogantly, they both thought they'd be able to duke it out while Gohan and Supreme Kai took care of Babidi and Dabura. This leads to Goku's second mistake: overestimating Gohan. His son was rusty from not training for seven years
, taking plenty of damage from Dabura and consequently having his energy sent to Majin Buu. Gohan needed his father's help for that battle and nearly died from not having it.
Had Goku realized this and taken the threat of Buu more seriously from the start, he could have transformed into Super Saiyan 3, defeated Vegeta quickly -- pride be damned -- then rushed over to help Gohan. Therefore Buu never hatches, the Earth and many other planets never blow up, and Goku's wife and friends are never into chocolate and eaten. But as Goku reveals later with Goten and Trunks
, he wanted the younger generation to get used to defending the Earth from evil beings on their own. Fate and circumstances decided rather differently in the end.
Why He Was Right
Had Goku gone Super Saiyan 3 and managed to defeat Majin Vegeta quickly, perhaps he could have ended Babidi's entire scheme before Buu hatched. But that was never a guarantee. Vegeta is nothing if not incredibly resilient, as he later reaffirmed in the fight against Kid Buu. Meanwhile, the situation could have worsened in a number of different ways -- which it did -- and then Goku would have had even less time to fight Buu, or train Goten and Trunks to fuse, or affect matters in another way before returning to the afterlife.
Also, Goku's transformation may well have permanently broken Vegeta. After sinking to such a point that he accepted help from an evil wizard to finally match his rival's strength again, only to discover that newfound power still doesn't even come close, how would Vegeta have reacted? Goku had all this in mind and felt the best way to handle a dark moment was to give Vegeta what he wanted: an even strength battle. Despite his numerous past transgressions and obsession with beating Goku, he didn't want to kill Vegeta or wound him permanently. Always looking for the best in others, he knows deep down Vegeta is a good person and valuable ally, as well as a loyal father and husband who loves his family.
Moving outside the specifics of right or wrong, if Goku uses Super Saiyan 3 and takes out Vegeta, the entire plot that leads to their teamwork and friendship -- likely the first in Vegeta's entire life -- can never happen. Likewise, Vegeta's emotional and fan-favorite atonement in sacrificing his own life for Bulma and Trunks also never occurs, nor does his heartfelt acceptance that Goku is the better fighter.
Was Goku's protection of his rival's pride -- and the friendship that formed in the end -- worth the eventual deaths of all their friends and family, and the destruction of Earth and many other planets along the way? Absolutely not -- unless you're in a universe where you can use dragon balls to restore an entire galaxy back to normal.