Originally Answered: Why is the octet rule true? I want quantum mechanics level answers, not hokey 'atoms want' answers?
Oooooooooh. GREAT question.
A complete and precise answer will vary a bit depending on what atom in what situation you're talking about. (For example, in the gas phase, a sodium cation Na+ is NOT more stable than a neutral sodium atom -- the ionization energy of neutral gas phase sodium atoms is a positive number, so that change is uphill and endothermic. In that sense, the octet rule *isn't* true. As you point out, it's not true for the d-block, and lots of p-block elements break it all the time anyway.)
Let's take the simplest case of a noble gas atom, like neon. The valence configuration is (2s)2(2p)6, closed shell, complete octet, both stable (in the sense that it it cannot, by itself, turn into anything else that is thermodynamically downhill) and unreactive (in the sense that there is nothing with which it can chemically combine to form new molecules that are energetically downhill).
A noble gas has, as you state, four atomic orbitals in that particular shell. They aren't the same energy (the 2s are below the 2p), but those orbitals are both far far higher in energy than the n=1 and far far lower in energy than the n=3. At the same time, Ne has more protons (10) than any other element with an n=2 valence shell, and a higher Z means that the electrons in Ne experience a higher Zeff (effective nuclear charge) than any other element in that row.
The phrase "atoms want 8 electrons" really means "atoms seem to form compounds that allow them to put 8 electrons into their 4 valence orbitals, either by giving electrons away to form a cation, taking electrons to form an anion, or sharing electrons to form a covalent bond".
Neon won't do any of those things.
Its Zeff is the highest in the row, so the ionization energy is the highest in the row. It won't form cations, because it takes far more energy to remove electrons than you could get back via the electrostatic attractive forces in an ionic complex. There's no stable complex of [Ne]+, because making [Ne]+ costs too much, because the Zeff and ionization energies are too high.
Neon won't form anions, either, because making [Ne]- requires adding an electron to the much higher energy n=3 shell. That's also uphill, and again that's too much of an energy cost to regain. There's no stable complex of [Ne]-, because making [Ne]- costs too much, because the n=3 is too high in energy and the electron affinity is too unfavourable.
And Ne won't share. Its electrons are already in low-energy orbitals, the lowest energy orbitals of any n=2 element, and they are all full. So the usual driving force for covalent bonds -- overlap two half filled orbitals to put both electrons in a more stable bonding interaction -- doesn't work. Any covalent bond you formed with neon would require the overlap of a full orbital, resulting in partial population of antibonding MOs, which is too destabilizing.
So neon won't give 'em, won't take 'em, and won't share 'em. Strike three, no way to bond, no compounds. You're stable.
For elements *not* in Group 18, at least one of the scenarios above is a good idea. Sodium has a low ionization energy -- uphill, but not too far -- so that the electrostatic attractive forces in an ionic complex make the formation of [Na]+ a net downhill process. Fluorine or oxygen have either slightly favourable electron affinities to make [F]- or [O]2-, or, again, uphill by small enough amounts that ionic complexes are overall downhill. Or they can share: overlap the singly-occupied orbitals with other singly occupied orbitals from other elements to form covalent bonds until all the singly occupied orbitals have a bond. That will lower the energy of those electrons and fully occupy every orbital -- fewer bonds (less than an octet) would leave some orbitals at least partially empty, so there are spots you could add more electrons to further lower the energy of the system, while more bonds would require orbitals you don't have available. Net result, elements usually share to form the octet, and then stop.
Basically, it comes down to the orbital energy structure you already understand. Filling the valence level, either by taking or sharing electrons, is downhill, adding to the one above it is uphill.
Hope that makes sense.