When PC's first came on the scene they had no more than 640k ram, and
those were the deluxe models.
The first IBM XT (Affectionately named 'The Tank' because it was light
military green and had a solid steel case) only had an 8k BIOS chip, with
512k (Half a megabyte) becoming the standard when the compatible wars
With these first PC's there was no concept of the hard disk, hard disks
where large multi platter things that looked like a cake shelf with a glass
cover over (Like you might see in a coffee shop) and were usually so heavy
that they took 2 or 3 people to lift them.
Often these large platter cases where only ever connected to mainframes
at the time, and where way too large to even consider being provided for a
desktop PC, so the floppy disk was used instead.
The first ever round of these floppy disks could hold no more than
approx 300k and were 5.25 inches square by about 2.5 millimeters thick,
some were double sided, so could hold 600k. There was also a range of
different types of software and disk encoder chips that could read/write
different densities, but the bottom line was space on them was very, very
Couple that with the fact, that most BIOSes at the time where only
around about 16k to 32k in size, you had to fit as much as possible to get
the machine up and running , and in the case of the IBM fit a rom based
basic interpreter in there too so that with no external operating system,
the computer was still usable for general computing tasks.
All of these restrictions meant that smaller was better.
Rather than have a flattened disk size with a large monolithic loader
on, it was better to 'format' these floppy disks so that most of the space
was user space, and allowed the end user to effectively customize the boot
software (EG: remove parts of the OS they didn't use) so the initial loader
that kick started everything was restricted to the first 512 bytes of the
The other main reason was the sheer verity of different disk systems
that were available back then (Remember this was way before the industry
was standardized) so putting something right at the beginning of the disk
was guaranteed to be found no matter how weirdly the running OS set the
rest of the disk up for it's own use, because no seeking was involved, you
didn't have to look at format marks and attempt to understand some weird
directory format. You simply moved the drive head to it's rest position,
then read 512 bytes, simple as that.
Once phoenix produced the first clone of the IBM Bios, and won over IBM
in court when the company tried to sue them for theft of intellectual
property, the flood gates opened. Almost overnight, everyone started to
make BIOS systems, and the PC market as we know to day exploded into a mess
of standards in interfaces of all different types.
Pretty soon, the vendor lock-in started, so the IEE/ANSI/ISO and other
standards bodies started to lay down the law, by making a specification on
how EVERYONE had to remain compatible with everyone else, those standards
have held true right through too today's modern era of computing.
By the time we got massive hard drives on the scene and bootable CD's,
USB's and all manner of other things, these standards (of which the 512b
boot sector was part) where SO DEEPLY entrenched in the grander scheme of
things, that it was impossible to change them.
Only a few brave companies dared venture into that territory and had
limited success, Sun Micro Systems for example were one of the brave few.
If you look at a Sun Raq3 (You can pick them up on Ebay for next to
nothing) they have a boot loader that mimics exactly what the PC disk based
loader does, but it boots out of rom until the second stage , where up on
it immediately looks for a /boot partition on a standard Linux based disk
layout, thus even though disks in these machines still have the standard
512 byte boot block, it's actually not used.
Hope that gives you the insight you want, given that I lived through a
lot of it I can remember a lot of it too including the machines that didn't
use the 512 byte boot block.