Feedback happens when an output, like speakers, gets fed back into an input, like a microphone. Typically, the point of a speaker is to amplify its input, so the output gets louder, which goes into the input, and comes out louder, and goes into the input... Eventually, the signal reaches the limits of either the input or output system, and things stabilize (or break). I was curious though, why feedback has its characteristic sound. White noise is so-called because it's an equal distribution across all frequencies, just like white light. Feedback has a specific tone because of the properties of microphones and speakers.
The sounds we hear are rarely single-frequency – They're usually spread over a range with different intensities. This is how you can tell the difference between two instruments playing the same note. Different shapes and materials have different harmonics.
Microphones on computers are designed to pick up human speech, which is mostly in the range 150 Hz to 4 kHz:
where L is the loudness of the sound, p_rms is the average pressure of the sound wave, and p_ref is a reference pressure. Every 10 decibels corresponds to a pressure 10x greater.
Computer microphones are designed to pick up the same range of frequencies:
Notice though, there's a slight uptick around 5 kHz. We can compare that to the output of computer speakers,
This also rises around 5 kHz, which leads to the typical high-pitch – For reference, the highest note on a piano is around 4 kHz.
Of course, understanding how it works doesn't make it any less irritating!