I guess the delay is largely the cumulative effect of buffering at the various steps in the distribution network. It appears that my player (Xmms/Linux) uses a 24kbyte network buffer; that is 12 seconds of data. The servers in the distribution network have something like 8kbyte buffers, good for 4 seconds of delay.
The delay is mostly due to the encoding on the source end and the decoding at the client end. But there is some additive delay as the stream goes from the origin server out to the slave (listener) servers.
If you absolutely must have minimal delay try the alt.liveatc.net server - it should be very low delay (just checked at 1-2 seconds):http://alt.liveatc.net/kbos_twr
(note: do not use the .m3u extension on this server)