I have a USB camera attached to a remote computer. To listen to its audio simply install SoX on the remote machine and locally (via MacPorts on OS X), and run the following command:
ssh XXX.XX.XX.XXX "sox -t alsa plughw:1,0 -p" | sox - -tcoreaudio
where XXX.XX.XX.XXX is the IP of the remote machine and plughw:1,0 is an Alsa device name on that machine.