This “affective caricature” approach involves musical symbols, affective notations, and caricature humor. First, rather than inventing new symbols, I adopted existing symbols in Western music notation — including music notes, dynamics, and breath marks. Second, I played around with visual primitives as position, size, shape, and color to elicit a strong emotional response.The third idea of ‘caricature humor’ encouraged me to exaggerate the visual elements to more immediately appeal to emotions.
This example conveys auditory salience with visually striking objects. For instance, very high or very low pitches are brought out by using an “exponent” (inverse-log) frequency scale on the vertical axis, such that if a woman makes a shrill, high-pitched scream in her laughter, it should standout by its positive vertical offset. Saliency in the perceived intensity of laugh pulses (syllables) and breathing noises are expressed using the exaggerated size of red note-heads and blue breath-marks, respectively. The anomalies in the quality of the laugh pulses (syllables) can be conveyed by the unexpected coloring of the note-heads. (For instance, the differences from statistical norm of the first, second, and third formants of the pulse can each be mapped to the R, G, B components, respectively, of the note-head color.) Finally, the possibly-exaggerated intensity contour of the pulses can be conveyed by the black enveloped regions inside the red note-heads, as illustrated above.
This transcription system is clearly a waste of energy if our goal is to accurately represent the laughter signal. However, if we take a perceptual (or perceptually salient) perspective, to communicate visually what is acoustically striking about a laughter, this may be an interesting and fun direction to take, one that complements the traditional approach using waveforms and spectrograms.
