This summer I had the chance to attend BTS's Arirang tour stops in Brussels, Belgium and I found a few people to wear sensors with me to the show. Wearing Equivital sensor vests and movesense units, they shared their breathing, body motion, and heart rates during the 3.5 hour performance. The data collected was so compelling that it inspired a new strategy for capturing concurrent musical behaviour between participants.

At some kinds of music concert, the audience is very active. They sing along, move to the beat, and cheer together for the people on stage. These active behaviours can be highly synchronised, but they don't have to be, at least not all of the time. In Kpop, there is usually a prescribed part for the audience to shout and sing with every track, and attendees commonly practice these parts before attending a show. However, the audience doesn't confine their participation to these parts, instead singing along to every lyric they can remember and waving their light sticks throughout.

So a question I could ask of these audience measurements is WHEN during the concert these 3-5 participants (depending on the signal, wee sensor issues) happened to synchronise in each of these measurements.

One participant's measurements during the first hour of the concert, showing changes in heart rate, body temperature, respiration wave and rate, body and wrist motion and body and wrist frequencies over matched constant Q spectrograms of the songs (reconstructed audio)

The Audience Physiology Signals

The audience behaviour signals to consider were:

  • Respiration wave: the chest stretch recordings from 3 participants (*4 if add second concert measurements from one participant), recorded at 25.6 Hz. This measurement particularly privileges inspiration timing and depth.

  • Body Quantity of Motion: 4 Hz jerk magnitude from 3D accelerometer recordings from 4 (*5) participants' upper torso. This captures when there are changes in body movement intensity.

  • Body Frequencies of motion: Power spectrum (FFT) on 256 Hz (*104Hz for one) accelerometer measurements, cut to frequency range [0.25-15Hz]. This captures when participants' bodies are expressing the beat.

  • Wrist Quantity of motion: 4 Hz jerk magnitude from 3D accelerometer recordings from 4 participants' wrists, mostly from their lightstick waving arm.

  • Wrist frequencies of motion: same feature as body frequencies.

  • Heart rate: specifically IBI timing, recorded from 3 (*4) and converted to 4 Hz beat-wise heart rate (after filtering for sensor noise and ectopic beats).

It's fair to wonder whether 3-4 participants is even enough to assess any kind of physiological synchrony. Research using seated participants usually depends on 2 to 20 times more bodies to find measureable shared information. But at a kpop concert, we expect audience members to spend a fair bit of time actively behaving in synchrony because participation is part of the experience. So few recordings might be enough to find something some long as there is a suitable comparison.

We want criteria for detecting when any given signal shows an "unexpected" amount of shared information across the participants. A simple way to do summarise that shared information is to take the average correlation between each pair of signals within a given time window. I've been critical of average correlations in other contexts, specifically across dozen of continuous ratings while applying inapproriate estimate of significance (Upham, 2012), but over a small time window and a small number of measurements and with a suitable null, this calculation can be quite helpful.

The Null reference set

Rather than parametricly estimate the likelihood of 4-way correlation values for each signal, we can use real measurements.

These participants wore sensors for at least an hour before the show started. During that time, they were walking, talking, standing in line, mostly waiting to enter the venue. They were all separated physically, not subject to the same environmental cues, and experienced some excitement and stress in anticipation of the performance. For each of the signals, participants produced a range of values during the time with natural oscillatory rates for each system. While the range of physical states are not quite as wide as captured in the venue, it's a start/

Calculating 4-way (and 3-way) correlation values on this pre-concert interval gives us a practical reference distribution for coincidental coherence rates in each signal. The 95th and 99th percentile coherence values are an empirical threshold for what can happen by chance, without some shared influence. If participants are showing a lot more coherence (quantity of time, not higher correlation values), that is a good reason to this concert is having a shared influence.

The Time Scale

If the goal is to identify WHEN coherence is high, the time interval over which coherence is assessed has to be relatively small. Taking these correlations over a whole song would be missing the temporal accuracy needed. Section changes in music tends to be quite fast and frequent, particularly in this genre, so if the point is to capture differences within songs, a time scale of 4-16 seconds would be useful.

It would be reasonable to consider different time-window sizes from different features, but for convenience, I've chosen the interval of 8 seconds for all.

Subset number (2-3-4)

To validate the relevance of the correlation values on this subset of the audience, we can actually test the distribution differences between the reference Null and the concert recordings for each signal as more recordings are added.

Distribution of coherence values on respiration signals across Null reference set (blue) and BTS Music intervals (red) for correlations in pairs, trios and sets of four on 8 second windows. The vertical solid and dotted lines mark the 95th and 99th percentile values, with accompanying numbers reporting ratio of concert data exceeding it.
Distribution of coherence values on Body quantity of motion across Null reference set (blue) and BTS Music intervals (red) for correlations in pairs, trios and sets of four on 8 second windows. The vertical solid and dotted lines mark the 95th and 99th percentile values, with accompanying numbers reporting ratio of concert data exceeding it.
Distribution of coherence values on Heart rates across Null reference set (blue) and BTS Music intervals (red) for correlations in pairs, trios and sets of four on 8 second windows. The vertical solid and dotted lines mark the 95th and 99th percentile values, with accompanying numbers reporting ratio of concert data exceeding it.
Distribution of coherence values on Body motion frequencies across Null reference set (blue) and BTS Music intervals (red) for correlations in pairs, trios and sets of four on 8 second windows. The vertical solid and dotted lines mark the 95th and 99th percentile values, with accompanying numbers reporting ratio of concert data exceeding it.

By considering correlations across a subset, this measure of coherence becomes more sensitive, detecting shared information through a bit more noise. Taken to the extreme, with dozens of measurements correlated at once, this sensitivity may work against interpretability, but at 3 or 4, the signal may be close enough to the surface to relate to recognisable behaviours.

Subset coherence during a song

To demonstrate how these calculations produce interpretable results, here is a summary figure of the measurements and coherence values for one track, Body to Body. This song is high energy, with moments when the audience is actively dancing ("I need the whole stadium to jump") but also includes an interval where they are asked to sing Arirang, a famous Korean folk song. Subset coherence successfully detects this switch of modalities for audience synchrony.

Composite graph showing measurement signals from participants during Body to Body at the July 1st Arirang Tour show and the coherence captured across each signal: Respiration wave, Heart rate, Body Quantity of Motion, Body frequencies, Wrist Quantity of Motion, and Wrist frequencies. On signal plots, the colour highlights mark the structure of the song via which BTS member is on mic and the x ticks indicate the onset of fanchant parts. The 8s corr values thicken at moments that exceed the 95th threshold and are red when they exceed 99th, with x ticks every 8 seconds to give a scale to the time windows of assessment.

This plot conveys a lot at once, but I'll highlight a few bits.

The body motion in this example also shows the benefits of the two measures: quantity of motion (BACC4Hz) and frequencies (BACCSpec). The quantity of motion coherence is sensitive to change in motion intensity, not reporting whether they are moving the same amount. This means coherence is fleeting, capturing the transitions rather than then intervals of motion, and we see QoM coherence go up with segment shifts, most dramatically at the end (155s) after the folk song section. The frequencies of motion show coherence when the dance motion is highest, showing similarity in between the transition when there is beat wise motion to capture. The wrist measurements show less over all coherence but follow similar trajectories.

The heart rate signals during this song are really interesting, but maybe not what we'd expect from the literature. Participants were perfroming a lot of coordinate behaviour during this song however, its expression through local changes in heart rate weren't so aligned. The body motion driven increases at the start and end of the song get picked up, but in between, similarity is mostly lost in the mixture of influences on heart rate across these three captured. When extreme respiration, body motion, and other factors of excitement are concurrently acting on an audience member's heart, the trajectory produced is depends on how these factors balance: how much they are moving, how hard they are singing, and how their specific bodies handle these demands.

For a song like this, the synchrony in behaviour is probably much more important for the crowds feeling of togetherness and collective effervescence than any parallels in cardiac activity. And yet, in this datasets, there are other songs where the cardiac coherence is very high! but more on that another time.

Conclusion

When the audience is active, coherence across 3 or 4 audience members is enough to find what is going on to the music. Coherence as measured with correlations on a small time window do not capture all similarities between participants, but it can highlight transitions in behaviour and musically relevant patterns within suitable physiological measurements.

Extending this analysis process to datasets with more participants is also quite easy, and will get written up soon.