Exploring nonlinear time within interactive and adaptive electronic music composition

          This practice as research PhD will employ game development software to compose interactive and adaptive electronic music compositions. The outcome will be a portfolio of music presented as a mobile music app where users can actively engage with and affect the music output through body movement.

The interactivity and adaptability will provide a basis to study nonlinearity in music. “Linearity and nonlinearity are the two fundamental means by which music structures time and by which time structures music.” (Kramer, 1988: 20). The following aspects of nonlinearity in music will be studied: moment form, modules, gestural, and vertical time. Moment form is a compositional style built on a“sequence of short, self-contained sections ("Moments"), which do not depend on a previous or a following Moment” (Chang, n.d). Modules are described as “passages that derive from the iterationand reiteration of a pattern or set of patterns” (Paynter 2015: 50). Gestural time “depends partially on temporal logic that is nonlinear and subjective” (Kramer, 1988: 11). Vertical time is “static, sonically orconceptually, and thus depends wholly on the listener to create its hierarchies and contrasts” (Arauco,1990: 155).

Nonlinearity itself can be thought of as events determined by general principles, as opposed to beingdirectly affected by previous events (Kramer, 1988) and is a wider subject than that of nonlinear time in music. De Landa (2000) considers nonlinear elements in history and philosophy. This research project will study philosophical ideologies in nonlinearity and alongside will examine the temporal spaces created through the active engagement of participants. Kinetic gestural interaction will be used to define the engagement of participants through body movement. Collins describes kinetic gestural interaction as bodily participation with sound through physical and gestural mimicry and determines we should re-evaluate the connection of participants, as the player is no longer a passive listener (Collins, 2007).

There is crossover between interactive game audio and nonlinear time, primarily through the requirement for interactive audio to adapt to circumstance within gameplay resulting in a nonlinear approach to composition. Game development software has built-in functionality to design interactive and adaptive audio making it an important exploratory device within this project. Unreal Engine 4 (UE4)*, Wwise** middleware and digital audio workstation (DAW) Ableton Live*** will be used in the app creation. The software will function as a research tool offering insights into nonlinear time from both composers and listeners perspectives.

The result is not intended to be a game, instead, the principle will be based on a ludic system where participants experience the music through play. This ludic system will create temporal structures for“interactive participant[s] within the ludic world” (Lindley, 2004: 183). 

There is a strong correlation between the interactive loop-based nature of game music and Ableton Live’s loop-based interface. At GDC 2017, Senior Audio Programmer Aaron McLeran presented the future of audio within UE4 and showcased its integration with Ableton Live, demonstrating the possibility of creating a bespoke software instrument within UE4 using the synthesis plug-in (UnrealEngine, 2017). It will be possible to extrapolate the music insights gained and apply them in other areas within interactive and electronic music. Examples from further afield include deep learning technology (artificial intelligence) in music composition (Weng & Chen, 2020; BBC, 2020) and in-game concerts (Arrigo, 2020).

IMPORTANCE AND ORIGINALITY OF THE RESEARCH

“The typical computer program does not exemplify solely linear thinking... [a] program consists of doubling back, of loops within loops, of branching off in different directions. It is thus an apt symbol for contemporary temporality” (Kramer, 1988: 13). Kramer’s work was at a time when there was a “lack of analytic tools for the study of nonlinear music.” (Arauco, 1990: 155) Since then, the advancement of computer software has created a tangible opportunity to investigate nonlinear structures in electronic music composition.

Notable research in nonlinearity includes Kaae’s chapter in From Pac-Man to Pop Music: Interactive Audio in Games and New Media (Kaae, 2008), Paynter’s Form and Process in Morton Feldman’s Springof Chosroes (Paynter, 2015), and Kramer’s Postmodern Concepts of Time (Kramer, 1996). Eugene Holland’s Deleuze and Guattari’s A Thousand Plateaus discusses the wider relationship between music and nonlinearity comparing jazz improvisation to a rhizome, which in itself has no linear progression (Holland, 2013).

Being Time presents insights into the temporal perception of music and covers recent electronic music composition. The authors state their work is “an investigation on musical behaviours that have an impact on temporality” which “provides evidence of the continued validity and utility of exploring particular works in-depth and developing an ongoing conversation about them” (Glover Gottschalk & Harrison, 2018: 9). An investigation into temporality and nonlinearity, especially the electronic music field, by using game development software, will provide key insights and add positively to the research already conducted.

As well as offering a clear definition to kinetic gestural interaction, An Introduction to the Participatory and Non-Linear Aspects of Video Games Audio is a relevant paper commenting on the research conducted into non-linear time in music thus far. “Although there has been significant academic research into related areas and audio in terms of technology, communication and development, work into the sonic aspects of audio-visual media has neglected games” (Collins, 2007: 1).

Reviewers (Arauco, 1990; Butler, 1990; Brown, 1990) consider Kramer’s The Time of Music a key and in-depth work on nonlinearity. The book discusses aspects of nonlinearity in music including the impact of technology, temporal multiplicity, duration, and proportion. Since publication, the subject has evolved and the work does not cover electronic music in depth, mention video games, or discuss interactive and adaptive music, which is the area this PhD project seeks to address.