Music scholars who engage in performance analysis are interested in examining performance characteristics such as style and expressiveness. They will typically collect multiple recorded performances of the same piece and compare their nuances. They will also often compare recordings of the same piece from different decades in order to discover historical trends—and if the performances are by the same artist, they might also use these recordings to study this artist’s stylistic changes over time.
Currently, it is the case that music scholars will frequently conduct such research by manually annotating musical elements such as downbeats, pitch, and note onset times in performance recordings. The arguably most popular software tool for annotating performances is Sonic Visualiser, which allows multiple annotations to be visualized together and all annotations to be exported into an Excel file for further quantitative analyses. However, such manual approaches to annotation can be tedious and inefficient. And more importantly, existing tools do not incorporate musical score information, an element that is often vital when analysing performances.
Dr. Yucong Jiang, seeking to provide better tools for music performance analysis, conducted a 13-month research project hosted at the mdw’s Department of Music Acoustics and funded by a prestigious EU Marie Skłodowska-Curie Actions (MSCA) postdoctoral fellowship. The main output of this project, which concluded successfully in July, is a prototype open-source software application called Piano Precision that is designed to facilitate performance analysis. When provided with a digital musical score and a performance of that score, this software automatically generates meaningful AI-driven annotations concerning note onsets and tempi that are fully integrated with the score. With its easy-to-use graphical user interface, the software displays and visualizes the score, the audio, and the annotations in intuitive ways. For example, to quickly jump to the playback time of a note in the score, users may simply click on that note in the score. The image below shows Piano Precision’s user interface.
The automatic annotation functionality is accomplished by an AI component in the software that runs an audio-to-score alignment algorithm. Given a digital musical score and a recording thereof, the algorithm automatically finds the onset time of each note in the score. Based on these onset times and the score information, the software calculates the local tempo at each note and displays an intuitive tempo curve to show tempo fluctuations throughout the performance.
Of course, no AI algorithm is perfect—so in cases where detected onsets are misaligned with the actual performance, the software allows users to correct them in an intuitive manner by dragging the onset indicators with the assistance of highlighted notes in the score and text labels representing the positions of selected notes. Users can then export these validated onsets and tempo annotations to an Excel file for further analysis. These annotations are valuable to performance-related research as timing is an essential aspect of musical expression.
Piano Precision is provided under a public license and is free for anyone to use; see the project page.
The audio-to-score alignment technology comes from the Music Information Retrieval (MIR) research community, where leading music technologies such as automatic music transcription and optical music recognition are invented and developed. To make it easier to connect with the MIR community, the audio-to-score alignment component in Piano Precision is designed as a plugin—meaning that it can be easily replaced by or switched to other MIR community alignment algorithms without modifying the underlying code. This project represents a meaningful step toward bridging the divide between MIR and performance analysis research.
Piano Precision uses the MEI (Music Encoding Initiative) digital score input format, which is becoming increasingly popular among music scholars. This makes for good synergy at the Department of Music Acoustics – Wiener Klangstil, as the department has multiple MEI-related projects and can boast leading experts in this area such as Werner Goebl and David M. Weigl.
Yucong Jiang conducted a user study of Piano Precision with 15 prospective users. The feedback was broadly positive, confirming this tool’s value for research in the field of performance analysis, and it also provided important insights for the future refinement of such tools.
