A systematic procedure to develop sonifications
The amount of data being processed today is steadily increasing, and both socienty and science need new ways to explore their data. In SysSon, we worked on a systematic procedure to introduce sonification - the acoustic analogue to visualization - to new scientific fields. Sonification takes advantage of human perceptual capabilities of the sense of hearing.
We tested our approach in a case study with climate scientists at the Wegener Center for Climate and Global Change. Climate data are an ideal use case for sonification because of typically large and multivariate data sets which are difficult to visualize; data are inherently dynamic (just as sound is); and the appealing nature of sound supports communication to a broad public.
SysSon software has been released in November 2015 (Vs. 1.7.3.) as open source software resulting from the project (see github for information and download). It allows to sonify and plot climate data (format: NetCDF files; the tool allows also to convert spreadsheets into NetCDF). Three template sonifications are provided online for immediate use with an interactive GUI. Furthermore, the tool is extendable with any sonification patch over an interactive shell. The SysSon wiki includes the User Manual, Software installation, files for the Science By Ear Workshop 2014, and the Release Workshop 2015.
SysSon was a cooperative project with three institutions involved:
Institute for Electronic Music and Acoustics (IEM)
The project SysSon was mainly hosted at the IEM - a multidisciplinary institution, whose general mission is to bridge the gap between science and the arts in the field of audio engineering. In the last years, the institute developed a research focus on sonification.
Hanns Holger Rutz
Wegener Center for Climate and Global Change (WegCenter)
The WegCenter is an interdisciplinary, internationally oriented research center at the University of Graz focusing on "Climate, Environmental, and Global Change". In total, 25 members of its staff took part in the SysSon project as test subjects.
Andrea K. Steiner
Centre for Systematic Musicology (SysMus Graz)
SysMus Graz conducts research and teaching in all areas of systematic musicology, focusing on music psychology. The centre also promotes interdisciplinary interactions among systematic musicology, its “parent disciplines” (e.g. psychology), and other musical disciplines (e.g. ethnomusicology).
- Vogt K., H. H. Rutz, V. Goudarzi, R. Parncutt, A. K. Steiner, M. Jury & R. Höldrich, Enabling scientists to sonify their data: General considerations based on a case study with climate scientists, submitted to Displays (ISSN 0141-9382), 2015.
- Goudarzi, V. Contextual Inquiry for a Climate Audio Interface, In Human-Computer Interfaces and Interactivity: Emergent Research and Applications, Eds.: Pedro Isaías, Blashki, Katherine, 2014.
- Vogt, K., Goudarzi, V., & Parncutt, R. Evaluating Aesthetics in Sonifications. Eds. P. Strumiłło, M. Bujacz, & M. Popielata, in Proceedings of the International Conference on Auditory Display ICAD-'13 (pp. 175 - 179). Łódź: Łódź University of Technology Press, 2013.
- Goudarzi, V. & Vogt, K. Contextual Inquiry for a Climate Audio Interface, In Proceedings of the Interfaces and Human Computer Interaction, IHCI 2013, Prague, Czech Republic.
- Goudarzi, V., Rutz, H. & Vogt, K., User Centered Audio Interface for Climate Science, In Proceedings of the Interactive Sonification Workshop, ISon 2013, Erlangen, Germany.
- H. H. Rutz, K. Vogt, R. Höldrich, The SysSon platform: A computer music perspective of sonification, in Proc. of the 21st International Conference on Auditory Display (ICAD-'15), Graz, 2015, pp. 188–192.
- V. Goudarzi, K. Vogt, R. Höldrich, Reflections on an interdisciplinary design process using a sonification framework, in Proc. of the 21st International Conference on Auditory Display (ICAD-'15), Graz, 2015.
- Goudarzi, V., SYSSON: A Systematic Procedure to Develop Sonifications. ACM CHI Extended Abstracts, CHI 2013, Paris, France.
- Vogt, K., Goudarzi, V. & Höldrich, R., SYSSON: A Systematic Procedure to Develop Sonifications, Ext. Abstract in Proceedings of the International Conference on Auditory Display, ICAD 2012, Atlanta, USA.
- Vogt K., H. H. Rutz, V. Goudarzi, R. Höldrich, A. K. Steiner, M. Jury, R. Walter, & R. Parncutt, SysSon - A Framework for Systematic Sonification Design, Poster presented at EGU-European Geosciences Union, Vienna, Austria, 2015.
- Goudarzi, V., Rutz, H. , & Vogt, K., SysSon: A Sonification Platform for Climate Data, Poster presented at EGU-European Geosciences Union, Vienna, Austria, 2014.
- Rutz, H. H., V. Goudarzi, K. Vogt, A. K. Steiner, & M. Jury, SysSon: A Systematic design for the Sonification of radio occultation data, Poster presented at the International Workshop on Occultations for Probing Atmosphere and Climate - OPAC-IROWG 2013 , Seggau Castle, Austria, September 2013.
- Vogt, K. & Goudarzi, V., Sonification of Climate Data, Poster presented at EGU-European Geosciences Union, Vienna, Austria, 2013.
Workshop - Science By Ear III
An Interdisciplinary Sonification Workshop
Date: September 3-5, 2014
Location: IEM Graz, Inffeldgasse 10/III
This workshop will bring together international experts in the field of sonification with distinguished scientists from climate science. Relevant sonification tools and example data sets will be discussed. Afterwards, participants will work in interdisciplinary teams of sonification experts, data scientists, and audio programmers.The hack session will entail the development of two tasks that include sonification strategies and experimentation with the SysSon platform via iterative coding. Researchers from Wegener Center for Climate and Global Change will have the opportunity to make firsthand experiences with SysSon platform for research questions from their fields; sonification experts will receive valuable feedback on their sonification design approaches. Furthermore, we hope to gain insights on interdisciplinary collaborative approaches.
Results are discussed in this paper.
Turbulence. A Climate Sound Portrait
Forum Stadtpark, 08 – 29 November 2014
Tue-Fri 11 AM – 6 PM, Sat 11 AM – 4 PM
Opening: Fr 07.11.2014, 7:00 PM
How could the complex global phenomenon of climate be experienced in an individual and sensory way? This is the question explored by this exhibit, evolved as a cooperation between the research project SysSon and the ADRIART – Life Long Learning project. A floating topography invites the public to an immersive and interactive experience. The space is pervaded by a composition of audible climate data, where sensors generate impulses to modulate the sound, capturing physical displacements and tangible contact.
SysSon Release Workshop
Release Workshop with three template sonifications
A workshop on using the SysSon software (in Vs. 1.7.3) was given at Wegener Center for Climate and Global Change in November 2015. The software was introduced by the examples of the template sonifications (a tour of plots, sounds, and videos of the templates is here):
- Augmented Audification: (Download and description; Example Sounds)
The sonification targets one-dimensional data. It can be played directly as so-called "audification", using a normally high sampling rate (in the order of several thousand samples per second), or zoom into a region of the data set with a pitch shift and modulation, leading to an "auditory graph".
You may find details on the sound synthesis in this paper.
- Resonant Rain: (Download and description; Example Sounds)
This sonification model produces an overall sound texture based on two different but synchronised (time wise) variables. It was created for data centred around zero, i.e. anomaly data. The sound is that of a rain with resonant frequencies, whereby one variable controls the rain texture and the other controls the type of resonances.
- Pitch Layers: (Download and description; Example Sounds)
This sonification targets anomaly data with at least two dimensions (e.g., time and height levels). The variable is considered symmetrical around a threshold, for example zero for anomaly data. The variable's magnitude is used to define the amplitude (volume) of each sound component. Sound components are sine oscillators whose frequency depends on the signum of the variable and the layer.
SysSon is now installed on the server of Wegener Center. It is freely downloadable here.
Basic sonification designs for the SysSon software
This page gives example sounds for each of the templates.
The templates (as well as the SysSon software) may be downloaded from github, where you also find their documentation.
Augmented Audification - Example Sounds
The sonification targets one-dimensional data. It can be played directly as so-called „audification“, using a normally high sampling rate (in the order of several thousand samples per second), or zoom into a region of the data set with a pitch shift and modulation, leading to an „auditory graph“.
You may find details on the sound synthesis in this paper.
For the examples here, we used data from the WegenerNet:
- Air temperature (at 2m) and surface temperature in Feldbach from 11/2007 to 11/2008
- Air temperature (at 2m) and surface temperature in Feldbach from 11/2011 to 11/2012
(hourly values over one year; missing values have been interpolated before)
Pure audification of hourly air temperature over one year with a sampling rate of 44.1kHz:
Reading out the data slower, e.g., with one fourth of the original sampling rate (11 kHz) is still a highly compressed signal in time, but some differences can be heards (year 07-08 seems to be a rougher sound).
Zooming into the data set, e.g. listening to 732 data points (corresponding to one month) per second would lead frequencies far beyong the human audible range. Therefore, we use augmented audification, with a frequency shift of 300Hz and a pitch modulation factor of 1 octave. With this setting, the yearly cycles can be compared to each other.
With an even slower read-ut of 168 data points (i.e., one week) per second, we compare the two Januaries. The more jittering („complaining“) sound of the year 07-08 as compared to the year 11-12 hints to a large storm that took place in the beginning of January 08, with more quick temperature changes involved.
Pitch Layers - Example Sounds
The PitchLayers template was created for anomaly data, and may show the quasi biennal oscillation (QBO), which is found in higher altitudes (given as pressure levels) around the equator, and results in an a-synchronous rhythm of roughly 2.5 years as compared to the ticking signal that indicates one year.
When playing all pressure levels are played, the lower glissandi are not clear anymore.
This behaviour is not at all reproduced in mid-latitudes (15 deg N - 35 deg N).
Raster - Resonant Rain - Example Sounds
Resonant Rain is the most complex template provided here. It can be used for any two-dimensional data set with the anomalies of two variables: our example plays precipitation and temperature data, who influence the sound quality in various ways.
The calibration file is adjustable to understand the different sounds of the sonification.
The basic sound with both anomalies around zero. This is the neutral, rather calm sound.
If both rain and termperature are unusually low, the sound is scarcer and has lower resonances.
If both rain and temperature anomalies are high, the resulting sound is dense and high-pitched.
Accordingly, unusually low precipitation paired with high temperatures, results in a scarce, high pitched sound…
… and vice versa.
These sounds span the timbre space in which the sonification realizes the sounds from real data.
We may compare the anomalies in the Alps region (3 months per second, from 2015 to 2020):
…with the same period in Indonesia, where anomalies are much more pronounced:
- We have set up a Soundcloud account where you can listen to sound samples.
- Klima|Anlage - Performing Climate Data: the interactive sound installation Klima|Anlage has been developed in cooperation with the SysSon project (involved: Katharina Vogt, Hanns Holger Rutz, Andrea K. Steiner, Martin Jury).
- The University for Music and Performing Arts Graz and the University of Graz launched a call for climate related artwork in November 2015, KlimARS2016. (Involved from SysSon: Katharina Vogt, Andrea K. Steiner)
- Deutsches Klimarechenzentrum
- The Human Cost of Poverty and Climate Change
- Wegener Net - local climate measurement stations in the region of Feldbach (Styria, Austria)