Sketch a doodle of a drum or a saxophone to conjure a multi-instrumental composition. Look right into a webcam, converse, and watch your mouth go bouncing throughout the display screen — the enter for a collection of charmingly clunky chain reactions.
This is what guests to the MIT Lewis Music Library encounter after they work together with two new digital installations, “Doodle Tunes” and “Sounds from the Mouth,” created by 2022-23 Center for Art and Technology (CAST) Visiting Artist Andreas Refsgaard in collaboration with Music Technology and Digital Media Librarian Caleb Hall. The residency was initiated by Avery Boddie, Lewis Music Library division head, who acknowledged Refsgaard’s aptitude for revealing the playfulness of rising applied sciences. The intricacies of coding and machine studying can appear formidable to newcomers, however Refsgaard’s follow as a artistic coder, interplay designer, and educator seeks to open the sector to all. Encompassing workshops, an artist speak, class visits, and an exhibition, the residency was infused along with his distinctive humorousness — a mix of vigorous eccentricity and easygoing relatability.
Machine Learning and the Arts with MIT CAST Visiting Artist Andreas Refsgaard
Learning by means of laughter
Refsgaard, who relies in Copenhagen, is a real maverick of machine studying. “I’m interested in the ways we can express ourselves through code,” he explains. “I like to make unconventional connections between inputs and outputs, with the computer serving as a translator — a tool might allow you to play music with your eyes, or it might generate a love poem from a photo of a burrito.” Refsgaard’s explicit spin on innovation isn’t about instantly fixing issues or launching world-changing startups. Instead, he merely seeks to “poke at what can be done,” offering accessible open-source templates to immediate new artistic concepts and functions.
Programmed by Refsgaard and that includes a customized set of sounds created by Hall, “Doodle Tunes” and “Sounds from the Mouth” exhibit how unique compositions might be generated by means of a mixture of spontaneous human gestures and algorithmically produced outputs. In “Doodle Tunes,” a machine studying algorithm is educated on a dataset of drawings of various devices: a piano, drums, bass guitar, or saxophone. When the consumer sketches one among these photos on a touchscreen, a sound is generated; the extra devices you add, the extra complicated the composition. “Sounds from the Mouth” works by means of facial monitoring and self-capturing photos. When the participant faces a webcam and opens their mouth, an autonomous snapshot is created which bounces off the notes of a piano. To attempt the tasks for your self, scroll to the tip of this text.
Libraries, limitless
Saxophone squeals and digital drum beats aren’t the one sounds issuing from the areas the place the tasks are put in. “My office is close by,” says Hall. “So when I suddenly hear laughter, I know exactly what’s up.” This new sonic dimension of the Lewis Music Library matches with the ethos of the atmosphere as a complete — designed as a campus hub for audio experimentation, the library was by no means meant to be wholly silent. Refsgaard’s residency exemplifies a brand new emphasis on progressive programming spearheaded by Boddie, because the technique of the library shifts towards a give attention to digital collections and music know-how.
“In addition to serving as a space for quiet study and access to physical resources, we want the library to be a place where users congregate, collaborate, and explore together,” says Boddie. “This residency was very successful in that regard. Through the workshops, we were able to connect individuals from across the MIT community and their unique disciplines. We had people from the Sloan School of Management, from the Schwarzman College of Computing, from Music and Theater Arts, all working together, getting messy, creating tools that sometimes worked … and sometimes didn’t.”
Error and serendipity
The integration of error is a key high quality of Refgaard’s work. Occasional glitches are a part of the artistry, and so they additionally serve to softly undermine the hype round AI; an algorithm is just pretty much as good as its dataset, and that set is inflected by human biases and oversights. During a public artist speak, “Machine Learning and the Arts,” viewers members had been initiated into Refsgaard’s offbeat creative paradigm, offered with tasks similar to Booksby.ai (a web-based bookstore for AI-produced sci-fi novels), Is it FUNKY? (an try to tell apart between “fun” and “boring” photos), and Eye Conductor (an interface to play music through eye actions and facial gestures). Glitches within the exhibit installations had been frankly admitted (it’s true that “Doodle Tunes” often errors a drawing of a saxophone for a squirrel), and Refsgaard inspired viewers members to recommend potential enhancements.
This open-minded perspective set the tone of the workshops “Art, Algorithms and Artificial Intelligence” and “Machine Learning for Interaction Designers,” meant to be appropriate for newcomers in addition to curious specialists. Refsgaard’s visits to music know-how lessons explored the ways in which human creativity might be amplified by machine studying, and learn how to navigate the sliding scale between creative intention and surprising outcomes. “As I see it, success is when participants engage with the material and come up with new ideas. The first step of learning is to understand what is being taught — the next is to apply that understanding in ways that the teacher couldn’t have foreseen.”
Uncertainty and alternative
Refsgaard’s work exemplifies a few of the core values and questions central to the evolution of MIT Libraries — problems with digitization, computation, and open entry. By selecting to make his lighthearted demos freely accessible, he renounces possession of his concepts; a machine studying mannequin would possibly function a studying machine for a scholar, and it would equally be monetized by an organization. For Refsgaard, play is a method of partaking with the moral implications of rising applied sciences, and Hall discovered himself grappling with these questions within the course of of making the sounds for the 2 installations. “If I wrote the sound samples, but someone else arranged them as a composition, then who owns the music? Or does the AI own the music? It’s an incredibly interesting time to be working in music technology; we’re entering into unknown territory.”
For Refsgaard, uncertainty is the key sauce of his algorithmic artistry. “I like to make things where I’m surprised by the end result,” he says. “I’m seeking that sweet spot between something familiar and something unexpected.” As he explains, an excessive amount of shock merely quantities to noise, however there’s one thing joyful within the risk {that a} machine would possibly mistake a saxophone for a squirrel. The activity of a artistic coder is to repeatedly tune the connection between human and machine capabilities — to seek out and observe the music.
“Doodle Tunes” and “Sounds from the Mouth” are on show within the MIT Lewis Music Library (14E-109) till Dec. 20. Click the hyperlinks to work together with the tasks on-line.