Carnegie Mellon University’s Robotics Institute has a brand new artist-in-residence.
FRIDA, a robotic arm with a paintbrush taped to it, makes use of synthetic intelligence to collaborate with people on artistic endeavors. Ask FRIDA to color an image, and it will get to work placing brush to canvas.
“There’s this one portray of a frog ballerina that I feel turned out actually properly,” mentioned Peter Schaldenbrand, a School of Computer Science Ph.D. pupil within the Robotics Institute working with FRIDA and exploring AI and creativity. “It is absolutely foolish and enjoyable, and I feel the shock of what FRIDA generated based mostly on my enter was actually enjoyable to see.”
FRIDA, named after Frida Kahlo, stands for Framework and Robotics Initiative for Developing Arts. The undertaking is led by Schaldenbrand with RI college members Jean Oh and Jim McCann, and has attracted college students and researchers throughout CMU.
Users can direct FRIDA by inputting a textual content description, submitting different artistic endeavors to encourage its fashion, or importing {a photograph} and asking it to color a illustration of it. The crew is experimenting with different inputs as nicely, together with audio. They performed ABBA’s “Dancing Queen” and requested FRIDA to color it.
“FRIDA is a robotic portray system, however FRIDA shouldn’t be an artist,” Schaldenbrand mentioned. “FRIDA shouldn’t be producing the concepts to speak. FRIDA is a system that an artist may collaborate with. The artist can specify high-level objectives for FRIDA after which FRIDA can execute them.”
The robotic makes use of AI fashions much like these powering instruments like OpenAI’s ChatGPT and DALL-E 2, which generate textual content or a picture, respectively, in response to a immediate. FRIDA simulates how it could paint a picture with brush strokes and makes use of machine studying to guage its progress as it really works.
FRIDA’s remaining merchandise are impressionistic and kooky. The brushstrokes are daring. They lack the precision sought so usually in robotic endeavors. If FRIDA makes a mistake, it riffs on it, incorporating the errant splotch of paint into the tip consequence.
“FRIDA is a undertaking exploring the intersection of human and robotic creativity,” McCann mentioned. “FRIDA is utilizing the sort of AI fashions which have been developed to do issues like caption pictures and perceive scene content material and making use of it to this creative generative downside.”
FRIDA faucets into AI and machine studying a number of instances throughout its creative course of. First, it spends an hour or extra studying how one can use its paintbrush. Then, it makes use of giant vision-language fashions educated on large datasets that pair textual content and pictures scraped from the web, corresponding to OpenAI’s Contrastive Language-Image Pre-Training (CLIP), to know the enter. AI techniques use these fashions to generate new textual content or pictures based mostly on a immediate.
Other image-generating instruments corresponding to OpenAI’s DALL-E 2, use giant vision-language fashions to provide digital pictures. FRIDA takes {that a} step additional and makes use of its embodied robotic system to provide bodily work. One of the largest technical challenges in producing a bodily picture is decreasing the simulation-to-real hole, the distinction between what FRIDA composes in simulation and what it paints on the canvas. FRIDA makes use of an concept often called real2sim2real. The robotic’s precise brush strokes are used to coach the simulator to mirror and mimic the bodily capabilities of the robotic and portray supplies.
FRIDA’s crew additionally seeks to handle among the limitations in present giant vision-language fashions by frequently refining those they use. The crew fed the fashions the headlines from information articles to offer it a way of what was taking place on the earth and additional educated them on pictures and textual content extra consultant of numerous cultures to keep away from an American or Western bias. This multicultural collaboration effort is led by Zhixuan Liu and Beverley-Claire Okogwu, first-year RI grasp’s college students, and Youeun Shin and Youngsik Yun, visiting grasp’s college students from Dongguk University in Korea. Their efforts embrace coaching information contributions from China, Japan, Korea, Mexico, Nigeria, Norway, Vietnam and different international locations.
Once FRIDA’s human person has specified a high-level idea of the portray they need to create, the robotic makes use of machine studying to create its simulation and develop a plan to make a portray to attain the person’s objectives. FRIDA shows a colour pallet on a pc display screen for a human to combine and supply to the robotic. Automatic paint mixing is presently being developed, led by Jiaying Wei, a grasp’s pupil within the School of Architecture, with Eunsu Kang, college within the Machine Learning Department.
Armed with a brush and paint, FRIDA will make its first strokes. Every so usually, the robotic makes use of an overhead digital camera to seize a picture of the portray. The picture helps FRIDA consider its progress and refine its plan, if wanted. The entire course of takes hours.
“People marvel if FRIDA goes to take artists’ jobs, however the principle objective of the FRIDA undertaking is kind of the alternative. We need to actually promote human creativity by FRIDA,” Oh mentioned. “For occasion, I personally wished to be an artist. Now, I can really collaborate with FRIDA to precise my concepts in portray.”
More details about FRIDA is obtainable on its web site. The crew will current its newest analysis from the undertaking, “FRIDA: A Collaborative Robot Painter With a Differentiable, Real2Sim2Real Planning Environment” on the 2023 IEEE International Conference on Robotics and Automation this May in London. FRIDA resides within the RI’s Bot Intelligence Group (BIG) lab within the Squirrel Hill neighborhood of Pittsburgh.