RedPajama replicates LLaMA dataset to construct open supply, state-of-the-art LLMs

Thought the open supply AI references to camelids had been completed? Think once more: Yesterday, Together, a Menlo Park, California-based firm centered on constructing a decentralized cloud and open supply fashions, introduced RedPajama (sure, like Llama Llama Red Pajama) yesterday.

“In many ways, AI is having its Linux moment,” the corporate mentioned in a weblog submit, linking to a January submit written by Chris Re, co-founder of Together, Stanford affiliate professor and co-founder of SambaNova, Snorkel.ai and Factory.

RedPajama is a collaborative undertaking between Together, Ontocord.ai, ETH DS3Lab, Stanford CRFM, Hazy Research, and MILA Québec AI Institute to create main, totally open-source massive language fashions (LLMs). Its effort started with yesterday’s launch of a 1.2 trillion token dataset that follows the LLaMA recipe. The knowledge allows any group to pre-train fashions that may be permissively licensed. The full dataset is out there on Hugging Face and customers can reproduce outcomes with Apache 2.0 scripts out there on Github.

LLaMA is a state-of-the-art foundational LLM launched in February by Meta with gated entry to researchers. Several different fashions based mostly on LLaMA have come out in current weeks, together with Alpaca, Vicuna and Koala — however these fashions haven’t been out there for industrial use. There was additionally some LLaMA-drama when the LLaMA mannequin was leaked on 4chan.

In the approaching weeks, Together will launch a full suite of LLMs and instruction tuned variations based mostly on the RedPajama dataset. The firm emphasised that the forthcoming fashions shall be totally open-source and commercially viable. In a tweet, the corporate mentioned, “We hope this can be a clean-room, drama-free version. The RedPajama models we release, starting in the coming weeks, will be released under the Apache 2.0 license.”

RedPajama a part of a wave of open supply AI

As VentureBeat reported final week, open supply AI has been having a second over the previous few weeks, following the wave of LLM releases and an effort by startups, collectives and teachers to push again on the shift in AI to closed, proprietary LLMs. 

And a camelid-adjacent mannequin, Dolly 2.0 (as in Dolly the Sheep), additionally made headlines final week when its developer, Databricks, known as it the primary open, instruction-following LLM for industrial use.

But the biggest, state-of-the-art open supply LLMs like LLaMA have been restricted to the analysis group. “They are limited in that you can’t build real applications and ship them,” mentioned Vipul Ved Prakash, founder and CEO of Together and beforehand cofounder of Cloudmark and Topsy. “We think having permissively licensed models is a critical aspect of open source AI.”

Replicating the LLaMA dataset was no small job

The firm began with LLaMa, which it known as the “leading suite of open base models,” as a result of it was skilled on a “very large dataset that was carefully filtered for quality.” Also, the 7 billion parameter LLaMA mannequin is “trained for much longer, well beyond the Chinchilla-optimal point, to ensure the best quality at that model size.”

While neither the dataset nor the mannequin shall be an identical, the builders intention to create a completely open supply copy of LLaMA which might be out there for industrial purposes, and supply a “more transparent pipeline for research.”

The builders didn’t have entry to the LLaMA dataset however had sufficient of a recipe to go on. “We followed the recipe very carefully to essentially recreate [the LLaMA dataset] from scratch,” mentioned Prakash. The dataset consists of seven knowledge slices, together with knowledge from Common Crawl, arxiv, Github, Wikipedia and a corpus of open books.

“For each data slice, we conduct careful data pre-processing and filtering, and tune our quality filters to roughly match the number of tokens as reported by Meta AI in the LLaMA paper,” learn the weblog submit.

“All of the data LLaMA was trained on is openly available data, but the challenge was that they they didn’t provide the actual data set — there’s a lot of work to go from the overview to the actual data set,” mentioned Prakash. For instance, he defined, the paper may describe how they picked the very best 10,000 from one million paperwork, however they didn’t provide the 10,000. “So we followed the recipe to repeat all that work to create an equivalent dataset,” he mentioned.

The debate over constructing clear programs

Prakash mentioned that the RedPajama undertaking collaborators imagine it’s essential that programs are clear. “You know exactly how this model was built, what went into it,” he mentioned. “If you’re trying to improve it, you can start from the dataset.”

The undertaking additionally brings collectively a bigger group to those fashions, he added. “I would say academia has really been cut out of foundation model research because of the level of resources required, starting from data to the compute,” he mentioned. He added that there’s a small variety of folks on the earth engaged on these massive fashions immediately, and if there was broader entry, “a lot of brilliant people” around the globe would be capable to discover totally different instructions of neural architectures, coaching algorithms and security analysis.

“Also, this is one of the first really general AI which can be adapted to different tasks, and we think the applicability is very broad,” he mentioned. “But many different applications are possible only if you have access to the model, the model weights, and adapt them to different computing environments. We see a lot of this happen because of open source AI.”

There are one other facet to the open supply AI debate, nevertheless. For instance, Ilya Sutskever, OpenAI’s chief scientist and co-founder, not too long ago mentioned it was “wrong” to share analysis so brazenly, saying worry of competitors and fears over security — had been “self-evident.” He added that “at some point it will be quite easy, if one wanted, to cause a great deal of harm with those models.”

And in a current interview with VentureBeat, Joelle Pineau, VP of AI analysis at Meta, mentioned that whereas accountability and transparency in AI fashions is crucial, the important thing for Meta is to stability the extent of entry, which might differ relying on the potential hurt of the mannequin.

“My hope, and it’s reflected in our strategy for data access, is to figure out how to allow transparency for verifiability audits of these models,” she mentioned, including that entry may very well be determined based mostly on the extent of potential hurt of the mannequin.

On the opposite hand, she mentioned that some ranges of openness go too far. “That’s why the LLaMA model had a gated release,” she defined. “Many people would have been very happy to go totally open. I don’t think that’s the responsible thing to do today.”

Debates round moral datasets as properly

There have additionally been debates concerning the ethics of the datasets themselves, whether or not the fashions are open or closed. An article final week in The Guardian mentioned that the “enormous datasets used to train the latest generation of these AI systems, like those behind ChatGPT and Stable Diffusion, are likely to contain billions of images scraped from the internet, millions of pirated ebooks, the entire proceedings of 16 years of the European parliament and the whole of English-language Wikipedia.”

But Prakash says that he thinks “these models capture in some ways the output of human society and there is a sort of obligation to make them open and usable by everyone.” He added that “most of the magic” of those fashions comes from the truth that they’re skilled on “really broad and vast” knowledge.

He additionally identified that the unique knowledge is compressed considerably within the precise mannequin. The RedPajama dataset is 5 terabytes, and the fashions could be as small as 14 GB, ~500x smaller than the unique knowledge they’re modeling.

“This means that knowledge from the data is abstracted, transformed and modeled in a very different representation of weights and biases of parameters in the neural network model, and not stored and used in its original form,” mentioned Prakash. So, it’s “not reproducing the training data — it is derivative work on top of that. From our understanding, it is considered fair use as long as the model is not reproducing the data — it’s learning from it.”

There is little question that the open supply AI debates are highly-complex. But when requested why the corporate known as the brand new undertaking RedPajama, the reply was way more easy. “A lot of us have small children,” mentioned Prakash. “It just seemed fun.”