Jerome Pesenti has a few reasons to celebrate Meta’s decision last week to release Llama 3, a powerful open source large language model that anyone can download, run, and build on.
Pesenti used to be vice president of artificial intelligence at Meta and says he often pushed the company to consider releasing its technology for others to use and build on. But his main reason to rejoice is that his new startup will get access to an AI model that he says is very close in power to OpenAI’s industry-leading text generator GPT-4, but considerably cheaper to run and more open to outside scrutiny and modification.
“The release last Friday really feels like a game-changer,” Pesenti says. His new company, Sizzle, an AI tutor, currently uses GPT-4 and other AI models, both closed and open, to craft problem sets and curricula for students. His engineers are evaluating whether Llama 3 could replace OpenAI’s model in many cases.
Sizzle’s story may augur a broader shift in the balance of power in AI. OpenAI changed the world with ChatGPT, setting off a wave of AI investment and drawing more than 2 million developers to its cloud APIs. But if open source models prove competitive, developers and entrepreneurs may decide to stop paying to access the latest model from OpenAI or Google and use Llama 3 or one of the other increasingly powerful open source models that are popping up.
“It’s going to be an interesting horse race,” Pesenti says of competition between open models like Llama 3 and closed ones such as GPT-4 and Google’s Gemini.
Meta’s previous model, Llama 2, was already influential, but the company says it made the latest version more powerful by feeding it larger amounts of higher-quality training data, with new techniques developed to filter out redundant or garbled content and to select the best mixture of datasets to use.
Pesenti says running Llama 3 on a cloud platform such as Fireworks.ai costs just a 20th of the cost of accessing GPT-4 through an API. He adds that Llama 3 can be configured to respond to queries extremely quickly, a key consideration for developers at companies like his that rely on tapping into models from different providers. “It’s an equation between latency, cost, and accuracy,” he says.
Open models appear to be dropping at an impressive clip. A couple of weeks ago, I went inside startup Databricks to witness the final stages of an effort to build DBRX, a language model built that was briefly the best open one around. That crown is now Llama 3’s. Ali Ghodsi, CEO of Databricks, also describes Llama 3 as “game-changing” and says the larger model “is approaching the quality of GPT 4—that levels the playing field between open and closed-source LLMs.”
Llama 3 also showcases the potential for making AI models smaller, so they can be run on less powerful hardware. Meta released two versions of its latest model, one with 70 billion parameters—a measure of the variables it uses to learn from training data—and another with 8 billion. The smaller model is compact enough to run on a laptop but is remarkably capable, at least in WIRED’s testing.
Two days before Meta’s release, Mistral, a French AI company founded by alumni of Pesenti’s team at Meta, open sourced Mixtral 8x22B. It has 141 billion parameters but uses only 39 billion of them at any one time, a design known as a mixture of experts. Thanks to this trick, the model is considerably more capable than some models that are much larger.
Meta isn’t the only tech giant releasing open source AI. This week Microsoft released Phi-3-mini and Apple released OpenELM, two tiny but capable free-to-use language models that can run on a smartphone.
Coming months will show whether Llama 3 and other open models really can displace premium AI models like GPT-4 for some developers. And even more powerful open source AI is coming. The company is working on a massive 400-billion-parameter version of Llama 3 that chief AI scientist Yann LeCun says should be one of the most capable in the world.
Of course all this openness is not purely altruistic. Meta CEO Mark Zuckerberg says opening up its AI models should ultimately benefit the company by lowering the cost of technologies it relies on, for example by spawning compatible tools and services that Meta can use for itself. He left unsaid that it may also be to Meta’s benefit to prevent OpenAI, Microsoft, or Google from dominating the field.
Meta, the parent company of Facebook, recently unveiled its latest open-source language model called Llama 3, which is poised to pose a strong challenge to OpenAI’s dominance in the field. This development comes as a significant milestone in the ongoing race to develop advanced artificial intelligence (AI) models that can understand and generate human-like text.
OpenAI, founded by Elon Musk and Sam Altman, has been at the forefront of AI research and development for several years. Their language model, GPT-3 (Generative Pre-trained Transformer 3), has garnered immense attention and acclaim for its ability to generate coherent and contextually relevant text across a wide range of topics. However, Meta’s Llama 3 aims to surpass GPT-3’s capabilities and establish itself as a formidable competitor in the AI landscape.
Llama 3 builds upon the advancements made by its predecessors, Llama and Llama 2, which were also open-source language models developed by Meta. These models were designed to improve natural language understanding and generation, with a focus on addressing biases and ethical concerns associated with AI systems. Llama 3 takes this mission further by incorporating state-of-the-art techniques and leveraging Meta’s vast resources to enhance its performance.
One of the key differentiators of Llama 3 is its ability to understand and generate text in multiple languages. While GPT-3 supports several languages, Llama 3 boasts an even broader linguistic coverage, making it more accessible and useful for users worldwide. This feature opens up new possibilities for cross-cultural communication, translation services, and global collaboration.
Another notable aspect of Llama 3 is its improved contextual understanding. Language models often struggle with accurately interpreting context and producing coherent responses. However, Llama 3 leverages Meta’s extensive data resources and advanced training techniques to better comprehend nuanced contexts, resulting in more accurate and contextually appropriate outputs.
Furthermore, Llama 3 aims to address the issue of bias in AI systems. OpenAI’s GPT-3 has faced criticism for perpetuating biases present in the training data, leading to potentially harmful or discriminatory outputs. Meta has taken a proactive approach to mitigate this problem in Llama 3 by implementing rigorous bias detection and mitigation techniques. By actively working towards reducing biases, Meta aims to create a more inclusive and equitable AI model.
Meta’s decision to release Llama 3 as an open-source project is also significant. This move allows researchers, developers, and enthusiasts worldwide to contribute to the model’s development, making it a collaborative effort. Open-source models foster innovation and knowledge sharing, enabling the AI community to collectively improve upon existing models and push the boundaries of AI research.
While Llama 3 shows great promise, it is important to note that the competition in the AI landscape is fierce. OpenAI’s GPT-3 has already established itself as a benchmark for language models, and other companies like Google and Microsoft are also actively working on their own AI models. The success of Llama 3 will depend on its ability to outperform existing models, gain widespread adoption, and continuously evolve to meet the ever-growing demands of the AI industry.
In conclusion, Meta’s open-source language model, Llama 3, presents a formidable challenge to OpenAI’s dominance in the field of AI. With its enhanced linguistic coverage, improved contextual understanding, and focus on addressing biases, Llama 3 aims to push the boundaries of what AI models can achieve. As the competition intensifies, it will be fascinating to witness how Llama 3 evolves and contributes to the advancement of AI technology.
