A Tectonic Shift in AI: How One Million Tokens Can Change Everything

Breaking Down the Million-Token Wall: A New Era for AI?

There's been a tectonic shift in AI that's poised to change everything. The "memory wall" of language models, traditionally limited to just a few thousand tokens, is about to get demolished.

Picture this: a language model capable of remembering an entire conversation, the full text of a novel, or even an extensive medical history. This isn't just tech fantasy anymore. A recent study has painted a vivid picture of an AI future where models remember not just a few thousand, but a staggering one million tokens.

For those not steeped in the jargon of AI, let me unravel the term "token". It's essentially a unit of data that an AI processes, often a word or part of a word. As it stands, models like GPT-3.5 and GPT-4 can remember 4096 and 32000 tokens respectively, which equates to roughly the content of 6 to 48 pages of text. Imagine, then, a language model capable of remembering the equivalent of 20 novels or 1000 legal case briefs. We're talking about an AI that could read and retain a patient's entire health history. Such a leap forward could unlock a cornucopia of opportunities across various sectors.

So, what are these possibilities? Let's take a deep dive.

1. Personalized Recommendations
Imagine an AI that doesn't just suggest random products or services, but understands your past preferences and tailors its suggestions accordingly. No more wading through irrelevant options. Just straight-up, "made-for-you" recommendations that hit the bullseye every time.

2. Medical Diagnosis Assistance
Doctors could have a new assistant that has read and remembers every detail of a patient's health history, making diagnoses faster and more accurate, leading to improved patient outcomes. It's like having Dr. House on your medical team, minus the sarcastic one-liners.

3. Legal Research Aid
Lawyers say goodbye to endless hours of trawling through case law. With a million-token memory, a language model could swiftly identify relevant precedents, building stronger cases and driving better outcomes for clients. Legal research could become a walk in the park, rather than a marathon.

4. Mental Health Counseling
Personalized mental health support could be transformed with an AI that remembers all past interactions, providing guidance tailored to individual needs. Imagine having a therapist in your pocket, ready to offer advice whenever you need it.

5. Creative Writing
For the writers among us, an AI capable of remembering a million tokens could maintain continuity in storytelling and character development. No more plot holes or inconsistent character arcs; just a seamless narrative from start to finish.

Of course, a future painted in broad strokes of optimism must also be tempered with caution.

The advent of a million-token memory model presents challenges, too. Privacy, data security, and misuse of technology are all concerns that need to be addressed as we venture into this brave new world of AI. The memory wall may be coming down, but we need to ensure that it's replaced with robust safeguards that protect users and their data.

Nonetheless, the transformative potential of a million-token memory model is undeniable. It's an exciting time in the world of AI as we stand on the cusp of a revolution that could redefine how we interact with technology. Buckle up, tech enthusiasts. It's going to be one hell of a ride.

AI replacing jobs in the wild…

Saw this automated floor cleaner in the Menards store today. Completely autonomous, running on BrainOS.

OpenAI: Vision and Volition

In a recent dialogue with Sam Altman, the president of OpenAI, the future trajectory of the organization was unveiled with an unexpected degree of candor. From GPU limitations to grand plans for GPT-4, the conversation covered a gamut of developer concerns and broader societal implications of AI.

The GPU Gordian Knot
A significant revelation was OpenAI's struggle with GPU limitations, a concern echoed by customers regarding the reliability and speed of the API​. The coveted 32k context, a feature eagerly anticipated by many, is currently held at bay by these constraints. The organization is grappling with the O(n^2) scaling of attention, implying the need for a significant research breakthrough to increase the token context windows beyond 100k - 1M, a goal they aim to achieve within the year​.

Additionally, the finetuning API is being bottlenecked by GPU availability, with efficient finetuning methods yet to be adopted. However, Altman has hinted at the future inclusion of a marketplace for community-contributed models, broadening the scope for shared innovation.

Roadmap Ruminations
Altman detailed OpenAI's near-term roadmap for the API, which included ambitions for a cheaper and faster GPT-4, longer context windows, an extended finetuning API, and a stateful API that remembers conversation history. The roadmap also indicated a move towards multimodality, but this would be dependent on the availability of more GPUs.

Apprehensions and Assurances
The conversation also addressed a common developer apprehension: fear of OpenAI becoming a competitor. Altman sought to assuage these fears by stating that OpenAI would not release more products beyond ChatGPT, thus avoiding competition with customers using OpenAI's APIs. The goal for ChatGPT is to be a "super-smart assistant for work," leaving plenty of other GPT use cases for developers to explore.

Regulation and Open Source: A Delicate Balance
Altman's views on the regulation of future AI models and the role of open-source were also thought-provoking. While he called for the regulation of future models, he didn't view existing models as dangerous, suggesting it would be a mistake to regulate or ban them. At the same time, he expressed skepticism about the feasibility of open-sourcing GPT-3, questioning how many entities could effectively host and serve large Language Models.

The Scaling Hypothesis: Not Over, Yet
Defying popular belief, Altman maintained that the era of giant AI models is far from over. OpenAI's internal data suggests that the scaling laws for model performance still hold. However, the rate of scaling can't be maintained as it was in the past. Instead of growing by orders of magnitude, models are likely to double or triple in size each year. The persistence of these scaling laws has significant implications for the timelines of AGI (Artificial General Intelligence) development, suggesting that AGI may be closer than we think.

In conclusion, OpenAI's future, as outlined by Sam Altman, is as ambitious as it is pragmatic.

With an acute awareness of the challenges and a clear vision for the future, the organization is set to navigate the complex landscape of AI development, keeping its mission of ensuring that artificial general intelligence (AGI) benefits all of humanity at the heart of its operations.

Changes coming this month

For members, I am starting up the live training sessions this month. Also if you would like to book a one-on-one call with me, let me know.

For email subscribers, good news (I think). I am going to be creating multiple levels within the Discord server for paid members versus Email Subscribers so we can create a better community around all of this and motivate me to put more stuff I find and experience up on the Discord group.

We have a good number of subscribers now, so I am hoping we can get some activity and engagement up there when I open it up.

Hope you enjoy your weekend! We are checking out for a couple days for a little R&R before my next treatment next Friday.

Until Monday,
Kevin Davis

P.S. If you noticed the broken link to the video walkthrough yesterday, check out the web version with the correct link in place to see the demo.