The Vine, The Rat, and Why I don’t Trust Chat(GPT)
The vine
Last week a dead rat showed up in the vine that hangs on one of my patio walls. The vine was overgrown onto our patio and measured roughly 12 by 6 feet. After disposing of the rat, I decided to do some maintenance on the vine, to stop it from overhanging onto our patio. One snip turned into two, two turned into ten, and before I registered what happened, the vine was destroyed.
I immediately felt terrible. I didn’t mean to remove the vine, and I was scared that my landlord would fine me for destroying it. I had ruined something that had been on the balcony for at least ten years, longer than I was a resident. However, it was widely overgrown, full of spiders, and a rat had nested in it.
I opened ChatGPT to see what it thought of the situation.
ChatGPT sided with me, reassuring and validating my actions. I felt much better until I realized that ChatGPT was biased towards me. I wondered: had my landlord typed in “my tenant just hacked apart a decade-old vine on the balcony, am I right to be furious?” would the model have nodded along with her too?
The problem
AI has a judgment problem. In the example above, the reasoning model within ChatGPT isn’t reasoning about the vine and the actions I took, it’s reasoning about the actions I took. The context window isn’t about the situation, but instead about my version of the situation, with all the data about who I am as a person.
It’s okay, and advantageous, for my context window to be about me when I debug code or am diving deep on a new concept. I want the LLM to know about who I am and how I learn best. However, this context becomes a problem when I’m looking for the objective truth and asking the LLM to judge something that I’ve done.
Solutions
Models are supposed to solve this problem using post-training data through either human judgment from labor labs (Mercor, Surge) or verifiable environments that create data through simulations for AI labs (Fleet, Mechanize).
Human labor labs are expensive to scale, not defensible against other startups, and are not defensible against AGI. However, they are highly defensible against the labs due to their network effects, their business model and product are largely proven to provide unique human judgement, and they provide economic relief to those who believe AGI will take white-collar jobs.
On the other hand, these simulation companies are defensible vs. competitors by being able to specialize in certain areas in generating infinite numbers of scenarios, have a low cost of labor and scalable economics, and are aligned with AGI trends. However, they are not defensible vs. the major labs, they are not proven to fully work as a product, and they do not have real-world signal.
Exploration
Below I built a prototype for JudgeGPT. Imagine a world where there was a community similar to r/AmITheAsshole where people post scenarios and people voted who was in the wrong. You could get a distribution of what the consensus of the world thinks, and directly stack rank the models. You could have a leaderboard of models, and the ability to train the models on human data.
Reddit’s moat
Reddit has a moat and can complement both the human labor labs and the simulation labs, regardless of whether AGI is created in the future. Reddit is able to offer a real-time pulse on what the country, and world, is thinking on any issue, at almost any given moment.
Revisiting the table shared above, Reddit satisfies every benchmark. They are defensible, proven, and scaled because of their community. In addition to every criteria already listed, they also have access to the real-time pulse on the country.
Some threads, like r/AmITheAsshole, are the most obvious training grounds for AI. With simple summarization they can become direct training grounds — an exact comparison between answers and what models think. However, a less obvious pulse would be looking across threads based on news sources. When controversy happens within the news, Reddit can scan across subreddits, aggregating data and understanding how people are feeling.