Companion Paradox: agency in the age of relatable AI

Weeknotes 352 - Companion Paradox: agency in the age of relatable AI - GPT-5 is everywhere in the news. It delivers and it underwhelms, depending who you ask. I dived in two the impact on being a relatable AI.

Interpretation by Midjourney

Dear reader!

Thanks for landing on the newsletter. The second edition of the new format. If you just arrived back from vacation, check last week’s newsletter for more background. The TL;DR is that I stopped using Target is New, and brought this more in line with what it always was: a weekly newsletter with my personal reflections on last week’s news, with — mainly — a lens on human-AI-thing-collaborations. After a general look back, I dive right into my triggered thought, and the rest follows after.

If you are in a place that is heating up in the coming days, enjoy, or take care, depending on your preferences and specific situation.

Week 352: Companion Paradox: agency in the age of relatable AI

GPT-5. It is there and it is all over the news, analyses, opinions, not only from tech critics but especially from the users. There is an inevitable new reality now, but in response to some pushback from people missing their trusted 4o model, there is a semi-rollback; you can choose, for now, to switch on the so-called legacy models. What does this all say? I try to make some sense out of the below in my triggered thought.

Another thing that got some attention in the Netherlands was a prediction of economic consequences by Andy Xie, predicting an AI arms race and an AI bubble-bursting. It was part of an online Tegenlicht edition, definitely worth watching. I especially find his argument about the importance of AI in the physical space relatable. That is, of course, the core of Cities of Things, and I am focusing on that even more now, with the immersive AI that I mentioned before.

I did not share last week that I made a last-minute proposal for the SXSW panel picker process (the voting is open now). I think it sums it up nicely:

When AI embeds in every object around us, it creates immersive worlds that fragment communities through addictive loops of immediacy, empathy, and physical presence. This workshop explores how "civic protocol economies" can redirect these same forces toward collective flourishing. (…). Together, we'll prototype how neighborhoods can transform AI from an extractive force into collective infrastructure.

I am not sure how it will resonate with the SXSW crowd, but it was a nice way to focus my thinking. There is much more to say about last week, but I keep that to the news captures below. And the triggered thought.

This week’s triggered thought

GPT-5's launch has sparked varied reactions—some find it substantial, others underwhelming. Rather than a revolutionary model, it's more an integration of existing capabilities made accessible to the masses through the free tier. The most intriguing aspect is how it affects our relationship with AI tools. Will it indeed become more of a partnering model? The rapid adoption of ChatGPT as a search engine alternative reveals a significant shift in how we access information. But beneath this functionality lies a deeper question: how relatable should our AI companions be?

With GPT-5, advanced reasoning capabilities once reserved for specialized, paid tools have been seamlessly integrated into everyday interactions. The system now invisibly decides which approach to take with our queries—sometimes employing deep reasoning, other times providing straightforward generative responses. This opacity raises concerns about user agency and transparency. DeepSeek demonstrated that many users appreciate seeing AI's reasoning process unfold—making the technology more relatable and trustworthy. This "co-performance" creates a partnership feeling that enhances our engagement. But OpenAI has chosen a different path: hiding the complexity beneath a smooth interface.

This design choice connects directly to the broader debate around AI's future. As discussions of "AI bubbles" and AGI (Artificial General Intelligence) intensify, we should question whether AGI is a meaningful concept or merely marketing terminology that serves corporate interests. The race toward superintelligence distracts from more immediate concerns about human agency.

The real concern isn't AI overtaking humanity through some paperclip-maximizing scenario, but rather humans gradually surrendering their agency by delegating an increasing number of decisions to AI systems. A danger lies in becoming dependent on technologies we no longer fully understand or control. As discussed last week, this is also part of the rhetoric of Sam Altman (from OpenAI), I think this is a topic to consider. An interesting counterpoint emerges from comparing Western and Eastern approaches to AI. While Western companies focus predominantly on language and knowledge models (brain-oriented), Eastern developments emphasize physical AI and robotics. This distinction matters profoundly.

Making and using tools represents a fundamental expression of human intelligence. The next evolutionary step may be AI systems that create their own tools—a capability that would further blur the boundaries between human and machine cognition. This self-extending functionality will reshape our relationship with technology. The ideal path forward combines transparency with agency. AI systems should provide debriefs about their reasoning processes, make assumptions explicit, and engage users in meaningful dialogue rather than simply delivering answers. I like to connect this to the notion of “thing-centered-design”, an approach where you learn more from human-technology interactions by impersonating the technology and observing the human in its behavior. When AI inevitably becomes an integral part of our environment, it will constantly capture, interpret, and respond to our world. Truly relatable AI must not only understand our context but reveal its own reasoning—creating a mutual interpretative relationship where both human and machine acknowledge their limitations and perspectives.

Physical AI may ultimately prove more transformative than language models alone. By embodying intelligence in the physical world, these systems must contend with real-world constraints and feedback—potentially creating more robust and genuinely helpful forms of artificial intelligence that complement rather than replace human capabilities. As we navigate this rapidly evolving landscape, the key question isn't whether AI will achieve human-level intelligence, but whether we'll maintain the agency and critical thinking necessary to shape these technologies according to truly human values and needs.

Notions from last week’s news

Human-AI partnerships

A collection on GPT-5 reactions. Positive, skeptical, and the backlash that made OpenAI bring back the possibility to choose the legacy models. It scores well in the hallucination index.

GPT-5 is being released to all ChatGPT users (The Verge)
GPT-5: Key characteristics, pricing and model card (Simon Willison)
The New ChatGPT Resets the AI Race (The Atlantic)
GPT-5: It Just Does Stuff (Mollick)
GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it. (Gary Marcus)
Apple brings OpenAI’s GPT-5 to iOS and macOS (Ars Technica)
Tests reveal that ChatGPT-5 hallucinates less than GPT-4o did – and Grok is still the king of making stuff up (Techradar)
OpenAI will not disclose GPT-5’s energy use. It could be higher than past models (The Guardian)
OpenAI Brings Back Fan-Favorite GPT-4o After a Massive User Revolt (Gizmodo)
Can GPT-5 build better dev tools for itself? Does it improve its coding performance? (Latent Space)
The Complete Guide to Personalizing ChatGPT-5, Including Fixes and Workarounds for the Top 10 Post-Launch Complaints (Nate B. Johnson)

Not only GPT-5 was released, but Google DeepMind also introduced Genie 3, “a new frontier for world models”. It impresses especially with the continuity in what is added to the models. That is one of the hardest parts.

Anthropic is now remembering your past chat history.

“The most important strength of AI is its ability to understand and act in the 3D physical world, not just answer questions.”

Doomprompting. A new concept is born.

Kevin Kelly is also connecting the real intelligence AI with acting in the physical space.

Context engineering as lens for a different way of prompting

Robotic performances

As more autonomous vehicles hit the road, it will make sense to combine people and package moving, much like the stagecoaches of the Wild West.

Alignment of AI could be inspired by ants, which make smart decisions together without a leader.

The apparently first store for humanoids. Of course, in China.

Immersive connectedness and collectivity

This work by Yuri Suzuki is visually pleasing as always, and connects to stories of collectivity and connectedness.

An update on the immersive connectedness standard Matter, focusing on reliability

Will Alexa benefit from an AI injection?

Wifi reached low-end household appliances

Tech societies

Studying their consistent behaviors could be a new kind of psychology, helping us understand how they influence our world.

Are AI disagreements within the community, are there hidden agendas?

Cloudflare’s ruling that Perplexity behaves sketchy, scraping the internet, makes it harder for Apple to do a possible acquisition. On the other hand, with Dia introducing a paid model, it might make sense to rebuild Safari as a Dia / Comet inspired browser soon.

Zitron is also discussing the AI bubble.

The future is unsure

Not a surprise, but the Trump AI strategy is not really effective..

How a new search can reshape democracy.

Use AI to give the dead a voice. It feels like a Black Mirror episode.

No surprises: the new Product Management will be AI-Native. Or something between embedded AI and team AI.

Will the glass slab finally be there?

For fun. And proving the wisdom of the crowds in a certain way:

Weekly paper to check

Many new papers arrive on different aspects of AI. One that stood out, curious to check the taxonomy sometime:

This report provides a comprehensive taxonomy of LLM hallucinations, beginning with a formal definition and a theoretical framework that posits its inherent inevitability in computable LLMs, irrespective of architecture or training. It explores core distinctions, differentiating between intrinsic (contradicting input context) and extrinsic (inconsistent with training data or reality), as well as factuality (absolute correctness) and faithfulness (adherence to input).

Cossio, M. (2025). A comprehensive taxonomy of hallucinations in Large Language Models. arXiv preprint arXiv:2508.01781.

What’s up for the coming week?

I might check this online gathering on the augmented city this Thursday. I did not notice anything else, but I might have missed some. Otoh, it will only start kicking in a couple of weeks.