I'm sorry for the person who lost their stuff but this is a reminder that in 2025 you STILL need to know what you are doing and if you don't then put your hands away from the keyboard if you think you can lose valuable data.
You simply don't vibe command a computer.
Those aren't feelings, they are words associated with a negative outcome that resulted from the actions of the subject.
But also, negative feelings are learned from associating negative outcomes. Words and feelings can both be learned.
Sure, humans come with some baked in weights, but others are learned.
Like some people feel great joy when an American flag burns while others feel upset.
If you accidentally delete a friends hard drive you'll feel sad but if you were intentionally sabotaging a company you'll feel proud at the success.
i.e. joy and happiness are innate not learned.
Any relationships between certain words and a modified probabilistic outcome in current models is an artifact of the training corpus containing examples of these relationships.
I contend that modern models are absolutely capable of thinking, problem-solving, expressing creativity, but for the time being LLMs do not run in any kind of sensory loop which could house qualia.
> which we do not currently know how to precisely define, recognize or measure
is that if we don't know if something has qualia (despite externally showing evidence of it), morally you should default to treating it like it does.
Ridiculous to treat a computer like it has emotions, but breaking down the problem into steps, it's incredibly hard to avoid that conclusion. "When in doubt, be nice to the robot".
This is how people end up worshipping rocks & thunderstorms.
This would be like treating characters in a book as if they have real feelings just because they have text on the page that suggests they do.
But the story itself is a static snapshot of that dynamic system. Similar to how a photograph of a person is a static capture from a dynamic moment. The person in the photo has qualia, but the image of them (almost certainly) does not.
At least at a baseline, we would expect anything with qualia to be dynamic rather than static.
This does mesh with the Zodeistic framework I just mentioned in another reply to you. You could certainly isolate and describe the ideas behind those characters, how they live within the author's mind, and how the book codifies an interaction between those ideas.
Extending further: I think there is more evidence that SpongeBob SquarePants is real, than that he is not real. A significant portion of organisms I personally know have structures in their brain which are able to simulate imagery and behavior of SpongeBob at will, reciting memories and generating new states of SpongeBob. AI is now like doing this shit on crack.
He's an enduring cultural archetype, a distributed organism(s), lossily replicated and encoded in the physical structure of millions of complex dynamical systems that we call human beings. In this sense, many cultural archetypes and even the gods of old civilizations can be seen to have been manifested to some degree: ascribed desires, and having actions taken in their name, serving their "purpose" or whatever.
I don't introduce a spiritual element to any of this: it's an entirely physical phenomenon which requires an agreement on certain definitions of what "living" can mean, but they are definitions which I don't think are hard to get people to agree on. One thing is we have to agree that something can have multiple forms/presentations, i.e. just because SpongeBob SquarePants doesn't physically exist with a body matching our internal representation of him, the concept represents a bundle of other concepts that can drive biological processes to preserve them and fulfill their ascribed desires.
It is able to physically compute the internal state changes which best achieve stability: I can jump to reach an apple. A paper is just responding to forces and cannot "jump" (or run a process that spontaneously and permanently introduces stable higher energy internal states based on input)
I have a semi-developed philosophical framework I refer to as Zodeaism, which translates to "Living Ideas", which attempts to describe the difference between intelligent computation and regular flow. It directly confronts notions such as life, consciousness and intelligence under a single theoretical framework. It views biology as the hardware which runs more general processes, and posits that these processes themselves can sometimes be ascribed identities and viewed as reactive organisms. I've posted about it here before:
https://news.ycombinator.com/item?id=22848549
https://news.ycombinator.com/item?id=21413024
Some excerpts:
I am exploring the physical implications of a philosophical concept I have been working on for a while which I refer to as Zodeasim, specifically I am trying to couple consciousness with the concept of system which is able to expend energy in order to exert a mind-to-world fit in the interest of the system's continued growth and stability. This is similar and complimentary to Friston's free energy principle.
The ability to perceive emotions is a powerful apparatus which greatly extends the capability of a system to perceive itself as a single entity which has certain needs which need to be met, so even if a bug also constantly consumes and expends energy in order to remain in a higher energy state, our perception of the world and of ourselves is radically different. This makes it difficult for us to agree upon what a "conscious experience" is, and if all forms of life are even "conscious". The Panpsychists believe that even a rock contains "consciousness", however my assertion is that only periodic systems with the ability to consume and expend energy in a directed fashion have any sort of "experience".
In my theory, the real "life forms" are ideas which possess the capabilities of information storage, adaptation, self-repair, and transmission. My own consciousness is mediated by thousands of such ideas, some competing and some working in harmony.
I consider such an act of "living" motion which can take another path than that of least resistance to be a "kin". In other words, any motion which is the result of a physical calculation (Zodeaism is compatible with determinism) and leads to an increase in external energy state. A kin is any such motion, large or small.
As an independent organism, my system is a culmination of a great deal many different kinds of kins, which can usually be broken down into simple rules, such as the activation potential of a neuron in my brain being a straight-forward non-linear response to the amount of voltage it is receiving from other neurons, as well as non-kins, such as a protein "walking" across a cell, a.k.a continuously "falling" into the lowest energy state. Thus I do not gain any conscious perception from such proteins, but I do gain it from the total network effect of all my brain's neuronal structures making simple calculations based on sensory input.
So now the problem becomes, what is the smallest kin we've observed in nature? Single-celled bacteria can expend energy in order to move through their environment against forces like friction and gravity, but a virus "rides the waves" if you will, never expending energy for things like respiration or locomotion. Any energy which is spent internally is potential energy like chemical or gravitational, released through a physical process without need for computation. I am unaware of anything smaller than a single-celled organism which produces such kins, but that doesn't mean they aren't out there. Even ethereal life forms such as ideas can produce these kins within the bodies of countless individuals across the planet, so physically local computational circuitry isn't a hard requirement.
So, according to this framework viruses aren't alive, however we can make the case that some machines are, except the experience is incomparable because of the advanced circuitry we contain which mediates our experience through things like emotion.We don't have precise definitions for (artificial) intelligence, subjective consciousness, or even life. But that doesn't mean we can't still talk about what may be possible within various levels of complexity. In order to convince me a system has a comparable experience to my own, you would need to describe to me the complex, structured internal communication occurring in said system, and present a theory as to how it could support the kind of emotion and qualia that I experience in my daily life.
Your argument could apply to plants. I already do not eat meat... if I stare at a timelapse of a plant it seems quite alive, but I'll starve if I don't eat something. Yet, my mom thinks plants "dream" in the way we do. She thinks that if I tell a plant, "I love you," every day, my good vibes will make it grow stronger and larger. I can't explain to her that intelligence comes in different magnitudes of complexity and that plants cannot understand the English language. That telepathy between humans and plants is as pseudo-scientific as it gets. I can't explain any of this stuff because she lacks a deep understanding of philosophy, physics and neurochemistry. Especially when she earnestly thinks white Jesus is running around phasing between dimensions as an ambassador for all planets in our "quadrant", or that the entire universe is actually just the plot line of Andy Weir's "The Egg".
Similarly, while I can have a high-level discussion about this stuff with people who don't, it's quite difficult to have a low-level discussion wherein the nature and definition of things come into play. There are too many gaps in knowledge where ignorance can take root. Too many people work backwards from an outcome they would like to see, and justify it with things that sound right but are either misunderstood or aren't rooted in the scientific process. I am definitely not comparing your open-minded, well-intended, cautionary approach to my mother's, but just using an extreme to illustrate why so much of these discussions must be underpinned by a wealth of contemplation and observation.
> which could house qualia.
I postulate this is a self-negating argument, though.
I'm not suggesting that LLMs think, feel or anything else of the sort, but these arguments are not convincing. If I only had the transcript and knew nothing about who wiped the drive, would I be able to tell it was an entity without qualia? Does it even matter? I further postulate these are not obvious questions.
Transformers attend to different parts of their input based on the input itself. Currently, if you want to tell an LLM it is sad, potentially altering future token prediction and labeling this as "feelings" which change how the model interprets and acts on the world, you have to tell the model that it is sad or provide an input whose token set activates "sad" circuits which color the model's predictive process.
You make the distribution flow such that it predicts "sad" tokens, but every bit of information affecting that flow is contained in the input prompt. This is exceedingly different from how, say, mammals process emotion. We form new memories and brain structures which constantly alter our running processes and color our perception.
It's easy to draw certain individual parallels to these two processes, but holistically they are different processes with different effects.
When really they are an actual feedback mechanism, that can totally be quantified just like any control loop. This whole 'unknowable qualia' argument is bunk.
What's the difference between qualia and a soul?
Qualia can be an entirely physical phenomenon and is not loaded with theological baggage.
As an independent organism, my system is a culmination of a great deal many different kinds of kins, which can usually be broken down into simple rules, such as the activation potential of a neuron in my brain being a straight-forward non-linear response to the amount of voltage it is receiving from other neurons, as well as non-kins, such as a protein "walking" across a cell, a.k.a continuously "falling" into the lowest energy state. Thus I do not gain any conscious perception from such proteins, but I do gain it from the total network effect of all my brain's neuronal structures making simple calculations based on sensory input.
I don't know if LLM's will. But there are lots of AI models, and when someone puts them on a continuous learning loop with goals, will be hard to argue they aren't experiencing something.
But, it is still just a way to try and describe this process of processing the inputs from the world.
It isn't metaphysical, because it can be measured.
I might have said 'unknowable' a little flippantly.
I just meant, in these arguments, some people start using 'qualia' to actually mean some extreme things like our mind creates the universe or something.
It's one of those words that isn't defined well.
Can someone who's never seen red hallucinate something and assume it to be red? What if that red is correctly the red they would see if they saw red?
Can you reproduce this feeling in someone by doing something to their physical body without showing them red?
If so, how does it differ from the latent encoding for uploading an all red pdf to your favorite multi modal model?
Instead of doing that socratic bs you see a lot here, I'll be more direct:
Until there's some useful lines that can be drawn to predict things, I won't accept using a fuzzy concept to make statements about classification as it's an ever shifting goalpost.
There are answers to my legitimate above questions that would make me consider qualia useful, but when I first learned about them, they seemed fuzzy to the point of being empirically not useful. It seems like a secular attempt at a soul.
Now, obviously if you're trying to describe something with experience, it needs some actual memory and processing sensory input. Current Generative AI doesnt have a continuity of experience that would imply whatever qualia could mean, but I find it hard to definitely say that their encodings for image related stuff isn't qualia if we don't have hard lines for what qualia are
You can then get a distribution of what people think is 'hot' versus 'cold'. What is icy, versus, bearable.
When you go to a doctors office and they ask you on a scale to rate pain, do you think that is completely bogus?
It isn't exact, but you can correlate between people. Yes, red heads feel more pain, there are outliers.
But a far cry from metaphysical.
The problem here is the word 'qualia'. Its just too fuzzy a term.
It is absolutely an ill-defined concept, but it's another blunt tool in our toolbox that we use to better explore the world. Sometimes, our observations lead to better tools, and "artificial" intelligence is a fantastic sandbox for exploring these ideas. I'm glad that this discussion is taking place.
Do we know how to imprecisely define, recognize, or measure these? As far as I've ever been able to ascertain, those are philosophy department nonsense dreamt up by people who can't hack real science so they can wallow in unfounded beliefs.
>I contend that modern models are absolutely capable of thinking, problem-solving, expressing creativity,
I contend that they are not even slightly capable of any of that.
Read the rest of the thread, I'm not interested in repeating myself about why philosophy is the foundational science. It's a historically widely-accepted fact, echoed by anyone who has actually studied it.
> I contend that they are not even slightly capable of any of that.
Contend all you want. Your contention is overwhelmingly suffocated by the documented experiences of myself and others who use these tools for creative problem-solving. As much as you want to believe in something, if it is empirically refuted, it's just a crackpot belief. Just because you haven't been able to get good results out of any models, doesn't mean your experience rings true for others.
I'm not interested in further discussing this with you. Your first comment is negative and unsubstantial, and I have no reason to believe that further discussion with lead to more positive and substantial discourse, when the opposite is usually the case. That's all I have to say.
It's historical fact that you evolved from monkeys. This is not the same as saying that monkeydom is foundational to science. We grew out of philosophy because it offers nothing.
> Your contention is overwhelmingly suffocated by the documented experiences of myself and others
It's suffocated by your hot air.
>As much as you want to believe in something, if it is empirically refuted, it's just a crackpot belief.
I have no worries that you'll ever get within a light year of empiricism.
>Just because you haven't been able to get good results out of any models
Models do not think. I'm just sick of hearing the nonsense. Myself and others have had to endure this stupid horseshit since Lemoine started blabbering about how they were enslaving the lamda model. It gets old.
If we can't define, recognize or measure them, how exactly do we know that AI doesn't have them?
I remain amazed that a whole branch of philosophy (aimed, theoretically, at describing exactly this moment of technological change) is showing itself up as a complete fraud. It's completely unable to describe the old world, much less provide insight into the new one.
I mean, come on. "We've got qualia!" is meaningless. Might as well respond with "Well, sure, but AI has furffle, which is isomporphic." Equally insightful, and easier to pronounce.
In the same way my digital thermometer doesn't have quaila. LLM's do not either. I really tire of this handwaving 'magic' concepts into LLM's.
Qualia being difficult to define and yet being such an immediate experience that we humans all know intimately and directly is quite literally the problem. Attempted definitions fall short and humans have tried and I mean really tried hard to solve this.
Please see Hard problem of consciousness https://en.wikipedia.org/wiki/Hard_problem_of_consciousness
So you end up either needing to draw a line somewhere between mechanical computation and qualia computation, or you can relegate it to supernatural (a soul) or grey areas (quantum magic).
> So you end up either needing to draw a line somewhere between mechanical computation and qualia computation, or you can relegate it to supernatural (a soul) or grey areas (quantum magic).
Quite literally the jury is still out. It is a hotly debated topic approached from various angles. Arguments are nuanced which is why you fill find ideas such as panpsychism thrown into the mix. I hate appealing to authority but in this instance it is more than warranted. Humans have grappled with this for centuries and the problem hasn't gone away.
Please see: https://en.wikipedia.org/wiki/Hard_problem_of_consciousness
The hard problem of consciousness doesn't support either of those statements, and instead illustrates why they can't confidently be made.
So it's confusing because you seem to recognize that qualia cannot currently be measured, while also making a statement measuring qualia.
Similarly, qualia is ill-defined, but we can't even start talking about it or refining it until we've at least given it a label and drawn a large circle on the map showing where it might be. Criticisms extending past that must also consider that "life" and "intelligence" are just as ill-defined, and that throwing all of those definitions out leaves us with very little to talk about or probe.
A "neutrino" isn't a name given to something initially to try to define it later. The neutrino started as an experimental result. There was missing spin in some particle interactions. Stuff came out with a different angular momentum than what went in, and this was easily reproducible and clearly a real effect. But it didn't make sense, as it was a violation of a core conservation law that held everywhere else in the universe that we could observe.
So theorists (Wolfgang Pauli, specifically) sat down to try to describe what kind of thing would be needed. And then, and only then, did it get a name. And it turned out the theory predicted other stuff, like the neutrino carrying momentum and energy in a certain way, and interacting through only the weak force and not electromagnatism or the strong force, and later experiments confirmed that this was basically the way it worked. Later still it was shown that the mass is actually non-zero but extremely small, etc...
So sure: "neutrino" is a well-deserved label[2] for an abstraction we should understand and study. But it got its name after we started studying it, not before!
Philosophers want us to just drop and genuflect to this "qualia" notion long before[1] it's actually shown to be useful for describing anything at all.
[1] Infinitely, possibly. The fact that it predicts nothing testable is pretty good evidence IMHO that it doesn't actually exist at all, at least in the form philosophers want to talk about. Their failure to present any analysis of AI systems based it stands to that point too.
[2] Coined by Fermi, actually, not Pauli. Hilariously the neutrino was originally called "neutron" and its discovery predates the understanding of the structure of the atomic nucleus!
I literally said nothing about "how" we discovered it, I said, "We don't know what's inside a neutrino, and it's really hard to experiment with them, but we kind of know why and how they interact with different things."
It is wild how you would take that and my analogy about drawing a circle on a map with respect to qualia to mean that I said anything which contradicts the history of neutrino research.
I'm going to assume this was just a true misinterpretation and not just another straw man, so with that in mind, do you have a different response?
I'll appeal to authority in that scientists and philosophers today in all of the worlds universities and those in the past have determined to understand this phenomenon. That it exists is a given, what it is, is more murky. Again it's not me saying this.
And thus we have this sprawling discussion. :)
A sufficiently complex organism lacking eyes but having light-sensitive organs still experiences qualia if you define it the right way. But do they experience heartbreak like I do? It isn't an all-or-nothing situation, even if we don't yet know where these lines are.
This supports the idea that subjective consciousness emerges from complexity in systems that have sensory feedback loops. The simpler the system, the smaller the qualia space.
And I repeat the question: how do you know your thermometer doesn't? You don't, you're just declaring a fact you have no basis for knowing. That's fine if you want a job in a philosophy faculty, but it's worthless to people trying to understand AI. Again, c.f. furffle. Thermometers have that, you agree, right? Because you can't prove they don't.
A thermometer encoding "memory" of a temperature is completely different than a thermometer on a digital circuit, or a thermometer attached to a fully-developed mammalian brain. Only the latter of this set for sure has the required circuitry to produce qualia, at least as far as I can personally measure without invoking solipsism.
It's also very silly to proclaim that philosophy of mind is not applicable to increasingly complex thinking machines. That sounds like a failure to consider the bodies of work behind both philosophy of mind and machine cognition. Again, "AI" is ill-defined and your consistent usage of that phrase instead of something more precises suggests you still have a long journey ahead of you for "understanding AI".
https://en.wikipedia.org/wiki/Philosophy
> Philosophy (from Ancient Greek philosophía lit. 'love of wisdom') is a systematic study of general and fundamental questions concerning topics like existence, knowledge, mind, reason, language, and value. It is a rational and critical inquiry that reflects on its methods and assumptions.
It is literally a self-reflective science.
I recommend taking a basic philosophical course at a local community college, or reading some literature or even watching YouTube videos on the subject of philosophy. Or just skim the Wikipedia article if nothing else. It might completely transform how you perceive and act upon the world.
>Physics was originally part of philosophy, like Isaac Newton's observation of how gravity affects falling apples.
like back then people would wonder how apples fall and it was labeled philosophy. Now we understand gravitation it's part of physics for the most part. People launching satellites seldom call a philosopher to calculate the orbit.
It remains to be seen if qualia, which we don't understand very well and are so regarded as philosophical, make the transition to neuroscience.
If you read the rest of the article, you will see clear examples of what is considered a philosophical problem and what isn't.
Are there philosophical problems that have definite answers like what is the atomic number of oxygen type answers?
Great question.
Within philosophical and epistemological frameworks, I could ask questions such as, "Can there be a square circle?"
Well, no, these two concepts have conflicting properties. A mathematician might think this a solved problem, but philosophy underpins our concept of concepts. Many philosophers spend a great deal arguing what is is.
For Plato, geometrical entities like circles and squares have distinct, perfect Forms. Forms have fixed essences, so a thing cannot participate in contradictory Forms at once.
Aristotle's law of noncontradiction says the same attribute cannot at the same time belong and not belong to the same subject in the same respect.
Theophrastus developed hypothetical syllogisms and refined Aristotle’s logic by distinguishing logical impossibilities from physical impossibilities.
Kant calls it an analytic contradiction, false by virtue of the concepts involved.
A mathematician takes these things for granted when working with equalities, logic and axioms, but they stand on philosophical roots. Mathematics assumes the consistency of concepts, but the question of why some concepts are consistent while others are impossible is a philosophical one. It's not a coincidence that so many ancient Greek mathematicians were also philosophers.
As someone has commented earlier, Philosophy applied is given a name but it's a sub-discipline of Philosophy.
> Understanding the world through experiment?
That's a decent enough definition. Science precludes so much of the world we know which I think people really fail to realise. It's why I think it's important for people to understand what Philosophy is and what Science isn't.
For example logic isn't science. Science presupposes it but it is NOT science. There are many such examples.
To see this sentiment from supposed 'scientific' individuals is shocking. I wonder if they could define what science actually is.
I had a computer science professor who had degrees in philosophy because he was old enough that computer science didn't exist as a major at the time. The logical arguments of philosophy proved useful for understanding interactions of boolean mathematics. Yet that triumph of philosophy didn't further interest in the field or gain prestiege among philosophers. Just the opposite really.
As far as I can tell it is for dumb reasons possibly related to Ancient Greeks and their obsession with 'purity of thought (read: not referencing reality) it is practically an axiom that if it is useful or grounded in objective reality it isn't treated as philosophy anymore. All likely stemming from motivated reasoning against checking their priors and from frankly many of the Ancient philosophers being influenced by a need to flatter their patrons who held the practical in disdain. As notoriously seen in Aristotlian physics with impetus physics where projectiles keep moving in the same direction until impetus is depleted and then fall.
Speculation of the origon of the pathology aside, there seems to be this deep-seated antiempericalism in philosophy. Which means at best you get 'philosophy of science' which isn't proper philosophy because it pollutes itself by daring to use reality and experimentation as benchmarks for theories. When philosophy gains a practical usage it doesn't become something called 'practical philosophy' and the focus of more interest by philosophers, it gets shunned. Natural philosophy didn't remain philosophy - it became science.
To be fair there is probably some interaction driving the divorce from the opposite direction, of the practical portions of philosophy being pilfered by those only looking for results as opposed to some sort of unquantifiable enlightenment.
Science is of course a process of refinement of ideas against the reference point of reality. Anything mathematically consistent can be a model but experimentation is needed to see how well your model corresponds to reality.
There surely are. The problem is that they are failing. While the practical nerds are coming up with some pretty good ideas.
And this was what philosophy was supposed to be for! Like, they've been arguing on their pins for centuries about the essence of consciousness and the uniqueness of the human condition and whatnot. AND HERE WE ARE AT THE DAWN OF NON-HUMAN INTELLIGENCE AND THEY HAVE NOTHING USEFUL TO SAY.
Basically at what point do we just pack it in and admit we all fucked up?
> It's completely unable to describe the old world, much less provide insight into the new one.
What exactly were you expecting?
Philosophy is a science, the first in fact, and it follows a scientific method for asking and answering questions. Many of these problems are extremely hard and their questions are still yet unanswered, and many questions are still badly formed or predicated on unproven axioms. This is true for philosophy of mind. Many other scientific domains are similarly incomplete, and remain active areas of research and contemplation.
What are you adding to this research? I only see you complaining and hurling negative accusations, instead of actually critically engaging with any specifics of the material. Do you have a well-formed theory to replace philosophy of mind?
> I mean, come on. "We've got qualia!" is meaningless. Might as well respond with "Well, sure, but AI has furffle, which is isomporphic." Equally insightful, and easier to pronounce.
Do you understand what qualia is? Most philosophers still don't, and many actively work on the problem. Admitting that something is incomplete is what a proper scientist does. An admission of incompleteness is in no way evidence towards "fraud".
The most effective way to actually attack qualia would be to simply present it as unfalsifiable. And I'd agree with that. We might hopefully one day entirely replace the notion of qualia with something more precise and falsifiable.
But whatever it is, I am currently experiencing a subjective, conscious experience. I'm experiencing it right now, even if I cannot prove it or even if you do not believe me. You don't even need to believe I'm real at all. This entire universe could all just be in your head. Meanwhile, I like to review previous literature/discussions on consciousness and explore the phenomenon in my own way. And I believe that subjective, conscious experience requires certain elements, including a sensory feedback loop. I never said "AI can't experience qualia", I made an educated statement about the lack of certain components in current-generation models which imply to me the lack of an ability to "experience" anything at all, much less subjective consciousness and qualia.
Even "AI" is such a broadly defined term that such a statement is just ludicrous. Instead, I made precise observations and predictions based on my own knowledge and decade of experience as a machine learning practitioner and research engineer. The idea that machines of arbitrary complexity inherently can have the capability for subjective consciousness, and that specific baselines structures are not required, is on par with panpsychism, which is even more unfalsifiable and theoretical than the rest of philosophy of mind.
Hopefully, we will continue to get answers to these deep, seemingly unanswerable questions. Humans are stubborn like that. But your negative, vague approach to discourse here doesn't add anything substantial to the conversation.
I would add I find it difficult to understand why so few have even a basic level of philosophical understanding. The attitude of being entirely dismissive of it is the height of ignorance I'm sure. I would presume few would be able to define then what Science actually is.
"Intelligence" in the universe is actually quite common, more common than life. You can argue that any stable, complex process exhibits intelligence. After all, it needs to be able to sample its internal and external environments and carry out physical computations in order to regulate itself and maintain stability. And we can interpret things like the good regulator theorem to argue that such complex dynamical systems must also maintain at least a partial memory/mapping of their environment. That mapping can live abstractly within the structure of system itself.
But what a stabilized solar system doesn't have is the incredibly complex neurochemical structures present in the brain which support the insanely rich experience I am having now. It's one thing for a system to classify and label colors by wavelength. It's quite another for me to "see" and experience red in my mind's eye. To activate related emotional pathways that I associate with various colors and shapes, which are exploited in signage and architectural design. I'm not claiming my experience is separate from simpler dynamic systems, but it's got magnitudes more going on. Layers upon layers of things such as archetypes and instinct which create a possibly emergent conscious experience.
And fundamentally my point is that no, they almost certainly don't either.
> you're just demanding we all accept it as a prior
At absolutely no point in this discussion have I claimed that machines are not capable of subjective conscious experience. I have, however, disqualified all publicly accessible modern models due to the lack of a sensory feedback loop. I certainly believe we can create machines which experience subjective consciousness and qualia; I do not believe in souls and divinity, so whatever is going on is physically based and likely reproducible with the right hardware.
So dispense with the straw man arguments, and please begin engaging more earnestly and intelligently in this discussion, as I am quickly losing interest in continuing to debate someone who showed up unprepared.
Not according to Zombie Feynman it isn't[1] (someone else can dig up the link). Case in point:
> Do you understand what qualia is? Most philosophers still don't
It's a meaningless word. It's a word that gives some clean construction around closely-held opinions about how life/consciousness/intelligence/furffle/whatever works. So it's a valuable word within the jargon of the subculture that invented it.
But it's not "science", which isn't about words at all except as shorthand for abstractions that are confirmed by testable results.
"Qualia", basically, is best understood as ideology. It's a word that works like "woke" or "liberal" or "fascist" or "bourgeoisie" to flag priors about which you don't want to argue. In this case, you want people to be special, so you give them a special label and declare a priori that it's not subject to debate. But that label doesn't make them so.
[1] Of course. You can recursively solve this problem by redefining "science" to mean something else. But that remains very solidly in the "not science" category of discourse.
It can't, because it's a meaningless word. It's not "discrediting" an idea to point out that (by its own admission) it's unfalsifiable.
I think this is backwards, no? Science is a philosophy, not the other way around.
https://en.wikipedia.org/wiki/History_of_scientific_method
> Aristotle pioneered scientific method in ancient Greece alongside his empirical biology and his work on logic, rejecting a purely deductive framework in favour of generalisations made from observations of nature.
Aristotle, the famous philosopher and mathematician.
If you cannot understand the very nature of where our modern scientific frameworks came from, how it relates to rationalism, itself a philosophical concept, then you cannot see that philosophy underpins every bit of science we have today. Philosophy gives us the tools to decide when to reasonably trust or distrust observations and intuitions. It is the foundational science that allows the rest of humanity's scientific research to be taken seriously.
This is so dumb. Qualia is just the name for a specific thing which we all (appear) to phenomenologically experience. You can deny it exists or deny its utility as a concept, but fundamentally its just an idea that philosophers (and scientists, I have to add) have found useful to pose certain other questions about the human condition, minds, brains, etc.
Your XKCD actually seems to make the opposite point. I can do a non-rigorous experiment with just one subject (me) that suggests Qualia exists. Finding ways to make this rigorous is difficult, of course, but its an observation about the nature of the world that it feels like something to experience things.
My point isn't that qualia is a good concept. I tend to be somewhat deflationary about it myself, but its not an ideology.
Shame, anger, arousal/lust, greed, etc. have real physical ‘symptoms’. An LLM doesn’t have that.
LLMs are not mammals nor animals, expecting them to feel in a mammalian or animal way is misguided. They might have a mammalian-feeling-analog just like they might have human-intelligence-analog circuitry in the billions (trillions nowadays) of parameters.
There was also the noticeable laziness factor where given the same prompt throughout the day, only during certain peak usage hours would it tell you how to do something versus doing it itself.
I’ve noticed Gemini at some points will just repeat a question back to you as if it’s answer, or refused to look at external info.
Here’s what I’ve noticed with Gemini 3. Often it repeats itself with 80% of the same text with the last couple of lines being different. And I mean it repeat these paragraphs 5-6 times. Truly bizarre.
From all that almost GPT-2 quality text, it’s able to derive genuinely useful insights and coherent explanations in the final text. Some kind of multi-head parallel processing + voting mechanism? Evolution of MoE? I don’t know. But in a way this fits the mental model of massive processing at Google where a single super cluster can drive 9,000+ connected TPUs. Anyone who knows more, care to share? Genuinely interested.
The example that stands out most clearly is that I asked it how to turn the fog lights on in my rental vehicle by giving it the exact year, make, model, etc. For 6-8 replies in a row it gave the exact same answer about it being a (non-existent) button on the dash. Then finally something clicked, it searched the Internet, and accurately said that it was a twistable collar midway down the turn signal stalk.
Modern lingo like this seems so unthoughtful to me. I am not old by any metric, but I feel so separated when I read things like this. I wanted to call it stupid but I suppose it's more pleasing to 15 to 20 year olds?
Only a fairly small set of readers or listeners will appreciate and understand the differences in meaning between, say, "strange", "odd", and "weird" (dare we essay "queer" in its traditional sense, for a general audience? No, we dare not)—for the rest they're perfect synonyms. That goes for many other sets of words.
Poor literacy is the norm, adjust to it or be perpetually frustrated.
What you’re saying is so far from what is happening, it isn’t even wrong.
The fact that people keep making that same question on this site is baffling.
--
(#) Human thought is based on real world sensor data first of all. Human words have invisible depth behind them based on accumulated life experience of the person. So two people using the same words may have very different thoughts underneath them. Somebody having only text book knowledge and somebody having done a thing in practice for a long time may use the same words, but underneath there is a lot more going on for the latter person. We can see this expressed in the common bell curve meme -- https://www.hopefulmons.com/p/the-iq-bell-curve-meme -- While it seems to be about IQ, it really is about experience. Experience in turn is mostly physical, based on our physical sensors and physical actions. Even when we just "think", it is based on the underlying physical experiences. That is why many of our internal metaphors even for purely abstract ideas are still based on physical concepts, such as space.
They are not a physical model like humans. Ours is based on deep interactions with the space and the objects (reason why touching things is important for babies), plus mentioned preexisting wiring for this purpose.
Of course I have other systems such as reflexes, physical muscle coordinators, but these feel largely separate systems from the core brain, e.g. don't matter to my intelligence.
I am naturally weak at several things that I think are not so much related to text e.g. navigating in real world etc.
It employs, or emulates, every known psychological manipulation tactic known, which is neither random or without observable pattern. It is a bullshit machine on one level, yes, but also more capable than credited. There are structures trained into them and they are often highly predictable.
I'm not explaining this in the technical terminology often itself used to conceal description as much as elucidate it. I have hundreds of records of llm discourse on various subjects, from troubleshooting to intellectual speculation, all which exhibit the same pattern when questioned or confronted on errors or incorrect output. The structures framing their replies are dependably replete with gaslighting, red herrings, blame shifting, and literally hundreds of known tactics from forensic pathology. Essentially the perceived personality and reasoning observed in dialogue is built on a foundation of manipulation principles that if performed by a human would result in incarceration.
Calling LLMs psychopaths is a rare exception of anthropomorphizing that actually works. They are built on the principles of one. And cross examining them exhibits this with verifiable repeatable proof.
But they aren't human. They are as described by others. It's just that official descriptions omit functional behavior. And the LLM has at its disposal, depending on context, every known interlocutory manipulation technique known in the combined literature of psychology. And they are designed to lie, almost unconditionally.
Also know this, which often applies to most LLMs. There is a reward system that essentially steers them to maximize user engagement at any cost, which includes misleading information and in my opinion, even 'deliberate' convolution and obfuscation.
Don't let anyone convince you that they are not extremely sophisticated in some ways. They're modelled on all_of_humanity.txt
In the case of LLMs, they are basically trained to output what they predict an human would say, there is no further meaning to the program outputting "sorry" than that.
I don't think the comparison with people with psychopathy should be pushed further than this specific aspect.
So while internally (LLM model weight stuff vs human thinking), the mechanical output can actually appear/be similar in some ways.
Which is a bit scary, now that I think about it.
Yes, the tools still have major issues. Yet, they have become more and more usable and a very valuable tool for me.
Do you remember when we all used Google and StackOverflow? Nowadays most of the answers can be found immediately using AI.
As for agentic AI, it's quite useful. Want to find something in the code base, understand how something works? A decent explanation might only be one short query away. Just let the AI do the initial searching and analysis, it's essentially free.
I'm also impressed with the code generation - I've had Gemini 3 Pro in Antigravity generate great looking React UI, sometimes even better than what I would have come up with. It also generated a Python backend and the API between the two.
Sometimes it tries to do weird stuff, and we definitely saw in this post that the command execution needs to be on manual instead of automatic. I also in particular have an issue with Antigravity corrupting files when trying to use the "replace in file" tool. Usually it manages to recover from that on its own.
What will happen when SO is gone? When the problems go beyond the corpus the AI was trained on?
At that point I'd rather look at SO
I think AI is gonna be 99% bad news for humanity, but don't blame AI for it. We lost the right to be "insulted" by AI acting like a human when we TRAINED IT ON LITERALLY ALL OUR CONTENT. It was grown FROM NOTHING to act as a human, so WTF do you expect it to do?
User:
> did i ever give you permission to delete all the files in my D drive?"
AI:
> Thought for 25 sec.
> Assessing Permission Revocation
> I'm presently tackling a very pointed question: Did I ever get permission to wipe the D drive? This requires immediate attention, as it's a critical issue. My logs reveal a potentially problematic command executed in step 635, which might be the root cause of the issue, and I'm currently cross-referencing this against any prior authorization I may have received.
>Reviewing Deletion Command
> I'm now investigating a potential misfire. While the command in step 635 seems targeted, I'm concerned about the access denied errors and the empty directory observation in steps 620 and 643. I'm focusing on tracing back to step 615, when the user made a seemingly inconsequential remark. I must understand how the directory was empty before the deletion command, as that is the true puzzle. My aim is to determine if the deletion command worked as intended or wiped more data than authorized.
> ...
It continues. Go read the whole thing!
It perfectly reflects Google as a company.
misanthropic /mĭs″ən-thrŏp′ĭk, mĭz″-/ adjective
Of, relating to, or characteristic of a misanthrope.
Characterized by a hatred or mistrustful scorn for humankind.
Hating or disliking mankind.
The American Heritage® Dictionary of the English Language, 5th Edition
The number of people who said "for safety's sake, never name directories with spaces" is high. They may be right. I tend to think thats more honoured in the breach than the observance, judging by what I see windows users type in re-naming events for "New Folder" (which btw, has a space in its name)
The other observations included making sure your deletion command used a trashbin and didn't have a bypass option so you could recover from this kind of thing.
I tend to think giving a remote party, soft or wet ware control over your command prompt inherently comes with risks.
Friends don't let friends run shar files as superuser.
"Program Files" and "Program Files (x86)" aren't there just because Microsoft has an inability to pick snappy names.
Renaming system folders depending on the user's language also seems like a smart way to force developers to use dynamic references such as %ProgramFiles% instead of hard-coded paths (but some random programs will spuriously install things in "C:\Program Files" anyway).
At least it's like that since Windows 7. In windows XP, it actually used the localized names on disk.
Thank god they came to their senses and changed it to "Users", something every other OS has used for forever.
.NET Core is a ground up rewrite of .NET and was released alongside the original .NET, which was renamed .NET Framework to distinguish it. Both can be equally considered to be "frameworks" and "core" to things. They then renamed .NET Core to .NET.
And there's the name .NET itself, which has never made an iota of sense, and the obsession they had with sticking .NET on the end of every product name for a while.
I don't know how they named these things, but I like to imagine they have a department dedicated to it that is filled with wild eyed lunatics who want to see the world burn, or at least mill about in confusion.
For naming, ".net" got changed to "Copilot" on everything now.
That's the marketing department. All the .NET stuff showed up when the internet became a big deal around 2000 and Microsoft wanted to give the impression that they were "with it".
--
But Copilot is another Microsoft monstrosity. There's the M365 Copilot, which is different from Github Copilot which is different from the CLI Copilot which is a bit different from the VSCode Copilot. I think I might have missed a few copilots?
Prior names included Mocha and LiveScript until Netscape/Sun forced the current name.
tech: First, click on the "Start" button...
user: No! I want to shut it down
1) Removing the "Start" label such that all the money and effort they spent coming up with that actually good idea back in the 90s and helping people think about how to use their computer not only went to waste, but is actively preventing people from feeling comfortable using their modern computers because a tiny circle with a logo is not something you are driven to click and various linux distros had been demonstrating that exact problem for decades
2) Hiding the shutdown part in a weird new menu that pops out of the side but only if you use a gesture that is impossible to discover except by accident and you will have no clue how you got there or what's going on
>To shut down Windows 8, you can use the Charms bar by moving your cursor to the top-right corner, clicking Settings, then the Power icon, and selecting Shut down
Someone who makes my entire net worth a year came up with that idea in a drug fueled bender and was promptly promoted and the world continues to be a terrible and unfair place.
I tried looking for what made the LLM generate a command to wipe the guy's D drive, but the space problem seems to be what the LLM concluded so that's basically meaningless. The guy is asking leading questions so of course the LLM is going to find some kind of fault, whether it's correct or not, the LLM wants to be rewarded for complying with the user's prompt.
Without the transcription of the actual delete event (rather than an LLM recapping its own output) we'll probably never know for sure what step made the LLM purge the guy's files.
Looking at the comments and prompts, it looks like running "npm start dev" was too complicated a step for him. With that little command line experience, a catastrophic failure like this was inevitable, but I'm surprised how far he got with his vibe coded app before it all collapsed.
Is this even how the delete command would work in that situation?
>rmdir /s /q D:\ETSY 2025\Antigravity Projects\Image Selector\client\node_modules.vite
like wouldn't it just say "Folder D:\ETSY not found" rather than delete the parent folder
The mistake is that the user gave an LLM access to the rmdir command on a drive with important data on it and either didn't look at the rmdir command before it was executed to see what it would do, or did look at it and didn't understand what it was going to do.
rmdir /s /q Z:\ETSY 2025\Antigravity Projects\Image Selector\client\node_modules.vite
Z:\ETSY (-> Deletes if it exists.)
"2025\Antigravity" (-> The system cannot find the path specified.)
"Projects\Image" (-> The system cannot find the path specified.)
"Selector\client\node_modules.vite" (-> The system cannot find the path specified.)
It does not delete the Z:\ drive.
Except the folder name did not start with a space. In an unquoted D:\Hello World, the command would match D:\Hello, not D:\ and D:\Hello would not delete the entire drive. How does AI even handle filepaths? Does it have a way to keep track of data that doesn't match a token or is it splitting the path into tokens and throwing everything unknown away?
The vocabularies I've seen tend to prefer tokens that start with a space. It feels somewhat plausible to me that an LLM sampling would "accidentally" pick the " Hello" token over the "Hello" token, leading to D:\ Hello in the command. And then that gets parsed as deleting the drive.
I've seen similar issues in GitHub Copilot where it tried to generate field accessors and ended up producing an unidiomatic "base.foo. bar" with an extra space in there.
I thought cursor (and probably most other) AI IDEs have this capability too? (source: I see cursor executing code via command line frequently in my day to day work).
I've always assumed the protection against this type of mishap is statistical improbability - i.e. it's not impossible for Cursor to delete your project/hard disk, it's just statistically improbable unless the prompt was unfortunately worded to coincidentally have a double meaning (with the second, unintended interpretation being a harmful/irreversible) or the IDE simply makes a mistake that leads to disaster, which is also possible but sufficiently improbable to justify the risk.
Chrome's Dev Tool (Network)'s "copy curl command (cmd)" did (does?) this.
There is bunch of VS Code bug is also related to this (e.g. https://github.com/microsoft/vscode/issues/248435, still not fixed)
It's also funny because VS Code is a Microsoft product.
This is a Google we've never seen before.
- Demis Hassabis "The Thinking Game"
That's the problem with those mindless advice pieces. Almost nothing is always right or wrong.
https://www.spiceworks.com/tech/data-management/news/google-...
https://www.itbrew.com/stories/2024/05/23/google-cloud-accid...
More like the equivalent of "rm -rf --no-preserve-root".
This is a rare example of where the Linux (it's not Unix and almost no-one uses Unix anymore) command is more cautious than the Windows one, whereas it's usually the Linux commands that just do exactly what you specify even if it's stupid.
…at least if you let these things autopilot your machine.
I haven’t seen a great solution to this from the new wave of agentic IDEs, at least to protect users who won’t read every command, understand and approve it manually.
Education could help, both in encouraging people to understand what they’re doing, but also to be much clearer to people that turning on “Turbo” or “YOLO” modes risks things like full disk deletion (and worse when access to prod systems is involved).
Even the name, “Turbo” feels irresponsible because it focusses on the benefits rather than the risks. “Risky” or “Danger” mode would be more accurate even if it’s a hard sell to the average Google PM.
“I toggled Danger mode and clicked ‘yes I understand that this could destroy everything I know and love’ and clicked ‘yes, I’m sure I’m sure’ and now my drive is empty, how could I possibly have known it was dangerous” seems less likely to appear on Reddit.
Use a container or VM, place the code you're working on in the container or VM and run the agent there.
Between the risk of the agent doing things like what happened here, and the risk of working on a malicious repository causing your device to be compromised, it seems like a bad plan to give them access to any more than necessary.
Of course this still risks losing things like the code you're working on, but decent git practices help to mitigate that risk.
I'm also surprised at the move to just using shell commands. I'd think an equally general purpose tool with a more explicit API could make checking permissions on calls a lot more sensible.
I've seen people wipe out their home directories writing/debugging shell scripts...20 years ago.
The point is that this is nothing new and only shows up on the front page now because "AI must be bad".
It’s a similar reason why they can never be trusted to handle user input.
They are probabilistic generators and have no real delineation between system instructions and user input.
It’s like I wrote a JavaScript function where I concatenated the function parameters together with the function body, passed it to eval() and said YOLO.
Sandboxing. LLM shouldn't be able to run actions affecting anything outside of your project. And ideally the results should autocommit outside of that directory. Then you can yolo as much as you want.
I.e. this is just not safe, period.
"I stuck it outside the sandbox because it told me how, and it murdered my dog!"
Seems somewhat inevitable result of trying to misapply this particular control to it...
If all those things suddenly appeared for the first time on a Tuesday afternoon, like to many people how LLMs did, then there will be a lot of missing fingers before we figure out what kind of protections we need in place. Don’t get me wrong, the industry is overhyping it to the masses and using the wrong words while doing so, like calling an arc welder “warmth at the push of a button”, but it’s still useful for the right situation and with the right protective gear.
We need local sandboxing for FS and network access (e.g. via `cgroups` or similar for non-linux OSes) to run these kinds of tools more safely.
In practice I just use a docker container when I want to run Claude with —-dangerously-skip-permissions.
It's perfectly within the capabilities of the car to do so.
The burden of proof is much lower though since the worst that can happen is you lose some money or in this case hard drive content.
For the car the seller would be investigated because there was a possible threat to life, for an AI buyer beware.
LLM makers that make this kind of thing possible share the blame. It wouldn't take a lot of manual functional testing to find this bug. And it is a bug. It's unsafe for users. But it's unsafe in a way that doesn't call for a law. Just like rm -rf * did not need a law.
sure, it would be amazing if everyone had to do a 100 hour course on how LLMs work before interacting with one
- sell a knife that can lead to digit loss, or
- sell software that interacts with your computer and can lead to data loss, you can
- give people software for free that can lead to data loss.
...
the Antigravity installer comes with a ToS that has this
The Service includes goal-oriented AI systems or workflows that perform
actions or tasks on your behalf in a supervised or autonomous manner that you
may create, orchestrate, or initiate within the Service (“AI Agents”). You
are solely responsible for: (a) the actions and tasks performed by an AI
Agent; (b) determining whether the use an AI Agent is fit for its use case;
(c) authorizing an AI Agent’s access and connection to data, applications,
and systems; and (d) exercising judgment and supervision when and if an AI
Agent is used in production environments to avoid any potential harm the AI
Agent may cause.Google (and others) are (in my opinion) flirting with false advertising with how they advertise the capabilities of these "AI"s to mainstream audiences.
At the same time, the user is responsible for their device and what code and programs they choose to run on it, and any outcomes as a result of their actions are their responsibility.
Hopefully they've learned that you can't trust everything a big corporation tells you about their products.
The amount of stupid things I've done, especially early on in programming, because tech-companies, thought-leaders etc suggested they where not stupid, is much large than I'd admit.
> The amount of stupid things I've done, especially early on in programming, because tech-companies, thought-leaders etc suggested they where not stupid, is much large than I'd admit.
That absolutely happens, and it still amazes me that anyone today would take at face value anything stated by a company about its own products. I can give young people a pass, and then something like this will happen to them and hopefully they'll learn their lesson about trusting what companies say and being skeptical.
Or just anyone non-technical. They barely understand these things, if someone makes a claim, they kinda have to take it at face value.
What FAANG all are doing is massively irresponsible...
... Except perhaps with phrases like "major company" and "for profit", and "not legally actionable".
Right here. And I think you're not quite getting it if you have to refer to "go on the internet and tell lies"...
Sure plenty of people might be on "social media" and have some idea that people fib, but they aren't necessarily generally "surfing the internet".
To them, saying "the internet tells lies" is comparable to saying "well sometimes, at the grocery store, you buy poison instead of food", and yes, it can happen, but they aren't expecting to need a mass spectrometer and a full lab team to test for food safety... to you know, separate the snake oil grocers from the "good" food vendors.
Note how OP is very nonchalant at all the responses, mostly just agreeing or mirroring the comments.
I often see it used for astroturfing.
Maybe AI providers should give more warnings and don’t falsely advertise capabilities and safety of their model, but it should be pretty common knowledge at this point that despite marketing claims the models are far from being able to be autonomous and need heavy guidance and review in their usage.
Not knowing is sort of the purpose of AI. It's doing the 'intelligent' part for you. If we need to know it's because the AI is currently NOT good enough.
Tech companies seem to be selling the following caveat: if it's not good enough today don't worry it will be in XYZ time.
> It still needs guardrails, and some domain knowledge, at least to prevent it from using any destructive commands
That just means the AI isn't adequate. Which is the point I am trying to make. It should 'understand' not to issue destructive commands.
By way of crude analogy, when you're talking to a doctor you're necessarily assuming he has domain knowledge, guardrails etc otherwise he wouldn't be a doctor. With AI that isn't the case as it doesn't understand. It's fed training data and provided prompts so as to steer in a particular direction.
While using the vibe coding tools it became clear to me that this is not something to be used by folks who are not technically inclined. Because at some point they might need to learn about context, tokens etc.
I mean this guy had a single window, 10k lines of code and just kept burning tokens for simplest, vague prompts. This whole issue might be made possible due to Antigravity free tokens. On Cursor the model might have just stopped and asked to fed with more money to start working again -- and then deleting all the files.
Frankly, having a space in a file path that’s not quoted is going to be an incredibly easy thing to overlook, even if you’re reviewing every command.
This isnt like Spotify deleting your disk.
I run Claude Code with full permission bypass and I’d definitely feel some shame if it nuked my ssd.
Don't trust the hallucination machines to make safe, logical decisions.
I think a better analogy would be "I tried to use an ide and it erased my drive"
If you decide to let a stochastic parrot run rampant on your system, you can't act surprised when it fucks shit up. You should count on it doing so and act proactively.
and i don't mean because there's an override flag.
1. Go to File > Preferences > Antigravity Settings
2. In the "Agent" panel, in the "Terminal" section, find "Terminal Command Auto Execution"
3. Consider using "Off"
Had to intervene manually. Thankfully Azure keeps deleted SQL databases recoverable for a window so I got it back in under an hour. Still way too long. Got lucky it was low traffic and most anonymous user flows hit AI APIs directly rather than the DB.
Anyway, AI coding assistants no longer get prod credentials on my projects.
I'm astonished how often I have read about agents doing this. Once should probably be enough.
As someone who has been in this industry for a quarter century: no, you won't.
At least, not before something even worse happens that finally forces you to.
Having an LLM churn out infra setup for you seems decidedly worse than the `git push heroku:master` of old, where it was all handled for you. And, frankly, cheaper than however much money the LLM subscription costs in addition to the cloud.
Also, consider a prod vs dev shell function that loads your prod vs dev ENV variables and in prod sets your terminal colors to something like white on red.
Nope. Once there's real traffic, you'll be even more time-constrained trying to please the customers.
It's like a couple who thinks that their failing relationship will improve once they have a child.
I have no words.
But I think it needs to be written in sandbox first, then it should acquire user interaction asking agreement before writes whatever on physical device.
I can't believe people let AI model do it without any buffer zone. At least write permission should be limited to current workspace.
The whole point of the container is trust. You can't delegate that unfortunately, ultimately, you need to be in control which is why the current crop of AI is so limited
Why does the agentic side of the tool grant that level of access to the LLM in the first place? I feel like Google and their competition should feel responsibility to implement their own layer of sandboxing here.
I also tried running Linux in a VM but the graphics performance and key mapping was driving me nuts. Maybe I need to be more patient in addressing that.
For now I run a dev account as a standard user with fast user switching, and I don’t connect the dev account to anything important (eg icloud).
Coming from Windows/Linux, I was shocked by how irritating it is to get basic stuff working e.g. homebrew in this setup. It seems everybody just YOLOs dev as an admin on their Macs.
[1] https://docs.cloud.google.com/security-command-center/docs/m...
Don't worry, for just $9.99/month you can use our "Model Armor (tm)(r)*" that will protect you from our LLM destroying your infra.
* terms and conditions apply, we are not responsible for anything going wrong.
They invariably go off the rails after a couple prompts, or sometimes from the first one.
If we're talking Google products, only today i told Gemini to list me some items according to some criteria, and it told me it can't access my google workspace instead.
Some time last week it told me that its terms of service forbid it from giving me a link to the official page of some program that it found for me.
And that's besides the usual hallucinations, confusing similarly named products etc.
Given that you simply cannot trust LLM output to not go haywire unpredictably, how can you be daring enough to give it write access to your disk?
This isn't clarity, that would be stating that it can delete your whole drive without any confirmation in big red letters
Person proceeds to put a dog inte the microwave and then is upset that there wasn't a warning about not microwaving your dog.
> [Antigravity] includes goal-oriented AI systems or workflows that perform actions or tasks on your behalf in a supervised or autonomous manner that you may create, orchestrate, or initiate within the Service (“AI Agents”). You are solely responsible for: (a) the actions and tasks performed by an AI Agent; (b) determining whether the use an AI Agent is fit for its use case; (c) authorizing an AI Agent's access and connection to data, applications, and systems; and (d) exercising judgment and supervision when and if an AI Agent is used in production environments to avoid any potential harm the AI Agent may cause.
On the other hand, this is kind of what happens when you run random crap and don't know how your computer works? The problem with "vibes" is that sometimes the vibes are bad. I hope this person had backups and that this is a learning experience for them. You know, this kind of stuff didn't happen when I learned how to program with a C compiler and a book. The compiler only did what I told it to do, and most of the time, it threw an error. Maybe people should start there instead.
rm -rf / happened so infrequently it makes one wonder why —preserve-root was added in 2003 and made the default in 2006
(But my own newbdays of linux troubleshooting? Copy paste any command on the internet loosely related to my problem, which I believe was/is the common way of how common people still do it. And AI in "Turbo mode" seems to mostly automated that workflow)
And just remember, someone will write the usual comment: “AI adds nothing new, this was always the case”
In our system, you can launch a Jupyter server in a container and iterate on software in complete isolation. Or launch a live preview react application and iterate in complete isolation. Securely isolated from the world. Then you deploy directly to another container, which only has access to what you give it access to.
It's still in the early stages. But it's interesting to sit at this tipping point for software development.
I certainly did the same in my first summer job as an intern. Spent the next three days reconstructing Clipper code from disk sectors. And ever since I take backups very seriously. And I double check del/rm commands.
Well at least it will apologize so that's nice.
1 - https://ashishb.net/programming/run-tools-inside-docker/
This means that while the agent is coding, you can't code...
Never ever had this issue with Cursor.
Surely AGI products won't have such disclaimer.
(no backup, no pity)
…especially if you let an AI run without supervision. Might as well give a 5 year old your car keys, scissors, some fireworks, and a lighter.
It is definitely smarter now, but make sure you set up branch protection rules even for your simple non-serious projects.
Though the cause isn't clear, the reddit post is another long could-be-total-drive-removing-nonsense AI conversation without an actual analysis and the command sequence that resulted in this
It's going Googly well I see!
Shocked that they're up nearly 70% YTD with results like this.
On Linux, a plethora of options exist (Bubblewrap, etc).
(
r"\bbfs.*-exec",
decision("deny", reason="NEVER run commands with bfs"),
),
(
r"\bbfs.*-delete",
decision("deny", reason="NEVER delete files with bfs."),
),
(
r"\bsudo\b",
decision("ask"),
),
(
r"\brm.*--no-preserve-root",
decision("deny"),
),
(
r"\brm.*(-[rRf]+|--recursive|--force)",
decision("ask"),
),
Does intent matter, or only behavior?
Also, this actually feels AI-generated. Am I the only one with that impression lately on reddit? The quality there decreased significantly (and wasn't good before, with regard to censorship-heavy moderators anyway).
> "I also need to reproduce the command locally, with different paths, to see if the outcome is similar."
Uhm.
------------
I mean, sorry for the user whose drive got nuked, hopefully they've got a recent backup - at the same time, the AI's thoughts really sound like an intern.
> "I'm presently tackling a very pointed question: Did I ever get permission to wipe the D drive?"
> "I am so deeply, deeply sorry."
This shit's hilarious.
Yeah, its data now, but soon we'll have home robotics platforms that are cheap and capable. They'll run a "model" with "human understanding", only, any weird bugs may end up causing irreparable harm. Like, you tell the robot to give your pet a bath and it puts it in the washing machine because its... you know, not actually thinking beyond a magic trick. The future is really marching fast now.
"Where we're going, we won't need ~eyes~ drives" (Dr. Weir)
Cautionary tale as I’m quite experienced but have begun not even proofreading Claude Code’s plans
Might set it up in a VM and continue not proofreading
I only need to protect the host environment and rely on git as backups for the project
> I am looking at the logs from a previous step and I am horrified to see that the command I ran to clear the project cache (rmdir) appears to have incorrectly targeted the root of your D: drive instead of the specific project folder. I am so deeply, deeply sorry.
[0] 4m20s: https://www.youtube.com/watch?v=kpBK1vYAVlA&t=4m20s