The output is very problematic. It breaks itself all the time, makes the same mistakes multiple times, I have to retread my steps. I’m going to have it write tests so it can better tell what it’s breaking.
But being able to say “take this GTK app and add a web server and browser based mode” and it just kinda does it with minimal manual debugging is something remarkable. I don’t fully understand it, it is a new capability. I do robotics and I wish we had this for PCB design and mechanical CAD, but those will take much longer to solve. Still, I am eager to point Claude at my hand written python robotics stack from my last major project [1] and have it clean up and document what was a years long chaotic prototyping process with results I was reasonably happy with.
The current systems have flaws but if you look at where LLMs were five years ago and you see the potential value in fixing the flaws with agentic coding, it is easy to imagine that those flaws will be addressed. There will be higher level flaws and those will eventually be addressed, etc. Maybe not, but I’m quite curious to see where this goes, and what it means for engineering as a human being at these times.
[1] https://github.com/sequoia-hope/acorn-precision-farming-rove...
Implementation plans, intermediate bot/hunan review to control for complexity, convention adherence, actual task completion, then provide guidance, manage the context and a ton of other things to cage and harness the agent.
Then, what it produces, it almost passes the sniff test. Add further bot and human code review, and we've got something that passes muster.
The siren song of "just do it/fix it" is hard to avoid sometimes, especially as deadlines loom, but that way lies pain. Not a problem for a quick prototype or something throwaway (and OP is right, that that works at all is nothing short of marvelous), but to create output to be used in long term maintainable software a lot has to happen, and even that it's sometimes a crap shoot.
But why not do it by hand then? To me it still accelerates and opens up the possibility space tremendously.
Overall I'm bullish on agents improving past the current necessary operator-driven handholding sooner than later. Right now we have the largest collection of developer agent RL data ever, all the labs sucking up that juicy dev data. I think that will improve today's tooling tremendously.
I have no doubt that agents will become meaningfully useful for some things at some point. This just hasn't really happened yet, aside of the really simple stuff perhaps.
The shitty code it comes up with helps me a lot, because fixing broken stuff and unraveling the first principles of why it's broken is how I learn best. It helps me think and stay focused. When learning new areas, the goal is to grasp the subject matter enough to find out what's wrong with the generated code - and it's still a pretty safe bet that it will be wrong, one way or another. Whenever I attempt to outsource the actual thinking (because of feeling lazy or even just to check the abilities of the model), the results are pretty bad and absolutely nowhere near the quality level of anything I'd want to sign my name under.
Of course, some people don't mind and end up wasting other people's time with their generated patches. It's not that hard to find them around. I have higher standards than replying "dunno, that's what AI wrote" when a reviewer asks why is something done this particular way. Agentic tools bring down the walls which could let you stop for a moment and notice the sloppiness of that output even further. They just let the model do more of the things it's not very good at, and encourage it to flood the code with yet another workaround for an issue that would disappear completely had you spent two minutes pondering about the root cause (which you won't do, because you don't have a mental model of what the code does, because you let the agent do it for you).
I use Claude Code with a dozen different hand tuned subagent specs and a comprehensive CLAUDE.md specifying how to use them. I review every line of code before committing (turning off the auto commit was the very first instruction). It is now the case that it is able to make a full PR that needs no major changes. Often just one or two follow up tweak requests.
With subagents it can sometimes be running for an hour or more before it is done, but I don’t have to babysit it anymore.
Tried it once and it really sucked.
I think I've had it available with the separate website ("research preview"?) for months, but yeah, last few weeks it's been directly in ChatGPT.com, and I'm within the EU.
It did research for ages, not really finding anything useful. And then it took it ages to create a very generic presentation that looked like from PowerPoint 95 with very generic tips.
[0] https://chatgpt.com/share/68953a55-c5d8-8003-a817-663f565c6f...
What is ChatGPT Agent?
Just give us the API and stop trying OpenAI.
Is for example Google’s crawl bot an agent?
Is there a prominent successful agent that I could test myself?
So many questions…
For instance, you could have an "agent" that can read/edit files on your computer by adding something like "to read a file, issue the `read_file $path`" to your prompt, and whenever a line of LLM output that starts with `read_file` is finished, the script running on your computer will read that file, paste it into the prompt, and let the LLM continue its autocomplete-on-steroids.
If you write enough tools and a complicated enough prompt, you end up with an LLM that can do stuff. By default, smart tools usually require user confirmation before actually doing stuff, but if you run the LLM in full agent mode, you trust the LLM not to do anything it shouldn't. curl2bash with LLMs, basically.
An LLM with significant training and access to file access, HTTP(S) API access, and access to some OS APIs can do a lot of work for you if you prompt it right. My experience with Claude/Copilot/etc. is that 75% of the time, the LLM will fail to do what it should be doing without manually repairing its mistakes, but in the other 25% of the time it does look rather sci-fi-ish.
With some tools you can tell your computer "take this directory, examine the EXIF data of each image, map the coordinates to the country and nearest town the picture was taken in, then make directories for each town and move the pictures to their corresponding locations". The LLM will type out shell commands (`ls /some/directory`), interpret the results as part of the prompt response that your computer sends back, and repeat that until its task has been completed. If you prepare a specific prompt and set of tools for the purpose of managing files, you could call that a "file management agent".
Generally, this works best for things you can do by hand in a couple of minutes or maybe an hour if it's a big set of images, but something the computer can now probably take care of you for you. That said, you're basically spending enough CO2 to drive to the store and back, so until we get more energy efficient data centers I'm not too fond of using these tools for banal interactions like that.
You can chain agents together into a string to accomplish larger tasks.
Think of everything involved in booking travel. You have set a budget, pick dates, chose a destinations, etc…. Each step can be defined as an agent and then you chain them together into a tool that handles the entire task for you.
Arranging these in a workflow to automate processes is common, but not agentic.
prompt = user_input()
while prompt != "exit":
prompt = replace_tool_calls_with_results(call_llm(prompt))With LLMs, this went through two phases of shittifaction: first, there was a window where the safety people were hopeful about LLMs because the weren’t agents, so everyone and their mother declared that they would create an agent out if an LLM explicitly because they heard it was dangerous.
This pleased the VCs.
Second, they failed to satisfy the original definition, so they changed the definition of agent to the thing that they made and declared victory. This pleased the VCs
""" 2.9 Potential for Risky Emergent Behaviors Novel capabilities often emerge in more powerful models.[61, 62] Some that are particularly concerning are the ability to create and act on long-term plans,[63] to accrue power and resources (“power- seeking”),[64] and to exhibit behavior that is increasingly “agentic.”[65] Agentic in this context does not intend to humanize language models or refer to sentience but rather refers to systems characterized by ability to, e.g., accomplish goals which may not have been concretely specified and 54 which have not appeared in training; focus on achieving specific, quantifiable objectives; and do long-term planning. Some evidence already exists of such emergent behavior in models.[66, 67, 65] For most possible objectives, the best plans involve auxiliary power-seeking actions because this is inherently useful for furthering the objectives and avoiding changes or threats to them.19[68, 69] More specifically, power-seeking is optimal for most reward functions and many types of agents;[70, 71, 72] and there is evidence that existing models can identify power-seeking as an instrumentally useful strategy.[29] We are thus particularly interested in evaluating power-seeking behavior due to the high risks it could present.[73, 74]"""
Ref: Artificial Intelligence - A Modern Approach.
Were you thinking along these lines?
https://medium.com/@tahirbalarabe2/five-types-of-ai-agents-e...
>Agent originally meant an ai that made decisions to optimize some utility function.
That's not what agents originally referred to, and I don't understand how your circling back to LLMs is relevant to the original definition of agent?
Can’t say much about the usage as I haven’t tried it yet.