AI Human In The Loop

AI is a big buzzword right now, and has proponents and opponents with some very strong opinions. In this post I will try to describe how I use AI (in my case, Anthropic’s Claude) as someone who works in IT security and networking, where “Be paranoid” is point one in my job description.

Let’s not misunderstand each other here, AI can be a very useful and powerful tool for complex jobs like mine, it can save you many hours of time. But, AI can also do significant damage if it’s treated as a “team member” and trusted to just get on with the job.

What I use AI for

My work involves managing and designing some pretty complex configurations on a not-small fleet of networking devices, there are many private networks involved, lots of interconnect links between them and many routers and firewalls. With this level of complexity it’s easy to miss stuff or make mistakes, and trying to avoid mistakes takes up a lot of my work time.

My use of AI involves putting configurations of a set of devices, with any sensitive info like encryption keys and passwords removed, in a Claude Cowork project and then working with Cowork to understand those configurations and design any changes that need to be made in Cowork sessions. The AI reads and understands the configurations, looks for potential problems, maintains a todo list, maintains a project notes file detailing what’s happened in each session, advises changes that I should make, and verifies those changes with updated configuration files from the devices.

My role before: Work out what needs to be done where in hundreds of lines of configurations across several devices, sanity check my own work, apply those changes, hope I didn’t make an error, and fix the inevitable errors that slip in.

My role now: Tell the AI what I need to do, have a conversation with it to refine the plan as needed, check the suggested changes for sanity, apply the changes, have the AI check that the new configuration is what is expected.

What I do not use AI for

1. I don’t let the AI touch the devices, it can never apply changes to anything.
2. I don’t use it as a personal assistant. The AI does not have any links to things like my email or my calendar, all it can see is files I put in the Cowork folder on my computer.
3. I never take actions suggested by it without first making sure the actions are valid and sane.
4. Vibe coding (using AI to write code without really understanding it). I have used Claude Code to write some software, but only very simple stuff that I understand properly and that I could have written myself over a longer period of time.

Some hints for people who would like to use AI the way I do

Note: I use Claude Desktop, so these are geared specifically to that.

1. Turn memory on in Capabilities - For big projects, this is very useful to avoid having to explain everything to the AI in every session. It “remembers” (it uses internal Markdown files) stuff and applies that memory in later sessions.
2. Use Claude Cowork - Cowork lets you collect files related to a specific project together in a folder and then work with them as a unit.
3. Use projects in Claude Cowork - Don’t just dump everything in one folder and work from there. I have projects (Cowork folders) for stuff like router management and I have a project called “General” where I do random stuff like drafting user documentation.
4. Look into the CLAUDE.md mechanism for Cowork - A file called CLAUDE.md in the project file contains general instructions for Claude Cowork related to that project. Mine has stuff in it like “this device has this unusual configuration because of this custom design requirement”, and “do these 5 steps every time I tell you we are at the end of a session” or even “Bob is my network cabling guy, so I might refer to him by name in prompts”
5. Use Claude Cowork to maintain CLAUDE.md - CLAUDE.md is read at the start of each session, so manual edits you make mid-session won’t be picked up until the next one. To avoid confusion, I prompt Claude to make changes to CLAUDE.md directly rather than editing it myself.
6. Maintain a project notes file. Early on in the lifetime of a Claude Cowork project, prompt it with something like “Maintain a project notes file that details things done during each session, this file is both for myself and other team members or my manager to read”. Then in CLAUDE.md, as part of the session-end steps, I have “Update the project notes file and make sure the file is written to disk”.

Using AI as a tool has sped up my work while also improving the quality. A key thought is that I see AI as a tool in my toolbox for working out what I should do, not as a team member who will do the work.

And always: Be the human in the loop. AI can make mistakes, so it’s important to always be the human in the loop and make sure the output makes sense.