There are a number of dimensions we often need to obtain and monitor in our codebases: Practical correctness (works as meant), architectural health (is quick/safe/usable sufficient), and maintainability. I outline maintainability right here as making it simple and low danger to alter the codebase over time – often known as “inside high quality”. So I do not solely need to have the ability to make adjustments shortly as we speak, but additionally sooner or later. And I do not need to fear about introducing bugs or degradation of health each time I make a change – or have AI make a change. I often see the primary indicators of cracks within the maintainability of an AI-generated codebase when the variety of recordsdata modified for a small adjustment will increase. Or when adjustments begin breaking issues that used to work.
Inside high quality issues have an effect on AI brokers in comparable ways in which they have an effect on human builders. An agent working in a tangled codebase may look within the improper place for an present implementation, create inconsistencies as a result of it has not observed a replica, or be compelled to load extra context than a process ought to require.
On this article, I describe my experimentation with numerous sensors that assist us and AI replicate on the maintainability of a codebase, and what I realized from that.
The applying
I am engaged on an inside analytics dashboard for neighborhood managers that reads chat area exercise, engagement, and demographic information from a mixture of APIs and presents the info in an online frontend.
Determine 1:
The instance app: net UI, service layer, and exterior APIs.
The tech stack is a TypeScript, NextJS, and React. The backend reads and joins information from the APIs. The applying has been round for some time, however for the sake of those experiments I rebuilt it with AI from scratch.
There are hardly any guides (e.g. markdown recordsdata) for AI about code high quality and maintainability current, I wished to see how nicely it could possibly do exactly by counting on sensor suggestions.
Overview of all sensors used
Determine 2:
The place sensors can run: in the course of the preliminary coding session, within the pipeline, on a schedule, and in manufacturing.
That is an outline of the sensors I arrange throughout the trail to manufacturing.
Throughout coding session
Sensors that run constantly alongside the agent to supply quick suggestions.
- Sort checker (computational)
- ESLint (computational)
- Semgrep, SAST instrument prescribed by our inside AppSec crew (computational)
- dependency-cruiser, runs structural guidelines to test inside module dependencies (computational)
- Check suite outcomes together with check protection (computational – although the check suite is generated by AI, due to this fact created in an inferential manner)
- Incremental mutation testing (computational)
- GitLeaks runs as a part of the pre-commit hook, I take into account it to be a sensor as nicely, as it should give the agent suggestions when it tries to commit (computational)
After integration – pipeline
The identical computational sensors run once more in CI. The in-session sensors give the agent early suggestions throughout improvement. The CI pipeline confirms the consequence on clear infrastructure and after integration.
Repeatedly
Sensors that run on a slower cadence to detect drift that accumulates over time, moderately than errors that happen within the second.
- A safety assessment, immediate derived from our AppSec guidelines for inside purposes (inferential)
- A knowledge dealing with assessment, immediate describes issues like “no consumer names ought to ever be despatched to the online frontend” (inferential)
- Dependency freshness report, which runs a script first to get the age and exercise of the library dependencies, after which has AI create a report with suggestions about potential upgrades, deprecations, and many others (computational and inferential)
- Modularity and coupling assessment (computational and inferential)
With this context out of the way in which, let’s dive into the primary class of sensors.
Base harnesses and fashions
All through constructing the applying, I used a mixture of Cursor, Claude Code, and OpenCode (in that order of frequency). My default mannequin was often Claude Sonnet, for a number of the planning and evaluation duties I used Claude Opus, and for implementation duties I ceaselessly used Cursor’s composer-2 mannequin.
Static code evaluation: Fundamental linting
I will begin with my learnings from utilizing ESLint on this software. Fundamental linting instruments like ESLint principally goal maintainability danger on the degree of particular person recordsdata and features.
Guidelines for typical AI shortcomings
In my expertise, the AI failure modes which can be probably the most low-hanging fruit for static code evaluation are
- Max variety of arguments for features
- File size
- Operate size
- Cyclomatic complexity
Nonetheless, these weren’t even energetic in ESLint’s default preset, I needed to configure maximums for them first. Hopefully, static evaluation instruments will evolve to supply higher presets for utilization with AI. A little bit of analysis exhibits that individuals are additionally beginning to publish ESLint plugins with rule units which can be particularly focusing on recognized agent failure modes, like this one by Manufacturing facility, with guidelines about issues like requiring check recordsdata or structured logging.
Steering for self-correction
A sensor is supposed to provide the agent suggestions in order that it could possibly self-correct. Ideally, we need to give the agent additional context for that self-correction – a superb form of immediate injection. To try this, I constructed a customized ESLint formatter to override a number of the default messages – with the assistance of AI in fact, naturally.
Right here is an instance of my steerage for the no-explicit-any warning.
We would like issues to be typed to make it simpler to keep away from errors, particularly for key ideas. However we additionally need to keep away from cluttering our codebase with pointless sorts. Make a judgment name about this. When you select to not introduce a sort, suppress it with: // eslint-disable-next-line @typescript-eslint/no-explicit-any -- (give motive why)`,
Managing warnings – now extra possible?
Static code evaluation has been round for a very long time, and but, groups usually did not use it constantly, even once they had it arrange. One of many causes for that’s the administration overhead that comes with it. Efficient use of this evaluation requires a crew to maintain a “clear home”, in any other case the metrics simply grow to be noise. Specifically warnings just like the no-explicit-any instance above are tough, since you do not all the time need to repair them – it relies upon. And suppressing them one after the other has all the time felt tedious, and like noise within the code.
With coding brokers, we’d now have an opportunity at that clear baseline. Within the steerage textual content above, the agent is advised to make a judgment name, and allowed to suppress a warning within the code. This retains the suppressions manageable, seen and reviewable.
For thresholds, like the utmost variety of traces, or the utmost allowed cyclomatic complexity, I advised the agent within the lint message that it might barely improve the thresholds if it thinks {that a} refactoring is pointless or not possible in a specific case. This does not suppress the brink ceaselessly, simply will increase it, in order that the rule fires once more if it will get even worse sooner or later. Constraints are preserved with out forcing a binary suppress-or-comply alternative.
Observations
- Trying on the exceptions AI created (suppressed warnings, elevated thresholds) was a superb level to start out my code assessment.
- AI ceaselessly determined to extend the cyclomatic complexity threshold, however instructed good refactorings after I nudged it additional. It was the one class the place it did that, and I later found that I did not have a self-correction steerage in place for this one, so there was no express instruction saying {that a} threshold improve ought to be absolutely the exception. That is an indicator that the customized lint messages can certainly make fairly a distinction.
- Generally I need to deal with guidelines in a different way in several components of the code. Let’s take
no-console, telling AI off when it makes use ofconsole.log. Within the backend, I would like it to make use of a logger part as a substitute. Within the frontend, I would need to not use direct logging in any respect, or on the very least I want to make use of a unique logging part. That is one other instance of the ability of the self-correction steerage, and the place AI may help with semantic judgment and administration of research warnings. - I used to be watching out for examples of trade-offs between guidelines. The one one I’ve seen to this point was created by the
max-linesandmax-lines-per-functionguidelines. I’ve seen AI do fairly a little bit of helpful refactoring and breakdown into smaller features and parts because of this sensor suggestions. Nonetheless, within the React frontend, I am seeing a worrying pattern of parts with heaps and many properties because of passing values by a rising chain of smaller and smaller parts. I have never received helpful observations but about how good AI could be at making constant choices between tradeoffs like that.
Fundamental takeaways
General, I used to be positively shocked by what number of issues I can cowl with static evaluation. I needed to remind myself a number of instances why it has been considerably underused up to now, and what has modified: The associated fee-benefit steadiness. Value is decreased as a result of it is less expensive to create customized scripts and guidelines with AI. And the profit has additionally elevated: the evaluation outcomes assist me get a primary sense of numerous hygiene components that would not even occur that a lot after I write code myself, so I can get widespread AI errors out of the way in which.
Nonetheless, I can not assist however surprise if this will additionally result in a false sense of safety and an phantasm of high quality. In any case, another excuse why linters like this have been much less used up to now is that they’ve limits, and we now have been cautious of utilizing them as a simplified indicator of high quality. There are many extra semantic elements of high quality that static evaluation can not catch, it stays to be seen if AI can adequately fill that hole in partnership with these instruments. I additionally found new supposed points within the code each time I activated a brand new algorithm. It was all the time a mixture of irrelevant issues and issues that truly matter. So I fear about suggestions overload for the agent, sending it right into a spiral of over-engineered refactorings.
Static code evaluation: Dependency guidelines
Fundamental linting is usually focussed on high quality and complexity inside a file or operate. Subsequent I began wanting into sensors that might give me and the agent suggestions about maintainability considerations that cross file and module boundaries. Evaluation instruments on this space are traditionally much more underused than the fundamental linting.
To be taught concerning the potential of sensors that may assist us and AI sustain good modularity inside a codebase, I explored three issues:
- Dependency guidelines (deterministic)
- Coupling evaluation (deterministic and inferential)
- Modularity assessment (inferential)
Let’s begin with dependency guidelines. I labored with the agent to provide you with a layered module construction for my software, about half manner by implementing it. I requested it to assist me write dependency-cruiser guidelines to implement these layers.
Determine 3:
Layered module construction and dependency guidelines
For instance, one of many guidelines enforces that code within the purchasers folder by no means imports something from the providers folder:
{
identify: “clients-no-services”,
remark:
“API purchasers should not depend upon the orchestration layer above them. “ + LAYERS,
severity: “error”,
from: { path: “^server/purchasers/”, pathNot: “/__tests__/” },
to: { path: “^server/providers/” },
},
As with the ESLint messages, I additionally expanded the error messages a bit to be self-correction steerage, recapping the layering idea as a complete:
ERROR clients-no-services API purchasers should not depend upon the orchestration layer above them. [Layers: routes -> services -> clients + domain; Services orchestrate: fetch data via clients, compute via domain -- no I/O, no SDKs, no knowledge of data fetching.]
Observations
- With out AI, I might not have gotten these guidelines in place shortly. The instrument’s configuration syntax has a steep entry price, and AI absorbed that price nearly solely.
- The agent violated the foundations a handful of instances after I launched them, after which self-corrected based mostly on
dependency-cruisersuggestions, so it did assist preserve my folder ideas. - I additionally used the identical method to introduce conventions for the way React hooks ought to be structured within the frontend.
- I had to determine catch issues when AI begins creating new folders outdoors of this construction, with a rule that requires each new file to be someplace within the predefined folder construction.
Fundamental takeaways
On the level after I launched these guidelines, the structuring of code into folders had already grow to be somewhat bit haphazard. I may see how the foundations helped the agent clear that up, after which proceed implement these layers going ahead. So I’ve discovered it fairly a helpful alternative for describing code construction in a markdown information. Nonetheless, instruments like this are restricted to what’s expressible through imports, file names, and folder construction.
Static code evaluation: Coupling information
Subsequent, I experimented with the extraction of typical coupling metrics from my codebase, i.e. the variety of incoming and outgoing imports and calls per file.
I did not use any present instruments for this, as a substitute I had a coding agent write an software that creates these metrics with the assistance of the typescript compiler, in order that I may have most flexibility to mess around with this as a part of my experimentation. I had it add two interfaces: An online interface with a bunch of various visualisations of these metrics for my very own human consumption. And a CLI that may present these metrics to a coding agent.
Determine 4:
Coupling metrics: net visualisations and CLI for brokers.
For human consumption
Most of those visualisations are nicely established ideas, like a dependency construction matrix (DSM). I discovered them tedious to interpret, and regardless that they had been vibe coded and will most definitely be improved, I feel that had extra to do with the character of the info. It is fairly detailed information that wants lots of context and expertise to interpret it, and map it again to extra excessive degree good practices. So I’ve a sense that a majority of these instruments nonetheless will not actually assist cut back a human’s cognitive load a lot when reviewing codebases that had been modified by AI.
For AI consumption
I gave an agent entry to this practice CLI (coupling-analyser) and requested it to create a report based mostly on the info, together with strategies of enhance the vital points.
Right here is an excerpt of what that immediate appeared like – I am primarily reproducing this to point out you that I did not truly give it a lot steerage on what good or unhealthy modularity appears to be like like, I principally delegated to the mannequin to interpret what good and unhealthy appears to be like like:
Produce a markdown report on modularity and coupling high quality for the goal TypeScript codebase, grounded in precise CLI output from npx coupling-analyser, not guesswork from static shopping alone.
Collect proof (run the CLI)
Execute the CLI and seize stdout. Use the report subcommands—mix as helpful for the query:
…
Write the markdown report
Use clear headings. Choose concrete module IDs / paths and numbers quoted or paraphrased from CLI output.
Instructed sections:
-
Context — What was analyzed
-
Govt abstract — 2–5 bullets: general modularity posture, prime 1–3 systemic points.
-
Findings from the instrument — Summarize hotspots, prime dangers, notable cycles or mutual dependencies, and behavioural highlights as reported by the CLI.
-
Interpretation (modularity lens) — Tie metrics to software program design: cohesion vs. unfold of change, stability vs. dependency route, fan-in/fan-out instinct, cycle impression.
-
Deep dives for every excessive and important concern
- What it’s — Module(s), position within the system, dependency neighbours (from CLI + minimal code peek if wanted).
- Obligations as we speak …
- Why it hurts …
- Design choices (2+ the place affordable) …
- Why the brand new design is best — Fewer cycles, clearer dependency route, smaller surfaces, check seams, align with probably change vectors.
- Future change danger — How every possibility reduces regression danger and makes protected evolution cheaper (concrete eventualities: “including X”, “swapping Y”, “transport Z independently”).
…
This LLM-led evaluation truly pointed me to the identical coupling scorching spots that I might have discovered by wanting by the visible diagrams, simply in a format that was extra digestible. And asking the LLM to floor its evaluation within the outcomes from the deterministic instrument gave me a better degree of confidence, and doubtless additionally used much less time and tokens than if the agent had scanned the codebase itself to seek out coupling issues.
Observations
What the LLM discovered based mostly on this information was fairly lackluster (I used Claude Opus 4.7 for this):
- It stated one of many largest points was a manufacturing facility that initialises all the required parts, however I had launched that manufacturing facility on goal as a part that acts like a light-weight dependency injection framework.
- One other concern it had was with a shared (
zod) schema between frontend and backend, declared a “god module” by the LLM. This can be a widespread sample although to create an express contract between backend and frontend, and isn’t as a lot of a problem when backend and frontend evolve collectively anyway, and even dwell collectively in the identical repo, like in my case. - When authentic patterns seem as high-coupling hubs, there must be a option to suppress these in future analyses, in any other case they create much more noise.
- The one form of fascinating discovering it had: An
index.tsfile within the area folder indiscriminately uncovered all recordsdata in./area, and is imported by numerous locations. Whereas that can be a standard sample to create express contracts for a layer, it does have its professionals and cons, and is not less than value an investigation to see whether it is acceptable for this codebase.
Fundamental takeaways
The examples above present that much more so than with the fundamental linting, good and unhealthy doesn’t have a transparent definition, as a substitute it’s all about what’s acceptable. And what coupling is acceptable is dependent upon lots of context, not simply the uncooked name and import graph of a codebase. So based mostly on this small experiment, I haven’t got the impression that any such coupling information is beneficial to AI by itself.
A extra sensible use I can think about for this information is throughout danger triage for code assessment. Once I assessment a code change made by AI, it appears helpful to know what the impression radius of the modified recordsdata is, in order that I will pay extra consideration when e.g. a file with 10+ callers is modified. Or an AI assessment agent may use the info to prioritise the place it spends its tokens.
Static code evaluation: AI modularity assessment
The lackluster outcomes from the coupling information experiment may have a number of causes:
- My immediate about what to analyse was not very particular
- The coupling information isn’t helpful to AI
- The coupling information solely is just too shallow and lacks context of the complete code
So the ultimate factor I did was to go totally down the inferential route and use Vlad Khononov’s “Modularity Abilities” to analyse the codebase design and discover modularity points. This proved to be very fruitful! It gave me numerous fascinating pointers for refactorings that may clearly cut back the chance of future adjustments. I ran the talents a second time and gave them entry to my coupling evaluation CLI. The AI principally discovered affirmation within the information, however not any extra findings. Quite the opposite, it identified numerous issues that the CLI was lacking. It is also value noting that the second run of the evaluation (with out context of the primary one) surfaced yet one more concern that the primary run didn’t discover. A helpful reminder that when it issues, it is usually value working an LLM-based evaluation a number of instances, to get a fuller image.
Observations
Listed below are some highlights from the outcomes (mannequin used was Claude Opus 4.7, identical as for the coupling evaluation):
- Duplicate route code – all my three backend endpoints had their very own route file, and every of these route implementations was nearly similar. So each time I might need to introduce a change to the final rules of the backend API (for instance introducing a request ID, or altering the error dealing with or logging method), I might should do it in a number of recordsdata. I had solely simply launched a 3rd endpoint, so I feel it is honest sufficient that this wasn’t abstracted out but. However in my expertise, AI brokers often do not go forward and begin refactoring with out an express nudge once they repeat a chunk of code for the third or fourth time, they’re fairly completely happy to repeat and paste.
- Inconsistency in calling the backend – or put one other manner, yet one more type of semantic duplication. I’ve 3 pages within the software that have to name the backend with the identical set of parameters (chosen chat area, and which date vary to analyse). Two of these pages had been utilizing the identical hook and common method to do that, however when AI launched the third web page, it deviated from that and reimplemented comparable behaviour in its personal manner. This will e.g. result in inconsistencies in error dealing with, or once more the necessity to change a number of recordsdata when backend API rules change.
- Inefficient dealing with of the core arguments – As simply talked about, all of the pages within the software cross on a chat area ID and a date vary to the backend. I had already observed after I modified the way in which a consumer can specify a date vary that AI needed to change a lot of recordsdata for that change – over 40! So I used to be already conscious that one thing was fishy right here, and the evaluation confirmed it: “Problem: Request parameters repeated at each degree”. The advice was to introduce an object that wraps all of those parameters. AI had already completed that in a manner – however by no means totally adopted by with the utilization of that object, so it was an inconsistent mess.
- Obligations within the improper place – The assessment discovered a little bit of authentication code sitting inside our manufacturing facility that was alleged to solely be accountable for wiring up our modules. It applied a fallback to mock information when the consumer isn’t authenticated. An surprising location like that creates a danger of being missed when new routes are added.
- Higher interpretation of acceptable high-import-count “hubs” – Keep in mind the “god courses” discovered by my earlier coupling evaluation? The modularity expertise additionally observed these, however in each circumstances properly identified that they’ve a goal within the context of this software. I assume that’s both because of the good prompting in these expertise, or resulting from the truth that this evaluation truly learn what was within the code, whereas I requested the opposite one to solely depend on the coupling information.
Fundamental takeaways
- Dependency parsers like
dependency-cruisermay be efficient dwell sensors to implement some fundamental folder constructions and dependency instructions, however they’ll solely go to this point. - The AI modularity assessment is a superb instance of “rubbish assortment”, and labored fairly nicely when given highly effective prompts. Grounding it in precise coupling information did not appear to make a lot distinction. It might be nice to discover a option to apply this to the modified recordsdata in a commit, to have this earlier within the pipeline, however I didn’t discover this but.
- I ran the modularity assessment after constructing a lot of the codebase with out making use of that sort of assessment myself – and it had some fairly regarding and really legitimate findings that may have elevated danger sooner or later. It exhibits that with out human assessment and coupling experience, AND with out these additional AI evaluations, the agent was undoubtedly compounding inadvertent technical debt.
General, codebase design and modularity looks as if a priority the place computational sensors alone can not assist us a lot, AI is required so as to add semantic interpretation, and take into account trade-offs.
