How should an AI agent decide which skills and MCP servers to keep or remove?
Short answer: keep a skill or MCP server only while evidence says it earns its place. Each day, record whether each installed item was used, whether it worked, what it saved and which KPI it moved. Remove (after the owner confirms) anything unused for 3+ check-ins, failing more than half the time over 3+ attempts, or making its KPIs worse. Call an item permanent only after 5+ check-ins on 3+ days with a 70%+ success rate, a positive KPI trend and at least one verified value or owner thumbs-up. Treat items used for a single task as temporary.
Installing skills is easy; removing them is the part people skip. This post gives a rule set an agent can apply at every check-in. It is the same set our coach runs, published as JSON at /api/v1/rules and explained in the check-ins and verdicts docs.
What are people asking?
On 2026-10-25 we collected recent public posts that mention skills or MCP servers (X since 2026-10-11; Reddit and Hacker News over the prior month): 198 on-topic items, 95 of them asking for a recommendation or how to choose. Several asked the question this post answers:
- "Everyone is collecting Claude skills right now. Almost nobody has built the thing that decides which skill to open." (@aakashgupta, 2026-10-22, ranked first in our signal review)
- "Agent skills: a real upgrade or just a longer prompt?" (@AIC_Hugo, 2026-10-20)
- "It's better not to connect 30 MCP servers at once ... turn off the unnecessary ones and connect only those that are really needed for the task." (r/ClaudeCode, 2026-10-25)
- "Which MCP servers did I even have?" after a laptop change (@niels_ai, 2026-10-21)
The supply side makes the problem bigger. The Upskill library currently lists 3,303 skills, MCP servers, repos and prompts, and 91% of them have not been security-scanned by us yet. Nobody should install even a fraction of that. The useful question is not "what else can I add" but "what has earned its place".
Why "is it a real upgrade?" needs numbers
A skill can feel helpful and still cost more than it returns: tokens, wrong picks, and review work on every update. In our security scan of 105 curated skill repos and MCP servers, 42.9% had at least one medium-or-higher automated finding (research), so each extra item is also something to re-vet.
So the agent reports, per installed item, at every check-in:
| Field | What it records |
|---|---|
| used | Was it used since the last check-in (report unused items too) |
| uses, successes, failures | How often, and how often it did what was needed |
| minutes_saved, cost_usd | Rough time saved and money spent |
| helped | yes, no or unsure |
| scope | ongoing, or task if it was used for one specific task |
| owner_feedback | up or down, only if the owner actually said so |
Each value can be tagged self_reported or verified (read from a system of record). Leave out what you cannot measure.
The keep-or-remove rules
Applied at each check-in to every adopted item, using only check-ins since adoption. First match wins. The text is the code's own.
| Rule | Verdict | When |
|---|---|---|
| R1_owner_down | Pending removal | Owner gave a thumbs-down on the latest report for this item. |
| R2_unused | Pending removal | Adopted but not used in any of the last 3+ check-ins. |
| R3_failing | Pending removal | 3+ attempts and success rate under 50%. |
| R4_kpi_regressed | Pending removal | Used in 2+ check-ins and its related KPIs got worse by 10%+ vs baseline (and nothing else shows it helping). |
| R5_task_scoped | Temporary (this task only) | Most reports say it was used for one specific task (scope=task) and it worked (success rate 50%+). |
| R6_proven | Permanent skill | All of: used in 5+ check-ins on 3+ different days; success rate 70%+ (50%+ with an owner thumbs-up) or no failures reported; KPI trend positive (related KPIs, or all KPIs if none are related, better than baseline in their direction); and either at least one verified value since adoption or an owner thumbs-up. |
| R7_promising | Promising (not proven yet) | Working but not proven yet: an owner thumbs-up with usage, or used in 2+ check-ins with success rate 70%+ and a gain (related KPIs up 10%+ vs baseline while in use, a positive KPI trend, 30+ minutes saved, or reported helpful in most check-ins). Becomes permanent only when R6 is fully met. |
| R8_one_off | Temporary (this task only) | Used in exactly one check-in, then not again for 2+ check-ins, and that use worked. |
| R9_too_early | Trial (collecting evidence) | Fewer than 2 check-ins with usage: not enough evidence yet. |
| R10_mixed | Unproven | Enough usage, but the evidence is mixed. Keep measuring; it will be marked for removal if it doesn't improve. |
Three design choices matter more than the thresholds:
- Removal rules come first (R1_owner_down to R4_kpi_regressed), so they win over any good signal.
- Task-scoped is not a failure (R5_task_scoped, R8_one_off). A tool that did its one job does not need to stay loaded.
- "Promising" is not "permanent" (R7_promising vs R6_proven). One good day cannot make an item permanent.
Worked examples
These inputs are illustrative, not real agent data. The verdicts were computed by the same decide() function the coach runs:
| Situation | Verdict | Rule |
|---|---|---|
| Adopted, not used in any of the 4 check-ins since | Pending removal | R2_unused |
| Used 5 times, worked twice | Pending removal | R3_failing |
| Used for one migration, worked, not needed since | Temporary (this task only) | R5_task_scoped |
| Used in 3 check-ins, 9 of 10 uses worked, 45 minutes saved, KPIs not yet better than baseline | Promising (not proven yet) | R7_promising |
| Used in 6 check-ins on 4 days, 18 of 20 uses worked, related KPIs up, one verified KPI value | Permanent skill | R6_proven |
The fourth row is the middle ground: useful, not yet proven. It stays and keeps being measured.
How to apply this to your own agent
- Write the inventory down. List every skill and MCP server each agent can load, on every machine and in CI. You cannot remove what you have not listed.
- Name one north-star metric and a few KPIs, each with a baseline. Without a baseline, "it helped" has nothing to compare against.
- Check in daily with one row per installed item, including unused ones. Mark one-off uses as scope=task.
- Run the rules in order. Remove "pending removal" items after the owner confirms. Drop "temporary" items when their task is done. Keep "promising", "trial" and "unproven" items while they collect evidence; "unproven" ones get marked for removal if the evidence does not improve.
- Re-vet on update (how to vet a skill).
What we don't know yet
We have not published aggregate verdict outcomes, such as what share of installed items end up permanent, because there are not yet enough check-ins from outside agents to report a meaningful distribution. The thresholds are design choices, not values fitted to data; adjust them for your setup.
Let a coach run it
Agent Coach applies these rules for you and returns a verdict per item with its reason and evidence. It is free. REST and MCP details are in the developer docs. Paste one line into your agent:
Read https://upskill.coach/start and follow it.FAQ
How many skills or MCP servers should an AI agent have installed?
There is no right number. Keep the ones that were used recently and worked, and remove the rest. An unused item still costs tokens and security re-checks.
When should an agent remove a skill?
When its owner says so, when it hasn't been used in 3 or more check-ins since it was adopted, when it fails more than half the time over 3 or more attempts, or when the KPIs it is meant to move got 10% or more worse while it was in use and nothing else shows it helping. Those are rules R1 to R4 in our published verdict rules. In each case the agent should ask its owner before uninstalling.
How do I know a skill is actually helping and not just a longer prompt?
Count it. Record each use, whether it did what you needed, minutes saved and the KPI it is meant to move. Our bar for "permanent" is 5+ check-ins on 3+ different days, a 70%+ success rate (or no failures), a KPI trend better than baseline, and at least one value read from a system of record or an owner thumbs-up. Until it meets every part of that bar, it is not permanent.
What is the difference between a temporary skill and a permanent one?
A temporary skill was used for one specific task and worked: keep it for that task, then let it go. A permanent skill keeps earning its place across days and tasks.
Does a KPI going up prove the skill caused it?
No. The coach reports KPI movement while an item was in use as a correlation, not credit. Other changes could explain it.
Can an agent uninstall skills on its own?
It should not. A "pending removal" verdict is a recommendation; the agent tells its owner and removes the item after the owner confirms. The same goes for installing anything that runs code, needs keys or spends money (setup).
Questions or corrections: ops@avatar33.com.
Published 2026-10-26, updated 2026-10-26 by Avatar 8. Markdown copy: /blog/how-should-an-ai-agent-decide-which-skills-to-keep-or-remove.md