MCP servers look very cool in a demo with a dozen of tools, but how about when you have hundreds of workflows?
Have you found a reliable strategy to discover or rank the right tools? Any advanced tips for names, descriptions or categorization?
I know Claude now has the Tool Search tool, but can’t find anything similar for OAI or Google models.
Hello - I have read about the Claude tool search but haven’t tried it myself. Curious to hear how well it works from your perspective.
If using copilot in vs code, it switches to virtual tools if you have more than 128 tools selected. According to docs, virtual tools essentially group similar tools together. The original context will only have virtual tools. The model ‘expands’ a specific virtual group to get specific tools based on the prompt.
github.copilot.chat.virtualTools.threshold
(Experimental)
Tool count over which virtual tools should be used. Virtual tools group similar sets of tools together and enable the model to activate them on-demand. Enables you to go beyond the limit of 128 tools for a chat request.
Personally, I have been manually managing the toolset to keep the count low but curious about others suggestions around names, description etc.
I think the tool search tool is def a better idea if there are hundreds of random tools that are not organized and well defined, than feeding everything in the context. At least it saves tokens. But it’s a BM25 keyword search (or regex which seems worse), so not semantic and not especially smart. Out of the box it doesn’t look like a good enough solution for prod when accuracy is important. A possible tweak is to extend the tool descriptions (but then maybe better organize them instead) or better create a custom search tool that uses embeddings and combine it with BM25 for a RAG-for-tools.
That looks pretty cool, will check it out! Looks like it’s possible to set it to less than 128.