Evaluation of LLM Performance in Tool Selection: A Feasibility Study on AI-Based Tool Development

초록

The objective of this study is to evaluate how effectively large language models (LLMs), particularly ChatGPT, can understand and select mechanical tools. Two research questions are addressed: (1) to what extent LLMs possess fundamental tool understanding and visual recognition capabilities, and (2) how effectively they can recommend appropriate tools based on user-defined situational variables. To evaluate these aspects, a systematic framework was designed using text- and image-based questions with Chain-of-Thought (CoT) prompting to assess tool function description, usage explanation, and visual differentiation between structurally similar tools. Experimental results show that while ChatGPT performs well in functional descriptions and image recognition, its performance degrades in more detailed tasks such as usage explanation and structural differentiation. Applying CoT prompting significantly improves response quality, increasing the accuracy of tool usage explanations from 71.4% to 84.3% and context-based tool recommendations from 83.3% to 91.9%. Based on these findings, this study develops a custom GPT, termed Tool Guide LLM, which automatically categorizes tool-related queries and applies optimized CoT strategies for each category. In a five-point user satisfaction evaluation, GPT-4o achieved an average score of 3.96, whereas Tool Guide LLM achieved a higher average score of 4.31. In addition, Tool Guide LLM reduced output token usage by approximately 60% on average, indicating improved response efficiency and potential reductions in response generation time. These results demonstrate the potential of LLMs as personalized tool recommendation systems and their applicability to AI-based engineering design methodologies.

키워드