Speaker
Description
The increasing complexity of software development in the Comprehensive Nuclear-Test-Ban Treaty Organization (CTBTO) necessitates the adoption of innovative tools and techniques to enhance productivity and efficiency. LLM-based AI code assistants have emerged as a promising solution, offering capabilities such as autocomplete, code refactoring, error detection and correction, chat based question answering, and context-aware suggestions. However, evaluating the effectiveness of these AI-powered tools in the CTBTO context is crucial to ensure their reliability, security, and compliance with organizational standards. In this work we present the results of the evaluation of open source AI code assistants in CTBTO, focusing on a modular approach that incorporates multiple components, including autocomplete models, chat models, local and remote context engines, filtering mechanisms, and training engines. We will explore the trade-offs between model size, latency, and suggestion quality, as well as the importance of integrating these components with existing toolchains and software development lifecycle data. By adopting a modular approach to AI code assistant evaluation, the CTBTO can minimize vendor lock-in, stay up to date with rapid advancements in AI, and tailor solutions to specific organizational needs.
| [email protected] |
