MODULAR FRAMEWORK FOR ENHANCED PERCEPTION AND DECISION-MAKING - EXPRO+

Objective: To develop a software module based on visual agents to integrate and combine visual inputsprocessing and decision-making capabilities.Description:The use of artificial intelligence for computer vision tasks has been extensively explored and applied over the lastdecade, resulting in a myriad of industrial applications. However, these approaches have predominantly relied on traditional CNN-based systems, which are often rigid, task-specific, and lack decision-making capabilities. Recent advancements, such as multimodal large language models, visual transformers, and agentic workflows, have demonstrated the potential for more flexible and adaptable solutions, that can also combine both visual input processing and decision making. This activity aims to develop a software module that will:- Define and implement a software layer (execution layer) to support the inference of visual agents and enable easy integration of new agents.- Design and implement an agentic workflow to execute various computer vision tasks (i.e.: object localization,image segmentation) on top of the defined execution layer.- Integrate decision-making capabilities into the agent implementation, leveraging the output of the vision tasks.For instance, planning and scheduling tasks or specific instruction to be executed, considering the output ofthe computer vison task processed by the agent.- Identify and deploy the solution on space-representative hardware.The software solution will integrate visual perception with decision-making capabilities that surpasses currentfunctionalities provided by traditional convolutional neural network (CNN). Unlike traditional CNNs, the agentic software developed in this activity will utilize multimodal inputs that may include various types of sensors outputs,(e.g. optical, radar, time series, astextual instructions), enabling agents to perform diverse tasks. The softwaredeveloped in this activity can impact multiple application domains, like Earth Observation missions. Additionally,the solution will enable onboard capabilities to be presented as functions or services that can be adapted based on evolving ground requirements. Therefore, at the system level (flight-ground), this solution will influence current interaction paradigms, allowing the ground segment to define new goals for a satellite dynamically. This is a key technology for the Space Domain Awareness (SDA) and for tracking activities on Earth, supporting disasters monitoring, civil protection, border control, and security applications.The activity shall encompass the following tasks:- Review and analyse traditional AI algorithms applied to computer vision tasks in space and identify new trendsusing multimodal transformers, visual transformers, and agentic workflows.- Identify use cases where the visual agent framework will be applied. At least two different use cases shall beproposed to demonstrate the framework's flexibility to adapt to different situations and covering both the visualtaskand reasoning task to be implemented by agents.- Design and develop the visual agent framework, including the final selection of onboard space hardware and the design specifications necessary for deploying the algorithm on it.- Integrate the framework into the different use cases and deploy it onto the selected hardware.- Verify and validate the framework, including performance assessment to evaluate the framework's efficiency, adaptability, and robustness compared with traditional algorithms.