Aims: The aims of the project are: (a) the design and implementation of an autonomous innovative data acquisition system in-field to detect HH and other pests as well, (b) the design and implement a classification system to detect fruits damages not visible at naked eye, (c) a software solution, based on ML techniques, to derive as furthest as we can the epidemiological model of HH or/and other pests, and (d) a logbook available to further steps of the fruit production chain.
UNIPG: project leader, UAV paths-planning design and development, Cooperation UAVs and IoT devices, Logbook, coordination, coordination evaluation, integration, dissemination
UNIMORE: scientific coordination on the bug biology and behaviour, contact and dissemination with agro-food chain stakeholders, object recognition and detection, feature extraction activities, post-harvest fruit-processing hyperspectral image acquisition, evaluation, integration, dissemination;
TUBS: IoT devices, sensing, data communication and networking, logbook, evaluation,integration, dissemination;
TNI: UAV path-planning design, bug biology (Teagasc), data communication and networking, evaluation, integration, dissemination;
UPB: UAV path-planning design and development, bug biology and behaviour, ML data processing for i) object recognition and detection, ii) estimating abundance of HH, and iii) possibly epidemiological model, evaluation, integration, dissemination;
IMEC: post-harvest fruit-processing hyperspectral image acquisition, exploration of complementary imaging technologies, evaluation, integration, dissemination;
UOWM: UAV path-planning design, ML data processing for i) object recognition and detection, ii) estimating abundance of HH, and iii) possibly epidemiological model, evaluation, integration, dissemination.
This project is structured in 5 WPs (WPs). In each WPs, except WP0, a task ‘Evaluation, Integration, & Demonstration’ (TEID) that covers the last year is introduced.
WP0 (M1-36) (Coordination, Dissemination, Exploitation and Communication): The output of WP0 are the reports, the web-site, and the symposium. The effectiveness of WP will be measured by the scientific visibility and the publications. It consists of two tasks.
- Task 0.1 (T0.1) [M1-36] (UNIPG, All): Coordination. The activities include:
- Conceptual map and set-up of a collaborative platform and repository for knowledge management. This appears as a web-site.
- Establishment and management of internal and external communication procedures, as explained Section Impact ‘c’.
- Establishment of the roadmap of the scientific and technological expected outcomes.
- Administrative, legal and financial management.
- Task 0.2 (T0.2) [M1-36]: Dissemination, Exploitation and Communication
- Intra-partners dissemination of the selected intra-communication procedures, web-page.
- Communication Promotion and dissemination of the Project’s outcomes (All).
- Organization symposium at the end of the second year (All).
- Contacting stakeholders such as companies and farmers’ associations to promote the project (UNIMORE, All).
- Deliverable 0.1 [M36]: Dissemination of systems’ technical details, open tools, and results.
WP1 (M1-36) (Algorithms for data acquisition): The WP is in charge of designing the algorithms to be used for acquiring the data using Unmanned Aerial Vehicles (UAVs) and Internet of Things (IoT) devices. The output of WP1 are the algorithms to be used in WP2 to implement the data collection and the logbook. The work package consists of five tasks, plus Task TEID. WP1 and WP2 will interoperate.
- Task 1.1 (T1.1) (M1-6): Hardware Selection:
- Vision-chips (UNIPG, UOWM, UPB, TNI, TUBS).
- IoT components (TUBS, TNI).
- Task 1.2 (T1.2) (M3-16): Orchard- and Tree-Navigation Algorithms for UAVs: initially the main interest will be the high quality of the images and the simplicity of use of the proposed algorithms. The algorithm will be designed and developed by the ICT experts, but they will be utilized on the fields by plant health operators (UNIPG, UOWM, UPB).
- Task 1.3 (T1.3) (M16-30): Advanced Navigation Algorithms: The UAV trajectories will be defined with respect to points of interest. For example, a navigation algorithm for an abundance map of HH must create a coverage model obtained from samples of different locations/trees (GPS establishing) using a consensus-based algorithm. The consensus mechanism can indicate problematic measurement zones that will be solved by a furthest ML approach (UNIPG, UOWM, UPB).
- Task 1.4 (T1.4) (M1-15): IoT Protocol Design: Design of the protocols for distributing the data requests and for collecting the data, possibly aggregated (TNI, TUBS).
- Task 1.5 (T1.5) (M12-24): IoT Implementation and UAV integration: Cooperation UAVs and IoTs (TNI, TUB,UNIPG).
- Task TEID [M24-36] : Demonstration Navigation Algorithms and IoT protocols (all).
- Deliverable 1.1 (D1.1) [M16]: Hardware selection, functional navigation algorithms and IoT architecture.
- Milestone 1 (MS1) [M16]: Basic UAV algorithms and IoT protocol design architecture.
WP2 (M1-36) (Data Collection and Data Processing): As a first step, this WP collects data including images using the UAV algorithms released by WP1. Then, the data are elaborated using Machine Learning techniques. First goal of ML in WP2 is to output a novel algorithm for the detection of HH, built upon expertise in supervised learning, neural networks and decision fusion in combining NN classifiers (deep CNN, combined with efficient features) for detection, classification and segmentation of objects of interest. The second goal of MLin WP2 is to discover a method to estimate the numerousness of the HH population in selected farms involved by the partners. The numerousness will be the input for a tool to make decisions on the countermeasures. For example, we can use the tool to decide whether to perform a treatment or postpone it. The most ambitious goal of WP2 is to obtain an epidemiological model, which is extremely difficult for HH due its high mobility and thus fast changes in the scenario. A possibility is to make a kind of experiment in silico in regions with mapped orchards with historical data on: weather conditions, phenology of the trees, HH field monitoring and known overwintering sites. The algorithms will be tuned under the expertise of the entomologists. The output of this WP are the ML algorithms adapted for detection, recognition, and investigation of HH population features.
The WP consists of 4 tasks and it is completed with the Task TEID.
- Task 2.1 (T2.1) [M4-8, M16-20 (seasonal)]: Implementation on the fields of the data collection using the algorithms designed in WP1 (UNIMORE, UPB, UOWM, TNI, TUBS).
- Task 2.2 (T2.2) [M4-12]: Feature extraction algorithms, creation data-set shared among the partners, HH vision detection algorithm (UOWM, UNIMORE, UPB).
- Task 2.3 (T2.3) [M12-30]: Data analyses, ML techniques for deriving a model for abundance of the pest, including a tool for short-term daily decisions (UOWM, UNIMORE, UPB).
- Task 2.4 (T2.4) [M24-36] Epidemiological model: This task will work towards a reference epidemiological model for the scientific and agriculture communities (UPB, UOWM,UNIMORE).
- Task TEID [M24-36]: Demonstration of data collection and ML results and their integration/impact on the system (all).
- Deliverable 2.1 (D2.1) [M24]: Functional data acquisition and ML processing systems.
- Deliverable 2.2 (D2.2) [M36]: Epidemiological model feasibility evaluation and proof-of-concept.
- Milestone 2 (MS2) [M24]: ML algorithms for HH detection and feature extraction.
WP3 (M1-36) (Fruit Classification): Two or three different fruit types will be selected to develop the automatic post-harvest fruit classification system, based on the economic impact of HH pest and on the ripening period. During the first year, near infrared hyperspectral imaging (NIR-HSI) will be used to acquire images. The feasibility of using complementary technologies, for which the potential in fruit sorting have been recently demonstrated, will be explored for improving defect detection ability. In particular, we will investigate the use of terahertz spectroscopic imaging to detect (sub)surface defects, and microwave imaging or ultrasound probing for the detection of deeper defects. The feasibility of a unified classification model will then be evaluated by considering data fusion strategies. In the second year, the models will be further optimized using the data acquired, making the classification models more robust with respect to between-years variability. All the models will be fully validated on the samples harvested in the third year, in collaboration with specific fruit storage/sorting industries involved by the partners. The output of WP3 is an assessment of the fruit damages, which is also an implicit further assessment of the size of the pest infestation. WP3 consists of three tasks. It is completed by the Task TEID.
- Task 3.1 (T3.1) [M1-12]: Develop classification models from NIR-HSI to detect fruit defects (IMEC, UNIMORE).
- Task 3.2 (T3.2) [M1-12]: Investigate the feasibility of complementary technologies for detection of fruit defects (IMEC).
- Task 3.3 (T3.3) [M12-24]: Data fusion and optimization of robustness of fruit defect detection models (IMEC, UNIMORE).
- Task TEID [M24-36]: Evaluation & Demonstration fruit-processing. Integrate the results of fruit processing to indirectly evaluate the effectiveness of the tools devised in WP2.
- Deliverable 3.1 (D3.1) [M12]: First classification model based on HSI and report on feasibility of complementary technologies for detection of fruit defects.
- Deliverable 3.2 (D3.2) [M24]: Improved classification model and report on data fusion approach.
- Deliverable 3.3 (D3.3) [M36]: Fruit classification system based on data fusion of NIR-HSI and selected complementary technology.
- Milestone 3 (MS3) [M12]: Selection of most suitable complementary technology.
WP4 (M1-36): (Data Management): We anticipate that various data sets collected will be of interest to others, e.g., images to entomologists and UAV path data to computer scientists & engineers. The organization of a logbook of the data will be studied. The use of trusted execution and blockchains for assessing the data truth will be investigated. This can ensure the consumers and other later stages of the food-production chain about the food production. WP4 consists of two tasks:
- Task 4.1 (T4.1) [M1-12]: A Data Management Plan (DMP) (TUBS, UNIPG).
- Task 4.2 (T4.2) [M12-36]: A seasonal case study logbook system will be provided. We will investigate whether it can be prepared in a tamper-proof manner, for example, using blockchain techniques, since solutions for certified data-log may increase the final consumer confidence in the fruit production process and may satisfy the criteria of the integrated pest policy (TUBS, UNIPG).
- Task TEID [M24-36]: Evaluation, Integration, & Demonstration of DMP and logbook in HALY.ID project.
- Deliverable D4.1 [M3]: DMP creation and updating daily logbook system.
- Deliverable D4.2 [M30]: Logbook system release.
- Milestone 4 (MS4) [M12]: Logbook system architecture and implementation.