RESEARCH GROUPS

The use of AI for decision-making in, for example, financial or legal situations, has raised numerous ethical issues regarding both fairness and privacy. In particular, disadvantaged groups may receive unfair decisions due to biases in the datasets which are transferred to the AI models. In addition, da-tasets used for AI often contain sensitive information which can be transferred to the AI model, re-sulting in potential privacy breaches against individuals. The problem of how to address both privacy and fairness in a principled manner is of critical importance and remains unresolved. The objective of this collaboration is to develop principled approaches for the ethical use of data in AI technologies, by addressing issues of privacy and fairness, and their interaction.

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 568154038

Funding Body

Funding Body

DFG - Deutsche Forschungsgemeinschaft

In ILLUMINATION, a toolbox of technical privacy methods and interdisciplinary recommendations for privacy-preserving use of centralized LLMs in the healthcare sector is being developed. The technical methods enable LLM users to implement appropriate data protection for their users in the spirit of “Privacy by Design.” The recommendations—based on technical, legal, human-centered, and application-specific perspectives—contribute to strengthening responsible and privacy-compliant LLM practices, support navigation through the complex landscape of LLM implementation, and lay the foundation for legally compliant privacy methods in LLM-based applications.

Funding Body

Funding Body

Bundesministerium für Forschung, Technologie und Raumfahrt

Partner

Partner

Freie Universität Berlin

Freie Universität Berlin

Partner

Universität Heidelberg

Universität Heidelberg

Partner

Charité – Universitätsmedizin Berlin

Charité – Universitätsmedizin Berlin

Partner

Algonaut (KMU)

Algonaut (KMU)

While there is a long-standing tradition of training various machine learning models for different application tasks on visual data, only the most recent advances in the domain of foundation models managed to unify those endeavors into obtaining highly powerful multi-purpose deep learning models. These models, such as DINO_v21, or SAM,2 are pre-trained on large amounts of public data which turns them into efficient feature extractors. Using only small amounts of sensitive downstream data and reduced compute resources in comparison to a from-scratch training, these feature extractors can then be adapted (through full or partial fine-tuning, transfer learning, or prompting approaches) to solve a wide range of downstream applications. Our goal is to bring foundation models and the new and powerful learning paradigms for their adaptation to the complex and sensitive medical domain with a focus on CT and MRI data.

Funding Body

Funding Body

Helmholtz - Artificial Intelligence Cooperation Unit

Partner

Partner

Deutsches Krebsforschungszentrum (DKFZ)

Novel foundation models (FMs) like GPT, LLaMA, and Stable Diffusion are achieving exceptional performance across diverse tasks, generating high-quality text, images, and audio, and driving industry innovations. This progress stems from a shift in machine learning paradigm: instead of training task-specific models on curated datasets, FMs are first pretrained on vast, uncurated data to become strong general-purpose models, then adapted on smaller, domain-specific datasets for specific tasks.

However, FMs leak information from their training data. For example, recent studies reveal that they can re-create individual data points from their pretraining and adaptation datasets. This poses serious privacy risks when private data is involved. Preventing exposure requires developing methods to ensure privacy-preservation throughout FMs' lifecycle, from pretraining to deployment. To achieve this, our project will identify sources of privacy leakage, provide privacy guarantees over both pretraining and adaptation, and audit FMs to detect privacy violations. Therefore, we must overcome three major challenges: the limited understanding of privacy risks in FM pretraining, the lack of formal joint privacy guarantees for pretraining and adaptation, and the ineffectiveness of current privacy auditing methods.

The solution that we propose will establish a novel theoretical framework for privacy guarantees in FMs under the pretrain-adapt paradigm. Our fundamental innovations rely on the insight that, due to complex interdependencies between pretraining and adaptation data, different data points require individual levels of protection to prevent leakage. Advancing methods for identifying, achieving, and accounting for such individual guarantees will enable us to formally bound privacy leakage over both training stages and to detect violations. These innovations will allow society to benefit from technological advancements through FMs without compromising individuals' privacy.

Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or [name of the granting authority]. Neither the European Union nor the granting authority can be held responsible for them.

Funding Body

Funding Body

European Union - EU – Starting Grant (ERC-2025-StG)

Self-Supervised Learning (SSL) has emerged as a new and powerful learning paradigm in machine learning. In contrast to standard supervised learning, which requires labeled data, SSL relies on unlabeled data to train high-performing feature encoders. This gives SSL the potential to unlock the value of the vast amounts of unlabeled data that remain unused in supervised learning. This is particularly relevant in sensitive domains such as medical imaging or biometrics, where labeling data is inherently difficult and expensive.

Despite its promising performance, the applicability of SSL encoders in sensitive areas has so far been limited. A major reason for this is that SSL encoders have been shown to leak sensitive information about their training data, posing a significant privacy risk. To date, there are no targeted methods to mitigate this risk while maintaining encoder performance.

In this project, we leverage the concept of memorization—that is, the ability of a machine learning model to retain information about its training data—to systematically analyze and mitigate privacy risks in SSL. First, we develop a fundamental understanding of memorization in SSL and identify where information about individual training data points is stored in SSL encoders. We then quantify the resulting privacy risks, identify their causes, and formally link the privacy risk of individual data points to their degree of memorization.

Finally, based on our insights into why certain data points exhibit high memorization and where they are stored within the encoders, we design targeted strategies to reduce privacy risks while still enabling the training of high-performing encoders. Our approach opens a new path toward deploying powerful SSL encoders in sensitive domains while preserving the privacy of their training data.

Funding Body

Funding Body

Gefördert durch die Deutsche Forschungsgemeinschaft (DFG) - 550224287

The project aims to enable software developers to quickly gather information about code snippets they reuse in their codebase. This includes notifications about changes to the code sources, alerts about security issues and bugs, or summaries of discussions related to such code snippets. The Field Study Fellowship is intended to adapt the software to the needs of developers and to improve the effectiveness of code reuse.

Funding Body

Funding Body

Helmholtz - Field Study Fellowship Modul

Communication efficiency is one of the central challenges for cryptography. Modern distributed computing techniques work on large quantities of data, and critically depend on keeping the amount of information exchanged between parties as low as possible. However, classical cryptographic protocols for secure distributed computation cause a prohibitive blow-up of communication in this setting. Laconic cryptography is an emerging paradigm in cryptography aiming to realize protocols for complex tasks with a minimal amount of interaction and a sub-linear overall communication complexity. If we manage to construct truly efficient laconic protocols, we could add a cryptographic layer of protection to modern data-driven techniques in computing. My initial results in laconic cryptography did not just demonstrate the potential of this paradigm, but proved to be a game-changer in solving several long standing open problems in cryptography, e.g. enabling me to construct identity-based encryption from weak assumptions. However, the field faces two major challenges: (a) Current constructions employ techniques that are inherently inefficient. (b) The most advanced notions in laconic cryptography are only known from very specific combinations of assumptions, and are therefore just one cryptanalytic breakthrough away from becoming void. This project will make a leap forward in both challenges. I will systematically address these challenges in a work program which pursues the following objectives: (i) Develop new tools and mechanisms to realize crucial cryptographic primitives in a compact way. (ii) Design efficient protocols for advanced laconic functionalities which sidestep the need for inherently inefficient low-level techniques and widen the foundation of underlying assumptions. (iii) Strengthen the conceptual bridge between laconic cryptography and cryptographically secure obfuscation, transferring new techniques and ideas between these domains.

Partner

Funding Body

European Research Council

In recent times, generative machine learning (ML) models, like ChatGPT and Stable diffusion, have grown rapidly in creating high-quality texts and images. Despite their remarkable achievements in producing high-quality outputs, this project underscores the imperative of prioritizing their safety, a facet frequently understudied. Safety here means ensuring the models operate without causing unintended harm, especially in terms of protecting the confidentiality and integrity of the processed data, and guarding against the generation of misleading or harmful content. Additionally, it refers to the safety of the models themselves against illegitimate usage. Our project is built upon a fundamental hypothesis that underscores the vital connection between the safety of generative models and how their training data is managed: the bedrock of reliable and legally compliant AI systems lies in the nature and handling of the data that these systems are trained on. To develop this connection, we have structured our research into three parts. Our first objective is on exploring the learning mechanisms of generative models, specifically focusing on their capabilities to “memorize” training data. This aspect is particularly critical as it often leads to unintended exposure of sensitive or copyrighted material during the models' inference stage. Our second objective is centered around devising innovative verification techniques designed to serve two primary scenarios: The first scenario empowers individuals, such as artists, to determine if their creative work, even not explicitly memorized by the model, was improperly used in its training. The second scenario, termed output verification, aims at enabling model owners to confirm or refute the generation of potentially harmful content by their models. Finally, we propose groundbreaking strategies for safeguarding both generative models and the parties who trained them (usually API providers) using high amounts of computational power and manual labor. Our approach involves the development of active defenses to thwart attempts at stealing the models. In cases where the theft still occurs, we explore ownership resolution techniques to legally challenge such actions. In summary, our project aims to enhance the safety of generative models by investigating memorization, devising verification techniques, and proposing strategies for safeguarding both models and data. By compiling our methods into a safe generative ML framework and validating it in real-life applications, we aspire to contribute to the responsible deployment of generative models, ensuring their integrity and confidentiality in the evolving landscape of artificial intelligence. We plan to disseminate our framework through educational materials and an open-source library as well as validate the effectiveness of our methods on real-life applications.

Funding Body

Funding Body

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 545047250

Increasing connectivity and the use of AI in industrial control systems (ICS) create a variety of chal-lenges. Attackers use AI-supported methods to automatically identify and exploit vulnerabilities. Tra-ditional security measures such as rule-based or behavior-based intrusion detection systems (IDS) are often unable to detect these dynamic threats. Outdated legacy systems, insecure protocols, and inad-equate encryption exacerbate the problem. In addition, concerns about downtime and compatibility issues make it difficult to implement updates and patches in a timely manner.

From a scientific and technical perspective, the project aims to address these challenges as it builds on established methods in AI-based cybersecurity while filling clearly defined gaps in industrial and OT environments. In the short to medium term, the project advances hybrid detection architectures com-bining signature-based and AI-based methods, adversarial robustness analysis of AI models, federated learning under industrial constraints with heterogeneous data and limited compute, and the system-atic investigation of data poisoning and AI-generated attack patterns.

Our research is directed towards reducing dependence on regular software updates, minimizes down-time risks, and enables companies to maintain system integrity without constant manual intervention. Furthermore, the work systematically examines how such approaches can contribute to alignment with stringent regulatory frameworks, including Critical Infrastructure and IT Security Act 2.0. Within this context, the project develops and experimentally validates an innovative cybersecurity architec-ture aimed at strengthening the resilience of critical infrastructures while analyzing its potential impact on operational stability and continuity.

Funding Body

Funding Body

Federal Ministry of Research, Technology and Space (BMFTR)

In January 2023, the European Digital Innovation Hub for AI and Cybersecurity (initially called DIH4AISec, and soon later updated to DAISEC, www.daisec.de/en) was established for the Metropolitan Region Hannover-Göttingen-Braunschweig-Wolfsburg, combining a consortium with internationally leading expertise in AI , cybersecurity (including AI safety) , production , and skilled crafts .

Furthermore DAISEC will offer training and support for legal aspects of AI deployment, with a dedicated legal expert position. In the first two years of operation, DAISEC established itself as a crucial component of the existing regional digital innovation landscape, enabling the application of state-of-the-art AI and Cybersecurity technology to gain competitive advantages. With more than 8000 customers served so far in over 600 individual and group services, DAISEC has significantly contributed to digital innovation in companies and the public sector regionally, nationally, and on the European level, and established an excellent reputation for highquality offerings, as seen in the overwhelmingly positive feedback to our services.

By now, DAISEC is fully integrated in the economic development infrastructure, with strong collaborative ties that have been established with over 30 associated partners including regional government organizations, regional and national economic sector organizations, chambers of commerce and skilled crafts, economic networks, and innovation funding entities. Based on these successes, DAISEC has now become the go-to place for AI and Cybersecurity support in the region.

We propose to continue and extend the DAISEC operations for the next EDIH funding period (2026-2028) and move it during that period to an increasingly self-sustaining state that will ensure a long-term operation beyond the EU funding period, partially through the gradual shift from free to paid services and in addition securing partial regional and national funding by the end of 2028.

Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or [name of the granting authority]. Neither the European Union nor the granting authority can be held responsible for them.

Funding Body

Funding Body

European Union

Partner

Partner/ Coordinator

Gottfried Wilhelm Leibniz Universität Hannover LUH

Gottfried Wilhelm Leibniz Universität Hannover LUH

Partner

IT-Dienstleitungsgesellschaft mbH Emsland

IT-Dienstleitungsgesellschaft mbH Emsland

Partner

Interessengemeinschaft des Heinz-Piest-Instituts für Handwerks-technik an der Leibniz Universität Hannover e.V.

Interessengemeinschaft des Heinz-Piest-Instituts für Handwerks-technik an der Leibniz Universität Hannover e.V.

Partner

Technische Universität Braunschweig

Technische Universität Braunschweig

Modern industrial and production environments are becoming increasingly computerized and networked; thus, cybersecurity is becoming increasingly relevant for them. Cybersecurity is a necessary prerequisite as well as an enabler for novel applications. Attacks against industrial and production systems can cause downtime, damage to machines, and injury to human operators, potentially impacting the entire supply chain. It can also facilitate industrial espionage by stealing vast amounts of digital information.
Achieving cybersecurity for complex and interconnected industrial and production environments is a complex challenge that requires a multi-faceted approach and a multitude of perspectives, ranging from “classical” software-, hardware-, and network security over formal analysis and formal guarantees, and human factors to actual experiences in production systems and related workflows.
In the proposed research project we aim to tackle these challenges with a team covering diverse backgrounds and showing excellent scientific quality. The team integrates the joint expertise of key international researchers from the CISPA Helmholtz Center for Information Security (CISPA) and Leibniz University Hannover (LUH). Key goals include excellent research advancing the currently very limited state-of-the-art and fostering cooperation between CISPA and LUH broadly.

Funding Body

Funding Body

Niedersächsisches Ministerium für Wissenschaft und Kultur - zukunft.niedersachsen

Partner

Partner

Leibniz University Hannover (Lead)

The quest for a science of perspicuous computing continues. With the results that were achieved in the first funding period, we are spearheading a larger movement towards building and reasoning about software-based systems that are understandable and predictable. As a result, CPEC is gradually enlarging its scope in its second funding period. This pertains to three interrelated facets of our research:

·      Deepening investigation of various feedback loops within the system lifecycle which are required to feed system analysis insights – in particular, insights from inspection-time justification – back into the design-time engineering of perspicuous systems.

·      Emphasising human-centred and psychological research regarding the human-in-the-loop, reflecting the need to investigate the interaction of perspicuous systems with various groups of human stakeholders.

·      Interfacing to the societal dimension of perspicuity – society-in-the-loop – echoing the increasing number of regulatory requirements regarding perspicuity put forward in recent years.

CPEC joins the pertinent forces at Saarbrücken and Dresden that are apt to master the challenge of putting perspicuous computing research into the loop. It comprises computer science experts and links to psychology and juridical expertise at Universität des Saarlandes and Technische Universität Dresden as well as the Max Planck Institute for Software Systems and the CISPA Helmholtz Center for Information Security. The participating institutions have developed a joint research agenda to deepen the transregional network of experts in perspicuous systems. It will serve our society in its need to stay in well-informed control over the computerised systems we all interact with. It enables comprehension and control in a cyber-physical world.

Source: https://www.perspicuous-computing.science/research/ (03.05.2023)

Funding Body

Deutsche Forschungsgemeinschaft

Partner

Saarland University

Technical University Dresden

Max Planck Institute for Software Systems

The central role of information technology in all aspects of our private and professional lives has led to a fundamental change in the type of program properties we care about. Up to now, the focus has been on functional correctness; in the future, requirements that reflect our societal values, like privacy, fairness. The central role of information technology in all aspects of our private and professional lives has led to a fundamental change in the type of program properties we care about. Up to now, the focus has been on functional correctness; in the future, requirements that reflect our societal values, like privacy, fairness, and explainability will be far more important. These properties belong to the class of hyperproperties, which represent sets of sets of execution traces and can therefore specify the relationship between different computations of a reactive system. Previous work has focussed on individual hyperproperties like noninterference or restricted classes such as k-hypersafety; this project sets out to develop a unified theory for general hyperproperties. We will develop a formal specification language and effective algorithms for logical reasoning, verification, and program synthesis. The central idea is to use the type and alternation structure of the logical quantifiers, ranging from classic first-order and second-order quantification to quantifiers over rich data domains and quantitative operators for statistical analysis, as the fundamental structure that partitions the broad concept of hyperproperties into specific property classes; each particular class is then supported by algorithms that provide a uniform solution for all the properties within the class. The project will bring the analysis of hyperproperties to the level of traditional notions of safety and reliability, and provide a rigorous foundation for the debate about standards for privacy, fairness, and explainability that future software-based systems will be measured against.

Funding Body

European Union

ELLE proposes a European doctoral network aimed to train a new generation of AI scientists and professionals that will promote and reinforce the European leadership in the development of secure and safe AI technology. Within ELLE, we will tackle four main scientific and technical challenges that are hindering the concrete imple-mentation of legally-compliant AI models with the provisions of the EU AI Act and related regulations, espe-cially after the recent paradigm shift that has fostered the wide adoption of foundation models. These chal-lenges include ensuring the main trustworthiness properties of learning algorithms, i.e., (i) technical robustness and safety; (ii) privacy and data governance; and (iii) interpretability and transparency; while considering mod-els that can be (iv) continuously adapted in security-sensitive and safety-critical tasks where the data is

streaming, non-stationary and showing unexpected changes of its statistical characteristics. ELLE aims to tackle these challenges jointly, building upon three main pillars: (1) the proposal of 15 distinct, multi-disciplinary Ph.D. research programs that are deeply intertwined within the four aforementioned challenges via carefully defined co-supervisions and secondments; (2) the development of open courses, educational material, and workshops, with an emphasis on hands-on, collaborative learning and attention to transferable skills; and (3) the promo-tion of mobility and knowledge transfer; all within a training network that includes a selection of Europe's fin-est researchers, supported by the ELLIS Society and stemming from the ELSA European Network of Excellence. ELLE will integrate novel and rigorous science developments with training skilled doctoral researchers to nur-ture and sustain the development and deployment of AI technology in compliance with the EU AI Act and re-lated regulations, serving and promoting foundational EU values.

Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or [name of the granting authority]. Neither the European Union nor the granting authority can be held responsible for them.

Funding Body

Funding Body

European Union - Horizon Europe

Partner

Partner/Coordinator

Università degli Studi di Cagliari (IT)

Università degli Studi di Cagliari (IT)

For improving the capabilities of general-purpose AI models and for extending their applicability to domains where the temporal dimension–among several others–is of importance, we will target the development of the next generation of Multimodal Space-Time Foundation Models (MSTFMs). These will combine spatio-temporal understanding, which is important even for modalities such as the visual one that have already been introduced in large generative models, with the effective management of new time-relevant modalities that are yet to be supported in foundation models, such as industrial time series data, remote sensing data and health-related measurements. Real and synthetic data, to mitigate data scarcity, will be leveraged for training general-purpose MSTFMs and for further adapting them for specific downstream tasks. Real data used for training will include data directly provided by members of the consortium as well as data from relevant European Data Spaces, while complementary synthetic data will be generated by exploiting existing generative AI  capabilities as well as new ones developed in the project. European HPC infrastructure is directly included in the consortium to ensure the availability of the necessary computing resources.

Funding Body

Funding Body

European Union

Subproject: Federated Analytics and Machine Learning, Privacy Models, and Benchmarking: CISPA

The goal of PrivateAIM is to develop a federated platform for machine learning (ML) and data analysis within the framework of the Medical Informatics Initiative (MII), in which analyses are brought to the data rather than the data being brought to the analyses. Methods that enable distributed data processing in the Data Integration Centers (DIZ) established by the MII are important for several reasons:

1. Patient data may only be used without consent if anonymity is ensured;

2. Federated technologies can help connect the MII with other health data networks.

However, the mechanisms currently established within the MII have significant limitations and, for example, are not suitable for complex ML and data science tasks. Moreover, federated platforms developed in other contexts are:

1. complicated to set up and operate,

2. support only a limited number of analysis or ML methods,

3. do not implement modern privacy-preserving technologies, and

4. are not scalable or mature.

The PrivateAIM consortium, supported by all MII consortia, brings together experts to develop the next generation of federated analysis and ML platforms for the MII. The Federated Learning and Analysis Methods Platform (FLAME) will combine state-of-the-art federated methods with innovative privacy models for multimodal data. Using components for monitoring and controlling the level of protection, these will be integrated into a distributed infrastructure that can be easily adopted by the Data Integration Centers.

The practical implementation will be supported by addressing challenges at the intersection of technology and law, developing concepts for operation by hospital IT departments, and coordinating with ethics committees and data protection officers.

Funding Body

Funding Body

Bundesministerium für Forschung, Technologie und Raumfahrt

Partner

Project Partner

Universität zu Köln

Universität zu Köln

Project Partner

Universität Leipzig

Universität Leipzig

Project Partner

Deutsches Krebsforschungszentrum (DKFZ)

Project Partner

Universität Ulm

Universität Ulm

Project Partner

Universität Heidelberg

Universität Heidelberg

Project Partner

Albert-Ludwigs-Universität Freiburg

Albert-Ludwigs-Universität Freiburg

Project Partner

Charité - Universitätsmedizin Berlin

Charité - Universitätsmedizin Berlin

Project Partner

Technische Universität München

Technische Universität München

Project Partner

Eberhard Karls Universität Tübingen

Eberhard Karls Universität Tübingen

Project Partner

TMF - Technologie- und Methodenplattform für die vernetzte medizinische Forschung e. V. Berlin

TMF - Technologie- und Methodenplattform für die vernetzte medizinische Forschung e. V. Berlin

Project Partner

Friedrich-Alexander-Universität Erlangen-Nürnberg

Friedrich-Alexander-Universität Erlangen-Nürnberg

Project Partner

Ludwig-Maximilians-Universität München

Ludwig-Maximilians-Universität München

Modern software systems are complex, interconnected, and continuously evolving, resulting in a wide range of vulnerabilities introduced by human development errors and the use of AI coding tools. Exist-ing static and dynamic security tools struggle to detect context-dependent issues such as business logic flaws and authorization bypasses and often produce high numbers of false positives. As a result, many real-world vulnerabilities remain undetected, exposing organizations to significant security, regula-tory, and operational risks and creating a strong need for an accurate, context-aware software security approach that integrates seamlessly into real-world development workflows.

The project “C-Mind-Security” aims to develop and validate an AI-native, context-aware software se-curity system that reliably detects, triages, and remediates complex vulnerabilities missed by conven-tional tools. The system combines a specialized AI model, program analyzers, and an evolving knowledge base, enabling AI agents to autonomously perform vulnerability analysis and remediation. Development follows an iterative process with continuous benchmarking against state-of-the-art tools and validation in real-world application environments.

By combining AI-native reasoning with software analysis techniques, the project introduces a novel approach to software security. Key innovations include holistic context modeling of software, a policy-aware AI model, and AI agents capable of precise detection and remediation of vulnerabilities. The resulting architecture provides a scalable foundation for broader code analysis tasks and contributes to digital sovereignty through on-premises and EU-based deployment.

Funding Body

Funding Body

Federal Ministry of Research, Technology and Space (BMFTR)

The "AlgenCY" project aims to thoroughly explore and evaluate the diverse possibilities and challenges that generative AI methods bring to the field of cybersecurity. We seek to understand how these technologies can be effectively used to defend against cyber threats, while also identifying the potential vulnerabilities and risks they may pose themselves. The goal of this research is to make well-founded predictions about the future impact of generative AI on cybersecurity. Based on these insights, targeted strategies and solutions will be developed to strengthen digital security. As part of this subproject, CISPA will specifically investigate the security aspects of large language models (LLMs) and other generative AI methods. The focus will be both on securing these technologies themselves and on exploring how they can be applied within the context of IT security.

Funding Body

Funding Body

Bundesministerium für Forschung, Technologie und Raumfahrt

Partner

Project Partner

Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung eingetragener Verein, München

Project Partner

Freie Universität Berlin, Berlin

Freie Universität Berlin, Berlin

Project Partner

Technische Universität Berlin, Berlin

Technische Universität Berlin, Berlin

Associate partner

Aleph Alpha GmbH

Aleph Alpha GmbH

Computer systems in banks and insurance companies, as well as in autonomous vehicles or satellites, are promising targets for cyberattacks and must be protected to withstand such attacks or to endure them while continuing to operate safely. Unfortunately, attackers gain increasing opportunities as these systems grow more complex, which means that in defending them, we must assume that some attacks could be successful. Fortunately, resilience techniques already exist, such as triple replication of the computer system along with the protocol, allowing the correct result to be determined through consensus even if one instance delivers a faulty result due to a successful cyberattack. However, for these techniques to be applied, they must be adapted to the system’s structure, especially how the components of the system interact with each other. The resilience techniques developed so far are limited to certain forms of interaction, and developing new techniques for more complex forms of interaction remains a difficult and error-prone task. In particular, tools that support the development of such protocols through correctness checks can usually only be applied to fully completed protocols, and their use typically requires rare expert knowledge. In the FM-CReST project, researchers from CISPA, Germany, and the SnT at the University of Luxembourg have joined forces to develop a new class of highly automated and easy-to-use tools that assist in designing provably correct resilience protocols. To achieve this, we rely on co-design of protocols with complex interaction patterns and base the development of our tools on observations made during protocol design, aiming to simplify similar tasks in the future.

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 513487900.

Funding Body

Funding Body

Deutsche Forschungsgemeinschaft

Novel on-body devices offer new, scalable user interfaces that are more intuitive and direct to use. However, the close proximity of input and output to the body introduces serious new privacy risks for users: the large hand and finger gestures typically used for input are significantly more susceptible to observation by third parties than established forms of touch input. This is even more true for visual output on the body. This is particularly problematic since on-body devices are typically used during mobile activities in non-private environments. The primary goal of this project is to contribute to the scalability of on-body computing in public spaces by developing interaction techniques for input and output of private information that provide improved resistance to privacy violations. Our approach focuses on leveraging the unique interaction properties of the human body: high manual dexterity, high tactile sensitivity, and a large available surface for input and output, combined with the ability to flexibly shield input and output through variable body posture. These properties can form the basis for new body-based input and output techniques that are scalable and (practically) unobservable. This goal is largely unexplored so far. It is very challenging due to the new and highly diverse forms and scales of on-body devices as well as novel forms of multimodal input and output. These challenges are further complicated by the inherent complexity of social environments, respective proxemics, and the attention of users and bystanders. To create a design space for these interactions, we will empirically investigate the privacy of tactile input, visual, and haptic output at various body locations, depending on body posture and proxemic configurations. Subsequently, we will systematically design and implement body-related input gestures and scalable techniques for multimodal interaction that preserve privacy in social environments according to a generalized threat model. We will use attention models that incorporate the human body. The new interaction techniques will be empirically evaluated with users in realistic scenarios and in the laboratory to assess how their properties affect usability, privacy, and scalability. Both will help us understand the internal and external validity of our approach. We expect the results of this project to make a significant contribution to establishing the foundations for scalable body-based interactions that preserve privacy.

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 521601028

Funding Body

Funding Body

Deutsche Forschungsgemeinschaft

Many of today’s critical infrastructures such as power grids or cellular networks are distributed systems which are comprised of autonomous nodes connected over a network. To avoid single points of failure, a central goal in distributed systems is to implement such systems in a faulttolerant manner. Fault-tolerant systems remain secure and available even if some minority of nodes crash or spread incorrect information. To improve robustness and scalability, cuttingedge systems frequently rely on cryptography. In spite of its benefits, careless use of cryptography can incur performance penalties or lead to vulnerabilities. Both of these aspects significantly complicate its use in practice. As a result, many real-world systems use cryptography sparingly and therefore lack both robustness and scalability. To improve this unsatisfying state of affairs, the objectives of CRYPTOSYSTEMS are as follows:

• New Formal Models. Established formal models from cryptography and distributed systems pursue independent security goals and therefore lack compatibility. CRYPTOSYSTEMS will develop new, more appropriate formal models for analyzing the security of cryptographic distributed systems.

• Efficient and Robust Distributed Algorithms. The use of cryptography is currently underexplored in distributed systems. CRYPTOSYSTEMS will present exciting new applications of cryptography that will lead to the development of more robust and scalable distributed systems.

• Cryptography for Distributed Algorithms. Cryptography is seldom developed with distributed algorithms as its primary use-case in mind. This results in inefficient or unwieldy cryptography which hampers the efficiency of distributed algorithms. To counteract this, CRYPTOSYSTEMS will develop cutting-edge cryptography such as compact signatures and communication efficient distributed randomness generation routines. Importantly, these tools will be specifically geared toward use in distributed algorithms.

Funding Body

European Union

The specific objectives for the fellowship include developing novel methods and systems to autonomously exploit binaries by combining program analysis, symbolic reasoning, and generative AI models, while also exploring countermeasures and defense mechanisms. By the end of the fellowship, Nanda Rani is expected to demonstrate a comprehensive suite of re-search outcomes, including (i) autonomous binary exploitation prototypes powered by LLM-driven reasoning, (ii) methodologies for benchmarking and evaluating autonomous exploitation systems on CTF-style and real-world binaries, (iii) curated datasets and reproducible pipe-lines for training and testing AI-driven exploit generation. 

Funding Body

Funding Body

Impulse and Networking Fund from the Helmholtz Association

The research project “SisWiss” aims to enable companies to use large language models (LLMs) securely and in compliance with legal regulations. This technology offers great potential for efficiently leveraging distributed knowledge bases and gaining insights into business processes. At the same time, LLMs pose a risk, as they can expose sensitive information. Therefore, the project team is developing practical tools and methods for evaluating, improving, and certifying language models—based on existing open-source models.

Core components of the project include a software library and a testing module to assess robustness against attacks, such as manipulative input prompts. In addition, theoretical approaches to certification according to EU legal requirements, such as the AI Act and the Data Governance Act, are being explored. The project’s outcomes are intended to help small and medium-sized enterprises, in particular, process confidential data within their own infrastructure without relying on external services.

Overall, the project combines theoretical approaches to the security certification of LLMs with concrete security testing, enabling companies to use language models safely and in accordance with existing and upcoming EU regulations.

Funding Body

Funding Body

Bundesministerium für Forschung, Technologie und Raumfahrt

Partner

Partner

Leibniz Universität Hannover

Leibniz Universität Hannover

Partner

L3S Research Center (L3S)

L3S Research Center (L3S)

Partner

AandBI.com GmbH (AandBI®)

AandBI.com GmbH (AandBI®)

Partner

Deutsches Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)

Deutsches Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)

Partner

Laverana GmbH & Co. KG (Laverana)

Laverana GmbH & Co. KG (Laverana)

Machine learning models are growing larger and more complex, making training increasingly resource-demanding. Concurrently, our world, and hence the training data is perpetually evolving. This requires continual model updating or retraining to address changing training data. Presently, the most reliable course to handle such distribution shifts is to retrain models from scratch on new training data. This results in substantial resource usage, increased CO2 footprint, elevated energy consumption, and limits the decisive ML progress to large-scale industry players. Imagine a world in which models help each other learn. When the data distribution changes, a complete retraining of models could be avoided if the new model could learn from the outdated one by using reliable and provably effective methods. Furthermore, the convention of relying on large, versatile monolithic models could then give way to a consortium of smaller specialized models, with each contributing its specific domain knowledge when needed. By encouraging this form of decentralization, we could reduce resource consumption as the individual components can be updated independently of each other. Drawing on groundbreaking research in distributed ML model training, CollectiveMinds aspires to design adaptable ML models. These models can effectively manage updates in training data and task modifications, while also enabling efficient knowledge exchange across various models, thereby fostering widescale collaborative learning and constructing a sustainable framework for collaborative machine intelligence. This initiative could revolutionize sectors like healthcare, where there is limited training data, and trustworthy AI that demands guarantees on data ownership and control. Furthermore, it could foster improved collaborative research within the realm of science. CollectiveMinds embodies a significant paradigm shift towards democratizing ML, focusing on cooperative intellectual efforts.

"Funded by the European Union. Views and opinions expressed are however those ofthe author(s) only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held re-sponsible for them."

Funding Body

Funding Body

Funded by the European Union

The exponential growth of datasets and models in machine learning necessitates scalable optimization methods to efficiently leverage distributed systems. This proposal addresses the challenges of distributed deep learning optimization, particularly in non-convex settings. Conventional optimization methods and insights from convex optimization do not generalize well to these scenarios, leading to inefficiencies and increased resource consumption. Our research aims to develop and validate new optimization techniques that are well suited to the unique challenges of distributed deep learning. We will focus on scalable federated learning and decentralized optimization methods, addressing critical issues such as drift correction, rules for parameter scaling, and the integration of advanced neural network architectures such as transformers. By benchmarking these techniques across various simulated environments and real-world applications, we aim to derive comprehensive guidelines to improve the efficiency and scalability of distributed deep learning. Our interdisciplinary team combines expertise in stochastic optimization, federated learning, and neural network training, ensuring a holistic approach to tackling these complex challenges. The outcomes of this project will make a significant contribution to the field of distributed machine learning by providing practical solutions for the efficient training of large-scale models, thereby helping to reduce the carbon footprint of deep learning.

Funding Body

Funding Body

Gefördert durch die Deutsche Forschungsgemeinschaft (DFG) – 553954458

Autonomous vehicles make steering decisions on the road independently based on AI-driven processing of various sensor data. Potential malicious attacks can lead to accidents due to incorrect maneuvers by autonomous vehicles and must therefore be systematically researched along with countermeasures when discussing the reliability of such systems.

This project analyzes the effects of manipulations on current sensors and sensor processing pipelines. Furthermore, secure platforms for sensor data processing will be designed and implemented, and their effectiveness in reliably defending against manipulation and compromise attempts will be demonstrated. The resulting design of a security-focused platform will serve as a reference for future research and product development in this area.

We approach the problem from three complementary research directions: defense against physical sensor manipulations, detection and prevention of manipulations in the sensor fusion pipeline, and trustworthy processing platforms for the automotive industry. Solutions for each topic will first be theoretically analyzed and then tested for effectiveness in simulations. In parallel with threat analysis and the development of countermeasures, we will develop and implement our practical demonstrator platform.

Funding Body

Funding Body

Bundesministerium für Digitales und Verkehr

The goal of this RTG is to develop novel neuroexplicit models that accurately solve tasks in natural language processing, computer vision, and action-decision making, and to investigate the theoretical and practical principles of designing effective neuroexplicit models. We will train three cohorts of PhD students in carrying out research on these models at the highest international level, producing graduates who are highly qualified for both the academic and the industrial job market. Over the past ten years, deep neural models have revolutionized computer science and set new standards for what artificial intelligence can achieve. They are in contrast to explicit models, which are designed to capture knowledge about a domain or a task with representations that can be understood and authored by human experts. Neuroexplicit models combine neural and explicit components, with the goal of inheriting the strengths of both model types. Explicit models include symbolic models, e.g. in computational linguistics and action planning, but also approaches to computer vision that model the physics of the world with differential equations. Thus neuroexplicit models encompass neurosymbolic models, which have recently been gaining in popularity across many areas of AI. Despite widespread and growing international interest, this RTG will be the first research center and, in particular, the first PhD training program for neuroexplicit methods in Europe. Neural models greatly outperform purely explicit models in terms of accuracy and scalability. At the same time, they have their own limitations, in particular with respect to generalization (learning abstractions which will transfer to related tasks), robustness to perturbations of the input, and interpretability. As the participating researchers of this RTG and others have shown, neuroexplicit models can help overcome these limitations. However, the design challenges of neuroexplicit models currently must be carefully addressed from scratch for each new task, and the fundamental design principles of neuroexplicit models are not yet well understood. This makes the development of novel neuroexplicit models cumbersome, requiring extensive experimentation to get the model right. In this RTG, we will work out design principles for effective neuroexplicit models through the interdisciplinary collaboration of PhD students from multiple fields of AI, and thus accelerate the development of efficient and accurate neuroexplicit models in the future.

Funding Body

Funding Body

Funded by the Deutsche Forschungsgemeinschaft (DFG, Ger-man Research Foundation) – Project number: 471607914

Partner

Partner

Universität des Saarlandes

Acute ischemic stroke remains a leading cause of disability and death, with current reperfusion therapies constrained by narrow time windows and variable efficacy. We propose an explainable U-Net-based model that integrates multimodal data — demographics, clinical scores, biomarkers, imaging— within a decentralized Swarm Learning infrastructure to stratify patients for mechanical thrombectomy based on 90-day outcome predictions. By establishing a scalable Swarm Learning network across centres of the German Stroke Registry, we will enable privacy-preserving usage of datasets while improving the generalizability of the model on recent heterogeneous real-world data. For the models’ explainability and trustworthy, we will compare feature‐importance across decentralized, local, and central models, use saliency maps to localize image-driven predictions, and identify underperforming subgroups to guide model specialization. Altogether, our framework delivers personalized risk assessment to optimize patient selection while providing a sustainable and scalable pipeline for AI-driven diagnostics and prognostics across stroke centres via our Swarm Learning network. 

Funding Body

Funding Body

Impuls- und Vernetzungsfonds der Helmholtz-Gemeinschaft

In recent years, novel AI coding assistants based on large language models (LLMs) have become ex-tremely popular in creating code. However, at least 50% of the effort of software development goes into understanding and maintaining existing software, tasks that still pose grand challenges for LLMs. This is because most maintenance tasks are related to the dynamic behavior of programs, which is insufficiently captured by learning from static code alone. Understanding how the individual pieces of a large program work together can take human experts months to years. 

As part of our ERC Advanced Grant "S3 - Semantics of Software Systems", we have developed innova-tive techniques to systematically test software systems, comprehensively exploring their behavior. This allows us to capture the dynamic features of inputs, outputs, and executions and train program-specific machine learning models that relate them to each other. The resulting model can then an-swer maintenance questions such as "What configuration input do I need to make this button green?" or "What input triggers this error message?" 

Our underlying research has already shown the principal feasibility of this approach, but has been limited to small examples. The question is whether these approaches scale to the multitude of fea-tures that occur during executing a complex program. This is the aim of this proof of concept pro-posal: To create a demonstrator of a Program-Specific Agent that is automatically trained from a 

nontrivial piece of software and that can answer maintenance-related questions with high accuracy. 

Training such an agent on even a single piece of software will require substantial resources, hence this grant proposal. However, any such trained agent will act as an expert on the software's behavior as long as it exists, promising substantial cost savings for the future of software development. 

Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or [name of the granting authority]. Neither the European Union nor the granting authority can be held responsible for them.

Funding Body

Funding Body

European Union - EU – Proof of Concept (ERC-2025-PoC)

Programs use strings to represent all kinds of textual data: names, credit card numbers, email addresses, URLs, bank accounts, color codes, and much more.

However, programming languages offer only limited support for verifying whether the contents of these strings actually meet expectations. This can lead not only to functional errors but also to frequent attacks such as script or SQL injections. In this proposal, we introduce string types—a means to express the valid values of strings using formal languages such as regular expressions and grammars. We develop methods to specify which sets of strings are acceptable as values and to dynamically and statically check whether the program is correct with respect to the specified string types. Since these are formal languages, string types also enable the generation of instances from these specifications.

This makes massive automated testing of string-processing functions with valid inputs possible, while string types, in turn, verify string outputs for lexical, syntactic, and semantic correctness. Finally, we introduce means to learn such specifications from code and its executions so that string types can be introduced easily. The consortium brings extensive experience in static analysis of parsing code, unit test and oracle generation, as well as language-based specification and testing. Their combined expertise will ensure the success of this project.

Funding Body

Funding Body

Gefördert durch die Deutsche Forschungsgemeinschaft (DFG) -535740946

Hepatitis D is by far the most severe form of chronic viral hepatitis frequently leading to liver failure, hepatocellular carcinoma and death. Hepatitis D is caused by coinfection of hepatitis B patients with the hepatitis D virus (HDV). Up to 20 Million individuals are infected with HDV worldwide including about 250.000 patients in the European Union. There is very limited knowledge on disease pathophysiology and host-virus interactions explaining the large interindividual variability in the course of hepatitis D. It is inparticular unknown why 20-50% are spontaneously able to control HDV replication, why the majority but not all patients progress to advanced stages of liver disease and why only some patients show off-treatment responses to antiviral treatment with either pegylated interferon alpha or the novel HBV/HDV entry inhibitor bulevirtide. As HDV is an orphan disease, no multicenter cohorts of HDV infected patients are available with appropriate biobanking. There is also no reliable animal model available allowing to study host responses. Thus, there is an urgent clinical, social and economic need to better understand individual factors determining the outcome of infection and to identify subjects benefitting from currently available treatments. Hepatitis D is a protype infection which could hugely benefit from a novel individualized infectious medicine approach. We here aim to perform an unbiased screening of a large multicenter cohort of well-defined HDV-infected patients followed by mechanistic studies to determine the functional role of distinct molecules. Identified specific parameters could have an immediate impact on the personalized surveillance strategies and antiviral treatment approaches. D-SOLVE aims to reduce disease burden, improve patient's quality of life and safe direct and indirect costs caused by HDV infection by combining exceptional clinical, immunological, bioinformatical and virological expertise from leading centers in Europe.

Funding Body

Funding Body

European Health and Digital Executive Agency (HADEA)

This project will develop the foundations of privacy-preserving AI safety verification, enabling the verification of AI safety guarantees without revealing any sensitive details about the underlying model or data. The technology will deliver mathematical assurances that AI systems adhere to safety specifications and uphold privacy of their models and data, crucial in applications where both safety and privacy are critical.

Funding Body

Funding Body

Advanced Research and Invention Agency (ARIA, UK)

Partner

Partner

University of Birmingham