Federated Systems and Data
Head of the division Federated Systems and Data
Phone: +49 2461 61-2433
Fax: +49 2461 61-6656
The Federated Systems and Data (FSD) division develops and operates services that enable easy and secure access to distributed resources like HPC systems and data. Guiding principles are to preserve site autonomy, to drive and implement open standards, and to simplify usage and administration of the services.
FSD is the core development partner of the UNICORE Grid middleware. Division members work in the Research Data Alliance, the Open Grid Forum, and OASIS in order to create open, standardised interfaces for distributed resources and data. The division runs federated infrastructure services for several German and European e-infrastructures that allow users to access JSC computing and data resources. Users, administrators and application developers are supported in enabling applications and resources for federated e-infrastructures. The research and development activities cover data management, data analysis, resource management, virtualisation, security, workflows, and scheduling in all fields of distributed systems.
Research and development on distributed software and services (such as UNICORE) includes the following topics
- Security: single sign-on, federated identity management, resource sharing
- Web portals
- Access to cloud resources such as OpenStack, S3 and CDMI
- User defined application environments using virtual machines and application containers
- Data and metadata management
- Scientific workflows using federated, heterogeneous resources
- High-performance, firewall-friendly file transfer (UFTP)
- Alternative programming models such as map/reduce, stream processing or tuple spaces
- RESTful APIs
- Continuous evolution and maintenance of UNICORE
The adoption of standards (OASIS, W3C, OGF), re-use of existing open-source software and effective, lightweight quality assurance measures (strong focus on unit testing, code coverage, and continuous integration) are used in order to produce high-quality and maintainable open source software.
These activities are pursued in collaboration with partner institutes and in the context of international projects such as PRACE, EUDAT, EGI, HBP and AARC.
This group pursues the integration of various technologies developed or available within FSD and supports user communities to make the best use of the services at hand.
- Application Integration: Develop integration solutions specifically tailored for user communities and their use cases. This includes application specific user interfaces as well as consultancy regarding adequate solutions for the use cases in question. Solutions may be comprised of traditional batch-processing as well as cloud-based approaches including storage.
- Cloud Application Enablement: A number of user communities approach us with already defined solutions to their problems, most often making use of virtual resources. We help them optimize their solutions to enable them to be deployed at scale. Generic patterns of cloud application architecture are derived and contribute to our general set of solutions.
- Cloud Services: Operation of cloud compute and storage services as part of the EGI federated cloud. Our interest is in defining a profile for heterogeneous cloud federations based on open standards, including, but not limited to Open Cloud Computing Interface (OCCI), Cloud Data Management Interface (CDMI), and SAML based AAI federations.
Operations and Support
UNICORE as well as other services are operated in a variety of eInfrastructures and support is given to users as well as administrators.
- Operation of UNICORE Services: The UNICORE services are installed to provide access to JSC computing and data resources for users of Forschungszentrum Jülich, the research infrastructures PRACE and EGI as well as infrastructures for research projects.
- Support: Assistance is given to users and administrators of the UNICORE services. For example support is provided for application integration and workflow modelling. Training courses are offered to interested parties on usage and administration of UNICORE.
- Data Management: Initiated by the research infrastructure project EUDAT (European Collaborative Data Infrastructure) safe data replication between JSC resources and other infrastructure resources has been realized and is maintained.
High Productivity Data Processing Research Group
The research group investigates solutions to overcome problems and challenges arising from the ever increasing amount of ‘big data’:
- Investigate Generic Data Methods: Explore user-centered programming models and data analytics methods that overcome limitations when processing and analyzing large quantities of data (e.g. in-memory database, data privacy methods, query processing)
- Learn from Data: Apply and extend (serial or parallel) machine learning and data mining techniques in order to increase the productivity of data analysis processes employed by user communities from diverse scientific disciplines (e.g. classification, clustering, regression, automatic detection of trends, outliers, patterns, anomalies, etc.)
- Enable Smart Data Analytics Applications: Examine suitable applications in the light of general data analytics techniques that enable high productivity data processing and analysis (e.g. explore guidelines of a useful mix of resources, algorithms, and tools)
The research topics worked on by the Federated Systems and Data devision are presented under the Distrubuted Computing research section.