cluster/grid

Cluster of servers (computer), cluster computing farm, grid
Higher Edu - Research dev card
Development from the higher education and research community
  • Creation or important update: 02/08/12
  • Minor correction: 02/08/12

Stratuslab : complete IaaS cloud distribution

This software was developed (or is under development) within the higher education and research community. Its stability can vary (see fields below) and its working state is not guaranteed.
  • Web site
  • System:
  • Current version: v2.0 - 25 June 2012
  • License(s): Other - Apache-2, AGPL
  • Status: stable release
  • Support: maintained, ongoing development
  • Designer(s): StratusLab Collaboration (CNRS, UCM, GRNET, SixSq, TID, et TCD)
  • Contact designer(s): support@stratuslab.eu
  • Laboratory, service: Universidad Complutense de Madrid (Madrid, Spain), GRNET (Athens, Greece), SixSq (Geneva, Switzerland), Telefónica I+D (Madrid, Spain), Trinity College Dublin (Dublin, Irland)

 

General software features

logo stratuslab
The distribution contains all of the necessary for deploying an Infrastructure-as-a-Service (IaaS) cloud: network, storage, and virtual machine management. Moreover, it provides innovative features like the Marketplace that facilitates sharing of virtual appliances, service management that allows deployment and autoscaling of multi-machine services, and support for multi-cloud scenarios. The distribution supports multiple operating systems (CentOS 6.2, Fedora16, and OpenSuSE 12.1) and is ideal for both public and private cloud deployments. The StratusLab client, written in Python, provides a simple command line interface to access to StratusLab cloud infrastructures from GNU/Linux, Mac OS X, and Windows machines.

Source code for the distribution can be found on GitHub.

Context in which the software is used

StratusLab is used at LAL to provide one of the StratusLab reference cloud infrastructures, a public cloud open to anyone for non-commercial use. The other StratusLab reference cloud is operated by GRNET in Greece. LAL also operates a second, private cloud infrastructure for deployment of laboratory services; existing services are gradually being migrated to this cloud infrastructure.

CNRS/IBCP operates a StratusLab cloud to support bioinformatics research and services. This public cloud infrastructure is available to users of the ReNaBi network. A portal, customized for bioinformatics users, facilitates use of the IBCP cloud and simplifies access to relevant virtual appliances and databases.

There are also commercial deployments of the StratusLab cloud distribution that support software engineering processes (such as deployment of ESA's SCOS-2000 platform) and scientific use (like the Atos Helix Nebula cloud infrastructure).

Publications related to the software

All of the related publications of the project are available from :

The chapter of the book "European Research Activities in Cloud Computing" contains a general description of the StratusLab project and its cloud distribution.

Higher Edu - Research dev card
Development from the higher education and research community
  • Creation or important update: 20/03/12
  • Minor correction: 20/03/12

HINTS : network on demand diagnostic tool

This software was developed (or is under development) within the higher education and research community. Its stability can vary (see fields below) and its working state is not guaranteed.

 

General software features

HINTS is a tool that provides quick network troubleshooting between two "probes", which are usually installed in remote geographical locations.
Although HINTS was primarily designed to diagnose potential network problems between two nodes of a grid, it is completely usable in a classical context, between two standard sites not involved in a grid infrastructure.

Historically, HINTS was born as part of the european project Enabling Grids for E-sciencE - EGEE, and continues today its evolution within the European Grid Infrastructure - EGI. HINTS is developed and maintained by CNRS, partner of France Grilles, the French National Grid Initiative and GARR, partner of the Italian Grid Initiative.

HINTS allows a user, from a centralized web interface, to initiate basic network tests between two specific agents (probes) previously deployed within local networks one want to qualify the link.

The qualification of the network between two probes is done through the following traditional system tools:

  • Ping
  • Traceroute,
  • Nmap,
  • DNS Resolution,
  • Iperf.

The main benefits of HINTS are:

  • A user may (subject to accreditation) test the quality of the network between its site and another site without involving an administrator of that other site,
  • All tests are symmetric (made in both directions between the selected probes),
  • Direct access to the probes (and thus access to the various diagnostic tools) is not possible, everything is done only via a centralized interface,
  • Fine and flexible user management (certification, delegation of rights),
  • The probes are based on those proposed by the well known PerfSONAR project.
Context in which the software is used

HINTS becomes useful when a user wants to diagnose the network between two sites without involving a network administrator and above all without disturbing the network administrator at the remote site.

If HINTS highlights some dysfunction between these two sites, network administrators will then not be disturbed for nothing.

If HINTS finds no problem, then the problems faced by users are likely not due to the network and is the reason to be searched elsewhere (application layers).

Today, HINTS is available in Scientific Linux 6.x RPM format.

Publications related to the software

  • Presentation at the EGI Technical forum, Lyon, September 2011
  • Poster for the EGI Community Forum, Munich, March 2012
Higher Edu - Research dev card
Development from the higher education and research community
  • Creation or important update: 03/01/12
  • Minor correction: 03/01/12

XtremWeb-HEP : middleware for distributed data processing

This software was developed (or is under development) within the higher education and research community. Its stability can vary (see fields below) and its working state is not guaranteed.
  • Web site
  • System:
  • Current version: 7.6.4 - 12/12/2011
  • License(s): GPL
  • Status: validated (according to PLUME)
  • Support: maintained, ongoing development
  • Designer(s): Oleg LODYGENSKY
  • Contact designer(s): xtremweb (at) lal.in2p3.fr
  • Laboratory, service:

 

General software features
  • ’XtremWeb-HEP’ is a middleware for Distributed Data Processing (grids) :
    –  It permits Administrators to :
        - manage various Users and Applications, by providing adequate access rights to them,
        - catalog various Data and Computing Resources :
           Â· PC farms managed by an IT department,
           Â· PC grids contributed by volunteer citizens,
    –  It permits Users to submit Jobs referencing these Applications,
    –  From Job descriptions, it dynamically deploys and executes these Applications on available Computing Resources, then it provides the results to authorized Users,
    –  It protects the Computing Resources running Mac OS X by starting the Application inside MAC OS X Sandbox,
    –  For the access to data, it permits the usage of HTTP, HTTPS, and any URI scheme whose driver is provided by the User.

Secured three tiers Architecture.  Scheduler and data repository managed by a software administrator on a server;  Client installed on the machine of each User (for ex. scientist);  Worker installed on the resource of each contributor.

  • Soon will come in production the version of XtremWeb-HEP additionally managing the submission of complete virtual machines for execution inside VirtualBox.

  • Interoperability with other grid middleware stacks :
    –  XtremWeb-HEP accepts X509 certificates and proxies for user management, in particular those of the DEGISCO international project.
    –  XtremWeb-HEP integrates a bridge permitting suitable XtremWeb-HEP jobs to be accepted by the gLite middleware in order to be executed by the EGI European infrastructure.
    –  On the other way, thanks to the 3G Bridge of the EDGI European project, the resources gathered by XtremWeb-HEP are available for the many users of the EGI infrastructure (gLite, ARC and Unicore middleware stacks).

  • Domain, Infrastructures, Documentation and Maintenance :
    –  In spite of its name, XtremWeb-HEP is used way beyond High Energy Physics :  Biology,  ADN Research,  Mathematics,  Physics of Solids,  Signal Processing.
    –  XtremWeb-HEP is powering at least 2 production grids (For each grid, look at the 'Statistics' page) :
        - http://www.xtremweb-hep.org/lal/xw_lal/
        - http://xw.lri.fr:4330/XWHEP
    –  XtremWeb-HEP has a complete up to date set of user manuals, presented at http://www.xtremweb-hep.org/spip.php?rubrique16
    –  XtremWeb-HEP is maintained by the software team presented at http://www.xtremweb-hep.org/spip.php?rubrique35 and is strongly supported by Institut des Grilles et du Cloud, INRIA, ENS Lyon, GRID5000

Context in which the software is used
  • Distributed Data Processing
  • Distributed Computing
  • Resource Sharing
  • Computing Grid (PC Grid)
  • Job Submission
Publications related to the software
  • Hybrid Distributed Computing Infrastructure Experiments in Grid5000 : Supporting QoS in Desktop Grids with Cloud Resources   http://users.lal.in2p3.fr/lodygens/gc/g5k.pdf
    G. Fedak, S. Delamare, O. Lodygensky.   Grid 5000 School, Reims, France - April 18-21, 2011

  • Extending the EGEE grid with XtremWeb-HEP Desktop Grid   http://users.lal.in2p3.fr/lodygens/gc/PCGrid2010.pdf
    H. He, G. Fedak, P. Kacsuk, Z. Farkas, Z. Balaton, O. Lodygensky, E. Urbah, G. Caillat, F. Aurajo, A. Emmen.   4th Workshop on Desktop Grids and Volunteer Computing Systems, Melbourne, Australia - May 17-20, 2010

  • EDGeS : Bridging EGEE to BOINC and XtremWeb   http://users.lal.in2p3.fr/lodygens/gc/EDGeS-Bridgi...
    E. Urbah, P. Kacsuk, Z. Farkas, G. Fedak, G. Kecskemeti, O. Lodygensky, A. Marosi, Z. Balaton, G. Caillat, G. Gombas, A. Kornafeld, J. Kovacs, H. He, and R. Lovas.   JoGC Journal of Grid Computing, Volume 7, Number 3, 2009.

Higher Edu - Research dev card
Development from the higher education and research community
  • Creation or important update: 25/10/10
  • Minor correction: 28/10/10

EDNA : framework for plugin-based applications for online data analysis

This software was developed (or is under development) within the higher education and research community. Its stability can vary (see fields below) and its working state is not guaranteed.
  • Web site
  • System:
  • Current version: 1.0 (Application MXv1 june 2009) - revision SVN 1930
  • License(s): GPL -

    EDNA kernel and many plugins are LGPLv3+

    EDNA applications (like MXv1) and some plugins are GPLv3+

  • Status: stable release, internal use, under development
  • Support: maintained, ongoing development
  • Designer(s):

    Executive Committee Chair: Andrew Leslie (MRC-LMB)

    Project Manager: Olof Svensson (ESRF)

    Project Coordinator: Alun Ashton (Diamond)

    Currently around 10 developers but 40 people have been involved in the project.

  • Contact designer(s): edna-support@esrf.fr
  • Laboratory, service: DLS, MRC-LMB, CCP4, Bessy (HZB), Max Lab. NSLS, SLS-PSI, Univ Sydney, Univ York, Global Phasing

 

General software features

Platform
The EDNA kernel runs on all platforms which can provide a Python interpreter. The MXv1 application runs on all Unix/Linux platforms that have bash, Python (version 2.5 and later) and which can run the programs MOSFLM, LABELIT, RADDOSE and BEST. The EDNA framework and MXv1 application are installed and regularly used at the Diamond Light Source (Didcot), EMBL Hamburg, ESRF Grenoble and NSLS Brookhaven National Laboratory (New York).

EDNA-based applications are designed and developed with the aim of being easily configurable, extensible and smoothly maintainable. This has been made possible thanks to the technical facilities that the EDNA framework provides, including configuration facilities, a library of re-usable components, datamodel-driven code generation machinery and a testing framework.

Modularity
The components are organized in a logical class hierarchy that makes it straightforward to develop new functional plugins by deriving them from the appropriate parent. Two families of plugins have been designed: the first branch contains the execution plugins (EDPluginExec classes) that are responsible for the execution of a particular action (e.g. execution of third-party software); the second branch contains the control plugins (EDPluginControl classes) that are responsible for the data flow (propagation of the data), the workflow (sequential or parallel execution of appropriate execution plugins) and the error-tracking mechanism (propagation of the errors).

Datamodel
The EDNA kernel provides a data model tool kit that allows the construction of elaborate data models needed by advanced applications. In addition, it allows the design of unitary data models that can be unit-tested, so that a plugin can be launched and tested independently of any application context. This tool kit consists of generic low-level class definitions including general types (XSDataString, XSDataFloat etc.) and X-ray experiment classes which can be re-used when designing specific components data models. It is available in several standard data formats including XMI (XML metadata interchange) and XSD (XML schema definition) in order to facilitate importing into UML (unified modeling language) data modeling tools and/or XSD files. The framework also provides (external) code generation machinery generateDS that allows automatic code generation from UML diagrams to Python code via XSD format.

Testing framework
A testing framework has been developed and integrated with the kernel in order to test the applications and the components easily and efficiently. To ensure the reliability and robustness of the components, the testing framework provides all the necessary utilities to check a class (EDTestCase) and a plugin (EDTestCasePlugin) either in a unitary manner (EDTestCasePluginUnit) or by testing its execution in an application context (EDTestCasePluginExecute). These families of tests can be automatically launched via test suites. An automatic analysis of the test results is performed by comparing the obtained result with the expected one (assert mechanism), so that a successful test proves that a result conforms to the expectation. This allows a high degree of confidence when implementing new components or re-implementing features of existing components.

Configuration facilities
Each plugin is configured (path of the controlled executable, batch queuing system configuration, ...) by the mean of XML files. The configuration is selected according to the EDNA_SITE environment variable.

Context in which the software is used

EDNA is used to create online data analysis applications on synchrotron beam-lines (but not only), among them:

  • Mxv1 application for fast characterization of protein crystals; in regular use at the ESRF, the DLS and the NSLS.
  • MXv2 application is in preparation to take advantage of kappa goniostats.
  • Dimple application has been written by CCP4 for doing molecular replacement and ligand location in proteins.
  • Diffraction computed tomography application with azimuthal integration and online sinogram generation with dynamic region of interest for Nano-Analysis Beamlines.
  • Saxs pre-processing (azimuthal integration and averaging) and downstream processing for BioSaxs Beamline.
  • A Tutorial for Raw digital camera development that explains how to take advantage of the parallel capabilities of EDNA.
Publications related to the software

EDNA: a framework for plugin-based applications applied to X-ray experiment online data analysis
M.-F. Incardona, G. P. Bourenkov, K. Levik, R. A. Pieritz, A. N. Popov and O. Svensson
J. Synchrotron Rad. (2009). 16, 872-879 [ doi:10.1107/S0909049509036681 ]

Abstract: EDNA is a framework for developing plugin-based applications especially for online data analysis in the X-ray experiments field. This article describes the features provided by the EDNA framework to ease the development of extensible scientific applications. This framework includes a plugins class hierarchy, configuration and application facilities, a mechanism to generate data classes and a testing framework. These utilities allow rapid development and integration in which robustness and quality play a fundamental role. A first prototype, designed for macromolecular crystallography experiments and tested at several synchrotrons, is presented.

Higher Edu - Research dev card
Development from the higher education and research community
  • Creation or important update: 16/08/10
  • Minor correction: 01/06/11

CiGri : lightweight computing grid

This software was developed (or is under development) within the higher education and research community. Its stability can vary (see fields below) and its working state is not guaranteed.
  • Web site
  • System:
  • Current version: 1.3 - Aout 2009
  • License(s): GPL - v2
  • Status: validated (according to PLUME), stable release, under development
  • Support: maintained, ongoing development
  • Designer(s): Bruno Bzeznik, Nicolas Capit, Olivier Richard, Elton Nicoletti Mathias, Yiannis Georgiou, and various contributors (internships, google summer of code)
  • Contact designer(s): Bruno.Bzeznik@imag.fr
  • Laboratory, service: CIMENT (University of Grenoble Computing center)

 

General software features

The CiGri software allows to set up a grid center to exploit a pre-existing set of super-computers. It is specialised on the management of "bag-of-tasks" jobs. It gathers the unused computing resources from an intranet infrastructure and makes it available for large set of tasks.

More information (in French) at fiche logiciel Fiche Plume.

Context in which the software is used

CiGri software is used at the computing center at Joseph Fourier University of Grenoble (CIMENT) since 2002.

Publications related to the software
  • Yiannis Georgiou, Olivier Richard, et Nicolas Capit.
    Evaluations of the lightweight grid cigri upon the grid5000 platform. In E-SCIENCE '07: Proceedings of the Third IEEE International Conference on e-Science and Grid Computing, pages 279-286, Washington, DC, USA, 2007. IEEE Computer Society.
  • Yiannis Georgiou, Nicolas Capit, Bruno Bzeznik, et Olivier Richard.
    Simple, fault tolerant, lightweight grid computing approach for bag-of-tasks applications. 3rd EGEE User Forum, 2008.
    http://indico.cern.ch/contributionDisplay.py?contr....
  • Yvan Calas, Nicolas Capit, et Estelle Gabarron.
    Cigri : Expériences autour de l’exploitation d’une grille légère. JRES, 2005.
    http://2005.jres.org/paper/90.pdf.
  • F. Dupros, F. Boulahya, J. Vairon, P. Lombard, N. Capit, et J-F. Méhaut.
    Iggi, a computing framework for large scale parametric simulations: Application to uncertainty analysis with toughreact. TOUGH Symposium, 2006.
    http://esd.lbl.gov/TOUGHsymposium/pdf/Dupros_IGGI.pdf.
  • J. Aoun, V. Breton, L. Desbat, B. Bzeznik, M. Leabadand, et J. Dimastromatteo.
    Validation of the Small Animal Biospace Gamma Imager Model Using GATE Monte Carlo Simulations on the Grid. In J. Montagnat S. D. Olabarriaga, D. Lingrand, editor, Proceedings of MICCAI-Grid Workshop Medical imaging on grids: achievements and perspectives, MICCAI-Grid Workshop, New York États-Unis d'Amérique, 2008.
Higher Edu - Research dev card
Development from the higher education and research community
  • Creation or important update: 05/04/10
  • Minor correction: 04/04/13

Paraloop : distributing parallel jobs

This software was developed (or is under development) within the higher education and research community. Its stability can vary (see fields below) and its working state is not guaranteed.
  • Web site
  • System:
  • Current version: 1.3 - 2008 September
  • License(s): CeCILL
  • Status: stable release
  • Support: maintained, ongoing development
  • Designer(s): Emmanuel COURCELLE
  • Contact designer(s): emmanuel.courcelle at toulouse.inra.fr
  • Laboratory, service: Service Bioinformatique

 

General software features

Paraloop distributes your jobs on several processors of a machine, independently of its architecture: it may be a single SMP computer with shared memory, as well as a cluster, or even a network of workstations.

Paraloop is best suited for the use cases when we have a high number of independant tasks to execute, as is often the case in the data treatment pipelines found in bioinformatics projects.

Paraloop is a tool for programmers, who are able to easily distribute their jobs, while using the same script whatever the machine they run on. It is a perl object program: data treatment is wrapped inside an object (called a "plugin"), the code responsible for the machine interaction is wrapped inside another object (called "scheduler"). It is thus relatively easy to adapt paraloop to a new architecture: it just means writing a new scheduler (in fact, just a few methods). The same is true for plugins: they are able to read and treat data, using some particular format.

A few plugins are delivered with paraloop: some of them are specific to the bioinformatics field (one of them is useful to execute BLAST in parallel, for instance), while others are completely generic (reading a text file, ...). However, writing plugins dedicated for other thematic fields would be a quite useful task.

When used in a queue context, with a limited cpu time per job, it is possible to configure paraloop so that the current job is interrupted before being killed by the system; the job is resubmitted to the queue just before the interruption, so that it will be resumed as soon as permitted by the system.

Besides, paraloop includes a command to print the progress report of each job.

Finally, a "load balancing mode" is available: it may be used to insure that all the jobs take approximately the same time to execute.

Context in which the software is used

We currently use paraloop for our bioinformatics computations, whether on SMP servers or computer clusters. Besides, paraloop is integrated to our bioinformatics projects (eugene, LeARN, Narcisse, ...)

Publications related to the software

Paraloop was described in a poster session a J-RES 2005 http://2005.jres.org/resume/poster/138.pdf and at JOBIM, in 2005 too http://pbil.univ-lyon1.fr/events/jobim2005/proceedings/P64Courcelle.pdf

Paraloop is currently hosted on the SourceSup forge: http://sourcesup.cru.fr/projects/paraloop

Higher Edu - Research dev card
Development from the higher education and research community
  • Creation or important update: 03/07/09
  • Minor correction: 22/03/10

SOFA : interactive mechanical simulation library in C++

This software was developed (or is under development) within the higher education and research community. Its stability can vary (see fields below) and its working state is not guaranteed.
  • Web site
  • System:
  • License(s): LGPL
  • Status: beta release
  • Support: maintained, ongoing development
  • Designer(s): INRIA, LJK
  • Contact designer(s):
  • Laboratory, service:

 

General software features

SOFA is an open-source, modular mechanical simulation library developed in C++.
It includes geometries such as polygonal and volumetric meshes, internal forces such as FEM or springs, ODE solvers such as Explicit or implicit Euler, collision detection methods, OpenGL viewing, and many other features.
GPU implementations are available for some force fields, such as springs and tetrahedral FEM. The automatic deployment on multi-core architectures is possible.
Its design allows contributors to focus on their domain of competence while reusing other work.

More detail on http://www.sofa-framework.org

Context in which the software is used

Sofa has been applied to interactive medical simulations, CAD assembly, as well as off-line character animation or civil engineering simulations.

Publications related to the software
Syndicate content