- Monitoring and Measurement in the Next Generation Technologies
Monitoring and Measurement in the Next Generation Technologies (MOMENT) - is a project aimed at integrating different platforms for network monitoring and measurement to develop a common and open pan-European infrastructure. The system will include both passive and active monitoring and measurement techniques via a common web services interface and ontology that allows semantic queries.http://www.fp7-moment.eu/
Concept and objectives
It is clear that measurement and monitoring will be a key element in the management of future network infrastructures, both at the level of network equipment and also in the overall distributed control of the large scale Internet infrastructure. In the future, interoperability of monitoring and data collection capabilities can provide the support basis of seamless end-to-end network and service composition and operation across multiple operators and business domains. During the last five years important research initiatives emerged worldwide to tackle problems of Internet monitoring and data collection. In FP6 the EU ICT programme started several successful Internet measurement and monitoring efforts to boost European leadership within this emerging area. These projects already passed the proof of the concept stage and now key applications in future network management can be developed on their basis.
This project is aimed at integrating existing measurement and monitoring infrastructures towards a common and open pan European platform. This will be achieved through harmonisation of individual components, definition of common data format, development of unified interface, to provide the flexibility to the design of future Internet applications. On the other hand, the project will allow semantic representation and retrieval of measurement and monitoring information. Additionally the project will develop and demonstrate a set of tools and applications for next generation networks taking advantage from the integrated approach.
Background and Concept
Internet researchers face many daunting challenges, including keeping up with the conditions of ever changing operational environments, privacy concerns, legal complications, and resource access. One of the most fundamental problems remains access to current operational data on Internet infrastructure. For many projects the relevant datasets simply do not exist, and researchers must go through a laborious process of securing permission and deploying measurement infrastructure before they can begin to study a problem. For others, the necessary data may exist and even be available. Unfortunately, if word-of-mouth has insufficiently propagated the information about the data ownership and access procedures, researchers may waste time and effort creating a new dataset, use a dataset inappropriate for a given research problem, or possibly even abandon the research.
In addition, the dearth of centralized knowledge about the few datasets that are known to exist in the community leads researchers to use these datasets well past their window of representativeness. Correspondingly, lack of awareness of datasets limits longitudinal study ofnetwork conditions, since comparable datasets that span months or years are difficult to find.While the resource, legal, and privacy concerns limiting new Internet data collection efforts remain largely intractable, significant research could be promoted through more widespread use of existing data. To that end, CAIDA began developing an Internet Measurement Data Catalog — an index of existing datasets possibly available for research.
Within Europe, the main ingredients of the integrated efforts already made a significant contribution to integrate and validate state-of-the-art technology that is essential for preparing the future upgrades in the infrastructure deployed across Europe: LOBSTER (http://www.ist-lobster.org) is a pilot European infrastructure for accurate Internet traffic monitoring. Based on passive monitoring, and capitalising on previous experience, the LOBSTER infrastructure is unique in Europe and one of only three similar infrastructures that exist in the world today. DIMES (http://www.netdimes.org) is a distributed scientific research project, aimed to study the structure and topology of the Internet, with the help of a volunteer community (similar in spirit to projects such as SETI@Home) . The about 12000 DIMES agents worldwide perform Internet measurements from the volunteer sites. Recently a measurement planner has been added which enables a web interface to coordinated global measurements of bandwidth and delay. ETOMIC (http://www.etomic.org) is a measurement infrastructure in Europe that is able to carry out high temporal resolution (~10 nano second), globally synchronised, active measurements between the measurement boxes. It provides a high resolution, spatially extended dynamic picture of fast changes in the network traffic and opens up the possibility of network tomography, where traffic far from the measurement devices can be reconstructed and monitored. Both DIMES and ETOMIC were developed within the IST FET Integrated Project Evergrow. MOME (http://www.ist-mome.org) is a coordination action in IST that offered a platform for knowledge and tool exchange and for coordinating activities in the field of IP monitoring and measurement between IST projects and other European partners. The platform provides information on the interoperability of monitoring and measurement tools, as well as measurement data in a common format. Motivated by the success of the MOME database, a similar initiative started at its main US competitor (http://www.caida.org) recently.
The demand for network measurement and monitoring
The Future Internet must provide capabilities for QoS guarantees – usually in the form of bandwidth guarantees and sometimes for bounded message latency – that let distributed resources communicate at high speeds during critical times and in which operating experiments are connected to large-scale computing and data, to do real-time remote control. New kinds of data archive systems must have access servers that allow for reliable, high-speed, wide-area network data transfer. Storage, computing, and network resources must have support for the detailed monitoring that is essential for fault detection and recovery in widely distributed systems. Such services must be developed, installed, and integrated into the operational environments of the individual systems.
The network services requirement evolution is following a new paradigm of storage, computer cache and computing functionalities in delivering the new high-speed services.The most important new service the network should provide is the one that indicates how much bandwidth is available at a given point in time between any specified set of end points. With this information, applications and middleware can have the ability to adapt to current and future network conditions, even in the absence of any bandwidth guarantees. Ideally, this network service should provide both end-to-end and hop-by-hop information and should include information on network capacity, available bandwidth, delay, loss, and jitter. A mechanism for layer 2 network topology discovery also would be useful for network engineers to better understand and debug network problems and for middleware services to efficiently utilize the network.
Monitoring and measurement needs
Simply scaling up today’s approaches will not suffice to cope with new requirements. Although some monitoring can be done passively, other information can be collected only by using active probes. A new kind of scalable monitoring and measurement infrastructure is needed to avoid too much measurement traffic. The best solution would be to look up measurement data and measurement services in a distributed database, similar to the way that hostnames are resolved using the domain name server. There are many open research issues in designing such a monitoring system. One of the hardest problems is to separate network issues from host and application issues. It is also difficult to separate physical layer issues from protocol layer issues in the network. Archiving measurement data is also important. It allows one to compare current and previous performance and to determine what has changed. To make this network measurement data more accessible, work is also required to enhance SNMP. Shortcomings of the SNMP include the lack of privacy, authentication, and access control, which limit the protocol’s usefulness across domain boundaries. It also falls short in describing the available measurement services, which demand a semantic representation.
Project objectives
Future generation networks are heterogeneous as never before. Constantly emerging devices and applications result into enhanced resourses, topology and QoS requirements, as well as they arise a set of issues concerned with interoperability, cooperation and communication. Network structure becomes more and more complicated thus requiring thorough and comprehensive measurement and monitoringReaching optimising control, management and flexibility of the future network infrastructures requires actions to retrieve a good understanding of network and application behaviour. In previous EU-funded FP6 projects like EVERGROW and LOBSTER, large efforts have been taken to successfully develop and deploy measurement infrastructures (e.g. ETOMIC, DIMES) to gain insight into the operational network, without being intrusive from the perspective of commercial ISPs. Meta-repositories like MOME were dedicated to keep an overview on available measurement data, tools, infrastructures and projects. Each of the mentioned existing solutions for network monitoring provides single-side view of the network, thus, building of overall picture of network performance is rather complicated. Applying separately several techniques doesn’t seem to be a perfect solution.This stipulates the necessity in the single common architecture that would comprise the advantages of previous solutions and provide the comprehensive picture of network structure and functioning.The innovation solution should be based on the current techniques to ensure the compliance with the existing network infrastructure. The achieved advances are integrated within the MOMENT project towards a common and open, pan-European platform by confederating participants from various FP5, FP6 and other measurement-related projects. The main objective of the MOMENT project is to design and implement a mediator architecture offering a unified interface for measurement services, able to use all data and functionalities from the existing measurement infrastructures. Main innovation of the project is the use of a measurement-specific ontology, allowing semantic representation and retrieval of measurement and monitoring information, as well as providing the flexibility of a service oriented architecture for future Internet applications. Mediator basis will provide interoperability of MOMENT platform with the existing techniques and at the same time the platform will be open for the new methods to ensure that the proposed solutions for the network monitoring can be integrated to provide more enhanced functionality of the platform.To validate the benefits of the integrated approach, the project will develop and demonstrate a set of tools and applications presenting the added value of combining measurement data collected from different infrastructuresDue to the flexible design, the achievements of the project will be open to future network architectures, rather than limited to the current Internet protocols. By liaison with international activities like CAIDA, the project ensures to have a major impact in the contribution to standardisation bodies, as well as to the overall measurement community in the Internet.The MOMENT project is aimed at developing a common and open pan-European infrastructure to integrate different platforms for network monitoring and measurement. The system will include both passive and active monitoring and measurement techniques via a common web services interface and ontology that allows semantic queries. It will continuously monitor the macroscopic status of the network and individual domains by leveraging network tomography techniques. It will be able to collect information about the temporal variation of reachability and synthetic QoS indicators. It will offer a pan-European platform for detecting macroscopically relevant events like outages, attacks, world-scale infections, and large anomalies (e.g. BGP storms). The system will be non-intrusive from the perspective of commercial ISPs: monitoring probes will be installed outside the commercial network domains. The system will protect the privacy of end-users: the data obtained by passive monitoring will be made anonymous in a certifiable way and middleware solutions will be deployed inside the system to properly control and manage access to data subject to privacy concerns. Not only the unified web services interface, measurement results and the anonymous traces will be publicly accessible, but the system will also implement a role-based access control middleware devised to flexibly provide and manage access and/or processing permissions on either raw (i.e. not anonymised at all) or on data traces with reduced protection. This will allow duly authenticated and authorised users and monitoring applications to perform more advanced data mining, queries, and correlation tasks without being impaired by the data protection mechanisms set forth. At the same time, the semantic control exerted by the middleware will guarantee that disclosure - if any - of sensible information will occur only to the minimal extent required to operate the monitoring application, and only to specific, well identified, and duly authorised subjects (users or applications). Research groups will also be able to use the infrastructure, both the passive and active components, in order to perform their own experiments. Technology turns on to a new dimension – requiring new means for management of innovation. MOMENT is a tool, which facilitates the task of network management, while providing more usability and transparency of network processes and infrastructure.
Progress beyond the state-of-the-art
Network monitoring and measurement is of high interest since the IP networks have started to grow into the converged communication network, to help optimise the network design and operation. There are number of research projects financed by the European Union, National Science Foundation (NSF in USA) and by Internet-2 with number of regional projects defined. In this section we highlight some of the work already done, which will be used as background information in MOMENT project. It is worth noticing that MOMENT builds on top the current measurement infrastructures, by providing a unified access to data and services. On the other hand, several measurement infrastructures which were funded by the European Sixth Framework Programme have been developed by MOMENT partners. The aim of the project is to provide a middleware or mediation engine that serves not only to provide a common interface to those applications, but also to provide taxonomy of such monitoring services, and semantic-based querying capabilities. As a result, a middleware will be implemented which basically offers a mediation engine to those applications requiring monitoring and measurement services. A diagram of the proposed model is presented in Figure 2, and a more detailed representation of the mediation engine is depicted in Figure 3. The platform will provide a semantic mediation layer where monitoring services can advertise themselves, and monitoring data consumers can submit their requests and queries. The mediating framework will include a description model with which monitoring service providers can describe the supplied measurements in a detailed way. Consumers can similarly define their requirements, in response to which the mediator retrieves a ranked list of services that match the client needs. The mediation layer will furthermore include a description model for the consumer access roles and permissions, and will implement the run-time support procedures to allow and control processing of the data with reduced level of protection (e.g. no anonymisation).The applications requiring monitoring services will communicate with the mediator through the query interface. Such interface will be performed according to the SOA paradigm, which is a commonly accepted architecture for the deployment of web services. On the other hand, the subscription interface serves for the monitoring services to subscribe. In this project, we will integrate a number of well-known monitoring tools, which have been previously developed within EU projects (LOBSTER, DIMES, ETOMIC, etc.). However, our architecture is not limited to those monitoring platforms and can be extended to any other platform through the subscription interface. Furthermore, there is a configuration interface that allows remote management tasks. The subscription model will aim to overcome the limitations of current standards for web service registry and subscription such as UDDI, by means of rich and flexible metadata specific to the network monitoring domain, enabling semantic-aware query capabilities beyond mere service lookup.
Within the mediator architecture, there is a wrapper in charge of collecting the monitoring services architecture and convert it to a format understood by the inference engine. Note that the wrapper not only collects raw data from the applications but also the monitoring service description itself. Then, an ontology is designed that provides a catalogue of monitoring services and data. Such ontology constitutes the cornerstone to implement a database which can be addressed by content. Ontology-based descriptions will draw from existing semantic-based knowledge technologies and standards, such as the OWL language from the W3C for ontology definition.
The advantages of using ontologies as a supporting technology for the database are the improved scalability, interoperability, and the support of semantic queries. Such ontology allows, along with the adequate inference engine, to produce automated replies to complex queries for monitoring services. It also allows defining a database structure that can be indexed by semantic queries. It will be assessed whether a full inference engine (such as Racer or Pellet), or simpler inferencing functionalities such as the ones integrated in public-domain ontology APIs (e.g. Jena or Sesame), or even custom-programmed ontology graph processing features, are most appropriate for the needs and particularities of the envisioned mediation system. Complementarily, semantic-based query execution will rely on the available standards, such as SPARQL or RDQL, and the corresponding query engines provided by the ontology access APIs. Note that the above architecture does not imply that the control interface of the different monitoring tools has to be integrated in the mediation engine, even though it does not preclude that some tools can be controlled from the engine. As a use case example, let us consider the BART tool, which can be installed on a general-purpose PC. Our mediation service provides the means for the tool to register in the database. Then, applications can query the engine in order to discover whether a certain Internet link is monitored by BART for available bandwidth measurements. In a way, the engine provides all the monitoring features that can be exploited when analysing a certain network segment.
On the other hand, the engine also provides the data that is periodically being provided by the measurement infrastructures. Such data is provides with an XML wrapper to enable applications to understand and use data coming from different formats.
Therefore, the MOMENT mediation engine not only provides universal access to the available measurement and monitoring tools, but also facilitates working with collected data.
The proposed approach can be compared to the semantic Web services vision. Recent research and standardisation efforts in that area have addressed the service modelling and discovery problems from the broadest, domain-independent perspective. In order to progress beyond current achievements, we believe further specialisation is needed at this point, as is envisioned here. The semantic models to be researched and developed here will be thus defined from a domain-specific perspective, specifically oriented to the network monitoring domain.
MOMENT will advance beyond the state-of-the-art by covering the full portfolio of network measurements by integrating existing methodologies, services and tools performing active and passive on both, traffic and topology level, as depicted in.
MOME
The MOME Database supports access to measurement traces and tools published by different projects. The MOME database consists of a database backend (based on MySQL), a front-end web GUI (based on Apache/PHP), an analysis engine for trace analysis, and a local storage for selected traces. The MOME database is located at: http://www.ist-mome.org/database
The MOME database knows three kinds of actors:
* Unregistered users: able to view datasets
* Registered users: able to add, analyse and delete their datasets
* Administrators: full access to all functions provided by the GUI.Currently, the following trace types are considered, all having a common structure and some trace type specific attributes and analysis results:
* Packet trace
* Flow trace
* QoS measurement
* Routing trace
* HTTP trace
* Web based data repositories can also be linked within MOMELOBSTER
The LOBSTER project has built an advanced pilot European Internet traffic monitoring infrastructure based on passive network monitoring sensors. LOBSTER has also developed novel performance and security monitoring applications, which have been enabled by the availability of the passive network monitoring infrastructure, and has realised the appropriate data anonymisation tools for prohibiting unauthorised access or tampering of the original traffic data.Currently, more than 25 LOBSTER sensors have been deployed across Europe by several organisations. Using the monitoring applications developed within the project, researchers and administrators are able to monitor the Internet traffic for gaining a better understanding of its performance, as well as to spot security incidents. LOBSTER is based on passive network traffic monitoring. Instead of collecting flow-level traffic summaries or actively probing the network, passive network monitoring records all IP packets (both headers and payloads) that flow through the monitored link. This enables passive monitoring methods to record complete information about the actual traffic of the network, which allows for tackling monitoring problems more accurately compared to methods based on flow-level statistics or active monitoring.The passive monitoring applications running on the sensors have been developed on top of MAPI (Monitoring Application Programming Interface), an expressive programming interface for building network monitoring applications, which has been developed in the context of the SCAMPI and LOBSTER projects. MAPI enables application programmers to express complex monitoring needs, choose only the amount of information they are interested in, and therefore balance the monitoring overhead with the amount of the received information. Furthermore, MAPI gives the ability for building remote and distributed passive network monitoring applications that can receive monitoring data from multiple remote monitoring sensors.The LOBSTER sensors operated by the various organisations monitor the network traffic using different measurement applications. All applications have been developed within the LOBSTER project using MAPI, according to the needs of each organisation. Some of the developed applications are: APPMON http://lobster.ics.forth.gr/~appmon (Appmon is an application for Accurate Per-Application Network Traffic Classification), Stager http://stager.uninett.no/ (Stager is a system for aggregating and presenting network statistics ) and ABW https://perfmon.cesnet.cz/abw-intro.html (ABW is an application written on top of LOBSTER DiMAPI (Distributed Monitoring Application Interface) and tracklib library).
ETOMIC
The European Traffic Observatory Measurement InfrastruCture (ETOMIC) was created in 2004-05 within the EVERGROW Integrated Project launched by the Future and Emergent Technologies Programme of the European Union VI framework. UPNA, UAM, and CB, who are also partners in this proposal, have developed the ETOMIC platform. Its goal is to provide an open access, public testbed for researchers investigating the Internet with active measurement methods, to serve as a Virtual Observatory active measurement data on the European part of the Internet. The developed measurement nodes are fully reconfigurable, extremely accurate (nanoseconds) and GPS-synchronised, which properties make the ETOMIC as a measurement infrastructure with unique capabilities. The ETOMIC infrastructure was awarded by the Best Testbed Award on the TridentCom 2005 Conference, and it was opened and introduced to the community on the INFOCOM 2006 Conference.
Currently 18 active probing nodes are deployed Europe wide. The location of the measurement nodes are illustrated in Figure 6. The measurement nodes are hosted by European research groups collaborating in the EVERGROW project and in Eurolabs testbed (deployed by Telscom).
DIMES
DIMES (http://www.netDimes.org) is a subproject of the EVERGROW Integrated Project (http://www.evergrow.org) in the EU Information Society Technologies, Future and Emerging Technologies programme. DIMES is studying the structure and topology of the Internet to obtain a very accurate map and annotate it with delay, loss, and in the future available bandwidth and link capacity.The DIMES project is based on measurement by software agents that are downloaded by volunteers and installed on their privately owned machines. Once installed at user premises the agent operates at a very low rate so as to have minimal impact on the machine performance and on its network connection. DIMES follows the Internet dynamics and growth, the project intends to explore possible relationships between the data gathered on the Internet’s growth with geographical and socio-economic data, in particular for fast developing countries, to see if they can provide a measure of economic development and societal openness. The project is committed to openness and thus publishes period maps at several aggregation levels on the web. It also includes web interfaces for running remote coordinated measurements, which is used by researches around the world.As of April 2007, over 12500 agents have been installed by over 5500 users residing in about 95 nations, and in several hundreds of ASes. The project has collected over 2.2 billion measurements. These figures make DIMES the largest Internet topology measurement project in the world.BART
BART is a method developed by SICS and EAB for measuring available bandwidth in real time. Monitoring available bandwidth for an end-to-end network path would theoretically be possible without active probing, by having access to management data from all the network nodes in the path. However, network equipment owners do not usually make such data available. Measurement is only feasible by actively probing the network path, in order to determine at which probe rate the path shows signs of congestion. In BART, care is taken to minimize negative effects on the real network traffic, and the packets experiencing congestion are temporarily stored in the router buffers.BART uses the inter-packet separation strain as a convenient, dimensionless measure of the interaction between the cross traffic and the probe traffic. The expectation value for the strain is zero for an uncongested state, but rises linearly with the overload rate when congestion occurs. BART applies a Kalman filter, in order to maintain and update the estimate of the available bandwidth for each new measurement point. Although the system is non-linear over the whole range of probe rates, it is linear in the overload range; BART uses separate Kalman filters for the linear subsections.Although BART was originally intended for available-bandwidth measurements, it can be used for general-purpose, end-to-end QoS measurements.
METRIC SERVICE (MS-Policy)
Metric service is a customisable engine which allows data manipulation for a wide variety of management applications. It allows such applications to manipulate raw metrics and to work with data coming from heterogeneous sources. MS-Policy is the XML-based policy specification language that constitutes the core of the engine. The data manipulation and integration can be performed by metrics collection, aggregation and composition.
This mediation engine is the closest example that we find in the literature to the MOMENT proposal. It has been developed by the IBM India Research Laboratory. However, MOMENT goes several steps further: MOMENT provides not only data integration but also integration of the different monitoring services in a directory service. Due to the limited functionality of Metric Service, it turns out that no semantic capabilities are necessary. On the other hand ontological languages capture information elements along with relationships among them. Thus, they allow reasoning over available information, which is required to perform semantic queries for monitoring services. In contrast, the Metric Service does not provide query processing or dynamic source selection capabilities. Consequently, the MOMENT semantic database and offered functionality is much richer than the one reported in Metric Service. Furthermore, MOMENT builds on top of currently existing measurement and monitoring infrastructures. For all of them, an integration effort will be performed in terms of defining the service and data in a format which is amenable for integration in the MOMENT mediation engine database.
RIPE Test traffic Monitoring (RIPE TTM)
RIPE NCC offers Test Traffic Measurements as a service, to its users. The system continuously records one-way delay and packet-loss measurements as well as router-level paths ("traceroutes") between a large set of probe machines ("test boxes"). The test boxes are hosted by many different organisations, including NRENs, commercial ISPs, universities and others, and usually maintained by RIPE NCC as part of the service. While the vast majority of the test boxes is in Europe, there are a couple of machines in other continents, including outside the RIPE NCC's service area. Every test box includes a precise time source - typically a GPS receiver - so that accurate one-way delay measurements can be provided. Measurement data are entered in a central database at RIPE NCC's premises every night. The database is based on CERN's ROOT system. Measurement results can be retrieved over the Web using various presentations, both pre-generated and "on-demand". The TTM project has provided extremely high-quality and stable delay and loss measurements for a large set of network paths (IPv4 and IPv6) throughout Europe. These paths cover an interesting mix of research and commercial networks. The Web interface to the collected measurements supports in-depth exploration quite well, such as looking at the delay/loss evolution of a specific path over both a wide range of intervals from the very short to the very long. The raw data collected by the RIPE TTM infrastructure is not generally available, but access can be obtained for scientists access to the measurement infrastructure for research.However, RIPE NCC's Routing Information Service (RIS) raw data is openly available in a well-defined and relatively easy-to-use format, and hence it is possible for third parties to develop innovative and useful ways to look at that data.
GEANT2/perfSONAR
GEANT2 network is one of the largest wide area network supporting National Research and Education Networks (NRENs) of Europe is the largest research network, linking all research networks in the world. perfSONAR as a sub-project of GEANT2 is a structure for performance monitoring data exchange between networks, making it easier to solve performance problems occurring between 2 hosts, interconnected through several networks. It contains a set of services delivering performance measurements in a multi-domain environment. These services act as an intermediate layer, between the performance measurement tools and the visualisation applications. This layer is aimed at exchanging performance measurements between networks, using well defined protocols.
The perfSONAR project has major services to offer:
* Measurement Point Service: Creates and/or publishes monitoring information related to active and passive measurements
* Measurement Archive Service: Stores and publishes monitoring information retrieved from Measurement Point Services
* Lookup Service: Registers all participating services and their capabilities
* Authentication Service: Manages domain-level access to services via tokens
* Transformation Service: Offers custom data manipulation of existing archived measurements
* Resource Protector Service: Manages granular details regarding system resource consumption
* Toplogy Service: Offers topological information on networks Note that perfSONAR includes capabilities to integrate monitoring services using SOAP-XML. However, the measurement data and services database does not support semantic queries. In MOMENT, the design approach will reuse existing protocols for publication of services and data, such as the ones deployed in perfSONAR, but the mediation engine will have radically different architecture. Actually, the MOMENT mediation engine will incorporate an ontology, along with support for semantic query languages (SPARQL), thus allowing the use of semantic queries. Another distinctively feature of MOMENT is the use of en inference engine. The use of rules (in SWRL or other language) allows the inference of new instances in the ontology. For instance, if a rule defines a concrete condition (congestion, DDoS, etc.) based on a given measurements, it is possible to obtain those nodes that comply with that condition.COMO Project
The COMO project aims to design the fundamental building block of a network monitoring infrastructure that allows researchers and network operators to process and share network data across multiple sites. CoMo supports i) arbitrary traffic queries that run continuously on the live data streams, ii) retrospective queries that analyse past traffic data to enable network forensics. Data streams may have different formats (e.g., packet sequences, flow summaries, etc.) and originate from different platforms (e.g, passive link monitors, routers, wireless access points, etc.). CoMo can operate in the presence of different devices and data sources and provide a unified data interface to queries. Multiple CoMo systems will also cooperate to rapidly disseminate queries throughout the network of monitors, allowing operators to "drill down" to relevant data locations in the network.
Internet2/The Abeline observatory
The Abeline observatory, an activity within Internet2, has built up different data base with variety of network related data. The data resides on different databases that are available online through a variety of programmatic interfaces. Taken as a whole, the databases comprise a large correlated database for use by the research community, at international level.
* Abilene Usage Data
* Abilene Netflow Data
* Abilene Routing Data
* Abilene Latency Data
* Abilene Throughput Data
* Abilene Router Data
* Abilene Syslog DataThey include Planet lab and Passive measurement activities.PlanetLab is a global overlay network for developing and accessing new network services. The goal is to grow to 1000 geographically distributed nodes, connected by a diverse collection of links. Toward this end, PlanetLab nodes are into edge sites, co-location centres, and routing centres (e.g., the Abilene backbone). PlanetLab is designed to support both short-term experiments and long-running services. Currently running services include network weather maps, network-embedded storage, peer-to-peer networks, routing and multicast overlays, and content distribution networks. The objective of passive measurement and analysis (PMA) is to deliver new insights into the operation, behaviour, and health of the Internet, for the benefit of network users and operations. Passive header trace data provides the means to study workload profiles for a number of strategically located measurement points in high speed environments. PMA is collecting daily packet header trace samples from about two dozen sites within the US HPC networks, primarily Internet2/Abilene and its connectors.
UPNA, UAM, and CB, who are also partners in this proposal, have developed ETOMIC platform. Telscom has deployed ETOMIC in the Eurolabs testbed, which is accessible to user projects.
Technical infrastructure state of the art
The Internet architecture is in a perpetual state of transition. A decentralised, global mesh of several thousand autonomous systems (ASes), its providers are highly competitive, facing relatively low profit margins and few economic or business models by which they can differentiate themselves or their services. The challenges inherent in Internet operational support, particularly given its underlying best effort protocol, fully consume the attention of these Internet Service Providers (ISPs). Given its absence from the list of critical ISP priorities, data collection across individual backbones and at peering points continues to languish, both for end-to-end data (which require measurement across IP clouds) and actual traffic flows, e.g., the application (web, e-mail, real-audio, FTP...); packet origin, destination, and size; and the duration of flows. Yet it is detailed traffic and performance measurement and analysis that has heretofore been essential to identifying and ameliorating network problems. Trend analysis and accurate network system monitoring permit network managers to identify hot spots (overloaded paths), predict problems before they occur, and avoid congestion and outages via efficient deployment of resources and optimised network configurations. As our dependence on the Internet increases, we must deployed mechanisms that enable Internet infrastructure-wide planning and analysis and promote efficient scaling. User communities will also serve an important role in driving this process through their demands for verifiable service guarantees that are not readily available under the current Internet. This is particularly true for users engaged in Just-in-time manufacturing, such as the automotive industry, and users deploying high bandwidth applications and distance education, such as that proposed in the higher education community.
IETF
A first step in achieving measurements that are not only relevant, but also comparable, is the development of common definitions of IP metrics. The Internet Engineering Task Force (IETF) IP performance metrics (IPPM) working group was chartered to develop a more rigorous theoretical framework and guidelines for designing robust measurement tools for the Internet's wide variety of disparate signal sources. In late 1996, draft requests for comments (RFCs) were issued delineating metrics for connectivity [Mahdavi and Paxson] , one-way delay [Almes and Kalidindi] , and empirical bulk transfer capacity [Mathis] . (http://www.advanced.org/IPPM).
CAIDA
CAIDA is one of the organisations established in USA that provides a neutral framework to support cooperative research and operational measurement/analysis endeavours. Their activities include:
* creating a set of Internet performance metrics, in collaboration with IETF/IPPM and other organisations, and working with industry, consumer, regulatory, and other representatives to assure their utility and universal acceptance;
* creating a collaborative research environment in which commercial providers can share performance and engineering data confidentially, or in a desensitised form; and
* fostering the development of advanced networking technologies such as
** multicast and the MBONE
** web caching protocols/hierarchies
** traffic performance and flow characterisation tools
** bandwidth reservation and related QoS
** traffic visualisations, simulations and analyses
** BGP instability diagnosis
** next generation protocols/technologies, e.g. IPv6Issues of interest to MOMENT
The work in the past had shown the weaknesses in current tools and end-user measurement initiatives that are daunting, most of them lacking:
* well-defined traffic metrics
* uniformly applied methodologies, that account for varied/dynamic topologies
* clear definition of hypothesis or goal
* measurement scalability
* ability to explain phenomena: topology changes, routing loops, black holes
* relevance to actual ISP problems or mechanism for fixing measured problems
* communication of useful results to ISPs, vendors, and other usersTable 3: Internet visualisation tools
In addition to the lack of data on Internet traffic flows and performance, a similar dearth exists in quality analysis, modelling, and simulations tools, particularly those capable of addressing Internet scalability. Those commercial tools which are available currently are generally viewed by users as sorely inadequate. Few of these tools are designed for today's Internet environment and are therefore incapable of assisting Internet engineers and architects to reasonably plan for backbone expansions or substantive changes in protocols, technology or routing.
The IETF GROW working group is working on a standard definition of the binary format for BGP-4 route collectors, which tries to encompass developments in BGP-4 to adapt to the Internet's growth (i.e. use of 4 byte Autonomous System Identifiers, IPv6, etc. ) and applications of BGP-4 in new network technologies (i.e. MPLS and GMPLS applications). The MRT binary format specification, is still defined in an IETF draft.
Network Tomography
Network Tomography has attracted increasing attention in the past few years as a powerful technique that aims at determining – via statistical inference – link specific information, such as loss rates and delay statistics, using solely end-to-end measurements. Measurements may be passive (monitoring traffic flows) or active (generating probe traffic). In either case, the goal is to collect measurements statistics and/or observe external events that indirectly relate to the performance measures, e.g., link bandwidth, delay statistics of the network internal links. As such, network tomography has significantly improved the capabilities of measurement infrastructures which would otherwise be limited to capture only end-to-end path behaviour.
Active Network Tomography
In the field of active tomography, most of the proposed techniques rely on probing flows sent by a single source to multiple receivers, using either multicast traffic (where possible) or suitable unicast patterns, e.g., packet pairs, stripes, etc. The measured data are then fed to an inference engine which allows inferring, with some degree of confidence, the topology and loss/delay characteristics of intermediate links. One of the legacy infrastructures (ETOMIC) has already built in a network tomography engine that allows monitoring internal link delay behaviour from end-to-end measurements. However, all the previously proposed methods require cooperation between the sender and the receiver end-hosts participating in the measurement. This limits somewhat the scope of the paths over which measurements can be made, which in turn limits the number of links whose performance can be inferred. To improve upon this point we propose to exploit and extend recent ideas in the network tomography research area which attempt to overcome these limitations, and to include them into the MOMENT framework. More specifically we propose to investigate the use of RTT measurements – which eliminates the need for receiver cooperation – to infer internal link characteristics. The basic idea is to use either TTL expire techniques or SYN-ACK correlation to extract RTT measurement from a single vantage point. By properly designing a set of unicast active measurements – basically packets pairs to different routers and/or hosts - it is possible to infer the characteristics of the individual links in the traversed paths reusing well-established network tomography inference algorithms, but without explicit cooperation by the receiver end-hosts. By eliminating the requirement of receiver’ cooperation, the proposed approach would allow to greatly extend the range of paths over which tomographic measurements can be used. Moreover, it would avoid a number of practical complications, e.g. synchronization and data exchange between sender and receivers. In the project we plan to carry on original research on this topic and assess to which extent such techniques are applicable in practice. The planned research tasks include:
* Investigating/advancing the state of art algorithms for network tomography via RTT measurements;
* Developing a tool/module implementing these type of measurements as part of the existing infrastructures and on top of the proposed mediator.Passive Network Tomography
Besides the classical approach based on active measurements (packet probing), more recently new tomographic techniques are emerging based on purely passive monitoring. The key idea is that the traffic data captured by passive probes at key location inside a network (or at its edge) may reveal the presence of network problems (e.g. congested bottlenecks, route or server failures, etc.). When merged with external information available from other measurement sources (e.g. routing data), it becomes possible also to identify the location the event. A promising approach to passive network tomography relies on the inference of TCP-related micro-events (retransmission, timeout expirations, semi-RTT from DATA-ACK pairs, etc.) and on the subsequent statistical processing of such data. Such approach has been explored in some recent works1 and yield very interesting potential for practical application into real networks. Alternative approaches can be envisioned that combine the passive and active measurement paradigm, where the probing packets (active component) are captured at different points into the network (passive component) and then processed to reveal the inner status of the network. In MOMENT we are willing to explore the applicability of purely passive tomography techniques as well as integrated passive/active methods. Working task include:
* Investigating the monitoring methodology and inference/processing algorithms.
* Develop modules to implement such kind of measurements.
* Validate the method on top of real data.Privacy issues
Internet traffic passive measurement raises important privacy and anonymity concerns. RFC1262 explicitly states: “…an individual attempting a network measurement activity should ensure that […] the data collected will not violate privacy, security, or acceptable use concerns…”, and such recommendation is a proof that the issue of privacy in network measurement is indeed recognised by the Internet (IETF) community. This problem is made worse when traces are gathered at different points in the networks and from different measurement systems, as the ability of linking each other, and correlate possibly very detailed logs increases the possibility to draw a general picture of the end user network behaviour.
To face these issues, MOMENT will include, in its design requirements and in the specified system architecture, both i) data anonymity mechanisms amenable to preserve properties that are later on exploited by measurement and monitoring applications (e.g., prefix-preserving anonymisation mechanisms), as well as privacy preserving operation on gathered data and specifically ii) access control and iii) privacy-preserving elaboration and data extraction.
While anonymisation mechanisms have received considerable attention, for what concerns the last two categories of privacy-preserving solutions, to date, only a limited amount of work has been specifically targeted to monitoring and measurement systems. Nevertheless MOMENT will build on, adapt, and improve, approaches proposed in the last decade by the privacy research community on these subject matters. For instance, a large amount of work has been carried out in the field of Privacy Preserving Data Mining (PPDM) [LIN00, UMB06, CMU04, UAL04] , and although most of this work has been discussed in general terms and has not been specifically addressed to the area of network monitoring, nevertheless some proposed solutions may provide a very good starting point for being adapted to the specific context tackled in the MOMENT project. Specifically, the work carried out in the area of Selective Private Function Evaluation (SPFE) [CAN00] , consisting in computing statistics of selected data stored inside a remote database in a privacy preserving manner is of significant interest for our project. More precisely, it is possible for a certain user to calculate a function f of some unknown values xi hold by a remote server (ore more servers), which will learn nothing about the client. The client will learn only the desired statistics, but not the values of the data nor partial computations; the server will not learn the indices i involved in the computation. Guidelines to design cryptographic protocols to accomplish these tasks in the most efficient way have been provided in [SUB04, NAO01, KRU06] and an application of these techniques to large scale network measurements has been recently proposed in [ROU06] . An important class of PPDM techniques of interest for MOMENT have been studied, in general terms, in [AGR00, AGR01, KAR03, LIU06, AGR05] , with the intent to modify original data values in order to have a new version of the database which can be safely released to the public. Various methods have been proposed with that purpose, such as additive perturbation, replacement with meaningless symbols, aggregation to a coarser granularity, or even sampling. The crucial point with all this approach is then to accurately reconstruct the aggregate distributions and to easily perform data mining, as data perturbation usually results in a degradation of the database performance.
Access control models specifically tailored towards privacy protection is another important research area which has received considerable “general” attention but limited application in the measurement and monitoring area, and a further goal of the MOMENT project is to assess the applicability and the adaptability of these models to the system under design. To that respect, several approaches have been proposed, that go beyond conventional Role-Based Access Control (RBAC) [FER01] , incorporating different criteria in access control decisions, rather than just which user, having which role, is performing which action on which data object. An important technology is that of Hippocratic Databases [AGR02, LEF04, MAS06] , which allow to associate a “purpose” to every table in the Database. When a query is submitted to the database, the system answers only queries for which the purpose is equal to that for which the information has been collected. A sophisticated approach to purpose hierarchies has led to the definition of a purpose Ontology in [BYU05] . Key ideas from this approach will be exploited in the definition of the MOMENT ontology. Furthermore, MOMENT, in its architecture specification, will also leverage on existing languages and XML extensions specifically devised for access control and privacy protection. These include the OASIS eXtensible Access Control Markup Language (XACML) [OAS04] , a general-purpose access control standard written in XML, and its Privacy Policy Profile specified in XACML v2.0 [MOS05] .
References
* [LIN00] Y. Lindell and B. Pinkas, Privacy preserving data mining. In Advances in Cryptology - CRYPTO '00, volume 1880 of Lecture Notes in Computer Science, pages 36--54. Springer-Verlag, 2000.
* [UMB06] Privacy Preserving Data Mining Bibliography: http://www.csee.umbc.edu/~kunliu1/research/privacy_review.html
* [CMU04] References to Privacy-Preserving Data Mining Literature: http://privacy.cs.cmu.edu/dataprivacy/papers/ppdm/
* [UAL04] Privacy Preserving Data Mining Publications: http://www.cs.ualberta.ca/%7Eoliveira/psdm/pub_by_year.html
* [CAN00] R. Canetti, Y. Ishai, R. Kumar, M. K. Reiter, R. Rubinfeld, and R. N. Wright, “Selective private function evaluation with applications to private statistics”, Proc. of the 20th ACM Symposium on Principles of Distributed Computing (PODC), 2001.
* [SUB04] H Subramaniam, RN Wright, Z Yang, Experimental Analysis of Privacy-Preserving Statistics Computation. Proc. of the VLDB Worshop on Secure Data Management, 2004 – Springer
* [NAO01] M. Naor, and K. Nissim. Communication Preserving Protocols for Secure Function Evaluation. To appear, STOC 2001.
* [KRU06] Louis Kruger, Somesh Jha, Eu-Jin Goh, Dan Boneh: Secure function evaluation with ordered binary decision diagrams. ACM Conference on Computer and Communications Security 2006: 410-420.
* Matthew Roughan and Yin Zhang. Secure distributed data-mining and its application to large-scale network measurements. ACM SIGCOMM Computer Communication Review, Volume 36 , Issue 1 (January 2006).
* [AGR00] Rakesh Agrawal and Ramakrishnan Srikant, Privacy-preserving data mining, In Proceedings of the ACM SIGMOD Conference on Management of Data (2000), 439–450.
* [AGR01] D. Agrawal and C. C. Aggarwal. On the design and quantification of privacy preserving data mining algorithms. In Proceedings of the Twentieth ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems, Santa Barbara, California, USA, May 21-23 2001. ACM.
* [KAR03] Hillol Kargupta, Souptik Datta, Qi Wang, Krishnamoorthy Sivakumar: On the Privacy Preserving Properties of Random Data Perturbation Techniques. ICDM 2003: 99-106
* [LIU06] Kun Liu, Hillol Kargupta, Jessica Ryan, "Random Projection-Based Multiplicative Data Perturbation for Privacy Preserving Distributed Data Mining," IEEE Transactions on Knowledge and Data Engineering, vol. 18, no. 1, pp. 92-106, Jan., 2006.
* [AGR05] S. Agrawal and J.R. Haritsa, “A Framework for High-Accuracy Privacy-Preserving Mining,” Proc. 21st Int’l Conf. Data Eng. (ICDE’05), pp. 193-204, Apr. 2005.
* [FER01] D. F. Ferraiolo, R. Sandhu, S. Gavrila, D. R. Kuhn and R. Chandramouli, “Proposed NIST Standard for Role-Based Access Control”, ACM Transactions on Information and System Security (TISSEC), Vol. 4 No. 3, p.224-274, August 2001.
* [AGR02] R. Agrawal, J. Kienrnan, R. Srikant and Y. Xu, “Hippocratic Databases”, in Proc. of the 28th International Conference on Very Large Databases (VLDB 2002), Hong Kong, China, Aug. 2002.
* [LEF04] K. LeFevre, R. Agrawal, V. Ercegovac, R. Ramakrishnan, Y. Xu and D. J. DeWitt, “Limiting Disclosure in Hippocratic Databases”, in Proc. of the 30th International Conference on Very Large Databases (VLDB 2004), Toronto, Canada, Aug. 2004.
* [MAS06] F. Massacci, J. Mylopoulos and N. Zannone, “Hierarchical hippocratic databases with minimal disclosure for virtual organizations”, The International Journal on Very Large Data Bases, Vol. 15, No. 4, pp. 370-387, Springer, Nov. 2006.
* [BYU05] J. Byun, E. Bertino and N. Li, “Purpose Based Access Control for Privacy Protection in Relational Database Systems”, in Proc. of the 10th ACM symposium on Access control models and technologies, Stockholm, Sweden, June 2005.
* [OASIS04] Organization for the Advancement of Structured Information Standards (OASIS), “OASIS eXtensible Access Control Markup Language (XACML) TC”, 2004, http://www.oasis-open.org/committees/xacml/.
* [Moses05] T. Moses, “OASIS Privacy Policy Profile of XACML v2.0”, OASIS Standard, Feb. 2005.
Wikimedia Foundation. 2010.
