[Community Feedback] Thoughts and Recommendations from the ACM SIGCOMM 2017 Reproducibility Workshop

This paper has been submitted to CCR. This is a draft version of the paper that has not been peer-reviewed. Comments on the paper are encouraged through the comment facility at the bottom of this page before December 15th, 2017.
Damien Saucez, Luigi Iannone

Abstract

Ensuring the reproducibility of results is an essential part of experimental sciences, including computer networking. Unfortunately, as highlighted recently, a large portion of research results are hardly, if not at all, reproducible, raising reasonable lack of conviction on the research carried out around the world.

Recent years have shown an increasing awareness about reproducibility of results as an essential part of research car- ried out by members of the ACM SIGCOMM community. To address this important issue, ACM has introduced a new policy on result and artifact review and badging. The policy defines the terminology to be used to assess results and artifacts but does not specify the review process or how to make research reproducible.

During SIGCOMM’17 a side workshop has been organized with the specific purpose to tackle the issue. The objective being to trigger discussion and activity in order to craft recommendations on how to introduce incentives for authors to share their artifacts, and the details on how to use them, as well as defining the process to be used.

This editorial overviews the workshop activity and summarizes the main discussions and outcomes.

Draft article

The October 2017 Issue

Computer Communication Review (CCR) continues to promote reproducible research by encouraging the submission of papers providing artifacts (software, datasets, . . . ). As in the previous issue, all the accepted technical papers have released their artifacts. These artifacts will help other researchers to improve the results published in CCR by easily comparing their new ideas with those described in the related work.

As announced in the last issue, the CCR Online website, https://ccronline. sigcomm.org has been enhanced with a Community Comments section to encourage interactions between readers and authors. Several submitted articles have been posted in this section and I encourage you to look at them and provide constructive comments to their authors. Providing open and constructive comments is a nice way to be involved in the SIGCOMM community.

Three technical papers were accepted from the open call. In Measuring YouTube Content Delivery over IPv6 Vaibhav Bajpai et al. report on large scale measurements that compared the performance of Youtube over IPv4 and IPv6. As IPv6 gets widely deployed, it is interesting to study whether the two network stacks provide similar performance. They release both the measurement software and the collected dataset.

In Inside the Walled Garden: Deconstructing Facebook’s Free Basics Program, Rijurekha Sen et al. study the operation of Facebook’s recent Free Basics program in Pakistan and South Africa. Their software and measurement dataset will probably serve as a baseline for researchers who will explore the evolution of similar programs in the future. These artifacts have been significantly improved by the authors based on interactions with reviewers.

In Dissecting Last-mile Latency Characteristics Vaibhav Bajpai et al. use measurements on two different platforms to analyse the factors that influence the performance of access networks. They release both measurement scripts and datasets.

Two editorial papers report on recent workshops. In The 9th Workshop on Active Internet Measurements (AIMS-9) Re- port, kc Claffy and David Clark summarise the AIMS-9 workshop that was held in March 2017 at CAIDA. In Report on Net- working and Programming Languages 2017, Nikolaj Bjorner et al. summarise the SIGCOMM’17 NetPL workshop.

In addition to the papers accepted from the open call, this issue also contains the best papers from the SIGCOMM’17 workshops:

I hope that you will enjoy reading this new issue and welcome comments and suggestions on CCR Online or by email at ccr-editor at sigcomm.org.

Olivier Bonaventure

CCR Editor

Measuring YouTube Content Delivery over IPv6

Vaibhav Bajpai, Saba Ahsan, Jürgen Schönwälder, Jörg Ott

Abstract

We measure YouTube content delivery over IPv6 using ∼100 SamKnows probes connected to dual-stacked networks representing 66 different origin ASes. Using a 34-months long (Aug 2014-Jun 2017) dataset, we show that success rates of streaming a stall-free version of a video over IPv6 have improved over time. We show that a Happy Eyeballs (HE) race during initial TCP connection establishment leads to a strong (more than 97%) preference over IPv6. However, even though clients prefer streaming videos over IPv6, we observe worse performance over IPv6 than over IPv4. We witness consistently higher TCP connection establishment times and startup delays (∼100 ms or more) over IPv6. We also observe consistently lower achieved throughput both for audio and video over IPv6. We observe less than 1% stall rates over both address families. Due to lower stall rates, bitrates that can be reliably streamed over both address families are comparable. However, in situations, where a stall does occur, 80% of the samples experience higher stall durations that are at least 1s longer over IPv6 and have not reduced over time. The worse performance over IPv6 is due to the disparity in the availability of Google Global Caches (GGC) over IPv6. The measurements performed in this work using the youtube test and the entire dataset is made available to the measurement community.

Download the full article DOI: 10.1145/3155055.3155057

Inside the Walled Garden: Deconstructing Facebook’s Free Basics Program

Rijurekha Sen, Sohaib Ahmad, Amreesh Phokeer, Zaid Ahmed Farooq, Ihsan Ayyub Qazi, David Choffnes, Krishna P. Gummadi
Abstract

Free Basics is a Facebook initiative to provide zero-rated web services in developing countries. The program has grown rapidly to 60+ countries in the past two years. But it has also seen strong opposition from Internet activists and has been banned in some countries like India. Facebook highlights the societal benefits of providing low-income populations with free Internet access, while detractors point to concerns about privacy and network neutrality.

In this paper, we provide the first independent analysis of such claims regarding the Free Basics service, using both the perspective of a Free Basics service provider and of web clients visiting the service via cellular phones providing access to Free Basics in Pakistan and South Africa.

Specifically, with control of both endpoints, we not only provide a more detailed view of how the Free Basics service is architected, but also can isolate the likely causes of network performance impairments. Our analysis reveals that Free Basics services experience 4 to 12 times worse network performance than their paid counterparts. We isolate the root causes using factors such as network path inflation and throttling policies by Facebook and telecom service providers.

The Free Basics service and its restrictions are designed with assumptions about users’ device capabilities (e.g., lack of JavaScript support). To evaluate such assumptions, we infer the types of mobile devices that generated 47K unique visitors to our Free Basics services between Sep 2016 and Jan 2017. We find that there are large numbers of requests from constrained WAP browsers, but also large fractions of high-capability mobile phones that send Free Basics requests.

We discuss the implications of our observations, with the hope to aid more informed debates on such telecom policies.

Download the full article DOI: 10.1145/3155055.3155058

Dissecting Last-mile Latency Characteristics

Vaibhav Bajpai, Steffie Jacob Eravuchira, Jürgen Schönwälder
Abstract

Recent research has shown that last-mile latency is a key network performance indicator that contributes heavily to DNS lookup and page load times. Using a month-long dataset collected from 696 residential RIPE Atlas probes and 1245 SamKnows probes, we measure last-mile latencies from 19 ISPs (RIPE Atlas) in the US and the EU, and 9 ISPs (SamKnows) in the UK. We show that DSL deployments not only tend to enable interleaving on the last-mile, but also employ multiple depth levels that change over time. We also witness that last-mile latency is considerably stable over time and not affected by diurnal load patterns. Unlike observations from prior studies, we show that cable providers in the US do not generally exhibit lower last-mile latencies when compared to that of DSL. We instead identify that last-mile latencies vary by subscriber location and show that last-mile latencies of cable providers in the US are considerably different across the US east and west coast. We further show that last-mile latencies vary depending on the access technology used by the DSL modem wherein VDSL deployments show last-mile latencies lower than ADSL1/ADSL2+ broadband speeds. The entire dataset and software used in this study is made available to the measurement community.

Download the full article DOI: 10.1145/3155055.3155059

The 9th Workshop on Active Internet Measurements (AIMS-9) Report

kc Claffy, David Clark
Abstract

For almost a decade” CAIDA has hosted its Workshop on Active Internet Measurements (AIMS-9). This workshop series provides a forum for stakeholders in Internet active measurement projects to communicate their interests and concerns, and explore cooperative approaches to maximizing the collective benefit of deployed infrastructure and gathered measurements. On 1-3 March 2017, CAIDA hosted the ninth Workshop on Active Internet Measurements (AIMS-9). Materials related to the workshop are at http://www.caida.org/workshops/aims/1703/.

 

Download the full article DOI: 10.1145/3155055.3155060

Perceived Performance of Top Retail Webpages In the Wild

Qingzhu, Prasenjit Dey, Parvez Ahammad
Abstract

Clearly, no one likes webpages with poor quality of experience (QoE). Being perceived as slow or fast is a key element in the overall perceived QoE of web applications. While extensive effort has been put into optimizing web applications (both in industry and academia), not a lot of work exists in characterizing what aspects of webpage loading process truly influence human end-user’s perception of the Speed of a page. In this paper we present SpeedPerception, a large-scale web performance crowdsourcing framework focused on understanding the perceived loading performance of above-the-fold (ATF) webpage content. Our end goal is to create free open-source benchmarking datasets to advance the systematic analysis of how humans perceive webpage loading process.

In Phase-1 of our SpeedPerception study using Internet Retailer Top 500 (IR 500) websites, we found that commonly used navigation metrics such as onLoad and Time To First Byte (TTFB) fail (less than 60% match) to represent majority human perception when comparing the speed of two webpages. We present a simple 3-variable-based machine learning model that explains the majority end-user choices better (with 87 +- 2% accuracy). In addition, our results suggest that the time needed by end-users to evaluate relative perceived speed of webpage is far less than the time of its visualComplete event.

Download the full article DOI: 10.1145/3155055.3155062

Hierarchical IP flow clustering

Kamal Shadi, Preethi Natarajan, Constantine Dovrolis
Abstract

The analysis of flow traces can help to understand a network’s usage patterns.
We present a hierarchical clustering algorithm for network flow data that can summarize terabytes of IP traffic  into a parsimonious tree model.  The method automatically finds an appropriate scale of aggregation so  that each cluster represents a local maximum of the traffic density from a block of source addresses
to a block of destination addresses. We apply this clustering method on NetFlow data from an enterprise network, find the largest traffic clusters,  and analyze their stationarity across time. The existence of heavy-volume clusters that persist over long time scales can help network operators to perform usage-based accounting, capacity provisioning and traffic engineering. Also, changes in the layout of hierarchical clusters can facilitate the detection of anomalies and significant changes in the network workload.

Download the full article DOI: 10.1145/3155055.3155063

Contain-ed: An NFV Micro-Service System for Containing e2e Latency

Amit Sheoran, Puneet Sharma, Sonia Fahey, Vinay Saxena
Abstract

Network Functions Virtualization (NFV) has enabled operators to dynamically place and allocate resources for network services to match workload requirements. However, unbounded end-to-end (e2e) latency of Service Function Chains (SFCs) resulting from distributed Virtualized Network Function (VNF) deployments can severely degrade performance. In particular, SFC instantiations with inter-data center links can incur high e2e latencies and Service Level Agreement (SLA) violations. These latencies can trigger timeouts and protocol errors with latency-sensitive operations.

Traditional solutions to reduce e2e latency involve physical deployment of service elements in close proximity. These solutions are, however, no longer viable in the NFV era. In this paper, we present our solution that bounds the e2e latency in SFCs and inter-VNF control message exchanges by creating micro-service aggregates based on the affinity between VNFs. Our system, Contain-ed, dynamically creates and manages affinity aggregates using light-weight virtualization technologies like containers, allowing them to be placed in close proximity and hence bounding the e2e latency. We have applied Contain-ed to the Clearwater IP Multimedia Subsystem and built a proof-of-concept. Our results demonstrate that, by utilizing application and protocol specific knowledge, affinity aggregates can effectively bound SFC delays and significantly reduce protocol errors and service disruptions.

Download the full article DOI: 10.1145/3155055.3155064

 

How to Measure the Killer Microsecond

Mia Primorac, Edouard Bugnion, Katerina Argyraki
Abstract

Datacenter-networking research requires tools to both generate traffic and accurately measure latency and throughput. While hardware-based tools have long existed commercially, they are primarily used to validate ASICs and lack flexibility, e.g. to study new protocols. They are also too expensive for academics. The recent development of kernel-bypass networking and advanced NIC features such as hardware timestamping have created new opportunities for accurate latency measurements.

This paper compares these two approaches, and in particular whether commodity servers and NICs, when properly configured, can measure the latency distributions as precisely as specialized hardware.

Our work shows that well-designed commodity solutions can capture subtle differences in the tail latency of stateless UDP traffic. We use hardware devices as the ground truth, both to measure latency and to forward traffic. We compare the ground truth with observations that combine five latency-measuring clients and five different port forwarding solutions and configurations. State-of-the-art software such as MoonGen that uses NIC hardware timestamping provides sufficient visibility into tail latencies to study the effect of subtle operating system configuration changes. We also observe that the kernel-bypass-based TRex software, that only relies on the CPU to timestamp traffic, can also provide solid results when NIC timestamps are not available for a particular protocol or device.

Download the full article DOI: 10.1145/3155055.3155065