OONI-dev meeting and hackathon 2016

Oonitarians are spread out across the globe, and OONI’s core team is no exception. We recently met in Berlin to hack on OONI’s software, work on our roadmap, and to have all those discussions that are more fruitful to have in person than on IRC. We also held a hackathon to engage new individuals with our project!

Below we document some of the things that we worked on as part of our OONI-dev meeting and hackathon.

OONI-dev meeting

OONI’s internal meeting was an opportunity for the project’s core team members to meet in person, discuss priorities over the next months, and to adjust the project’s roadmap accordingly. This meeting took place between Monday, 7th November and Thursday, 10th November.

We started off by mapping all of the ideas, questions, and problems that we wanted to collaboratively discuss and work on as part of the meeting. We then proceeded with voting amongst us on the ones that we believed were of highest priority, and subsequently creating sessions out of them.

As a result, we spent the first two days of the OONI-dev meeting having discussions as part of the following six sessions.

Session 1: OONI Explorer user stories.

Session 2: Probe orchestration.

Session 3: PCAPs and throttling as a form of censorship as opposed to IP blocking.

Session 4: UX design.

Session 5: The future of testing clients: How should we ship software to end users?

Session 6: OONI mobile client.

We then spent the third day working on our roadmap for the next six months, adjusting it to the project’s needs and priorities based on the discussions from the sessions of the previous days. During the fourth (and final) day of our internal meeting we held smaller, ad hoc discussions based on the rest of the ideas/questions/problems that were mapped out in the first day, and which didn’t fit into any of the six core sessions of the first two days. We also spent the last day of the internal meeting hacking on ooniprobe and measurement-kit (a network measurement library that ooniprobe is being integrated in).

OONI Hackathon

The OONI hackathon, held at Berlin’s OnionSpace on 11th and 12th November, was an opportunity for us to meet new people interested in engaging with our project. We were excited to see that individuals with diverse skills and backgrounds showed up at the hackathon, including data scientists, programmers, researchers, and activists from various countries around the world.

Below is a summary of the things that we worked on as part of the two-day hackathon.

Data analysis

OONI Explorer provides a web interface to explore all of OONI’s network measurements. However, OONI’s data processing pipeline still needs some improvement to make OONI’s data more accessible and easier to navigate. As such, we were thrilled to have data scientists join the hackathon and offer suggestions on how to improve our data analysis techniques.

We discussed and defined queries for data analysis that can help improve the next iteration of OONI’s pipeline. This included:

Furthermore, we discussed and implemented techniques for identifying and filtering false positives that emerge in the measurements. As part of this, we discussed ways to filter Cloudflare false positives, as well as techniques for better identifying block pages within measurements. We also discussed a methodology for detecting URLs that fail in the same way across measurements, across time, and across countries, and filtering them as false positives.

OONI’s measurement API

In addition to OONI Explorer, OONI’s network measurements are published in JSON format. One of our priorities though is to also build an API to the measurements that enables data analysis, allowing users to query the API to retrieve measurements based on certain characteristics (to query, for example, all anomalous measurements).

As part of the hackathon, we continued our work on OONI’s new measurement API and created a staging version of it.

Web development

Several web developers joined the hackathon and we worked with them on the following:

Measurement-kit

Measurement-kit is a network measurement library that ooniprobe is gradually being integrated in. As part of the hackathon, we worked on improving the DNS capabilities of measurement-kit along the following lines:

In addition, we also started implementing support for cloud-fronted collectors in measurement-kit to allow users to submit the result of OONI measurements to our servers through a cloud-based service (e.g. Amazon), thus masking the fact that they are communicating with OONI servers. We also committed enhancements and fixes to support the development of mobile applications based on measurement-kit.

OONI mobile

Spoiler: Over the next few months we will be releasing a mobile app (for Android and iOS) for running ooniprobe! During the hackathon, we worked on improving the design of OONI’s mobile app.

Here’s a sneak peak of OONI’s upcoming mobile app:

OONI mobile

Screenshot: OONI mobile

OONI mobile testing a site

Screenshot: OONI mobile testing http://neubot.org

Test lists

An important part of identifying censorship is determining which websites to examine for blocking.

OONI bases its testing on the Citizen Lab’s test list Github repository which includes a wide range of country-specific lists of URLs. These lists constantly need to be updated with new URLs and, in many cases, new lists need to be created for countries that don’t already have their own test lists.

We were happy to see that various participants joined the hackathon to help contribute to existing test lists and to learn how to create new ones. As part of the hackathon, participants worked on updating the Palestinian test list to include many new URLs. We also prioritized on updating test lists for countries with upcoming elections, such as the test list for Gambia.

Running ooniprobe

Amongst the new faces that joined us at the hackathon, we were also happy to meet people curious to learn about OONI and how to run ooniprobe. We discussed OONI’s approaches and methodologies, and taught participants how to run various ooniprobe software tests. As an outcome, we hope that participants will share the skills and knowledge they acquired through the hackathon with others too, and help increase transparency on internet censorship around the world.


The OONI team would like to send a warm thank you to all the participants who joined the hackathon, to the OnionSpace for sharing its space with us for a week, and to the Ford Foundation for covering some of the OONI team’s travel costs, thus making this meeting and hackathon possible.