All Our Ideas

screenshot of allourideas.org

About the project

All Our Ideas is a research project created by Matthew Salganik, Professor of Sociology at Princeton University. The goal of the project is to explore new methods of collecting social data via “wiki surveys”. The primary face of the project is the website allourideas.org, where anyone can create their own wiki surveys. A variety of organizations have created their own wiki surveys including: Wikipedia, The Washington Post, United Nations Division for Sustainable Development, and the New York City Mayor’s Office. The site itself is open source, which allows anyone to run their own version of All Our Ideas.

The technology

All Our Ideas is written using the Ruby on Rails framework. The underlying database is MySQL, while Redis serves as the primary cache datastore.

There are two Ruby on Rails projects that make up the site:

  1. allourideas.org serves the All Our Ideas site.
  2. pairwise-api provides the underlying API that powers the All Our Ideas site.

My role

After the initial development had been completed, I became the primary software developer on the project and responsible for the smooth running of allourideas.org, as well as new feature and maintenance development.

The allourideas.org site sees traffic levels that are highly variable due to the fact that anyone can create and promote their own wiki survey. As the site became more popular, I worked to ensure that we could handle these bursts of traffic by identifying and optimizing bottlenecks in the code. Some of the optimizations included: adding appropriate indexes to the MySQL database, reducing the number of database queries, and adding caching to computationally intensive portions of the voting process.

The maintenance ranged from small tasks like getting old tests working again to larger architectural changes. I setup the project to use Travis CI to automate the running of the tests with each commit. I also worked with Matt and a designer to update the site to be responsive using Bootstrap. The site had already been internationalized but the process for adding additional translations was cumbersome and frustrating to translators. I update that process to allow translators to use Google Sheets to suggest their translations. The process then took in that data as a CSV and updated the appropriate files in the Ruby on Rails application.

One larger architectural change was to alter how CSV data from wiki surveys is generated. The CSV data that is generated involves both the pairwise-api and allourideas.org applications. The previous architecture relied on having a shared location (e.g., the file system or Redis) that would allow the allourideas.org application to fetch the data generated by the pairwise-api application. This architecture caused problems when scaling the service with more servers and complicated the installation process for other users of the software. The solution that we settled on was to add another API call to the pairiwse-api application which would allow the allourideas.org to fetch the CSV data it needed to complete the CSV data export. This fixed the problems with the old architecture and fit nicely with the architecture of the rest of the application.

Finally, I supported the open source project by writing documentation, assisting users who wanted to run their own version of All Our Ideas, and simplified the installation requirements.