At the time of this project, although their UI and visuals were rough and low fidelity, Couchers.org already had a preliminarily defined structure, navigation, and user flow within the app. By conducting a usability test early on, we could solve problems that may arise later before the team commits to a high fidelity UI- by providing feedback early on in the process.
Above: State of Couchers app at the time of testing
Understanding the culture, evaluating the competition.
With none of us within the team having couch surfing experience, the best way to learn about couch surfing was to try it firsthand ourselves- however the restrictions as a result of the pandemic meant this was not an option for us.
To fill some of the gaps I had within this niche and to supplement discussion we had with the Couchers.org team, I browsed through various couch surfing subreddits and discord channels to learn more about the topic and chat directly with enthusiasts within the realm.
From a couple of conversations with those within the couch surfing and hospitality exchange scene, CouchSurfing and Trustroots were the most commonly used mobile applications. We signed up and looked thoroughly through these apps to understand how users currently interact with couch surfing platforms.
While Airbnb is not exactly a couch surfing or hospitality exchange service, we took cues on several of their design decisions. As one of the most popular travel apps, Airbnb set precedence to a lot of successful design patterns that were already familiar to users.
With more context of the couch surfing scene, we felt a lot more comfortable in how we were to execute our study plan.
In designing our usability study, our goal was to learn:
How users would utilize core interactions
Whether participants were able to complete specific tasks provided successfully
The pain points throughout this process
Identify existing behaviors and see how we can leverage them within the app
Current flow of requesting to surf
Above: Screenshots of the existing low fidelity flow for requesting to surf from a host
Creating evaluation tasks & scenarios for testing
To test the product's functionality, we decided to do task-based usability testings on the current prototype. This allowed us to observe how the users interact with the product as they attempt to complete the given tasks.
The participants were assigned to do 5 tasks, each of which was doable through the Community page. We began by giving them a scenario of a user persona who may be interested in using the Couchers app.
Asking the right questions
To have a better understanding of the demographic, we began the study with the pre-test questions and prepared a different set for experienced versus novice users. For each testing, we concluded a post-test questionnaire to de-brief and collect feedback from the participants.
The questions were based on how users felt about the tasks, specific features in the app, and the prototype in general. While we primarily seeked for qualitative data through open ended questions, we included basic scaling questions for in order to gain quantitative data that could support our qualitative findings.
Recruiting the participants.
We recruited experienced users through various subreddits and discord channels. For this portion of our participant pool, we aimed to recruit more females as they have generally reported to have more negative experiences with certain aspects of Couchsurfing.
We also recruited potential novice users (Those who have not couch surfed before) as they would provide insight on the app usability overall, and offer a different and unbiased perspective from current couch surfers.
Making sure of validity within testing
To ensure internal validity we provided a quick on-boarding of the mobile web app prior to the testing itself in order to reduce the learning curve and novelty effects for all participants.
To ensure external validity, we conducted the study in the same environment for all of our participants. We also mixed our pool of participants with experienced and novice users, randomly selected with various countries and backgrounds in order for our study to be generalized to a larger population.
Now that we have the data...
1. How do we make sense of it
2. How can we leverage it?
We used affinity diagramming to analyze our data collaboratively. Issues highlighted in the usability tests were grouped, categorized, and then assigned a problem severity rating on how critical the issue was in terms of the interface usability based on Nielsen’s severity scale.
This allowed us to draw conclusions on higher-level themes, recurring patterns, and usability problems encountered within the Couchers app.
We also calculated averages for success rates based on our completion criteria for each task as well as the mean results of our likert scaling questions.
This quantitative data was compared alongside our qualitative findings to confirm whether their experience and verbal comments were indicative of how well they were able to complete the tasks.
Back to the drawing board 📝
With data-backed insight of flaws and usability issues within the Couchers.org interface, using the old fashion pen & paper, as a team we brainstormed ways we can reflect these in the interface and pooled our thoughts together.
