What are they?
And how are they useful in UX research?
Diary studies are a longitudinal research method, that is they allow you to capture observations about users over a period of time.
You might find that shorter cross-sectional studies, such as usability testing, are limiting when you’re trying to understand user’s evolving behavioural pattern or interaction with a product on a repeated basis. And if that’s the case, a longitudinal method such as a diary study will help to fill in those blanks.
Having experienced some of the common pitfalls with this method, I’ve come up with some useful suggestions on planning and conducting a diary study.
Things to consider
Although diary studies are extremely useful at gathering insight on naturalistic behaviours and utilisation of products over time, they’re often not the simplest, and certainly not the fastest type of study to run.
And there are two main reasons for this:
In order to capture longitudinal data, your study will need to run for at least a couple of weeks, if not more. Longer studies take more time to plan and organise, it’s harder to find engaged participants, and recruitment costs and incentives are generally more expensive because the study is longer
Over the course of your study, you’ll capture an overwhelming amount of data, if you have ten participants on a two week study, writing one diary entry a day, that’s 140 potentially lengthy documents you have to read. The review and analysis of diary study data takes time and can be all consuming!
Don’t let these things put you off though, the incredible insights you’ll gather will be worth it!
Planning
The most successful diary studies I have run are the ones that have been well planned, and I think this is something you learn to get good at as you go along - it’s true that we learn from our mistakes! Spending a good amount of time and effort on planning for all eventualities will mean your study runs smoothly and is pleasant for your subjects to participate in.
I like to use the ‘5 bums on a rugby post’ method to ensure I have covered all bases when planning:
5 bums on a rugby post. Nope, I haven’t lost my mind. That’s how I remember who, what, where, when, why, and (the rugby post) how!
Who will participate in the study?
Start thinking about your participants right away, and consider who do you want to observe. This is important as your study details may differ depending on what type of participants you use. For example the way you plan a study with teenagers would be different from a study with pensioners. Its also worth thinking about people beyond those directly taking part in your study - who do your participants share their experience with, and how do they communicate with them? Keep your users in mind from the start!
What do you want to capture?
Consider what kind of behaviours are you interested in observing and think about how you’ll communicate the purpose of your study to participants and capture these. You need to be clear on exactly what is is that you’re interested in to get relevant diary entries and avoid tons of info about unrelated or unimportant topics. You’ll also need to be clear with participants on your expectations in terms of what is required of them in order to qualify for their incentive (ie MUST complete all 14 days, MUST write complete entries, MUST NOT submit one word answers etc)
Where will the study take place?
Context is key to a lot of research insights, and even more so in diary studies. Thinking about where your participants are when they’re trying to complete their goals and use your product is really important for two reasons. Firstly, how might you enable participants to capture that contextual insight for you, could you allow them to add photos to their diary entries? And secondly, does your diary collection method suit their context, if they’re likely to be participating during their commute is your diary entry mechanism mobile friendly?
When are participants expected to write entries?
Are you particularly interested in your subjects thoughts when they first wake up in the morning? Or maybe theres a particular time of day that your users get the most from your product? Consider that when designing your study and again make sure you make it clear that you require diaries to be recorded at those times and make it easy for your participants to do so. Its also important to consider the dates your study will run from and to.
Why, why, why, why, why?
Remember Toyota’s 5 whys technique, and find a way to mitigate against factual diary entries. “I went here”, or “I did this” aren’t particularly useful insights by themselves, but often these can be the sort of entries that participants default to. We want to understand why they make choices and what their thought process is, so be creative with your diary collection method to encourage lots of detail about thoughts and feelings behind actions.
How will participants record their behaviours and experience?
You’ll need to plan a method of recording diary entries that is accessible and easy for your participants to use. If there is a barrier to creating diary entries, participants will drop out of the study and your results will weaken. Trust me, I have had this problem before! Make it easy simple and above all fun and rewarding to be part of your study!
Top Tips
Here are some suggestions I have for ensuring a study runs smoothly.
Use a recruitment agency
As I mentioned before, diary studies can be taxing in terms of organisation and analysis. Take some of the strain away by using an agency to recruit participants for you, as they’ll be able to handle the comms and people organisation element for you.
Start and end your study with short user interviews
Opening interviews are a great way for you to reiterate your intentions, the purpose of the study and your expectations of the participants right at the start, and you’ll be able to capture a bit more detail about your subjects and build rapport. Closing interviews are a good way to hoover up any miscellaneous details, or clarify any entries you didn’t quite understand before you let your participants go at the end of the study.
Send out a study details pack
Create a detailed study pack including all the information about the project (dates, contact details, links etc). Include info about your work, who you are and why you’re running the study, and how much effort you have put into planning it (this can help reduce drop outs). A study pack is a great resource for participants to look at if they’re stuck or lost and can reduce the amount of questions you have to field (particularly if your study is running over weekend, when you may not be in the office to deal with any issues that arise).
Get creative with your collection method
Pen and paper diary entries are all but dead. They make it impossible to analyse as you first have to type up someone else’s badly written scrappy bits of paper, my advice is to avoid pen and paper like the plague! But there are plenty of other great ways to capture diary data. I’ve used WhatsApp a number of times in previous diary studies, its great as participants can send a message whenever they like as if they were messaging a friend, and include images or even voice notes!
Incentivise great work, incentivise a mid point
The offer of an additional incentive for those participants who send the ‘best’ diary entries is a magic way to get everyone to write full and detailed diaries. Consider giving an example of what a great diary entry looks like in your study pack and let participants know that if they consistently meet the criteria they’ll be entered into a prize draw to win an extra boost to their incentive package. It’s also useful to do a midway incentive to encourage participant to follow through to the end of the study.
Final thoughts
This post has been full of strange concepts (5 bums on a rugby post and the 5 whys!), but I have a final one to finish off.
P. P. P. P. P
Prior preparation prevents poor performance.
Plan your study well, think about who, what, where, when, why and how!