Questions & Answers – DigiCivics @ APL

What to consider in preparing usability tests?

Towards the end of a design project, we prepare prototypes and product mock-ups and use those to gather feedback from potential users. In preparing for usability tests of prototypes and user interface designs, the following pointers may be useful to consider.

Focus, focus, focus: What’s the most important aspect of your story board to prototype? It’s likely not important to test processes such as sign-ups and log-ins. You may assume that the user has already logged in, they are already at the point of use, where the part of the action states you want to test. Remember the structure of the five-step interview, before you show your prototype, provide a little introduction, e.g., by saying “Imagine you are in the following situation…” (develop context) Then introduce your app.
Clarify the user path: As with story boards, think about the sequence of actions you intend your users to follow as they use the prototype (user paths). How can you design for this sequence? For interface design consider, e.g., appropriate navigation, buttons that give away what to do next, explanatory text (sometimes called ‘copy’) that indicates what users are to do…. Ask yourself what are the five key steps that users need to go through to complete the experience you design for. You can also look at the project repository on the Marvel site, where there are many great examples.
Make it real: The tricky bit of usability testing with the kinds of prototypes we have is to make users believe they interact with the real thing. So consider how you deliver your prototype (e.g. on a tablet?) and what levels and kinds of interactions the prototype efforts users to do.
Speaking aloud: Remember to remind users to speak aloud when you observe their use of the prototype. The ‘speak aloud’ method is a key method in usability testing. Before you give any prompts, hints, or tips, let the user explore your prototype and play with it. Observe what they do and listen what they say. Ask them to things like “What do you think you should do here on this page?” or “What do you wish you could do on this page but can’t?”.
Document and capture: Usability tests are all about testing whether your app works as intended. Keep notes what users do, where they struggle, where they get lost. Remember to make arrangements for how you document the user interactions. For example, in one group we discussed the possibility to develop a template to record observations. This template would list the different pages of your prototype and offer a field to capture written notes. Usability tests also usually depend on protocol analysis, that is alike to a transcript of all statements made by the participant. It would be very good to capture any insightful statements users make.
End on a high note: Do a brief debrief. Have a set of questions read to try sum up the users’ experience and overall impression of the prototype. Always great is the question “If you had a magic want, what three things would you change or improve about this app?” OR “How useful did you find this app” Explore general statements what was useful and what wasn’t, and if the app was usable or not.

What were the team tasks per week again?

A core part of your progress on your design projects is to keep design log with regular reflective entries on your progress. Entries for your design log will also help you to fulfil one of the two major assessment components: a final version of the design log, an edited collection of all the design log entries you have been making. This article provides an overview of the prompts we set each week for you to use as basis of design log entries. But the design log is yours, so we strongly encourage you to write additional entries if and when needed. Ps: If you are unsure what makes a good design log entry, have a look here.

Note: the reflective prompts outlined for each week are due within one week after they were posted; for marking purposes, a submission within two weeks will count.

Schedule for 16/17

Please note that prompts may be updated slightly in response to our overall progress.

Week 1 (for ± 11th Oct)

TEAM INTRODUCTION: Briefly introduce your team, the members and what you think brings you together around this project.
DISCUSS YOUR BRIEF: describe your goals/expectations for the client meeting; what ‘digital civics’ might your project involve and what challenges do you see in relating to those civic groupings? (feel free to relate to the two pieces of reading for the first lecture)

Week 2 (due ± 18th Oct)

USER MAPPING: With the help of the client, prepare a map that depicts a process and indicate what activities your project stakeholders engage in (or not) as part of this. Notice tensions, conflicts (in resources or views).
Goals and aims: What are your aims and in response what questions to do you plan to cover?
User interview & research plan: Who do you involved in your user research and what activities might you do? What other documents / evidence might you consider? (based on clients’ advice)

Week 3 (due ± 25th Oct)

Briefly note what evidence and interviews you planned for
As you do your user interviews, write a blog post on high level insights from for your user interviews. You can subdivide this task amongst members in your team.
Documentation: Consider how you planned for documenting insights from your interviews? (e.g. a list of user stories?)

Week 4 (due ± 1st Nov)

Revisit your goals: Have they changed and how?
Report on activity: In your team, individually collate a list of products/services/devices that may serve as inspiration for your projects. Then can come from the literature or the Internet. Each member of the team should select two examples and say (1) why selected, (2) how it provides inspiration for the project. In the team, discuss the examples you came up with and note which features about interesting.
What examples have you discussed and why?
What of these examples has worked well and serves as inspiration for your project?

Week 5 (due 8th Nov)

Sketch out concept ideas: Individually develop a set of alternatives. Describe your set of alternative concept ideas and discuss the potential weaknesses and limitations of each. Finally settle on a final concept you’ll go on to prototype. Discuss why and how you choose it?

Week 6 (due 15th Nov)

Write a blog post: Develop a story board for your idea / concept (also covered in our seminar for this week) and document your progress by writing a blog post about it (about 300 words)
Helpful reading: Truong, K., Hayes, G., & Abowd, G. D. (2006). Storyboarding: An Empirical Determination of Best Practices and Effective Guidelines (pp. 1–10). Presented at the DIS.

Week 7 (due 22nd Nov)

For this week (until 22nd November), I encourage you to think and write an entry (200 — 400 words) about either of the following:

Reflect on the discussion with Joanna & Matthias and your story board designs, what key concerns did they mention in the deployment of their applications that might be relevant to your —> What problems do you need to address if you wanted to deploy your app with your project client? (approx. 300 words).
Review the resources for prototyping mentioned in the module guide (section “Resources bank for your design sprint”) and write a post about what kind of tools you want to use for designing the prototype app / service? Referring to your story board, describe what you are going to build. Bare in mind you’ll present your eventual prototype to user representatives in week 9/10 to gather feedback.

Week 8 (due 29th Nov)

Prototyping: What did you take away from this weeks lecture about prototyping? What aspect of your story board will you prototype and why? What materials, software, techniques do you plan to use to build your prototype? What level of fidelity do you aim to achieve with your prototype? Document your prototype designs to demonstrate how you successively refined your prototype to arrive at the final result.

Week 9 (due 6th Nov)

User test preparation: Discuss how you prepare for user tests in the next week. Where and how will you run your user interviews? What questions do you answer yourself and how do you plan to evaluate the user responses to your designs?

Week 10 (anytime during week 11)

User tests: Who were your participants and which audiences do they represent?How did they respond to the prototypes? What interesting statements did they make that help answer your project goals / questions? Did they use the prototype as intended by you? What did they note for further improvement? What would you do differently if you were to do the user test again?

Week 11 (due as part of the final design log, 13th Jan)

Next steps: Discuss where you think your project could go next? What are the next steps for your proposition? Based on the user tests what do you think needs to change in your proposition if you were to make it a successful product?

How to plan for your user interviews?

So you have set yourself some goals for your project (e.g. reduced air pollution on Gosforth highstreet, more people take up volunteering in Walker, etc) and developed some questions you like your project to answer to (e.g. Can air quality data, displayed at street level, encourage behaviour change in pedestrians are car drivers?)… and you have begun to map out stakeholders, and key activities those are involved in as relevant to your challenge… So what’s next?

For our projects, we do assume that nobody knows everything, knowledge is distributed. Somebody knows more about the local area, somebody may know aspects of technological interventions that might work, etc, etc. As suggested in the lecture, we approach our designs with the idea that the participants listed in our briefs are experts in their own lifes, and that their views in relation to our projects are interesting and important (see Halskov et al., 2015).

So one way to approach this is to say, given our goal, how might our user informants (listed on our brief) help us collect the insight required to develop better idea of the priorities, needs, expectations that they may place towards our projects. At this point, we are not yet so focused on specific solutions or approaches, we still keep an open mind and let the users we involve speak.

Preparing your interviews

So with your goals in mind (e.g. reduced air pollution on Gosforth highstreet, more people take up volunteering in Walker, etc), prepare your interviews around the following points.

What might the person tell you about?: Who is the person you are speaking to and which organisation they belong to? Note down a few points: e.g. community organiser in Walker, likely interests in x, y, z.
Do you prepare to bring some prompt materials? For example, you could bring a map of the area; flip chart paper and post-its are good to capture notes during the meeting.
Decide who is going to ask questions?: If you meet, consider who is going to lead and ask the questions? Decide on who is going to take notes.
What do you plan to cover and perhaps in which order?
- Start with an introduction: Introduce your project and indicate what you wish individuals to discuss.
- Invite your participant to introduce themselves: Why is he/she interested in the project and how does he/she relate to the, e.g., client & other users?
- Given them the opportunity to express challenges relating to your project: What key concerns do they share / talk about?
- Consider taking notes into opportunities: The sprint book suggests how might we questions (e.g. how might we reduce car traffic on Gosforth highstreet).
Thank your participant for their time
Sign the consent form: Bring with you a copy of the consent form template (on Blackboard shortly)

Example interview schedule

This morning, I am meeting with XXXX (Organisation B) and XXXX (from Newcastle City Council). The meeting will be to discuss the accessibility technology that we are working on with which accessibility concerns in public spaces will be automatically logged. Below is a brief plan for the interview.

Item	What questions
Introduction to us	Thanks for inviting us, I am Sebastian (lecturer at the planning school) and this here is XXXX (PhD at the OpenLab, part of the school of computing). We have won some funding to develop and test a system to collate feedback from elderly wheel chair users. Our goal is to explore alternative measures of accessibility that compliment council’s work. We are therefore understand your interested in this project as well as how accessibility concerns are presently raised.
Discussion of XXXX’s work	To start, please tell us a little more about your day-to-day work and what it involves. Could you tell us how you presently assess the accessibility of spaces and any challenges? Are there examples of ‘inaccessibility’ that are particularly difficult to identify? And in which ways do you presently hear from or interact with mobile impaired and other disadvantaged groups?
Discussion of the organisation	Could you briefly explain what your organisation does in the local area? How does it interface with the council officers?
Next steps	Thank you for your feedback.

After your interview

After your interview, share and review your notes, and write up some high-level interview notes. What was discussed? What challenges (pain points) does the participant face that the design project can or cannot address? What needs does your participant discuss that could be translated into requirements for your design concept?

Source: Halskov, K., & Hansen, N. B. (2015). The diversity of participatory design research practice at PDC 2002–2012. Journal of Human Computer Studies, 74(C), 81–92.

What makes a good design log entry?

On this module, we use an online reflective log to share stories, notes, ideas coming from the success and failures of our design practice (across teams). This post gives some guidance on how to evaluate your project activities through a series of blog entries.

Reflective writing helps your design projects in three ways (Zubizarreta, 2009). First, it enables you to document and evidence actions and activities. Second, it makes it easy to support feedback, mentoring, and collaboration. Finally, it helps to learn from success and failure, develop insight into why those occurred. Over time, it helps to make sense of complex issues. As you make entries over time, you capture observation through images and document notes, you’ll help revise your insight and reflection. So do review older posts and refer to those in newer entries for example by commenting on new insights gained.

Zubizarreta (2009, p. 46) mentions a number of aspects to consider for log entries:

Dimensions	Comments
Reflective depth	Logs often focus on future improvements and change: to learn from an action, indicate what improvement is desirable and critically assess why and how! “What change was necessary?”, “What process did we use to facilitate change?”, “Why was the change desirable?”
Source of evidence	(1) “Information from self” (teams’ set goals, earlier reflective entries); (2) “Information from others” (reflection on feedback from peers, the tech advisor, lecturer etc); (3) “Products of learning” (a note, or summary of, an achievement produced; interviews, requirements gathered, for instance).
Evidence	(1) From field notes: images, videos; (2) From feedback: emails, classroom discussions; (3) From research: concept maps, references to published material
Consistency	Posts should be of sufficient length to make their point, so at last a substantial paragraph in length; and an indicative entry per week would be around 500 words at least. Referencing to theory, literature, activities is desirable (see point ‘evidence’).

As a general note, try to start each log entry with a brief introduction (to give a bit of context to the post as a whole). This will help the reader understand where you are coming from; and you may also discover that your posts connections to reflections from previous weeks.

An example of a good reflective log entry can be found here: https://blogs.ncl.ac.uk/digicivics-at-apl/2017/10/25/2-statements-of-community-consultation-meeting-with-the-stakeholders/

Sources for follow-up: Zubizarreta (2009). The Learning Portfolio. John Wiley & Sons

What is a story board?

Design processes can be messy and unstructured and that is particularly so in the fast-paced digital industries, where requirements, user expectations, technologies may turn over quickly. Good product design rests on the foundation of user-centred design, which means it always keeps sight of the users’ wishes, needs, demands. User-centred design is to design for the user. It is true that technological specifications may influence the design space, but in this module we are looking at designing bespoke interventions where this is not so much the case.

Story boards can be seen as part of the preparation work necessary before starting on any detailed designs or prototypes. Story boards are telling a story of how this digital product of yours will eventually be used. However, it also outlines a rational for why it would be used. Story boards depend on articulation of the use context, so where and why the practice you encourage is performed. Then, story boards detail a story of use. Here, a story is comprised of a sequence of events of use that coherently explain what practices happing when and in which order. Above all, stories assume the interactions in the story have a purpose for the user, a desired outcome without which the story would be incomplete.

Below is an excellent example from a student team on TCP2031 2016/17. Please note that the story board will develop alongside to your creative proposition and it may thus be updated, refined, polished, until you arrive at a final version.

What are design briefs?

The initial seed for your project, if you will, is a written ‘design brief’. Design briefs are common in the industry and stipulate a challenge, a range of constraints applying to the challenge, potential stakeholders, a bit of the back story to the project, perhaps also information on the socio-economical context giving relevance to your challenge. The brief is designed to help you get started. It will come with a number of reference documents and resources to firm your understanding.

In advance to the module start, I will have been working hard to establish and elaborate project briefs that are sufficiently bounded. The briefs will likely contain the following information: The project partner, project description, list of other stakeholders, information on your mentor, key datasets, a statement on the ideal outcome (from partner’s perspective), a set of keywords, and related files.

How will you be paired to briefs? In pairing projects and challenge partners with you, I make an effort to consider your topic interests in advance. A survey will be sent around to the students registered on the module to establish interests along, for example, the following topics: cycling, mobility, public health, air pollution, open data, data stores, planning, planning applications, engagement.

Design sprints

What is design? Defining ‘design’ is not straight forward. At present, in the tech, it is related to addressing of complex problems devoid of immediate clear solutions. What has been termed as ‘wicket problems’ (Rittel & Webber, 1973) is presently encountered in practice and theory and in this module relates to the appropriation of digital technologies within public services. Both the appropriation as well as ‘public services’ are complex matters in their own right. To address this, part of the design ethos, as I understand, appreciates no solution can ever be perfect. As we explore a solution, we explore the problem, resulting in learning about each. Design is about failing fast, testing small interventions, seeing if they work. That’s why design is about ‘action and change’ (Baskerville & Myers, 2014). In the tech industry, design has also been associated with aesthetics, a sort of appreciation of human needs, both, in terms of the usability of features and how they are presented. That’s how, digital ‘interventions’ should not result in artefacts that irritate but thought through concepts informed by context and therefore ready to blend in (Ackerman, 2000). However, we can recognize that design is ‘technicist’ but rather a philosophy, a way of doing, and approach, sometimes an outcome.

What is a ‘sprint’? Let’s move on to ‘sprints’. It sounds very sporty and there’s a reason for it. For this module, a ‘sprint’ sets the pace and the content of activities. In my view, a sprint is a set of (design) activities delimited by a problem question that we subsequently seek to address through development and testing of a solution concept. A sprint presupposes setting a structure, assembling a design context, and an eventual means of testing propositions developed. This module adapts “Google Design sprints” into a method for teaching digital interventions in public services. Usually design sprints are comprised of five days with each day reserved to a specific activity for a team of designers. Those include: (1) background research and requirements gathering; (2) appraisal of options and selection of one; (3) refinement of selected option as a concept; (4) visual articulation of the user story of the concept; (5) eventual testing of the concept on users.

How do we use the ‘design sprint’ throughout this module? In this module we will stretch the usual five-day schedule for a design sprint over the 12 weeks of the semester. Within a team of four or five peers, you will be paired with a partner from outside of the university. The partner, for example in local government or from a community group, will set a ‘challenge’ for your team, something they presently grapple with, where they think a well-informed digital intervention may be useful. Together with your team, you will then be introduced to a set of activities that present a full design sprint. At the end of term, you will present a concept you explored, researched, and ideally tested back to your challenge partner. For this you will be supported by your peers in the team, a structured reflective blog, and at least one mentor from APL or the School of Computing.

How does this relate to urbanism? The rapid diffusion of digital media and other information communication technologies has influence across all sectors of society. In the past couple of years, with other contextual factors, such as austerity, the first generation of ‘digital natives’ going into work, and generally advancing pace of change, the public sector faces many challenges that call for innovation and revision of established service delivery. At OpenLab’s Centre for Digital Civics, which this module is linked to, Newcastle University trains a new breed of interdisciplinary researchers and practitioners that can answer to the challenges ahead. In terms of urban planning and architecture, the influences relate back to our discipline, requiring a change in both the content of planning education as well as its delivery.