Impact Analysis

You will need to measure the impact your project has or has not made against your original intent.  Reflect on the social values and ethical principles you have or have not promoted.   Finally you will need to think about how to follow up so your sensing project can continue to create value for your city.

Summary

In the end, Crowd Sensing should be about co-creating social impact with the public. You need to show quantitative and qualitative results in the right ways to keep momentum, your stakeholders involved, and the public engaged.

Questions

Now is the time to reflect on your crowd sensing project.  Among the many questions and considerations that will help you at this stage here are three central themes. 

  • What was the impact on public space and peoples lives? 
  • Did I conduct my project in responsible and ethical ways? 
  • How can I improve upon my experience and continue or conclude?
See Questions

Resources

Whether your project comes to an end or continues, this is the time to think critically about the impact you have made with crowd sensing in your city.  Success or failure can be framed in many ways. 

  • Project Retrospective
  • Reflecting Social Impact 
  • Evaluating Ethical Performance
  • Follow-up Strategy
See Resources

Video Library

Visit the video library for expert interviews and advice about Impact and Analysis. 

In this section we’ll be speaking with experts:

  1. Ido Nap – Program Manager Sensing Netherlands Police
  2. Paul Manwaring – Founder IoT Living Lab and Co-founder CITIXL
Watch Videos

6.1 Questions

Making people aware and getting them involved to the point where they want to participate is difficult so you have to show quantitative and qualitative results in the right way to keep the public engaged. At this point there are also operational, tactical and strategic considerations that relate to accountability issues and follow-up.  For example re-establishing ownership, governance, escalation paths, roles, and responsibilities among other considerations such as: 

  • Can you continue by designing interventions and involving the public?
  • Are you measuring changes, results, and social impact? 
  • Are you being transparent and publishing the data and results with visualisations and engaging stories? 
  • Have you been speaking with the public to generate a “data plus dialogue”?
  • Are you communicating your project, creating awareness and accountability? 
  • How can I reflect on the impact I have made in the context of my project goals as well as in an ethical context taking into account social values? 
  • Is it possible to keep momentum and enable a continuous discussion? 
  • Who will now own this project and how will they be accountable?
  • Have you established a follow-up plan and strategy?

6.2 – Project Retrospective

Now is a good time to reflect on your experience, look at the original intent, review your goals and revisit the Decision Canvass you worked on in the beginning of your journey to responsible and ethical crowd sensing.  Project post-mortems and assessing lessons learned is important but think about the unique aspects of crowd sensing and your project in a broader sense.  It’s not just about measuring performance against KPIs, it’s also about how you did or did not achieve your goals in responsible and ethical ways.  Here are some suggestions to help you assess your project’s impact:

  • Revisit your original Decision Canvas with project stakeholders.
  • Think about Lessons Learned. Document and share your experience.
  • Measure the changes you have made with an impact analysis report to quantity the before and after of your project and any interventions you conducted. 
  • Publish the results and tell your story.  It is important to be transparent and share your results. 

No doubt your organisation will have its own way of reflecting on a project. We hope to add some value at this stage by emphasising the ethical aspects specifically related to crowd sensing projects.  Now would be a good time to (re) visit The Responsible Sensing Lab https://www.ams-institute.org/urban-challenges/urban-data-intelligence/responsible-sensing-lab/  and read the cases: Human Scan Car, Transparent Charging Station, and Camera Shutter. Three well documented projects you can refer to. 

 

6.2.1 Reflecting: Social and Ethical Impact

It is difficult to measure success in a crowd sensing project you may or may not have achieved the goals you established. You may have had some serious problems or surprising results but what is most important is understanding how your project impacted the public. Did you change perceptions or behaviour and what does this mean? After evaluating the quantitative results of your project you should consider engaging the public again to get some feedback and gather the qualitative data about social impact.  

 

.

6.2.2 Evaluating Ethical Performance 

Responsible and ethical crowd sensing as a practice is just now beginning to take shape. If we design and implement our projects keeping social values and ethical principles in mind we are pioneering important work. The impact we create in our cities is much more than solving problems that have to do with how people move in public spaces but how we approach solving these problems with privacy and dignity at the same time.

Now is a good time to hold an Ethical Performance Workshop with your project stakeholders. You can revisit the Applying an Ethical Framework and Measuring Ethical Principles in Step 3 (LINK) or visit some of the data ethics links below to help you think about how to evaluate your ethical performance:

https://gdpr-info.eu/issues/privacy-by-design/
https://tada.city
https://www.ams-institute.org/urban-challenges/urban-data-intelligence/responsible-sensing-lab/
https://citiesfordigitalrights.org
https://en.wikipedia.org/wiki/Value_sensitive_design
https://dataschool.nl/en/deda/app/

For a specific example of a TADA Data Ethics Workshop to evaluate ethical performance, download this report from the Recreational Crowd Study conducted by CITIXL in collaboration with Marineterrien Amsterdam, City of Amsterdam CTO office and TADA.

See the example workshop results:  TADA Recreational Space Study Workshop PDF

Project stakeholders should be evaluating the ethical performance for each of their experiments. Teams are free to define their own values and principles, but most importantly see how they can continually improve on responsible sensing during the project life cycle. In the illustration below, you see a comparison of how teams ranked themselves on a hypothetical research project (Based on the TADA ethical framework). 

.

Cafe owners using Computer Vision to measure social distancing on their terrace

Business district impact analysis of 3D Sensors in a street during lockdown

Museum managers using WiFi Sniffers for smart ticketing to improve waiting times

Health department using MM Waves to detect after hour activities of playgrounds

Campus security using Sound Sensors to classify sounds like guns or breaking glass

Facility managers using Motion Sensors to determine safe capacity of student assembly  

6.2.3 Follow-up

At this point there are also operational, tactical and strategic considerations that relate to accountability issues and follow-up.  For example re-establishing ownership, governance, escalation paths, roles, and responsibilities among other considerations.

In agile projects there are various approaches to continuous improvement.  Specific to responsible and ethical crowd sensing it is useful to map out your journey, consider what impact you have had, and determine a path forward or a productive conclusion. Here are some considerations:

  • Can you amplify the positive impact you have had with your project?
  • Is it possible to scale, duplicate, and/or distribute your project?
  • If you continue, how can you improve your project responsibly and ethically?
  • If you conclude your project, how can you and the public benefit from lessons learned? 

When we talk about agile development, design thinking, or experimentation in Living Labs it’s important to remember that these are all non-linear approaches to creative collaboration and conclusions are less important than applying lessons learned to improving your project or designing a better solution to test again. Here are some links to help you think about follow up so you can build on your experience and continue to create impact with your project. 

.

6.3 Video Library

At the end of this process, we suggest you revisit your original goals, evaluate your progress, and reflect on your experience. You can do this by applying the Ethical Framework example provided in Step 1. But for now, please take a few minutes to learn from personal experience and insights that will help you think about Measuring Impact for your crowd monitoring project.

All expert interviews ask these three questions:

What are the biggest challenges for crowd monitoring in public spaces?
What are some approaches to address these challenges?
What advice can you give innovators thinking about crowd monitoring?

Ido Nap

Program Manager Sensing Netherlands Police

Ido believes the challenge is in the way we interpret the results and how we follow up (with the public). Only with continuous measurement and feedback loops can we understand the consequences of our designs, interventions, and actions and how we can inform the public when we assess the impact of our project.

Paul Manwaring

Cofounder CITIXL & Founder IoT Living Lab

It’s important to look at the big picture and by using data for the common good we will create positive social impact. Paul reminds us that we need to make sure that our digital rights are protected in public spaces; that measuring impact is not just about getting “results”, but also about working together to reclaim our public spaces and protect the digital commons.