A framework for rigorously evaluating digital mental health tools

Technology will play a vital role in addressing the worldwide need for better access to mental health services. And while new digital mental health tools are developed every day, decision makers struggle to determine which tools should be used and scaled with confidence in healthcare systems.

To help solve this problem, HRI has built a Framework to guide the development, evaluation, and regulation of top-quality digital mental health tools backed by solid science.

With funding from The RBC Foundation, HRI engaged faculty members from Harvard Medical School, including Yuri Quintana, Ph.D., Chief of the Division of Clinical Informatics at Beth Israel Deaconess Medical Center (BIDMC), and John Torous, M.D., Chief of the Division of Digital Psychiatry at BIDMC, to design the Framework. Drawing on input from a distinguished panel of Canadian and international experts in medicine and mobile health, the team produced a groundbreaking report entitled Framework for Evaluation of Mobile Apps for Youth Mental Health. While the report focuses on youth, the Framework is applicable to tools designed for adults.

The Framework was informed by a previous project led by Dr. Quintana, entitled Youth Mental Health Apps in the Digital Age: A Scoping Review of Trends and Evaluations. That project generated a comprehensive report, which explores current trends in youth mental health and looks more closely at several popular mental health apps. The report identifies strengths and limitations of available frameworks for evaluating apps and demonstrates a clear need for the science-backed Framework that has now been developed.

Who benefits from this Framework?

“This Framework will be of high value to consumers, healthcare providers, government leaders, as well as to those who design, evaluate, or invest in mental health apps,” says Dr. Quintana, an expert in clinical informatics and digital health services.

“Equally important, it will pave the way for improvements in regulations and policies related to mental health apps that can help guide how systems are selected, implemented, monitored, and evaluated. This work will help Canada and other countries develop a more scientifically informed process for strategic funding decisions and roadmaps for youth mental health needs.”

Click here to view the full report and framework.

HRI aims to move this work forward in partnership with other organizations interested in digital mental health. For more information on partnering with us to advance this important work, email

To receive updates on this work, join our email list:

Stay Connected Button

Follow us:


Building the foundation for recovery monitoring in Ontario

Ontario recently launched Roadmap to Wellness: A Plan to Build Ontario’s Mental Health and Addictions System. The plan provides a clear path forward to improve mental health and addiction services and calls for a standardized approach to measuring performance of the current system – a call echoed by both the Auditor General and Addictions and Mental Health Ontario.

HRI is bringing experts together to answer that call.

On March 12, HRI invited key stakeholders from the mental health and addiction sector across Ontario to an exploratory workshop. The meeting was a crucial first step to co-creating a vision and exploring opportunities to advance recovery monitoring in Ontario.

Participants included addiction treatment providers from across the province, and representatives from key organizations, including:

  • The Mental Health and Addictions Centre of Excellence
  • Addiction and Mental Health Ontario
  • CAMH’s Provincial Support Services Program
  • ConnexOntario, and
  • Ontario Health

The group explored the feasibility of a province-wide data system that would help Ontario answer vital questions about its services: Are people getting better? Will proposed changes to care actually make a difference? Are investments in addiction care paying off?

A performance-measurement system would also enable:

  • Evidence-based improvements to care that will help people get better, sooner
  • More efficient treatment services so more people can access care
  • Improved cost efficiency in government spending

A foundational framework created by HRI

To open the meeting, HRI scientists Dr. Jean Costello and Dr. Brian Rush provided an overview of HRI’s Recovery Journey Project. The Project is a recovery-monitoring system for addiction treatment developed and tested over the past five years in partnership with Homewood Health.

Discussions followed about how such a system could be built upon for broader application. The group identified outstanding needs, available supports for implementing a province-wide system, the alignment of this initiative with provincial priorities, and next steps to build capacity for a system-level approach.

The group referenced performance-measurement systems currently in place in other healthcare areas — such as cancer treatment – that have generated invaluable data to guide improvements to care and ultimately, to save lives.

HRI is now exploring existing projects relevant to this initiative and seeking collaborative opportunities to advance this work in measuring outcomes on a broad scale.

To receive updates on these efforts, join our email list:

Stay Connected Button

Follow us:


$1M awarded to HRI scientists studying new treatment for PTSD in first responders

Over the next three years, HRI scientists will receive nearly $1 million to support a study aimed at helping Canada’s public safety personnel (PSP).

A $990,000 grant from the Canadian Institutes of Health Research (CIHR) was awarded to Dr. Margaret McKinnon, Homewood Research Chair in Mental Health and Trauma, and Dr. Ruth Lanius, HRI Consulting Scientist, for their study exploring a new approach to treating PTSD in PSP.

PSP include paramedics, police, firefighters, nurses, correctional officers, and others working in emergency service and first response roles.

The study will test the utility of a cognitive therapy program, called Goal Management Training, in reducing the cognitive and emotional symptoms of PTSD, such as difficulty with memory, concentration and planning, as well as impulsivity and anger. The researchers will measure not only symptom changes among PSP receiving the treatment but also functional outcomes such as return to work and functioning in the family. They will also use MRI technology to examine changes in the brain structure and in brain function before and after treatment.

“The ability to capture very high-resolution images of the brain before and after treatment makes this study unique, and allows us to make the invisible wound of PTSD visible” says Dr. McKinnon.

“Not only can we measure how symptoms change in PSP with PTSD, but we can observe physical changes in the brain structure and function as a result of this treatment.”

A timely announcement

Funding for the study was announced as the COVID-19 pandemic took hold and Canada’s PSP were thrust into the spotlight.

“First responders and healthcare workers are dedicating their lives to society by working on the frontlines amid this global emergency,” says McKinnon.

“We know the pandemic will bring overwhelming trauma to many people working in the healthcare and public safety sectors at this time. It is very powerful to be able to offer hope to those who may be affected by PTSD and to be able to offer hope to our first responder and healthcare communities impacted by COVID-19.”

Focus on real-world application

To date, studies on trauma among PSP have focused on reducing the emotional symptoms of PTSD, such as hypervigilance and anxiety; however, many people are left with cognitive deficits that severely impact day-to-day functioning. These cognitive and emotional symptoms interact leading to difficulties such as poor planning, impulsivity, and problems with anger management”

“Very few studies have actually looked at real-world functioning – for example, the ability to return to work,” says Dr. Ruth Lanius, co-investigator and a Professor at Western University.

“If an individual struggles to process new information, maintain focus, or make a timely decision, their chances of returning to work and normal life are slim to none. We want to find treatments that help to close that gap – new approaches that help people get their quality of life back.”

Generous funding from Military Casualty Support Foundation, the Cowan Foundation, and the Bickell Foundation has been critical to the development of this program of research. Additional funding from the Worker’s Safety Insurance Board of Ontario will allow the team to look at various formats for delivery of GMT, and associated adjunctive treatments.

For future updates on this story, join our email list:

Stay Connected Button

Follow us:


Contributing to national discussions about COVID-19 and mental health

Protecting the psychological well-being of our nation during the COVID-19 pandemic has been a priority for anyone working in the field of mental health and addiction.

As the novel coronavirus spread throughout our communities, so too did our anxiety about the future. Prolonged isolation, the loss of loved ones, and financial hardship have placed many at risk of mental illness. For healthcare workers, another threat persists – the threat of moral injury.

Recently, HRI scientists have added to important discussions across Canada about the trauma-related impacts of COVID-19:

To follow our experts as the COVID-19 pandemic continues, join our email list or follow us today:Stay Connected Button

Follow us:


Moral dilemmas and traumatic stress during COVID-19

The COVID-19 pandemic places many people at risk for exposure to potentially traumatic events. Some of us have already faced the loss of a loved one, long-term isolation from family and friends, or perhaps a near-death experience ourselves. But this pandemic brings another type of trauma-related risk: the risk of sustaining a moral injury.

What is moral injury?

A moral injury may occur when a person witnesses, fails to prevent, or engages in an event that violates their personal moral beliefs. Exposure to morally injurious situations has been linked to post-traumatic stress disorder and can bring profound shame, anguish, and guilt that severely disrupts one’s quality of life.

In the context of COVID-19, a moral injury could occur after an individual faces an ethical dilemma that results in harm to another or to oneself. For example:

  • A doctor who must make difficult decisions about how to deploy limited life-saving resources, such as ventilators
  • A healthcare worker who may put their own children at risk by continuing to work
  • An individual who cannot be with a terminally ill loved one due to self-illness or facility restrictions
  • An undiagnosed person who inadvertently infects vulnerable residents in a nursing home

These are wrenching situations no one wants to face.  And they can leave us scarred.

What to do if you experience traumatic events

“The first thing to do if you are exposed to a traumatic event is to talk about it,” says Dr. Margaret McKinnon, an HRI-supported scientist and trauma expert from McMaster University.

Dr. McKinnon serves as Homewood Research Chair in Mental Health and Trauma and is leading important research related to moral injury.

“Talk to your colleagues or close personal network. Failing to acknowledge a morally injurious event can make matters worse, so share your experiences and support one another. The next thing to do is to practice self-compassion and self-care as you move through these challenging times. Many virtual tools are now available to support our resilience and mental health.”

And finally, if you experience persistent and overwhelming feelings related to a traumatic experience that are severely disrupting your quality of life, talk to a mental health professional.

“Help is available, and protecting our mental wellbeing is of the utmost importance,” says McKinnon.

Mental health resources during COVID-19

If you are an Ontario healthcare worker, you can access free phone therapy sessions provided by licensed mental health workers who are volunteering their services during the COVID-19 pandemic.

For anyone experiencing mental health concerns, many online resources are available, including:

How to practice self-compassion and self-care

  • Accept your feelings without judgement. It is normal to feel negative emotions during uncertain times or after experiencing a traumatic event. Name your feelings and be gentle with yourself as you sit with them and process them.
  • Do your best to maintain healthy lifestyle habits. If you are able, maintain a regular sleep, exercise and meal schedule. Take breaks, communicate with your peers and family, and use mental health tools that work for you. A list of resources is available below.
  • Practice mindfulness techniques. A variety of mindfulness tools exist online. Apps such as Calm and Insight Timer provide guided recordings, breathing exercises, and other tools to stay grounded in the here and now.
  • Continue doing all you can. Follow public health recommendations to slow the spread of COVID-19. Stay informed via reputable news sources (but don’t be afraid to take a media break if you need one.) Remind yourself of the positive outcomes you contribute to each day.

Follow HRI or subscribe for the latest news:

Stay Connected Button

Follow us:


Mobilizing experts to address sexual misconduct in the military

In October 2019, 50 stakeholders from across Canada attended a workshop at the annual Canadian Institute for Military and Veteran Health Research Forum. The workshop focused on the psychological consequences of sexual misconduct during military service.

Researchers, policy makers, military members, veterans, and clinicians convened to explore the relation between exposure to sexual misconduct during military service and the onset of moral injury among female military members and veterans.

To mobilize action on these discussions, HRI hosted a symposium on December 3, 2019. Funded by the Government of Canada’s Defence Engagement Program, the symposium brought Canadian military and non-military experts together with a common goal of supporting the Canadian Armed Forces in addressing sexual misconduct.

The role of moral injury

Canada patch flags on soldiers arm

Work at HRI is exploring whether exposure to sexual misconduct in military environments is related to the onset of moral injury. For example, sexual misconduct may result in moral injury when an incident involves a perceived betrayal by those within a circle of trust or by an authority figure.

Similarly, a military member could be morally wounded by witnessing – but being unable to stop – a colleague from sexually harassing another military member.

The interplay between sexual misconduct and moral injury is complex and requires further investigation. What is known, however, is that military members affected by sexual misconduct in the workplace are at increased risk of post-traumatic stress disorder, anxiety, depression, and suicide.

Moving research, prevention, and intervention forward

During the December 3 symposium, discussions were diverse but focused on barriers, resources, and actionable items to affect change, including:

  • The rise of the internet and social media in skewing views of normal sexual behaviour and consent
  • Optimizing available resources for affected members, including the Sexual Misconduct Response Centre
  • Celebrating current training and education initiatives, including Operation Honour
  • Expanding education focused on emotional intelligence, boundary setting, assertiveness, bystander roles, and moral dilemmas
  • Understanding the needs of LGBTQ+ members, indigenous members, and members with disabilities
  • Improving screening tools to identify affected members or those at high risk of perpetrating sexual misconduct
  • Improving treatment and intervention approaches to help affected members
  • Vital research needs, including the need to explore where best to target resources, the link between sexual misconduct and moral injury, how to facilitate culture change among all levels and ranks, the value of peer support, and the need to expand research to include male, LGBTQ+ and non-binary members

Group of symposium attendees, December 3, 2019The symposium was attended by 18 representatives of leading universities, research centres, and treatment providers, as well as the Department of National Defence, Veteran’s Affairs Canada, Servicewomen’s Salute, and the Canadian Institute for Military and Veteran Health Research.

Dr. Margaret McKinnon, Homewood Research Chair in Mental Health and Trauma, is the lead investigator on this research. Key collaborators include:

Subscribe to HRI and follow us for the latest updates on this project:

Stay Connected Button

Follow us:


International study explores how AA contributes to recovery

Treatment of alcohol use disorder is multi-faceted and often includes participation in mutual-support groups, the oldest and largest of which is Alcoholics Anonymous (AA).

Although frequent AA meeting attendance is associated with greater abstinence from alcohol, there is little understanding of the specific mechanisms by which AA benefits people. In other words, we know that AA works, but we aren’t sure how.

Recent studies suggest that AA may prompt important behavioural changes by positively impacting impulsivity and social networks. New research from the Peter Boris Centre for Addictions Research will take a deeper dive into this theory. The study will be led by Dr. James MacKillop, Peter Boris Chair in Addictions and HRI Senior Scientist, and Dr. John Kelly, Elizabeth R. Spallin Professor of Psychiatry in Addiction Medicine at Harvard Medical School. Their team will examine impulsivity and social networks to see how these mechanisms drive recovery success in AA attendees.

The process

Researchers will follow people seeking treatment for alcohol use disorder in both inpatient and outpatient settings. The study will involve participants from multiple treatment programs based in Guelph and Hamilton, Ontario, and Boston, Massachusetts.

Participants will be assessed when they enter treatment, at 4-6 weeks into treatment, and at four more time points over a one-year period. Assessments will ask about alcohol use, personality traits, family and peer behaviour, and general personal and health information. Novel approaches will be employed to measure impulsive behaviour and analyze social networks, allowing researchers to determine how these mechanisms relate to abstinence and other markers of recovery.

Generating new knowledge to help recovery

This study will shed light on the psychological and social mechanisms of recovery and how AA activates these mechanisms. Researchers also hope to discover factors that could predict positive or negative experiences with AA. Clinicians and scientists may also use findings to identify therapeutic targets for people receiving treatment in an effort to boost recovery success.

Funding and collaboration

This study, entitled Investigating Impulsivity and Social Network Changes as Novel Mechanisms of Behavioural Change for Alcoholics Anonymous’ (AA) Positive Effects, is funded by the US National Institute of Alcohol Abuse and Alcoholism (NIAAA).

Collaborators on this project include:

  • John Kelly (Massachusetts General Hospital, Harvard Medical School)
  • Robert Stout (PIRE Decision Sciences Institute, Providence, RI USA)
  • Allan Clifton (Vassar College, New York, USA)

Would you like to learn more about mental health and addiction research in Canada?

Stay Connected Button

Follow us:


Evidence 101: Q&A with an expert

Dr. David Streiner has attempted to retire three times, but this biostatistics expert is in high demand.

Dr. David Streiner headshot

Dr. David Streiner

Having trained as a clinical psychologist more than 50 years ago, Dr. Streiner is now Professor Emeritus in both the Department of Psychiatry and Behavioural Neurosciences and the Department of Clinical Epidemiology and Biostatistics at McMaster University, and a Professor in the Department of Psychiatry at the University of Toronto.

His research expertise is sought by scientists, faculty, residents, and students alike. A star behind the scenes, Dr. Streiner frequently consults with HRI scientists and trainees on everything from study design to research analysis.

In this question-and-answer feature, Dr. Streiner provides clarity about some of the most common questions we receive at HRI about evidence.

What does “evidence” mean in the context of healthcare?

Evidence is the available body of facts that healthcare providers can draw from to best treat patients. The most common method of obtaining healthcare evidence today is through research studies. Prior to the rise of scientific healthcare research, evidence was based primarily on clinical expertise and opinion.

What is “evidence-based practice” in healthcare?

The term “evidence-based practice” originated from Dr. David Sackett, an American-Canadian physician who defined the term as “…the conscientious, explicit, and judicious use of the current best evidence in making decisions about the care of patients.”

So when using an evidence-based approach, one would base healthcare decisions solely on evidence from scientific research.

How is “evidence-informed practice” different from “evidence-based practice”?

Early critics of evidence-based practice argued that its definition limited medicine to a “cookbook approach” that fails to consider each patient’s unique values and preferences, and downplays the clinician’s experience and knowledge of the patient. A new term began to circulate, which encompassed a more person-centred approach:  evidence-informed practice.

Over time, the concept of evidence-based practice has evolved to include not only scientific research but also clinical expertise and the patient perspective brought to light by proponents of evidence-informed practice. Today, we see both terms used interchangeably.

How do we create evidence in healthcare?

First of all, we need research. Without research, we have no evidence. But to do research, we also need participants. We know that research plays a central role in designing and evaluating new treatments, but we often forget that patients and families play a central role in the research itself.

When answering research questions, we use a variety of methodologies. For example, if we want to know about the prevalence of a medical condition in a population, a cross-sectional study would be appropriate. On the other hand, if we want to find out whether a proposed treatment would help people with a specific condition, a randomized controlled trial would be used. At the end of the day, the research methodology depends on the question being asked.

The following infographic outlines some of the most common research approaches used at HRI:

Infographic explaining several types of research studies