Towards Hybrid EEG Analysis: There is good evidence to support that human intelligence combined with artificial intelligence is superior to artificial intelligence alone when it comes to solving complex problems. The CrowdEEG team is working to develop a hybrid system of EEG analysis which combines and compares each of the following to fully-automated approaches:
- A general set of algorithms to decompose EEG analysis into a set of micro tasks
- The responses of expert EEG technicians
- The responses of non-expert crowd workers
Our eventual aim is to deploy and iterate our hybrid system in real-world hospital settings for the application of clinical EEG interpretation, and make freely available our prototype system for the use of health care organizations abroad.
The CrowdEEG team is currently conducting two ongoing studies in support of our overarching research aims:
- The Nature of Expert Disagreement
- Active Learning
We are also actively engaged in an ongoing collaborative project with Harvard Medical School:
3. The Bhutan Epilepsy Project
Read more about our ongoing studies and projects below:
1. The Nature of Expert Disagreement
Low inter-rater agreement has been deemed “the norm” in the interpretation of medical data[6]
When multiple trained experts disagree on how to interpret patient data, it isn’t obvious how this disagreement should be resolved, and thus such inter-subjective cases often remain ambiguous. A popular solution put forward to resolve ambiguity from exert disagreement is group deliberation, where those experts that disagree come together to adjudicate ambiguous cases and arrive at a consensus.
The CrowdEEG team has placed a special focus on mining expert disagreement in the medical domain, specifically in the context of medical time series analysis. Mike Schaekermann, the student investigator on the CrowdEEG Project, is currently conducting a study on the effects of implicit contextual information in multi-channel biosignal time series data on agreement rates between experts. The aim of the study is to identify the different types of information that trained experts evaluate implicitly to inform their classification decisions, with the greater goal of arriving at a better understanding of the causes, effects, properties, and perception of expert disagreement and adjudication processes in the context of medical time series analysis.
The interpretation of medical time series analysis commonly relies on visual analysis of subjective criteria. Sleep stage classification, an EEG interpretation task, is one such example, and the sub-domain in which we embed our study. Sleep stage classification is a special case because it is sequential in nature—the interpretation of one case will affect the interpretation of subsequent cases, and thus expert disagreement surrounding one case may result in a cascade of disagreements about the data that follows. The average agreement rate among experts in the context of sleep stage classification is 82.6%2.
In addition to the problem of expert disagreement, there is disagreement in the field about what exactly should be done with those cases where experts disagree. Some researchers view expert disagreement as a problem to be resolved; some view expert disagreement as something to be leveraged towards a useful end gaol. Our researchers take the latter approach.
In our study on the nature of expert disagreement, we deploy a web-based system of structured group deliberation in which expert sleep technologists are invited to participate in a sleep stage classification task: annotating and classifying a full-length EEG recording (i.e., a polysomnogram) into discrete sleep stages. Our expert participants score a single recording independently, and then are brought together remotely via our web platform to deliberate and adjudicate disagreement cases, providing rationale for their individual classification decisions based on evidence criteria. Here, the end goal is to arrive at an adjudicated hypnogram with a higher agreement rate than the original. In addition, the structured output of our system collected throughout the study will constitute a data set that can be used for the purposes of active learning.
2. Active Learning
The overall vision of the CrowdEEG project is to combine human and machine intelligence for the scalable and accurate analysis of human clinical EEG data.
In support of our overall goal to combine the joint effort of algorithms, experts, and crowd workers to make amenable the tasks of EEG interpretation to human-computational methods, active learning is an ongoing research study with the CrowdEEG project.
For this purpose, we have designed a computational framework which will selectively elicit feedback from clinical experts and non-expert crowds to train machine learning algorithms for highly accurate classifications of human clinical EEG data. Innovation and development of this platform is an ongoing concern, both for the purposes of our research, and for those of our collaborators. Namely, in the context of sleep stage classification and seizure detection.
3. The Bhutan Epilepsy Project
In joint collaboration with Harvard Medical School, the CrowdEEG team aims to integrate our algorithms with smartphone-based EEG recording devices for the purpose of remote EEG analysis.
In remote and/or resource-poor clinical settings, trained neurologists or EEG technicians are both rare and expensive. Thus, EEG recording and interpretation must occasionally be outsourced to outside experts for patient diagnosis.
To tackle this problem and help make EEG a viable and affordable diagnostic tool for clinical settings abroad, the multi-national venture, known as the Bhutan Epilepsy Project, was born. Specialized devices which allow for clinical EEG to be recorded via smartphone are being tested in under-resourced clinics in Bhutan. However, these smartphone-based devices currently do not allow for EEG interpretation.
One of our translational aims is to integrate our hybrid system of EEG analysis with the smartphone-based devices developed by the Bhutan Epilepsy Project to allow for both EEG recording and interpretation in remote, resource-poor clinical settings. Local non-experts (e.g., nurses) would work with the help of our algorithms running locally on the device to complete micro-tasks associated with EEG interpretation, and with the help of an internet connection, expert technicians can access the system remotely to complete high-level interpretation and classification tasks when queried. In addition to saving money, time, and expediting patient diagnosis, this will allow us to improve and iterate our system, along with improving the performance of non-expert EEG readers working in health care.
Visit the following links to read more about the Bhutan Epilepsy Project as a whole:
https://www.ilae.org/journals/epigraph/epigraph-vol-17-issue-3-fall-2015/the-bhutan-epilepsy-project
In the course of the CrowdEEG project as a whole, we plan to make publicly available a high-quality dataset of human clinical polysomnograms (i.e., multi-channel biosignal recordings of human subjects during sleep) to support research endeavours of other research groups in this field.
2. Penzel, T.; Zhang, X.; and Fietze, I. 2013. Inter-scorer reliability between sleep centers can teach us what to improve in the scoring rules. Journal of Clinical Sleep Medicine 9(1):81–87.
[6] Schaekermann, M., Law, E., Larson, K., & Lim, A. (2018). Expert Disagreement in Sequential Labeling: A Case Study on Adjudication in Medical Time Series Analysis. In 1st Workshop on Subjectivity, Ambiguity and Disagreement (SAD) in Crowdsourcing, at HCOMP 2018. Zurich, Switzerland.