The push towards automated assistance in sleep staging and seizure detection.

Limitations of EEG:

Electroencephalography (EEG) is a measurement technique crucial to the diagnosis of brain pathology, where changes in underlying brain activity (i.e. brain-generated electrical potentials) reflect illness or disorder.

Despite its diagnostic power, EEG has several limitations within a clinical context:

1. Time — EEG is a continuous measurement technique, a single recording of which can be anywhere from 8 hours (e.g., in the context of sleep) to days (e.g., in the context of seizure detection) in duration. Analyzing these extensive recordings is a time-intensive endeavour, and delays patient diagnosis.
2. Expert Availability & Cost — Clinical experts like neurologists or EEG technicians are rare and expensive. In clinics that that cannot afford to staff regular EEG technologists, EEG recordings must be outsourced for analysis, further delaying patient diagnosis.
3. Inter-Rater Reliability — EEG interpretation is a laborious task based on visual analysis of subjective criteria, the nature of which leads to variability and disagreement between readers. Agreement between experts is important for the reliability of EEG as a diagnostic tool.

These limitations point to the need for for a method of EEG interpretation that is a) fast and scalable, b) not heavily dependent on highly trained experts, and c) leverages variable interpretations to increase inter-rater reliability.

Barriers to Automated EEG Analysis:

All of the considerations above have motivated efforts to develop assisted technologies for EEG analysis. However, several factors contribute to the computational difficulty of EEG interpretation under standard machine learning methods:

• Feature detection — Distinguishing features of an EEG recording are typically brief, rare, and visually subjective
• Classification — Even when relevant features are identified, the classification of segments/epochs in normal or abnormal states often depend on ambiguous criteria
• Noise & artifact — In order to accurately classify an EEG recording, noise and artifact must be differentiated from the signal of interest
• Data distribution — Especially in the context of seizure detection, EEG data is often both highly unbalanced and non-stationary

While there have been many past attempts at automated methods of EEG interpretation—especially for the purposes of seizure detection, the performance of these algorithms has been historically poor, falling short of that of an expert. Moreover, these automated algorithms are typically performed on EEG data that is artificially clean (i.e., artifact-free), and not what would be seen from patients a real-world clinical setting.

An Alternate to Automation: Crowd Work

In a 2014 paper¹ published in Nature Methods, Warby et al. report on a study in which a group on non-expert crowd workers performed a sleep-spindle detection task with an EEG recording. This study found that the aggregate performance of crowd workers not only exceeded that of existing automated algorithms, but approached that of trained experts in the field. In a separate endeavour, when the performance of trained experts was aggregated and compared to a benchmark, the end result exceeded the performance of any one expert alone.

While limited in scope, these findings demonstrate the potential for a new method of EEG interpretation.

A hybrid system of EEG interpretation

For human experts, EEG interpretation is laborious and time-intensive. For automated algorithms, EEG interpretation can be made scalable and efficient, but at the cost of performance, and at the mercy of several design challenges. When groups of crowdworkers come together, they can exceed the performance of modern algorithms. These three insights point to a hybrid approach to EEG interpretation: human computation.

Human computation is a fledgling field in computer science in which human intelligence is combined with artificial intelligence (AI) in order to harness the strengths and overcome the weaknesses of both for the purposes of complex tasks. Typically, micro-tasks amenable to AI are automated, while humans work on those that are not—and in the end, the output from both is aggregated to solve a problem.

This approach has proven successful with many complex tasks in a variety of scientific fields: machine translation, object recognition, and protein folding. These are all tasks where automation is desirable, but intractable by AI alone—much like EEG interpretation in the context of sleep stage classification and seizure detection.

A prime example of human computation is crowdsourcing — where groups of non-experts perform micro tasks through a computational framework which in the aggregate amount to a solution to a complex tasks. Studies like the one in Nature Methods described above suggest that a carefully designed system which efficiently integrates the input of crowd workers, augmented by AI algorithms, can approach or exceed that of fully automated approaches, and experts alone.

This is precisely what we aim to achieve with CrowdEEG as a research project and application: combining human and machine intelligence—the joint efforts of experts, non-experts, and algorithms—for the rapid, scalable, and accurate analysis of human clinical EEG data.

1. Warby, S. C., Wendt, S. L., Welinder, P., Munk, E. G. S., Carrillo, O., Sorensen, H. B. D., … Mignot, E. (2014). Sleep-spindle detection: crowdsourcing and evaluating performance of experts, non-experts and automated methods. Nature Methods, 11(4), 385–392. http://doi.org/10.1038/nmeth.2855