A device that measures several of the physiological responses accompanying emotion.

13.6 Examples

Despite the challenges, numerous HCI researchers have used physiological data to observe user interactions in ways that would not otherwise be possible. An examination of some of these studies indicates the common theme of using these techniques to record real-time observations of a task in progress, as opposed to subjective, posttest response.

A study of cognitive load and multimodal interfaces used three different traffic control interfaces with three different task complexity levels to investigate the possibility of using galvanic skin response (GSR) to measure cognitive load. Participants used gesture-based, speech-based, or multimodal (speech and gesture) interfaces to complete tasks. Initial analysis of data from five participants indicated that average response levels were lowest for the multimodal interface, followed by speech and then gesture interfaces. For all three interfaces, the total response increased with task complexity. This was interpreted as providing evidence for the utility of using GSR to indicate cognitive loads. Analysis of specific recordings found GSR peaks to be correlated with stressful or frustrating events, with responses decreasing over time. Peaks were also correlated with major events that were thought to be cognitively challenging, including reading instructions and competing tasks (Shi et al., 2007).

Another study used both galvanic skin response (GSR) and blood-volume pressure (BVP) to measure user frustration in an explicit attempt to develop methods for using multiple sensing technologies. The experimental design involved a game with several puzzles. Participants were told that the experimenters were interested in how brightly colored graphics would influence physiological variables in an online game. Unbeknown to the participants, the game software was rigged to randomly introduce episodes of unresponsiveness. As participants were being timed and had been offered a reward if they had the fastest task completion times, these delays would presumably cause frustration.1 BVP and GSR responses were used to develop models that could distinguish between frustrating and nonfrustrating states (Scheirer et al., 2002).

Interaction with computer games is a natural topic for physiological data. As anyone who has played video games knows, players can become excited while driving race cars, hunting aliens, or playing basketball on the computer. However, the fast-paced nature of these games limits the applicability of many techniques. Intrusive data collection techniques, such as “think-aloud” descriptions, interfere with the game-playing experience and posttest questionnaires fail to recapture all of the nuances of the playing experience (Mandryk and Inkpen, 2004).

One study used various physiological data sources—GSR, EKG, cardiovascular rate, respiration rate, and facial EMG—to measure responses to computer games played against a computer and against a friend. Starting from the premise that the physiological data would provide objective measures that would be correlated to players' subjective reports of experiences with video games, the researchers hypothesized that preferences and physiological responses would differ when comparing playing against a computer to playing against a friend. Specifically, they hypothesized that participants would prefer playing against friends, GSR and EMG values would be higher (due to increased competition), and that differences between GSR readings in the two conditions would correspond to subjective ratings (Mandryk and Inkpen, 2004).

To test these hypotheses, they asked participants to play a hockey video game, against the computer and against a friend. Participants were recruited in pairs of friends, so each person knew their opponent. The hypotheses were generally confirmed: participants found playing against a friend to be more exciting, and most had higher GSR and facial EMG levels when playing with a friend. Cardiovascular and respiratory measures did not show any differences. Investigation of specific incidents also revealed differences—participants had a greater response to a fight when playing a friend. Examination of the relationship between GSR, fun, and frustration revealed a positive correlation with fun and a negative correlation with frustration (Mandryk and Inkpen, 2004). The use of multiple coordinated sensors to measure frustration in game playing continues to be an active area of research, with more recent papers exploring topics such as the impact of system delays (Taylor et al., 2015).

EEGs have been also used by HCI researchers to develop brain-computer interfaces that use measurable brain activity to control computers (Millán, 2003). Machine-learning algorithms applied to EEG signals have been used to distinguish between different types of activity. Similar to the study of cooperative gaming described earlier (Mandryk and Inkpen, 2004), one study found that EEG signals could be used to distinguish between resting states, solo game play, and playing against an expert player (Lee and Tan, 2006). Other HCI applications involving EEG signals include identifying images of interest from a large set (Mathan et al., 2006) and measurement of memory and cognitive load in a military command-and-control environment (Berka et al., 2004).

Electromyography has been used to measure a variety of emotional responses to computer interfaces. One study of web surfing tasks found strong correlations between facial EMG measures of frustration and incorrectly completed tasks or home pages that required greater effort to navigate (Hazlett, 2003). Similar studies used EMG to measure emotional responses to videos describing new software features, tension in using media-player software (Hazlett and Benedek, 2006), and task difficulty or frustration in word processing (Branco et al., 2005). An experiment involving boys playing racing games on the Microsoft Xbox established the validity of facial EMG for distinguishing between positive and negative events (Hazlett, 2006). Combinations of multiple physiological measures, including EMG, have also been used to study emotional responses (Mahlke et al., 2006).

A broad body of work has explored the use of body sensing in a variety of healthcare domains, including assessment of disability, rehabilitation, and in use by clinicians. Several of these applications have been discussed in this chapter; for a more in-depth discussion, see “Body Tracking in Healthcare” in O'Hara et al. (2016).

3.1.7 Galvanic skin response

There is an increase in eccrine sweat gland activity when saddening, threatening, joyful, etc. actions are observed. GSR measures the change in electrical activity, which takes place due to the shift in sweat gland activity (as shown in Fig. 6). An increase in sweat gland activity can take place due to both positive (“joyful”) and negative (“scary”) events. Therefore the GSR signal does not represent the type of emotions [11]. The hands have a high density of sensitive sweat glands; thus GSR data are collected from the finger, wrist, or palm. A well-known application of GSR is a lie detection test.

From the earlier discussion, it is clear that EEG and MEG both are used to identify disorders that involve the brain, but MEG is more superior. ECG is used to identify disturbances in the heart. EMG and MMG are used to identify complications that affect the muscles, but MMG is more preferred. EOG detects the eye movement. GSR records electrical activity in the skin, which is due to variation of moisture level in the body as a result of sweating. This significant information assists in identifying the various abnormality of the human body. In addition, the various signals are collected for addressing particular health issue, and thus same signal may or may not provide the satisfactory results for other health problem.

Biological and Medical Sensor Technologies

Beyond position tracking, other physiological body attributes can also be monitored and used as inputs to control various aspects of a virtual world. These aspects include body functions, such as temperature, perspiration (galvanic skin response), heart rate, respiration rate, emotional state, and brain waves (Fig. 4-34). These functions might be measured simply to monitor the participant’s condition as they experience a world, or they might be used to alter the world—to determine what experiences are most relaxing, for instance, and use that feedback to guide the user down a calmer path [Addison et al. 1995], or to use the respiration rate to control vertical movement in a scuba diving experience [Davies and Harrison 1996] (see also Craig et al. Chapter 8, Case Study 8.1 [Craig et al. 2009]).

Technology for sensing user biometrics is increasingly becoming available as medical tools, as well as for personal fitness (e.g., the Fitbit) and entertainment purposes (e.g., the Myo Gesture Control Armband). Toys with simple EEG measurements used to manipulate physical properties of the toy (fly a helicopter or move a ball) have been used in some do-it-yourself interfaces (Fig. 4-35).

A Case for Quant

The large amounts of empirical data that are the result of quantitative studies can be used to validate designs. The risks are that they can sometimes be too narrow and not holistic enough, so they leave out important factors that are part of the bigger picture. This can be mitigated by using quantitative data in a strategic way, keeping the overall user and business goals in mind while still analyzing the details in the data.

In the many successful cases, quantitative data can be triangulated with qualitative data to deepen the understanding of the user and the system. This combination can provide rich data that can identify the big picture issues, patterns, and more detailed findings for specific issues.

The following case study is an example of the power of triangulating data. This case study considers many user touch points. Furthermore, it provides a case for biometrics, using eye-tracking and galvanic skin response, combined with qualitative feedback and self-reporting to better understand users’ emotional engagement with a product.

User Experience Across Platforms

How Did We Use UX to Solve It?

We coordinated five tools—electrodermal activity (EDA), eye tracking, Microsoft Product Reaction Cards, net promoter scores (NPSs), and qualitative feedback—to understand the emotional impact of product packaging, digital ads, and Web site design.

Tools

Electrodermal Activity

The use of biometrics provides information about affective responses as they occur. EDA data were the most appropriate choice for this study, since the measurement device was non-invasive and mobile. The Affectiva Q Sensor is a wearable, wireless biosensor that measures emotional arousal via skin conductance. The unit of measure is EDA that increases when the user is in a state of excitement, attention or anxiety, and reduces when the user experiences boredom or relaxation (Figure 7.3).

Figure 7.3. Eyetracking glasses and EDA gloves.

EDA, also known as skin conductance or galvanic skin response (GSR), is a method of measuring the electrical conductance of the skin, which varies with its moisture level. Sweat glands are controlled by the sympathetic nervous system (Martini and Bartholomew, 2003); therefore, skin conductance is used as an indication of psychological or physiological arousal.

Eye Tracking

In addition to biometrics, eye tracking can be used to pinpoint emotionally charged experiences. Observers look earlier and longer at emotionally charged images than neutral images, perhaps to prepare for rapid defensive responses (Calvo and Lang, 2004). This study used both SensoMotoric Instruments (SMIs) RED computer monitor eye tracking system and mobile eye tracking glasses.

Microsoft Product Reaction Cards

The Microsoft product cards were used to form the basis for discussion about a product (Benedek and Miner, 2002) and to access the participant’s perception of her emotional responses. The main advantage of this technique is that it does not rely on a questionnaire or rating scales, and users do not have to generate words themselves. The 118 product reaction cards targeted a 60% positive and 40% neutral balance.

Net Promoter Scores

The NPSs were used to understand the appeal of the interface. This surveying tool asks participants about their willingness to promote a company or product, indicating their loyalty and future growth (Reichheld, 2003).

Qualitative Feedback

This research used a “think-aloud” protocol to encourage feedback from participants throughout the study.

We conducted a study on natural outdoor skincare products to:

understand what engages consumers on the client’s product packaging and Web site relative to the competition;

make recommendations for branding and Web site design based on feedback;

improve the user experience across multiple interfaces by effectively addressing concerns of users.

During each 90-min, moderated session the participants wore the Q sensor, which monitored their EDA. Depending on the task, the participants either wore eye-tracking glasses or worked on a computer monitor with an eye-tracking system, both of which monitored their eye movements. Participants also performed the think-aloud protocol. They performed the following tasks:

Analyze product packaging. Participants were asked to view a shelf of four sunscreen products and then decide which product(s), if any they would purchase. They picked up and examined the product packaging while completing this task. The participants verbally indicated which product(s) they would purchase and why.

At the end of the task, they gave a NPS for each of the four products. Participants performed a similar task for insect repellents.

Evaluate the Web site. Participants performed several tasks on Web site A. At the end of the tasks, they gave a NPS for the Web site, and they chose ten words from the Microsoft Product Reaction Cards. Participants performed the same tasks on Web site B. The A/B testing for the Web sites was counterbalanced, with 50% of the participants viewing Web site A first, and 50% viewing Web site B first.

Evaluate the effectiveness of magazine advertisements. Participants viewed four images for ads.

We analyzed the data collected from the five tools that provided insight into what participants were thinking about the interfaces. The data from the research tools can be combined in many ways. On this project, we mainly used the following combinations:

First, moments of interest—or peaks—in the EDA data indicate where there is a heightened emotional response. While EDA data allow practitioners to pinpoint exact moments of emotions, it does not reveal what type of engagement or emotion the participant was experiencing. We needed to view the EDA data alongside data from other tools. These moments of heightened emotional response can be synchronized with eye tracking data to pinpoint what the participant was viewing at the exact moment of heightened EDA. Qualitative data can provide an additional level of understanding of the emotional response. Specifically, we reviewed the EDA data and identified the time of the peaks. Then the eye tracking data were reviewed for the specific stimulus the participants were viewing at the time of the EDA peak. Video recordings were then reviewed to hear what feedback the participants provided at the time of the EDA peak.

Second, the emotional activation level for a specific task or Web site can be combined with Microsoft Product Reaction Card data to determine the intensity and type of the emotional engagement.

Third, the participant’s cognitive experience can be understood by combining data from the Microsoft Product Reaction Cards, NPS, and qualitative feedback. The participant’s own interpretation of their emotional responses can be determined from this combination. Analysis of these three data sets provided insight into what the participants were thinking about the interfaces. We reviewed the most frequently chosen words on the Microsoft Product Reaction cards, and compared these positive and negative words with the NPSs and the qualitative feedback.

And fourth, aggregated eye tracking data can be analyzed to determine where participants focused most intensely.

6.2.1 Activity Recognition Systems

A typical activity-recognition system contains three components: a sensor infrastructure, a feature extraction component, and a learning module for recognizing known activities to be used for future prediction.

The sensor infrastructure uses relevant sensors to collect data from the environment and/or people. A wide range of sensor types have been used during the past decade. Early work in activity recognition has mainly focused on using static cameras (e.g., see Chellappa [11], Duong et al. [18,19]) or circuits embedded indoors [64,65]. However, these types of sensors are limited to some fixed location and only provide the ability to classify activities that have occurred previously. Often they cannot be used to predict future activities, which is more important, especially in healthcare assistance systems.

Much of the recent work has shifted focus to wearable sensors due to their convenience and ease of use. Typical sensing types include GPS [40], accelerometers [24,30,36,38,46,57], gyroscopes [24,38], galvanic skin response (GSR) sensors, and electrocardiogram (ECG) sensors [33,38] to detect body movement and physical states. These sensors have been integrated into pervasive commercial wearable devices such as Sociometric Badge [47]2 or SHIMMER [10].3 These types of sensors and signals are becoming more and more prevalent, creating new opportunities and challenges in activity-recognition research.

The purpose of the feature extraction component is to extract the most relevant and informative features from the raw signals. For example, in vision-based recognition, popular features include scale-invariant feature transform (SIFT) descriptors, silhouettes, contours, edges, pose estimates, velocities, and optical flow. For temporal signals, such as those collected from accelerometers or gyroscopes, one might use basic descriptive statistics derived from each window (e.g., mean, variance, standard deviation, energy, entropy, FFT, and wavelet coefficients) [2,57,59]. Those features might be processed further to remove noise or reduce the dimensionality of the data using machine learning techniques, such as Principal Component Analysis (PCA) [31] or Kernel PCA [66,68], to provide a better description of the data.

The third and perhaps the most important component of an activity-recognition system is the learning module. This module learns predefined activities from training data with labels to be used for future activity prediction. Activity labels are often manually annotated during the training phase and supplied to a supervised learning method of choice. Once trained, the system can start to predict the activity label based on acquired signals. However, manually annotating activity labels is a time-consuming process. In addition, the activity patterns may grow and change over time, and this traditional method of supervised training might fail to address real-world problems. Our work presents an effort to address this problem.

A relevant chapter in this book by Rashidi [56] also recognizes this challenge and proposes a sequential data mining framework for human activity discovery. However, Rashidi’s work focuses on discovering dynamic activities from raw signals collected from a data stream, whereas by using Bayesian nonparametric models, our work aims to discover higher-order latent activities such as sitting, walking, and people gathering, referred to as situations or goals in Rashidi’s chapter.

Bio-metrics to detect physiological responses to emotional impact

The use of instrumented measurement of physiological responses in participants is called biometrics. Biometrics are about detection and measurement of autonomic or involuntary bodily changes triggered by nervous system responses to emotional impact within interaction events. Examples include changes in heart rate, respiration, perspiration, and eye pupil dilation. Changes in perspiration are measured by galvanic skin response measurements to detect changes in electrical conductivity.

Such nervous system changes can be correlated with emotional responses to interaction events. Pupillary dilation is an autonomous indication especially of interest, engagement, and excitement and is known to correlate with a number of emotional states (Tullis & Albert, 2008).

The downside of biometrics is the need for specialized monitoring equipment. If you can get some good measuring instruments and are trained to use them to get good measures, it does not get more “embodied” than this. But most equipment for measuring physiological changes is out of reach for the average UX practitioner.

It is possible to adapt a polygraph or lie detector, for example, to detect changes in pulse, respiration, and skin conductivity that could be correlated with emotional responses to interaction events. However, the operation of most of this equipment requires skills and experience in medical technology, and interpretation of raw data can require specialized training in psychology, all beyond our scope. Finally, the extent of culture independence of facial expressions and other physiological responses is not entirely known.

4.2.3.3 Simultaneous signal acquisition

The PPG waveform provides a wealth of information about cardiovascular hemodynamics and can be used to obtain several vital parameters such as heart rate, respiration rate, arterial oxygen saturation, and blood pressure. However, the true potential of the PPG is realized when other physiological signals such as the electrocardiogram (ECG), accelerometry, and galvanic skin response (GSR - electrodermal activity) signals are recorded and analyzed concurrently.

Simultaneous acquisition of multiple PPG signals at different body sites facilitates measurement of pulse transit time (PTT), the time taken for the pulse wave to propagate along an arterial path. PTT is related to blood pressure (Mukkamala et al., 2015), and can be used to calculate pulse wave velocity (PWV), a marker of cardiovascular risk (Mattace-Raso et al., 2010).

Simultaneous acquisition of the ECG can serve as a cardiac timing reference for PPG signals. Pulse arrival time (PAT) can be calculated from simultaneous PPG and ECG signals, which can be used for blood pressure assessment (similarly to PTT), and to assess cardiovascular risk factors such as arterial stiffness and hypertension (Rundo et al., 2018).

The GSR signal correlates with sympathetic nervous system activity and is a valuable instrument for measuring arousal and certain facets of autonomic control. This information complements related information derived from pulse rate variability analysis of the PPG signal (Goshvarpour and Goshvarpour, 2019). Hence, combined acquisition and analysis of these signals offers opportunity for deeper mental stress assessment, along with cardiovascular hemodynamics.

The accelerometry signal can be used to remove motion interference from PPG signals, which is a particularly important issue when using portable/wearable devices. The simultaneous reference motion measurements provided by accelerometers can be used to remove such interference (Boloursaz Mashhadi et al., 2015). However, accelerometers do not differentiate between acceleration due to movement and acceleration due to gravity. Hence, it has been suggested that gyroscopes could be used for better removal of motion interference from PPG signals (Casson et al., 2016; Lee et al., 2019).

Artificial Intelligence

The data produced from sensors will be continuous and in real time. This means that it is difficult to represent specific outcomes using a rule-based approach (i.e., one where expert knowledge is coded into predefined outcomes). With pattern recognition, the system learns to recognize patterns directly from real data sets which can result from supervised and unsupervised learning. In these cases, the data is used to train the AI algorithm (called the training set) to recognize patterns. The data are typically from samples from the problem domain, for example, a series of images or real values representing heart rate. In supervised approaches, the data can be labeled as the items in the data set are associated with a distinct category. An error signal can be incorporated during the analysis which refines the algorithm, improving its performance at classifying the pattern correctly. Examples of supervised AI algorithms are artificial neural networks (ANNs).

In ANN, training involves presenting an object from the data set to the network and calculating a response. Algorithms such as Backpropagation (Rumelhart, Hinton, & Williams, 1986) can be used to minimize errors that are created in the training set resulting from the network. Training ceases once the overall error in the network for the training set has reached an acceptable minimum. Neural networks have been used by Yuen, San, Rizon, and Seong (2009) to classify human emotions from EEG signals. Their network trained by the Backpropagation algorithm demonstrated a classification rate of 95% in five types of human emotion: anger, sadness, surprise, happiness, and neutral. Kobayashi and Hara (1991) used a feed forward network trained by using the Backpropagation algorithm to recognize facial expressions including surprise, fear, disgust, anger, happiness, and sadness. Lee et al. (2005) described an approach to develop a “Multilayer Perceptron” to recognize the emotions sadness, calm pleasure, interesting pleasure, and fear. They used electrocardiographs and galvanic skin response data representing heart rate variability and skin response respectively as inputs to the network. Their network resulted in 80% accuracy when determining emotions with fear being predicted with the highest level of accuracy. Using a trained feed-forward network, which is a process whereby the data passes through the nodes from signal to output in one direction, reduces the computational overhead when classifying new data. Training for this would need to take place on a high performance computer. The resulting trained network could then be used on a variety of devices including mobiles ones.

A limitation of supervised methods is that they require labeled data in order for the algorithm to learn. Therefore unsupervised learning is used to cluster data based on an algorithm that determines classes in the data. These identify groups of similar objects within a greater data set. A clustering algorithm would ascertain what emotions exist, as well as their similarities and differences, and would subsequently organize them into appropriate clusters.

Khosrowabadi, Quek, Wahab, and Ang (2010) used a Self-Organizing Map (SOM) in an unsupervised ANN to recognize emotions from EEG data. Pictures from the International Affective Picture System (IAPS) set, coupled with synthesized musical excerpts, were used to invoke emotion stimuli in subjects whilst capturing their EEG responses. The emotions that they were trying to ascertain were calm, happy, sad, and fear. They found that SOM was better able to separate out the four emotions. Quazi et al. (2012) used a k-means clustering algorithm, one which aims to separate observations into appropriate groups or clusters, in order to identify emotions from skin conductance and heart-rate data. This algorithm is similar to SOMs where data is grouped into k cluster. However, this algorithm can determine the number of groupings whereas in SOM, the network structure is typically fixed.

Once the information has been processed, ideally the technology would be able to adapt to the identified responses. This ability of technology to recognize and respond to human states is known as affective computing. This has been identified as a possible method for developing intelligent systems that can respond to patients' emotions. In a review by Luneski, Konstantinidis, and Bamidis (2010), they suggest that multimodal emotional expressions (speech, facial expressions, body gestures, and physiological reactions) can be used to facilitate communication between patients and healthcare teams.

Intelligent systems are also an important part of computer games and simulations. They provide a way of responding to players in order to provide them with more challenges and personalize their experience. Where computer games have been traditionally associated with leisure or recreational activities, there is increasing interest in how they can be used in educational settings. Game-based learning, serious games, and gamification all take advantage of the constituent parts of games—their mechanics—to create fun and pleasurable experiences. By combining these with educational content, games with a purpose can be created. Each time an individual plays a game, the outcome may be different. The player can therefore experiment with and experience different ways of solving particular problems. This allows them to construct knowledge about a situation or solve a problem in their own time and at their own pace.

5.2 Technical analysis of the Internet of Things-based pharmaceutical industry

IoT technologies improve QoS, reduce errors, and detect health anomalies using vital signs. IoT technology can be applied in a pharmaceutical system to identify Adverse Drug Reactions (ADRs), harmful side effects at the time of pregnancy or lactation, allergies, and complications with liver/renal defects. The physicians would feel more comfortable in drug prescribing and making clinical decisions.

5.2.1 Remote monitoring system

A remote monitoring system (RMS) uses Raspberry Pi3 to acquire data from sensors such as air flow, pulse oximeter, body temperature, and a galvanic skin and blood pressure monitor. Graphical User Interface (GUI) assists in analyzing the historical data and records the information by visualizing it. This can be accessed either locally or remotely using touch display or web devices. An IoT sensor-based platform provides solutions using historical patient data that trigger live action locally and remotely. A Remote Patient Monitoring (RPM) system explores several ways for classifying physiological data [2]. However, the sensor data vary significantly according to the physical activities engaged in by patients. In order to evade erroneous conditions, the acquisition of physiological data from body sensors is performed along with the activity using sensors. RPM enhances the medication process by enabling doctors to prescribe medication remotely and allowing caretakers to verify that the correct medication is provided to the right patient [3]. The interface with a WSN permits caregivers to monitor patient-related information continuously, where a body area network comprises of multiple sensors which are fixed on or in the body. In Refs. [4,5], Raspberry Pi 3 board and Arduino Uno boards are used for measuring the ECG signal for cardiovascular diseases. The data acquired from different types of sensors are described below and depicted in Fig. 5.2.

Figure 5.2. Data acquisition from different sensors.

•

Air flow checks the breathing intensity;

•

Pulse oximeter (e.g., Contec CMS50D+) monitors the pulse rate and oxygen saturation level in the blood;

•

Body temperature sensors track the temperature of the body;

•

Galvanic skin response verifies skin conductance and resistance;

•

Blood pressure monitor (e.g., Kodea KD-202F) detects the pulse rate and diastolic blood pressure.

The pulse oximeter can be directly connected to USB ports and the rest of the sensors are connected to the sensor platform which is embedded with a Raspberry Pi and Arduino.

The communication is enabled both synchronously and asynchronously to gather, save, and visualize the sensor data. Data gathered from the sensors includes air flow, pulse oximeter, and galvanic skin response, which are recorded at specific time intervals. With a body temperature sensor, the current value is compared with the previous one over a set amount of time. If temperature varies and is above the threshold level, the time interval is reset and the new value is recorded when the variation is below the threshold.

5.2.2 Drug-delivery system for neurological disorders

In the unified Drug-Delivery System (DDS) framework, drug injection is provided based on the seizure detection. An Electromagnetic Actuated Valveless Micropump (EAVM) with diaphragm consists of polydimethylsiloxane and is used for delivering drugs. Epilepsy refers to the neurological disorder in which brain activity becomes abnormal, causing seizures or unusual behavior, resulting in convulsions or lack of consciousness. An average of 1% of the world’s population suffers from this problem which can be controlled via surgery and antiepileptic drugs (AEDs). A very small fraction of people are willing to undergo epilepsy surgery and one-third of patients do not respond to AEDs. An effective solution for controlling seizures is vital to prevent serious consequences. The DDS is capable of detecting a seizure and injecting an AED in the respective zone of the brain at the right time. This localized and responsive injection improves the efficacy of the drug, providing an effective solution. Connectivity with other healthcare devices is established with the IoT and it enables effective analysis of health behavior and remote health monitoring. In DDS, drugs are injected into the epileptogenic zone based on seizure detection [6].

An IoT-based DDS framework provides regulated drug injection and better detection accuracy. Generally, the system consists of two units, namely seizure detection and drug-delivery units.

•

Discrete wavelet transform (DWT) basically provides frequency determination and conjoint time to capture complex electroencephalography (EEG) dynamics that leads to accurate seizure detection.

•

The electrostatic and piezoelectric actuation requires high voltage, while the electromagnetic actuation operates at a low voltage for membrane displacement. This reduces power consumption and paves the way for using it in low-power medical applications.

•

DDS features low-power consumption, universal connectivity, and better detection accuracy. An IoT-based RMS facilitates considerable enhancement of the epilepsy patient’s quality of life.

Seizure prediction, detection, and control are hot topics in on-going research. Some methodologies have been proposed using focal cooling, electrical simulation, or drug delivery for controlling seizures [7]. An electrophoretic drug-delivery device is proposed for on-demand drug delivery that functions based on the underlying principle of an organic electron ion chip. An AED can be injected directly into a particular brain region to control seizures. A seizure-detecting smartwatch detects seizures and notifies physicians about such occurrences. In DDS, initially the EEG signals are decomposed using DWT, statistical features are extracted and fed into an k-NN classifier for detection. Once the seizure is detected, AEDs are injected into the targeted area in order to end the seizure propagation. The complete processing of a DDS is shown in Fig. 5.3.

Figure 5.3. Drug-delivery system.

5.2.2.1 Seizure detection unit

The seizure detection unit is comprised of the following subparts: feature extraction using DWT and a k-NN classifier. EEG signals are decomposed using DWT and time frequency (TF) localization is provided. The features are extracted and given to the k-NN for classification.

DWT with TF localization is useful for analyzing nonstationary and complex EEG signals. A k-NN classifier is used in training and testing phase with a distinct data set. A Euclidean distance metric is used to compute the nearness of the data.

5.2.2.2 Drug-delivery unit

An EAVM offers advantages when compared to other micropumps including:

1.

It requires lower actuation voltage;

2.

It leads to faster response, higher actuation force, and deflection;

3.

Elimination of valves makes it more simplified and reliable.

In the smart DDS, the k-NN classifier is followed for seizure detection and an electromagnetic micropump is used for drug delivery. This reduces the power consumption and enhances the detection accuracy. Moreover, it is most suitable for epilepsy treatment and can be preferably implemented in commercial biomedical applications.

Automated Tracking of Affective States During Learning

The automated classification of affect states varies in the extent to which they are invasive. Noninvasive sensing methods do not attach sensing devices to the learner and do not disrupt the normal stream of learning by probing the students with questions about their emotions or learning. The computers detect emotions of the learner by analyzing different communication channels and their interactions (Calvo & D’Mello, 2010; D’Mello & Graesser, 2010; Picard, 1997). The common communication channels include facial expression (D’Mello & Graesser, 2010; Grafsgaard, Boyer, Phillips, & Lester, 2011; Kapoor et al., 2007); speech parameters (Litman & Forbes-Riley, 2006); body posture (D’Mello, Dale, & Graesser, 2012); and language and discourse interaction (D’Mello & Graesser, 2012). The accuracy of these automated detection methods is modest, but so is the agreement among human judges, as discussed above. Some biological channels of affect sensing are minimally invasive, such as a wrist band, but most are maximally invasive, such as an EEG or fMRI. These methods include the recording of heart rate, movements of the muscles, galvanic skin response, and brain activity (Calvo & D’Mello, 2010). However, most of these methods are invasive, in the sense that it is obvious to the students that they are being recorded by physical instruments that have contact with their bodies.

Once again, there is no gold standard for measuring what emotions the learners are actually experiencing during learning, because all measures are imperfect windows into emotional experience. The various measures correlate only modestly (kappas ranging from 0.2 to 0.5, see D’Mello & Graesser, 2010), with each measure having both virtues and liabilities. In light of these indeterminacies in measurement validity, researchers often collect multiple measures and adjust the confidence in their conclusions according to the consistency of the results.

Most of our work on automated emotion detection has concentrated on three channels: the discourse interaction history; facial actions; and body movements—and combinations of these three channels (D’Mello & Graesser, 2010, 2012). The discourse interaction history includes events stored in the AutoTutor log file, the speech acts of student and tutor turns, and the knowledge states achieved by the student during the tutorial dialog. An analysis of the discourse interaction history provides a model of the context of an emotional expression. The facial actions and expressions and body pressure measurement systems are tracked by particular systems that are beyond the scope of this chapter. The point we wish to convey here is that the discourse history goes a long way (90% or higher, compared with all other sensing devices) in classifying students’ affect states (defined as a composite of many measures). As one example, Graesser and D’Mello (2012) reported that dialog interaction history showed accuracies of 63%, 77%, 64%, 70%, and 74% (50% is chance), in discriminating confusion, frustration, boredom, flow, and delight from neutral. The average across emotions was 70%. If we were to transform these scores to values comparable with kappa scores [i.e., 2*(score − 0.5)], the quantities would be 0.26, 0.54, 0.28, 0.40, and 0.48, respectively, or 0.39 overall. Such kappa scores are comparable with accuracy scores reported by other researchers in the literature who have attempted automated emotion detecting systems. They are also comparable with the reliability of human judgments.

It is beyond the scope of this chapter to specify the mechanisms of the computer automatically detecting affective states from the different channels of communication (see D’Mello & Graesser, 2010, 2012, for a description of these mechanisms); however, we will present some highlights of the channel that involves the language and discourse interaction history. Advances in computational linguistics (Jurafsky & Martin, 2008) have made it possible to interpret students’ natural language by segmenting the language within conversational turns into segments (such as speech acts), classifying the segments into categories (such as questions, assertions, expressive evaluations, and other speech act categories), and performing semantic evaluations of the quality of the student assertions. Quality can be assessed automatically by matching the verbal input of the student to representations expected by the computer. The semantic match algorithms go beyond keywords and into inferences derived from large text corpora and higher dimensional semantic spaces (Landauer, McNamara, Dennis, & Kintsch, 2007). The history of these interpreted interactions in learning sessions is stored in log files, so that data mining and machine learning analyses can be performed. This allows researchers to discover what language and discourse patterns are diagnostic of particular learner emotions.

According to D’Mello and Graesser (2010), the dialog cues that trigger the emotions are quite different for the different emotions. The cues that accompany confusion tend to be short student responses, frozen student expressions (such as “I don’t know”; “Uh huh”), speech acts by the tutor that are indirect (such as hints), and early time phases during the student’s initial attempts to solve the problem or answer the questions posed by the tutor. In contrast, the cues that accompany frustration are negative tutor feedback and student responses that are locally good ideas but not globally good ideas. Flow/engagement tends to occur with lengthier answers, during early phases of the dialog, and after positive tutor feedback. Boredom tends to occur in later phases in the session or a particular problem and when the tutor tends to lecture with direct assertions. These dialog cues are important when we track emotions of adult learners in our conversational learning environments.