Machine Learning Predictive Models Will Not Replace Clinical Judgment Anytime Soon

Beth Krone, Ph.D.
Icahn School of Medicine at Mount Sinai

In the spirit of full disclosure, I am a technophile. My age-cohort was the first to have desktop computers as children. I first learned to program in binary. After a decade as an end-user, I still have the muscle memory of a programmer. The concept behind Machine Learning predictive models in mental health diagnostics – the idea that we can train computers to be ‘smart’ enough to recognize patterns in data and ‘learn’ to classify and predict outcomes from reading the data without any prior information or rule specification – does not intimidate me. I welcome our computer overlords. So far, though, computers have not been out-performing clinicians in separating ADHD from typically developing youth using brain-based biomarkers.

The ADHD 200 global competition freely gave a moderately large fMRI dataset to researchers and statisticians, who responded to the call and flexed their creativity in developing algorithms and models to distinguish between the dataset’s ADHD patients and healthy controls. Several teams have published on the data, using pieces of the dataset to test their theories and in search of the elusive definitive confirmatory biomarker of disease state that, so far, seems not to exist. In 2012, for example, Sato and his team created a classification model using brain region homogeneity as a measure of volume, Fractional amplitude of low frequency fluctuations (fALFF) as a measure of spontaneous brain activity at rest, and network maps of the default mode (positive values) and task-positive networks (negative values). The model returned a median predictive accuracy of 54% for discriminating ADHD from controls, providing no additive clinical value to the diagnostic process at that time.

Recently, Sen and his team (2018) published a general prediction model using the ADHD 200, then tested in the ABIDE dataset, which is also freely available. From MRI data, their team generated 3 dimensional representations of brain volumes, or ‘texture’, that discriminated between ADHD and typical development with 63% accuracy. Adding to the dataset information about 45 independent intrinsic connectivity networks (ICNs) derived from the resting state fMRI data (networks thought to underlie functions such as mind-wandering, and planning), raised the predictive accuracy of their model to 67% in the ADHD 200 dataset, and retested with 64% accuracy in the ABIDE dataset. An accuracy rate in the mid to high 60’s is still far below the expected performance of a well-trained human diagnostician, but not much different than the overall predictive validity of the Continuous Performance Test (CPT-II; Fazio, 2014). Given the differential between CPT and fMRI in terms of time, cost, and resources, the CPT is not likely to soon be replaced by fMRI for augmentation of clinical judgment as standard of care.

Other recently published works highlight the diversity of methods employed within machine learning and the range of quality control procedures in data acquisition and analysis, against the larger clinical backdrop of heterogeneity in ADHD and the value of clinical training in mental health care diagnostics. In 2017, for example, Lirong Tan and his team developed a Support Vector Machine (SVM) model to separate youth with ADHD from controls based on the volume of brain regions as measured by fMRI rather than using the more traditional approach of looking at the volume of brain regions measured as physical structures via MRI. The advantage of using the functional measure here was to capture how much a task caused activation in and around a particular structure of the brain. The team entered demographic data into their model, looking at socio-cultural contributors to the overall presentation of ADHD. They found that, brain-wide, functional volumes discriminated ADHD with equivalent accuracy (59.6% accuracy) to age and sex (58.5% accuracy), with neither being of strong clinical value. Tweaking the model by entering information about 10 brain regions of interest in ADHD pathology improved accuracy to 67%, correlating to subtle differences across the brain, rather than to a significant difference in any one particular region that could identify a group.

Xun-Heng Wang (2018) and his team also examined 10-ICNs, including an executive control network and a cerebellar network, for their predictive value. Their approach was to measure variability in the networks’ functional connectivity when not performing a task. Unfortunately, they included demographics in their model without examining the independent predictive quality of network variability. Since these were the same demographic features that Lirong found independently predicted with 58.5% accuracy, and we cannot determine the actual independent value of the connectivity analysis, we cannot be certain that their model truly achieves 75% accuracy with which the team presents us. These are claims that science will prove with replication, or not.

In the end, scientists will keep searching, fueled by the strong desire and public need to find ‘the’ biomarker or biomarkers that definitively separate ADHD from typical development. For the foreseeable future, though, clinical judgment is in no danger of being replaced by machine intelligence. Through more than a decade’s work as a clinician for a clinical and translational research group, I have frequently had to tell patients that, “No, I’m sorry, but we cannot use your fMRI/MRI to diagnose you. The science just is not there. No one can do that, yet.” Yet.

References:
Fazio, R., Dole, L. & King, J. (2014). CPT-II versus TOVA: Assessing the Diagnostic Power of Continuous Performance Tests. Archives of Clinical Neuropsychology 29(6):540
Sato, J.R., Hoexter, M.Q., Fujita, A., & Rohde, L.A. (2012). Evaluation of Pattern Recognition and Feature Extraction Methods in ADHD Prediction. Frontiers in Systems Neuroscience, 6
Sen, B., Borle, N., Greiner, R. & Brown, M.R.G. (2018). A General Prediction Model for the Detection of ADHD and Autism Using Structural and Functional Imaging. Plos One, 13(4), e0194856.
Tan, L., Guo, X., Ren, S., Epstein, J. & Lu, L.J. (2017) A Computational Model for the Automatic Diagnosis of Attention Deficit Hyperactivity Disorder Based on Functional Brain Volume. Frontiers in Computational Neuroscience, 11, 75. DOI 10.3389/fncom.2017.00075
Wang, X-H., Jiao, Y & Li, L. (2018) Identifying Individuals with Attention Deficit Hyperactivity Disorder Based on Temporal Variability of Dynamic Functional Connectivity. Nature Scientific Reports, 8/11789.

Managing ADHD: What is Your Implementation Plan?

As part of the mission of the APSARD Psychosocial Treatment Committee, the committee members will share blogs related to issues relevant to non-medical treatments for ADHD.

J. Russell Ramsay, Ph.D.
Associate Professor of Clinical Psychology
University of Pennsylvania, Perelman School of Medicine

Implementation intentions stem from a line of self-regulation research focused on the observation that a purely goal-focused approach to behavior change does not inexorably produce actions necessary to achieve that goal (Gollwitzer, 1999). Consequently, more specific, action-oriented plans designed for specific contexts (and tied to an overarching goal) have been found to improve follow through. These plans are framed in “If X, then Y” conditional statements in which a specific action or obstacle is tied with a specific setting, such that the setting itself then provides a cue for the desired action: “If situation X is encountered, then I will perform the goal-directed response Y!” (Gollwitzer & Oettingen, 2016, p. 223). Thus, someone with the goal of losing weight might identify that he is prone to snacking after seeing someone at work eating candy from the vending machine. His implementation intention might be “If I have the urge to go to the vending machine, then I will go get water or coffee instead.” The theory is that the vending machine becomes a cue for the coping response.

Although not yet studied in adults with ADHD, non-clinical studies of implementation intentions with children with ADHD indicate that they promote better task follow through (Gawrilow, Gollwitzer, & Oettingen, 2011a, 2011b; Gawrilow et al., 2013). The “If X, Then Y” (or “When X, Then Y,” if using “possibility language”) strategy has been a facet of our CBT approach for adult ADHD and it has been a valuable intervention domain (Ramsay & Rostain, 2015). In a small open study of adults with ADHD who completed CBT without medication, significant improvements on an activation measure were achieved, consistent with this and other implementation tenets (Ramsay & Rostain, 2011).

Procrastination is arguably the most common problem faced by adults with ADHD. Specific implementation plans can be designed for the initial behavioral step of task engagement, such as for a college student: “If I can get to the library, then I can open my economics assignment.” These statements are particularly useful for managing the pivot points within a task plan, such as defining the step for returning to the task after a brief break (“When I finish my coffee, then I can re-read the last few sentences I wrote to get re-engaged.”). Implementation plans can also be designed for re-engaging in tasks after being distracted, dealing with rationalizations for escaping a task, managing obstacles (“If the main section of the library is crowded, then I will take the elevator to the 5th floor stacks.”), or navigating other “tipping points” that represent risks for somehow getting distracted from one’s objectives.

The implementation plan or the “If/When X-Then Y plan,” is a useful, “sticky” take-away skill. These reminders can be externalized if the form of coping cards or other means to increase the likelihood of engagement in and follow through on tasks and endeavors.

References
Gawrilow, C., Gollwitzer, P. M., & Oettingen, G. (2011a). If-then plan benefit delay of gratification performance in children with and without ADHD. Cognitive Therapy and Research, 35, 442-455. doi: 10.1007/s10608-010-9309-z
Gawrilow, C., Gollwitzer, P. M., & Oettingen, G. (2011b). If-then plans benefit executive functions in children with ADHD. Journal of Social and Clinical Psychology, 30, 616-646.
Gawrilow, C., Morgenroth, K., Schultz, R., Oettingen, G., & Gollwitzer, P. M. (2013). Mental contrasting with implementation intentions enhances self-regulation of goal pursuit in schoolchildren at risk for ADHD. Motivation and Emotion, 37, 134-145. doi: 10.1007/s11031-012-9288-3
Gollwitzer, P. M. (1999). Implementation intentions: Strong effects of simple plans. American Psychologist, 54, 493-503.
Gollwitzer, P. M., & Oettingen, G. (2016). Planning promotes goal striving. In K. D. Vohs, & R. F. Baumeister (Eds.), Handbook of self-regulation: Research, theory, and applications (3rd ed.) (pp. 223-244). New York: Guilford.
Ramsay, J. R., & Rostain, A. L. (2011). CBT without medications for adult ADHD: An open pilot study of five patients. Journal of Cognitive Psychotherapy: An International Quarterly, 25, 277-286. doi: 10.1891/0889-8391.25.4.277
Ramsay, J. R., & Rostain, A. L. (2015). Cognitive behavioral therapy for adult ADHD: An integrative psychosocial and medical approach (2nd ed.). New York: Routledge.

Friendships and Social Networks of Adolescents With ADHD

Here is a blog based on a poster presented by Barbara Wise, Ph.D. from Indiana Wesleyan University at the 2018 APSARD conference

Although much research has examined social difficulties of children and adolescents with ADHD (Barkley, 2014; Glass, Flory, & Hankin, 2012; Marton, Wiener, Rogers, & Moore, 2015; Normand et al., 2011; Rokeach & Wiener, 2017; Storebø et al., 2011), no sociometric or social network analysis has been located that examined the friendship networks of adolescents with ADHD. Peer problems in adolescents with ADHD have been linked to risker sexual behaviors and substance abuse (Barkley, 2014; Umberson, Crosnoe, & Reczek, 2010). Many other adolescent health behaviors have been linked to peer influence. For clinicians working with adolescents with ADHD, an understanding of their friendships and social networks will assist in understanding their types of social difficulties, providing accurate anticipatory guidance, and serve as a foundation for building effective interventions for youth with ADHD that are struggling socially.

In this study, I looked at two questions:

  1. How do adolescents with ADHD compare with adolescents without ADHD on measures of perceived social acceptance, strength of ties, presence of one mutual same gender friend, social network measures, and extracurricular activity participation?
  2. Are there differences in these measures among the ADHD subtypes of inattentive, hyperactive, and combined?

This was a descriptive study utilizing secondary analysis of school social network data from the National Longitudinal Study of Adolescent to Adult Health, a nationally representative sample initially collected in 7th through 12th grades. Friendship nominations were collected in Wave I for all students in 122 schools; each student could nominate up to 5 male and 5 female friends. This allowed for whole network analysis of schools with >50% participation. Adolescents with ADHD symptoms in childhood were identified by retrospective self-report in wave III (N=703).
Table one gives the characteristics of the participants, comparing those who did and did not meet ADHD diagnostic criteria.

The findings are summarized below:

Perceived social acceptance. Youth with ADHD inattentive and impulsive self-reported significantly less social acceptance than those without ADHD.
Friendships. The presence of one mutual friend and time spent with friends increased with age among all participants, and this trajectory was not significantly different among those with ADHD symptoms. Multiple linear and logistic regressions demonstrated that those with ADHD were no more likely to be isolates or pendants (to have no or only one social tie) than others. Those with ADHD had similar strengths of ties with their friends as others, with those with hyperactive ADHD only reporting more time spent with friends than average.
Social network measures. There was no difference in popularity (in-degree) among those with ADHD than others, although those with inattentive ADHD reported fewer friends (out-degree) on average than others. Those with inattentive ADHD also had lower centrality and reach within their social networks.
Extracurricular involvement. Youth with ADHD had no significant difference in the total number of extracurricular activities with which they were involved than others, but were significantly less likely to be involved in an academically focused extracurricular activity.

Table 2: comparison of those meeting ADHD criteria with those not meeting, by subtype, for selected social network measures

ns=no significant differences

Conclusions
This study found fewer social deficits in adolescents than suggested in the literature, possibly because most studies of childhood social networks were carried out in clinical samples, where ADHD severity is likely higher. These findings may also reflect that no previous study examined whole social networks of high school students, so that analyses of adolescent social ties were based on self-report, teacher report and parent report. Discrepancies between teacher, parent, and adolescent reports were considered to indicate that the adolescent with ADHD had an inaccurate representation of their own popularity. However, when the student nominated by an adolescent with ADHD as a friend reciprocates that nomination, then this is the most accurate way to measure the existence of a social tie. Perceived lack of social acceptance was striking among ADHD subgroups. This may reflect that while those with ADHD had friends, those friends may not have been in prestigious cliques; social acceptance can mean prestige to an adolescent rather than number of people that name the youth as a friend (Borgatti, Everett, & Johnson, 2013). A more detailed analysis of who adolescents with ADHD are friends with, rather than simply that they have friends, would be valuable.
In clinical practice, the results of this study allow clinicians to offer reassurance to children and parents of children with ADHD that most of the participants who had significant childhood ADHD symptoms appeared to be functioning well socially in adolescence, despite most differences in their perception about their level of social acceptance. The lack of significant differences on most measures, including presence and strength of social ties suggests that there is room for a positive, strengths based approach to the social problems of adolescents with ADHD.
This study does not predict outcomes. Further research is needed to identify to what extent friendship networks and characteristics predict future health behaviors and academic and career success. Further research is needed to explore the effects of comorbidities such as depression and Oppositional Defiant Disorder on the social networks of those with ADHD, as well as specific environmental factors that might be associated with better social outcomes, such as the size of the school and participation in specific types of extracurricular activities. Lastly, there is a noticeable gap in research addressing effective interventions for helping the minority of adolescents with ADHD who have more significant social difficulties affecting their quality of life.
A major strength of this study is that it is the largest population based examination of the social position of adolescents with ADHD in school social networks to date, and the only one describing specific social network characteristics. Limitations of the study include the age of the data, the lack of longitudinal whole network data, and the self-reported nature of the ADHD symptoms.

References
Barkley, R. A. (2014). Attention-deficit hyperactivity disorder: A handbook for diagnosis and treatment. (4rth ed.). New York: Guilford Press.
Glass, K., Flory, K., & Hankin, B. L. (2012). Symptoms of ADHD and close friendships in adolescence. Journal of Attention Disorders, 16(5), 406-417. doi:10.1177/1087054710390865
Marton, I., Wiener, J., Rogers, M., & Moore, C. (2015). Friendship characteristics of children with ADHD. Journal of Attention Disorders, 19(10), 872-881. doi:10.1177/1087054712458971
Normand, S., Schneider, B. H., Lee, M. D., Maisonneuve, M., Kuehn, S. M., & Robaey, P. (2011). How do children with ADHD (mis)manage their real-life dyadic friendships? A multi-method investigation. Journal of Abnormal Child Psychology, 39(2), 293-305. doi:10.1007/s10802-010-9450-x
Rokeach, A., & Wiener, J. (2017). Friendship quality in adolescents with ADHD. Journal of Attention Disorders, , 1087054717735380. doi:10.1177/1087054717735380
Storebø, O. J., Skoog, S., Damm, D., Thomsen, P. H., Simonsen, E., & Gluud, C. (2011). Social skills training for attention deficit hyperactivity
disorder (ADHD) in children aged 5 to 18 years . Cochrane Database of Systematic Reviews, 12(Art. No.: CD008223), 1-89. doi:DOI: 10.1002/14651858.CD008223.pub2.
Umberson, D., Crosnoe, R., & Reczek, C. (2010). Social relationships and health behaviors across the life course. Annual Review of Sociology, 36, 139-157. doi:10.1146/annurev-soc-070308-120011

Brain Stimulation and Impulsivity in Adult ADHD

J. Russell Ramsay, Ph.D.
Associate Professor of Clinical Psychology
University of Pennsylvania, Perelman School of Medicine

Impulsivity is one of the core symptoms of Attention-Deficit/Hyperactivity Disorder (ADHD). A recent study published online ahead of print in the journal Brain Stimulation reported on the results of a sham controlled study of the potential benefit of transcranial direct current stimulation (tDCS) for adults with ADHD. The stimulation targeted the dorsolateral prefrontal cortex to promote downstream effects on cognitive control circuits in this region in order to improve impulse control.
Thirty-seven adult participants with ADHD completed two periods of three sessions of either active tDCS or sham. Sessions occurred two weeks apart in a within-subject, double-blind, counterbalanced order. Outcome measures of impulsivity were Conners Continuous Performance Task (CPT) scores and stop signal task (SST) reaction times. Measures were obtained at baseline, at the end of treatment, and 3-day post-stimulation follow-up.
Results indicated a significant stimulation condition-by-session interaction for change in CPT false positives, driven primarily by a reduction of CPT false positives at the end of treatment; this effect did not persist at 3-day follow-up. There was no significant change in CPT false negative errors, hit response time, or SST reaction time. Nonetheless, these preliminary findings suggest a potential therapeutic benefit of tDCS targeting the DLPFC for reducing impulsivity symptoms.

Note: APSARD Board Members Anthony L. Rostain, M.D. and J. Russell Ramsay, Ph.D. are co-authors on this study.

Reference: Allenby C, Falcone M, Bernardo L, Wileyto P, Rostain A, Ramsay JR, Lerman C, Loughead J (2018). Transcranial direct current brain stimulation decreases impulsivity in ADHD, Brain Stimulation, online ahead of print. doi: 10.1016/j.brs.2018.04.016

Link to study: https://www.brainstimjrnl.com/article/S1935-861X(18)30138-4/fulltext

APSARD Blog: Sluggish Cognitive Tempo in Adult Outpatients Seeking an ADHD Evaluation

Jessica Lunsford-Avery, Ph.D.

By Jessica R. Lunsford-Avery & John T. Mitchell

Sluggish Cognitive Tempo (SCT) is a set of symptoms including mental fogginess, slowed cognition and behavior, and daydreaming. Researchers are currently uncertain whether SCT is a transdiagnostic construct or a separate diagnosis, and if it is a separate diagnosis, there’s discussion about whether to call it SCT or something else, such as Concentration Deficit Disorder. Either way, this set of symptoms is distinct from ADHD and uniquely contributes to psychiatric and functional outcomes in children. Measures of SCT have also been found to be reliable and valid for use with both children and college students. However, no studies have examined the psychometric properties of a SCT measure among adult outpatients, which limits clinicians’ ability to use it with their patients. Our study sought to examine the reliability and validity of an SCT measure in an applied clinical setting using both self and other report (e.g., spouse or parent) as well as links between SCT and home, occupational, academic, social, and community functioning in an adult sample.

John T. Mitchell, Ph.D.

One-hundred twenty-four adults participated in our study. All participants presented to an outpatient psychiatry clinic for an evaluation for ADHD. Participants completed a thorough psychiatric evaluation. In addition, participants and a person who knows them well, such as a spouse or parent, completed measures of SCT (9-item Barkley Adult ADHD Rating Scale-IV SCT subscale; BAARS-IV) and ADHD symptom severity (Conners’ Adult ADHD Rating Scale; CAARS). Eighty participants met criteria for ADHD and 44 received other primary diagnoses, such as depression and anxiety disorders. Among individuals receiving an ADHD diagnosis, comorbid diagnoses were common, occurring in 53% of adults with ADHD.

Across raters, the SCT subscale demonstrated good internal consistency and yielded three factors: Slow/Daydreamy, Sleepy/Sluggish, and Low Initiation/Persistence. Total SCT score exhibited convergent validity with ADHD symptoms on the CAARS by both reporters. SCT factors were also associated with ADHD symptoms with the exception of the Sleepy/Sluggish factor, which was associated with increased inattention but not with hyperactivity. Adults with ADHD were rated more highly on the SCT measure than those with other psychiatric disorders by the other reporter but not by their own report. When comorbidity was considered, however, adults with ADHD rated themselves more highly on the SCT measure than those without ADHD, but only if comorbidity was present. Finally, greater SCT severity was associated with poorer functioning in home, social, and academic settings by other report, and with deficits in social and community functioning by self-report, after accounting for ADHD.

Our study highlights the potential of SCT measures to inform diagnostic presentation and treatment planning in clinical settings. Specifically, the BAARS-IV SCT subscale can be reliability and validly collected adult outpatient settings and provides important clinical information related to poor functioning over and above the assessment of ADHD. Further, we found that SCT symptoms cluster into three separate factors, as opposed to one factor. Finally, this study underscores the importance of collecting SCT information from someone who knows the patient well, such as a spouse or parent, as their report detected SCT symptoms contributing to functional impairment in additional areas not captured by self-report.

We appreciated the opportunity to present our work at the 2018 APSARD meeting and would like to thank our colleagues at the Duke ADHD Program – Drs. Scott Kollins, Naomi Davis, Julia Schechter, Maggie Sweitzer, Cara Lusby, and Jessica Sloan – as well as Kayla McCoy and Michelle Lepsch-Halligan for their contributions to this study.

 

Internet Communities and Self Medication Among Young Adults with ADHD

Beth Krone, Ph.D.
Icahn School of Medicine at Mount Sinai

In March 2017, a group of young adults emailed me a link to a crowd-funding website to ask my thoughts on a new, direct to consumer health drink being marketed as a cognition boosting, anxiolytic, nutritional supplement. The makers were a group of neuroscience students without clinical licensure. Their all natural product was listed as being a mixture of antioxidants, b-vitamins, and Phenibut.

So what is Phenibut?
Phenibut is a Latvian produced, Soviet developed pharmaceutical that has not been licensed in the United States. In 2018, Australia joined several European nations in regulating Phenibut. Since it has not been licensed in the U.S., it remains unregulated. This means that a quick Google search will lead to a variety of Phenibut products for sale from U.S. based internet merchants or health food stores, since it is not illegal to buy, sell, or possess it.

Phenibut is a gabapentinoid, β-phenyl-γ-aminobutyric acid (β-phenyl-GABA), or a GABA analogue. Known to cross the blood–brain barrier, it is dopamine enhancing in the striatum, has some effects similar to benzodiazepines, may induce euphoria, and there is mixed information about its potential for inducing seizure activity based on its effects on calcium channel activity. There is also mixed information regarding the addiction potential of Phenibut, with at least one published case study supporting many anecdotal reports of it being highly addictive with quick tolerance and intense withdrawal symptoms. When overdosed, Phenibut has also been implicated in Eosinophilia and some fairly severe kidney and liver problems.

With its known side effect profile and abuse potential, Phenibut is a fairly potent psychotropic with pharmalogical properties that require dosing by age and health status. Certainly a health professional would expect this to preferably be done by a competent clinician with prescription privileges and with some type of compliance monitoring for safety.

Bulletin boards and a self-medication community
A quick review of some of the most often used bulletin board sites like Reddit and drugs-forum.com, though clearly shows that young adults do not necessarily share this perspective. Although some users name their preferred formulations, these sites do not advertise a particular drug, company, or formulation. People form communities in support of self-medication on these sites. Lacking resources or unhappy with the care they have received for their ADHD and comorbidities, they share information about unregulated psychotropics. They recount personal experiences of their trials. They freely discuss tweaking their medication regimens with various combinations of prescribed drugs, unregulated supplements, and illicit drugs.

Their discussions are, from a clinical management perspective, very educational. From a developmental or social psychological perspective, it is not surprising that young adults would accept prescribing information from peers without clinical degrees (or names), who often provide quite sophisticated scientific sounding information without references. From a systems or health literacy perspective, though, this highlights the enormity of the clinician’s task in building trust, reaching, and successfully serving the unserved or underserved who cannot find relief through safer channels. This task may be increasingly difficult in context of the larger social conflicts between the medical imperative to prescribe approved and well validated treatments, and the perspectives of a vocal and growing population of people outside the medical community.

To see what some of your patients are reading, check out:
https://www.reddit.com/r/phenibut/comments/4kvbq3/the_beginners_guide_to_phenibut/
https://drugs-forum.com/threads/pharmaceutical-phenibut-physician-patient-instructions-translated-from-russian.221041/
https://nootropicuniverse.com/phenibut-legal-status/