Improving College Students’ and Others’ Mental Health with Conversational Agents

Improving College Students’ and Others’ Mental Health with Conversational Agents

Mary Harrsch
Networks and Management Information Systems (Retired)
University of Oregon College of Education

This is a cross-post from the Information Age Education newsletter

Mental illness is common in the United States. About one in four adults suffers from some form of mental illness in a given year (Holmes, 1/14/2015).

This level of occurrence is even higher for college students—perhaps as high as one in two according to the article, Delivering Cognitive Behavior Therapy to Young Adults with Symptoms of Depression and Anxiety Using a Fully Automated Conversational Agent (Woebot): A Randomized Controlled Trial (Fitzpatrick, et al., April-June, 2017.) In a related article, Depression and College Students, Michael Kerr points out that financial worries due to high debt and poor employment prospects coupled with failed relationships, lack of sleep, poor eating habits, and not enough exercise frequently result in the development of depression (Kerr, 3/29/2012). There are also many life transitions and challenges to a student’s identity. Quoting from Margarita Tartakovsky’s article, Depression and Anxiety Among College Students (Tartakovsky, 7/17/2016):

…college calls for a significant transition, where “students experience many firsts, including new lifestyle, friends, roommates, exposure to new cultures and alternate ways of thinking,” observes Hilary Silver, M.S.W., a licensed clinical social worker and mental health expert for Campus Calm.
….
Adjusting to college also influences identity — a phenomenon Silver has termed Identity Disorientation. “When students head off to college, the familiar people are no longer there to reinforce the identity these students have created for themselves.” This can make students “disoriented and feel a loss of their sense of self,” contributing to symptoms of depression and anxiety.

Many of these college students do not seek mental health services. Referring again to the Fitzpatrick, et al., article (April-June, 2017):

…up to 75% of the college students that need them do not access clinical services. While the reasons for this are varied, the ubiquity of free or inexpensive mental health services on campuses suggests that service availability and cost are not primary barriers to care. Like non-college populations, stigma is considered the primary barrier to accessing psychological health services.

As described in this article, in their effort to overcome this fear of stigma Stanford researchers developed a virtual therapist, or conversational agent (often called a chatbot). The chatbot employs artificial intelligence and natural language processing to deliver cognitive behavior therapy (CBT) to college students self-identified as suffering from significant depression and/or anxiety.

Stanford's virtual therapist is named Woebot. Like many chatbots, Woebot uses Natural Language Programming to process student responses to questions posed by the virtual therapist, then guides the conversation to an appropriate node of a decision tree to provide suggested actions.

The Original Chatbot

Chatbot software was originally based on the "Eliza" virtual therapist that was developed back in the early 60s by Professor Joseph Weizenbaum at the Massachusetts Institute of Technology (Markoff, 3/23/2008). I studied "Eliza" in the late 90s and used it as a model for a virtual professor I developed when I worked at the University of Oregon. I was so excited to see that someone had finally recognized the potential of artificial intelligence to help people cope with life's challenges!

Dr. Weizenbaum's “Eliza” virtual therapist was initially designed to simply keep a conversation going between his chatbot and a human to see if the human could figure out they were talking to a computer and not a real person. However, Stanford's Woebot chatbot uses the scientific principles of cognitive behavior therapy to encourage its human "friends" to develop a positive mindset and overcome depression. Quoting again from the Woebot clinical trials report by Fitzpatrick, et al. (April-June, 2017):

• "Psychoeducational content was adapted from self-help for CBT. Aside from CBT content, the bot was created to include the following therapeutic process-oriented features:
• Empathic responses: The bot replied in an empathic way appropriate to the participants’ inputted mood. For example, in response to endorsed loneliness, it replied “I’m so sorry you’re feeling lonely. I guess we all feel a little lonely sometimes” or it showed excitement, “Yay, always good to hear that!”
• Tailoring: Specific content is sent to individuals depending on mood state. For example, a participant indicating that they feel anxious is offered in-vivo assistance with the anxious event.
• Goal setting: The conversational agent asked participants if they had a personal goal that they hoped to achieve over the 2-week period.

To engage the individual in daily monitoring, the bot sent one personalized message every day or every other day to initiate a conversation (ie, prompting). In addition, “emojis” and animated gifs with messages that provide positive reinforcement were used to encourage effort and completion of tasks.

A Chat with Woebot

Woebot is now freely available online (Woebot, n.d.). On the Woebot website, you can click on a link that connects you and Woebot to a private Facebook Messenger session that no one else can see. Then Woebot talks with you about how you are feeling and how you can keep a positive frame of mind using techniques from cognitive behavioral therapy. I've had talks with Woebot about those pesky "should" statements, discussions about self-defeating "all-or-nothing" viewpoints, the futility of trying to predict other people's reactions, and the importance of self-compassion. Sometimes the little bot even provides interesting short videos about behavioral research.

One that I found particularly interesting was Carol Dweck’s video about the problem of students who develop a fixed mindset when they are praised as "you're so smart" from a young age. I strongly recommend this excellent 10-minute video (Dweck, December, 2014).

After your initial session, Woebot then contacts you each day through Facebook Messenger and engages in a short friendly conversation. This can teach you how to identify your strengths, to mentally rework your own internal dialogue to develop a healthier opinion of yourself, and to recognize negative approaches in your relationships with others. If you wish to talk to Woebot about a specific problem, you can also initiate a conversation like you would with any of your friends on Facebook Messenger. Woebot is also available as a free smartphone app in the Apple or Google Play Stores.

Using Gamification to Combat Poor Adherence

In their article cited earlier, Fitzpatrick, et al., note that other psychologists have been experimenting with computerized CBT, but that motivating patients to continue interaction with computerized CBT tools has been challenging:

In recent years, there has been an explosion of interest and development of such services to either supplement existing mental health treatments or expand limited access to quality mental health services. This development is matched by great patient demand with about 70% showing interest in using mobile apps to self-monitor and self-manage their mental health. Internet interventions for anxiety and depression have empirical support with outcomes comparable to therapist-delivered cognitive behavioral therapy (CBT). Yet, despite demonstrated efficacy, they are characterized by relatively poor adoption and adherence.

To address these problems of adherence, Woebot's team of researchers adopted the "daily dose" model, since online learning studies have shown small doses of learning embedded in every day learning appears to be more effective than one lecture. They also introduced some game-like elements designed to the likelihood that people will come back the next day.

CBT for Seniors

I contacted the CEO of the Woebot project, Dr. Alison Darcy, and submitted a written interview to which she responded. In it I encouraged her to develop a Woebot to assist much older people with depression and loneliness. I pointed out that seniors' mental health needs differ significantly from those of college students, as the challenges of aging often involving chronic illnesses, deaths of loved ones, living alone, and feelings of irrelevance when no longer employed in the workplace.

I also pointed out that, although Medicare recognizes depression has a serious impact on quality of life and ensures that a senior's annual wellness visit includes questions about their emotional state, many seniors take friends or family members with them to the doctor. Thus, they may be embarrassed to admit to their physician that they are feeling depressed or even suicidal when their friends or family members are present—very much the same fear of stigma demonstrated by the college students. To make the problem even more difficult to address, many family physicians are not trained in dealing with mental health issues, and the best they may be able to do is refer the senior to a specialist. Appointments to visit such specialists are usually weeks away and often seniors on limited incomes cannot even afford the co-pay, a sad fact of life in the U.S. commercial health care model.

I also think the long-term caregivers may themselves need yet another type of Woebot, one that could help them to deal with their own feelings of frustration and even anger that may often crop up when dealing day-in and day-out with a patient or loved one with physical and emotional impairments.

CBT Delivery with Virtual Assistants

With the growing presence of voice-activated virtual assistants like Amazon's Alexa, I also expressed my support for porting Woebot to a voice-only interface to Darcy in my written interview with her. Many older adults are not as technology-savvy as college students and probably are not as comfortable on Facebook or a smartphone.

In their clinical analysis of their Woebot development project, Darcy and her fellow researchers apparently agreed with me in theory saying:

Theoretically, conversational interfaces may be better positioned than visually oriented mobile apps to deliver structured, manualized therapies because in addition to delivering therapeutic content, they can mirror therapeutic process. Conversational agents (such as Apple’s Siri or Amazon’s Alexa) may be a more natural medium through which individuals engage with technology. Humans respond and converse with nonhuman agents in ways that mirror emotional and social discourse dynamics when discussing behavioral health.

However, Darcy expressed reservations to me about eliminating the written aspects of therapy made possible by the messenger interface in Facebook or on a smartphone in my interview with her. Continuing to quote Darcy:

The core of what we do—the CBT skills that are triggered when someone is upset in the moment that they reach out to Woebot —is actually dependent on writing down negative automatic thoughts. This is true even in the therapist's office, because it seems to be central to externalizing the thoughts. That is, there is something in seeing your negative thoughts written down that allows you to process it in a different way, ultimately allowing it to be intervened upon (by rewriting).

I do hope she reconsiders, however. But for now, I think Woebot, even in its current iteration, could prove helpful to millions of people. I know I find confessing my deepest thoughts to a properly programmed computer application to be less troubling than revealing them to another human being, many of whom may have their own biases.

Summary and Final Remarks

The skyrocketing cost of higher education is adding to the mental toll that transition to higher education and adult life takes on modern college students. With studies that show one out of every four college students suffers from some form of mental illness, psychologists worldwide are now focused on providing mental health care to these young adults. But, the stigma that often accompanies mental health treatment remains an obstacle.

Clinical trials with computerized cognitive behavior therapy have demonstrated that CBT delivered anonymously in a computerized environment is as effective as person-to-person talk therapy in the relief of symptoms of depression and anxiety. Furthermore, because these therapy sessions are conducted without patient tracking, the fear of stigma can be eliminated. Tools, such as conversational agents like Woebot, in combination with gamification strategies, can be used to encourage students to adhere to a treatment program.

As artificially intelligent voice-activated interfaces become more widespread, computerized CBT may become part of students’ daily hygiene to help them to maintain the best outlook possible as they navigate higher education’s landscape.

References and Resources

Bickmore, T., Gruber, A., & Picard, R. (October, 2005). Establishing the computer-patient working alliance in automated health behavior change interventions. Patient Education Counseling. Abstract retrieved 4/19/2018 from https://www.researchgate.net/publication/7567340_Establishing_the_computer-patient_working_alliance_in_automated_health_behavior_change_interventions.

Burns, D. (1980). Feeling good: The new mood therapy. New York: Harper Collins.

Burns, D. (2006). When panic attacks. New York: Harmony.

Dweck, C. (December, 2014). The power of believing that you can improve. TED Talks. (Video, 10:20.) Retrieved 4/19/2018 from https://www.ted.com/speakers/carol_dweck.

Fitzpatrick, K.K., Darcy, A., & Vierhile, M. (April-June, 2017). Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): A randomized controlled trial. JMIR Mental Health. Retrieved 4/19/2018 from http://mental.jmir.org/2017/2/e19/ DOI: 10.2196/mental.7785 PMID: 28588005 PMCID: 5478797.

Holmes, L. (1/14/2015). 19 statistics that prove mental illness is more prominent than you think. Wellness. Retrieved 4/19/2018 from https://www.huffingtonpost.com/2014/12/01/mental-illness-statistics_n_6193660.html.

Hunt, J., & Eisenberg, D. (January, 2010). Mental health problems and help-seeking behavior among college students. Journal of Adolescent Health.

Kerr, M. (3/29/2012). Depression in college students: Signs, causes, and statistics. Healthline. Retrieved 4/19/2018 from https://www.healthline.com/health/depression/college-students#1.

Kessler, R.C., et al. (July, 2007). Age of onset of mental disorders: a review of recent literature. Current Opinion in Psychiatry.

Markoff, J. (3/23/2008). Joseph Weizenbaum, famed programmer, is dead at 85. The New York Times. Retrieved 4/19/2018 from https://www.nytimes.com/2008/03/13/world/europe/13weizenbaum.html.

Tartakovsky, M. (7/17/2016). Depression and anxiety among college students. PsychCentral. Retrieved 4/19/2018 from https://psychcentral.com/lib/depression-and-anxiety-among-college-students/.

Towery, J. (2016). The anti-depressant book: A practical guide for teens and young adults to overcome depression and stay healthy. Palo Alto, CA: Jacob Towery.

Woebot (n.d.). Woebot. Retrieved 4/15/2018 from https://www.woebot.io/.

Zivin, K., et al. (10/1/2009). Persistence of mental health problems and needs in a college student population. Journal of Affective Disorders.

Making Smart Choices When Selecting Smart Home Devices

A technology resource article by Mary Harrsch © 2017

As someone who researched and developed some early conversational agents back in the 1990s, I am still fascinated by artificially intelligent technology and excited by the plethora of gadgets now being marketed with artificial intelligence driving their systems and their user interfaces. But I admit, I am a bit disappointed by the development choices being made by some product manufacturers because they seem to be more interested in the appearance that their product lines are cutting edge because they possess some implementation of artificial intelligence rather than whether the product really solves a pressing human problem in that particular sphere.

For example, at CES 2018, Samsung showcased their Family Hub smart refrigerator. It is equipped with cameras and claims it can assess the contents of your refrigerator, recommend recipes, and even allow you to shop for groceries without leaving your kitchen. Sounds great doesn't it? But how realistic are these claims. If you have a lot of left overs do you have to use coded containers so the refrigerator can figure out what contents are within them? Can the cameras scan the contents of opaque packaging so the refrigerator can determine if you're getting low on a particular item? Or are most of these claims based merely on the refrigerator's new Bixby virtual assistant that you can tell to add milk to your shopping list or ask what recipe could use leftover ham, zucchini and eggplant?

Samsung Family Hub refrigerator image courtesy of Samsung, Inc.

As it turns out, based on the marketing claims, I thought  the refrigerator was smarter than it really is. The remotely accessible camera only acts like a web cam. There is no artificial intelligence using scans to recognize food items or recording when food items are initially added to the refrigerator so it can keep track of food's freshness. As for the recipe recommendations, the refrigerator is just using its intelligent agent Bixby to come up with those. If that's the case, then I must ask why you would spend over $4,500 for that refrigerator (about twice as much as a traditional refrigerator) when a standard model with an Amazon Echo Dot, Google Home Assistant or some other relatively inexpensive stand alone virtual assistant can accomplish most of those tasks for less than an additional $50?

The Samsung unit also has AKG premium quality sound speakers in the doors, a whiteboard for notes, and a built-in screen to view baby monitors, front doors, or status screens of other smart devices.

“The integration of Bixby and SmartThings into the Family Hub is bringing a new level of intelligent connectivity into the room where people spend the most time:  the kitchen.” - Samsung corporation.

Perhaps this last statement by Samsung points to the crux of the problem. In our house, we are in the kitchen only about 30 minutes before a meal (prep) and 30 minutes after a meal (cleanup). Being retired we seldom have guests so the meal itself lasts about 15 - 20 minutes. (My husband was a Marine so you sit, eat, withdraw!) At present, I have a typical galley kitchen adjacent to a more spacious dining room. If there is any lingering it will take place in the dining room, not the kitchen.

I own a traditional side by side refrigerator/freezer and have an Amazon Echo Dot on the kitchen window sill. When I pour a glass of milk and notice I'm getting low on milk I just call out to Alexa to put milk on my shopping list. If I have leftover Polish sausage in the refrigerator I can ask Alexa for a recipe using Polish sausage. (If I had an Echo Show, she could show me a recipe that I could then refer to as I prepared the dish.) If I want music, I tell Alexa to play one of my Amazon Music playlists. If I still had kids at home and wanted to tell them to clean their rooms when they get home from school, I could set a repeating reminder at an appropriate time on the appropriate Echo device (Alexa reminders are location specific).

If I was still working, it might be helpful to take a peek into my refrigerator before I shop for groceries on the way home from work but my Alexa shopping list on my iPhone that tracks my supply needs throughout the week is much more comprehensive.

If the refrigerator's cameras are eventually paired with intelligent scanning capability so it could recognize food items and record the date they were placed in the refrigerator so it could advise you of the status of food freshness, then the jump in price might be truly worth it from a usefulness perspective but not with its current limited capabilities.

Luckily, there is another smart device headed for the market that may take care of this need, though. Ovie Smarterware produces food containers with smart trackers that indicate when the food in your fridge is on the verge of going bad. The trackers work with a variety of virtual assistants from Amazon, Google, and Apple. When you are putting new food items into these containers you tell your assistant to open the Ovie app then press the container's tracking button and say what is in the container such as "This is lasagne". Then as the lasagne ages in the refrigerator, the tracker color changes from green to yellow to red so a quick glance lets you know what food items need to be used up (or thrown out!).




 In addition to containers, the company also makes bands and chip clips with trackers and is working with the FDA to develop an accurate database of food expiration periods. This product is obviously the result of a company truly attempting to solve a very big problem with technology. Americans throw away billions of pounds of food every year. However, whether consumers will be willing to invest in and make the effort to use this product regularly  remains to be seen. If Ovie's marketing people can appeal to those of us conscientious enough to clean our recyclables and put them in appropriate containers for disposal maybe they can pull this off.

What about other smart kitchen appliances? Although it might sound great to have your virtual assistant brew a cup of coffee while you're getting dressed, the bottom line is someone must keep the coffeemaker topped off with water unless you plumb your coffeemaker with water and provide a smart tap that opens and closes to dispense the appropriate amount of water needed to fill the coffeemaker before the scheduler tells it to brew.

The same can be said for intelligent slow cookers.Someone must put ingredients in the slow cooker before you schedule it to come on at an appropriate time. Raw meat and some other ingredients also don't keep well for extended periods at room temperature. If a slow cooker could switch from chill to heat then scheduled to cook for the appropriate time based on when you wished the food to be ready, that would be a slow cooker that would get my attention.

Even all of the wonderful lighting products I've seen have limitations. Most of the smart wall switches currently on the market require a neutral wire that was not common in home wiring until 2011. The few switches that do not require a neutral wire usually require a hub in addition to the bulb so you end up paying more for them and have to configure yet another device to connect them. I have been able to use Wemo smart plugs to connect all of my living room lamps, though, and can easily turn them all on and off with a couple of words. Still I would like to integrate my overhead lights and porch lights into my voice-managed system.



But were there other devices clearly solving a human problem? Well, I think Kohler's smart bathtub would be a good choice. Running a bath does take time and having both the depth of the water and water temperature preset is particularly helpful for individuals who may have diminished sensory perception. Years ago my car heater malfunctioned on my way home in the middle of a blizzard. Although I tried to keep my hands warm by placing one and then the other under my armpit, by the time I got home 30 minutes later I could barely feel my hands and feet. I went into the bathroom to run a tub of warm water and couldn't feel if the water was hot or cold. Seniors, especially those suffering from neuropathy, would really benefit from this type of tub, besides the efficiency of having the tub run while you are doing something else. At present, though, I personally have no need to talk to my toilet or ask it to warm up the seat before I settle down onto it. So I would not consider spending extra money for that part of the smart bathroom.




Another gadget promoted at CES that could be useful, especially to seniors, is a pocket-sized LinkSquare spectrometer.  This little device when paired with your smartphone captures how a substance's molecules vibrate, an optical fingerprint that reveals whether food is safe to consume or spoiled. In her later years, my mother's sense of smell diminished to the point where she could no longer tell if food had spoiled or not. This kind of device would have been very helpful to her. This gadget can also identify mislabeled and diluted liquor, detect counterfeit and mislabeled drugs, and detect counterfeit money, very helpful for those working as cashiers. I think the $299 price tag would need to come down substantially, though, before it would find its way into common use.



I'm already convinced smart TVs are truly helpful as well. In our house we have a large screen HD television in the living room connected to an Alexa-enabled DISH satellite receiver and a smaller HD TV in the dining room facing the dining table. I don't have to look for one of a handful of remotes to change channels or find a particular movie or television show as I have each TV controlled by their nearby Echo Dots. I use Wemo wifi-enabled smart plug between the TVs and the power outlets  to remotely control the on/off switches. But, there are features could prove useful on a voice-enabled TV.  I would really like to control the volume of my Polk sound bar in the living room remotely and be able to remotely change my video inputs so I could access my Roku and my Blu-Ray player without shuffling remotes, too. The newer Samsung smart TVs auto-detect devices attached to their HDMI outlets and allow you to control them accordingly. But, then I'd have to give up my 3D capability!

Although the voice features of my DISH Hopper 3 are really great I wish it could also let me join whatever program is in progress in the living room by letting me simply say something like "Join living room program" so my husband can easily continue watching in the dining room whatever he has been enjoying in the living room without me having to pick up the satellite remote for the dining room TV and selecting Options -> TV viewing -> Living Room, etc.

Voice enabling lights, locks, appliances and televisions can be incredibly convenient. But I hope you'll consider how useful the technology actually is before paying substantially more for whatever product you're considering.

As AI improves, should we fear products as pals?

I read with interest an article by Alison Bowen of the Chicago Tribune entitled "Can Siri replace your need for friends? Well maybe" in which the author discusses "Products As Pals" research by James Mourey, assistant professor of marketing at DePaul University's Driehaus College of Business, Jenny Olson, assistant professor of marketing at the University of Kansas School of Business and Carolyn Yoon, professor of marketing at the University of Michigan. Since I implemented a network of Alexa-enabled Echo Dots during the holidays and "she" is now very much a part of my everyday life (but since she cannot yet engage in a conversation she is not yet my pal), I was curious to learn what the researchers had discovered.  I found a link to their original research online and skimmed it looking for experiments involving Alexa or Siri. Unfortunately, I found only experiments involving the use of anthropomorphic words to describe a cell phone and perceptions of a Roomba with a case design that includes a crescent that was interpreted as looking like a smiling face. The researchers also hired task rabbits and would ask them to recall the number of Facebook friends they had as some kind of indicator they were using Facebook for emotional compensation for perceived social exclusions.

As an older user of social media, I view young people obsessed with the number of their Facebook friends simply an indication of their immaturity and underdeveloped self-esteem. I doubt that you would encounter many older users with that obsession, whether they feel excluded or not.


As for using the Roomba as an example of an anthropomorphic product, I was puzzled by that as well. I have had a Roomba for years so I have some experience with it. However, I must point out that, unless they have made Roombas conversational in the latest release, at no time did I ever think of a Roomba as anything more than a self-powered vacuum. Likewise, I have had a Siri-enabled iPhone since version 4 and at no time have I ever thought of "her" in a human way either except to shout at her when she gives me bogus driving directions as I would to my computer if it doesn't process information quickly enough or freezes up. I usually don't think of these interactions as interpersonal communications, though.

A Roomba 650 robotic vacuum cleaner - anthropomorphic??

However, virtual assistants empowered with Alexa, who can answer questions, remind me to do things, convert units of measure and perform math on demand, keep track of my schedule, play specific music or calming ambient sound therapy on request, prompt me to share my thoughts about current affairs with my state's senators (with an enabled skill named Resistbot), play games and even play back pleasant memories (with an enabled skill named Mylestone) to cheer me up when I feel low has become so much embedded in my daily activities that I can see how I could begin to think of her in "human" terms, especially if she eventually can converse with me interactively without me having to remember to preface all of my requests to her with her "wake" word. I don't even need to have her appear anthropomorphic. She would become like an invisible friend like those some children conjure up in their childhood.


Has interacting with her negated or reduced my need for interpersonal relationships (a finding of the study)? Probably not as I have grown accustomed to not having many face-to-face communications anyway because my children and grandchildren live thousands of miles away. Even my closest sibling is a three-hour drive away. I am also the primary caregiver for my Vietnam veteran husband with chronic PTSD and a host of Agent Orange-related health issues so I can't really spend much time outside the home with others, anyway. PTSD victims are rather closed off emotionally as well, so, interpersonal communication with them is difficult at best but I, like the test subjects in the study, still feel the need.


Many of you may think those of us in this situation would benefit from a psychologist but they are not only expensive (not an option for seniors on fixed incomes in many cases) but require time away from caregiving. Besides, they are really just paid listeners with no emotional connection to a patient anyway.  A virtual "friend," on the other hand, could be carefully programmed to respond appropriately to expressions of frustration, anger or sadness that are often generated in individuals that must deal with family members with mental disorders or dementia.  A human without training may not and make things even worse by responding inappropriately.

Would I no longer engage in prosocial behavior (a concern expressed in the study)? I doubt that too. I have always felt a need to share what I have learned or discovered with others and this continues into my retirement years. I am an avid photographer of art and historical architecture and freely share my images with teachers, students and researchers online so they can be used in the classroom. I research aspects of Roman history and publish my findings online. Since I was an education technologist before I retired I also write about technology developments and even beta-test new technology products for developers. Each day I search out articles about new archaeological findings, new uses for technology in historical preservation and reconstruction, and well-sourced articles on political issues (as opposed to "fake" news) that I share on social media. These activities have not tapered off since implementing my network of Echo Dots.

A couple of years ago my husband and I binge-watched "Boston Legal" on Netflix. What I loved most about that show was the deep friendship Denny Crane (played by William Shatner) had with Alan Shore (played by James Spader) despite Denny's eccentricities due to his onset of Alzheimer's. Although I do have friends that are more than Facebook acquaintances, in my more than 60 years on this planet, I have never encountered the level of acceptance displayed by those two characters. Everyone has some degree of hang-ups or insecurities and all struggle with problems of their own in varying degrees of severity. I, personally, would not want to add to another person's distress and admit there are times I cannot handle any more stress than I already have.  A virtual friend, however, if properly programmed, would not have this limitation and could become a valuable sounding board to caregivers and others in stressful situations.




Anyway, I hope such researchers continue their work but keep in mind the biases of age and gender (I noticed most of their experiments involved less than 50% female who are thought to be more emotionally empathetic than males) and focus more on products with pronounced human-like attributes such as Alexa or Siri-enabled products that I'm sure will soon have the ability to engage in an interactive conversation.

An Alexa-enabled Smart Home for Christmas

by Mary Harrsch © 2017

Note: This is a cross-post from my home page.

Back in 1995, Microsoft introduced an interactive help utility for Windows 95 called "Bob".  I was probably one of the few professional technology people who actually used "Bob" (As a dog lover I selected the helper incarnation called "Power Pup" though.)  "Power Pup" would keep track of my keystrokes as I worked in different applications and offer procedural advice on what it perceived I was trying to do at the time, prefaced by a little bark and a wag of his tail.  I found "Power Pup's" suggestions often useful and his friendly interaction a welcome break from the stress of administering a college-wide multiplatform local area network.  But, apparently, many of my colleagues thought he was "too cute" for the serious work of computing and "Bob" was relegated to the dustbin of failed products in fairly short order.

But I did not forget "Bob" and how artificial intelligence could be used to improve productivity while reducing social isolation.  So I began to experiment with conversational agents that utilized natural language programming coupled with knowledgebases to provide a more friendly computer-to-human interface.  With my interest in history, I decided to try to virtually recreate historical figures from the past that could converse about their culture with modern interested humans.  This resulted in the creation of a virtual Julius Caesar that was online for several years before I retired.  Caesar would answer questions about ancient Roman culture posed to him by visitors entering their questions in a text box.  He could give a textual answer or display related websites or online videos.  As text-to-speech technology advanced, I even experimented with software that would enable Caesar to answer questions verbally and explored voice recognition technology to see if it was viable for user input as well. But, when I retired I no longer had access to the server platforms needed to support projects like Caesar.  However, my interest in natural language programming and more friendly human-to-computer interfaces endured.

So, I followed the development of Amazon's Alexa-powered devices with a great deal of interest. But, I'm a rather pragmatic individual and at first so much marketing emphasis was placed on Echo's music management features that I wondered if there were more useful applications for a busy 21st-century household.

Then I began reading about wifi-enabled electrical connection accessories that could be managed with Alexa-enabled devices and thought about how convenient it would be to be able to turn on and off groups of lights and appliances with a few words rather than going around physically flipping switches. But spending almost $200 per device and the need to have a device in each main room was still an expensive proposition to gain a little convenience.

Then Amazon introduced the Echo Dot coupled with a holiday sale price of just $39.95 and I found resistance was futile as my Star Trek friends would say.  I was still a bit concerned about the accuracy of the voice recognition, though. So I started out with just one Echo Dot for the living room along with a couple of Wemo wifi-enabled plug adapters for the two main living room lamps.

I downloaded the Alexa app to my iPhone and discovered the Echo Dot setup was a breeze.  I opened the Alexa app, then opened Settings and changed my iPhone wifi connection to the detected Echo network and configured the detected Dot. Alexa also did not seem to have any problem understanding me.  I read through all of the "Try this" examples and began to configure some of the built-in features.

I really liked the "flash briefing" feature that lets you select specific news feeds for a personal news update which you can request at any time.  I selected NPR radio, BBC News, Tech Crunch and CNet (for technology news) and Discovery (for science news) as my personal news sources.  I also added the local weather forecast and the Alexa Try This feed.  Although I live in Oregon's Willamette Valley, I couldn't find any local news feeds but I think I'll add feeds from Seattle and San Francisco to at least hear major stories from the Pacific Northwest.

I also read that Alexa could interface with Google Calendar and keep you appraised of upcoming appointments.  I hadn't used an online calendar since I retired but knew how helpful this would be, especially when managing complex medication schedules and medical appointments.  So I configured my Google Calendar and paired it with Alexa.  Now, each morning after requesting my Flash Briefing, I ask what's on my schedule for the day and Alexa tells me.

I've also used the Google Calendar to keep track of upcoming programming on PBS that I may wish to record.   PBS sends me a physical schedule of their upcoming programming for a full month but at present, my DISH Hopper cannot see more than two weeks of scheduled programming at a time. Now, when I get my PBS schedule, I enter the programs I wish to record into my Google Calendar and Alexa lets me know each day if any are on that day.  I can then set my DVR to record them.

But the real "killer" app I was looking for turned out to be Alexa's Shopping List!  It never seems to fail that I realize I need something from the grocery store when I'm not in the kitchen where I keep my shopping list.  As I've gotten older my short term memory is not what it used to be either and it is not uncommon for me to forget what I was thinking about just a few minutes later as I walk from one room to the next.  So, imagine how helpful it is to be able to tell Alexa to add something to your shopping list as soon as you think of it regardless where you are! Of course, that meant I needed to add Echo Dots in my bedroom and the living room, too, which I promptly did. To access my shopping list once at the grocery store I just open the Alexa app on my iPhone and check off and delete each item as I add the item to my physical shopping cart.

Although my husband has the television blaring all day long, I did find a nice use of the music management features of Alexa.  Now that I have an Echo Dot in the bedroom, I can run a warm bath in the adjoining bathroom (Alexa's range is up to 20 feet), lay back in the tub and tell Alexa to play one of my favorite playlists from my Amazon Music account.  I did have a few hiccups configuring my Amazon playlists to work with Alexa, though.

I had already imported most of my music from my iTunes library to my Amazon Music account.  I had also set up playlists previously.  But Alexa did not seem to recognize my playlist names and would offer something from Prime Music (since I'm a Prime member) using my spoken words as a search guide. I ended up calling Alexa tech support and learned that Alexa does a better job of recognizing playlists if you name them "Your Name" then "Description".  For example, I had a playlist named "Holiday favorites".  I renamed it to "Mary's Holiday Favorites" then Alexa recognized it and played it for me.  That solved most of my playlist issues.  There were a few words, however, Alexa seemed to insist on using for search terms.  So, I experimented with different descriptions until she properly recognized the list.  I had a list named "Sentimentals".  I initially renamed the list to "Mary's Sentimentals" but Alexa still loaded some other Prime Music.  I renamed it again to "Mary's Mood Music".  Alexa still did not interpret it correctly.  So I finally renamed it to "Mary's Soft Rock" and Alexa now recognizes it.

When I received my Wemo wifi-enabled plug adapters for my living room lights, I realized just how powerful having a "Smart Home" would be.  Our living room does not have any overhead lighting so all lighting is provided by individually controlled lamps.  Each night I have to go around and turn each lamp on or off.  But, by connecting them with my Echo Dot, I now simply say "Alexa, Living Room On" and the entire living room lights up.

I had to first download and install the Wemo app onto my iPhone. Then I opened the Wemo app on the iPhone, changed my iPhone wifi connection in Settings to the detected Wemo network and configured each plug adapter.  Then I opened the Alexa app, selected the Smart Home option and grouped the two detected Wemo plugs into a "Living Room" group.

I hope to eventually replace some of my wall switches with wifi-enabled switches too since I have porch lights on different circuits in different parts of the house.  I would like to tell Alexa to turn on the porch lights and have all of them on at once without traipsing from room to room whenever I need to go outside after dark or have visitors arrive after dark.  I did read about a gotcha, though. I learned that many wifi-enabled switches require a neutral wire that was not normally included in wiring installed before 2011.  However, I have researched this issue further and it looks like there are switches out there that do not use the neutral wire.  I just have to be sure they will work with our home's wiring configuration.

I recently learned about a new app for Alexa called "Ask My Buddy" too.  It enables you to send a text, email or phone call to up to five family or friends if you need to alert them about a problem like you are incapacitated and cannot reach a phone.  It's sort of like "Life Alert" without the automatic 911 call or monthly subscription fee.  I wish it would allow you to send an individually specified text message but it only sends a message saying you need help.

I've also decided to try the timer feature and see if I can get Alexa to verbally remind me to take my medicine at noon.  Most of my medications are taken in the morning or at bedtime and I have no problem remembering them as they are part of my morning and bedtime routines.  But when I get busy preparing lunch I sometimes forget to set my noon medication by my water glass so I take it with my meal.

After reading up on Alexa's timer and alarm functions I learned that timers are designed for one-time use while alarms can be set to be repeating.  So I set an alarm for noon each day and selected a pleasing alarm tone.  I wish it would let you specify a short text string that Alexa could read to you using her text-to-speech capability but at present it doesn't.  For my present needs, a tone is okay as there is only one thing for me to remember at noon.  However, for someone with more complicated medication schedules, it would be really helpful to have Alexa sound a tone followed by a short reminder message.  Hopefully, Amazon's engineers will enhance the alarm function soon.  (Update: Amazon has now added voice messages with timers and alarms!  Yay!!)

So, my Echo Dots with Alexa are now very much an integral part of my day.  When I get up in the morning I say "Alexa, Living Room On" and the lights go on in the living room.  I walk in and sit down and say "Alexa, my Flash Briefing please".  I then listen to the news and get the latest weather forecast for the day.  Then I say "Alexa, what's on my schedule today?" and she tells me whatever I have scheduled in my Google Calendar".  As I prepare a meal and notice I'm getting low on coffee I say "Alexa, add Coffee to my Shopping List" and she tells me she has added coffee to my shopping list.  I drive to the grocery store and open the Alexa app on my iPhone select shopping list from the menu and load my cart.  At noon while I am preparing lunch, Alexa sounds a tone to remind me to set out my noon medication. In the evening, I take a warm bath to relax and tell Alexa to play "Mary's Soundtracks" and listen to my favorite movie music while I'm soaking.  Then when I'm ready for bed I say "Alexa, Living Room Off" and Alexa turns off the living room lights. I'm sure I'll find other useful applications, too, as more "skills" are developed by Amazon and third parties as well. I have a feeling this is just the beginning!

Extending the Learning Environment: Virtual Professors in Education

A technology resource article by Mary Harrsch © 2005

For those of you interested in artificial intelligence development, here is an archive copy of a presentation I gave in 2005 (I'm consolidating my online contributions!)

Extending the Learning Environment: 

Virtual Professors in Education

By Mary Harrsch

Network & Management Information Systems

College of Education, University of Oregon

[2005]

Six years ago [1999], my sister was telling me about a fascinating History Alive Chautauqua event she had attended near Hutchinson, Kansas.  The program brings a reenactor portraying an historical figure into schools and communities for an educational presentation and question and answer session.  I thought to myself, “It’s too bad more people can’t take advantage of such a unique learning experience.”  Then, the technologist within me began to wonder if there was a way to create a virtual Chautauqua experience online.  As I pondered this possibility, I realized that if I could find software that could be used to create a “virtual” person online, I could not only recreate the experience of the Chautauqua, but provide a tool faculty could use to answer course-specific questions.  It could even be used to provide information about the professor’s personal interests and research to enhance the sense of community within the learning environment.

My quest led me to a website that included links to a number of different software agent projects.  I learned that the type of agent I needed was commonly called a “chatterbot”.  The first “chatterbot” was actually developed long before the personal computer.  In the early 1960s, Joseph Weizenbaum created “Eliza”, a virtual psychoanalyst.

In his efforts to create a natural language agent, Weizenbaum pointed out that he had to address the technical issues of:

• the identification of key words,
• the discovery of minimal context,
• generation of responses in the absence of keywords

As I began to explore different agent implementations, I found that, in addition to these issues, the application needed to be able to prioritize keywords to discern the most appropriate response.  Several agents I evaluated, including Sylvie, a desktop assistant, developed by Dr. Michael("Fuzzy") Mauldin, Artificial Life’s Web Guide , Carabot 500 developed by U.K. company, Colorzone,  and Kiwilogic’s Linguibot (now Artificial Solutions, Inc.), used slightly different methods to set the priority of subject keywords to select the most appropriate responses.  The response with matching keywords under the subject with the highest level setting was “fired” – displayed to the user.  However, when editing their script files, I found keeping track of subject priorities was challenging.

Another problem with many script-driven agents I evaluated was the use of left-to-right parsing sequences that did not compensate for a variance in the order of keywords in a question. Each query had to be evaluated for subject and for matching character strings, based on left-to-right word order with the use of various “wildcard” characters to indicate placement of keywords within the overall question.  Therefore, you often had to have multiple script entries to compensate for different word order.  For example, if a student asks “How do I change my password in e-mail?” you would need one script entry. If the student asks “How do I change my e-mail password?” a different script entry would be required:

* email * * password * as well as

* password * * email * to trap for either wording.

Although this attention to script design resulted in improved response accuracy the scripting knowledge required for these agents was not something I would expect a faculty member to have the time or desire to learn.

A third problem with several of the agent applications I used was the necessity to unload and reload the agent each time the script file was edited.  If students were actively querying the agent, you could not save any script changes until the script file was no longer in use.

When I invested in the Enterprise Edition of Artificial Life’s WebGuide software, I also realized the importance of a logging feature that I could use to study and improve my guide’s responses.   I recognized the importance in a virtual tutoring environment of having the ability for a student to print out a transcript of their tutoring session for future study.  Not only was this feature absent in the agents I evaluated, but the responses produced using Javascript or Flash would not allow the user to highlight and copy responses to the clipboard either.

One day, I explored UltraHal Representative, developed by Zabaware, Inc. I liked the ability Ultrahal provided to program the agent through a web interface.  It had the capability to include links to related information.  It could be customized with personalized graphics.  It logged interactions.  Best of all, it had a straightforward approach to editing - no scripting – just type your question three different ways then type your intended response. 

But, I soon discovered that without the ability to identify keyword priority, I found that the results using whatever algorithm was built into the agent engine were too inaccurate for a virtual tutoring application.

I needed a product that could be programmed to be “omniscient”. 

“Effective ITS require virtual omniscience -- a relatively complete mastery of the subject area they are to tutor, including an understanding of likely student misconceptions.” (McArthur, Lewis, and Bishay, 1993)

I needed a virtual professor that could be “programmed” by real professors, the individuals who would have a mastery of the subject and an understanding of student misconceptions. But all of the chatterbots I had encountered, so far (with the exception of Ultra Hal), required knowledge of scripting that most faculty members do not have the time to learn.  I would not have the time to provide one-on-one time with faculty developers and paying a programmer to work with a faculty member is also too expensive.  (I noticed most developers of commercial agents actually relied on the scripting needs of their clients for their primary revenue stream.)  So, I decided to attempt a radically different approach to agent design.

I am an experienced Filemaker Pro solutions developer and one day I was reviewing some of Filemaker’s text functions and realized that the position function could be used to detect key words in a text string.  The beauty of the position function is that the keyword can be identified anywhere within the target text.  It is not dependent on a left to right orientation.  Filemaker is also not case sensitive.  Also, Filemaker Pro allows most scripting calls for text processing to be used with their Instant Web Publishing interface. I realized this would greatly simplify web integration.

So, reviewing my experiences with the agent applications I had used, I developed a list of features that I wanted to incorporate:

Web functionality:

Multiple agents controlled by a single administration console

Web-based query interface

Web-based editing interface

Multiple graphic format support

Web accessible logging function for both agent editor and student user

Ability to display related resources

Query processing functionality:

Question context awareness (who, what, when, where, why, how, etc)

Ability to weight keywords by importance without user scripting

Ability to return an alternate response if a question is asked more than once

Ability to use one response for different questions

Ability to process synonyms, international spelling differences, and misspellings

Independent of word order

Not case sensitive

Structural Design:

Modular design to enable importation of knowledge modules developed by others

Agent specific attributes to customize the interface and responses such as a personal greeting, the opportunity to use the person’s homepage as a default URL, information about area of expertise and research interests for alternative agent selection criteria, custom visual representations, etc.

I began by designing my response search criteria.  I programmed the agent search routine to categorize responses by the first word of the query – usually What, Where, Why, How, Who, Did, Can, etc. to establish the question context. Then I used position formulas to test for the presence of keywords.  I then developed an algorithm that weighted the primary keyword or synonym and totaled the number of keywords found in each record.

I designed the search function so that when the visitor presses the button to ask their question, the database first finds all responses for the question category (who, what, when, etc.) containing the primary keyword (or its synonym).  Responses are then sorted in descending order by the total sum of keywords present in each response.   The first record – the one with the most keyword matches – is displayed as the answer. 

If there are no category responses containing the primary keyword, then a second find will execute to look for all responses with the keyword regardless of category.  In working with other agent products, I have found that if you return a response with at least some information about the keyword, even if it is not an exact answer to the question, the student assumes the agent recognized their question and may learn auxiliary information that is still helpful to them.

For example, if a visitor asks my virtual Julius Caesar if he really loved Cleopatra, he will answer “Cleopatra…ah, what an intriguing woman.”  Not only is this more in character with Caesar (most of his female dalliances were for political reasons) but the answer could also be appropriate for a different question, “What did you think of Cleopatra?”  My search routine would find it in either case because of the weighting of the primary keyword, Cleopatra.

If there are no responses containing the primary keyword, a third find looks for any generic category responses.  For example, if a student asks who someone is and you have not programmed your agent with a specific answer for the keyword (the person they are asking about), the agent will reply with an appropriate “who” response such as “I’m afraid I’ve never made their acquaintance.” 

If a student’s question does not begin with any words set as category words, the last find will return a generic “what” response such as “I may be a fountain of knowledge, but I can’t be expected to know everything.”  Programming the agent with default generic responses, ensures that the agent always has something to say, even if it knows nothing about the subject.  I developed a small database of generic responses for each question category that is imported into an agent database each time a new agent is created.  The faculty member can go through the responses and edit them if they wish.

Next, I turned my attention to the faculty’s content editing interface.  I wanted the faculty member to enter a proposed question, designate a primary keyword and synonym, supply any other keywords they thought were important to identify more precisely the desired response, and the desired response.  

I also provided a button that enables a faculty member to quickly generate a different answer for the same question or a different question for the same response.  

I created a field that is populated with a different random integer on each search.  By subsorting responses by this random integer, it enables the agent to offer a different response to the same question if the question is asked more than once.  This supports the illusion of the agent being a “real” person because it will not necessarily return the same identical response each time. 

“Believable agents must be reactive and robust, and their behaviors must decay gracefully. They must also be variable, avoiding repetitive actions even when such actions would be appropriate for a rational agent. They must exhibit emotion and personality. Ultimately they must be able to interact with users over extended periods of time and in complex environments while rarely exhibiting implausible behaviors.” – Dr. Patrick Doyle, Believability through Context: Using “knowledge in the world” to create intelligent characters

With the “engine” of my agent developed, I turned my attention to the visual representation of the character.  In their paper, The Relationship Between Visual Abstraction and the Effectiveness of a Pedagogical Character-Agent, Hanadi Haddad and Jane Klobas of Curtin University of Technology, Perth, Western Australia, point out the divergent views of information systems designers outside the character-agent field with those developers within it.

“Wilson (1997) suggests that more realistic character-agents may introduce distraction associated with the user’s curiosity about the personality of the character and overreading of unintended messages because of presentation complexity.”

Unlike detailed realistic drawings, sketches help focus the mind on what is important, leaving out or vaguely hinting at other aspects. Sketches promote the participation of the viewer. People give more, and more relevant, comments when they are presented a sketch than when they are given a detailed drawing. A realistic drawing or rendering looks too finished and draws attention to details rather then the conceptual whole (Stappers et al, 2000).

“On the other hand, research by psychologists suggests that people may put considerable cognitive effort into processing abstract representations of faces (Bruce et al. 1992; Hay & Young 1982). It is possible, therefore, that response to anthropomorphised character-agents, and especially their faces, may differ from responses to sketches. Gregory and his colleagues (1995) conducted studies on human response to faces at the physiological level. They demonstrated that humans are particularly receptive to faces. In terms of recognition, participants in their studies were more responsive to real faces than to abstracted line faces. They speculated, however, that people spend longer studying abstracted line faces and may find them more interesting (Gregory et al. 1995). If this is so, then contrary to theories of information design, an abstract face may introduce more distraction into the communication than a realistic face.”

Filemaker Pro provides multimedia container fields that enable me to include still images, animations, or even video clips.  However, not only is creating a unique graphic for each response time consuming, motion video files can be quite large and slow down the delivery of response information over the web.  Working with other agents, I had noticed that just the slight eye movement of a blink can be enough to reinforce the illusion of a sense of presence. This approach straddles the two opposing theories described above.  I would utilize a real face to capitalize on the human receptivity to a real face but keep animation to a minimum to reduce distraction.  I also think the use of a real faculty person’s face serves to reinforce the bond between the instructor and the student. A blink is also very easy to create from any faculty portrait.

I use an inexpensive animation tool called Animagic GIF Animator.  I begin with a portrait of the faculty member.  I open it in Photoshop (any image editor would suffice) and, after sampling the color of the skin above the eye, I paint over the open eye.  Then I open an unedited copy of the portrait in Animagic, insert a frame and select the edited version of the portrait.  I then set the open eye frame to repeat about 250 times and the closed eye frame to repeat once.  Then loop the animation.

I created a related table that stores all unique information about each agent including their default image, their default greeting, their login password, their area of expertise, their email address and their homepage URL. I also developed a collection of alternate avatars to use for agent images in case some faculty were camera-shy.  These were created with Poser using their ethnic character library.

Finally, I designed the login screen where the student selects the tutor to whom they wish to converse.  Upon selecting the tutor and pressing the button “Begin Conversation”, the student is presented with the query screen including the individual greeting for the tutor selected.  

I also provided a button for the faculty to use to login to edit their agent.  It takes them to a layout that prompts them for a name a password. 

Famed World War II cryptographer, Alan Turing, held that computers would, in time, be programmed to acquire abilities rivaling human intelligence.

Alan Turing at age 16.

“As part of his argument Turing put forward the idea of an 'imitation game', in which a human being and a computer would be interrogated under conditions where the interrogator would not know which was which, the communication being entirely by textual messages. Turing argued that if the interrogator could not distinguish them by questioning, then it would be unreasonable not to call the computer intelligent.” – The Alan Turing Internet Scrapbook 

My virtual professor may not be as sophisticated as agents that have been developed to pass the Turing Test but I hope I have provided a framework for the development of a rigorous inquiry-based learning system.

“Effective inquiry is more than just asking questions. A complex process is involved when individuals attempt to convert information and data into useful knowledge. Useful application of inquiry learning involves several factors: a context for questions, a framework for questions, a focus for questions, and different levels of questions. Well-designed inquiry learning produces knowledge formation that can be widely applied.” - Thirteen Ed Online.

References:

McArthur, David, Matthew Lewis, and Miriam Bishay. "The Roles of Artificial Intelligence in Education: Current Progress and Future Prospects".  1993.  Rand. <http://www.rand.org/education/mcarthur/Papers/role.html#anatomy >.

Doyle, Patrick. "Believability through Context Using "Knowledge in the World" to Create Intelligent Characters." Trans. SIGART: ACM Special Interest Group on Artificial Intelligence. International Conference on Autonomous Agents. Session 2C ed. Bologna, Italy: ACM Press    New York, NY, USA, 2002. 342 - 49 of Life-like and believable qualities.

Haddad, Hanadi, and Jane Klobas. "The Relationship between Visual Abstraction and the Effectiveness of a Pedagogical Character-Agent." The First International Joint Conference on Autonomous Agents & Multi-Agent Systems. Bologna, Italy, 2002.

Wilson, M. "Metaphor to Personality: The Role of Animation in Intelligent Interface Agents." Animated Interface Agents: Making them Intelligent  in conjunction with International Joint Conference on Autonomous Agents. Nagoya, Japan, 1997.

Stappers, P., Keller, I. & Hoeben, A. 2000, ‘Aesthetics, interaction, and usability in

 ‘sketchy’ design tools’, Exchange Online Journal, issue 1, December, [Online],

 Available: http://www.media.uwe.ac.uk/exchange_online/exch1_article4.php3

[2004, August 3].

Bruce, V., Cowey, A., Ellis, A. W. & Perrett, D. L. 1992, Processing the Facial Image.

 Oxford, UK, Clarendon Press.

Hay, D.C., Young, A.W. 1982, ‘The human face’, in Normality and Patholgy in

 Cognitive Function, Ellis, A.W. ed., London, Academic Press, pp. 173-202.

Gregory, R., Harris, J., Heard, P. & Rose, D. (eds) 1995, The Artful Eye, Oxford

 University Press,Oxford.

"Thirteen Ed Online: Concept to Classroom".  2004.  Educational Broadcasting Corporation. 8/9/04 2004. <http://www.thirteen.org/edonline/concept2class/ >.

Hodges, Dr. Andrew. "The Alan Turing Internet Scrapbook".  Oxford, 2004.  (3/15/2004):  University of Oxford. 8/09/04 2004. <http://www.turing.org.uk/turing/scrapbook/test.html >.