Watson from IBM: Why semantic text tech helps analytics

This example also illustrates nicely why this is a valuable computation technique. If you’re able to “lazily” (a technical term) leave data unstructured until it’s value is certain, you can eliminate the significant design, storage, and data entry associated with database schemas.

We briefly touched on IBM’s Watson previously in our article on Cisco’s CES talk on Internet of Things. The Internet of Things will create a lot of data. Some of it, especially in hindsight, will not be optimally structured. And, as our example above illustrated, it is actually more efficient to leave data unstructured when there is a great deal of it and the relative future importance of various features is uncertain. That’s where Watson will come to play.

Apparently, IBM Watson could mine text very well for Jeopardy answers, somehow. But what is this technology, exactly? Can it do something besides play a mean game of Jeopardy?

Medicine like playing Jeopardy!?

IBM’s first killer application for Watson is assisting medical doctors in keeping up with the latest research. A huge amount of medical research is published each year. Articles sometimes provide new insights into diagnosis and treatments of disease, especially the more exotic and interesting cases. But the amount of new literature is vast and nearly impossible for specialists to keep up with, let alone your average general practitioner.

Enter Watson, which can understand large quantities text almost like a human. It can also answer “natural language” questions from humans (medical doctors) and respond to those questions in a natural way, as proved on Jeopardy. (It doesn’t really yet “understand” the way a human does. But it is able to create statistical models of the meanings of questions and text. So, when it is asked a question by a human, it is able to find the medical articles that are statistically most likely to be relevant in answering that question, and present that result back to the doctor.

Now, to be perfectly fair, this may not be an entirely new concept. (More on why tech to search scientific articles isn’t new in a bit.) Moreover, medical articles play to the computer’s strengths (much like the game of chess in that other famous IBM exhibition). While they are pretty fair along in the continuum of structured versus unstructured text, medical articles still have more structure than an average newspaper article or short story. Researchers go through a precise ritual when writing a medical article. There’s an abstract, introduction, conclusion, and so on. Space is very limited, so research describe concisely what they are doing in a set number of words in each section following a pre-set scientific style. (This is unlike a classic novel by, say, Agatha Christie or Lewis Carrol, which might jump from first-person to third-person narration or switch to prose mid-story.) Scientific articles use a lot of jargon. This also plays to the strength of computers, which can have a potentially unlimited vocabulary.

Searching scientific articles isn’t new tech

Scientific articles use citations to link paragraphs and sentences to other scientific articles. These citations also follow one of a small number of allowed formats, which provide a standardized reference intended to retrieve the cited article. The vast majority of these other articles, going back several decades, will already be on-line. Again, advantage computer, since it will be able to instantly retrieve and scan each citation to learn more about the meaning of the article. The poor human specialize must either already be familiar with the article (as is sometimes the case with highly cited articles in specialized fields) or spend time reading and retrieving it.

Moreover, in many cases the National Library of Medicine (NLM) and similar groups have electronically annotated cited articles in a machine-readable way. (Each discipline has their own system, but medicine and biology often use the MeSH ontology.) This was originally intended to speed researcher’s searching for related articles in PubMed/Medline (the online electronic article abstract searching system set up by the NLM, which took the place of multiple similar commercial services in the 1990s). If you knew (or know) the MeSH terms for the subjects you are interested in, you can pull their abstracts over the Internet via Medline. This, in turn, sometimes allows access to the full-text articles on publisher sites.

1 2 3 4