Watson from IBM: Why semantic text tech helps analytics

More significantly, the “bad guys” (if you will) are aware of these databases, and may take steps to evade automated surveillance. They may constantly change their calling phone number to avoid being blacklisted. Thus, real humans are still needed to go into these databases and study the unstructured text. If you see people writing descriptions of the same kind of illegal phone call coming from constantly changing numbers, you want to assign someone to investigate further. This is why unstructured text is required: there is no way to know in advance how scammers or criminals will behave. If you tried to anticipate every possible situation with a pre-designed form, the “bad guys” would modify their behavior to “game” the predesigned form an avoid surveillance.

FBI and Crimestoppers databases and Dr. Watson

The more important examples are things like the FBI IC3 database and the Crimestoppers database. In part for the reasons we just mentioned, unstructured text plays an essential role in the complaint used by citizens. So, at present, a human is still required to read each complaints. Since any one human can only read a small part of these databases, they might miss complex crime trends. This is where a technology like Watson might come in. Watson could read every complaint in FBI IC3 or Crimestoppers, and potentially catch significant details in the aggregate unstructured text that would be overlooked by necessarily more myopic humans. (Note that since every scam involving the Internet is potentially an FBI IC3 complaint, this database of citizen complaints is huge. Most complaints are ultimately ignored due to lack of resources. Watson could help here by connecting dots that otherwise are overlooked.)

We’ve mentioned Yelp for finding restaurants. Although it’s controversial (and there may be legal hazards to those making complaints) it also serves as a repository for citizen complaints against businesses. (There are many other companies recording these kinds of complaints, with various levels of complaint vetting, different business models, and different amounts of controversy.) Yelp and similar databases are often largely public, so the government (or Yelp’s owners) could have Watson analyze this data to find previously undiscovered trends as well.

The IBM Watson team is asking people with suggestions for how Watson’s technology might be used to make submissions to them. Since this application occurred to us, we thought we’d just throw it out here. (IBM has specific guidelines. For example, the application must involve unstructured text, as well as involving human language questions for Watson to answer. In this case, the questions to Watson would be about ideal allocation of resources, or whether there were unusual new patterns appearing in the aggregate submissions.)

Dr. Watson the spook

We’re limiting our discussion here to unclassified government databases, which must be huge and full of unstructured text. Of course, the government also has those “other” databases we’ve heard so much about recently. We’re guessing Watson already passed the background checks for a security clearance, so that’s one’s probably covered. It’s pretty obvious the technology hasn’t made it around to “more civilian” government applications yet, however.

Forming a posse in the digital age

Maybe semantic processing of text and crowd-sourced law enforcement is how you form a posse in the digital age. (Or maybe we just think that because Americans located out West with an engineering bent. Posses have obvious legal and ethical hazards — we not being entirely serious in our enthusiasm for the crowd-sourced posse thing here. Perhaps on the emerging risks side, these new technologies might facilitate a dangerous new vigilantism.)

The problem with unstructured text

To date, computers have been mainly good at processing structured text. This means a designer or software developed needs to anticipate in advance how data will be used. They need to carefully design a database schema or web form that captures these use cases in advance.

Humans, on the other hand, are especially good at processing unstructured data. (We may create a structure around that data after the fact.) From an evolutionary standpoint, this makes sense. The importance of things we encounter may not become clear until after the fact. Therefore, it’s usually not possible to design a structure around this data in advance.

There’s a bear in the woods, or, Watson come quickly.

Usually, in these kinds of evolutionary examples a lion is used. We’ll use a bear instead. Let’s say there’s a bear in the woods. You thought the bear was tame, but in hindsight it turned out very dangerous. You previously thought the information you had about bears was not valuable, so you didn’t bother to organize it since that would involve unnecessary effort. If you’re able to go back and extract crucial information about bears from this previous unstructured data, your genes are much more likely to be passed on. Similarly, the vast majority of information streaming in to your senses will later to out to be extraneous. By being able to leave this data unstructured until its importance becomes clearer, you’re able to save energy (computation effort) and will have an evolutionary advantage.

1 2 3 4