Have you heard of searching for "statistical improbable phrases" or SIPs in books on Amazon? Bierma's linguistics column in the Chicago Tribune writes about these searches. Apparently this is a feature they have instituted that finds word pairs that are unusually common in one book, compared to others. For example, if you find "Cilice belt," "Sweater pocket" and "Corporal mortification," the book is probably Brown's "The Da Vinci Code."

According to Amazon, the aim of the feature is to use mathematical, algorithm-derived phrases that occur more frequently in some books than in other books, and that will be interesting to customers. It's derived by math, not humans.

Does anyone know how to find this feature? I tried going to the Amazon site, but I couldn't find it. It sounds fun!

quote:

Originally posted by Kalleh:
Have you heard of searching for "statistical improbable phrases" or SIPs in books on Amazon? Bierma's linguistics column in the Chicago Tribune writes about these searches.

Does anyone know how to find this feature? I tried going to the Amazon site, but I couldn't find it. It sounds fun!

Thanks for bringing up this subject, I'd never heard of SIPs till now, but I agree that it does sound fun .

I couldn't find much on the Amazon site either, but I found a few other references.

See here and click on the link at the bottom.

This is a discussion about them.

These are the rest of the Google matches on the subject.

I just wonder where that Amazon link to SIPs is. The article I read seemed to say it was on the Amazon Web site, but I surely didn't find it.

Apparently, 2 other new Amazon features are "Concordance," which lists the 100 most common words in a book, and "Text Stats," which lists several word statistics, including "words per sentence," "words per dollar" and the percentage of "complex words," defined as words with three or more syllables. "Text Stats" also lists the "Fog Index," a formula that uses the average sentence length and the number of complex words to generate an approximate grade level for the reading.

I'd love to find books with the most complex words. I wonder if they'd be biochemistry books or the like. I couldn't find those links, either, on the Amazon site.

quote:

Originally posted by Kalleh:
I just wonder where that Amazon link to SIPs is. The article I read seemed to say it was on the Amazon Web site, but I surely didn't find it.

Apparently, 2 other new Amazon features are "Concordance," which lists the 100 most common words in a book, and "Text Stats," which lists several word statistics.

I couldn't find those links, either, on the Amazon site.

I've given up and decided to contact them to ask how to find these features. If all goes well, I should get a reply "within 24 hours". It's now 11.17 here, so it should arrive before this time tomorrow morning.

Oh, thanks, Dianthus! I can't wait to find out. I especially want to see those books with the most complex words.

quote:

Originally posted by Kalleh:
Oh, thanks, Dianthus! I can't wait to find out. I especially want to see those books with the most complex words.

I got a reply from them, but not the one I'd hoped for .

"From: Amazon.com Customer Service
Date: 05/27/05 13:57:41
To:
Subject: Your Amazon.com Inquiry

Thank you for writing to us at Amazon.com.

I'm very sorry, but I was unable to determine what type of assistance you need from the content of your e-mail message.

Please note that you can find the answers to most of your questions in our online Help Department:

http://www.amazon.com/help/

If you still require assistance after consulting our online resources, please click the link below to write back to us.

We look forward to hearing from you, and thank you for shopping at Amazon.com.

Please visit the following link to provide the information we requested:

http://www.amazon.com/rsvp-mi?c=awewvcfe3397847447&q=o2r

Please note: this e-mail was sent from an address that cannot accept incoming e-mail. Please use the link above to send us your reply.

Best regards,

Debnath Rana
Amazon.com Customer Service
http://www.amazon.com
==============================
Check your order and more: http://www.amazon.com/your-account"


This was my original enquiry:


"Date: Fri May 27 10:15:45 UTC 2005
Subject: Other Questions & Comments
To: int-primary__4r4t4y@amazon.com
From:
---------------
05/27/05 03:14:49
NAME:
COMMENTS: I've come across several references to Amazon's new 'Statistically Improbable Phrases', 'Concordance' and 'Text Stats' features but, despite conducting an intensive search of your site, I can't find any of them.

Could you please tell me how to access them? I know several other people who have also heard of them and would find them useful but can't find them either.

Thank you.
Caroline"

So I've given them this URL and copied the text from that page and sent that to them. I don't expect to hear from them before Tuesday because it's a public holiday in America, so they probably won't be around till then.

We shall see.

In the meantime, I will contact Nathan Bierma, the author of the column in the Chicago Tribune talking about it.

Thanks for all your hard work!

quote:

Originally posted by Kalleh:
In the meantime, I will contact Nathan Bierma, the author of the column in the Chicago Tribune talking about it.

Thanks for all your hard work!

I've got another email from Amazon and this time it's answered my questions:

"Greetings from Amazon.com.

Thank you for contacting us regarding our Statistically Improbable Phrases (SIPs).

SIPs are phrases that appear in the text we scan for those books that have the "Search Inside!" displayed on their detail page. To identify SIPs, our computers scan the text of all books in Search Inside. If they find a phrase that occurs a large number of times in a particular book relative to all Search Inside books, that phrase is a SIP in that book.

To use SIPs, simply type in the keywords, of the subject in which you are inquiring, into the Search field at the top of our homepage. When you click "Go!" you will be given all of the results for those keywords.

In the results, you will notice that some books have an icon above the picture of the book that states "Search Inside!" By clicking on the title, or image of the book, you will be shown all the possible SIPs related to your keywords under the title and author.

You may then click on a SIP to view a list of books in which the phrase occurs. You can also view a list of references to the phrase in each book, and you can learn even more about the phrase by clicking on the A9.com search link beneath it.

If you would like to learn more about SIPs, please visit:

http://www.amazon.com/gp/search-inside/sipshelp.html

We hope you find this feature helpful! Thank you for shopping at Amazon.com.

Please let us know if this e-mail resolved your question:

If yes, click here: http://www.amazon.com/rsvp-y?c=bagwbgbe3390441787
If not, click here: http://www.amazon.com/rsvp-n?c=bagwbgbe3390441787

Please note: this e-mail was sent from an address that cannot accept incoming e-mail.

To contact us about an unrelated issue, please visit the Help section of our web site.

Best regards,

Vipul Yadav
Amazon.com Customer Service
http://www.amazon.com"

So I decided to put it to the test and typed in "crinolines bustles" and got these results.

Oh, thanks, Di.

Similarly, I got this note from Nathan Bierma:

"Amazon doesn't have an SIP-only search function; it just has SIPs listed under books that have Search Inside activated. So you have to search title-by-title. All the books I mentioned in my column have this capability.

Glad you enjoyed the piece! You can tell readers to sign up to receive my column by e-mail if they're interested:

http://www.nbierma.com/language/column/email/"

If any of you wish to receive Nathan's language columns automatically, you can do so in with the above URL. I very much enjoy receiving them, and they have been the inspiration for several posts.