202: Accepted

EOY 2007 Data Analysis

2008-01-01T16:44:00Z

It's a new year and time for some dumb data analysis.

Most interesting thing to me this year is that most of the traffic
to this site and my other sites (notably epcostello.net) is from
automated agents: search engines, random webcrawlers, SEO's link injectors.

HTTP Status Codes:

155127	200
64662	304
11400	301
6352	302
5705	404
5308	202
2784	401
1609	405
126	400
63	414
31	500
30	403
3	501

Top Ten Hosts

4551	66.249.73.200	crawl-66-249-73-200.googlebot.com
2234	81.52.143.16	natcrawlbloc03.net.m1.fti.net
1890	81.52.143.15	natcrawlbloc01.net.m1.fti.net.
1492	64.152.34.36	jfk-lv3-n4.panthercdn.com
1380	38.99.203.110	Panscient_Data_Services.demarc.cogentco.com
991	128.194.135.94	web-crawler.irl.cs.tamu.edu
885	216.240.154.103
884	66.249.73.148	crawl-66-249-73-148.googlebot.com
800	64.92.162.210
773	72.30.177.225	wm509310.inktomisearch.com.

Raw Top Ten Requests

21547	/robots.txt
8030	/favicon.ico
6801	/articles/2005/12/27/a_practically_u/
6062	/articles/nav-commenters.gif
6055	/g/Google_logo_transparent.png
5582	/d/4/js/ajax/
5290	/202/2006/06/disabling_trackbacks_in_movabl/
4445	/
4217	/g/feed-icon-16×16.png
3993	/g/by-sa-3.0-88×31.png

Filtered Top Ten Requests

21547	/robots.txt
6801	/articles/2005/12/27/a_practically_u/
5290	/202/2006/06/disabling_trackbacks_in_movabl/
4445	/
2812	/202/
2664	/202/2006/12/google_reader_annoyances/
2121	/202/2006/10/social-bookmarking-and-attention/
1278	/202/2006/11/bloglines_new_features_playlis/
912	/articles/
890	/202/2006/07/yet_another_spam_retaliation_t/

Top 20 non-caching requestors of Robots.txt:

2353	crawl-66-249-73-200.googlebot.com	[66.249.73.200]
1404	natcrawlbloc03.net.m1.fti.net	[81.52.143.16]
1192	natcrawlbloc01.net.m1.fti.net	[81.52.143.15]
770	wm509310.inktomisearch.com	[72.30.177.225]
736	ct501085.crawl.yahoo.net		[74.6.86.230]
688	wm509458.inktomisearch.com	[74.6.74.202]
498	crawl-66-249-73-148.googlebot.com	[66.249.73.148]
496	livebot-65-55-213-74.search.live.com	[65.55.213.74]
491	wm508816.inktomisearch.com	[74.6.69.173]
342	lj512274.crawl.yahoo.net		[74.6.19.77]
342	wm511001.inktomisearch.com	[72.30.252.135]
327	natcrawlbloc02.net.s1.fti.net [193.252.149.15]
288	lm502044.crawl.yahoo.net		[72.30.226.173]
262	ct501101.crawl.yahoo.net		[74.6.86.207]
233	67.110.56.45.ptr.us.xo.net	[67.110.56.45]
224	crawl-66-249-73-132.googlebot.com	[66.249.73.132]
208	wm511565.inktomisearch.com	[72.30.226.209]
206	c02.entireweb.com	[89.150.197.130]
199	wm509426.inktomisearch.com	[74.6.75.46]
197	ip67-95-51-86.z51-95-67.customer.algx.net	[67.95.51.86]

Last time robots.txt changed: 23 March 2007

Top Ten Referrers (filtered):

97488	"-"
533	"http://neworder.box.sk/forum.php?page=last&did=multSecurity%20and%20Networking&thread=251392"
244	"http://www.zenatode.org.uk/ian/internet/hotmail.xhtml"
212	"http://my.yahoo.com/"
142	"http://www.google.com/search?hl=en&q=phx.gbl"
122	"http://www.stumbleupon.com/refer.php?url=http%3A%2F%2Fartific.com%2Farticles%2F2005%2F12%2F27%2Fa_practically_u%2F"
117	"http://www.google.com/search?q=phx.gbl&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a"
88	"http://www.google.com/search?hl=en&q=phx.gbl&btnG=Google+Search"
68	"http://neworder.box.sk/forum.php?did=multSecurity%20and%20Networking&thread=251392"
55	"http://www.dslreports.com/shownews/DNS-Hacks-Phishing-20-90182"

Internal referrers and obviously junk referrers have been filtered out.

Top Ten Raw Search Requests:

1474	q=phx.gbl
260	q=phx.gbl"
122	q=.phx.gbl
107	q=phx.gbl%3A1863
98	q=phx%2egbl"
76	q=crawler.bloglines.com
62	q=phx.gbl+domain
59	q=gbl+domain
53	q=gbl+tld
42	q=%22phx.gbl%22

Top 20 phx.gbl searches:

phx.gbl is a pseudo-domain used by Microsoft for a variety of services. I wrote about it in On The Importances of Revers DNS which I now realize is still using the previous design system for this site.

1474	q=phx.gbl
260	q=phx.gbl"
122	q=.phx.gbl
107	q=phx.gbl%3A1863
62	q=phx.gbl+domain
42	q=%22phx.gbl%22
30	q=.phx.gbl%3A1863
29	q=@phx.gbl
26	q=%40phx.gbl
23	q=phx.gbl+netstat
22	q=phx.gbl+1863
22	q=.phx.gbl"
19	q=phx.gbl%3A1863"
18	q=by2msg2204708.phx.gbl
17	q=what+is+phx.gbl
15	q=netstat+phx.gbl
15	q=by1msg4176104.phx.gbl
14	q=phx.gbl+msn
13	q=by2msg2204912.phx.gbl
13	q=%22phx.gbl%22"

Top 20 Non-phx.gbl searches:

76	q=crawler.bloglines.com
38	q=artific
16	q=infobackground
15	q=ed+costello
15	q=207.46.108.36
15	q=202+Accepted
13	q=crawler.bloglines.com"
13	q=207.46.111.86
13	q=202+accepted
11	q=spam+retaliation
11	q=InfoBackground
10	q=google+reader+rename+folder
10	q=Reverse+DNS
9	q=kb05474
9	q=importance+of+reverse+dns
9	q=crawler.bloglines.com+
8	q=nokia+espionage
8	q=iab+ad+units
8	q=hotmail+reverse+dns
7	q=tvpath.com

In March 2007 I wrote my own trackback endpoint in PHP which logs all of the trackback data to a file instead of beating up my MovableType installation and MySQL database.

Trackbacks Received since 21 March 2007: 10258

Number of Valid Trackbacks: 0

Top Ten Trackback Sources:

447	207-234-131-237.ptr.primarydns.com	[207.234.131.237]
156	movinglabs.com [195.242.99.80]
150	u15250532.onlinehome-server.com [74.208.14.63]
144	218.189.232.72.static.reverse.ltdomains.com	[72.232.189.218]
124	[206.123.73.15] [206.123.73.15]
122	server.camelotwealthcreation.com		[69.50.210.8]
113	giantlogic.net [208.101.35.52]
99	89-149-195-161.internetserviceteam.com [89.149.195.161]
96	u15251680.onlinehome-server.com [74.208.14.215]
95	210.219.232.72.static.reverse.ltdomains.com	[72.232.219.210]

Top Fifteen Trackback Titles:

169	"Tramadol."
151	"Phentermine."
119	"Xanax."
94	"Cialis."
62	"Lexapro."
56	"Ephedra."
52	"Valium."
52	"Ultram."
47	"Zoloft."
43	"Ambien."
42	"Fioricet."
37	"Percocet."
37	"Cheapphentermine."
37	"Adderall."
34	"Soma."

End of year domain cleanup

2007-12-03T03:04:32Z

I have the following domains available for sale through Sedo:

Acquire them through Sedo or contact me at sales @ artific.com.

Reboot

2007-11-06T16:12:00Z

I realized I sort of fell off the blog beat here. I've rebuilt the site (again) using the Yahoo! User Interface Library which I've been getting to know and use for various sites over the past year. I'm currently at the Defrag 2007 conference in Denver, CO and am enjoying it, it's serving (for me) as an introduction to the intersection of the current crop of social tools and enterprises.

I'll post some notes from Defrag later today.

Feedburner's Migration to Google notice

2007-06-02T17:39:49Z

I just attempted to log into my FeedBurner account and got the following notice (in addition to the login information):

NOTE: Service of FeedBurner publisher accounts will not be interrupted as a result of the acquisition by Google. You will have a 14-day interim period ending June 15, 2007 to opt-out of allowing Google to service your account. If you take no action by June 15, 2007, the rights to your data will transfer from FeedBurner to Google. Opting out will terminate your user agreement with FeedBurner, permanently delete your FeedBurner account, feeds, and all related statistical data and history, and prevent the transfer of your data rights to Google. To opt-out, contact us via accountx@feedburner.com, provide your FeedBurner account Username, and request to have your FeedBurner account deleted. We will contact you at your registered email address to confirm your deletion request before completing it.

While I don't object to the sale or to the company merging into the Google Borg, 14 days seems to be awfully short to give notice to those who don't want to continue using FeedBurner after it becomes part of Google. I have feeds which I ceased advertising years ago, which either return 301 redirects or 410 gone messages and they still get tens of hits per day. If you delete your FeedBurner account you can't redirect the subscribers using that account (unless you either redirect using a 302 or 307 redirect from a URL you control, or you used the feeds.yoursitename.tld service, which you could simply point back to a site you control). With summer vacations and the vagaries of feed updates my guess is that many people or organizations who do opt-out of the FeedBurner-Google migration will lose many readers, who will just get dropped.

FeedBurner should allow, for a limited time, say 60 days, the ability to opt-out in some way and redirect the feed elsewhere.

An interesting exercise would be to track the people who drop their FeedBurner feeds (and accounts), picking out the feeds with the highest traffic volume and registering them for yourself (unless FB has changed to blocking re-registration of a feed URL).

TwitterSense

2007-05-30T23:04:41Z

I noticed yesterday that twitter has added Google Adsense ads to the individual status message pages, but only if you are not logged into twitter. Here's two examples:

Unauthenticated:
Authenticated (no AdSense):

Twitter have also added a tab to your twitter home page showing "replies" (messages sent to @username.

Update: fixed images and thumbnails.

How Staples.com lost my business today

2007-05-23T18:05:15Z

My wife and I are in the process of moving. In the scheme of things, we're not moving all that far (approximately 660 meters if Google Earth can be trusted). In New York City you only need to move an avenue or two to be in a completely different neighborhood.

So, we're moving, and I want to get various things lined up. I used the U.S.P.S. online address change form, am forwarding all of our mail to a P.O. Box to "cleanse" our data trail in Corporate America's databanks, and am trying to figure out our broadband solution (the new building has a T1 we're told with no limit on the exclamations. That might have been cool in 1998, but we have and use a 7Mbs DSL line today, which makes a piddling T1 look downright like dialup. I digress.).

I have found it handy to have a stamp with our address on it for the rare times we actually send postal mail, so I went to Staples.com to order a stamp online. I have an account there, which is probably the only reason I thought to go there. After tooling around the site for 30 seconds I got asked if I wanted to go to the Staples Custom Printing Shop. On clicking "ok", I ended up at http://www.staples.marktheworld.com/browsercheck.asp, which is apparently where Staples has outsourced their custom stamp printing to. The browsercheck.asp in the URL should give away what happened next:

We are sorry for the inconvenience. Our site currently supports only Internet Explorer version 4.0 or higher. This is due to the advanced features used in the product customization process.

Come on. I mean, sure, they probably used an ActiveX control written in 1999 to show what the stamp would look like. And MSIE is used by, what, 80% of the worldwide marketplace? And they probably don't want to waste the precious investment in the ActiveX and ASP coding. The net result is that they lost me as a customer, there is no reason, today, to be designing web applications solely for one browser platform. None. I will accept that if you're on a tightly controlled intranet you might consider it, but really there's just no reason for this.

Twitter Bug or Mis-design: location applies to all tweets

2007-05-05T14:44:29Z

I had some unexpected travel this week, spending much of Thursday and Friday driving between NYC and DC on I-95. While in DC I was curious to see how Twitter handles changes to the user's location. Your location is echoed back in your Twitter homepage as well as in the various status feeds. Twittervision uses the location field to plot tweets on a Google map.

From what I can tell, Twitter only keeps the current location, and uses that to fill in the <location> field in the various feeds. I think this is a mistake and a design flaw:

Tweets are frequently location-specific, location is part of the "content" of the tweet, it helps set the context for the tweet.
If you can't depend on the location being "accurate", you can't build a service atop twitter which reliably uses the location to take action. If I only update twitter daily and I'm travelling around a lot, I don't want to get alerts related to a tweet I posted earlier in the week from one location when I'm in a completely different city and the topic/alert would no longer be relevant.

Now, whining aside, I can see why you wouldn't want to build a history of locations since the location is (currently) free-form. You'd probably want to normalize the locations, would you do that across the entire user base or on a user-by-user basis?

What I would do though is store the user's location with each tweet so that locational context is preserved.

Twittering Tweets Trouble The Times' Truthseekers

2007-04-22T20:29:39Z

I find the ongoing attempts to "get" twitter interesting, as if everything must have a defined purpose in order to be. I use twitter, play around with writing code for it, and I have absolutely no idea what it will be in a year, if it's even around.

The latest article is a New York Times piece From Many Tweets, One Loud Voice on the Internet. The article doesn't add much to the discussion, but I did like this statement attributed to Evan Williams: Twitter is best understood as a highly flexible messaging system that swiftly routes messages, composed on a variety of devices, to the people who have elected to receive them in the medium the recipients prefer.

I wrote elsewhere that I could see twitter becoming a sort of personal message queuing system. It's asynchronous, you just keep writing messages to the queue and your consumers, your readers, your receivers can get the messages on their cell or I/M immediately, or on their twitter page, your twitter page, or via a web feed.

The 140 character limit imposes brevity, there's a penalty if you try sending too much information. You just can't afford to wrap a SOAP or other XML language around your message.

It's tempting to make it into something big, something major, because it must be something major to get all the attention it's receiving, right? Why can't people just enjoy it for what it is, a simple messaging system. You don't have to use it, you don't have to subscribe to it, it isn't required. If you don't want to use it, then don't.

The Times' writer Jason Pontin writes:

But I also strongly disliked the radical self-revelation of Twitter. I wasn’t sure that it was good for my intimate circle to know so much about my daily rounds, or healthy for me to tell them. A little secretiveness is, perhaps, a necessary lubricant in our social relations. I wondered whether twittering could ever have broad appeal.

I recall a similar complaint about blogging. You reveal as much or as little about yourself as you want to, no one forces you to reveal more than you choose to.

My twitter account is twitter.com/epc. You can follow me to see when I'm walking the dogs. Apologies for the headline.

DOM templating with JavaScript?

2007-04-20T05:47:19Z

I've been doing more and more JavaScript programming. I've run into a situation where I want to create a fragment then clone it repeatedly, changing a couple of ids and node values (picture a form with maybe ten fields, I want to update the value of a couple of fields and the id of a couple of fields and then add the result to the DOM).

It feels like overkill to walk the tree for each change. I kind of want to do a sprintf(). I am wondering if I can fetch the element by the technically duplicate id and change the node value and id to something else, before appending it to the tree with the original node (where the duplicate id could conflict).

Twitter API ideas

2007-03-30T06:37:28Z

I am tinkering with Twitter and working on something which uses the API, here's some ideas for further API enhancements:

Adopt an authentication model like Flickr's for 3^rd party agents so that users can permit an agent to have access to their information or act on their behalf without having to have the user's userid and password passed to the 3rd party.
Add a method to retrieve and set a user's location. For bonus points, a method to retrieve any user's location information (or at least users who are your "friends" or "favorites"). This information is already exposed in the datastream returned from http://twitter.com/statuses/friends_timeline.xml.
Add a method to retrieve only the updates for a given user, or at least only "my" updates.
Add a method to retrieve updates since a given time.
Add a method to add, remove, and forget friends (though I realize this could be horribly abused, so maybe not a great idea).

I noticed one glitch while destroying the design of my twitter page: you can change the link color, and allegedly the background color, but the background color change does not apply to all relevant elements, while the link color affects all text elements. So for example you could set the link color to white and the background to black but certain fields will retain a background color of white and thus all links will become invisible.

I previously wrote about twitter in Twittering.

ETech 2007 Day 2 p.m. sessions

2007-03-28T23:10:03Z

Don MacAskill (SmugMug) -- Set Amazon's Servers on Fire, Not Yours

Smugmug 140MM photos, no debt, profitable since first year. 192TB stored at S3. Doubling yearly.

S3::Simple Storage Service. $0.15/Gb/month w/replicats. REST API. Fast, not 15K-SCSI fast, but internet fast.

Why use them?

not a lot of web scale expertise on planet earth
reputation for systems
[he] once competed with Amazon - fatbrain
They eat their own dogfood. Dozens of products.
Focus on the app, not the muck.

Show me the money!

Guestimate: ~$500k save per year
Actual:
- Growth: 64MM photos -> 140MM photos
- Disks would cost $40k -> $100k/month
- $922K would have been spent
- $230K spent instead
- $692K in cold, hard savings
Nasty taxes (on capital goods)! $295K 'saved' in cash flow. Bonus!
Reselling disks to recoupe dunk costs

this is a partial cost of ownership number

sweet spots

perfect for startups & small companies
ideal for store lots, serve little businesses of all sizes
not so great (yet) for serving lots if you're a medium or large sized business. Transfer costs high if you can buy bandwidth in 1Gbps+ chunks.
We're a store lots, serve lots company. What to do?

Like SmugFS

Architecture remarkably similar to internal Smug filesystem
Similar to lots of startups
Stupid we're all building the same thing
Easy to drop in
Started on Monday, live in on production on Friday

S3 evolution

started just doing secondary storage. Too cold!
Tried out as Primary. Too hot!
Finally hot & cold model == just right!
Amazon gets 100% of the data
Smugmug keeps hot data local (about 10%)
95% reduction in # of disks bought

Sample Request

they check for image in cache, if not there they log it and then retrieve the image from S3 and return to client

Proxy vs Redirect vs Direct Links

Build SmugMug->S3 with multiple mods
Can flip a switch to change
Nearly 100% served are proxy reads
Sometimes HTTP redirects
Rarely direct S3 links

Permissions

SmugMug has complicated permissions
Passwords, privacy, external links
Proxying allows strong protection

REST vs SOAP

loves rest, hates SOAP
Lightweight
Nothing useful added with SOAP's complexity

Reliability

not 100%, close though
more reliable than SmugFS
no service level agreement
Lots of failure points:
- SmugMug's datacenter
- Internet backbones
- Amazon's datacenter

No other software, hardware, or service [they] use is 100% reliable either

Handling failure

Build from day one with dailure in mind.
Stuff breaks, try again
Writes ail? Write locally, sync later.
Reads fail? Handle Intelligently. Alerts?

Performance

Fast for reads and writes
Mostly speed of light limited 20-80ms
Parallel I/O for massive throughput. 100s of Mbps
Machine measurable, human indistinguishable

CDN?

S3 is not a CDN[content delivery network]
it's storage
no global locations yet
limited edge caching
perhaps a future Amazon web service?

How do they do their proxy reads?

Store and forward vs stream

Store and forward
- great resiliency
- poor performance
- if it's a big file, really poor performance

Stream
- poor resiliency
- great performance
- do a quick HEAD first to verify

The speed of light problem

he was misquoted as saying Amazon was slow when trying to explain the speed of light
Amazon has not solved fasther than light data transmission. yet.
unavoidable, make sure your application can tolerate
parallelized I/O can mask problem
caching can help
streaming can help

Outages and Problems

not perfect, five major issues
3 outages of 15-30 minutes, 2 were core switch failures and one DNS problem. Amazon.com affected.
2 performance degradations. On a smugmug customer noticed, another wasn't noticed
Not a big deal, everything fails, expect it.

SLA, Service and Support

Smugmug do not care about SLA, but others might
Service Support: One area where Amazon is weak.
- This is a utility
- They need a service status dashboard
- Pro-active customer notifications
- Ability to get a hold of a human
Support for developers is quite good.

Amazon.com's customer service is good, AWS will likely catch up

Saving SmugMug's butts

knocked out power to ~70TB of storage. Oops!
Moved datacenters during normal business hours, customers not affected
Stupid bugs

Miscellaneous Tips

use cURL
- fasther
- more reliable
- storing vs streaming is simple

make stuff as asynchronous as possible
- hides speed of light issues
- hides or masks problems
- fast customer service

Elastic Compute Cloud (EC2)

Like S3 but for computing
- scale up or down via API
- web servers, procesing boxes, development test beds, etc

Launching large EC2 implementation "soon"
- image processing
- 500k-1M photos/day
- 10-20 terapixels/day processed
- peaky traffic on weekends, holidays
- ridiculously parallel

Simple Queue Service (SQS)

Simple, reliable queueing
Mates well with EC2 and S3
- Stick jobs in SQS
- retrieve jobs with EC2 instances using S3 data
- run jobs, report status to SQS

$0.10/1000 items
- Priced well for small projects
- gets costly for large ones (millions)

Missing Pieces

Database API or DB grade EC2 instances
- Fast (lots of local spindles, lots of RAM)
- Persistent

Load Balancer API
- Single IP in front of lots of EC2 instances
- Programmable to add/remove/change clusters
- Can be done with software on an EC2 instance, but painful

Slides to be at http://blogs.smugmug.com/

How are they using EC2?
- The EC2 instances invoke smugmug APIs to do work. The SmugMug servers don't really know much about EC2

I asked: Has their use of Amazon been an issue, either to outside investors or customers?
- Not an issue, they have no outside investors, and further they've talked with VCs to raise the issue that startups should be looking at Amazon's services (and if not, why not)

Superninja Privacy Techniques for web Application Developers

Marc Hedlud and Brad ...? from Wesabe. Wesabe is a personal finance web application.

Keep critical data local. If there's data you'd never ever ever want to lose, don't put it on a web site.

created a wesabe uploader for Mac/Windows to keep bank credentials on your computer. The uploader downloads data from bank sites, strips certain data out of the files, then uploads to Wesabe.
don't trust the site. sensitive data filtered before it ever reaches the server.
requires a download
puts burden on user to maintain a secure machine (same risk as using a web browser to bank)
if successful, risk of trojan targeting

Use a privacy wall to separate public and private data

use secret key as index in db
secret key is only computed when user is logged in (they use hash(password + salt))
secret key stored in session data
other paths through the db: need to ensure that if you're using a privacy wall all transactions must traverse the privacy wall
the data itself can leak information
logs and exception reports can capture leaked information

password changin and recovery becomes trickier
use a locker generate a one time key for user stored in locker
encrypt using the locker rather than the password
troubleshooting can be harder

Use partitioning to protect against breaches

keep pools of sensitive data separate
eg membership and financial records kept separate
no relationship between them other than status
reduces impact of any brach -- firewalls off anything truly identifiable
allow separate politices and approaches by data type
pretty much zero drawbacks other than implementation time

data fuzzing and log scrubbing

(currently) no requirement to retain specific data on users of a server (in the US)
Subpoena / warrant may require that you give up all data on a user
Different countries have different data retention politices (see epic.org)
filter key parameters from logs
remove some of the precision of IP addresses
remove precision from timestamps since they too can be used to identify someone (cf. example of whistleblower information)

prevents leakage of passwords
avoids giving attackers / law enforcement a way through the privacy wall
loss of certain private data may require you to notify your customers
best protection is to delete your logs
important to have a public policy in place (cf link to eff.org policy information)

no protection against wiretap orders
difficult to cover all your bases (use centralized logging)

use voting algorithms to determine public information

"the esp game" to tag things at CMU.edu. If two people tag something the same thing at the same time, maybe that's a good tag to apply.
look at google image labeller
when people agree on a term, it's common knowledge
if enough people agree, it's probably publicly known
private transactions shouldn't be shown on the site
lots of users naming a merchange probably means it's public
works on opaque information
reliable -- very few faults since launch
no manual work needed

drawbacks

information is hidden until threshold met (understates available info)
can leak data if threshhold is too low

miscellaneous

hash your passwords. don't store in plaintext.
random (non-sequential) database ids. Don't use auto-inc ids in public data.
data bill of rights -- your data is your data. can export, delete, etc.

more information

eff guidelines for online service providers http://www.eff.org/osp/
brad's blog: http://blog.footle.org/
privacy software tools for web apps http://dev.riseup.net/privacy/

ETech 2007 Day 2 a.m. sessions

2007-03-28T18:16:02Z

Mike Kuniavsky -- The Coming Age of Magic

Ubiquitous computing design studio

"the hidden middle of Moore's Law". The processing power in the middle is high and the price is at commodity levels.
eg: 486 was $1000 in 1989, is $0.53 in 2007.
now that CPU power is cheap enough, can put 486 power, information processing power, into commodity items
ubicom allows us to embed knowledge into our tools

the old paradigm of terminal oriented computing does not work for ubiquitous devices

animism represetnign at a gut level
animism means labeling inamimate objects as living, attrobuting characterisints of aniumate objects to inanimate objects (typically humans) and making predictions ore explanations about inanimate objects based on knownled about animate objects (again usually represented y human beings)

Sony MagicLink, running MagicCap by General Magic, circa 1995.
Core development principle was to couple portable computer with networked communications.
levereaged experience with Mac UI, extended desktop metaphor to portable network device metaphor
used literal desktop metaphor. stuck closely to the metaphore, like hallways, furniture
ie "Downtown" as a direction. tall shiny building as the Internet.

how does that metaphor allow people to get work done?
The desktop metaphor does not work for ubiquitous computing. Sticking a basic unmodified PC into an everyday object will not take off.
Creates an information management problem on top of all the other problems the user is trying to solve.

there must be a better way to deisgning devices for people

there must be an existing metaphor for how objects and devices interact with people

there is...it's magic. not the vast majority of magical concepts in each culture, but very specifically enchanted objects.

portable network aware information processing appliances
p. not advocating pretending the technology is magic or that the technology works by magic

what distinguishes them is the ability to have independent behavious

characteristics of "magical" objects

everyday
familiar
physical
no screen (no assumption of text output, graphic output)
Not human. they have some behavior but we don't expect that to be like us.
Not superhuman. ulitmately we're in control, they're not in control of us.
We don't believe in magic. Healthy disbelief in magic.

examples of ubicom devices

orb from ambient devices
nokia medallion
various wand devices. that people are already using the magic term wand to describe these devices
enchanted rabbit - nabaztag

Ubiquitous computing emergence is the byproduct of market forces and commodity information processing prices.
Easier to go with familiar patterns than to go with new ones.
Enchanted objects ar ethe most familiar of all.
metaphore is powerful, we need to be conscious of its power.

Magic, good magic, does not conceal, cripple or ?? , it explains.

q: can you express the dangerousness of a device to a user in terms of the metaphor
a: not really. in mythology there isn't that kind of implicit relationship either. you don't know how powerful something is just by looking at it.

danah boyd -- incantations for muggles

what happens when we start to mock the people we're creating for, does that make us "evil"
what are our responsibilities?

perhaps we're not the wizards, perhaps we're the muggles?

maybe we're trying to model things after the magic of everyday life.

what are the spells we cast on one another?
what are the spells that people through their practices and rituals cast upon us?

"if you build it they will come"
what is the backside of the technology we're developing? People
how to segment people to undersand that there are different people who are not like you
not see the world as one big homogeneous group

life stages

one way of segmenting people

identity formation and role-seeking
- you, teens, college students
integration and coupling
- 20-somethings
societal contribution
- professional life, marriage...
reflection and storytelling
- retirement

that teens look at their parents, peers, see model roles, model
behaviours, see social categories. Trying to figure out the social
world around them. That we're defined by the people around us, defined
by civilisation and social interaction.

that status used to be coupled with getting integrated into the work force, now they are separated

who am I and how can I contribute?
that many young people are engaged in jobs but not careers. Working to make ends meet.
real drive for coupling around mid-twenties

strong myth, 1950s idea of how to exist in society (but it was a social construction even in the 1950s)
gives us a goal to work towards, a cornerstone of what we're doing
as we get older roles shift
start to be more concerned about what matters to us, what life is about

it's a way of getting to the priorities people have

what do people care about? Their priorities shift depending on their life stage.

america is not like silicon valley
that teens are going to social networking sites to get friends, build status
move through different technologies because they meet different needs
linkedin -- no value to a teenager
note that there's still not many social networking sites dedicated to older generation
young people want a different kind of validation than older people
corporations have very different needs than technology dreamers or people.
that corporations answer to shareholders, gain more users, grow grow grow
that teens are ok with ads, they understand that that's how the service is free, but make the ads relevant
that growth has consequences
comparison of "My So Called Life" and BtVS. MSCL had stunning demos but not enough to be kept by ABC as it was getting only one demo and not a wide audience. WB on the other hand realized they could focus on niche audiences and draw in broader audiences.
technology companies have taken the ABC model instead of the WB model.
case study: Facebook. Used to be right of passage, then FB opened up to high schools. "Didn't want to be on a site with his younger brother". FB continues to open up to more and more audiences.
Expansion has costs: you start to get people angry with you. One response is lock-in, which in turn has further costs.
If you tell your users that they're not valuable, they respond: by protesting, by creating fake users, by leaving your service...

What happens when people and technology come together?
Example: soldiers in Iraq, etsy creating communities around art they create,
find other people who are passionate about what they are doing
started with Stage 3 people but occurring with Stage 4 folks as well
of course not everything is positive. what comes out of some user practices is not necessarily good for society
for better or worse, technology reflects what's going on in the real world and some of it is ugly.
sites dedicated to people who have died, becoming proof of existence for their peers who continue to interact and respond on places like the decedents myspace page

social media are people
social media reflects existing practices and modify these practices

some characteristics of social media

persistence -- that comment from 1995 can still be found
searchability -- your future employer can find your frat party hijinks
replicability -- can't tell the original from the duplicate.
invisible audiences -- no sense of who you are talking to when you're online. don't know the social context of what's going on.

what does it mean to apply these scales to individual interactions
offline we had physical walls
can determine what norms exist within certain boundaries
way we communicate determined by environment around us
what happens when it is online, you lose the context
how do you train a generation to speak to all people all the time
we are always "in public" but there are different modes of operation depending on the context.
online the social context is not as well-formed
can't go into ostrich mode

eveyrthing we understand about public and private separations is changing due to technology
young people are taking this and running with it, even if we don't understand all of the implications of the technology we have created

what does it mean that you have so many audiences, how many can you follow?
twitter: cognitive overload. is it really building social relationships?
will your twitter friends be there when you're in a crisis?
twitter gives a way to feel you have presence

what does it mean that we have all of these technologies that create more and more influx of information?

mobile brings with it location data, presence data. will radically shift practice.
people will figure things out, may not be the healthiest plans
we through technologyu are shifting and changing the architecture of society and people are figuring out how to work around it for better or worse
as technologists, do we go into ostrich mode or do we engage with how our architecture changes society
what happens when magic goes both ways, it is not all positive, it has repercussions on society. What happens to the star wars kid? What happesn to the guy fired for blogging?
What are the consequences and how do you prepare for them?

how do we make a community that we love to be a part of?

second half of morning

Raph Koster

things that work have underlying structure. They have a grammar. Example: Solar by Miles Davis.

When somthing works, it works at many levels.

Things can be desconstructed into smaller things. Songs are made of smaller songs. Sentences decompose into SVO, then words, then syllables, then phonemes, then letters.

Games are made out of games.

games are designed to evoke fun. Each individual game within a game has to evoke fun, to enterain.

Fun as a chemical response. Fun also arises because of certain characteristics. You're meeting a challenge, understanding how it is presented, mastering how the model works.

Research shows four types of fun:

hard
easy
visceral
social

or:

Hard	Easy
Visceral	Social

games focus on hard fun, tough problems and making you solve them

Hard fun grammar: Hard fun is about solving problems
games are about making choices, interactive design.

the magic ingredients

Territory
- Where you interact should matter
Preparation
- when you interact should matter, previous interactions should take into account
Core Mechancic
- how should matter
Range of challegense
- what
Choice of abilities
- With? should matter. What tools you have should matter, should feel different.
Variable Feedback
- For? What you are buying for should matter. There should be feedback
Bad return on investment
- Few? Sometimes you don't get what you want
Cost of failure
- Phooey: game needs to tell you that you failed at something. Fun comes from learning, and failure is important in learning.

How?

The core VERB has to be repeatable. if it isn't something people care to do over and over again people will cease to use it.
Something that requires Skill. In order to learn you have to feel you are growing more competent.
Something that can handle statistical variation. What you want is for the game to acknowledge the fact that some tasks are more difficult than others though they seem to be the same.
Something that is competitive. You're competing against others, against yourself. Ratings, metrics, and other people need to see it.

When

Everything you did before must matter. The system needs to remember what you did before.
Whatever the opponent last did to you should matter too. Never start an interaction with no context. What else has the user already done?
The user should be able to prepare for the encounter in different ways. Need to prepare different ways for taking on a challenge.

Where

The territory and topology should affect the outcome
doing the same thing in different places should make a difference. The same challenge in different locales should be a fresh scenario.

What

The same verb must be applicable to many different challenges.
think of a verb as a hammer. You should present users with lots of kinds of Nails.

with

Users should be able to solve the challenge with their choice of tools.
And you should reward them with different feedback for it. The feedback should be tailored to the tools and context even if the task has been performed before. The path by which users come should be acknowledged.

All of the above can be quantified. You can arrive at a statistical measure, a rating, for every challenge. You can them some up the difficulty.

For?

games with only one outcome are ...boring
variable feedback keeps things lively
usually the best feedback is a greater challenge presented by the opponent
sometimes it's a pleasant surprise.
either way, it has to be highly visible. Others need to know, it draws and drives participation.
bottomfeeding is bad for fun.
low risk activity for high reward is bad for fun.
you need to drive users to challenges at the edge of ability

phooey

making a wrong choice has to be a setback. need to realize you've done a wrong thing.

The absence of any of these features makes any form of interaction less fun.

Chad Dickerson -- Hacking Yahoo!

Hack Day at Yahoo!, extending the formula inside out

hack day rules

build something in 24 hours, no power points
present it to everyone in 90 seconds
no prior review of projects, anything goes
that's it
100s of prototypes built

Why do Open Hack day?

build more engaged community around YDN
share the spirit of internal hack days
create excitement about Yahoo's APIs
leverage the capabilityes and scale of Yahoo! to deliver uniqely compelling event
We had been opening Yahoo! APIS, why not open Yahoo itself
show yahoo on a personal level

hack day cycle of innovation

yahoo creates new apis
battle tested APIs exposed to new users
YDN developers hack on the apis to create new things
and a bullet that went by too quickly to capture

made Yahoo! developers available, stayed up all night with the "hackers".

brought in entertainment, Beck

open request for invite for only 400 slots, no fee
simple criterion:
- will you be building something using at least on Yahoo! API
geographic diversity -- not just the usual Silicon Valley crowd
rapid fire demos, no prescreen, no powerpoints

hacking your company's cultural "software"
invited competitors
got strong emotional responses from attendees

overclocking your facility's "hardware"

a lot of amazing talent held within your organization
they let people camp on the holy grass at Yahoo! Convincing groundskeepers that sleeping on grass was ok.
had to make sure environment supported having hacking contest onsite, turning off sprinklers for example

Surprises

left event really open
class friday, hack saturday, present hacks saturday evening

prizes

Blogging in Motion: purse using pedometer and cell phone to upload photo every 100 steps to flickr www.blogginginmotion.com
ybox turned old tv into image display www.uncommonprojects.org/ybox
the puppets, made fun of Yahoo! and Silicon Valley work culture in Beck provided video
q. to be open you have to be open to people making fun of you and your culture

Another beck video http://yodel.yahoo.com/2006/09/30/the-hackers-are-here/ [um, potentially nsfw, masturbating puppet video]

feedback

Yahoo! as Punk Rock (Brooke Maury)
one of the most difficult aspects was getting the local news to understand what was going on.

Notes from etech Day 1, Morning Session [27 March 2007]

2007-03-28T16:49:44Z

These are my raw notes from Day 1's morning session I think I have edited out any extraneous comments.

Jeff Jonas -- IBM

Real time trip wires: black list + all other known possible problem makers and mix it into a mashup

Problem of enterprise amnesia: database silos, thief databases not merged with employee dbs or marketing dbs.
"Perception isolation" -- leading cause of enterprise unintelligence
"Perception integration" means integrated context
Want a system to observe and publish discoveries.
more intelligent systems created when you put data and persistent queries into the same data space.

notion of "perceptual analytics"
"relevance finds the consumer"
"when data and queries are treated the same then queries find queries"
observation that .6% difference between humans andchimpanzees, no wonder information systems are so screwed.
federated search hopeless for just in time context if trying to deliver enterpsie intelligence
must have enough features for context construction
can't bulk load data, must incrementally lean
"sequence neutrality" -- no matter the order of arrival of the data, the end state remains the same.
need to fix the past, fix connections as you load data.

notion that dreaming is deep recontextualisation
has a blog

how to handle ambiguity -- add more observations

Bill Janeway, Peter Bloom, Tim O'Reilly

bloom: that there's three themes:
- extracting signal from noise
- reduction of latency

That NYSE trades average 30 ms, hedge funds demand 1000 trades/second
Even in the era of the internet location matters, example of company moving transaction server closer to NYSE to shave 10ms off tp.
Example of India prohibiting electronic communication between markets. Solution was to have guys watching two screens and typing in to two systems at the same time to do the arbitrage. Ie, Mechanical Turk

generation focused on transactional efficiency

war game over exposure to quais-proprietary information, who owns that and what to do with that?
notion that hedge funds give up their "clickstream" to prime brokers, hedge funds use alternative trading systems to try to hide their trading data from others
preserving anonymity by breaking apart transactions across multiple markets
"dark pools of liquidity"

"tradestation" -- all of the former proprietary data is available now in consumer package for $100. users can trade models with each other. darwinian evolution of trading models.
reminder that in answer o the 87 crash due to program trading imposition of hard stops.
slow down of markets during crises, is it a problem or a way of enforcing liquidity?
removing latency means there's less time for you to reconsider your decision if it was a mistake

werner vogels, cto of amazon

"web scale computing" , building your business around ideas and not resources
sales pitch for amazon's services

computing, storage, queuing
pay-go
GWA for real systems
claims 99.9% availability
amazon machine image -- how does this compare to media temple's grid service?
sharing of machine images by community
switch to rightscale.com demo

Jane McGonigal

url: www.avantgame.com/happiness . hacking the real world to be more like a game

What are the design strategies for showing users alternate realities?

technologists as happiness hackers

future forecast

quality of life as the primary metric for evaluating everyday technologies
positive psychology as a principla explcit influence on design
public expectation that tech companies have a vision for life worth living

science of happiness

"three realms of happiness"

pleasure - satisfaction
engagement - immersion, responsive systems
meaning: having a powerful role in the world at large

how does engagement in second life, games, influence real life?
ubiquitous, alternate reality games, activating the world around us

example: ministry of reshelving (flickrtag:reshelving) move "1984" from fiction to U.S. history, current events, etc.

example: assassin with random acts of kindness.
using games to teach interaction in the social environment
"world without oil" alternate reality for the public good. 4/30/07
worldwithoutoil.org

"the new games are supergames"

super sized, massive scale, lot of people engaged, online collective intelligences
superimposed (hybrid experience)
suerpheroic (real place, not role play)
supercomputing (parallel problem solving and collective action)

notion of being part of something bigger
one common design trait is that there is a call to action to work towards a larger goal.
search for shared meaning
ubiquitous games more popular with female players than you see in video games

happiness measures

UPenn measures, via questions/statements:
- "I am optimistic about the future
- I feel close to most people even if I do not know them well
- I feel that ....this makes the world a better place.

Six happiness classes

wisdom
courage
humanity
justice
temperence
transcendance

do our technologies pass the "death bed test"

happiness gained by assessing life backwards
How will the time spent with our technologies look in retrsospect? What kind of life will our technologies have enabled, supported?

Invest a portion of your timeenergy and resources towards undersand and innovating happieness, it's the new capital.

make your technology not only feel good but also do good and expose good
...two more that i didn't catch, but slides at http://avantgame.com/happiness

jeff hawkins

cofounder of numenta
why can't a computer be more like a brain?

have no computers today that you can show a picture and say what is this, where humans can do
no computers that can understand language or talk like a human

why are computers unable to do what brains do?

suppositons:

computers are not powerful enough
brains are too complex to understand
brains work on quantum principles
brains are "magic"

representations built hierarchically, need to build model of the world
built from the sensory data, learned.

hierarchical temporal memory (HTM)

a theory of how neocortex works
HTM also a technology
- hierarchy of memory nodes
- learns by exposure to sensory patterns

how does htm work?

creats a model of its world
recognizes new patterns
predicts
generates behavior,

numenta platform for intelligent computing
kind of losing the thread here.
demo of pattern recognition.

applications against precise timing or high order temporal data. See http://www.numenta.com

Some ideas about authentication

2007-03-28T09:07:31Z

Several years ago a friend posed this problem: her corporate intranet team was mandated to use the in-house search engine. This search engine had a fatal flaw: it had no way to index entitled or authenticated content, thus much of the company's intranet was invisible and not showing up in the search engine. Some solutions had been discussed: dual path the authentication code so permit access by the crawler, hardcode userids and passwords into the crawler, temporarily turn off authentication when the crawler did an index run. None were really practical: the company had a distributed intranet, there were hundreds of web sites with different sorts of authentication in use; hardcoding information is just asking for trouble as is temporarily turning off the security.

Although my friend has since left that company, I have been thinking about the problem since then, toying with solutions, thinking up variations, but never really putting ink to paper so to speak.

What started as a question about allowing robots access to entitled content digressed into issues about automatically registering and authenticating any user agent to such content, and in turn authenticating agents between web services and authenticating agents as proxies for users.

Although identity and authentication are of interest to me, they're not my specialty and I readily concede not being up to the minute with the latest information about OpenID or Cardspace. So I offer the following ideas with this disclaimer: they're just ideas, I do not claim any originality, I just would like to see if we can push solutions a bit quicker.

The ideas are briefly:

Automating the registration and authentication of robots and web crawlers to entitled content or what I call "robots.xml".
Automating the registration and authentication of consumers to entitled content, "agents.xml".
Use public keys to sign requests from agents to web APIs instead of asking users to submit their userid and passwords to third party sites.
Use browser based public keys to sign agent requests to authorize agents to act as proxies for the consumer.

I outline the first two ideas below and plan to expand on them in future posts. The last two (using public keys for API requests) need some more thought put into them. I think where my head is at is using the flickr API as a starting point, but standardizing on a PGP or S/MIME signature to sign requests rather than have each service come up with a different signature method.

Note: this has been modified to correct an error about which status code could return authentication metadata (from 403 to 401).

robots.xml

My initial solution to my friend's problem was to update the Robot Exclusion Protocol. Move from a flat ASCII text file format to an XML format, add in some elements to designate protected URLs and other elements to designate authentication methods, mix in a few complaints which have arisen over the years about robots.txt, simmer with a few demonstration APIs and then try to convince every automated agent to adapt a new REP.

As I have thought about it, instead I think what I would propose is an extension to Google's Sitemaps protocol to add in elements to denote what URLs are protected (perhaps <authenticate> or <entitle>?) and where the agent should go to register and request an authentication token. I would add a few additional stanzas or elements requesting (and presenting) contact and support information, something sorely lacking in the relationship between robots and sites today. Finally you'd want an element describing what the robot can do with the content (index? cache?).

A complying server would respond to a request for an authenticated document with a standard HTTP ~~403~~ 401 status, but include a reference to the robots.xml or sitemaps file covering the URL in a <link> element. Perhaps something like:

<link rel="authentication" href="http://example.com/sitemap.xml">

A complying user agent would not request the sitemap file until it received a ~~403~~ 401 status response. Only then would it look for a sitemap file (relying either on a Link: HTTP header or a <link> element in the response).

I will write about this in more detail in a separate post. My goal with this proposal would be to create a standard, XML based, method and process for granting access to authenticated content.

agents.xml

Once you have an automated means to request and grant access to authenticated content for use by robots, why not extend it to users as well? For starters, you likely want to collect different information from a robot than from a user. Secondly users are more concerned about their privacy and may not want to be automatically registered or logged into a web site.

Many browsers today support semi-automated authentication: they offer to store a "userid" and "password" if they encounter a form with an input element of type password. However, in some testing I did last year I discovered that browsers will typically take the field immediately "next to" the password field, without regard to whether it really is or is not the "userid" that matches the password.

What I propose then is a variation on the extended sitemaps.xml file, call it agents.xml if you need a name.

A complying server would respond to a request for authenticated content with a 403 status. In addition to the <link> element provided for robots, we add a second <link> element for non-robot user agents.

A complying user agent would retrieve the agents.xml file, parse it, determine whether or not it has registered for the site previously (if so, use the cached identification information to attempt to log into the site). If it hasn't registered before it should prompt the user to confirm if it should register automatically, disclosing what information the site is requesting and submitting a registration request once confirmed.

This is very similar to what Kim Cameron of Microsoft has proposed Cardspace. I need to delve into the Cardspace documents because I think that there is a there there that I've ignored due to an aversion to all things WS-* and Soap.

One difference is that I personally think it is bad web server form to use redirects to deny access to a resource. Where Cardspace uses redirects to a login form, I would use a 401 status.

A second difference is that I think that we should provide a way to automate registration and authentication to sites which are not ready to jump to third party authentication. If a site wishes to use userid / password authentication (granting access either through Basic / Digest authentication or Cookie based authentication) then so be it. Yes it is a pain, and it does not relieve us of the multiple userid/password problem, but if a site could cogently describe the endpoints for authentication and the rules for the userid, password, and other registration fields, browsers could intelligently managed one's account space and create smarter browser based account managers.

Nothing in this solution should preclude using OpenID, Cardspace, or whatever else comes along. The server should indicate what authentication methods it supports. The user agent should determine which methods it supports and either automatically authenticate or display the site's 401 error page which presumably would contain links to a non-automated registration scheme.

Again, I plan to write this up in more detail and work on some prototype implementations. I would like to read some more about Cardspace and the simple registration extensions to OpenID.