Tonight, I really feel like the world is changing.

The typical problem scientists and scholars in general have is that we need to be able to predict paradigms changes, or at least study them. But how can you know that things are changing while you are in it? Can humans study humans? Well, I’m not a social scientist, so I don’t have to worry, officially, with such issues… but all scholars are affected to some level by this paradox.

Well, I’ve been using inDiscover.net. Yes, I’m linking to my own project, well, it isn’t my project, but I’m involved from the side. So, it is self-promotion. Fine. Still, using inDiscover.net has made me realized how the world has changed. A bit like when Stephen Downes worked on his MuniMall portal project and, while the project was a failure, he realized that the world was changing and he embarked on a mission (see his value statement on his site).

Look on the right-hand-side of my blog, you should see my current playlist from inDiscover.net. All of this music is free. It is out there. You can download the MP3s and listen to the same music I listen to. No matter where you are in the world. You can then share your playlist with the world. You can have my playlist, live, as XML, that you can incorporate in any application, any web site.

I hope to write later on why I think this is a paradigm shift. We are beyond the world of blogs, beyond the Web… this is deep. I think it will eventually change society all the way.

Ok, I’m making many claims here… I need to write this up, but it is late…

Ok, I’m not attending WWW 2004, but Elliotte Rusty Harold (ERH) does and he reports very well on what he sees and hear. Anna and I are really kicking ourselves for not going to WWW2004… but this is a long story. Damn.

Data storage issues?

According to ERH, Rick Rashid who I think is head of Microsoft Research, asked an interesting question… what happens when everyone has 1 TeraByte of storage. It now costs US$1000 to have that much storage, but it will be dirt cheap soon enough.

First, how much is 1 TeraByte… I think it is 1024 GB… Ok… How many 1MB picture can I store… that’s 1024*1024… so about a million pictures… over a year… that’s about a picture every 30 seconds. So, yes, that’s a lot.

I’m reminded of this cool interview with Jim Gray, also from Microsoft Research… and a Turing award recipient… and a model for me as a researcher… Jim wrote this…

Today disk-capacity growth continues at this blistering rate, maybe a little slower. But disk access, which is to say, “Move the disk arm to the right cylinder and rotate the disk to the right block,” has improved about tenfold. The rotation speed has gone up from 3,000 to 15,000 RPM, and the access times have gone from 50 milliseconds down to 5 milliseconds. That’s a factor of 10. Bandwidth has improved about 40-fold, from 1 megabyte per second to 40 megabytes per second. Access times are improving about 7 to 10 percent per year. Meanwhile, densities have been improving at 100 percent per year.

So, we have to be careful: yes, we’ll all have 1TB of storage soon. In fact, the geeks like myself will soon have far more, I think. Just because I think I always had slightly more storage and CPU power than the average Joe (not much, but a little more). However, this unlimited storage world brings its own flock of problems. If you can record a million 1MB pictures a year… what if you want to trace back something in this giant collection of pictures? You can’t possibly look at all of them, it would be boring… plus, your computer might not have the bandwidth to display all of this in a very short time, even if it were useful.

Other people have blogged about this because it has other consequences. For the last few years, for example, I’ve been storing all of my important files, and their revision history, using CVS, on daniel-lemire.com. I can afford this because my host has affordable storage. This means that in several years, I’ll be able to go back in time and precisely see what I had in terms of research results on a given day. This is not exactly new: experimental researchers keep lab books. However, this is electronic data… which means that software can go, parse it, analyse it. Here’s a dream: a piece of software will go over my logs, learn how I do research and assist me in some way.

Is there anything solid to Semantic Web?

Here’s an interesting quote from ERH:

RDF and OWL enable standard, machine interpretable semantics. XML enables only syntax. Or at least so he claims. I agree about XML, but so far I’ve yet to see evidence that there’s more semantics in RDF/OWL than in plain XML. There’s a huge number of big words being tossed around this morning (business inferencing, .NET tier, dynamic applications, reclassify corporate data, proprietary metadata markup, “align the semantics of federated distributed sources”, “rich, automatic, service orchestration”, etc.) which mostly seems to obscure the fact that none of this does anything.

or

They’re investing in and exploring a broad range of applications: Semantic blogging, information portals, and SMILE Joint. He’s describing several application areas, but again I don’t see how RDF/OWL have anything to do with what he’s talking about. I realized part of the problem: no one is showing any code. I feel like I’m a mechanical engineer in 1904 listening to a bunch of other engineers talks about airplanes, but nobody’s willing to show me how they actually expect to get their flying machines into the air. Maybe they can do it, but I won’t believe it until I see a plane in the air, and even then I really want to take the machine apart before I believe it isn’t a disguised hot air balloon. A lot of what I’m hearing this morning sounds like it could float a few balloons.

I think that ERH think exactly like I think. You want to semantically mark up something using XML, you do this…

<team>
<player>joe</player>
<player>jill</player>
</team>

I’ve seen no evidence that there is any thing more out there you can do. Oh… there are extensive specifications, nice diagrams, but like ERH says, I haven’t seen the plane fly yet, just engineers talking about it.

Of course, I should point to this article which also goes in the same direction. And I’m currently talking with a new colleague, Jean Robillard who has his own arguments.

I had lunch with Anna today. Anna is the computational linguist I designed the slope one collaborative filtering algorithm with. What is slope one? No published paper yet, but that’s the algorithm Sean implemented in PHP/SQL for indiscover.net. It is a very neat algorithm and we hope to have a paper out on it soon!

Anna has a Ph.D. in linguistics from McGill… in Montr�al… the city we both love… She recently left a researcher job at NRC, she was my neighbour, for a job in some cool Montréal start-up. This start-up seems soooo cool. They might just change the world… well, a tiny bit of it. Of course, Anna couldn’t tell me more than what’s public knowledge, but it sounded so cool.

Unavoidable subject: Ph.D.s leaving pure research for industry and doing so totally on purpose. Anna chose to go back to industry. That’s a concept most tenured or half-tenured scholars don’t understand… But watch it! Anna is one hell of scholar, wherever she is…

We talked about the fact that academia doesn’t have words to describe scholars who go outside of academia… it is like… like they fall off the world.

Note: Some might object that NRC is not academia. And they’d be right… but somehow, NRC manages to walk a fine line and remain mostly an academic creature while being a giant government lab. So the difference is more of a nuance, really.

After meeting Seb last week in Montréal, now Anna… I’m having so much fun! The next one I want to see in Montréal is Harold: quit chatting with Tim Berners-Lee and come by Montréal. I might not be as smart as Tim, but I know some cool cafés Tim doesn’t know! (Explanation: Harold is attending WWW2004, and I think he is in a workshop together with Tim so he’ll probably talk to him. Harold is famous, you know. You get that way after working 16 hours a day for the last 20 years.)

People who are happy with whatever operating system they are offered probably find it much easier to change job. When you are a linux addict, it means that you have to secretly install Linux and then reverse-engineer the network configuration so that you can, well, print.

This time I installed Gentoo in my office. As it turns out, it was rather painless, but as an overworked prof., it was still hurtful to waste a day configuring the machine. The toughest part this time was getting the printing to work. As it turns out, I had to the cups to use smb://mylogin:mypassword@tlmnt4/uer_com instead of smb://tlmnt4/uer_com. Somehow, cups couldn’t just reuse my known username and password. Hmmm… I wonder why I never had this problem before? I really wish cups was easier to configure. But at least, it works now.

There is also a special java applet system called explor@ here. But I think I mostly figured out how to get it running “ok” under linux (in French), though I had to waste another day on it.

I have good reasons to believe I must be the only prof. around using Linux. My addiction to command line interfaces has a thing or two to do with it. You can emulate pretty much the unix environment under Windows, but it is never quite the same in terms of productivity.

I’ve been told that MacOS X would be a good choice too. Except that I couldn’t have done what I just did here: take the “free” PC they put in my office and transform it in a Linux box.

Some facts people often don’t know:

• Networking is mostly painless under Linux. Since most networks use DHCP, the configuration is a joke. With samba, you can access pretty much all of the network services you need even when they are hosted on a Windows server.
• With OpenOffice and latex2rtf, you can pretty live within the MS Word universe and not get noticed. People will complain that the documents don’t look quite as they expect, but I’m a prof. and I can always claim that I’m not very good with word processing. You can consume and produce Word documents. Not very good ones, but unless you do secretarial work, it will be ok.
• Email is not a problem even in a supposedly windows-only world: just use the exchange server as a POP server and you’ll be fine. Microsoft is not yet crazy enough to prevent people from checking their mail using the POP protocol. You might not get all the features from Exchange, but what you are missing won’t hurt you, much.

In the end, you can be very productive with Linux.

Now, if I could find how to turn the system bell off once and for all, I’d be happy.

Update: it turns out that we can turn off the system bell easily under Linux. I never knew this. Just do “xset b off”. You can put this in your .xsession file too.

This time, I found stats for Philosophy Ph.D.s. It would appear that the ratio of candidates to job advertize is shrinking quite a bit. I got this from Leiter Reports. In the same blog, we find evidence that 9 out 10 Philosophy Ph.D.s get a cool middle-class job, assuming they went to Princeton.

Well, there seems to be a lot more evidence than I thought that things are improving for new Ph.D.s

As it turns out, though, my claims are also supported by anedotes: Fang and Nielsen.

Sean has added RSS feeds to indiscover.net. You have the latests artists and
the latest songs. Seb even alluded to indiscover in his blog and hopefully, we’ll collaborate with him. After all, indiscover is meant to be an open house, not a closed group. Stephen Downes also mentions the new RSS feeds.

I recall that indiscover.net is a form of music recommender for independent musicians. It is the result of the last couple of years of work with a big team of folks from NRC where I was a researcher before coming back to academia.

Does academic research matter?

I’m not a very good historian, but I seem to recall that rresearch as we know it arose out of the German model. It proved invaluable at least in the Second World War. Or did it?

Of course, Tim Berners-Lee owe to academic research some of the ideas that lead to the Web. Some. But not that much, really. Tim Bray is not exactly from academia, is he? Yet, XML changed the world in a deep way.

It seems like academic research is more and more irrelevant… or is it progressively more underfunded, or mismanaged… or just simply totally irrelevant?

Here’s a theory: we’ve come to define success by the number of publications… yet, amazing folks like Tim Bray don’t necessarily go out of their way to submit papers. They listen, they talk, they write within communities and then they publish proposals. They probably hack some software too. So, maybe academic research is becoming irrelevant because we have success wrongly?

Why would the public respect people whose main achievement is a (smallish) number of 10 pages documents they get in books hardly anyboyd ever read.