Friday, April 17, 2015

Make it better, every day of the week

As programmers, if we haven't made our code base just a little bit better (preferably a whole lot better) for every new feature we add, we have failed. If no one but us can understand what a piece of code is doing or modify it in meaningful ways without bringing down the house, we have failed. If we take a shortcut/hack under pressure of deadlines or whatever but don't make it easy to spot/remove, we have failed.

As engineering management, if we don't deeply understand what makes our components good or bad, we have failed. If we can't set up processes and tools to monitor the health of our code, we have failed. If we can't motivate our teams to love their code and to want to make it better, we have failed.

More words don't make a better song. More brush strokes don't make a better painting. More features/LoC don't make a better product.

In his own (perhaps small) way, every good software engineer is a visionary. He pre-empts the future by building it today. When he can't predict the future, he build room in the present to handle it tomorrow. Our legacy is not the code we write today, but the code other will write using it tomorrow.

So what did you improve today?

Thursday, May 9, 2013

Why Anti-Authoritarians are Diagnosed as Mentally Ill

This is a repost of a fantastic article I found somehow.
In my career as a psychologist, I have talked with hundreds of people previously diagnosed by other professionals with oppositional defiant disorder, attention deficit hyperactive disorder, anxiety disorder and other psychiatric illnesses, and I am struck by (1) how many of those diagnosed are essentially anti-authoritarians, and (2) how those professionals who have diagnosed them are not.
Anti-authoritarians question whether an authority is a legitimate one before taking that authority seriously. Evaluating the legitimacy of authorities includes assessing whether or not authorities actually know what they are talking about, are honest, and care about those people who are respecting their authority. And when anti-authoritarians assess an authority to be illegitimate, they challenge and resist that authority—sometimes aggressively and sometimes passive-aggressively, sometimes wisely and sometimes not.
Some activists lament how few anti-authoritarians there appear to be in the United States. One reason could be that many natural anti-authoritarians are now psychopathologized and medicated before they achieve political consciousness of society’s most oppressive authorities.
Why Mental Health Professionals Diagnose Anti-Authoritarians with Mental Illness
Gaining acceptance into graduate school or medical school and achieving a PhD or MD and becoming a psychologist or psychiatrist means jumping through many hoops, all of which require much behavioral and attentional compliance to authorities, even to those authorities that one lacks respect for. The selection and socialization of mental health professionals tends to breed out many anti-authoritarians. Having steered the higher-education terrain for a decade of my life, I know that degrees and credentials are primarily badges of compliance. Those with extended schooling have lived for many years in a world where one routinely conforms to the demands of authorities. Thus for many MDs and PhDs, people different from them who reject this attentional and behavioral compliance appear to be from another world—a diagnosable one.
I have found that most psychologists, psychiatrists, and other mental health professionals are not only extraordinarily compliant with authorities but also unaware of the magnitude of their obedience. And it also has become clear to me that the anti-authoritarianism of their patients creates enormous anxiety for these professionals, and their anxiety fuels diagnoses and treatments.
In graduate school, I discovered that all it took to be labeled as having “issues with authority” was to not kiss up to a director of clinical training whose personality was a combination of Donald Trump, Newt Gingrich, and Howard Cosell. When I was told by some faculty that I had “issues with authority,” I had mixed feelings about being so labeled. On the one hand, I found it quite amusing, because among the working-class kids whom I had grown up with, I was considered relatively compliant with authorities. After all, I had done my homework, studied, and received good grades. However, while my new “issues with authority” label made me grin because I was now being seen as a “bad boy,” it also very much concerned me about just what kind of a profession that I had entered. Specifically, if somebody such as myself was being labeled with “issues with authority,” what were they calling the kids I grew up with who paid attention to many things that they cared about but didn’t care enough about school to comply there? Well, the answer soon became clear.
Mental Illness Diagnoses for Anti-Authoritarians
A 2009 Psychiatric Times article titled “ADHD & ODD: Confronting the Challenges of Disruptive Behavior” reports that “disruptive disorders,” which include attention deficit hyperactivity disorder (ADHD) and opposition defiant disorder (ODD), are the most common mental health problem of children and teenagers. ADHD is defined by poor attention and distractibility, poor self-control and impulsivity, and hyperactivity. ODD is defined as a “a pattern of negativistic, hostile, and defiant behavior without the more serious violations of the basic rights of others that are seen in conduct disorder”; and ODD symptoms include “often actively defies or refuses to comply with adult requests or rules” and “often argues with adults.”
Psychologist Russell Barkley, one of mainstream mental health’s leading authorities on ADHD, says that those afflicted with ADHD have deficits in what he calls “rule-governed behavior,” as they are less responsive to rules of established authorities and less sensitive to positive or negative consequences. ODD young people, according to mainstream mental health authorities, also have these so-called deficits in rule-governed behavior, and so it is extremely common for young people to have a “duel diagnosis” of AHDH and ODD.
Do we really want to diagnose and medicate everyone with “deficits in rule-governed behavior”?
Albert Einstein, as a youth, would have likely received an ADHD diagnosis, and maybe an ODD one as well. Albert didn’t pay attention to his teachers, failed his college entrance examinations twice, and had difficulty holding jobs. However, Einstein biographer Ronald Clark (Einstein: The Life and Times) asserts that Albert’s problems did not stem from attention deficits but rather from his hatred of authoritarian, Prussian discipline in his schools. Einstein said, “The teachers in the elementary school appeared to me like sergeants and in the Gymnasium the teachers were like lieutenants.” At age 13, Einstein read Kant’s difficult Critique of Pure Reason—because Albert was interested in it. Clark also tells us Einstein refused to prepare himself for his college admissions as a rebellion against his father’s “unbearable” path of a “practical profession.” After he did enter college, one professor told Einstein, “You have one fault; one can’t tell you anything.” The very characteristics of Einstein that upset authorities so much were exactly the ones that allowed him to excel.
By today’s standards, Saul Alinsky, the legendary organizer and author of Reveille for Radicals and Rules for Radicals, would have certainly been diagnosed with one or more disruptive disorders. Recalling his childhood, Alinsky said, “I never thought of walking on the grass until I saw a sign saying ‘Keep off the grass.’ Then I would stomp all over it.” Alinsky also recalls a time when he was ten or eleven and his rabbi was tutoring him in Hebrew:
One particular day I read three pages in a row without any errors in pronunciation, and suddenly a penny fell onto the Bible . . . Then the next day the rabbi turned up and he told me to start reading. And I wouldn’t; I just sat there in silence, refusing to read. He asked me why I was so quiet, and I said, “This time it’s a nickel or nothing.” He threw back his arm and slammed me across the room.
Many people with severe anxiety and/or depression are also anti-authoritarians. Often a major pain of their lives that fuels their anxiety and/or depression is fear that their contempt for illegitimate authorities will cause them to be financially and socially marginalized; but they fear that compliance with such illegitimate authorities will cause them existential death.
I have also spent a great deal of time with people who had at one time in their lives had thoughts and behavior that were so bizarre that they were extremely frightening for their families and even themselves; they were diagnosed with schizophrenia and other psychoses, but have fully recovered and have been, for many years, leading productive lives. Among this population, I have not met one person whom I would not consider a major anti-authoritarian. Once recovered, they have learned to channel their anti-authoritarianism into more constructive political ends, including reforming mental health treatment.
Many anti-authoritarians who earlier in their lives were diagnosed with mental illness tell me that once they were labeled with a psychiatric diagnosis, they got caught in a dilemma. Authoritarians, by definition, demand unquestioning obedience, and so any resistance to their diagnosis and treatment created enormous anxiety for authoritarian mental health professionals; and professionals, feeling out of control, labeled them “noncompliant with treatment,” increased the severity of their diagnosis, and jacked up their medications. This was enraging for these anti-authoritarians, sometimes so much so that they reacted in ways that made them appear even more frightening to their families.
There are anti-authoritarians who use psychiatric drugs to help them function, but they often reject psychiatric authorities’ explanations for why they have difficulty functioning. So, for example, they may take Adderall (an amphetamine prescribed for ADHD), but they know that their attentional problem is not a result of a biochemical brain imbalance but rather caused by a boring job. And similarly, many anti-authoritarians in highly stressful environments will occasionally take prescribed benzodiazepines such as Xanax even though they believe it would be safer to occasionally use marijuana but can’t because of drug testing on their job
It has been my experience that many anti-authoritarians labeled with psychiatric diagnoses usually don’t reject all authorities, simply those they’ve assessed to be illegitimate ones, which just happens to be a great deal of society’s authorities.
Maintaining the Societal Status Quo
Americans have been increasingly socialized to equate inattention, anger, anxiety, and immobilizing despair with a medical condition, and to seek medical treatment rather than political remedies. What better way to maintain the status quo than to view inattention, anger, anxiety, and depression as biochemical problems of those who are mentally ill rather than normal reactions to an increasingly authoritarian society.
The reality is that depression is highly associated with societal and financial pains. One is much more likely to be depressed if one is unemployed, underemployed, on public assistance, or in debt (for documentation, see “400% Rise in Anti-Depressant Pill Use”). And ADHD labeled kids do pay attention when they are getting paid, or when an activity is novel, interests them, or is chosen by them (documented in my book Commonsense Rebellion).
In an earlier dark age, authoritarian monarchies partnered with authoritarian religious institutions. When the world exited from this dark age and entered the Enlightenment, there was a burst of energy. Much of this revitalization had to do with risking skepticism about authoritarian and corrupt institutions and regaining confidence in one’s own mind. We are now in another dark age, only the institutions have changed. Americans desperately need anti-authoritarians to question, challenge, and resist new illegitimate authorities and regain confidence in their own common sense.
In every generation there will be authoritarians and anti-authoritarians. While it is unusual in American history for anti-authoritarians to take the kind of effective action that inspires others to successfully revolt, every once in a while a Tom Paine, Crazy Horse, or Malcolm X come along. So authoritarians financially marginalize those who buck the system, they criminalize anti-authoritarianism, they psychopathologize anti-authoritarians, and they market drugs for their “cure.”

Saturday, December 8, 2012

The Official Solomon Says blog

Here's announcing  the official blog for Solomon Says.

I am no hotshot blogger, but I understand when Robert Scoble complains about noise. We follow blogs for some particular topics, and if the author starts spamming us with a whole bunch  of  other stuff, it becomes outright annoying.Over the last few months, I have devoted all of my time and energy to building Solomon Says into a useful website, and consequently my blogging has also taken on the form of a journal of my efforts. This was not a conscious decision, but I write what I am thinking about, and what I was thinking about was the site.

On the other hand, I felt compelled to share with my readers what I was planning with Solomon Says and the reasons for doing what I was doing. Since this blog was the only web destination I had at that time, this is where all the posts came. But some of you may be reading for entirely different reason and do not care about what changes I am making to the homepage. Other may only be interested in the progress of Solomon Says.

To avoid alienating both set of reader, I decided to separate the two data streams. Starting today, I am moving all blogging related to Solomon Says over to its own home, which will be the hub for all news, discussions, and brainstorming.

Go check it out, and subscribe if you would to stay abreast of what's happening in the world of online reviews.

Friday, November 23, 2012

Alway keep a backup. Of everything.

Let me tell you a story of failures and backups and pain.

So last night I finished a bunch of changes to Solomon Says.  After the regular load of testing (that lasts 15 minutes and includes opening a bunch of pages on Firefox and Chrome), I uploaded the changes and tried to bring the server back up. Everything exploded in my face at about the same time. The only reason we are still in business is that I had backups. In decreasing order of importance, the following backups saved the day:
  1. Database
  2. Code/Configurations
  3. Images
So pretty much everything :)

At this point, a note on the deployment process is in order. Here’s how it goes:
  1. Stop python fcgi process and nginx service.
  2. Delete the production code.
  3. Run the DB migration script.
  4. Upload the entire code from my laptop to the production location.
  5. Start python and nginx
#2, #3, and #4 didn’t go too well.

#2 – My dev. environment is Windows, but production is on LINUX.  So there’s a bunch of stuff related to path handling (‘/’ vs ‘\’ etc.) that I change just for development. This is automatically handled in production by using a different configuration file. Alas, I ran the delete for #2 from one level higher in the directory structure. Boom goes the config. And on bringing the server up, I get a load of ‘access permission denied’ errors. I spent a half hour analyzing the arcane debug messages, then give up and restore the entire code base file by file and change by change.

#3 – I missed selecting a couple of ‘where’ conditions when running the migration script. Result – 2 of the main table got randomly changed. Considering how crappy the day had been so far, I realized it only on restarting the server. So bring the server down again, restore the DB to its previous avatar from the backup, and run the migration with extra precaution.

#4 – My development copy did not have quite a few of the images related to the newer reviews I had posted. And since I had deleted the production data in #2, the server started throwing ‘Suspicious Operation’ exception (What the hell is that? It should have said ‘File not found” or something). In view of the blunders I had made for #2 and #3, I assumed a mistake in the new configuration I had created and spent another hour debugging, then gave up and copied over the image folders from the back up to production.
All told, something that should have taken 15 minutes took 4 hours.
Lesson learnt. Always keep a backup. Of everything.

Wednesday, November 7, 2012

Looking just a little bit better

A lot of the feedback that I have received on Solomon Says (a big thank you to everyone who spent time and effort providing it) has been regarding the styling and design aspect of the website. Or rather, the lack of it styling and design aspects in the website. Now, I am no designer. CSS3 and templating were not quite my fortes when I started working on it. So operating out of my ignorance of these fields, I have been forced to improve the design of the site in increments. Get something working, make it usable, and put it out there. Then improve what it looks like in the next iteration.

I'd like to share with you some of the changes I'm currently working on to the layout of the review pages. This primarily involves improving the data panel just above the text of the review. For the uninitiated, this is what it currently looks like on book and travel reviews respectively.

Both look very cramped and difficult to interact with. The huge orange rating section is sort of a waste of space, and the images don’t get due prominence (especially harmful on travel reviews). So I thought through these problems and came up with a small redesign which hopefully makes everything cleaner and easier to access. Check it out.

The new version is only slightly different from the current one but I think it lends a much more spaced-out feel to the whole page. You would also have noted that there is a small panel of image thumbnails right above the ratings section. These are the images that currently show up below the text of the review, like so:

I never really this design because pics are cool and everybody loves them. So I moved the images right to the top in a combination of sliding thumbnail carousel and Fancybox. Now they are easily accessible, and clicking on the thumbnail gallery blows them up to full size too! Like this.

A lot more groovy, even if I say so myself! I am planning to roll out the changes in about two weeks after a few minor tweaks and testing.

So what do you think? Like the new look? Not quite? Let me know in the comments or drop me a line at Suggestions/flowery words of praise/hate mail are all welcome.

Solomon Says at ISB

First things first – Please fill out this short survey. This will help me in assessing what I can do to make Solomon Says more exciting and useful for its users. I really, really appreciate it.

Now for the news of the week.

Solomon Says is currently the subject of a marketing project/case study in a course on Entrepreneurial Decision Making (EnDM) at ISB (Indian School of Business). The project is being conducted by Varun Jain (a very close friend of mine from my undergrad days at NSIT) of the ISB class of 2013 under Prof. Arun Pereira. Over the course of the project, I will be working with the two aforementioned gentlemen (mostly with Varun) to conduct market surveys, audiences analysis and other analytical wizardry to refine SolomonSays into an even more awesome product.

Quick background on how this came about. Essentially Varun was looking for a start-up to whet his new-found marketing chops on.. I was going around writing reviews and hacking away to glory with no time for reaching out to the wide world and finding a place in it. We discussed the website one day, and agreed that it could use some MBA lovin’. So starting this week with survey mentioned above, we’ll be doing some basic scoping exercises to (hopefully) understand out audience and define our market with a lot more clarity than before. These efforts will also try to discover how readers interact with the website and what we can build into it to make that experience smooth.

I have written before that I do not have a proper business plan yet for SolomonSays. Throughout this ISB affair, my focus will continue to be on how to make this the best, most helpful reviews website on the web. No doubt there are parts of the project which demand an emphasis on revenue streams and sustainability, but those come later. Till then, the spotlight, my dear readers, is on you.

Don't forget the survey.

Tuesday, October 23, 2012

Meet SolomonSays : All the review you will ever need

This post has been coming for a few months, but somehow I could never get around to writing it. But a certain milestone has been reached today that makes me want to sit down and write this. A minor milestone, no doubt, but something is better than nothing right? Over the last 10 months or so I have been working on a website called SolomonSays. This is a blog-in-website-shape where I review books, travel destinations, and eateries (though I am yet to get any traction on the last one). The milestone – 100 likes on the facebook page.

I have referred earlier to the broken-ness of the crowd-sourced or community driven model of getting reviews of products. It is great at getting lots of info and scales insanely, but none of this data is in a very usable form (short of going through tons of reviews). And apparently, I’m not the only one cribbing either. The other problem is that one has to go to so many different sites to get reviews on different things despite the fact that all these websites are doing the exact same thing – gathering user generated review content.

I first faced this problem while looking up book recommendations and reviews. So I set out to do what I thought was the right of reviewing things. Which, I believe, is the old fashioned way of having a team of dedicated reviewers writing comprehensive reviews about whatever it is that is being reviewed. I believe that no one should have to go through 300 one-off personal experiences to get an idea what a new restaurant or car is like. These are critical elements of a discussion, but do not comprise a coherent review. Combining the two approaches, i.e. getting a community to offer their individual opinions around a central review, is the clear winner IMO.

My original goal was extraordinarily ambitious – would review absolutely _everything_ that can be reviewed. Books, Restaurants, Home appliances, vehicles, were all fair game. While I realized that actually doing this wouldn’t be possible all at once, that was where I believed I wanted to get to. Such a website, if it could be created, would occupy a unique position between Wikipedia and Amazon – the reviews would be free, and we would be able to facilitate any purchase the reader wanted through affiliate partners. But the content would be the business. Data would come first. ALWAYS. I still hold to the original unbounded scope of the website - but getting there is being done only in baby steps.

All MBA and otherwise savvy reader will have seen that the above isn’t much of a business plan – write reviews till you have enough content to drive steady traffic through SEO, and then try to convert that traffic into sales. In all honesty, I didn’t think too much in terms of a business idea (I still don’t). I was extremely excited by the idea of having all this data under one roof, the day job wasn’t too interesting at the time, so I just ran with it. One of my friends whom I initially discussed this with still hasn’t stopped harassing me about monetization.

The time spent working on this has been very exciting, and very tiring. I essentially have two jobs, so I get to hang out with my friends a lot less that I used to. For all that, the learning has been tremendous (I have written earlier about the technical experience). But now I am paid to read books (not yet really, but soon will be, I hope J), and writing about what I read clears things up in my head too. Besides, there’s no better feeling than when drops a line saying how useful they found something that I wrote. 

At this small occasion, I thank everyone who has supported me thus far (Special mention – Aditya Mangla, Sid Reddy, Sreejith, Kalpana, and Sin City). Do drop me a line with your ideas, suggestions, and review requests (or if you are willing to share your beer). Random gossip and philosophical musings are welcome too. 

Whatever I code or write is only half the answer. The rest of the awesomeness comes from your experiences and participation. I will continue to bring more information to this party. Hope you’ll stay. Let’s clean up this morass of online reviewing!

Wednesday, August 29, 2012

The problem with crowdsourced content

We have whole lot of sites featuring reviews written by their readers on pretty much all topics. Of these, at least the front runners attract large traffic volumes, have a devoted user base, and are, I assume, making money. However, I want to discuss what I think is wrong with them.

The problem is the method of generating content. All these sites rely on a bevy of users to come write reviews on the products/service that the site focuses on and that they have used/experienced. This can be restaurant, book, travel, gadgets, or whatever else. Via friendly UIs, facilitated social media interaction etc., users are encouraged to contribute data for each others benefit. This strategy is very effective in generating large amounts of data. However, it is very bad at generating cohesive data.

In general, I have three somewhat interconnected problems with this:
  1. Data is of poor quality – Since the website wants readers to submit reviews, it can rarely hold them accountable to the quality of their writing. The aim is to lower the barrier to writing and social media sharing. Get him to write. No matter he writes, get him to write. To be fair, most of the reputed website will intervene if you write inflammatory or profane material, but apart from that, pretty much anything goes. As a direct consequence, the quality of review in terms of both the content (what is written) and the form (how it is written) goes down. Most people write unbalanced reviews, either giving full marks and endless praise or griping about a very bad experience.This can be avoided to some extent by making sincere efforts at moderating and community building (StackOverflow is a great example), but none of the major commercial websites seem to be doing so.
  2. There is just too much data – This is the explicit result of successfully inducing readers to submit content to a site. Since anyone can submit anything, the data volumes are large, and it becomes well-nigh impossible to find information. This is what I like to call the Problem of 500 reviews. What I mean by that is that on any successful reviews site, today you can find 500 reviews for pretty much every single item. Too much data is not much better than no data. The best this deluge of disjointed reader inputs can give us is a general sense of how people like something. As an experiment, choose any famous book about which you know nothing, and try to find out about the book using only GoodReads reviews. I am fairly regular on the site, but I mostly do it for the bragging rights, and to let my friends know what I am up to.
  3. Data is without structure – I totally agree that a successful travel site of the kind that we are talking about will have all data about some destination. But how do we find this data? Since it is broken up across a large number of unconnected reviews, it is very difficult to present the information in a coherent, intuitive manner. It is now left to the reader to sift through the data that each reviewer has provided and collate the data he needs (when to get there, how to get there etc.)
The crowdsourced content model is like a group discussion where everyone is talking at the same time. There is no anchor or reference around which a discussion can be built.

IMO, a far better alternative is to have an informed member write one review, and then use that to gather all sorts of varied and personalized experiences regarding the topic of discussion. It may seem so up front, but such a model (critic-driven model, if you will) is not about classroom style information broadcast. The Expert has not spoken. It is about providing a structured core, the basic information, and then inviting the readers to extend that into a wider body of information. If you want people to spend time and effort sharing their opinion, it is only fair that you offer them something in return.

Monday, August 27, 2012

A sample Ladakh itinerary and a vote of thanks

So yours truly went to Leh two weeks ago. It was, as expected, a mind-blastingly awesome experience.

It is not the most routine of trips, so I pulled up my Google socks and got down to business. Although data is widely available on each individual destination in Leh, I found difficult to plan an itinerary since no one was very clear how long each place would take (I found Nubra Valley estimates from "Dont even bother going there" to "It will definitely take 4 days"). The tour operators will certainly make one for you, but the geeks (and the jock for that matter) _has_ to work it out for himself, right?

I eventually had to call up friends who had been there to get their itineraries. Not cool. I don't want to be real life social when I can be online social. So to save all you worthies this trouble, here is my itinerary:
Day 1
Arrive Leh and overnight Stay. Check out Leh Palace and Shanti Stupa. Chill in random cafes.
Day 2
Leh-Like Monastery - Gurudwaara Pathhar Sahib - Lamayuru Monastery-Ule :  
You can actually visit Alchi the same day and come back if you rush somewhat, but why bother? The nice Ule resorts are built overlooking the river. Ergo, overnight stay.
Day 3
Ule-Alchi-Leh : Alchi monastery is nice and old. You can visit Basgo Palace on the way back, but it is really just a pile of mud and stone now and not worth the climb. Also see Hall of Fame - an army memorial/museum for our high altitude warriors. Respect.
Day 4
Leh to Nubra Valley - The drive is awesome. Bragging rights are earned by clicking pics at Khardungla (the highest motorable pass in the world at 5600-ish metres). Then bactrian camel ride at Diskit. It's  beautiful here and a hard ride back, so stay overnight.
Day 5
Drive back to Leh. Relax.
Day 6
Leh - Chang La - Pangong Lake : Changla is the third highest motorable pass in the world (14500 ft. Reach camp around lunchtime. Sit on the banks of the beautiful lake. It's frikking cold and windy.
Day 7
Pangong Lake - Leh. Enroute visit Hemis, Thikse Monasteries and Shey Palace.
Day 8
Go home
Everything was done at a mellow pace, no rushing. Spending less time than this would not be doing justice to them IMO. Also, this is a fairly touristy plan, since this was my first time.

For all the research, we still booked a packaged tour (I'll hulk out the next time, I promise) via Escapades India. That almost got effed up due to the recent flash floods. We were stuck in Manali for 2 days, and missed the first two days of our itinerary. But the folks over at Escapades were EXTREMELY helpful, stayed in touch constantly to find out if we were doing okay. I called the unputdownable Mr. Parag more times than I called my family and he just might have done the same! A mail of theirs said "On adventure trips, you want to worry more about the reliability of your operator rather than what he is charging you" (paraphrased). Parag and Co. certainly came through on that front.

We extended our trip by two days to compensate for the things we had missed, and Escapades arranged it for no extra charge. Read that again. Slowly. No extra charge.

So all in all I strongly recommend anyone looking for the arranged travel option to go to Leh (or elsewhere, but I'm not very sure of  that) to get in touch with Escapades India.

Monday, May 28, 2012

Chance knowledge and the cumulative effect

As I was driving past a park this Saturday, I saw that a section of it was burning. I suddenly had a vivid image of the entire park burning with high flames (It is a very large park). This then caused a thought about how Rome would have looked to Nero as it burnt. The revelation – I now know why Nero (the software) is so named.

This sort of random revelation is not an unusual thing. The most important part here was that I wasn’t thinking about computers at all. The experience was quite Sherlock Holmes-ish – When he gives voice to Dr. Watson’s thoughts by saying “Such a waste” and then proceeds to explain the chain of thought in the latter’s mind. Richard Bach compared ideas to fractures running in a crystal – any could lead to any other. Others have referred to the oneness of knowledge – that given one thing, everything else could be discovered. Both seem somewhat limited ways of explaining the experience.

Chance discoveries happen in science all the time. Microwave background radiation and radioactivity  are but two examples.

I’m thinking now of the cumulative nature of scientific knowledge. Both socio-economic theorists (Farcis Fukuyama, Fareed Zakaria etc.) and scientists (Francis Bacon) assert that with the invention of the scientific method (hypothesis-experiment-prove/disprove), scientific knowledge has become cumulative. That later generation inherit the knowledge of their predecessors. And it is intuitively true. We don’t have to rediscover the gravitational principle, although we may have to prove it several times over (damned CBSE exams!).

This is brought into somewhat contrary focus by the revelational experience. The cumulative nature of the scientific method no doubt holds true when the steps are small. Not to belittle any discoveries, but some _are_ greater than others. It took a Newton to get us to gravity, and even more drastically, an Einstein to get to relativity. Both works are quite out of the league for their times. The tools they used were there, as was the a-priori knowledge. But the power of the method seems to wane when we consider efforts where accepted first principles have to be discarded.

The cumulativity assertion says that given a state of human knowledge and the scientific method – the future can be worked out again. But how does this apply when the prior knowledge has to be discarded by a leap of faith.  When accidents force the next step forward, how does the scientific principle handle it? We can say that the geniuses use informed intuition (Kekule and the structure of Benzene). But that would lead us into the myriad definitions of genius and intuition.

And that’s a discussion for some other time.