Social Media as Prediction Tool

Even as several social networks have surpassed the populations of most nations — more than 500 million people are now signed up for Facebook and 175 million for Twitter — we still tend to regard these sites in terms of their value to us as individual users. In the past year, however, social scientists have begun looking more broadly at the aggregate value of social media. According to a number of recent studies, it now seems possible that the networks’ millions of posts and status updates are adding up to something culturally and financially priceless.

This past April, for instance, Sitaram Asur and Bernardo Huberman at HP Labs demonstrated that by analyzing the positive or negative sentiments expressed in 2.8 million Twitter messages about 24 movies, they could predict how the films would perform at the box office. Their methodology — an algorithm, actually, that their company is now in the process of patenting — worked significantly better than the Hollywood Stock Exchange, another popular tool for predicting box-office success.

In October, a team led by Johan Bollen at Indiana University reported that by classifying 9.7 million Twitter posts as falling into one of six mood categories (happiness, kindness, alertness, sureness, vitality and calmness) they could predict changes in the Dow Jones Industrial Average. As Bollen explains, when he began his study, he expected that the mood on Twitter would be a reflection of up and down movements in the stock market. He never imagined it would be a precursor.

Twitter’s daily feed challenges some of the conventions of social science. Those old methods of fact-gathering — finding a small population for study, applying for a grant and then spending years doing observations or surveys — can seem quaint by comparison. Of course, it’s hard to ignore the mercenary aspects of big data, too. Any technology that might help predict the future does not come cheap. Twitter and Facebook — which last year used an analysis of posts to create a daily happiness index — can easily mine information about consumer products, in real time, from their data streams. “Given that there’s a population of 250 million people or more logging in every day, it seems ridiculous that you would try to get in touch with them through a phone,” says Cameron Marlow, the head of Facebook’s data-science team. “So it seems like it’s improbable that social media won’t be the way that we acquire opinion research.”

via

D.I.Y. Macroeconomics

Until recently, the economics profession largely controlled the production, dissemination and interpretation of economic data. Now there’s a new trend afoot: do-it-yourself macroeconomics, in which ordinary citizens pull apart the data and come to their own conclusions.

The democratization of economics owes much to the financial crisis that first hit in 2007. That ongoing catastrophe, which few economists predicted, tarnished the profession’s reputation, prompting some to look elsewhere for answers. They turned to — where else? — the Internet, where vast amounts of economic data that had once been hidden from public view were now online. Sites like FRED, maintained by the Federal Reserve Bank of St. Louis, enabled anyone with a connection to the Web to download data on everything from local home-price indexes to credit-card balances to weekly fluctuations in diesel prices.

At the same time, a growing army of knowledgeable “econo-bloggers” began analyzing the data available online. Strikingly, many of the authors of these blogs — the brains behind the Big Picture, Calculated Risk, Mish’s Global Economic Trend Analysis and others — aren’t academic economists but people with real-world experience in financial markets. Their Web sites offer sophisticated interpretations of economic data and hold passionate debates with their readers over the merits of the data. As a result, economic data that were formerly greeted with grudging acceptance by the public — the latest unemployment figures, for example — are now the catalyst for endless popular exegeses.

via

The Missing Near Future

The near future – let’s peg it 2020 and beyond -- is a blank because there is almost no vision of a near-future that seems both desirable and plausible. Most stories, “worlds,” and scenarios of say the year 2050 are dystopian. Take your pick of nuclear self-annihilation, mortal pandemics, planetary floods, robotic overthrow, alien invasion, or fascist apocalypse. They are all very plausible, but not desirable.

The advantage of the far future is we don’t have to be told the story of how we arrived there, of how we passed through the near future. It’s far away enough that the creators can punt past it. But the near future is such a conundrum that is it has disappeared from our culture.

Computer scientist and inventor Danny Hillis, born in 1956, noticed that when he was a child the future was ‘far away’ in the year 2000, but that as he grew older, the future remained rooted to the year 2000, as if newness could not move beyond that boundary. He describes it as feeling as if the future was “shrinking” year by year until in 1999 the future was only one year long. Now that we’ve passed through 2000, the future has effectively disappeared – except for the far far future.

This disappearance has been made more real by key science fiction authors, futurists, and the brainy, nerdy folks who ordinarily keep busy churning out visions of tomorrow. A common belief in this circle is that things are moving so fast and weird that it is physically impossible to imagine the future of 2050 and beyond. Many of these futurists believe this discontinuity, called the singularity, is eminently desirable, because it is sure to lead to great intelligence, greater wealth, greater heath and immortality. But because it is forecast to come about via the shattering of what we understand humans to be now, many others will resist this future at all costs. And others believe the singularity future is not only undesirable, but implausible.

Either way, we are left with a blank for the near future. We have no story of progress that fits in the next century. There is no vision of 50 years hence that billions of people on earth would say, yes, that is what I want. Billions of people in the developing world know what they want tomorrow: clean water, free education, self-governance, cheap consumer goods, and hope for their kids. But beyond that, what? What do the billion in developed nations want? A clean environment and opportunities for meaningful work and ……?

...there is no shared vision of what this is headed, or even where we’d like it to head. There’s no agreed way on measuring if in fact our actions are titled in the desirable direction – because there is no desired direction. Muddling through blind is the default scenario of the near future.

Via the Great Kevin Kelly. Read it all here.

The Dionysian Mysteries

The Dionysian Mysteries were a ritual of ancient Greece and Rome which used intoxicants and other trance-inducing techniques, such as dance and music, to remove inhibitions and artificial societal constraints, liberating the individual to return to a more natural and primal state. It also afforded a degree of liberation for the marginals of Greek society: women, slaves and foreigners.

In their final phase the Mysteries apparently shifted from a chthonic, primeval orientation to a transcendental, mystical one, with Dionysus altering his nature accordingly (much in the same way as happened in the cult of Shiva).

By its nature as a mystery religion reserved for the initiated, most aspects of the Dionysian cult remain unknown and were lost with the decline of Greco-Roman polytheism. Our current knowledge is largely based on descriptions, imagery and comparative cross-cultural studies.

The idea of a mystery religion consisted essentially of a series of initiations which benefited the individual or their society in some way. Initially associated with the passage from childhood to adulthood and maturity, they later became seen as what we might call an evolutionary rite. And it was in the form of a mystery religion that the Dionysus Cult was first channeled in a more civilized way, probably first in Minoan Crete.

The notion behind the Dionysian Mysteries seems to have been not only of the affirmation of the primeval bestial side of mankind, but also its mastery and integration into a civilized psychology and social culture. Given the dual role of Ariadne as the Mistress of the Minoan Labyrinth and consort of Dionysus, some have seen the Minotaur story as also partly deriving from the idea of the mastery of mankind's animal nature, though this remains controversial. The self mastery achieved in this way was not one of domination as in similar cults, most famously preserved in contemporary culture as George and the Dragon, and perhaps the original Minotaur myth, but one of acceptance and integration. Thus while the Mysteries did much to lighten the darker aspects of the cult they often failed to reassure its perhaps excessively civilized critics and continued to be regarded by many as dangerously liberative (particularly given its egalitarian tendencies as well).

In Athens, atavistic possession was also channeled into dramatic masked ritual within the Bacchic Thiasos (Greek equivalent of a 'coven' or 'lodge'), seeding the emergence of acting and theatre, crafts also sacred to Dionysos, particularly in the form of tragedy and comedy. Thus the Dionysian Mysteries came to be seen not only as a recognition and casting off the repressive, over-civilised masks we all wear, and the realisation of our true nature, but with the creation of new, more authentic masks as well, arguably also the deeper function of drama and comedy; in other words, the development of genuine character rather than socialised persona. In time, as Dionysos became regarded as less bestial and more mystical, with the general shift of Pagan orientation, this also came to be seen as the generation of a soul and the survival of death. These themes would become central to the later Orphic manifestations of Dionysianism that would influence early Christianity according to Roman commentators, denounced as devilish mockery of Christ by Justin Martyr.

Full article

Visualizing Medical Data

List of Human Error

FRAMING BIAS
Exactly how an issue is framed can affect how we think about it. Compare, for example, the two questions “Should Britain adopt the euro?” and “Should Britain abolish the pound?” Under perfect rationally, this framing should not matter.

REPRESENTATIVENESS
People have a bad habit of drawing big conclusions from very small and biased sample. For example, we might talk to three friends in the office, each of whom has coincidently had a bad day, and conclude that the company is falling apart.

AVAILABILITY BIAS
People tend to make decisions based on data that is easily available as opposed to finding the data that is really needed to make a good decision. This is, in effect, “looking for your lost keys under the lamppost” because that is where the light is best.

DIFFICULTIES JUDGING RISK
Most people have a tough time reasoning with probabilities and assessing risks. In October 2000, a train crashed in Hatfeild, England, tragically killing four people and injuring 34. In response, the British government proposed to invest an additional $3 billion in rail safety. However, as The Economist newspaper pointed out, the real probability of dying in a train crash is quite low compared with other forms of transportation, and this expenditure would mean spending 150 times as much per expected life saved traveling by railway versus what is spent on road safety.

SUPERSTITIOUS REASONING
We tend to only look for the most proximate causes of things and often confuse random chance with cause and effect. Examples range from sports stars wearing their “lucky socks” to governments trying to reduce unemployment by simply making it more difficult to fire people.

MENTAL ACCOUNTING
Traditional Economics treats all money the same. However, people tend to put money into different mental compartments. For example, many people make a monthly contribution to a retirement plan even if they have outstanding credit card balances. This is not economically rational, because the return on investment in the retirement account (Even after tax savings) will likely be less than the credit-card interest. Nevertheless, people often view their retirement contribution as sacred and wall it off from current spending. This mental compartmentalizing of different types of expenditures is deeply rooted—an anthropologist has even found this kind of behavior among Luo tribespeople in Africa.

(via)

A Masterclass in Screenwriting in one letter from David Mamet

CBS’s drama The Unit, about the lives of the highly trained members of a top-secret military division, was canceled in 2009, but a memo to its writing staff from its executive producer David Mamet has just surfaced online. The best part: it's in ALL CAPS. Via

TO THE WRITERS OF THE UNIT

GREETINGS.

AS WE LEARN HOW TO WRITE THIS SHOW, A RECURRING PROBLEM BECOMES CLEAR.

THE PROBLEM IS THIS: TO DIFFERENTIATE BETWEEN DRAMA AND NON-DRAMA. LET ME BREAK-IT-DOWN-NOW.

EVERYONE IN CREATION IS SCREAMING AT US TO MAKE THE SHOW CLEAR. WE ARE TASKED WITH, IT SEEMS, CRAMMING A SHITLOAD OFINFORMATION INTO A LITTLE BIT OF TIME.

OUR FRIENDS. THE PENGUINS, THINK THAT WE, THEREFORE, ARE EMPLOYED TO COMMUNICATE INFORMATION — AND, SO, AT TIMES, ITSEEMS TO US.

BUT NOTE:THE AUDIENCE WILL NOT TUNE IN TO WATCH INFORMATION. YOU WOULDN’T, I WOULDN’T. NO ONE WOULD OR WILL. THE AUDIENCE WILL ONLY TUNE IN AND STAY TUNED TO WATCH DRAMA.

QUESTION:WHAT IS DRAMA? DRAMA, AGAIN, IS THE QUEST OF THE HEROTO OVERCOME THOSE THINGS WHICH PREVENT HIM FROM ACHIEVING ASPECIFIC, ACUTE GOAL.

SO: WE, THE WRITERS, MUST ASK OURSELVES OF EVERY SCENE THESE THREE QUESTIONS.

1) WHO WANTS WHAT?
2) WHAT HAPPENS IF HER DON’T GET IT?
3) WHY NOW?

THE ANSWERS TO THESE QUESTIONS ARE LITMUS PAPER. APPLY THEM, AND THEIR ANSWER WILL TELL YOU IF THE SCENE IS DRAMATIC OR NOT.

IF THE SCENE IS NOT DRAMATICALLY WRITTEN, IT WILL NOT BEDRAMATICALLY ACTED.

THERE IS NO MAGIC FAIRY DUST WHICH WILL MAKE A BORING, USELESS, REDUNDANT, OR MERELY INFORMATIVE SCENE AFTER IT LEAVES YOUR TYPEWRITER. YOU THE WRITERS, ARE IN CHARGE OF MAKING SUREEVERY SCENE IS DRAMATIC.

THIS MEANS ALL THE “LITTLE” EXPOSITIONAL SCENES OF TWO PEOPLE TALKING ABOUT A THIRD. THIS BUSHWAH (AND WE ALL TEND TO WRITEIT ON THE FIRST DRAFT) IS LESS THAN USELESS, SHOULD IT FINALLY, GOD FORBID, GET FILMED.

IF THE SCENE BORES YOU WHEN YOU READ IT, REST ASSURED IT WILLBORE THE ACTORS, AND WILL, THEN, BORE THE AUDIENCE, AND WE’RE ALL GOING TO BE BACK IN THE BREADLINE.

SOMEONE HAS TO MAKE THE SCENE DRAMATIC. IT IS NOT THE ACTORS JOB (THE ACTORS JOB IS TO BE TRUTHFUL). IT IS NOT THE DIRECTORS JOB. HIS OR HER JOB IS TO FILM IT STRAIGHTFORWARDLY AND REMIND THE ACTORS TO TALK FAST. IT IS YOUR JOB.

EVERY SCENE MUST BE DRAMATIC. THAT MEANS: THE MAIN CHARACTER MUST HAVE A SIMPLE, STRAIGHTFORWARD, PRESSING NEED WHICH IMPELS HIM OR HER TO SHOW UP IN THE SCENE.

THIS NEED IS WHY THEY CAME. IT IS WHAT THE SCENE IS ABOUT. THEIR ATTEMPT TO GET THIS NEED MET WILL LEAD, AT THE END OF THE SCENE,TO FAILURE - THIS IS HOW THE SCENE IS OVER. IT, THIS FAILURE, WILL, THEN, OF NECESSITY, PROPEL US INTO THE NEXTSCENE.

ALL THESE ATTEMPTS, TAKEN TOGETHER, WILL, OVER THE COURSE OFTHE EPISODE, CONSTITUTE THE PLOT.

ANY SCENE, THUS, WHICH DOES NOT BOTH ADVANCE THE PLOT, AND STANDALONE (THAT IS, DRAMATICALLY, BY ITSELF, ON ITS OWN MERITS) IS EITHER SUPERFLUOUS, OR INCORRECTLY WRITTEN.

YES BUT YES BUT YES BUT, YOU SAY: WHAT ABOUT THE NECESSITY OFWRITING IN ALL THAT “INFORMATION?”

AND I RESPOND “FIGURE IT OUT” ANY DICKHEAD WITH A BLUESUIT CAN BE (AND IS) TAUGHT TO SAY “MAKE IT CLEARER”, AND “I WANT TOKNOW MORE ABOUT HIM”.

WHEN YOU’VE MADE IT SO CLEAR THAT EVEN THIS BLUESUITED PENGUIN IS HAPPY, BOTH YOU AND HE OR SHE WILL BE OUT OF A JOB.

THE JOB OF THE DRAMATIST IS TO MAKE THE AUDIENCE WONDER WHAT HAPPENS NEXT. NOT TO EXPLAIN TO THEM WHAT JUST HAPPENED, OR TO*SUGGEST* TO THEM WHAT HAPPENS NEXT.

ANY DICKHEAD, AS ABOVE, CAN WRITE, “BUT, JIM, IF WE DON’T ASSASSINATE THE PRIME MINISTER IN THE NEXT SCENE, ALL EUROPE WILL BE ENGULFED IN FLAME”

WE ARE NOT GETTING PAID TO REALIZE THAT THE AUDIENCE NEEDS THIS INFORMATION TO UNDERSTAND THE NEXT SCENE, BUT TO FIGURE OUT HOW TO WRITE THE SCENE BEFORE US SUCH THAT THE AUDIENCE WILL BE INTERESTED IN WHAT HAPPENS NEXT.

YES BUT, YES BUT YES BUT YOU REITERATE.

AND I RESPOND FIGURE IT OUT.

HOW DOES ONE STRIKE THE BALANCE BETWEEN WITHHOLDING AND VOUCHSAFING INFORMATION? THAT IS THE ESSENTIAL TASK OF THE DRAMATIST. AND THE ABILITY TO DO THAT IS WHAT SEPARATES YOU FROM THE LESSER SPECIES IN THEIR BLUE SUITS.

FIGURE IT OUT.

START, EVERY TIME, WITH THIS INVIOLABLE RULE: THE SCENE MUSTBE DRAMATIC. it must start because the hero HAS A PROBLEM, AND ITMUST CULMINATE WITH THE HERO FINDING HIM OR HERSELF EITHER THWARTED OR EDUCATED THAT ANOTHER WAY EXISTS.

LOOK AT YOUR LOG LINES. ANY LOGLINE READING “BOB AND SUE DISCUSS…” IS NOT DESCRIBING A DRAMATIC SCENE.

PLEASE NOTE THAT OUR OUTLINES ARE, GENERALLY, SPECTACULAR. THE DRAMA FLOWS OUT BETWEEN THE OUTLINE AND THE FIRST DRAFT.

THINK LIKE A FILMMAKER RATHER THAN A FUNCTIONARY, BECAUSE, INTRUTH, YOU ARE MAKING THE FILM. WHAT YOU WRITE, THEY WILL SHOOT.

HERE ARE THE DANGER SIGNALS. ANY TIME TWO CHARACTERS ARE TALKING ABOUT A THIRD, THE SCENE IS A CROCK OF SHIT.

ANY TIME ANY CHARACTER IS SAYING TO ANOTHER “AS YOU KNOW”,THAT IS, TELLING ANOTHER CHARACTER WHAT YOU, THE WRITER, NEED THE AUDIENCE TO KNOW, THE SCENE IS A CROCK OF SHIT.

DO NOT WRITE A CROCK OF SHIT. WRITE A RIPPING THREE, FOUR, SEVEN MINUTE SCENE WHICH MOVES THE STORY ALONG, AND YOU CAN, VERY SOON, BUY A HOUSE IN BEL AIR AND HIRE SOMEONE TO LIVE THERE FOR YOU.

REMEMBER YOU ARE WRITING FOR A VISUAL MEDIUM. MOSTTELEVISION WRITING, OURS INCLUDED, SOUNDS LIKE RADIO. THECAMERA CAN DO THE EXPLAINING FOR YOU. LET IT. WHAT ARE THE CHARACTERS DOING -*LITERALLY*. WHAT ARE THEY HANDLING, WHAT ARE THEY READING. WHAT ARE THEY WATCHING ON TELEVISION, WHAT ARE THEY SEEING.

IF YOU PRETEND THE CHARACTERS CANT SPEAK, AND WRITE A SILENT MOVIE, YOU WILL BE WRITING GREAT DRAMA.

IF YOU DEPRIVE YOURSELF OF THE CRUTCH OF NARRATION, EXPOSITION,INDEED, OF SPEECH. YOU WILL BE FORGED TO WORK IN ANEW MEDIUM - TELLING THE STORY IN PICTURES (ALSO KNOWN ASSCREENWRITING)

THIS IS A NEW SKILL. NO ONE DOES IT NATURALLY. YOU CAN TRAIN YOURSELVES TO DO IT, BUT YOU NEED TO START.

I CLOSE WITH THE ONE THOUGHT: LOOK AT THE SCENE AND ASK YOURSELF “IS IT DRAMATIC? IS IT ESSENTIAL? DOES IT ADVANCE THE PLOT?

ANSWER TRUTHFULLY.

IF THE ANSWER IS “NO” WRITE IT AGAIN OR THROW IT OUT. IF YOU’VE GOT ANY QUESTIONS, CALL ME UP.