Retrieving data day queries

Perhaps the most famous data retrieval case in the history of science comes from sixteenth century orbital mechanics. Copernicus had laid the foundations for a viable heliocentric system; Kepler stood ready to finalise it. Between the two, both problem and solution, lay the mysteries of Mars: "the wanderer planet". The data which Kepler needed already existed, in a database of naked eye observations painstakingly constructed over two decades by Danish philosopher Tycho Brahe.

The problem was twofold. Brahe had nailed his colours to a mixed system at odds with that of Copernicus; and his data were his claim to posterity. He employed Kepler as an assistant, but jealously guarded access to the full observational data set.

Kepler did, eventually, gain access to the data. It wasn’t easy, nor always amicable (though allegations that he murdered Brahe to achieve it have been discounted), but it was done. He still had to learn how to retrieve it productively, but six years of mining and analysis finally bridged the gap to produce a final, successful, validated model.

Things have changed almost unrecognisably over the four or five centuries since Copernicus, Kepler and Brahe, but some features recognisably remain amid the new. Investment in research is balanced against the advantages of shared access. Boundaries, proprietary or otherwise, remain between researchers and data repositories. Murder and less extreme espionage methods may be rare (though not unheard of) as means of gaining access to data stores, but Kepler would no doubt recognise in essence the processes of negotiation and persuasion which allow those boundaries to be permeated.

The biggest early twenty first century data retrieval issue, however, is a different one. Acquisition in large quantity is becoming ever easier. Storage is, in relative terms, becoming cheaper. The headache often becomes how to ensure that one retrieves the right data for particular purposes from the ever ballooning volumes which are thus becoming available.

And then there is the problem of storage format obsolescence. Unlikely as it may seem, digital information which is by definition recent and (you might think) ought to be more easily accessible, and more carefully curated, can sometimes be harder to reach than older analogue stores. [More...]

The joy of text

It's a mundane truism, not normally worth mentioning, that words and phrases as signification units in natural language have only the fuzziest of relations to that which they signify. It is, nevertheless, a live issue for the many researchers attempting to computerise data analytic activity using text as raw material. It's also a truism of which I have been reminded afresh as I discussed the topic with practitioners and consumers of textual analysis, no two of whom used the term in exactly the way.

Strictly speaking, textual analysis describes a social sciences methodology for examining and categorising communication content. In practice, though, it is widely used to cover a range of activities in which unstructured or partially structured textual material is submitted to rigorous analytic treatment. What they all have in common is a desire to wrestle the petabytes of potentially valuable information locked up in an ever inflating text reservoir (blogs, books, chat rooms, clinical notes, departmental minutes, emails, field journals, lab notebooks, patents, reports, specification sheets, web sites and a million other sources) into a form which is susceptible to useful, objective data analytic treatment. Temis, of whom more later, have on their website a headline which sums it up neatly: "Big data issue #1: a lot of content and no insights". Text mining, the consequent knowledge bases, and analysis of the results have become a major component of biomedical and pharmaceutical research.

For our purposes here, I have taken it to mean analysis whose purpose is to extract scientific value from texts, to examine those texts scientifically, or some combination of the two. [More]

EndNote X6

While there are several developments in X6, the big news has to be that EndNote has completed its development of a viable cloud integration. There are a number of benefits to this, but the two big ones are geographic access and platform independence. Like many users nowadays, I work from wherever I happen to be, using whatever machine is most appropriate; I like to keep and manage my core bibliographic databases on a single master computer at my base, but want to access and update those in use from anywhere in the world on any device – including a smartphone as I use spare time in a queue.

The online EndNote Web tool has been available, and becoming progressively stronger, for some time, but until now the link between it and the desktop was less than transparent. That problem has now been addressed. From within EndNote X6 on the desktop you set preferences for automatic synchronisation between local and web versions of your bibliographic database, then forget it. Every change you make on your desktop is reflected in your web image and vice versa. Storage capacity limit is now 50,000 records and five gigabytes, which ought to satisfy the needs of most users. [more]

Ode to OCR

Preparing an article on text analysis for the upcoming issue of Scientific Computing World (watch this space) I've been revisiting the various ways of getting physical text into digital form.

Optical Character Recognition (OCR) is the workhorse of text transcription and, while we all grumble about its very real real short comings shortcomings, it really does a remarkably good job, most of the time, of rendering graphic images of printed fonts into digitised text for analysis.

Even at the lowliest manual level, and with all its admitted faults, OCR is a useful tool. A colleague and I recently had to add a six hundred and fifty page eighteenth century text to a digitised textbase for analysis. The rare and valuable paper original which we located was in a library, and could not be removed. Filing a request for the digitisation to be carried out would take weeks. With the consent of the library we used a smartphone, a ten year old copy of ABBYY FineReader 5 (now in release 11, and correspondingly more developed, as part of a software range for different text tasks) and a netbook. Even allowing for manual error correction, we had our validated data within three hours and the library added a copy to its own digital records. A similarly sized text already available as graphic only PDF was transferred more quickly still.

Disrupted patterns

Well … having read all zillion pages of Google's new privacy policy, since it was published, but done nothing concrete about it, today I have started to gradually respond.

Not necessarily responding to Google, as such; responding to the totality of movements in development of the social web.

To be honest, Google's new privacy policy doesn't, in itself and at the moment, worry me overmuch. What does concern me is the idea that I might passively allow myself to be borne along by the development currents without exercising any decision making or volition of my own. Eighteen years ago (good grief, Charlie Brown … is it really eighteen years???) I wrote:

“The greatest improvement in the telephone system, over the last twenty years, is not digital technology but the little switch that lets me turn it off … … … Do you have your finger on your own selected off-switch? Do you know where, and what, your off-switch is? Have you planned how to ensure that it is never, ever, taken from within your reach? Do it now.”

Well, I have drifted into a relaxation of that vigilance and Google has prodded me awake.

Don't get me wrong. I don't think that Google are about to misuse my information; not now, anyway. FaceBook, yes; but Google, so far at least, is not immediately about to do anything which I don't regard as fair use under the general contract whereby I get free service in return for advertising around my edges. But that's not the point. The point is that I should not rely on Google's (or anyone else's) benevolence for maintenance of my freedom and privacy; I should take responsibility for it myself.

So, as of today, I have started to fragment those parts of my online footprint which I have in laziness allowed to coalesce.

Google provide me with the platform for this blog. That's an arrangement which works to their and my mutual benefit, and I can shift if it ever seems desirable. But have just altered the search box in FireFox to choose AltaVista as its default engine. In other browsers, I've set different defaults. In Chrome (where I am not routinely signed into my Google account) Google remains the default engine. So, my searches can no longer be tied up with other information which I give to other parts of Google.

Some of my on line material already resides with, Box or DropBox, Foliospaces or SafeSync, Wikispaces or other repositories, according to which is most appropriate for a given purpose. That satisfactorily breaks up into discrete chunks, with a disruptive camouflage pattern of natural firebreaks between them, the body of stuff which advertisers and others want (quite legitimately) to link up and over which I (equally legitimately) wish to retain control. I shall start to look, without any urgent hurry but with deliberation, for other ways to spread eggs into multiple baskets. Google Docs and Microsoft's SkyDrive are already play very minimal parts this patchwork, but I shall henceforth keep a particular eye on ensuing that they remain that way.

Be careful what you wish for

When I was very small, my father used to sing (at my own urgently repeated request) a little one stanza song when we were playing out in the open air. I render it here with an attempt to recapture the particular cadence of his delivery.

I'm a lit-tle prair-ie flow'r
Grow-ing wilder hour by hour.
No-one tries to cultivate me
So I'm as wild as wild can be!

This memory lies dormant for weeks, months, years at a time, then springs to front of stage for no obvious particular reason to dance in my conscious mind for a day or two before returning to the wings. Inconsequential though it may be, it embodies for me something very personally precious about my father, and his relationship with me. A conversation with my brother, a couple of days ago, somehow brought it out for a spin in the light and it is sparkling still at the edge of my day to day thoughts, son on the spur of the moment I just did a search for it.

There are several video clips and MP3 files in the Google listing. I stuck to text hits, though, and the first I found was a partial reference within a longer anecdote. There was a small discrepancy (shown here in red):

I'm a little prairie flower!
Growing wilder by the hour!

Then there is this version, from Mudcat, which provides a whole song of which mine is the first stanza. Again, there are minor differences:

I'm a little prairie flower,
Growing wilder every hour;
Nobody cares to cultivate me,
I'm as wild as wild can be.

The International Lions also give a whole song but shorter than and only partially resembling as Mudcat's. Subtitling it I'm a little lion cub, add a line repetition and a wordless twirl to the end of each stanza:

I'm a little prairie flower,
Growing wilder every hour;
Nobody cares to cultivate me,
I'm as wild as wild can be.
I'm as wild as wild can be,
Tu-ra-lu-ra, Tu-ra-le.

Wikipedia mentions the song only under its entry for Lesley Sarony, without attributing it to him, although there are a number of web pages which do make this attribution.

Courtesy of Google Books I find that The Rotarian, vol.13 #3 (September 1918) , gives exactly the same version as Mudcat but the previous month's issue (vol.13 #2, August 1918) adds the last line repeat (though not the wordless twirl) of the Lion's version. Exactly forty years later (vol.93 #3, September 1958), however, a couple of years after my father was singing it to me, has it as:

I'm a little prairie flower,
Growing wilder by the hour;
No one cares to cultivate me,
I'm as wild as I can be.

At this point, I realised that precious memory, whether accurate or flawed, was beginning to blur at the edges. So, I stopped looking. There can be such a thing as too much information, and there can be occasional limits to John's breezy (KJV 8:32) assertion that “the truth shall make you free”.

Virtual books at the British Library

After recently giving a talk on the use of text within visual art, I followed up some suggestions (thank you, Maureen) and questions from the very lively and participatory audience. One place to which this process took me was the Virtual Books index at the British library.

There is an excellent online Lindisfarne Gospels, a Qur'an and a Hebrew Bible (these links take you to static text/image pages; there are also animated page-turning versions). But I recommend looking through the whole list, in all its representative variety from Alice through botanical illustration to Leonardo, maps and Mozart.

My dear, I was literally metaphorical...

We all have our private irritations and exasperations, which may not be reasonable but are nonetheless real. Many of mine lie in the use of words, and telling which are reasonable and which are not can be a grey area.

Language is, I passionately believe, an evolving thing and we cannot tie it down with rules. Words change their meanings ... get used to it. On the other hand, its richness and complexity rely on the existence of rules ... the rules can (indeed, must) change with time, they can be broken with magnificent effect, but like (to borrow Robert Frost's analogy) a tennis net, we do need them. If they break down entirely, or even change too fast, the glorious moderated anarchy which is language falls apart and becomes a puddle on the floor.

My reason for wittering on like this is, of course, a particular exasperation which has just happened by ... though in this case it's amused me rather than irritating. Over a long period, now, I've noticed (and generally restrained my irritation over) misuse of the word "literally", for example “I was so embarrassed, I literally died!” Today I've heard the converse: “I was so frightened, I was almost metaphorically looking over my shoulder!”

---

• "Writing free verse is like playing tennis with the net down." Robert Frost, in an address to students. 1935, Milton, Massachusetts: Milton Academy.
• "I'd as soon write free verse as play tennis with the net down." In an interview with Edward C Lathem, 1966. (Manuscript, part of the boxed Papers of Edward C. Lathem, 1913 - 2009, Hanover, New Hampshire, USA: Rauner Special Collections Library, Dartmouth College)

Picking over half a trillion words

Following JSBlog's enthusiasm, yesterday, (“Google just blew my bibliographic socks off”) for Google's new Ngram viewer, I've been busily catching up.

First stop was the viewer itself. Then a start on downloading the raw data sets which lie behind it, for more detailed analysis than the online viewer can deliver. Finally, while the data downloaded in the background (almost two gigabytes of it just for single words in English, even in ZIP form ... nearer to ten when expanded), reading the associated Science article by Michel et al.

It's going to be a good while before anything significant comes of the downloads, but I've done a couple of test drives. They can be intuitively checked with a quick visit to the viewer.

First experiment, resulting from a recent off the cuff discussion amongst a group of students: correlating uses of the words "twat", "twit" and "twerp". It's interesting to find positive correlation between the first and last from 1935 to 1980, but negative between them and "twerp" over the same period – which then reverses so that all three positively correlate over the past thirty years.

Second: the tendency to concatenate "bigrams" into single words. This train of thought was started by Google's example comparison of "child care" with "nursery school" and "kindergarten" ... I tried it out, and then added "childcare" to see if it made a difference. As examples to cut a long story short, "child care" declines markedly as "childcare" slightly increases (a negative correlation) from 1996 to 2008; "brood mare" and "broodmare" show a similar negative correlation from 1960 to 2000 but then "brood mare" recovers and the correlation becomes positive through to the present.

Those are, of course, trivial investigations and show nothing ... I mention them only to show the sort of five finger exercises that I've been playing with since yesterday. Much more interesting is some of the investigation mentioned by the authors of the Science article.

For example:

“Suppression – of a person, or an idea – leaves quantifiable fingerprints... ... ... Such examples are found in many countries, including Russia (e.g. Trotsky), China (Tiananmen Square) and the US (the Hollywood Ten, blacklisted in 1947)...

We probed the impact of censorship on a person’s cultural influence in Nazi Germany. Led by such figures as the librarian Wolfgang Hermann, the Nazis created lists of authors and artists whose “undesirable”, “degenerate” work was banned from libraries and museums and publicly burned... We plotted median usage in German for five such lists ... ... ... The five suppressed groups exhibited a decline. This decline was modest for writers of history (9%) and literature (27%), but pronounced in politics (60%), philosophy (76%), and art (56%). The only group whose signal increased during the Third Reich was the Nazi party members [a 500% increase...].

Given such strong signals, we tested whether one could identify victims of Nazi repression de novo. We computed a “suppression index” s for each person by dividing their frequency from 1933 – 1945 by the mean frequency in 1925-1933 and in 1955-1965... In English, the distribution of suppression indices is tightly centered around unity. Fewer than 1% of individuals lie at the extremes... In German, the distribution in much wider, and skewed leftward: suppression in Nazi Germany was not the exception, but the rule... At the far left, 9.8% of individuals showed strong suppression... This population is highly enriched for documented victims of repression, such as Pablo Picasso..., the Bauhaus architect Walter Gropius, and Hermann Maas... ... ... At the other extreme, 1.5% of the population exhibited a dramatic rise... This subpopulation is highly enriched for Nazis and Nazi-supporters, who benefited immensely from government propaganda...

These results provide a strategy for rapidly identifying likely victims of censorship from a large pool of possibilities, and highlights how culturomic methods might complement existing historical approaches.”

---

• Jean-Baptiste Michel, et al., "Quantitative Analysis of Culture Using Millions of Digitized Books" in Science, 2010. DOI 10.1126/science.1199644
  

Analytically speaking

My first Damascene vision of what a wonderful tool data analysis can be was not in the physical sciences. In a holiday homework assignment, when I was fourteen, a maths teacher asked us to explore, using what he had taught us that term, the suggestion that Shakespeare's Hamlet might have been written by Marlowe. Two weeks of miscounted words, syllables, and parts of speech later, I understood the sheer intellectual thrill of using statistical analysis to explore the unknown.

Linguistics is a word used in many ways by different constituencies, but they have in common the scientific study of (usually, but not always, natural) language. This plurality of meaning makes it representative of language in general. Like all the words which make up natural languages and other means of human to human or communication (including, for example, financial currencies) it is, to the exasperation intrigue of science, an "arbitrary signifier". Its meaning lies entirely in the intersection set of associations between those who transmit and receive it, and is defined by difference from what it is not rather that what it is.

The lure of the unknown is science's greatest romantic pull. It can be along the banks of the Amazon or the Congo, it can be on the inaccessible ocean floors or in other galaxies, it can be down in the subatomic or up on the macrocosmic. But it can equally well be in the vast and ever shifting jungles of arbitrary sign systems with which we attempt to communicate that we seek and find - and scientific computing methods are just as central there as in more physical arenas.

At the same time, these qualities of language have strong practical importance. Language is how we become fully human - whether or not it is an attribute unique to our species, as consensus suggests, it is certainly a powerful component in our dominance. It is how we become socialised and acculturated. It is how some of us make the long climb from newborn tabula rasa to mature professional scientist - encountering, along the way, what Evelyn Rodriguez-Alamo[1] called " the content and the vehicles of learning and scientific research for the 21st century". Analytic approaches to language underpin the effectiveness of learning. Viewed from another perspective, language is how organisations are structured; analysis of effectiveness depends upon linguistic assumptions. As well as being itself an inviting subject for scientific enquiry, then, understanding how language does and doesn't work is vital to both efficiency and outcomes for every stage and component of the context within which science happens.

Computerisation of linguistic data analysis ... [more...]

I am Google ... sort of (continued)

An interesting response to yesterday's "I am Google ... sort of" comes in a comment from AcerOne which has had me thinking further about the implications of copying a text for my own use. He compares my scanning of a printed text into digital form to [illegal] downloading from a file sharing site of music already purchased on physical media. I'm not sure that this is a true analogy in every way, although I do agree that the immediate (not extended) moral issue is the same.

I've just spent a while discussing this with a lawyer. Not professionally, I hasten to add: just chewing the fat with a friend who happens to be a lawyer (and not one with any specific experience of copyright law), as a way of trying out thoughts.

To some extent, what I say now in response to AcerOne is playing devil's advocate. Other parts I think I probably believe. I won't try to sort out the two at this point; I'm just having an exploratory dialogue with myself, using his words as a sounding board. In no way am I implying any judgment of AcerOne, for whom (and for whose moral sense) I have the greatest respect. What you are reading here is an internal monologue on my own outlook.

AcerOne: What difference does it make to anyone, the author included, if you also own a copy of the text in a digital format?

Well ... in practice, I think that depends upon circumstances. In the case of Appleton (the text I am currently digitising), perhaps none. If the text stays out of print, then my digitisation cannot cause any loss to Jay Appleton or his estate. Even if the text is reprinted, that remains true as long as I keep my digitised copy entirely for my own use. If, on the other hand, the text is later digitised and offered for sale in that form, I have arguably deprived the author or his estate of income from the digital copy which I might have purchased.

This becomes more acute, though not different in principle, if I digitise Spirin who is still very much in print but not available in electronic form. It becomes a very live issue if I digitise a book which is available as a purchasable electronic copy.

An analogy might be this. AcerOne is, amongst other things, a visual artist who currently has on show an exhibition of his paintings. He offers for sale both those paintings and prints of them. If I go to his show, photograph them, then make prints of the resulting photographs to hang on my wall, I very definitely feel that I have behaved shabbily. I should have bought his own prints – I have deprived him of the income which he would thus have made. If I take the same photographs and use them as illustrations in a lecture, on the other hand, I feel comfortable with that provided that I give details of source: I have not replaced his work, and may through publicising it contribute indirectly to his sales income. Whether it is legal or not is a different matter; I think it comes safely within "fair use", and my friendly lawyer (without wanting to be pinned down or quoted, of course) agrees.

AcerOne goes on...

“...i have on occasions used file-sharing websites to find music (both moral and copyright issues definitely breached). But [... ... ...] to download music i have already purchased and legitimately own; but on vinyl. [... ... ...] The thought of transferring it to my computer so that it can be played through my home system or on my iPod seems like a long and laborious task [...]So i have on occasions sourced and downloaded a digital version from file-sharing websites... Legally i have broken the law. But all i have done is chosen to take a simpler and far easier route to gaining the same outcome - a digital version of music i already own.”

There are several issues which occur to me here. The first is a technical one: the true analogy for my digitisation of a print text is, I think, not downloading a new copy^[1] but the transfer from vinyl to computer file. And that prompts me to consider the fact that I routinely buy a CD which I then do not play: I immediately "rip" it to digital form and play it from an MP3 player because most of my musical listening is done on the move. I have done the same with such vinyl or tape as I cannot replace with CD. So, I realise, my concern of digitisation of a book is somewhat hypocritical when I have done the same with in excess of two thousand albums. (Ripping is, of course, so endemic and so universally enabled by mainstream software providers as to be an unstoppable fait accompli . That's not, on the other hand, the same thing as being morally right. Nevertheless, I personally feel morally comfortable with both my and AcerOne's digital duplication of our music sources. There are other philosophically problematic issues around use of a file sharing site ... but I'll not go there as I've already bitten off more that I can chew with this topic.)

The second issue is also technical, but in an important legal sense. AcerOne does not, in fact, own the music on his vinyl; nor I the music on my CDs. We own only the physical media upon which it is recorded, and have a licence to play the music (only for our own enjoyment at that; playing it for other people is, strictly speaking, breach of the licence). Similarly, when I buy a copy of Appleton's book I do not buy the content: only the physical paper and board construct within which my single copy is encoded. Calling again on the analogy of AcerOne's paintings: if I purchase one of his prints, I do not thereby own the image itself and have no right to reproduce it – the image remains the property of AcerOne.

On the other hand ... the legal copyright breach seems to lie not in reproduction but in "publishing". Going back to the technical breach of license when I play my CD to others (previous para), the issue is that legally I thereby "publish" the work to others. Software licences work on much the same basis, but make allowance for back up; my word processor, for instance, comes with a specific proviso that it can be copied for security back up purposes, and can be installed on more than one computer provided that it is only ever in use on one of them at the same time. By analogy, it would seem that photocopying^[2] or digitising a book stays within the fence so long as nobody else then sees the copy. But then again ... flipping open the nearest book to hand at this moment, I find inside the following unambiguous words: “no part of this publication may be reproduced or transmitted by any means, electronic, mechanical, photocopying or otherwise, without the prior permission of the publisher" ... which seems pretty unambiguous.

I could go on ... but enough, already. My feeling remains that I am staying within "fair use" so far, but it's not a subject which I can afford to regard as cut and dried. For my own peace of mind it needs revisiting at intervals, and in particular every time I consider digitising a new text.

---

1. In my impecunious teens and student years, I regularly did deals with friends whereby one of us bought a vinyl album and the others made a tape cassette copy. Later, when my wife and I divorced amicably in our thirties, we solved the music problem by copying everything to tape cassette so that we each kept the whole collection. Both of these, it seems to me, are the true equivalents of file sharing. I wouldn't do it now, but it would be hypocritical of me to righteously condemn young people now who do it in the new ways which new technology makes available to them.

2. I do, in fact, have some photocopied texts. They were made in the days before digitisation. They were not made by me, but by the owners of print copies, as helpful gestures when I couldn't obtain a physical print copy of something to which I needed long term access. Who actually made them, however, is irrelevant here; my acceptance and retention of them is another blurring hypocrisy in the debate which I am having with myself over selfdigitisation.

---

I am Google ... sort of

I am preparing a series of lectures on landscape imagery for next year, and find myself making frequent return to several books. Appleton's Experience of Landscape, for instance. Nash's Wilderness and the American mind. Spirin's The language of landscape.

As I go, I extract fragments for reference and/or quotation. Time was, I'd have hand written these on index cards; now I stick the book into a scanner and let OCR take the strain.

I'm also, of course, referring at the same time to numerous electronic sources – which is so much more convenient since, quite apart from instant searching, I just have to highlight, copy and paste ... or even "save page as" ... or, better still, just store a link to the page. I have, in my notes for this lecture, for example, links to recent posts by Ray Girvan (Old Park) and Julie Heyward (Instrumentality).

If only the printed texts were as instantly accessible in their entirety as the electronic sources.

Sometimes, of course, a printed text is also available in digital form. It's well worth my well, for something often used, to buy both forms: paper as definitive version, digital for flexible reference and quoting. But that's not possible for the above three examples.

As I scanned a paragraph today, it dawned on me that I am gradually and unintentionally compiling a fragmentary but increasingly complete digital copy of a book in this way.

How do I feel about this?

I take copyright very seriously. I would not knowingly do anything which in might in any perceivable way tend to undermine the author's income from the text. But it seems to me that as long as I keep the material solely for my on use, and no purchasable electronic version exists, I am not going beyond my own moral view of "fair use".

There is one difference between Appleton on one hand, Nash and Spirin on the other: Appleton is out of print. That is probably not a significant difference for any practical purpose ... but it feels different as a starting point. So, as of today, I have started deliberately scanning the whole chapter containing a section I need from Appleton, rather than just the section itself. When I have the whole lot, I'll combine it into one PDF file and bingo ... I will be my own miniature Google Books.

When I've done that, I'll consider what to do about the others.

---

• Jay Appleton, The Experience of Landscape. 1986, Hull: Hull University Press. 0859584615. [Originally 1975, London: Wiley. 0471032565.] [Most recent edition 1996, Chichester: Wiley. 0471962333 (hbk) or 047196235X (pbk).] [Now out of print.]
• Roderick Nash, Wilderness and the American mind. 1982, New Haven, Conn: Yale University Press. 0300029101 (pbk.). [originally 1967]
• Anne Whiston Spirin, The language of landscape. 1998, New Haven, Conn: Yale University Press. 0300077459, 9780300082944 (pbk).

Wikileaks

I don't have anything original to say on the subject, and certainly nothing worth stopping to listen to, but it's necessary (in however small and insignificant way) to stand and be counted ... so ... I applaud what Wikileaks has done today.

There is never any such thing as an unmitigated good but, despite all freely acknowledged reservations and messy grey moral fogs, I do believe that releasing of truth is, in this case at least, the best available route through ambient evils.

Wikileaks

ORG: Recording Casualties of Armed Conflict

Iraq Body Count

...and, while I'm about it...

SIPRI

CCR

Burrowing into the bran tub

Intelligence organs of a fairly small but technologically proficient country managed to clone the email and document bases of a fairly large but unsuspecting company with an active and diverse research operation. Not the research results bases themselves; just the email and document archives. The immediate benefits of the theft were obvious, mundane, and irrelevant here. More interesting from a scientific computing viewpoint (if no more morally or legally defensible) were the spinoffs from applying statistical information mining techniques to the combined database contents.

As a direct result of the operation, other companies, seen as friendly to the small country's interests, received repackaged and anonymised material suggesting productive new lines of enquiry. Patents resulting from these have already been filed; others are in the pipeline. The large company which was the target of the operation, significantly, remains unaware of the opportunities which exist within components of its own activities which have never been brought together.

The story illustrates a truth: that much knowledge is locked away in information stores assembled for one set of reasons and never reexamined in other ways.

Less melodramatically, and less dubiously, information openly published on the internet forms a huge field within which to prospect potential information seams - "The low user entry barrier of the Web has resulted in massive amounts of unstructured and weakly structured data referring to objects, concepts, user interests and communities", to quote the Digital Enterprise Research Institute at Galway. As SAS's David Smith points out, tweets and blog entries can contain pointers to early identification of potentially vital phenomena. This is an aspect of what is known in the industry as pharmacovigilance, which "can be defined as a set of practices aiming at the detection, understanding and assessment of risks related to the use of drugs in a population, and the prevention of consequential adverse effects [or] in a narrower sense ... postmarket surveillance"^[1]. [More...]

---

1. Langlitz, N., "Pharmacovigilance and Post-Black Market Surveillance" in Social Studies of Science, 2009. 39(3): p. 395.

EndNote X4

With the recently released 14th version, EndNote continues its highly successful trick of evolving as an information manager without compromising its core rôle as a bibliographic reference handler. There is new material in several areas, major additions have been made from a functional viewpoint, and a signpost erected to future expansion potential.

[More]

Re-gruntled

A couple of weeks ago, in an embarrassing example of blog as self indulgent whinge, I complained about problems with Google Calendar. For the record, they have been ironed out and were temporary propagation artefacts; I am now completely happy with the resulting system, hosted by Google and managed primarily from Mozilla Lightning. In fact, I shall be shifting fully onto it from August 1st, a month earlier than planned

I also said, at that time, that:

“Adding an item from the E63 is slightly less convenient than on the Palm, but to a degree with which I can live. Modifying an item from a phone is impossible, but I've found ways to deal with that too. (There is supposed to be a way of synchronising the E63's onboard calendar application with the Google calendar, but I've not sussed that yet.)”

That para, too, is now out of date. I have found several third party apps which integrate Google Calendar with the onboard calendar of Symbian phones (the Nokia E63 runs Symbian), of which I narrowed down to a personal choice of two with confusingly similar names: Goosync and Googasync.

The two offer slightly different approaches, marginally different facilities, and considerably different pricing structures, but both do an excellent job and I won't say which I plumped for in the end. (Just don't do as I did, and try them both at the same time ... at one point I was weeding out six copies of every appointment at both handheld and web ends! Try one at a time, them make up your mind and install your choice.) The result, whichever you choose, is a near perfect unified system running simultaneously in web browser, hand held and (if you go for the Lightning option) email client.

A shortage of gruntles

Two and a half weeks, or so, since my last post, and what do I return with? A disgruntlement. (Is disgruntlement a word? It ought to be ... and it fits well in the Growlery, so let it stand on both counts!)

I am not what the marketing people call an "early adopter". I never change my technology until what I already have can't do something I need to do. I value reliability, familiarity, dependability, transparent usability, over cutting edge capability. So, I have continued to use a PalmOS hand held computer of one sort or another long after they ceased to be current. My palm machine holds numerous vital databases managed by JFile (a program whose author saw the way the wind was blowing and moved on some years ago) and handled at the desktop by JFTrans (whose author has also shifted to a new platform). For PIM functions I have relied totally upon Agendus, which integrates and expands upon the basic PalmOS elements miraculously without messing in any way with their structure. I make frequent use of several calculator or other math programs including PowerOne graphic scientific calculator from Infinity Softworks (who appear to no longer be with us). There are numerous other utilities on there as well, without which life would be either very much harder or much less intuitive.

On the other hand, I am not blind to need for progression so things have to change. “Life must change, if life's to grow”, as my favourite quote provider has it^*. I always make these changes in the early summer, when I have a period of relative slack in which to internalise them before they are tested in the fire of a new academic year. Thus it is that a year ago I added a Nokia E63 to my bag, for web and text based communications. I looked at the Blackberry clan, and [shudder] Pocket Windows, and there are several Apple iPhones knocking around here somewhere, and various others ... but none worked as well for me as the E63.

And now I am in the process of shifting my diary/calendar out of Agendus, thus decoupling it from address book and all the other functions which Agendus handles so well. Why? Because my current PalmOS device will not last forever. And because if I am giving up Agendus, I might as well try to future proof myself to some extent by moving onto the web where it can be accessed regardless of future shifts in proprietary handheld formats. And because it will more easily allow me to synchronise calendars for different external purposes with my own view of things. I don't, to be honest, like keeping PIM information off my own platforms ... so I've spent six month planning how to secure content behind a tokenised wall. Contacts information will never be shifted onto the web; though now that the calendar has gone, they will probably migrate to the E63.

So my comings and goings are now contained in Google Calendar. Or, to be more precise, are increasingly contained there and will be wholly so from August 31st ... for the time being, both handheld and web versions are running in parallel. Detailed maintenance is done (either from the desktop or via a web book when on the move) through Mozilla Thunderbird's calendar management extension, Lightning and a Google Provider addon which works well. Multiple calendars can be integrated within a single Lightning view. Alterations made at either the Thunderbird or Google end are reflected at the other within a second or two. The result can be viewed in any internet connected browser, so is available ion the E63 or any subsequent replacement. Adding an item from the E63 is slightly less convenient than on the Palm, but to a degree with which I can live. Modifying an item from a phone is impossible, but I've found ways to deal with that too. (There is supposed to be a way of synchronising the E63's onboard calendar application with the Google calendar, but I've not sussed that yet.) All in all, I'm happy with the new set up and prepared to drop the handheld version in six week's time.

So, with all that, and for free too, what is my disgruntlement? It has to do with daylight saving time adjustments. I've moaned about these before (and for people who know me in person it's worse, as they get their ears bent on the subject at least twice a year) but now I have a new reason to be exasperated.

A week ago, the online calendar segments which interpenetrate with those of my students went live. Immediately, I started to get woeful messages asking why I was forcing them to attend lectures at eight in the morning throughout the winter. Surprised, I went and looked ... sure enough, all times from roughly November to March were an hour earlier than they should be. Shuffle back to September 2010 or forward to April 2011 and they are correct again. As long as we are in Summer time, Google presents appointment slots in Summer time terms. Presumably, when we shift to Winter time, summer time slots will appear to be an hour later than they ought to be.

Mutter mutter.

I assumed, at first, that this was something to do with harmonising the setting of time zones on both my own platforms and the Google calendars. In particular, I suspected that the problem had its root in my insistence on keeping all my ICT equipment permanently on Zulu time all year round. I started by setting all the calendars to Zulu time, but that created new problems. So I reset them, gritted my teeth, and set my PC to daylight adjustment; but that didn't work either. I tried various combinations, but nothing seemed to work ... Google Calendar simply cannot see across a time adjustment without this parallax error, so to speak. I just have to live with it.

A particularly bizarre oddity is that Thunderbird/Lightning manages to get around this and show times correctly all year around ... but only in some views.

BUT ... just to annoy me further ... having spent time writing this whinge, I have just gone into both Google and Thunderbird calendars ... to find that ... blow me down ... the problem has disappeared. September and December 2010, January and May 2011, regardless of view, all display 9am lectures as 9am. At least, they do on my desktop ... and on the E63 ... but not on the web book at the time I typed those last three dots, though the display is not OK there as well.

What's going on? Is this a sporadic problem (disgruntling both in itself and in its unpredictability) or a permanent one (disgruntling but allowable for) or a web propagation issue (downgrade to irritating and livable with) or has it solved itself (disgruntling to have been disgruntled for no reason ... and a source of paranoia incase it unsolves itself again)? Only time will tell ... by August 31, I should know which it seems to be, and whether I should stick with the PalmOS calendar for another year.

---

• Melanie Safka, "Ears to the ground" on Please love me. 1973, New York: Buddah. BDS5132

Only (dis)connect

In comment on the article to which my "Only connect" post links, Dr C says:

“I have been wondering lately some things, reflected in what you say in your article, whether someone who has been carried along by the flood (e.g. texting, Facebook, iPhone, GPS location, implantable chip) can't voluntarily turn the other way. Do a sort of Thoreau maneuver and walk away from it all. I feel actually chained to my cell phone and I am expected to be there, one heartbeat away from an anxious parent. I don't really mind it but I can't imagine all of society barreling down the road with our every action, every movement, analyzed by someone in Washington, London or Paris (actually in Bangladesh).”

Removing my scientist's hat (UFO shaped, in a padded fluorescent orange synthetic satin, carrying the words "Kiss me quick, Igor") and replacing it with my vigilant citizen headgear (a frayed and sunbleached beanie, with floral hippy insert) ... this is something that concerns me, too.

The answer, I think, is that (as with almost any aspect of social organisation) one cannot easily walk away from the ICT revolution as a whole and still remain connected to the main of society as a whole ... but one can pick and choose amongst its manifestations to a considerable extent.

I have indicated before my lack of personal enthusiasm for sites of the FaceBook variety (and an alternative approach to the same requirements). On the other hand, here I am making use of another online strand – the blog – and I am an enthusiastic exponent of wikis for many particular purposes. The thing is, in my view, to decide what serves your needs and what makes you serve it ... embrace the first, shun the second, and be aware that the distinction will be different for each person who asks the question.

In his third sentence (“...cell phone ... one heartbeat away from an anxious parent”) Dr C says, in my opinion, more about himself than about the technology. A warm and compassionate person, with a strong professional conscience, he feels that a device which makes him more available to a concerned parent must be used to do just that. My own cell phone, by contrast, is a vital component of my life but spends most of its time "on silent", taking messages which I check regularly but to which I respond when convenient (hmmm ... it suddenly occurs to me that perhaps I have painted myself into a corner here ... perhaps I am, also by contrast with Dr C, a cold and unfeeling...)

About a decade and a half ago, commenting on exactly this, I used the metaphor of “a free fall parachutist, compelled to fall downward but riding tiny variations to a choice of the available landing points”. It still seems to me, after all this time, a reasonable description of what needs to be done.

Kindling a new inequality

A TTMF post sent me to a Santa Fe Reporter article which is interesting to a European reader for the unusually European (as opposed to USAmerican) cast of its analysis. That's not why I mention it, however. What made be stop and think was the following interesting and thought provoking para in relation to a textbook:

“Should she get it from Amazon or download the Kindle version? Bowles quickly rules out the Kindle because it makes the text impossible to share. Reading between the lines, Bowles’ choice reveals the hidden symbolism of each medium: If the paperback is Karl Marx, the Kindle is Ayn Rand.”

I've always made my book format selections on personal and/or practical bases. I generally prefer paperbacks to hardbacks because they are lighter, more compact, easier to use on the fly. I æsthetially enjoy reading paper books more than electronic ones, which is important to me. Electronic text offers mass portability, instant access, powerful searching, so has the edge for reference. Though I have a Kindle, I never use it; other electronic platforms work better for me.

I've not considered, before, the impact of electronic texts on inequality of access to literacy (I use the word literacy, rather than literature, deliberately). I shall, in future, do so. The advantages of electronic access remain and, in the case of (for example) Gutenberg Project texts, sharing is in many important ways enhanced; but the restrictive downside will from now on be a factor on the other side of the scales.

Pilate asked: What is truth?

Yesterday my brother sent me, without comment, a Discovery Channel News link referencing the speculatively constructed and partially animated version of Tiger Woods' automotive mishap. This morning, I found that Thinking through my fingers takes this same video as the starting point for a consideration of where our opportunities for accessing “true” news are heading. (Looking back, I see that TTMF referenced the same video a week earlier; but I didn't pick up on it.)

On the face of it, a very obviously constructed fiction (and one in which Woods, and his wife, and even the physics of real life movement, are scarcely recognisable) should be of little concern to anyone except the Woods family and their lawyers. It exists in the same intellectual space as Ice age 3: the dawn of the dinosaurs about which I grumbled recently. But that is disingenuous; the reality is that very few news consumers give much critical consideration to their sources, and very many make only blurred distinctions between fiction and nonfiction.

On the other hand, to start crying that the dividing line is being eroded between imaginative interpretation and objective reportage is equally disingenuous. I doubt that there has ever been such a distinction; certainly I cannot remember a time when it existed outside the platonic ideal in the minds of its champions (of which I, for the record, am one – if not a very active or heroic one). "Briefing against" is at east as old as China's "warring states period (476-221 BCE); eighteenth and nineteenth century press in Europe and the Americas offered political commentary skewed and invented to a degree unrivalled today; and Ray Girvan recently offered, over at JSBlog, an example of a 19th century "tabloid smear" against ordinary people. Truth has always been an elusive, provisional, partial thing, and finding even a worthwhile approximation to it has always been a process of determined exploration and detective work.

That last sentence is where TTMF starts: that "truth" (of even the blurred and provisional variety) can only be found (even by those actively seeking it) if somebody is somewhere telling it. Or, more accurately, if enough sources genuinely trying to tell it can be found and compared.

What concerns me even more is that as the volume of information sources multiplies, the fragmentarily "true" is ever harder to find. Whether or not disinformation is proliferating faster than information is a moot point about which we can all argue forever on the head of a pin, but is irrelevant. As the haystack grows exponentially, the pins will become harder to find even if they proportionally keep pace.

I don't suggest that anything be done about this, even if that were possible (which it is not): I would deplore central censorship of dis/information flow even more. But I do dispute the widespread idea that ever greater flow of information form every greater numbers of sources is synonymous with "more truth". The vital importance of critical thinking by consumers about sources and their content increases as a direct function of information volume.

I realise, belatedly, that this is little more than a rehash of my Whither democracy post, almost two years ago ... ah, well; “I grow old ... I grow old”* ... at least I stopped short of requoting Lem!

---

• T S Eliot, "The love song of J Alfred Prufrock", line 120, in Prufrock and other observations. 1917, London: The Egoist Ltd.