Data too big? Try a walk in the dark

I live on the Isle of Wight so there’re are plenty of places to walk and think. One of my favourites takes me through a pleasant wooded area, through which Lukely Brook runs. I’ve yet to see a red squirrel here – the Island is one of the few places in the UK where they live in good numbers – but there’s lots of other wildlife to observe.

Recently, I decided to do this route in the later evening. Without artificial lighting and with no moon in the sky, the wooded section was very dark. I was dependent on a rather feeble torch, and my memory of the path.

What struck me most was that in all my previous walks I’d not realised that the path roughly follows three sides of a rectangle. In the daylight, I could more than clearly see where I was going and – distracted by everything else to look at – I largely navigated unconsciously. But with little to see at night I had to pick my way carefully.

I could hardly see the trees, let alone the wood. In a sense, with far less data to guide me I had gained an insight into the bigger picture.

As a metaphor for working with data, I think this can be useful. In this Big Data world we can have so many numbers and stats available that they can obscure or distract us from what we’re trying to find out.

So, if you’re faced with a big dataset and not sure where to start why not try ‘a walk in the dark’? Pare it down so broader characteristics can be observed. Even a rough sketch can be very helpful – not just for what it shows, but for what it forces you to focus on in the drawing process: the highs and lows, the general spread, and so on.

Leave a comment

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s