Jenett’s quick guide to evaluating information

I got nudged by a friend to do a ramble about information evaluation. It might have gotten a little away from me.

Basic principles:

  • We all have biases and things we know more about than others.
  • Some people are more up front about this than other people.
  • Ditto goals. We all have them, some people are more up front about them.
  • Be really suspicious of the people who claim they have the absolute truth and are telling you for your own good.

(They probably don’t and they probably aren’t. Especially if you don’t have a preexisting trusting relationship. Real world stuff has fewer absolutes, for one thing.)

Information: A quick guide to information evaluation (image of a fountain pen and blank lined notebook)

Who is this person (or What is this source?)

Start with the basics. Who’s telling you this thing? What’s their background? If it’s a website without an individual author, what do you know about the site?

You may need to file this in “Need to do some more research” but knowing you need to do that is a great first step. First thing: check out the ‘about’ page, or a bio. Usually this will give you some hints on what they’re about and what they care about most.

If you’re not sure where to start with that, try searching the person’s name (plus maybe a term from the topics they write about, if you need to narrow it down) or search on the name of the site. Sometimes adding in words like ‘review’ or ‘about’ will help.

Even just knowing what kind of source this is can help. Personal website? Newspaper that’s actually well-known and reasonably respected (even if you don’t agree with them)? Pocket of internet culture you weren’t previously aware of? Political group hidden behind astroturfing techniques?

I sort things into “Probably reasonably competent”, “Dubious” and “Need more information”, personally.

Probably reasonably competent sources are those I’ve checked out before, and came up reasonably well sourced. I still need to check the specifics here, but they get some starting benefit of the doubt. Dubious sources are those that have come up short before. Everything else gets filed in ‘need more info’.

What are their goals?

Education? Information? Sell something? Share something gorgeous or fun or amusing? Are they trying to persuade you of something?

What do they get out of you believing them and taking them (or their information) seriously? Are they being up front and honest about that?

Here’s an example: sales sites are not the most fun thing ever, but there is something refreshingly honest about “Buy this thing from me and here’s why.” It’s clear what the people want, and usually pretty clear what’s involved in getting it.

On the other hand, a lot of sources in the political realm are trying to persuade you of things, but it’s not always clear what they’re trying to persuade you of. (Or whether they’re not trying to persuade you at all, but are instead signalling to their core base what they care about.)

This is often where you see a lot of vagaries and unsourced information that plays on emotions rather than treating you like the intelligent, thoughtful, considerate person I want to think you are.

Where did they get their information?

This is where we get to the meat of things. People who are saying trustworthy things should give you a way to check, or more information about how they know that.

When we’re talking to a friend, we put what they tell us in the context of all the other things we know about them. They’re reliable as anything with a ride when it’s important, lousy at getting stuff to the post office.

They have a lot of specific experience in dealing with Mercutian rabbits, and the last fifty things they told you about those rabbits turned out to be right, but they’re not nearly so reliable about Venusian wombats. And they’re normally great about Saturnian leopards, but there’s this one weird quirk, don’t trust their grooming recommendations.

When we’re reading a random website, we don’t have that. We can’t put some of what they’re saying in context without more information.

That’s why their sources matter. Do they tell us where they’re getting their info? If it’s unnamed experts and sources, be dubious. (Though there’s a link below with some more about how to evaluate this with more nuance.)

If they claim specific expertise, can you verify that or does it seem in line with what someone with that expertise would say? (If someone claims to be a lawyer or doctor or librarian and says stuff that is way outside what you’d expect, be dubious without more specifics. Maybe a lot more.)

When is this information from? Is this a topic where currency matters a lot? Some topics change fast, some don’t. Sometimes the info that debunks a current thing has been around for a while (so older info may still be helpful in sorting this out.)

What kind of source is this, and is the information presented in a way consistent with quality information in that kind of source? Reputable newspapers don’t generally go in for explicit personal insults or completely unverified sources. (Unless they’re quoting someone who used one.) Less reputable current events sources might.

Expect better of where you go to learn things. If they’re not giving you meaningful information, go to sources that that will. You can do better than speculation and gossip.

Other key tips

Beware of absolutes, especially in complex situations.

There just aren’t that many absolutes in the world. This is especially true when looking at expert statements: few experts will give 100% certainty. If they do, they will likely also be explaining why. Look for that explanation.

If a media source says something absolute, check into what the experts actually said, and what information they looked at to get there. Chances are pretty good the expert was not nearly so absolute about things.

Be dubious of things that are too good to be true, too weird, or too perfect.

Again, the world just isn’t like that very often. The more we realise that we live in a world that has a lot of shades of colour and nuance and different experiences in it, the sooner we’re going to get better at evaluating information effectively and using it well.

Is this a situation where there are strong emotions?

Sourcing is often not the top priority in these cases. Which is understandable, but just because someone’s having emotions all over the place doesn’t mean you have to use everything they tell you as the basis of your decisions.

Emotions don’t mean someone’s wrong, mind you.

It is, for example, pretty reasonable for someone to be emotional about a topic that has a major impact on their daily life, health, safety, family, or religion, if other people are treating it as a purely intellectual discussion. But a story that’s playing on your emotions to make you feel upset or riled up or righteously victorious, you should be suspicious of that.

If emotions are in play, and you’re not in the middle of the discussion, it’s usually better to pause and take a moment to look at what’s being said.

Who has real experience with this thing? Who doesn’t? How does what people are saying match up with other kinds of information you can find or your experience of people or situations? Who has what at stake? Is this a real person who has specific experiences, or is it a made up storm of emotion that’s trying to get you to react a certain way?

Some additional resources:

Here are a few additional links worth reading

This is only a beginning – there are lots of nuanced issues involved in how we find and evaluate information I haven’t even touched on here (like who decides what gets researched that you can refer to later.)

Tacit knowledge : 5 things to start with

One of the hard things about talking about research (and especially research outside of structured academic work) is figuring out where to start. Figuring out where to start is difficult for a number of reasons, but one of the big ones has to do with what is sometimes called tacit knowledge. Those are the things that are obvious to you once you know about them – but utterly mystifying to people who don’t have experience with it.

Tacit knowledge isn’t just a thing for researchers or students or teachers! It’s also an issue for things like cooking or knitting or picking up any new hobby or craft. It’s true for many jobs.

And it’s true for Pagans – think about all the parts of going to a public ritual. Where is it? What do you bring? What should you expect? What’s polite? What’s rude? What do you do if you feel uncomfortable? What is it okay to ask about? Will anyone ask you to do something you really don’t want to do? And then, depending on the ritual, there may be a whole lot of assumed tacit knowledge about things like quarters and elements and the nature of deity and the theory of magic going on.

Clearly, tacit knowledge is something that deserves some quality time on this blog. This post is an introduction to the topic, and I’ll circle back to more specifics in future posts.

The photo for this post is a photo I took at the Greenwich Observatory on a trip to London in 2015. It’s a pocket astronomical device that does about half a dozen things, and is the size of a pocket watch when it’s closed. To use it, you need a tremendous amount of knowledge about the different tools it includes, and how to interpret the markings and settings, most of which are quite tiny – more reminders than information.

Photo of an astronomical pocket device: astrolabe, sextant, and other tools.

What is tacit knowledge?

Tacit knowledge is often defined as “the kind of knowledge that is difficult to transfer by writing it down or verbalising it.

Tacit knowledge is often about having experience with a thing, and trying it out for ourselves and figuring out how it works. (Guidance can help, but a lot of tacit knowledge things, we need to have that hands on experience before they make sense and stick.)

Librarians talk a lot about this kind of thing, because it affects how we teach people how to find and evaluate information. Information literacy skills rely on a lot of tacit knowledge – and to make things more complicated, what that knowledge is changes, sometimes very rapidly.

In the Pagan community, we often talk about religious mysteries, which are definitely a related idea.

In future posts, I’m going to be talking about different pieces in more detail, but here I want to lay out a few broad categories and give you things to think about.

  • Structure
  • Contextual cues
  • Expected audience
  • Citations and references
  • Location and orientation

If you’d like to read more, I got started thinking about this again by coming across Barbara Fister’s 2013 post about tacit knowledge in my saved bookmarks. Threshold concepts are a related approach, and here’s a good brief handout in PDF (aimed at librarians, so there’s a bit of jargon) designed for a webinar on the topic.

1) Structure

Two classic examples for tacit knowledge are understanding how a book works and how a print newspaper works.

For books, it’s how to use the table of contents, index, and references or bibliography to find information in the book or elsewhere that talks about what we’re interested in. For print newspapers, it’s things like how placement on the page or within the section indicates different things, what cues indicate something is an editorial or opinion, and much more.

On the web, we can look at things like how a page or site is structured, if there’s advertising (and what that indicates), whether there’s information about authorship or source, and how to find it if there is. On social media, it can involve understanding how posts get shared, memes get started, and things can change context over time.

2) Contextual cues

A lot of our society is designed to give us cues about how to do things or how seriously to take them. But we’re often not very consciously aware that that’s going on. Design choices can encourage us to use one door or path over another (or to buy one brand or type of product over another).

Many of us have a sense that Comic Sans is not a font choice with much gravitas, or that lots of blinking images across our screens may be someone trying to catch our attention with flash rather than substance. But we may not realise why we read those things that way.

There are also things that we learn by being in a space for a while. People who are active users of Tumblr or Twitter or Facebook will become familiar with the customs there (or at least in the circles they spend time with there) about how to tag things, what kinds of tags are considered useful or acceptable in that set of people, what customs are around things like warning for possibly upsetting content.

People online also signal things in other ways. The theme or account name we use on a blog or service. The avatars or icons or profile photos we choose. The email addresses we share in public. (An email with a legal name is different from WitchyChick333 is different from a magical name.) What sites are linked to (or not linked to). The terms different groups use for the same thing. Which hashtags or other tags get used. Which don’t.

3) Expected audience

One of the most complicated things about information is realising that different audiences sometimes matter a lot. It’s extremely hard to write material that is accessible to someone new to a subject, and to someone who knows a fair bit about it.

Aiming for a middle path, or including supplemental explanations can still leave lots of people out. (And that’s before we get into actual accessibility issues, which are many and also important!)

One of the things I see a lot in the Pagan community is a thing of not understanding how academic writing works – who academics are mostly writing for, what their goals are, how topics of research might be picked, and how work fits into an existing conversation in a particular field (or interdisciplinary conversation.) This all adds up to a lot of assumed tacit knowledge.

Likewise, the way we write for people in the Pagan community (and especially for people in our particular path, or at least general focus) is often a lot different than the way we’d explain things to a family member or non-Pagan friend. This can sometimes make it very hard to share information in an order that makes sense, or it can mean we need to circle back several times to explain things as people get comfortable with earlier topics.

This kind of thing is why time often helps – if you get introduced to a new concept, and then see it in action a couple of times, then talk about it again, you’ll often make more sense of it.

4) Citations and references

Citations are a whole bog of tacit knowledge of their own. Why do we have them? Why do they matter? How do we handle them?

They’re especially boggy when we talk about informal references – the kind of thing we do in a casual conversation, or an online social media space, where we’re often not going to trot out “Oh, this is on page 64 of The Best Book About Dinosaurs” and the full publisher and author information.

And yet, these things are very obscure unless you know how they work. What are all those things? How do you sort out abbreviations? Why do the different pieces make a difference?

(And that’s before we get into the sources themselves, and how we identify a scam journal or publisher, or someone who’s hiding their actual goals for some reason.)

5) Location and orientation

Most people know that there are methods to how libraries put things on shelves. However, many people don’t know (and why would you) where those systems come from, or why it’s hard to change terms, or what’s involved in doing so. Or they don’t realise that the two major systems in use in a lot of the English speaking world (Dewey Decimal and Library of Congress) both have their origins in a very Christian-centric and fairly colonialist worldview, and how that affects how subjects were set up.

The subject term “Wicca” is actually a really interesting case study here, and will be getting its own post and other materials in the future.

When I’m talking about orientation here, it’s also about how we move through information. Many people also don’t know all the different things your public library might be able to do for you, or what some options are if you move beyond what your public library can do. The same goes for online searches: there are a lot of things we can find with search engines, but what about the things the search engines don’t see? Or don’t show us? Poking at the tacit knowledge issues here can bring us huge benefits.

Conclusion

As you can see, these are all very large topics, and I’ve covered them only in the most general terms here, to give you an idea of what we’re going to dig at in future. If you’ve got a particular topic you’d like to see sooner than later, drop me a note on the contact form or in the comments.