Humanities Data Visualization

Signal problems in humanities data visualization:

  1. The specific is more interesting than the general
  2. The data shows its irregularities, if you ask it nicely.
![The data shows it's irregularities, if you ask it nicely.](/slides/images/ghostmaps/701.png)  

Deck 701: American Shipping, 1800-1850

Deck 892, US shipping 1980-1997

Decks of punchcards

0.0000001% of the ICOADS data set.

1848 6 1     3723 29038 02 4    10ISABE*_N   1   5                                                           
165 20779701 69 5 0 1                  FFFFFF77AAAAAAAAAAAA     99 0 790044118480601
3714N 6937W                                                                           NW     51 NW     57 NW   
51                                          201A.STEWART       NEW BEDFORD             WHALING V
OYAGE           2620 199

Digitized logbooks, c. 1930

Wallbrink and Koek

Logbook Digitization in the 1920s

Wallbrink, H. and F.B. Koek, Data Acquisition And Keypunching Codes For Marine Meteorological Observations At The Royal Netherlands Meteorological Institute, 1854–1968

Deck 720:German weather data

Deck 735:Soviet weather data

Soviet Closeup

Bookworm: Visualizing Libraries

Visualizing Libraries

  • Not visualizing texts.

Acknowledgements

Institutions

  • Northeastern University/Rice University Cultural Observatory

People

  • Erez Lieberman Aiden
  • Neva Cherniavsky, Martin Camacho, Matt Nicklay, Billy Janitsch, JB Michel, Piotr Organisciak.

Acknowledgements

Funders

  • Digital Public Library of America
  • Harvard Cultural Observatory
  • National Endowment for the Humanities
  • Mellon Foundation

Partners

  • Hathi Trust Research Center
  • github.com/Bookworm-project
  • bookworm.htrc.org
  • bookworm.culturomics.org

02138

02138

02138

Some Bookworm instances at Northeastern and Rice:

Yale University: full run of Vogue Magazine

http://bookworm.library.yale.edu/collections/vogue/

Medical Heritage Library (30,000 medical books)

US State Department

http://bookworm.htrc.illinois.edu/bookworm/

DSL for large text databases

  1. A query language modeled on MongoDB
  2. A visualization grammar modeled on ggplot

Grammar, part I: Creating a corpus

"search_limits": {
    "publish_country":["United States"],
   "year": {
      "$lte": 1920,
      "$gte": 1890
   },

Grammar, Part II: Defining texts and tokens

{
"plotType": "heatmap",
"search_limits": {
   "word": ["concentrate attention"],"year":{"$lte":1922,"$gte":1850}
},
"aesthetic":{"color":"WordsPerMillion","x":"decade","y":"BenSubject"},
"database":"presidio"
}

View

Grammar, Part II: Defining texts and tokens

Time interactions.

{
"database": "ChronAm",
"plotType": "heatmap",
"method": "return_json",
"search_limits": {
   "publish_year": {
      "$gte": 1860
  },
  "word": ["bicycle"]
},
"aesthetic": {
    "x": "publish_year",
    "y": "publish_day_year",
    "color": "WordsPerMillion"
}
}

View

Library composition

Language of the State of the Union

Comparing the Vocabulary of two TV shows

{
"database": "movies",
"plotType": "worddiv",
"search_limits": {
  "TV_show": ["Seinfeld"]
},
"compare_limits": {
  "TV_show": ["Cheers"]
},
"aesthetic": {
  "label": "unigram",
  "size": "Dunning"
}}

View

Comparing speeches by language

Reading texts in context

Incorporating

{"database": "ChronAm",
"plotType": "map",
"projection":"USA",
"method": "return_json",
"search_limits": {},
"aesthetic": {
  "point": "placeOfPublication_geo",
  "size": "TextCount"
}}

View

{"database": "ChronAm",
"plotType": "map",
"projection":"USA",
"method": "return_json",
"search_limits": {},
"aesthetic": {
  "point": "placeOfPublication_geo",
  "size": "TextCount",
  "time": "publish_year"}}

View

Newspaper flu coverage, 1917-1919

{"database": "ChronAm",
"plotType": "map",
"projection":"USA",
"method": "return_json",
"search_limits": {"word":["flu","influenza","pneumonia"],
           "publish_year":{"$lte":1920,"$gte":1918}},
"smoothingSpan":25,
"aesthetic": {
    "time":"publish_day",
  "point": "placeOfPublication_geo",
  "size": "TextPercent"}}

View

Mapping Places in the State of the Union

Data:

~14 million teaching evaluations from ~2003 to 2014

Scraped (slowly) from ratemyprofessors.com

Processed and tokenized using Bookworm

benschmidt.org/profGender

Same interaction easily adaptable to other collections

Most popular queries (male to female)

he, handsome, jerk, dick, arrogant, genius, gay, hilarious, entertaining, funny, sexy, clever, brilliant, old, cool, ass, intelligent, smart, fat, cute, interesting, weird, engaging, idiot, boring, great, knowledgeable, best, knowledgeable,

inspiring, awesome, good, challenging, pretty, excellent, passionate, fun, lazy, difficult, hot, good teacher, hard, stupid, fair, dumb, bad, competent, easy, approachable, kind, hate, useless, tough, demanding, attractive, creative, bad teacher, enthusiastic, angry, amazing,

love, clear, confusing, ugly, condescending, crazy, happy, understanding, worst, terrible, nice, friendly, helpful, mean, biased, harsh, horrible, unfair, awful, aggressive, organized, rude, incompetent, annoying, caring, strict, disorganized, evil, bossy, beautiful, sweet, she

Intelligence/unintelligence skews male

he, handsome, jerk, dick, arrogant, genius, gay, hilarious, entertaining, funny, sexy, clever, brilliant, old, cool, ass, intelligent, smart, fat, cute, interesting, weird, engaging, idiot, boring, great, knowledgeable, best, knowledgeable,

inspiring, awesome, good, challenging, pretty, excellent, passionate, fun, lazy, difficult, hot, good teacher, hard, stupid, fair, dumb, bad, competent, easy, approachable, kind, hate, useless, tough, demanding, attractive, creative, bad teacher, enthusiastic, angry, amazing,

love, clear, confusing, ugly, condescending, crazy, happy, understanding, worst, terrible, nice, friendly, helpful, mean, biased, harsh, horrible, unfair, awful, aggressive, organized, rude, incompetent, annoying, caring, strict, disorganized, evil, bossy, beautiful, sweet, she

Organization skews female

he, handsome, jerk, dick, arrogant, genius, gay, hilarious, entertaining, funny, sexy, clever, brilliant, old, cool, ass, intelligent, smart, fat, cute, interesting, weird, engaging, idiot, boring, great, knowledgeable, best, knowledgeable,

inspiring, awesome, good, challenging, pretty, excellent, passionate, fun, lazy, difficult, hot, good teacher, hard, stupid, fair, dumb, bad, competent, easy, approachable, kind, hate, useless, tough, demanding, attractive, creative, bad teacher, enthusiastic, angry, amazing,

love, clear, confusing, ugly, condescending, crazy, happy, understanding, worst, terrible, nice, friendly, helpful, mean, biased, harsh, horrible, unfair, awful, aggressive, organized, rude, incompetent, annoying, caring, strict, disorganized, evil, bossy, beautiful, sweet, she

Biased Language, Biased Data

Usage

"Such a nasty woman"

Usage: "consciousness raising"

Testing

A-B Testing

  • Web users randomly assigned to one of two groups; stored in locally in browser cache, persistent through multiple visits.
  • Page renders differently based on group

Adding related words

Subtracting Transitions

Confirmation bias

Top search terms on benschmidt.org/profGender

he, handsome, jerk, dick, arrogant, genius, gay, hilarious, entertaining, funny, sexy, clever, brilliant, old, cool, ass, intelligent, smart, fat, cute, interesting, weird, engaging, idiot, boring, great, knowledgeable, best, knowledgeable,

inspiring, awesome, good, challenging, pretty, excellent, passionate, fun, lazy, difficult, hot, good teacher, hard, stupid, fair, dumb, bad, competent, easy, approachable, kind, hate, useless, tough, demanding, attractive, creative, bad teacher, enthusiastic, angry, amazing,

love, clear, confusing, ugly, condescending, crazy, happy, understanding, worst, terrible, nice, friendly, helpful, mean, biased, harsh, horrible, unfair, awful, aggressive, organized, rude, incompetent, annoying, caring, strict, disorganized, evil, bossy, beautiful, sweet, she

Top search terms on benschmidt.org/profGender

he, handsome, jerk, dick, arrogant, genius, gay, hilarious, entertaining, funny, sexy, clever, brilliant, old, cool, ass, intelligent, smart, fat, cute, interesting, weird, engaging, idiot, boring, great, knowledgeable, best, knowledgeable,

inspiring, awesome, good, challenging, pretty, excellent, passionate, fun, lazy, difficult, hot, good teacher, hard, stupid, fair, dumb, bad, competent, easy, approachable, kind, hate, useless, tough, demanding, attractive, creative, bad teacher, enthusiastic, angry, amazing,

love, clear, confusing, ugly, condescending, crazy, happy, understanding, worst, terrible, nice, friendly, helpful, mean, biased, harsh, horrible, unfair, awful, aggressive, organized, rude, incompetent, annoying, caring, strict, disorganized, evil, bossy, beautiful, sweet, she

Users are (slightly) more likely to exit after looking at a chart that doesn't show bias

Narratives

Census Data

link

Boats

Deck 720:German weather data

Deck 735:Soviet weather data

Soviet Closeup

Shipping

The expansion of whaling

An elaborate API call to the Maury browser.

    [
    {
            "DCK":[701],
            "start":365.25*1840,
            "end":365.25*1940,
            "non_standard_fields":["origin","destination"],
            "color":function(d) {
                    if (d.origin.match("BOSTON|NEW BEDFORD|SALEM") !== null) {return true}
                    if (d.destination.match("BOSTON|NEW BEDFORD|SALEM") !== null) { return true}
                    return false;
            },
            "continuation":true,
            horizon:25,
            "projection":"oceanicHomolosine"
    },
    [...]

Minard

Minard reproduction