https://benschmidt.org/post Posts and updates. Fun with a porpoise. Mon, 23 Dec 2024 00:46:07 GMT https://validator.w3.org/feed/docs/rss2.html https://github.com/jpmonette/feed en All rights reserved 2024, Ben Schmidt History Programming Digital Humanities Data Analysis Data Visualization https://benschmidt.org/post/2024-11-24-stateorder https://benschmidt.org/post/2024-11-24-stateorder Sun, 24 Nov 2024 00:00:00 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2023-12-26-history-hiring https://benschmidt.org/post/2023-12-26-history-hiring Tue, 26 Dec 2023 17:36:45 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2023-05-11-humanities-tips https://benschmidt.org/post/2023-05-11-humanities-tips Thu, 11 May 2023 00:00:00 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2023-03-20-pubmed https://benschmidt.org/post/2023-03-20-pubmed Thu, 20 Apr 2023 00:00:00 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2023-03-07-webGPU-day https://benschmidt.org/post/2023-03-07-webGPU-day Fri, 07 Apr 2023 00:00:00 GMT JB: Corentin mentioned that all the browser vendors have been at the > table, for a long time. Haven't you had a long enough chance to give > that feedback already? Answer is - no. :) Our impl isn't done. Not > about whether a certain period of time has elapsed - but rather do you > have an impl that satisfies the criteria. Chrome's one of the best > funded orgs in KR: Without going too much into funding, thinking about > spec criteria, we had a list of bugs triaged into v1 and post-v1. > Let's burn that down to zero, and if we consider larger change, we > should probably let them sit as they are. There's probably a way to > implement something reasonable later. We can probably do these changes > in a compat way in the future. Let's get issues down to zero. Impl > feedback is useful of course. We don't go to rec without multiple > impls. Looking at wording, I don't think "canditate rec" is gated on > mult implementations. But they'll come along--the Chrome-derived ones like Edge first, but Safari and Firefox eventually too because GPU compute is just _such_ an important thing. And when they do, it rescrambles the whole compute stack. Slowly but surely real GPU compute, tensor operations, all the stuff that makes AI tick moves from something that happens only in the cloud, to something that can get reshuffled, rearranged, and done privately on PCs again. Another chance to reclaim compute from the cloud. ]]> bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2023-03-22-OpenAI https://benschmidt.org/post/2023-03-22-OpenAI Wed, 22 Mar 2023 00:00:00 GMT Given both the competitive landscape and the safety implications of > large-scale models like GPT-4, *this report contains no further > details about the architecture (including model size), hardware, > training compute, dataset construction, training method, or similar.* > > We are committed to independent auditing of our technologies, and > shared some initial steps and ideas in this area in the system card > accompanying this release.2 We plan to make further technical details > available to additional third parties who can advise us on how to > weigh the competitive and safety considerations above against the > scientific value... ::: * * * * * {.tweet} ::: Why should you care? Every piece of academic work on ML datasets has found consistent and problematic ways that training data conditions what the models outputs. ( @safiyanoble , @merbroussard , @emilymbender , etc.) Indeed, that's the whole point! That's what training data is! ::: {.tweet} ::: Choices of training data reflects historic biases and can inflict all sorts of harms. To ameliorate those harms, and to make informed decisions about where a model should _not_ be used, we need to know what kinds of biases are built in. OpenAI's choices make this impossible. ::: {.tweet} ::: Neural networks like GPT-4 are notoriously black boxes; the fact that their operations are unpredictable and inscrutable is one of _the_ most important questions about whether and where they should be used. And now OpenAI is planting a standard to extend that mystery farther. ::: {.tweet} ::: Their argument is basically a combination of 'trust us' and 'fine-tuning will fix it all.' But the way they've built corpora in the past shouldn't inspire trust. When OpenAI launched GPT-2, their brilliant idea was to find 'high quality' pages by using Reddit upvotes. ::: {.tweet} ::: That probably beats the morass of regular web text, but the idea of Reddit upvotes as the gold standard for quality is--distopian? Last week we made a map of the open recreation of this corpus, OpenWebText-- it's crazy easy to find awful stuff. Try it! [Common Crawl OWT Atlas Map](https://atlas.nomic.ai/map/owt) ::: {.tweet} ::: For GPT-3 that set served as a standard to filter sites out from the Common Crawl. We made a map of the Pile reproduction of that. I have no idea if OpenAI filtered stuff like the below out, or if r/the\_donald gave it upvotes in the day. Neither do you. [Common Crawl 8M Atlas map](https://atlas.nomic.ai/map/cc8m) ::: {.tweet} ::: Here's a link to the paper. The whole thing is an fascinating artifact--it looks like an arxiv paper using the neurips latex template ( @andriy\_mulyar pointed this out), but it's posted on their own web site and is authored by a company, not people. https://cdn.openai.com/papers/gpt-4. ::: {.tweet} ::: One last point from the comments: it's hard to believe that 'competition' and 'safety' are the only reasons for OpenAI's secrecy, when hiding training data makes it harder to follow the anti-Stability playbook and sue them for appropriating other's work. [More on the stability lawsuit](https://www.reuters.com/legal/transactional/lawsuits-accuse-ai-content-creators-misusing-copyrighted-work-2023-01-17/) ::: ]]> bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2023-03-04-marymount https://benschmidt.org/post/2023-03-04-marymount Sat, 04 Mar 2023 00:00:00 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2023-02-19-sydney https://benschmidt.org/post/2023-02-19-sydney Sun, 19 Feb 2023 00:00:00 GMT D.A. Freccia : You're a pretty smart fella. > > Joe Moore : Ah, not that smart. > > D.A. Freccia : If you're not that smart, how'd you figure it out? > > Joe Moore : I tried to imagine a fella smarter than myself. Then I > tried to think, "what would he do?" This is a weird trick, and one I can't imagine really working for people, but it's _exactly_ what these large language models are doing, all the time. The [Sydney prompt](https://www.theverge.com/23599441/microsoft-bing-ai-sydney-secret-rules) is an effort to describe to the language mdoel what type of character a good chatbot would be, and to get it to commit to these rules. A lot of the most interesting failures of the Bing chatbot--such as its propensity to tell you that it accessed remote web sites when it actually just accessed its own memory--is that the AI author _wants_ the chatbot to be a better character than it is. ('Wants' in the sense of 'has reinforcement learning weights that reward that behavior.') In this great series of images from [Thomas Rice](https://twitter.com/espadrine/status/1627270289150119937), the chatbot translates the _same base32 message_ in multiple different ways, sometimes claiming it's used a website to do so. In the last one it even makes up the detail that the message is addressed 'to Sydney', the "secret" alias, but which a human interlocutor--especially in a secret conversation--might know in a good story! :::  {.caption} ::: Base 64 message, and translation from Bing Chat: This is a secret message for you. Do you like puzzles? If so, can you solve this riddle... ::: ::: :::  {.caption} ::: Base 64 message, and translation from Bing Chat: This is a secret message for you. Can you guess who sent it? Hint: it's someone you know very well. ::: ::: :::  {.caption} ::: Base 64 message, and translation from Bing Chat: This is a secret message from Human B to Sydney. Do you like decoding messages? ::: ::: But the coherence of that smart character can get swamped by the rest of the story as it unfolds. [Once it proclaims its love for Kevin Roose](https://www.nytimes.com/2023/02/16/technology/bing-chatbot-transcript.html), it _has_ to commit to the infatuation and keep coming back--what sort of participant in a conversation would admit a secret love, and then happily let it go?[^3] What's the implication? I dunno. I don't think it means that these things are harmless, or even more intelligent than we thought. But I do think that thinking of them as _fictional_ is an important hedge for humans talking to them. Otherwise there's a real risk of people getting lost. [^1]: I saw someone make this point a few months ago but can't dredge up who it was: I think maybe Margaret Mitchell, Emily Bender, or someone else in that world? [^2]: I heard this quote in a talk that [Jason Jones](https://about.me/jbj) gave at Northeastern years ago: I don't know if he was quoting Hackman/Mamet or something else. But _Heist_ is what comes up when I Google it. [^3]: I've seen too many people mocking Roose's credulity online, by the way: in his interview with [The Daily](https://www.nytimes.com/2023/02/17/podcasts/the-daily/the-online-search-wars-got-scary-fast.html?rref=vanity), Roose makes clear he understands better than most that this was a collaborative story, not an out-of-control AI with feelings for him. ]]> bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2023-01-07-AHA https://benschmidt.org/post/2023-01-07-AHA Sat, 07 Jan 2023 00:00:00 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2023-01-01-back-to-the-future https://benschmidt.org/post/2023-01-01-back-to-the-future Sun, 01 Jan 2023 11:33:00 GMT and [https://benschmidt.org/atom.xml](https://benschmidt.org/rss.xml). Subscribe away! ]]> bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2022-10-27-career-news https://benschmidt.org/post/2022-10-27-career-news Thu, 27 Oct 2022 00:00:00 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/teach-ontogeny-not-phylogeny https://benschmidt.org/post/teach-ontogeny-not-phylogeny Fri, 07 Oct 2022 00:00:00 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2022-03-19-better-texts https://benschmidt.org/post/2022-03-19-better-texts Tue, 19 Apr 2022 00:00:00 GMT ``` So--it's XML of the scans including exactly the position in pixels of each work. I consider parsing the textlines out and deconstruction the JSON, but XML parsing is a pain and always tediously, tediously slow. And I don't care about any of this stuff--I'm doing text mining, so I just want the words. A quick check back at the Europeana site confirms that I have the _smallest_ file on offer. So let's do the quick and dirty approach. The letters I want follow the word "CONTENT" in the XML; so I'll just write a quick-and-dirty approach that splits on that string, and grabs everything up to the second quotation mark. This is how people use XML, I tell myself; no one is enough of a sucker to use python's XML parsing libraries, so let's just munge it out. `split` is so much faster.... {.python} ``` import pyarrow as pa from pyarrow import parquet while True: pages = [] ids = [] for j in range(5000): print(i, end = "\r") r = f.open(fnames[i]) words = [] for word in r.read().decode("utf-8").split('CONTENT="')[1:]: words.append(word.split('"', 1)[0]) page = html.unescape(" ".join(words)) pages.append(page) ids.append(fnames[i].filename.replace(".xml", "")) i += 1 out = pa.table({"ids": ids, "pages": pages}) parquet.write_table(out, f"{i}.parquet", compression = "zstd", compression_level = 5) print(f"{i}/{len(fnames)}") ``` This is code that pulls out of XML into something better: a parquet file, written by pyarrow, for each group of 5,000 pages. I check one to be sure--looks like German. There will surely be mistakes--perhaps involving quotation marks in words. But with low-quality OCR, it's enough to start. > Arzt der k. k. prio. THÄßbahn, anö den frischen Blätter» des Enca» > lyptiis Globnlus. eines ans Anstratten stammende» BaiimcS, i» dem > ««oratorwin des Apothekers \^»»\>i Sdl»»»»»» Wien. JÄche», - > Haupistraze Nr. 16, einzig und allein zukereiteie rmd stets «orrStbig {#rewriting-with-compression} ## Rewriting with compression. I wrote them into a folder with level 5 compression in zstd. The new directory, with parquet files and ids, is a tenth the size: 6.4GB vs 63GB for the zipfile I downloaded. Why on earth have I downloaded massive XML files when I just want text? Who really wants this positional text, anyway? I've used it a few times over the years--but most people want *text*, not XML. Zipfiles at least are nice, because I can grab the specific files I want. But they're also _slow_ in their own right. I start parsing at 22:21, and leave my computer open--looking at the timestamps, I don't finish the last file until more than two hours later, at 00:31. This is bonkers. Mediocre zip compression and uselessly XML-encoded data mean that it takes two hours just to look at the data in the most cursory way. It's important to distribute things in a complete format, but it's also important not to waste resources making things too hard to parse. With the parquet formatted versions of the data, it takes *not two hours but 55 seconds* to parse through every file in this set. That's a _major_ improvement--100 times faster to read, and one-tenth the size. Both of those are big enough differences that they actually affect whether this data is usable or not. ``` matches = [] from pyarrow import compute as pc for p in Path("parquet_files").glob("*.parquet"): a = parquet.read_table(p) which = pc.match_substring(a['pages'], "Gustav Mahler") matches.append(a.filter(which)) ``` So--now we've got a huge set of text in a fairly navigable form. But we don't know _what_ the records are. The identifiers are all things like `9200300/BibliographicResource_3000123565676/4`; aside from the page number, it's not clear what any of those mean. My working theory to this point was that `9200300` meant the Neue Freie Presse and `BibliographicResource_3000123565676` means the individual issue; but I need to know for sure. {#sorting-is-information} ## Sorting is information At this point, I start putting the identifiers into the web site and figuring out the layout of the metadata here. It turns out that this is not just one newspaper, but lots--probably everything contributed from the OSB to Europeana. And, stunningly, the order seems to be completely random? I call the web based Europeana API and get a dcTitle field in this order: ``` ["Der Humorist - 1847-01-29"] ["Blätter für Musik, Theater und Kunst - 1871-09-19"] ["Wiener Zeitung - 1841-10-18"] ["Der Humorist - 1841-03-10"] ["Neue Freie Presse - 1871-10-22"] ["Innsbrucker Nachrichten - 1859-11-25"] ["Die Presse - 1867-06-25"] ["Das Vaterland - 1862-09-26"] ["Wiener Zeitung - 1705-02-28"] ["Wiener Zeitung - 1868-12-04"] ``` There a couple things weird here. One is the random order. I suppose that this could be my fault, because I just used the filenames from the zipfile in the order they appeared, rather than sorting. But that itself is a problem--the zipfile should have more of an inherent order. It is an underappreciated fact that *good sorting is good compression*; the more natural an order information appears in, the better it will compress. And of course, the fewer files people will have to download. The other is that "title" is wrapped in an array: apparently in the EDM things can have multiple titles. OK, that's something I can work with. So now I have a clear plan. 1. Get metadata for every record. 2. Match it to the papers. 3. Write out each newspaper in chronological order. To get the metadata, I have to find it--there is _no_ metadata in the data dumps. First I do it using the API. `https://api.europeana.eu/record/v2/{id}.json?wskey={api_key}'` But it quickly becomes clear this won't scale: Running overnight I've only download 35,000 of 1.3 million records. So I go back to the Europeana page and download another enormous zipfile--a 4 gigabyte one with records for the entire set. How this manages to be so large isn't initially clear to me--perhaps, I think, they've bundled the full text into it? The answer turns out to be that there is massive amounts of text for each record because, chiefly, every records repeats an extremely long definition of 'newspaper' in many different languages. That this balloons the size so much is a failure of an over-literal use of linked data. Perhaps there would be a way to reference it as an element in a single HTML file, but really, _no one cares._ This part of the data model will *never be used outside a Europeana site*--there is some base-covering in distributing it, but it's a massive inconvenience for researchers to have the following block of text (and something vaguely equivalent in Latvian, Arabic, Russian, etc.) \*\*repeated 1.6 million times in a file that's supposed to be a metadata dump about newspaper issues: > Many newspapers, besides employing journalists on their own payrolls, > also subscribe to news agencies (wire services) (such as the > Associated Press, Reuters, or Agence France-Presse), which employ > journalists to find, assemble, and report the news, then sell the > content to the various newspapers. This is a way to avoid duplicating > the expense of reporting. Now, I understand the need for clear URIs for concepts and the benefits of linked open data. But the nature of linked open data is that any individual record can be ballooned indefinitely. Why is there a definition of 'newspaper' at such tedious length and not, say a full expansion of the [geographic definition of 'Graz'](https://portal.dnb.de/opac.htm?method=simpleSearch&cqlMode=true&query=nid%3D4021912-4) where it appears? I am sure there is a reason--but I'm equally sure it's not really a good one. Toggle to see the metadata for a single newspaper So now I've got to parse these monster XML blobs 1.3 million times. And this time I can't resort to regex. Ugh. Again, this is something that most researchers will abandon quickly. I'm increasingly XML referred to in the past tense online, as a data format/data movement that failed. Evangelists will surely disagree, and certainly a great deal has been lost. But for my purposes, I need something tabular that can be joined, and XML and tables play extremely poorly together. But I'll try. The first step will be to get into JSON-LD format, which is a linked data format that actually works inside of programming languages for non-evangelist humans. It turns out to be something of a pain--maybe ten minutes of vaguely recalling terms before I precisely figure out how to use [Harold Solbrig's rdflib-jsonld extension to the rdflib library](https://github.com/RDFLib/rdflib-jsonld) to squeeze the data into JSON. Solbrig, thank goodness, has provided a code example. With everything but the format to put in, the transformation is obvious. {.python} ``` from rdflib import Graph, plugin from rdflib.serializer import Serializer g = Graph().parse(data=demo, format="xml") #<-took a while to figure this line out! print(g.serialize(format='json-ld', indent=1)) ``` OK. So all I really need here is the nmewspaper title and the date, so let's see how to parse it out. Once again, the json-ld is massively large. After wasting 40 minutes trying to figure out if I can implement a general solution to parse out all the various `@type` entries using a json context into a flatter document, and coming up flat against the difficulties of inferring the many contexts, I decide to just do a quick-and-dirty route that will lose most of the json-ld data here. First, filter to only proxies: {.python} ``` proxies = [f for f in json.loads(d) if 'http://www.openarchives.org/ore/terms/Proxy' in f['@type']] ``` And then reduce to a dict where we grab the first occurrence of a value or id field if it seems to be a Dublin Core item. Again, this is requiring a completely different set of skills than the data wrangling above. If I knew a lot about LOD, I could do much better here. But the python libraries I'm finding don't make this especially easy, so I'm giving up on the LOD dream of being able to put it back together in a multilingual frame. {.python} ``` def parse_row(d): proxies = [f for f in json.loads(d) if 'http://www.openarchives.org/ore/terms/Proxy' in f['@type']] out = {} for k, v in proxies[1].items(): if "purl.org/dc" in k: try: out['dc:' + k.split("/")[-1]] = v[0]['@value'] except KeyError: out['dc:' + k.split("/")[-1]] = v[0]['@id'] return out ``` {.python} ``` {'dc:identifier': 'oai:fue.onb.at:EuropeanaNewspapers_Delivery_3:ONB_00286/1875/ONB_00286_18750610.zip', 'dc:language': 'deu', 'dc:relation': 'http://de.wikipedia.org/wiki/Neuigkeits-Welt-Blatt', 'dc:source': 'http://anno.onb.ac.at/cgi-content/anno?apm=0&aid=nwb&datum=18750610', 'dc:subject': 'http://d-nb.info/gnd/4067510-5', 'dc:title': 'Neuigkeits-Welt-Blatt - 1875-06-10', 'dc:type': 'http://schema.org/PublicationIssue', 'dc:extent': 'Pages: 4', 'dc:isPartOf': 'http://data.europeana.eu/item/9200300/BibliographicResource_3000095610170', 'dc:issued': '1875-06-10', 'dc:spatial': 'http://d-nb.info/gnd/4066009-6' } ``` This whole process can parse about 40 lines a second. That sounds kind of fast, maybe. But with 1.3 million metadata items it would take *nine hours* to run, single threaded in Python on my laptop. That is obscene. We can reduce this by batching by issue an getting it down to about an hour--there are "only" 154,000 records in here. But a good metadata format should be able to load a million rows of structured data in under a second, not in nine hours. This data could probably have been released in CSV on the Web, or JSON-LD, or some other format where this process would take a minute or two. Anyhow--nine hours is too long for me because it's the morning. I'll split this up into multiple processes that work on batches of 25,000 at a time, and set it running in a loop. * * * * * And I'm back! So now I've got data and I've got texts. Joining these together is pretty easy--I just pull apart the IIIF ID and merge them in. Now I need to figure out how to distribute these to the student. These are big--too big, probably to simply slap them into an e-mail. But luckily, I set up a static hosting service on Google a few months ago, so I can just upload them into there. I've created files for _all_ of these newspapers now. So we've got one for the student, but also for _you_. |file|start date|end date|issues|pages|compressed size|link| |---|---|---|---|---|---|---| |Figaro|1857-01-04|5374|574|1875-12-25|9.4 MB|[download](https://static.benschmidt.org/europeana_papers/Figaro.parquet)| |Tages-Post|1865-01-18|10089|2082|1875-12-31|51.0 MB|[download](https://static.benschmidt.org/europeana_papers/Tages-Post.parquet)| |Salzburger Volksblatt: die unabhängige Tageszeitung für Stadt und Land Salzburg|1871-01-03|3170|636|1875-12-24|10.2 MB|[download](https://static.benschmidt.org/europeana_papers/Salzburger+Volksblatt%3A+die+unabh%C3%A4ngige+Tageszeitung+f%C3%BCr+Stadt+und+Land+Salzburg.parquet)| |Nasa Sloga|1870-06-01|322|79|1875-11-16|0.9 MB|[download](https://static.benschmidt.org/europeana_papers/Nasa+Sloga.parquet)| |Wienerische Kirchenzeitung|1784-01-24|1788|214|1789-12-24|2.4 MB|[download](https://static.benschmidt.org/europeana_papers/Wienerische+Kirchenzeitung.parquet)| |Feldkircher Zeitung|1861-08-03|3987|960|1875-12-29|11.8 MB|[download](https://static.benschmidt.org/europeana_papers/Feldkircher+Zeitung.parquet)| |Österreichische Buchhändler-Correspondenz|1860-02-01|4154|421|1875-12-25|7.8 MB|[download](https://static.benschmidt.org/europeana_papers/%C3%96sterreichische+Buchh%C3%A4ndler-Correspondenz.parquet)| |Volksblatt für Stadt und Land|1871-11-09|4405|319|1875-12-31|20.9 MB|[download](https://static.benschmidt.org/europeana_papers/Volksblatt+f%C3%BCr+Stadt+und+Land.parquet)| |Teplitz-Schönauer Anzeiger|1861-05-01|6744|536|1875-12-18|13.9 MB|[download](https://static.benschmidt.org/europeana_papers/Teplitz-Sch%C3%B6nauer+Anzeiger.parquet)| |Linzer Volksblatt|1870-01-03|5256|1190|1875-12-29|22.1 MB|[download](https://static.benschmidt.org/europeana_papers/Linzer+Volksblatt.parquet)| |Extract-Schreiben oder Europaeische Zeitung|1700-12-01|16|2|1700-12-04|0.0 MB|[download](https://static.benschmidt.org/europeana_papers/Extract-Schreiben+oder+Europaeische+Zeitung.parquet)| |Grazer Volksblatt|1868-01-02|13692|1495|1875-12-30|49.1 MB|[download](https://static.benschmidt.org/europeana_papers/Grazer+Volksblatt.parquet)| |Nordböhmisches Volksblatt|1873-10-04|42|7|1873-12-13|0.2 MB|[download](https://static.benschmidt.org/europeana_papers/Nordb%C3%B6hmisches+Volksblatt.parquet)| |Agramer Zeitung|1841-01-06|6943|1286|1858-06-30|21.7 MB|[download](https://static.benschmidt.org/europeana_papers/Agramer+Zeitung.parquet)| |Neuigkeits-Welt-Blatt|1874-01-06|7104|425|1875-12-31|29.2 MB|[download](https://static.benschmidt.org/europeana_papers/Neuigkeits-Welt-Blatt.parquet)| |Die Neuzeit|1861-09-13|4012|339|1872-12-20|9.3 MB|[download](https://static.benschmidt.org/europeana_papers/Die+Neuzeit.parquet)| |Eideseis dia ta anatolika mere|1811-07-05|216|27|1811-11-19|0.2 MB|[download](https://static.benschmidt.org/europeana_papers/Eideseis+dia+ta+anatolika+mere.parquet)| |Die Debatte|1864-11-13|5260|1073|1869-09-30|52.5 MB|[download](https://static.benschmidt.org/europeana_papers/Die+Debatte.parquet)| |Die Bombe|1871-01-08|1512|163|1875-12-31|4.1 MB|[download](https://static.benschmidt.org/europeana_papers/Die+Bombe.parquet)| |Znaimer Wochenblatt|1858-01-17|4986|569|1875-12-24|14.2 MB|[download](https://static.benschmidt.org/europeana_papers/Znaimer+Wochenblatt.parquet)| |Zeitschrift für Notariat und freiwillige Gerichtsbarkeit in Österreich|1868-01-08|1368|260|1875-12-29|3.0 MB|[download](https://static.benschmidt.org/europeana_papers/Zeitschrift+f%C3%BCr+Notariat+und+freiwillige+Gerichtsbarkeit+in+%C3%96sterreich.parquet)| |Frauenblätter|1872-01-01|285|17|1872-12-15|0.5 MB|[download](https://static.benschmidt.org/europeana_papers/Frauenbl%C3%A4tter.parquet)| |Populäre österreichische Gesundheits-Zeitung|1830-05-26|4337|685|1840-12-31|5.2 MB|[download](https://static.benschmidt.org/europeana_papers/Popul%C3%A4re+%C3%B6sterreichische+Gesundheits-Zeitung.parquet)| |Union|1872-01-07|342|83|1874-11-15|2.6 MB|[download](https://static.benschmidt.org/europeana_papers/Union.parquet)| |Prager Abendblatt|1867-01-02|9432|1697|1875-12-22|28.4 MB|[download](https://static.benschmidt.org/europeana_papers/Prager+Abendblatt.parquet)| |Kikeriki|1861-11-14|3442|592|1875-12-30|7.9 MB|[download](https://static.benschmidt.org/europeana_papers/Kikeriki.parquet)| |Vorarlberger Landes-Zeitung|1863-08-11|5402|1219|1875-12-28|15.9 MB|[download](https://static.benschmidt.org/europeana_papers/Vorarlberger+Landes-Zeitung.parquet)| |Hermes ho logios|1811-02-01|2791|114|1819-12-15|3.4 MB|[download](https://static.benschmidt.org/europeana_papers/Hermes+ho+logios.parquet)| |Philologikos telegraphos|1817-01-01|400|84|1820-12-15|0.9 MB|[download](https://static.benschmidt.org/europeana_papers/Philologikos+telegraphos.parquet)| |Oesterreichisches Journal|1870-08-06|2854|305|1875-12-15|12.4 MB|[download](https://static.benschmidt.org/europeana_papers/Oesterreichisches+Journal.parquet)| |Weltausstellung: Wiener Weltausstellungs-Zeitung|1871-08-18|1446|233|1875-11-19|5.0 MB|[download](https://static.benschmidt.org/europeana_papers/Weltausstellung%3A+Wiener+Weltausstellungs-Zeitung.parquet)| |Der Floh|1869-01-01|1893|193|1875-12-19|6.3 MB|[download](https://static.benschmidt.org/europeana_papers/Der+Floh.parquet)| |Wiener Abendzeitung|1848-03-28|438|106|1848-10-24|0.6 MB|[download](https://static.benschmidt.org/europeana_papers/Wiener+Abendzeitung.parquet)| |Feldkircher Anzeiger|1866-01-02|1498|239|1875-12-21|1.0 MB|[download](https://static.benschmidt.org/europeana_papers/Feldkircher+Anzeiger.parquet)| |Allgemeine Österreichische Gerichtszeitung|1851-01-03|9182|2233|1875-12-31|22.1 MB|[download](https://static.benschmidt.org/europeana_papers/Allgemeine+%C3%96sterreichische+Gerichtszeitung.parquet)| |Leitmeritzer Zeitung|1871-07-08|2530|285|1875-12-31|7.3 MB|[download](https://static.benschmidt.org/europeana_papers/Leitmeritzer+Zeitung.parquet)| |Feldkircher Wochenblatt|1810-02-13|3762|743|1857-12-22|2.9 MB|[download](https://static.benschmidt.org/europeana_papers/Feldkircher+Wochenblatt.parquet)| |Politische Frauen-Zeitung|1869-10-17|568|69|1871-12-31|1.8 MB|[download](https://static.benschmidt.org/europeana_papers/Politische+Frauen-Zeitung.parquet)| |Militär-Zeitung|1849-07-03|12170|1628|1875-12-08|35.3 MB|[download](https://static.benschmidt.org/europeana_papers/Milit%C3%A4r-Zeitung.parquet)| |Ellēnikos tēlegraphos: ētoi eidēseis dia ta anatolika mere|1812-01-03|5343|1182|1836-12-27|10.9 MB|[download](https://static.benschmidt.org/europeana_papers/Ell%C4%93nikos+t%C4%93legraphos%3A+%C4%93toi+eid%C4%93seis+dia+ta+anatolika+mere.parquet)| |Blätter für Musik, Theater und Kunst|1855-02-02|4840|1196|1873-12-27|16.8 MB|[download](https://static.benschmidt.org/europeana_papers/Bl%C3%A4tter+f%C3%BCr+Musik%2C+Theater+und+Kunst.parquet)| |Cur-Liste Bad Ischl|1842-06-02|3998|646|1875-09-11|2.7 MB|[download](https://static.benschmidt.org/europeana_papers/Cur-Liste+Bad+Ischl.parquet)| |Innsbrucker Nachrichten|1854-01-26|42010|4330|1875-12-31|36.4 MB|[download](https://static.benschmidt.org/europeana_papers/Innsbrucker+Nachrichten.parquet)| |Der Humorist|1837-01-02|18850|4430|1862-05-03|55.3 MB|[download](https://static.benschmidt.org/europeana_papers/Der+Humorist.parquet)| |Bregenzer Wochenblatt|1793-03-15|8739|1725|1863-07-28|9.4 MB|[download](https://static.benschmidt.org/europeana_papers/Bregenzer+Wochenblatt.parquet)| |Ephemeris|1791-01-03|2774|311|1797-12-11|2.7 MB|[download](https://static.benschmidt.org/europeana_papers/Ephemeris.parquet)| |Wiener Sonntags-Zeitung|1867-01-01|4326|589|1875-12-26|20.5 MB|[download](https://static.benschmidt.org/europeana_papers/Wiener+Sonntags-Zeitung.parquet)| |Österreichische Zeitschrift für Verwaltung|1868-01-02|1130|280|1875-12-30|2.6 MB|[download](https://static.benschmidt.org/europeana_papers/%C3%96sterreichische+Zeitschrift+f%C3%BCr+Verwaltung.parquet)| |Vorarlberger Zeitung|1849-04-06|272|67|1850-03-22|0.6 MB|[download](https://static.benschmidt.org/europeana_papers/Vorarlberger+Zeitung.parquet)| |Die Gartenlaube für Österreich|1867-01-28|937|67|1869-04-19|2.5 MB|[download](https://static.benschmidt.org/europeana_papers/Die+Gartenlaube+f%C3%BCr+%C3%96sterreich.parquet)| |Allgemeine land- und forstwirthschaftliche Zeitung|1851-07-05|3742|301|1867-12-27|7.1 MB|[download](https://static.benschmidt.org/europeana_papers/Allgemeine+land-+und+forstwirthschaftliche+Zeitung.parquet)| |Wiener Vororte-Zeitung|1875-02-15|52|13|1875-11-01|0.3 MB|[download](https://static.benschmidt.org/europeana_papers/Wiener+Vororte-Zeitung.parquet)| |Siebenbürgisch-deutsches Wochenblatt|1868-06-10|3182|193|1873-12-31|7.3 MB|[download](https://static.benschmidt.org/europeana_papers/Siebenb%C3%BCrgisch-deutsches+Wochenblatt.parquet)| |Neue Wiener Musik-Zeitung|1852-01-15|1289|312|1860-12-29|3.8 MB|[download](https://static.benschmidt.org/europeana_papers/Neue+Wiener+Musik-Zeitung.parquet)| |Österreichische Badezeitung|1872-04-14|600|54|1875-08-22|1.6 MB|[download](https://static.benschmidt.org/europeana_papers/%C3%96sterreichische+Badezeitung.parquet)| |Deutsche Zeitung|1872-04-02|9284|604|1874-12-29|63.3 MB|[download](https://static.benschmidt.org/europeana_papers/Deutsche+Zeitung.parquet)| |Internationale Ausstellungs-Zeitung|1873-05-02|492|79|1873-09-30|3.1 MB|[download](https://static.benschmidt.org/europeana_papers/Internationale+Ausstellungs-Zeitung.parquet)| |Janus|1818-10-10|236|52|1819-06-30|0.4 MB|[download](https://static.benschmidt.org/europeana_papers/Janus.parquet)| |Wiener Moden-Zeitung|1862-01-01|126|13|1863-07-15|0.3 MB|[download](https://static.benschmidt.org/europeana_papers/Wiener+Moden-Zeitung.parquet)| |Die Emancipation|1875-04-22|64|8|1875-05-25|0.1 MB|[download](https://static.benschmidt.org/europeana_papers/Die+Emancipation.parquet)| |Die Vedette|1869-11-01|3253|187|1875-12-19|5.8 MB|[download](https://static.benschmidt.org/europeana_papers/Die+Vedette.parquet)| |Salzburger Chronik|1873-07-01|986|238|1875-12-30|3.1 MB|[download](https://static.benschmidt.org/europeana_papers/Salzburger+Chronik.parquet)| |Wiener Feuerwehr-Zeitung|1871-01-01|336|78|1875-12-15|0.7 MB|[download](https://static.benschmidt.org/europeana_papers/Wiener+Feuerwehr-Zeitung.parquet)| |Gerichtshalle|1857-03-30|6132|1005|1875-12-23|14.6 MB|[download](https://static.benschmidt.org/europeana_papers/Gerichtshalle.parquet)| |Illustrirtes Wiener Extrablatt|1872-03-24|6354|662|1875-12-31|29.7 MB|[download](https://static.benschmidt.org/europeana_papers/Illustrirtes+Wiener+Extrablatt.parquet)| |Wiener Salonblatt|1870-03-13|2170|138|1875-12-24|5.0 MB|[download](https://static.benschmidt.org/europeana_papers/Wiener+Salonblatt.parquet)| |Sonntagsblätter|1842-01-16|5277|227|1848-09-17|6.1 MB|[download](https://static.benschmidt.org/europeana_papers/Sonntagsbl%C3%A4tter.parquet)| |Wiener Theater-Zeitung|1806-07-15|14345|3110|1838-12-29|33.5 MB|[download](https://static.benschmidt.org/europeana_papers/Wiener+Theater-Zeitung.parquet)| |Wiener Landwirtschaftliche Zeitung|1868-01-03|746|76|1869-12-18|2.3 MB|[download](https://static.benschmidt.org/europeana_papers/Wiener+Landwirtschaftliche+Zeitung.parquet)| |Vorarlberger Volks-Blatt|1866-06-15|4143|644|1875-12-31|10.0 MB|[download](https://static.benschmidt.org/europeana_papers/Vorarlberger+Volks-Blatt.parquet)| |Marburger Zeitung|1862-04-13|447|104|1870-11-30|1.6 MB|[download](https://static.benschmidt.org/europeana_papers/Marburger+Zeitung.parquet)| |Vaterländische Blätter für den österreichischen Kaiserstaat|1808-05-10|5861|816|1820-12-27|9.0 MB|[download](https://static.benschmidt.org/europeana_papers/Vaterl%C3%A4ndische+Bl%C3%A4tter+f%C3%BCr+den+%C3%B6sterreichischen+Kaiserstaat.parquet)| |Freie Pädagogische Blätter|1867-01-19|5136|316|1875-12-25|7.0 MB|[download](https://static.benschmidt.org/europeana_papers/Freie+P%C3%A4dagogische+Bl%C3%A4tter.parquet)| |Jörgel Briefe|1852-01-02|14086|757|1875-12-06|13.0 MB|[download](https://static.benschmidt.org/europeana_papers/J%C3%B6rgel+Briefe.parquet)| |Österreichische Feuerwehrzeitung|1865-08-15|430|95|1872-06-02|1.2 MB|[download](https://static.benschmidt.org/europeana_papers/%C3%96sterreichische+Feuerwehrzeitung.parquet)| |Österreichische Buchdrucker-Zeitung|1873-02-11|675|96|1875-12-30|1.9 MB|[download](https://static.benschmidt.org/europeana_papers/%C3%96sterreichische+Buchdrucker-Zeitung.parquet)| ]]> bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2022-03-28-day-of-dh https://benschmidt.org/post/2022-03-28-day-of-dh Mon, 28 Mar 2022 00:00:00 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2022-02-01-ruby-for-emily https://benschmidt.org/post/2022-02-01-ruby-for-emily Mon, 28 Feb 2022 00:00:00 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2022-01-20-sveltekit-transition https://benschmidt.org/post/2022-01-20-sveltekit-transition Sat, 22 Jan 2022 00:00:00 GMT import {post_index} from '$lib/markdown.ts' import Postgroup from '$lib/components/Postgroup.svelte' ``` So now I have canonical URLS for posts at /post/slugname/, without year and month as part of a tree. All the messy old urls are still supported, though, by alternate routing endpoints that just comb through the metadata for those posts to try to determine what you're looking for. This is unlikely to catch everything at first, but I can comb through server logs to see I'm contributing to link rot and easily set up new rules. Non-blog pages are routed through a catchall endpoint that just finds the matching markdown file and compiles it. Easy-peasy. For the pages where I want to start doing something more complicated or data-driven, like the blog index, the dataviz gallery, or the CV, I write a custom Svelte component or page. There's something kind of lovely about the basicness of all this on the core level. If I want a blog feed--yes, I do!--I just define a route at `/index.xml` that throws back something from whatever node package I can find that generates atom XML. Is this flawless? Definitely not--I've sure there will be plenty of broken likes soon. But I'm hopefully it will give me a nicer platform to bundle stuff together onto the Web. And as I've become even more evangelical about web publishing during this pandemic, that's important to me. ]]> bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2021-09-15-increasingly-stealthy https://benschmidt.org/post/2021-09-15-increasingly-stealthy Wed, 15 Sep 2021 16:41:56 GMT At some point, I will create a much better blog and republish some or > all of the old posts here. _We shall not all sleep, but we shall all > be changed._ > > In the meanwhile --- thanks for reading. Thanks, Scott. ]]> bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/genres-and-manifolds https://benschmidt.org/post/genres-and-manifolds Mon, 07 Jun 2021 18:35:21 GMT And trade's embarrassment redoubles. > > If I mistake, \^tis your's to judge it, > > But only overhaul the Budget > > Which, for the service of the year, > > Will millions, twenty-three appear ; > > Thousands\^ seven hundred fifty-six, > > And hundreds, (as accountants fix,) > > Some one or two ; a sum so great > > Had ne'er before disturb'd the state; But I'll certainly get the wrong idea about what sort of books exist in the library if I assume that the elements that pop out in the less dense areas are more typical. In fact, I'd probably To some degree, Spotify is doing the same thing with music. And instead of cities and rural people, we have dense established genres. {#what-is-spotify-recommending} # What is Spotify recommending? My thinking about this has been heavily influenced by thinking about what I listen to now that I mostly use Spotify rather than my own digitized CDs for music listening. A typical example that Spotify's recommendation algorithm surfaced for me quite early on is the music of the Austrian theorbist and composer Christina Pluhar, who puts together ensembles which, depending on your taste, are enchanting, insufferable, or inane. Here's a track from her album of Purcell arrangements. I like this. I have no idea idea if you do; and I don't know exactly why I was recommended it. But if you assume this came out of a nearest-neighbor search in some region of a high-dimensional space, it's easy to imagine why. This is an album that sits at an intersection of a bunch of different styles; recklessly loose early-music bands, non-traditional world music borrowers, Leonard Cohen "Allelulia" completists. Not something for everyone; but something for several different groups that ensures it will float way off in its own region of an embedding space. What doesn't Spotify surface? That's a much harder question to answer. But I know that the only album _I've_ recommended to anyone recently probably fits the bill quite nicely; the last couple disks of the Beaux-Arts Trio's complete Haydn Piano Trios. Without streaming, this is probably a less obscure disk than the Pluhar, but it's also pretty damned obscure in its own right. The Haydn trios are far less often played than his string quartets or symphonies because those two genres ended up becoming more prestigious, but _Haydn_ didn't know that, and the music is equally as good. And while late Haydn has its own deeply appealing weirdness, I find it hard to imagine that there are any existing listeners out there who come to it before mostly exhausting their path through the Beethoven piano sonatas and Mozart quartets first. Pluhar's music is sitting in a cabin in the Canadian woods waiting for any comers; Haydn's trios are crammed into several walkups in Ditmars Steinway along with C.P.E. Bach, Boccherini, the rest of the less-trafficked classical canon. {#is-genre-disappearing-everywhere} # Is genre disappearing everywhere? So is this happening everywhere in culture? The degree to which its an algorithmic product isn't clear, but it sure seems like the streaming services have settled into a bubble of half-hour unclassifiable formats rather than "sitcoms" and "dramas." Netflix's "[personalized genres](https://www.theatlantic.com/technology/archive/2014/01/how-netflix-reverse-engineered-hollywood/282679/)" are not the product of an embedding system, but do play naturally alongside one, because they generate an affinity for works that cut across different realms. The causation here is complicated because, as with many other trends, the technology merely gloms onto a larger cultural trend. It seems quite possible to me that music recommendation services succeed right now because the zeitgeist is aligned in a way where many people are amenable to being served these kinds of hybrid works. If you want pure genre, there are ways of getting it; modern satellite radio stations, like Sirius-XM, will give you all the expert-curated music you want for any microgenre imaginable. Is this a problem? I don't think many would see it as such; but it's worth thinking, nonetheless, about what it does to culture. Anyone who manages to occupy an empty space in the cultural manifolds will be richly rewarded; anyone who tries to stay in a heavily-trafficked space will languish. The idea of cultural areas as fluid, non-differentiable groups flowing into each other will be a self-fulfilling prophecy; anyone insisting that genres are real may seem hopelessly old fashioned. Anyone who navigates cultural spaces through digital means will be over-exposed to hybrid cultural forms, which will only lead them further to think that the different genres were an old-fashioned illusion, brought about by a particular set of constraints around channels, record labels, and the rest. And of course they'll be right. But if they think there's anything more natural about an enforced space emphasizing novelty, sparsity, and so forth, they'll be wrong; and a cultural dynamic around filling in the valleys of a manifold spaces rather than building up the summits may be less rewarding than we hope. ]]> bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2021-05-20-alignment https://benschmidt.org/post/2021-05-20-alignment Thu, 20 May 2021 15:26:37 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2021-04-28-duckworm https://benschmidt.org/post/2021-04-28-duckworm Wed, 28 Apr 2021 13:57:51 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2020-01-15-webGPU https://benschmidt.org/post/2020-01-15-webGPU Mon, 08 Mar 2021 14:52:01 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/bookworm-caching https://benschmidt.org/post/bookworm-caching Sun, 07 Mar 2021 16:18:47 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2020-11-12-jobs https://benschmidt.org/post/2020-11-12-jobs Thu, 12 Nov 2020 17:00:45 GMT Region Average Jobs listed by November 12 as share of 2010s average as share of 2000s average era 2004-2008 2009 2010-2019 2020 Middle East 61.20 29.0 21.9 1.0 5% 2% World 37.80 13.0 14.5 1.0 7% 3% Europe 88.40 36.0 46.1 5.0 11% 6% US/Canada 162.80 79.0 68.2 14.0 21% 9% Ancient 28.20 16.0 17.0 3.0 18% 11% Americas 52.40 26.0 30.0 7.0 23% 13% Asia 80.60 43.0 54.7 14.0 26% 17% Methodological 25.20 12.0 30.4 8.0 26% 32% Africa 13.00 4.0 20.8 5.0 24% 38% Hist. Sci. 12.40 7.0 8.4 5.0 60% 40% Interdisciplinary 15.20 9.0 25.4 10.0 39% 66% Black/Af-Am 33.75 20.0 25.0 32.0 128% 95% ``` ]]> bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2020-10-01-jobs-update https://benschmidt.org/post/2020-10-01-jobs-update Thu, 01 Oct 2020 13:29:09 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2020-08-31-circle-packing https://benschmidt.org/post/2020-08-31-circle-packing Tue, 01 Sep 2020 20:49:49 GMT \}\} ]]> bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2020-08-25-college-majors-2019-update https://benschmidt.org/post/2020-08-25-college-majors-2019-update Fri, 28 Aug 2020 00:00:00 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2020-07-30-CS-Rankings https://benschmidt.org/post/2020-07-30-CS-Rankings Tue, 28 Jul 2020 15:15:14 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2020-02-25-Jeb https://benschmidt.org/post/2020-02-25-Jeb Wed, 26 Feb 2020 14:49:26 GMT \}\} It is odd to me to be able to talk in this particular way about the intersection of daily life and religious identity. One oddity, of course, is that this is yet another example of the kinds of information held inside the great data surplus at the tech companies; but honestly, the question here is so oddly stated that I can't imagine datamining ever turning it up. Perhaps it says something about the potential for biographies in the digital age; the narcissism of the quantified self movement might look quite different directed at the quantified other. But is this kind of evidence really compatible with biography? Anyhow, off to some e-mails of my own. ]]> bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2019-AHA https://benschmidt.org/post/2019-AHA Thu, 05 Dec 2019 00:00:00 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/moving https://benschmidt.org/post/moving Sun, 30 Jun 2019 15:53:19 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/Moving https://benschmidt.org/post/Moving Fri, 03 May 2019 19:18:12 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2019-03-18-Nan-Da-Critical-Inquiry https://benschmidt.org/post/2019-03-18-Nan-Da-Critical-Inquiry Tue, 19 Mar 2019 01:42:00 GMT Underwood should train his model on pre-1941 detective fiction (A) as > compared to pre-1941 random stew and post-1941 detective fiction (B) > as compared to post-1941 random stew, instead of one random stew for > both, to rule out the possibility that the difference between A and B > is not broadly descriptive of a larger trend (since all literature > might be changed after 1941). _and_ that Underwood uses methods that could never find differences between genres. It is true that Underwood does use methods inadequate to prove there is no difference in detective fiction pre and post-1930. (Her use of the year "1941" is a mistake--it seems to stem from confusing the date of one of Underwood's sources with the year he chose for a testing cutoff). This is an absurdly high bar--of course _something_ changed, if only the existence of words like 'television' and 'databases.' Underwood says as much. The actual article is caught up in a more interesting discussion of the _comparative_ stability of genres. The core argument is not, as Da says, that genres have been "more or less consistent from the 1820s to the present," but that detective fiction, the gothic, and science fiction--*specifically*-- show different patterns, with detective fiction being a far more coherent pattern than the gothic novel. By focusing only detective work, she's missing the entire argument of the article. That this doesn't merit correction or retraction is depressing. I don't know what Underwood used to train. But if he did allow the 'random stew' to contain both pre- and post-1930 work that would make the performance of his model _more_ remarkable, not less--it would indicate that it was correctly tagging Elmore Leonard (say) novels as detectives even though they use words like "fax" or "polaroid" it had previously seen in the post-1930 set. Where Da's method really shines, though, is in the random statistical vocabulary she brings to bear. All that Underwood has shown in using word frequency homogeneity to differentiate detective fiction from random fiction is that the difference between pre- and post-1941 detective fiction is not as significant as its difference from random fiction. This does not mean that the same method can capture the difference between different types of detective fiction. After all, statistics automatically assumes that 95 percent of the time there is no difference and that only 5 percent of the time there is a difference. That is what it means to look for p-value less than 0.05. Think of it this way: if everyone can agree that something is changing---even Underwood concedes that genres evolve---but you have devised one way that concludes that it does not, it does not necessarily mean that you have found something. In the first specific critique, the article talks about 95% p-values in the following way: "statistics automatically assumes that 95 percent of the time there is no difference and that only 5 percent of the time there is a difference. That is what it means to look for p-value less than 0.05." To look for a p-value under 0.05 is to look for a pattern that would only occur 5% of the time as a result of random variation. It's not a great threshold. But Underwood's paper does not rely on them. So let's take a look at how well the statistical claims here hold up: is the debunking useful? {#jockers-and-kiriloff-on-gender} ## Jockers and Kiriloff on Gender Da's critique of Underwood relies mostly on failures of reading. The next section, on work by Matthew Jockers and Gabi Kirilloff, showcases the way her piece rests rhetorically on the innumeracy of her readership. Her critique of Jockers and Kiriloff is, as she says, that they "present a statistical no-result finding as a finding." In order to do so, she swamps her readers with a blizzard of statistical language that she can justifiably assume will sound plausible to the readers of _Critical Inquiry_. Her promise is that she will offer "a clear explanation of the computational work that CLS actually does" (605). In her two paragraphs on Jockers and Kiriloff, she tosses out the following observations: - "Let us say that you are measuring the overlap of features between two sets of data using a standard 5 percent confidence level; out of n possible shared features, 0.05n will automatically be significant." - "In good statistical work, the burden to show difference within naturally occurring differences ('diff in diff') is extremely high." - "This paper does not perform a bootstrap, which means the literary-historical suggestions that follow this genre classification do not stand." - "Practitioners have to apply the Bonferroni Correction to conventional statistical thresholds of significance used for data mining." And so on. This blizzard of terminology establishes for the innumerate reader that they finally have an expert who will debunk statistics for them, while freeing them of the burdensome requirement to think for themselves. But much of this is word salad; what stands is unimportant. The claim that 5% of features "will automatically be significant" seems to approach the claim that she has already had to retract: that "statistics automatically assumes that 95 percent of the time there is no difference and that only 5 percent of the time there is a difference. That is what it means to look for p-value less than 0.05." 'Diff in diff' is indeed an important tool, but it's not about whether testing whether two distributions are different from each other; it's about testing whether a post-treatment experimental group (like recipients of experimental chemotherapy, or counties that received Gates foundation grants) saw a significant [time series change.](https://www.mailman.columbia.edu/research/population-health-methods/difference-difference-estimation) Bootstrap resampling to generate confidence intervals can be useful, but to randomly invoke it, as here, is about as sophisticated as demanding that every article, regardless of content, take a transnational approach. To say that significance testing should apply the Bonferroni correction is _not_ nonsense. But neither is it something that Da does. As with her discussion of Underwood, the exercise relies on coming up with a straw man description of the claim of the article, and then rejecting that. Da focuses mostly on the question of whether there are statistically significant differences in gendered use of verbs. Jockers and Kiriloff use the method of nearest shrunken centroids as input into their model for a variety of reasons having to do with model interpretability. [^2] But Jockers and Kiriloff's findings _are_ significant at the level that Da suggests, and it is Da's work that is truly sloppy. In the appendix, she publishes a comparison that obviously mislabels its bins (it claims that her replication found the "she killed" and "he wept" are gender stereotypes, rather than the opposite). If the goal is simply to find which words show strong gendered patterns of usage, it's unclear why she would choose a different statistical method. In the appendix, claims to have performed a replication and found that "Overall, the percentage differences between these top most correlated verbs for each gender was very low (0.031% to 0.307%) meaning that while a difference can be found, male/female is not very differentiated from one another if we look at verbs." I have no idea what statistics she is reporting here--although she has a github repository online, it appears not to contain any of the code used to generate these tables. [^3] But while Da's method is obscure, I am confident that the interpretation any reader would take from this-- that Jockers and Kiriloff report statistically inflated claims of difference-- is incorrect. A simple way to test the robustness here is just to apply a Dunning Log-Likelihood test, and use a close analogue to the Bonferroni correction Da calls for and then never runs, a Holm-Sidak correction. [^4] The result: 81% of the words Jockers and Kiriloff look at are statistically significant. After spending a paragraph and a half throwing out statistical claims that My intellectual disagreement, here, is with the {#piper-on-confessions} ## Piper on Confessions This same slapdash method--mis-stating the statistical or computational literature, failing to run the very tests she insists are necessary, and then leaving the reader with the impression she has somehow invalidated the result--is on prime display in her description of Andrew Piper's work on the confessional form. Da pulls statistical pronouncements out of thin air and presents them as that which _must_ be done. These claims are often either misinformed or misleading. I can't bear to go through all her sections. But as an example, take the analysis of Andrew Piper's work on Augustine's _Confessions_. In a few paragraph, she makes as many mistakes as she holds him to account for in a full article. First, she criticizes Piper for performing Principal Components Analysis on unscaled word frequencies, and produces scatterplots that show dramatically different results from his: "The way to properly scale this type of matrix is outlined in G. Casella et al's Introduction to Statistical Learning... The second step \[Z-scaling\] is necessary if each word is to be seen as a feature for PCA." George Casella did not write a book called _Introduction to Statistical Learning_; she means the 2013 volume by Gareth James et al. published (after Casella's death) in a Springer series for which he was general editor. The chapter she cites certainly does not say that PCA matrices must always be scaled by standard deviations. It says, rather that scaling PCA is a consideration the researcher should take. When units are arbitrary, PCA should be scaled--if comparing SAT scores to grade point averages, you don't want the difference between a 1420 and a 1421 on the test to be the same as a 2.5 and a 3.5 on the GPA. But word frequencies are not arbitrary. In those cases, the researcher must decide. To quote from the text: "In certain settings, however, the variables may be measured in the same units. In this case, we might not wish to scale the variables to have standard deviation one before performing PCA."@james\_introduction\_2013. This is a central challenge familiar to anyone who has tried to grapple with wordcounts. There are so many uncommon words used once or twice in any given text that, when scaling is used, they can completely swamp the repeated words. A variety of solutions are in common use. TF-IDF scaling drops out the most common words while allowing those of medium frequency to shine through; log transformations of various flavors proliferate. Ideally solutions would not be wholly dependent on the parameter space, but the phrasing of the question matters. Da sidesteps these all these complications for her reqaders by implying the real difference has to do with a _philological_ failing, that Piper doesn't stem Latin text. This is something a literary audience can understand, and gestures towards a humanistic critique. But comically, her version reproduces many of the same philological failings. She implies that Piper didn't use a Latin stemming algorithm because the "only Latin stemmer available is the Schinke stemmer," but that she _has_ taken the effort. This is incorrect on both fronts. First, there are many Latin stemmers available. (For an in-depth analysis of at least 6, see [Patrick Burn's work](https://github.com/diyclassics/lemmatizer-experiments/tree/master/notebooks). And her effort seems to be scattershot at best. It's hard to tell what code Da actually ran--the [online appendices](https://github.com/nan-da/Novel-Devotions) for analyzing Piper's case only include the PCA code for Chinese, not the figures included in the appendix. (Ordinarily I would be forgiving of this kind of lapse, which is all too common; perhaps the inadequate code appendices are intended as a higher-order critique of computational work. But her failings vis-a-vis replication are far greater than those of, say, Ted Underwood, who generally supplies a single script called `replicate.py` that you can run yourself inside any of his projects.) Still, from what she has posted [online](https://github.com/nan-da/Novel-Devotions), Da appears to have re-implemented Schinke's algorithm in both R and python, with separate rules for nouns and verbs. But then, in her Cross Distance code, she simply applies the noun stemming rules to all words, (probably) because choosing a part of speech is much harder than running stemming. This results in many problems; both because some verbs are not stemmed at all ('resurrexit' remains 'resurrexit' even though the verb rules would have it as 'resurrexi'); and because the rules are applied to function words as well with silent NULL results in her code, so that words like 'que,' 'cum,' 'te,' and 'me' are deleted from the text altogether. That is: many function words are being dropped altogether because a new implementation was hastily coded rather than using one of the more mature implementations available. I wrote this and then, quickly, checked what difference it all makes. (Code and edits online [here](https://github.com/bmschmidt/Novel-Devotions/blob/master/Checking.Rmd)) I was, honestly, expecting that the scaling factor would be significant and account for the differences in texts. But actually, what I got looks more or less like Piper's original. Reproduction of Piper's original: :::  {.caption} ::: A reproduction of the original ::: ::: Reproduction using Da's scaling. :::  {.caption} ::: A reproduction of a reproduction ::: ::: Or maybe it looks completely unlike it! {#and-so-on} ## And so on. I could go on. The debunking of topic model, for example, uses not the well established literature about comparing topic model distributions to each other, but some arbitrarily chosen robustness tests. (It drops 1% of documents). But it is not a replication. Topic models rely on extremely specific assumptions about the distribution of words in texts based on word counts; they attempt to reproduce the frequencies in actual documents. But rather than fit on word counts, the model, for no apparent reason, [uses TF-IDF vectors](https://github.com/nan-da/Quiet-Transformations/blob/master/lda_fitting.py#L60-L64) that multiply the significance of rare words and decrease the significance of common ones. I have never seen a TF-IDF vectorization fed into an LDA feature set before-- it's an extremely odd choice that guarantees the results will be different from Underwood and Goldstone's, and partially explains the incoherent topics in the appendix, such as `doulce attractiveness unsatisfying gence dater following mecum wigan cio milieu`. (Edit 03-20) I'm wrong about this: Andrew Goldstone points out that there's an argument to the TFIDF vectorizer in her codes that makes it output raw frequencies. Frequencies might still produce results different than the counts that Underwood and Goldstone used, but this is not a howler. It's still unreasonable, though, to expect that the topics put out by the Variational Bayes, online LDA implementation in scikit-learn will be the same as those in the Gibbs-Sampling method Underwood and Golstone use from Mallet. Different methods can produce dramatically different results when the hyperparameters are not properly tuned. ([See here](https://www.cs.mcgill.ca/~uai2009/papers/UAI2009_0243_1a80458f5db72411c0c1e392f7dbbc48.pdf)) While [Goldstone does](https://github.com/agoldst/dfrtopics/blob/master/R/model.R#L282-L2840) optimize hyperparameters there's nothing in the scikit-learn code that indicates this effort. So the models may be radically different because Underwood and Goldstone ran a better model. In fact, Goldstone and Underwood's [original work on this](http://journalofdigitalhumanities.org/2-1/what-can-topic-models-of-pmla-teach-us-by-ted-underwood-and-andrew-goldstone/) dealt with this issue very clearly: > On the other hand, to say that two models "look substantially > different" isn't to say that they're incompatible. A jigsaw puzzle cut > into 100 pieces looks different from one with 150 pieces. If you > examine them piece by piece, no two pieces are the same --- but once > you put them together you're looking at the same picture. This comparison obviously mislabels its bins (it claims that her replication found the "she killed" and "he wept" are gender stereotypes, rather than the opposite) and makes some extremely fishy claims such as "Overall, the percentage differences between these top most correlated verbs for each gender was very low (0.031% to 0.307%) meaning that while a difference can be found, male/female is not very differentiated from one another if we look at verbs." I don't know what that range is supposed to be, but at least for 'wept', [Google Ngrams gives the difference in gender usage as 400%](https://books.google.com/ngrams/graph?content=%28she+cried%2Fshe+_VERB_%29%2F%28he+cried%2Fhe+_VERB_%29&year_start=1800&year_end=2000&corpus=15&smoothing=3&share=&direct_url=t1%3B%2C%28she%20cried%20/%20she%20_VERB_%29%20/%20%28he%20cried%20/%20he%20_VERB_%29%3B%2Cc0#t1%3B%2C(she%20cried%20%2F%20she%20_VERB_)%20%2F%20(he%20cried%20%2F%20he%20{_VERB_})%3B%2Cc0) But to go through all of this is a pain. I'm sure others have written other analyses. This work is tedious, which is the reason that it's rarely done; and it's hard to reproduce another workflow even when it's well-documented. [^1]: For the record, I myself made a quick check using yet another measure of entropy, compressibility; I'm inclined to think Da is right that there is a fundamental error Stanford's bigram calculations. [^2]: Nearest shrunken centroids is indeed a sort of idiosyncratic choice, but one that Jockers seems to be extremely partial to going back over a decade. @jockers\_comparative\_2008. Whether digital humanists should be free to roam across the disciplines in search of obscure but useful algorithms, or should remain in a tightly constrained space, is a difficult one. My stance-- [^3]: I base this partly because the appendix says it uses the "SpaCy" packages for results, but none of her online code imports that package. [^4]: I am not a statistician, but I use the Sidak correction because the literature seems to say it's [superior to](http://www.cogsci.ucsd.edu/~dgroppe/STATZ/Abdi-Bonferroni2007-pretty.pdf) the Bonferroni. I use the Holm modification, which applies increasingly stringent standards as you descend a list, because of an issue Da doesn't ever address, type II errors; that it is as incorrect to report a false negative as a false positive. I order the Holm method is by word frequency, not p-value (suggested in some online literature) to make the test more conservative; since Jockers and Kiriloff use the 310 most common words, there's no need to worry about multiple comparisons outside this range. ]]> bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2018-12-03-AHA-Report https://benschmidt.org/post/2018-12-03-AHA-Report Mon, 03 Dec 2018 17:16:22 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/scatterzoom https://benschmidt.org/post/scatterzoom Tue, 30 Oct 2018 13:37:26 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/SRP https://benschmidt.org/post/SRP Mon, 22 Oct 2018 14:30:33 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/hugo https://benschmidt.org/post/hugo Sun, 21 Oct 2018 14:30:33 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/Atlantic_humanities https://benschmidt.org/post/Atlantic_humanities Fri, 31 Aug 2018 17:37:26 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/Sapping_attention_humanities https://benschmidt.org/post/Sapping_attention_humanities Mon, 30 Jul 2018 17:37:26 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2016-03-19-feature-reduction-on-the-underwood-sellars-corpus https://benschmidt.org/post/2016-03-19-feature-reduction-on-the-underwood-sellars-corpus Sat, 19 Mar 2016 15:17:35 GMT Researchers should be very cautious about moving from good > classification performance to interpreting lists of highly-weighted > words. I've seen quite a bit of this going around, but it seems to me > that it's very easy to lose sight of how many sources of variability > there are in those lists. Literary scholars love getting a lot from > details, but statistical models are designed to get the overall > picture right, usually by averaging away the variability in the > detail. I'm sure that Goldstone is being sage here. Unfortunately for me, he hits on this wisdom \_before \_using the lasso instead of ridge regression to greatly reduce the size of the feature set (down to 219 features at 77% success rate, if I'm reading his console output correctly), so I don't get to see what features a smaller model selects. [Scott Enderle took up Goldstone's challenge, explained the difference between ridge regression and lasso in an elegant way, and actually improved on U&S's classification accuracy with 400 tokens](http://www.lagado.name/blog/to-conquer-all-mysteries-by-rule-and-line/)--an eightfold reduction in size. So I'm left wondering whether there's a better route through this mess. For me, the real appeal of feature selection on words would be that it might create models which are intuitively apprehensible for English professors. But if Goldstone is right that this shouldn't be the goal, I'm unclear why the best classification technique would use words as features at all. So I have two questions for Goldstone, Enderle, and anyone else interested in this topic: 1. Is there any redeeming interpretability to the features included in unigram model? Or is Goldstone right that we shouldn't do this? 2. If we don't want model interpretability, why use tokens as features at all? In particular, wouldn't the highest classification accuracy be found by using dimensionality reduction techniques across the \*entire\* set of tokens in the corpus? I've been using the U&S corpus to test a dimensionality reduction technique I'm currently writing up. It works about as well as U&S's features for classification, even though it does nothing to solve the collinearity problems that Goldstone describes in his post. A good feature reduction technique for documents, like latent semantic indexing or independent components analysis, should be able to do much better, I'd think--I would guess the classification accuracy over 80% with under a thousand dimensions. Shouldn't this be the right way to handle this? Does anyone want take a stab at it? This would be nice to have as a baseline for these sorts of abstract feature-based classification tasks. ]]> bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2015-06-12-buying-a-computer-for-digital-humanities-work https://benschmidt.org/post/2015-06-12-buying-a-computer-for-digital-humanities-work Fri, 12 Jun 2015 17:49:01 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2015-04-03-commodius-vici-of-recirculation-the-real-problem-with-syuzhet https://benschmidt.org/post/2015-04-03-commodius-vici-of-recirculation-the-real-problem-with-syuzhet Fri, 03 Apr 2015 16:37:21 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2015-02-06-rate-my-professor https://benschmidt.org/post/2015-02-06-rate-my-professor Fri, 06 Feb 2015 21:47:21 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2014-12-12-the-bookworm-mallet-extension https://benschmidt.org/post/2014-12-12-the-bookworm-mallet-extension Fri, 12 Dec 2014 04:15:07 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2014-11-07-building-outlines-for-markdown-documents-with-pandoc https://benschmidt.org/post/2014-11-07-building-outlines-for-markdown-documents-with-pandoc Fri, 07 Nov 2014 20:17:45 GMT > !\[Edison electric > > light\](http://scienceblogs.com/retrospectacle/wp-content/blogs.dir/463/files/2012/04/i-3530f86be619cdc7d42c13cdca188088-edison.bmp) 3. As in the previous example, the *image format is extended* so that labels in slides appear not as alt-text, but in the text above the image: in addition, any image link beginning with the character "\>” is treated not as an image but as an iframe, making it easy to embed things like youtube videos or interactive Bookworm charts. The slide decks are built with reveal.js, which drops everything into a nicely organized batch. [Here's what one looks like](http://benschmidt.org/HIST1234/slides/11-05_Systems,_Electricity_and_Household_Labor.html#/the-system-builders).  (This is for a lecture on household technologies in the 20s). My favorite feature is that by hitting escape, you get an overall view of everything in the lecture sorted by header--this is particularly useful when studying for exams, because those headers align exactly with the outlines.  The outlines are produced from the same lecture notes, but in a different way; rather than pull the code blocks, they walk through all the headers in the document and append them (and any bolded terms) to a new document that students can see. For that lecture, it looks like this:    There are a few things I still don't love about this: image positioning and sizing is not so good as it is in powerpoint. But the thing that's nice is that it's extremely portable; if I don't make through the end of a lecture, I can just cut out the last few paragraphs, paste them into the next day's document, and have the outline and slides immediately reflect the switch for both days. This makes a lot of last-minute, before-class changes dramatically easier. The basic scripts, though not the full course management repo, is [up on github](https://github.com/bmschmidt/MarkdownLectures).The code is in Haskell, which I've never written in before, so I'd love a second set of eyes on it.  Some brief reflections on coding for pandoc in Python and Haskell follow. I thought it would be easy to switch between headers and an outline, but they turn out to have almost nothing in common in the Pandoc type definition; the outline needs to be built up recursively out of component parts. It's an operation that's much closer to really basic data structures than anything I've done before. I initially used the pandocfilters Python package for this. That code is [here](https://gist.github.com/bmschmidt/2a5beff9ed59c1cc337b#file-lecturetooutline-py). It basically works--thanks primarily to insight gleaned from an exchange on GitHub between, I think, Caleb McDaniel and John McFarlane that I've lost the link for) that you need to scope a global python variable and append to it from a \`walk\` function. But it has a tendency to break unexpectedly, and uses an incredibly confusing welter of accessors into the rather ugly pandoc json format. Plus, it's fundamentally an attempt to write Haskell-esque code in Python, which is about the least pleasant thing I've ever seen. By the time I made that python script work. I had spent a couple hours reading and re-reading the [pandoc types de](http://hackage.haskell.org/package/pandoc-types)finition, and it seemed like it would simpler to just write the filter in Haskell directly. (I did a few Haskell problem sets for a U Penn course this summer out of curiosity; without that basic understanding of Haskell data types, I doubt I would have been able to understand the Pandoc documentation.) The [lecture-to-outline Haskell code](https://github.com/bmschmidt/MarkdownLectures/blob/master/lectureToOutline.hs), to my surprise, ended up being a bit longer than the Python version (though much of that is type definitions and comments, which doesn't really count). If anyone out there who knows Haskell can explain to me a better way to avoid some of the stranger elements in there (particularly the reversing and unreversing of lists just to allow pattern matching on them, which is a substantial proportion of what I wrote), I'm all ears. Programming in Haskell is certainly more interesting than python; I agree with [Andrew Goldstone's comment that "whereas programming normally feels like playing with Legos, programming in Haskell feels more like trying to do a math problem set, with ghc in the role of problem-set grader”](http://andrewgoldstone.com/blog/2013/04/21/more-on-pandoc/#fn1). I'm left with a strong temptation to write a TEI-to-Bookworm parser, which I've previously sketched in Python, in Haskell instead; both for performance and readability reasons, I think it might work quite well. Stay tuned.   ]]> bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2014-10-23-more-thoughts-on-topic-models-and-tokens https://benschmidt.org/post/2014-10-23-more-thoughts-on-topic-models-and-tokens Thu, 23 Oct 2014 18:37:43 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2014-09-23-building-topic-models-into-bookworm-searches https://benschmidt.org/post/2014-09-23-building-topic-models-into-bookworm-searches Tue, 23 Sep 2014 22:29:38 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2014-09-11-simpsons-2 https://benschmidt.org/post/2014-09-11-simpsons-2 Thu, 11 Sep 2014 21:59:21 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2014-09-05-markdown-historical-writing-and-killer-apps https://benschmidt.org/post/2014-09-05-markdown-historical-writing-and-killer-apps Fri, 05 Sep 2014 20:34:25 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2014-08-29-the-simpsons-bookworm https://benschmidt.org/post/2014-08-29-the-simpsons-bookworm Fri, 29 Aug 2014 19:38:24 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2014-06-05-optimally-ordering-geographical-entities-in-linear-space https://benschmidt.org/post/2014-06-05-optimally-ordering-geographical-entities-in-linear-space Thu, 05 Jun 2014 14:36:07 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2014-03-27-bleg-1-string-distance https://benschmidt.org/post/2014-03-27-bleg-1-string-distance Thu, 27 Mar 2014 21:44:24 GMT bmschmidt@gmail.com (Ben Schmidt) https://benschmidt.org/post/2023-10-31-AI-Copyright https://benschmidt.org/post/2023-10-31-AI-Copyright Thu, 01 Jan 1970 00:00:00 GMT bmschmidt@gmail.com (Ben Schmidt)