Karen&Kevin about the media archaeology of touchscreens (based on a German untranslated chapter!)!)

Welcome to the Deep Dive. Today we're looking at screens. You know, the ones everywhere your phone, your tablet, your monitor, we're gonna dove deep into their history, which is, well, surprisingly complex. - Yeah, it really is. We're drawing on an article by Shintaro Miyazaki, a media scholar. He actually wrote it in German for a collection called, "Surfaces and Interfaces, "Estetics and Politics of Film Images." - Right, and we're sort of unpacking his ideas here in English for you, exploring how the actual screen surface, the thing you touch, has changed. - Mm-hmm. And the core idea is this shift he talks about. How screens went from being like a clear window you just looked through. - Right, like old TVs or cinema screens? - Exactly. To being more like a physical glass plate, you know, a surface you actually interact with. Touch screens are the obvious example. - So he wants to look at the filmish, the cinematic stuff, but see how it fits into these new tactile surfaces we use all the time. - Yeah, he's arguing. - He just disappear with digital tech. It sort of reappears, but in a different way on these surfaces. - Okay, so this sounds like it's gonna take us under the hood a bit, looking at the history and the actual materials. - Absolutely, let's get into it. - So first up, this idea that modern screens feel different. They're interactive, obviously touch sensitive, and they feel more solid, opaque sometimes, not just a clear pain. - Right, and that very change in how the screen feels and works makes it in Miyazaki's view and a epistemological thing. Basically something really crucial for media and film studies to grapple with. - Because it changes how we understand images, how we interact. - Precisely. If the surface is different, our relationship towards on it changes too. It kind of reminds me of, you know, avant-garde filmmakers. - Oh yeah, how they like scratch the film or show the projector. - Exactly. They highlighted the materiality of the whole setup. Miyazaki's saying we need to do something similar with modern screens. To understand the aesthetics, the politics, you can't just look at the image or the software. - You need the tech side too, the sociotechnical, sociopolitical, and the actual stuff it's made of. - Yes, the median technological material conditions as he puts it. You need the whole picture. - Okay, this next bit is really interesting. He talks about how the old signs of film's materiality, like the scratches you mentioned, they've kind of moved into the symbolic world, right? Like software glitches or digital artifacts. Right, they become code or data issues, but here's the paradox, he points out, that film-ish, the cinematic aspect, it doesn't actually vanish, it comes back. But on the surface of these new screens, these optoelectronically expanded screens. - So it didn't disappear, it just moved house. - Kind of, because the screen isn't just glass anymore, it's this incredibly complex integrated system. He calls them media agents, mits. - What goes into that? - Oh, layers upon layers, liquid crystals, metal oxides, plastics, semiconductors, thin films, optoelectronics, all orchestrated by, well, techno-mathematics, algorithms, basically. - Wow, it really is a whole ecosystem in there. So how do we unpack this? The article suggests looking at two key histories. - Yeah, two media archeological samples, he calls them. We'll dig into the liquid crystal display, the LCD and the capacitive touchscreen. - Okay, let's start with the LCD. Where does that story begin? It goes way back, doesn't it? - Surprisingly far, back to 1888, actually. It starts with scientists getting interested in these weird things called liquid crystals. - And the carrot juice connection, that was memorable. - It really is, a botanist, Friedrich Reignitzer in Prague. He was experimenting with something from carrot juice cholesterol benzoid, if you want the technical term. - Okay. - And he noticed it melted strangely. It didn't just go solid to liquid. It hit one temperature, 145.5 Celsius, turned into this cloudy, milky liquid. - Uh-huh. - And then at a higher temperature, one doesn't eat 0.5 Celsius, it finally went clear. That weird optical effect, that milky phase was the key first step. - This in between state, not quite solid, not quite liquid. - Exactly. He wrote about it to Otto Lehman, a physicist, who looked at it under a microscope around 1888, 1889. Lehman was fascinated and came up with the name Flessend Kajada. - Flowing crystals, makes sense. - It does. Then jump ahead a bit to 1911. A French physicist, Charles Victor Mugwin, finds something called the Twisted Namedic effect. Basically how these crystals could twist light polarization. - Listing light, okay. - And then the really crucial bit for displays. In the 1920s, a Russian physicist, Visevlad Friedrichs, figured out you could control that twisting effect, not just with magnets, but with electric voltage. - Ah, so now you can control light passing through just by applying electricity. - Bingo, that's the core principle of an electrically controlled light filter, the basis for an LCD. - Amazing that the core idea was there so early, but it took ages to become a real display, right? - Oh, decades, like 70 years after Ron Itzer. Things picked up again around 1962. When a British chemist, George William Gray, published a really important book, pulling together everything known about liquid crystals then. - And the 60s seemed pivotal for both LCDs and touch screens. - Definitely a key decade. So 1963, Richard Williams at RCA, you know, the big TV company back then. - Right. - He did important work figuring out the optoelectronic properties needed. The specific material conditions to make liquid crystals work for displays. - RCA was already thinking beyond the big tube TVs. - For sure. And building right on Williams' research and engineer there, George H. Heilmeyer, led the team that made the first actual LCD prototypes. - And then they announced it to the world. - Yep, 1968, big press conference in New York. They pitched it as this new display tech. Low power, super thin, light weight compared to CRTs. That really kicked off research everywhere. - But the first things you could actually buy weren't exactly like monitors, were they? - No, not at all. Took another five years, still 1973. Sharpen Seiko came out with a calculator and a wristwatch using LCDs, but it was super basic, just black segments lighting up to make numbers. - Like on an old digital watch, just blocks. - Exactly. Meanwhile, researchers were pushing for more. Folks like Hellfrich and Shat in Switzerland, Peter Wilde too, working on better materials, higher resolution. - And this is where Active Matrix comes in. - Yes, crucial step. 1972, a team led by T. Peter Brody at Westinghouse in the US. They developed a screen using thin film transistors. Basically tiny switches coated onto a film. That's the Active Matrix idea. - What does that do? - It means each pixel, each tiny dot on the screen gets its own little transistor to control it individually. That allows for much faster updates, clearer images, higher resolution. - Okay, so the materials improved and the way to control them got much better. When did this tech actually show up in things like computer monitors? - The big milestone there is often cited as 1988. That's when Sharp launched a 14 inch color computer monitor. Using this Active Matrix LCD tech, that really marked its arrival as a mainstream flat panel display. - So 100 years from weird, carat-use behavior to a computer screen. Incredible. What about the pixel itself as a modern LCD? How does that work? - Well, since about '88, the basic setup is, each pixel is made of three tiny cells, red, green and blue, on a thin transparent film. In those thin film transistors we mentioned, they control how much light gets through each little color cell by applying tiny rhythmic electrical pulses. - And the light comes from behind it. - Exactly. There's a backlight layer, usually LEDs nowadays. The liquid crystals themselves don't make light. They act like tiny shutters or filters for that backlight. - Got it. So the transistors pulse, the crystals twist or untwist, letting specific amounts of red, green, blue light through. And the mix of those three in each pixel gives you the color and brightness at that point. And then you coordinate all the pixels, changing them rapidly over time, these macro configurations, and that's how you get moving images. - It's like a tiny, super fast, coordinated light show. - That's a good way to think about it? All driven by precise electrical signals. - Okay, so that's light control. Let's switch gears to interaction, the touchscreen. This story also starts earlier than you might think. - It does, maybe even earlier, in a way. Back to 1920, Petersburg. Leon Therman demonstrates his aquatic synthesizer. - Therman, played without touching it. How does that connect to touch screens? Seems like the opposite. - It seems counterintuitive, yeah. But the principle is the key. The Therman works because moving your hand near its antennas changes the electrical capacitance of the circuit. - Capacitance, like storing an electric charge? - Sort of, yeah. Your body has a small electrical charge. When you bring your hand near the antenna, which is part of an oscillating circuit, you change the electromagnetic field around it, and that changes the circuit's capacitance. - Okay. - In the Therman, he used two high frequency circuits. Changing the capacitance of one changed its frequency. The difference or beat between the two frequencies created the audible sound. So the Therman is essentially a very sensitive capacitance detector. - Right, it detects the presence and distance of your hand via capacitance changes. I see the link now. - Exactly. - Now, fast-forward to 1965, a guy named Eric Arthur Johnson at the Royal Radar Establishment in the UK, he developed something explicitly called a touch display. - For what? - For air traffic control using computers. So notice the connection here. Radar and aviation, not the TV industry like LCDs. Different needs driving the tech. - Interesting. How did his work? - It also used capacitance. He had thin copper wires embedded in a plastic film laid over the radar screen, the CRT monitor. When you touch two wires at the same time with your finger, it changed the capacitance and the system could detect which pair was touched. - How many points could it detect? - His early one could register 16 touch points, but only one touch at a time, sequentially. - Still pretty useful for selecting things on a radar screen, I imagine. - Definitely. Then in the 70s, you saw other types, like resistive touch screens, pop up. But the article brings us back to capacitance and surprisingly, to CERN. CERN again, the physics lab. Yeah, another crossover. Around 1973, Bitstump and Frank Beckett CERN started improving on Johnson's design. They figured out better ways to print transparent electrodes directly onto glass, making it clear, and they improved the electronics for better stability. Making it more practical. And crucially, they managed to miniaturize the control electronics onto an integrated circuit. That was key. Their first 16-point touch screen went into the control room for the super proton secret-tron accelerator in 76. Wow! Controlling particle beams with touch. And by 77, they'd upgraded it to a 25 by 25 grid, 625 touch points. Much finer control. Okay, so capacitance tech is developing. What about multi-touch? Touching with more than one finger. Good question. While HP had an infrared system in '83, the big breakthrough for capacitive multi-touch came from research at the University of Toronto, also around 1983. Lee, Bucston, and Smith built a prototype. What was different about theirs? It was a big pad, maybe 40 by 30 centimeters, covered in a dense grid 64 by 32, so over 2000 electrodes. It wasn't transparent, set next to the keyboard. But the key was they could address each electrode individually. Why did that matter so much? Because they meant the system could detect and track multiple touch points at the same time. That's multi-touch. Johnson's needed two points for one touch. This could see many separate touches. But scanning over 2000 points must have been slow back then. It would have been. Yes. So they developed the clever software trick, a recursive area subdivision algorithm. Basically, a fast way to search the grid and pinpoint only the areas being touched, ignoring the rest. Smart. Using software to overcome hardware limits. Exactly. Now, jump forward again. We see things like the IBM Simon phone in 93 with a basic, resistive touch screen, one touch point pressure based. But the line leading to our modern screens really goes through Wayne Westerman. Okay, Westerman. Did PhD research filed a key patent in 1998 and started a company called Fingerworks? Hurt of them. Vagely. Didn't Apple buy them? They did. In 2006, Westerman built on the Toronto work, but made key improvements. He tweaked the electrode layout using parallelograms, not just a simple grid, which helped sense gestures better. And did he still need that complex search algorithm? No. And that's crucial. By the late 90s, processors were so much faster. He realized he could just do a brute force scan. Check every single electrode really quickly. No need for the complex software work around anymore. Hard work caught up. Right. And his patent described exactly what we do now. Recognizing single taps, swipes, multi-finger tabs for clicks, multi-finger slides for gestures like pinch to zoom. All the basic grammar of touch interfaces. Yeah. Pretty much. Fingerworks had products like the iGestra Pat around 2003. Then Apple bought them, filed the patents again. And boom, iPhone in 2007, iPad in 2010. Mass market multi-touch arrives. So these two streams controlling light with liquid crystals, detecting touch with capacitance, starting in different places, different industries, eventually merge into the devices we all use. Perfectly put. And look at the structure underneath. It's all about the matrix. The LCD has a matrix of pixels. The touch screen has a matrix of electrodes. An old iPad, maybe 124 by 768 pixels and around 600 touch electrodes. The grid, the matrix, it's fundamental. It really is. It's a technical principle with deep roots. Miyazaki connects it back to the idea of the table, the grid used for organizing data. Going back centuries, like in 13th century astronomy, drive from Arab scholars. It's persistent way to structure information showing up in weaving punch cards, digital memory. Wow. A medieval organizing principle built into our phones. Kind of. Yeah. And another point from Peter Burst, that Miyazaki mentions, digital images, even still ones, are really pure-time images. They're not static like a print. Because they have to be constantly refreshed by the screen. Exactly. They exist in time, their surface effects, created by algorithms, rhythmically pulsing energy through that matrix. The graphics chip tells the transistors, which control the crystals or sense the touch. It's all about rhythmic control. Constant, rapid, rhythmic generation. Right. And this is where Miyazaki brings back the filmish. He says the projector and film reel are sort of metaphorically imploded into the pixel itself. Each pixel becomes a tiny engine creating rhythmic light. And the surface isn't just showing it's listening to. Yes, it's an input surface, a keyboard, a drawing tablet. It captures your touch via algorithmic scanning of that electrode matrix, digitizing your gestures. Some early graphics tablets were even called digitizers. So that physical layering you mentioned earlier is key for both display and touch. Absolutely crucial. These screens are literally made using film technology, ultra-thin printed layers, plastics, liquid crystals, those crucial metal oxides like Indium 10 oxide, ITO, for conductivity and transparency. ITO, right. That's the stuff that makes touch work. It's essential for capacitive touch. Along with glass, insulators, Apple patents show these intricate layers of ITO. Layer upon layer upon layer. And here's that final paradox again. The old aesthetic materiality of film, the scratches, the flicker that mostly vanishes into software. But the material reality of film as these incredibly thin functional layers reappears right in the hardware. The film becomes the screen itself in a way. Yeah. The pixel isn't just light. It's an imploded cinematic device. It embodies time, the refresh, optoelectronics control, and a light filter, the crystals. Controlled by voltage pulses on film layers, not mechanical sprockets. And this whole convergence, you mentioned the 60s being important again. That decade was bubbling with related ideas. You had experimental film exploring pure light and rhythm like Kubelka or Conrad, Namjoon Paik messing with TVs, McCluhen talking about tactility and electronic media. The very concepts of software and algorithm taking shape. And at the same time, those first prototypes at RCA and the role radar establishment we talked about were being built. All these threads were starting to weave together. Leading to, well, this device in my hand. Pretty much. For the old idea of film to disappear into the window glass, for its function to be built into this touchable surface, it takes more than just hardware and software. It takes this incredibly complex, precisely timed electronic signals, rhythms, frequencies, pulses, constantly working within those layers. The hidden life of the screen. You could say that. These images are effects of algorithms driving rhythms in the matrix layers. Those optical rhythms hit our eyes, become bio-electrical signals in our brain. We perceive the image, decide to act. Send motor signals to our fingers, our charge fingers touch the screen, change the capacitance. And the screen detects it. It's a loop. It's a classic machine human feedback loop. Just like severed edicts describe back in the 40s, constantly cycling between algorithmic signals, optical rhythms, biological signals, motor actions, and electrical changes in the circuit. It's not just me using the screen, it's an interaction rhythmic exchange. Exactly. And doing this media archaeology digging into the history and materials, it shows us these underlying principles, these unexpected historical links, these reconfigurations from the margins. And it really emphasizes the physical stuff again. Definitely. The chemistry, the material science, can visitate as the article puts it, the thin film tech semi-conductors, it's all fundamental. It echoes juicy pericose call for an ecological media history. Looking at where the materials actually come from. Yeah, the rocks, the minerals, the geology behind the technology, grounding it all back in the physical world. Wow. Okay. So we've gone from 19th century chemistry labs in early radio tech through radar rooms and physics labs to these layered rhythmic surfaces we touched constantly. And the big takeaway is that the cinematic didn't die with digital. It just transformed. It's embedded in the screen now in its layers, its algorithms, its rhythms. So when you're holding your phone tapping away, it's not just glass, it's this incredibly complex device packed with history, science, and these constant hidden electronic pulses. A material algorithmic system deeply connected to a century of innovation and carrying these echoes of older media principles. So here's a thought to leave you with. The next time you swipe or tap, just think for a second about those invisible layers that history, those rhythmic algorithms, firing just beneath your fingertip, does knowing all that change how you feel about that simple interaction? It definitely makes you think about what's really going on beneath the surface. Absolutely. Well, thank you for joining us on this fascinating deep dive into the sources you've brought. It's been really illuminating.