We Speak Your Language S01 E06

Intro: Hello, welcome to ‘We Speak Your Language’, the podcast for computer language Geeks and nerds. This 6th episode is hosted by Jan Vitek and is brought to you by Raincode Labs. Enjoy.

Jan: It’s a pleasure to have Cliff Click on this podcast ‘We Speak Your Language’ which is a podcast about programming languages by nerds and for nerds. And I’m Jan Vitek and I am the host of this episode, and my guest is Dr. Cliff Click, I’m happy to have Cliff today, it will be a joy to hear from him.

If you need an introduction, Cliff worked for many years on the Java hotspot server virtual machine and the compiler in it. He also worked at Azul, on Java virtual machine with hardware assist for the garbage collection and the transactional memory, Cliff will tell us more. And H2O, which is a distributed machine learning company that that he started a few years back. So, Cliff, I thought we could start with origins.

Cliff: I wrote my first compiler when I was 15, Pascal to P codes from a byte magazine article. By that time, I’d been doing machine code for a while, so I went from Pascal to C on my TRS-80. It was a 2 Pass compiler using a cassette deck between passes, that’s the old school way you did these things on minimal hardware read in 16K.

Jan: Somehow you ended up at Rice working on Fortran. Is that correct?

Cliff: It was for any classic imperative scalar language; we didn’t pick on Fortran per se. I never wrote a Fortran parser. I did write a C parser. We were a back-end, taking an IR pre-LLVM but with a LLVM mindset. We had a classic IR that all grad students used.  We had C parsers that produced it. We weren’t excited about those parts. We took the IR and produced machine code.

Jan: The reason I had the impression it was somewhat linked to Fortran was that Rice was a big Fortran shop.

Cliff: Yes, Ken Kennedy was a main guy in getting Fortran vectorized and later paralysed and had a lot of research dollars which he was kind of free with. I got my grad student ways back then funded from Ken Kennedy, who was known for doing Fortran, but I wasn’t doing Fortran. I was with Dr. Cooper, and he was all about anybody’s scalar compiler as opposed to parallel or vectorizing.

Jan: What was the path that took you from Rice to work on the Hotspot JVM?

Cliff: In my PhD thesis the main part that my advisor signed off is this cool bit of math. The appendix that I couldn’t get a PhD approved on was how to write a fast optimizer and that was the sea of nodes that I was working on – you’re putting semantics on a graph IR and optimising it, and it worked out well, I had a lot of cool stuff, but it wasn’t efficient for PHD from my advisor’s point of view. I moved on and got a job at Motorola doing compilers for embedded systems which was great.

One day, I got a call out of the blue from Sun saying “Hey, I read your thesis, we’re looking for an optimizer, we’re doing this Java thing and we’re going to do run time code generation and probably do code generation on the fly.” Out of the blue I got called to go to Silicon Valley, and I’m a computer nerd-geek all the way, so I had to join Sun in Silicon Valley. When I get there, they had hired a team, it was a small start-up, mostly Europeans doing Smalltalk rewrite called SELF that got rewritten again, into doing a Java back end.

They had a pretty – no offence intended – appalling compiler and they were not happy with how they considered American compilers to look like and I’m not making up these words. And I thought, I can make a compiler go fast, but I can do good code-gen and it’s not pretty, it’s not elegant and it f***ing works! Pardon my French. So, that’s what I did, and I got a lot of push back from a lot of people saying you can’t do this at runtime, it’s too slow and it will never work. I had senior compiler people in and out of Sun, HP Labs, a bunch of places telling me that it will never work, but I knew it was going to work. Where did they get a hold of the appendix to my thesis? I have no idea.

Now not too many years after I got Sun and I was saving the world with Java, Urs Hoelzle called me up and said, “Come and join me at Google, we’re doing this cool thing – internet search” and I thought “Internet search? Who the hell is going to pay for internet search?” Being #50 at Google or staying on as a Sun employee, of course you know Sun did.

Jan: If you if you think back about that thesis appendix on the sea of nodes, was it a good idea or great idea?

Cliff: We’re asking a biased guy for a biased answer, what the hell do you think I’m going to say? Yeah, no. It’s obviously a great idea.

Jan: As it works for everything, is it a panacea or does it have limitations?

Cliff: Does it have limitations? It does what it does very well, so when you say does it do everything well, it doesn’t cure cancer.

Jan: I meant every kind of language that you can imagine.

Cliff: Well, people can imagine some pretty crazy s**t such as Prolog and other things. For an imperative language only with sort of a standard for full execution model, which would include all the Lispy guys and functional programming guys, I think it’s absolutely the right answer. For a Prolog guy, I couldn’t say, I really have no idea. For SQL queries, I doubt it, there may be some variation for SQL, but I don’t know. You’re doing C, you’re doing Lisp, you’re doing Fortran, you’re doing Java obviously, you’re doing Closure, and you’re doing Ruby on Rails and you’re doing Pascal you’re doing ML; I think it’s the right answer.

Jan: I had the impression that the folks at Oracle were using the sea of node IR?

Cliff: So, during my years I kept looking for people using sea nodes and there was one other guy in the same time frame, or maybe a little earlier in Europe somewhere, but he sank later, and I never saw him again. Then there was Oracle Graal VM who obviously promoted that they were doing it well. I want to claim that I got some indication from the V8 guys that there’s something going on there that looks like sea of node, but I don’t know at that point. I’ve worked on V8, I don’t know that they claim sea of nodes publicly or anything. Hopefully for a JavaScript model you would heavily change what your node structures look like, but I believe the graph IR is still the correct answer, but what the actual nodes and edges mean is obviously very tight.

Jan: Changing the topic a little, if you remember and you would think back to those years compiling Java, what was the most annoying part of Java?

Cliff I don’t know. It’s as if you bought into the theory you have to compile somebody else’s language, so essentially, you’re buying all their warts. Are they annoying or are they just a problem you have to deal with? You had a billion null cheque exception, that was pretty elegantly solved. You had range text exceptions, I took a heavy hammer to those and that worked. Was there a better way? Maybe, maybe not. The heavy hammer certainly did the job and ended up with the best possible code you can imagine out of that kind of scenario. I think the thing that showed up as an unexpected latent life giant problem was the observation that you couldn’t ask the runtime system for the state of the world, twice in a row and with the same answer twice.

Jan: When you say ‘state of the world’ what do you mean?

Cliff: How many subclasses does this class have? I’m looking at a call, it appears to call this one particular class, if it has no subclasses, I’m going to declare it final internally and in checkpoint. If it has a subclass and it’s abstract, then you must be calling the subclass etc. So, you ask the question ‘how many subclasses do you have and how many are concrete, not abstract and so on’ You’re going to answer and you compile along and across compilers running parses and they visit problems again and they come to the same spot, then they want to ask the same question again. The second time you ask it, you move forward in time and the class got loaded, then you have a different answer.

Well, the compiler itself everything it does is basically a giant theorem prover. You get 2 conflicting inputs, and one says yes, and one says no, one says false, and one says true. Suddenly you can prove false is true and your compiler blows up and it always blows up in a weird place. First you prove false is true then you executed false and true things in intermingling and quickly discovered that all code could and should be dead through 99% of the code way, and then something crashed because you had an unexpected situation and you can look at the program you’re compiling, and of course it’s been emptied out by optimisers, and you don’t know where it is.

That was a tough one and the fix was less easy than it sounds. Obvious fixes are when you ask the question you cache it if you ask it in cache answer. Then when you say you’re done compiling, you go emit the code, you test all your answers and if they fail, you atomically lock out class loading unit testing, you install code, you unlock which is fine. You get derived answers from the first answer about or implied, and they can be lies. So, somebody comes along and says, ‘do you know how many subclasses I have?’ and the answer comes back as two. So, you test for the one and then the else clause must be true. So you cache the answer two and later somebody loads a third one, and that’s not the question you’re asking. It was stick another way or something happened like the class you think you have got unloaded and reloaded a different class and so your alternative changed and you never asked for that correctly. There’s a bunch of these things where asking one question implied properties of other questions that you also independently asked. If you ask the one when you go to install the code, you have to validate all the dependent answers as well, even if you didn’t ask them.

Jan: Would that not be handled by de-optimization?

Cliff: No, the issue is that you failed to notice that you had an inconsistent set of questions.

Jan: Right, but there are certain answers that can change.

Cliff: Okay, but the answers you change are handled by de-opt, you’re correct. But they get de-opted because you record a dependency saying you depend on said fact being true. If you fail to ask all the right questions, and then you don’t record the dependency, then later the situation in the run time changes and because you didn’t tell it one time to look for this, he doesn’t look for it, so he doesn’t de-opt you. So essentially you die, crash and burn. Finding all of those took some amount of time.

Jan: This is definitely something that was new with Java, right?

Cliff: Totally. People have been doing profile directed compilation for a long time, nearly got them 5-10% better answers in C land, but it was a pain in the butt, because you had to run profiling only mode, watching profiles and then recompile. Vendors did it getting benchmark scores, but that wasn’t the normal mode the compiler ran in, so that mode is full of bugs, so all the clients, all customers of the vendors – I talked to as many as I could, and they were all definitely afraid of using profile directed feedback optimizations because it was an untested path through the runtime. If we run it through compiler, the code generator left and right, fine. So, in the land of hot spot we started using those profiles up front all the time, everywhere, so that was the normal path, not the rare reoptimized path. So, it became stable and good, you got great answers. But that was a whole new set of things that you could and couldn’t do.

As soon as I got de-opt working, I suddenly opened up a whole other can of things I could experiment with. Not bothering to execute these incredibly rare paths and then having to hold those very conservative results back into the main control flaw I could just tee off the execution through the interpreter and never I shall return from this path, if this is false, you’re out of here.

And that saved me from handling a lot of really weird bad situations, and it let me also take low frequency paths and just throw them off into Lala land. Then that made the main line code go that much faster, so we got a lot of performance gains out once that can had opened, and we started looking for other things and you get down this path of what we call heroic optimizations, we would heroically assume something had never happened before, wouldn’t ever happen in the future.

We put a test in and said never has this been null before and if it’s null now we’re going to bail out and have to recompile and de-opt and go again, but since it’s never been null, we’ll just test it now. These two things were always aliased, so this this array operation is always a list always aliased or always unaliased, is always 1/2 left or path right? I don’t know.

Jan: If you think back about all those years a Sun that you spent working on hotspot, how much Java code did you actually write?

Cliff: Almost none. I wrote, tiny little bits now then to test out things. But I remember when I first really started writing Java code and I’m trying to, it was a long time ago when I sat down and said **** it, I’m going to write some code to use this damn JVM I worked on forever, but I wrote no Java code for the first five years of Sun, much of my time at Sun, there’s no Java code.

Jan: So, C++ then?

Cliff: Yes, it’s all C++ and a very restricted subset. When we started hotspot, templates were new and unreliable. They would either crash compiler or emit horrendous code on some platforms and then work great in others. Some guys had these new features that were sensible, some folks went through C-front and had horrible debugging stories to find the bug, you couldn’t rely on them. So, we restricted ourselves to this minor subset of templates: square brackets for doing operator overloading, they were template-ized so they would be specialized to whatever primitive we had. Kind of that was it for the templates. We like the comment style, we like that you could create new variables anywhere in the method, not having all variables at the start and single inheritance, strictly single inheritance never multiple inheritance. With a lot of stuff with the new operator, because we have a lot of poolings. There’s no GC, we’re writing the GC. You did a lot of classic arena resource pooling operations where you did something in a pool, then you reset the pool in your garbage collection. That was kind of a very restricted subset of C++. The other one was that there was a volatile keyword, but it meant basically nothing for multithread programming. It had no language visible semantic meaning.

Jan: What made you choose Azul?

Cliff:  I was unhappy at Sun because I would claim management screwups were getting worse, already bad and getting worse. So, you know, **** rolls downhill, so that one starts at Scott McNealy and goes down. Maybe he just lost track of the things that had some mid to lower levels, but somewhere in the mid-level management tier I could see people were doing obviously stupid things, which impacted me in a negative way. I was unhappy with life, and I was starting to race my high-end sports car because I was bored and wanted some fun. I had a crash, brakes failed at 110mph on Thunderbolt Raceway, so that was that was sort of a life changing moment or your mental opening of choice, like what the hell am I doing here?  Brakes fail at 110mph on Thunder Hill Raceway, so literally within the week   I’m back from the hospital for two days and I got a call from Azul saying, they were doing custom silicon for doing Java and this is sort of like my dream job right there. The time I spent at Azul really was some of the most fun times in my life. The CPU designer sat next to me, and we would shout ideas back and forth through the cube walls as to what you could and couldn’t, should and shouldn’t do. I showed up there and all these crazy ideas were speaking about Java. I was like no, no, no, throw this this random weird *** away, with classic three address RISC that’s easy to get code for because it would be way faster than not getting the code. And all these things you worry about they all go away once you’ve checked. We had custom support for as you mentioned; hardware transactional memory, custom support for ranch cheques, custom support for read and write barriers, custom support for inline caches. We have basically custom support for CAS instructions for locking the high core count machine. Your CAS instruction could hit in your L1 cache. Which meant that if you took a lock and unlocked it and re locked it unlocked repeatedly back-to-back after the first one was hot in your L1 and you took out one hits for locking and unlocking a lot. Pretty much basically is a custom support in that era and eventually, Intel got in their L2 right as well, some years ago, but Azul had it for 10 years before Intel did.

Something else is mission range cheques, so there were like three or four others that we had custom 4/4, it was just a lot of fun, and the chip is one of the easiest ones I ever did code for. At that time, I had code generators for a dozen different architectures. It was one of the easiest ones and it made some of the cleanest looking code. It was like I said, it was a real joy to write code for, to generate code for, and then she could play all these fun tricks. And then we started down the path of making our hardware transactional memory work that was brand new and all kinds of weird semantic issues playing around with it that were fun to debug.

We had a guy doing the garbage collector and I did a different version with the stack-based allocations, the same hardware that you do read and write, write your checks for doing stack allocation which got interesting numbers, but they never beat the background using other core, our garbage collection numbers which were awesome and remain awesome to this day. You’re on the land of an X86 probably doing something with stack allocation is still beneficial because your caches are so much bigger than these old caches. And the heating in an L1 cache on an X86 is so much greater than on an Azul Silicon, but the background collector is still pretty awesome and on X86 remains pretty awesome as well. I don’t know where we started from, but I had a great time, I got to do all the things that I knew needed to be done for the JVM that Sun would not let me do.

The first six months at Azul, I rewrote an enormous chunk of the core runtime, but 30,000-line DEF, 25,000 lines removed 5000 lines added, replaced the threading model, replaced the exception model, replaced the de-opt hooks, replaced the Interpol between the tiers and all kinds of stuff. The end result of that was that we could then stop individual threads at a handful down segments that turned into low latency garbage collection and lower latency biased locking in those latents. A lot of low latency things came out of that, because of the ability to stop and start individual threads or have threads do self-service tasks at very low cost. So, it’s low cost being typically nanoseconds. We did profiling, we got real random stack sample profiles at very low cost. Probably a sample code would run you more like 1000 nanos, mostly cache hitting writes to mostly cache hitting on your calling your stack and background rights to an 8K buffer which maybe not in your cache. But the X86 background essentially writes for free. You put down some stack samples in a local storage buffer that has been recycled from under you periodically by a background guy who’s collecting the samples, then throwing them off to the side. That all fed into an awesome tool for looking at performance so we have hardware performance counter we could drag out, we could deoptimize, we could disassemble the code for you, with all the Java stuff lined up next to it and the field names and all sets of types, the compiler deduced and all the inlining stuff with perf counters on an instruction by instruction basis. Freaking awesome, and then Azul hid it behind a paywall which drove me nuts. While not behind a paywall, I used that tool on a regular basis to help other people with non-Azul BMs get their Java performance problems fixed, usually within 10 minutes you have to run it on my system, it needed to be a big enough job that you actually get a profile that came out. Then you would pull up visualizations, the people would be drooling telling me they’d needed this for years! And then immediately the problem pops right out, I’d tell them what they needed to change, and they’d get 30% to 5X just like that, the stupidest thing ever!

Jan: What was the reason Azul stalled? At least in the marketplace.

Cliff: The hardware was the wrong business model. Very fun, great hardware to work with. But so, it’s a combination of things that are not obvious in hindsight, but the time going forward I never spotted it. You had to go big to buy Azul, there was no small version. You could always start small on your laptop, then once you got to the point where your problem was exciting and everyone wanted more of it and it didn’t fit on your laptop anymore, then you went to a server. Once it got exciting on a server, you had to go multi on a server, and that was kind of hard. Once you sorted out multithreading on a server, it wasn’t a big jump to go to multi-node, where you switch from shared memory to pushing your communications through a pipe.

Now if the pipe was shared memory, that’s great, but that could have been a TCP pipe or a socket. Suddenly you’ve got a bunch of independent processes connected by sockets. Now they’re running on different nodes or not accordingly. Then you can have a round off switcher if the services, otherwise, invariant onto what service it’s doing or how it behaves, and now you could scale the service easily. So, you’ve got a scaling model where I went to Amazon, I said give me another one, and you get this different set of performance problems such as these micro service collections as a big hub which have various kinds of I would claim Classic Control 3 performance problem issues. I hit these problems in spades, and you watch all kind of performance things that full box would never have me pull over. You could creep your way into there, solve your problems one at a time instead of paying a million bucks for a giant box, followed by restructure to be shared memory and then you get some decent tools, though if you sort out your performance bottlenecks and then suddenly you go huge. A number of people did that, it can be made to work, but it wasn’t an incremental piece. You bite for money, you bite for time, you get it to run on the box and then you got the big win, as opposed to I bought a $1000 box, it worked for a while, I bought a $1000 server, it worked while, I went to Amazon, I rented them for $1000 for a year. But I bought a second one then a third, then you end up spending thousands, then you end up spending $1,000,000 at once and that’s the difference. That’s what I think what really failed out there, well half of. The other half was the hardware at Azul didn’t have intels, so the proxy was 110th and it made up for it by having a lot of support for performance, even for Java execution baked in, but individual processors are about four times slower. So immediately your programme ran four times slower, meaning if you had multithreading and shared blocks, then the shared block portion was four times as long effectively, and you had a couple issues getting on the locking problem.

Cliff: The Azul box would have 10 times the count of processors and four times slower. This meant it would be 2 1/2 times faster on the small scale and on the large scale easily 10 to 100 times faster, but you had to overcome a 4X slow down with more processors before you tie which meant you multi-threaded in a pretty big way right away. Like I said, there are some folks that model to really work with people running big web services. I think even to this day, some of them are still running on that kind of ****. The web portals of 30 gigs a second, 50 gigs a second of allocation on 1000, running threads with problems, and the X86 is of the day were just dying left and right.

They ended up getting out of the hardware model, and they are still selling low latency big servers to people who are buying that model, but now are X86s. High frequency is the common use and they do sell those to latency guys, but the smallest possible latency can’t be done in Java. But if you’re just one step up, you can get a lot smarter. You can rotate algorithms much faster in Java, so that was the main trick for running Java for these trader guys was that training algorithms can be flipped in Java 10 times faster, you can get the same one debugged and see. They got bought out some years ago, I got paid off for my stock they’re an ongoing continuous (enterprise), they have a business model that keeps them funded and floating and alive and well.

Jan: Transactional memory, in hindsight was it a bad idea or the worst?

Cliff: I talked about this in the past. Hardware transactional memory doesn’t work. I mean it does exactly what it says it does, and the Azul stuff was way better than what anyone could do at the time, and I honestly think it’s better than what you could get today. Basically, all your L1 was the size of your transaction. You started it; you ran the code. If you didn’t lose a line out of your L1 then no one else in the system with shared memory could tell what you had done in your L1, and then you could just flip under L1 and that was the new state of the world not with your transaction. If you crossed the line, you voided the transaction and you took all the lines that you’ve modified, and you just threw them away. And then you just cache, miss them back in later to get the correct answers. Your transactions could be up to 16K. You had associativity issues, so you could get accidental false losing and stuff like that. The real problem was people didn’t write transaction friendly code, and that meant they had gratuitous true data dependencies that would prevent transaction, the obvious one being mod count for a hash table. If I had unsynchronized hash table then I just want to make it behave like a fast hash table, mod count blows you out.

Jan: Why?

Cliff: So, imagine you have two guys. The second guy updates the word containing the mod count which updates the cache.

Cliff: Which blows a line out of your cache, and you declare non-transactional. Then you have to atomically increment the mod count and then atomically update the other lines. So, the obvious thing is it’s a big hash table, it’s a big array, you have two different portions of your array, you’re on different cache lines. So, you’re busy modifying two different cache lines, which works and then you both touch the mod count and one of you loses.

Typically, the other one going along is when somebody loses the readers during an atomic read as well also lose. So, the readers all get aborted out and then readers all pile in right away. Then when a writer comes through, he aborts all the readers. If the reader comes back in after the writers started and has some lines and the reader wants to grab the lines, the writers holding onto, then the reader aborts the writer. You did have this conflict mostly tied up with mod count, so we went through mod count in the JDK and have a version that turned off the mod count. And you ran into something else, and then you ran into something else, and you could look at them and see that there was no concurrent thinking going on, and so there was accidental conflict that aborted you.

You had to structure yourself to be concurrently safe in practice. In the land of Java locking, you would have had to put locks in around these places and then you would get bottlenecks for locking. You would have done some parallel thinking to get around the locking bottlenecks. With the Azul version you didn’t have to put the lock in, but you still had to have the apparently parallel work. And if you didn’t have the apparently parallel work, it wasn’t going to run in parallel, somebody’s going to write, and you were going to kick each other out.

As soon as you made to the effort of making your code work more or less without locks, you didn’t have to be perfect with your locks anymore, but you did have to have the same kind of thinking when designing it. That effort meant that you touched the code, and we didn’t just deal with legacy code. That effort got you to running with a locking model, which didn’t require custom Azul hardware. Half of that work gets you to a model where you could put it in a different process and go to cluster computing. It’s a little bit more work to go to cluster computing because you can’t talk about things indirectly, you have to go through the pipe, but if you did it to run in parallel, in practice you had to have some sort of things which were independent and that can be run in parallel. When you got to that step, you were halfway to putting it into a cluster, so it never it never really paid off. I also looked at a lot of other people’s transactional memory hardware school stories and I got the same story from three different companies that made a serious hard effort to use software transactional memory.

The story was that the STM didn’t handle their exact problem, their exact problem didn’t fit the STM, so they could improve the STM, and they could make their problem more parallel and then the two kind of wove together more and more so that after a year’s worth of effort they had a reasonably performant system that was not really STM and an application. It was a blended thing where the STM was just really part of the application now and it wasn’t really STM at all, it was more like a custom parallel situation using some bizarre low level memory tracking thing as the way to detect it.

Also, these systems turned out to be extremely performance sensitive. In the land of doing any sort of engineering on fluid systems, mechanical with fluid flows or electrical engineering with fluid flows, you have these instabilities which everyone gets taught how to handle, then you have control theory on over and under damping to handle positive feedback loops which are causing it to spiral out of control. A lot of these STM systems didn’t have that thinking going into it because that wasn’t the formal training, but the guys were in the problem! They had these highly unstable domains which would work great until they fell over hard, then the only exception was to reboot the application and get it out of its wonky situation, so you didn’t have a smoothly increasing performance curve as you added more cores and adding more load it wouldn’t stack. Instead, it would go up and fall and then you could pull it off and it would never recover until you reset and go again. By the way, this still hits the micro services guys too, they have similar things, although in the last several years, over watching people get a little better about control theory, manipulations and how you deal with that from an engineering point of view, so it’s better for the micro services guys now, but you had to actually put the formal theory to a problem, whether it’s an embedded system, whether it’s an STM, whether symbols or hardware, or whether it’s a micro services. All of them you have to think about that.

Jan: So now you’re working on a new language?

Cliff: Yes.

Jan: Why not Rust?

Cliff: Because I’ve spent all my life evolving somebody else’s language, so I have to try out my own. That is part of it, but part of it is that I like terse because you get more functionality provided. One of the things I did at Sun that still makes people twitch, but I stared hard at this, is that I looked at 100,000 commits, an enormous list of commits manually one by one. Over the course of several months, I missed many months of tests because I was frustrated with was going on with Sun management and what they were dictating, such as how policy works for programming and who was allowed to do what.

Out of that study of the Hotspot core runtime system and the 20/30 engineers working in core hotspots: 7 from the C2 compiler, maybe 10 from the GC and 10 from the runtime. It turns out that there’s a clear exponential power of two for speed of the programmers in terms of lines of code generated. It turns out that the next guy down was half my speed and then the next guy down was half his speed etc. The myth of the 10X hacker is totally validated by the data. The bug rate per engineer… Some engineers produce more bugs, some less bugs. But it wasn’t based on the exponential scale of writing code based on bugs per line of code, was it on a bumper? Then it broke the kind of code you wrote by three classes. There was new code and then there was tech debt elimination and there was bug fixing. And each of those have their own bell curve, but they were shifted by 10X from each other, so bugs rarely introduced new bugs. Fairly frequently you’ve got a partial fix of an existing bug where we didn’t quite get it and it needed more work at all about keeping more work on the same file. Occasionally you’d get a new bug, but it was pretty rare. Tech debt elimination occasionally also brought new bugs in, but it was more than bug on bug, but it still wasn’t very much. It happened, people didn’t understand the invariants, but the code got lot cleaner, you occasionally screwed something up.

New code 10X 4 bugs and that was the classic thing. If you look at that, if you wrote more code, you wrote more bugs. If you want more lines of code, you get more bugs, because it just had a simple linear scaling factor on that bell curve or adding more lines of code you land on the bell curve where you landed with more bugs. Of course, I wrote new code, and I wrote more code than anyone else, so I had more bugs than anyone else by a large amount. But I was middle of the road for bell curve on bugs per line of code, for new code and for tech debt and for bug on bug, middle of the bell curve. There are people who wrote more buggy code provided code and people who wrote less, but it wasn’t a dramatic effect per se. Instead, the dramatic effect was they didn’t ever write the new code. There were guys who only ever did tech debt elimination behind me, and then they would complain how many bugs I wrote. I looked when they wrote new code, they wrote bugs at the same rate I wrote bugs when they wrote new code, so you code and their product stagnated or somebody wrote the new code, and they got all the bugs.

Jan: It is more terse, so therefore fewer bugs?

Cliff: Yes, absolutely! That was the reasoning I came through. Fewer lines of code getting the same job done. Your bugs are on a bell curve, if you got more work per line of code, you got fewer bugs therefore better functionality. I’ve carried that philosophy forward ever since and I can see you grimace, there are people that like fluffy code. But I didn’t count comments. I didn’t count whitespace. I did count if you open your curlies on a line down.

Jan: I always open them on the same line and if I can fit everything in one line, I do that.

Cliff: Me too, and I’m saying there are people who just will not do that, and my experience with them is they have the same bell curve bug per line of code, they get more code vertically, they got the same rate of bugs, but they got less done so they got more bugs per functionality.

Jan: I like to see a lot of stuff on one screen.

Cliff: You can fit it in your head, and then as soon as you can see you can understand it. That’s my theory about why that works? I’m not a psychologist, I didn’t do a psychologically accurate test, but I claim I did a tonne of put backs. When I say a lot, I’m really serious.

Jan: What is a put back?

Cliff: A git commit. On Sun’s code manager it’s a commit, some were big, some were small, many were in between, we had big and small all over the map. So, there’s a couple of things running around with AA, one is terseness, another is that I like the things you can do with functional programming, while I also like to write imperative code. I’d like to have clean cost model like a drive, high level concepts down through hardware to get good performance out of it. I understand and I like the concept for us, but I understand GC is incredibly useful when you just don’t care about performance, and you just want to get the job done. So, there’s a lot of places where you want to use garbage collection, because you’re just going to fire and forget and walk away and if it becomes slow later and the GC is to blame then you can deal with that one. There are also times where you simply cannot do GC, no device drivers and kernels etc, where it’s just not reasonably possible and you have to be no loss and you need a true tracking. Other places where performance can be had by getting away from the GC – big performance numbers can be had, such as tenfold speedups – big numbers by going to a rough style model, but that’s a performance gain, not a functionality.

Another thing I’ve tried is an experiment where I observed that the C2 compiler at Java Sun mostly reversed all the typing information that was lost by generic erasure. And I wonder if I could do the same thing doing type inferencing using that style and the answer was no, although the reason why surprised me. So, I then switched onto a Hindley-Milner style, and I wanted anyhow just to learn, so that became a learning curve. Then I figured I could combine Hindley-Milner and forward flow typing. That I think, is theoretically sound now and I’m busy trying to get the implementation. I’ve splashed over what I call the theoretical typing pit of despair, and I’m climbing my way out of it now, planning something exciting with Hindley-Milner and classic C style or Java style flow typing. I’ve also got the Hindley-Milner doing standard struct records, polymorphic struct records. Got nominal typing in there, structural nano typing and this classic Java nominative typing. I’ve got null checking like you get out of Kotlin built into the type system, so if I know that you’ve done a null cheque, I don’t have to do things. Same with the instance of, if you don’t instance of, you don’t have to declare a variable. In Sun you declare a variable saying this is the name of the variable in Kotlin and C, if you’ve done your test, that’s the type of thing has and the positive side of the test and you don’t have to declare anything, you just use it as a kind of thing which I wanted real closures. I want more closures, so that lets you do things such as code generation, code routines and generators, 24 loops at one background, there’s a bunch of things you can do there, monads, planning etc.

Jan: Tail call optimization?

Cliff: Yeah, I’m going to have to do that, that’s really for a particular coding style. I definitely have loops, although right now my loops are just regular AA code that have blue tail call on itself. It looks like a full loop like you might see in C code, it’s pretty close to it, but it’s actually just a user variable, as the word four is not a keyword it’s just a user variable for a function that happens to do with tail call looping.

Will I keep that forever? I don’t know, maybe, maybe not. I’m doing a lot of fun things, such as the keywords, it’s terse and having looked at it for the past year, I’m getting more and more joy knowing what it looks like. Is it everyone’s cup of tea? I don’t know. One of the fun things here is that I’m self-funded now, I don’t need to ask anyone else if they care I’m doing what I want to do, and if you don’t like what I’m doing then you don’t have to use it. You can pay my boss – that’s me. I’ll take money to do something different, I’m sure I can be talked into things!

Jan: What would you do if you weren’t involved in programming languages?

Cliff: I’ve gone back and forth between flying cars and running for public office. I see a lot of stupid things coming out of public offices and I think I could do well.

Jan: How old were you when you wrote code for the first time?

Cliff: I was eight. I liked it; it was fun. I didn’t see another computer till I was 11 or 12, and then I was doing punch card Fortran. Somewhere around 15 I got a TRS-80 and that was a big step up, a big change in things. With the punch cards I did paper tape and wrote a Star Trek game on paper tape. Somewhere in the prime of the TRS-80 I was running the computer centre for Saint Lukes Blood Centre and downtown Houston Medical Centre. They had a computer, and they ran a job every night to print out people who had donated blood in the past and when six months had gone by, they could donate again.

Well, my story of that is at some point when I had come home for dinner – because I was a teenager in high school, you come home for dinner at a reasonable hour – I got the call and it was an emergency, and they needed a rare blood type and they wanted me to run a search for AB negative or something – a really rare blood type. So I said, let’s see if there’s a hit in the greater Houston area, and so I go back downtown – by the way at 15 in Texas, you can drive a car – not a problem. I drove downtown, ran the query you had to set up head punch cards. I ran the query and I got a hit. The medical person there standing over my shoulder called the cops, and then ran out to the address. It turned out to be the patient’s sister-in-law or sister Mary. The police drove her downtown, she donated blood for her sister, and she saved her life.

Jan: There a software project that you would have loved to be part of?

00:49:07 Cliff: Stupid as it was, or is, the call to do Sun versus Google, I’m very happy with how I turned out with C2 the more that went. On the other hand, Google did some pretty amazing **** then that I would have loved to have been a part of.

Jan: What do you think is the most important quality for somebody in our trade?

Cliff: Attention to detail? I think a good memory. I remember op codes for C80 machine code, XC0 was a basic call.

Jan: I think you should shed the burdens of the past.

Cliff: It got wired into my brain at that era. I have shipped many burdens of my past. I am a much happier person than I’ve ever been before in my life. I guess I was really happy as a kid already – off and on, and up and down, nerdy as hell, I had all the issues and teenage angst etc. but I was happy and that kind of went downhill in the marriage because that didn’t work out for a long time. And it took me a long time to recover from that marriage, but I’m back to being about as happy as I’ve ever been. Yeah, I have enough money do I want. I can travel the world when I want, I have a wife who dotes on me and kind of do it on her, we’re having a great time and we both love to travel and do our **** and I’m still kind of Cliff!

Jan: If there was one language you wished never existed, that you could erase from the planet, which one would have been?

Cliff: There’s some real losers out there, if I didn’t like it and didn’t think about how it shouldn’t be, then the obvious one is COBOL, but honestly, Java has replaced COBOL, is that a better replacement or worse? I don’t know, but it certainly gave me a leg up to fame, maybe fortune, maybe fame for sure. So yeah, I would claim your COBOL, but COBOL has one thing going for it that no other language on the planet has going for, and that is that. It was around and alive and well in the 60s. Name any other widely popular programming languages from the 60s? We had to start somewhere, as much as I think COBOL is COBOL, on the other hand it opened the way and it showed what could be done. Immediately you can say what’s better, but it’s the same with Java, they said they can do multithreading, new GC, class libraries. They do a lot of things which are really good and then now we can look back and say Java did a good thing, but surely we can do better – well of course, but you had to have somebody come along first, say and ‘here’s a good thing, let’s make it better.’

Jan: What would you like to be remembered for?

Cliff: That remains the open question. This is the don’t live your life with regrets, which is the same reason I would like to run for public office because I would like to be remembered for something outside of C2. I think I’ll be remembered for C2. I think that it will be a long time before that memory is forgotten, maybe it’s forgotten already, I don’t know. But you know that was a good thing, and it’s run its course, it’s service to humanity. Where does that come from? Well, some of that came from what I did with Java and helping Java do what it does right now. In Java there were some things that I didn’t make happen, but I was definitely a part of making it happen and I’m happy that I could be a part of that.

Are there other services to humanity I could do that would have a longer-term impact? Surely, but what are they? I chose to sidestep down for a minute, to let myself do the language thing, get that bug out of my bonnet, and get that bucket on my bucket list. I’d still like to go for public office and do something with flying cars, and that one I’ve kind of got burned out on and I’m frustrated because I know what to do technically. There’s a technology piece there that I have wired in my head. When I have a technology bit in my head, I have never been wrong on those. Everyone told me, you can’t write a compiler at runtime. But I knew it could, it was there. You can’t write compiler with this damn graph thing. I knew I could, it was there. That has repeated itself over and over again in my life dozens of times, I’ve looked at something I just know how it works. I can see all the ends and pieces. I would have to pile in on the VC money game again and in some fraction of that job requires public presentation overall not exactly public office but interacting with public. I would like to run an auto pilot on all aircrafts everywhere that would allow all non-pilots to fly freely, but everyone has to buy into the same auto pilot. But that’s public airspace, so you have to make a request to public airspace. That’s the big piece that you need from the public side. From the technology side, there’s a couple pieces that are obvious and I don’t want talk about right now, but if I went down the line of explaining to you, you would totally buy into them, they’re just straight forwards. Suddenly moving Autopilot would handle a billion planes in the air at the same time and barely anyone could ever crash into each other, which is basically cheating the system. But if you play by the rules and the rules are easy, don’t break the ****Don’t try to break it, you couldn’t break it and then you could fly and it would be totally safe.

Jan: So, there are two possible futures: Governor Cliff or jet pilot.

Cliff: Yeah, exactly. It’s an enabler for other people. To get the freedom of flight. It’s not jet pilot Cliff, I went through that phase, that was me on the racetrack with my car having a great time, more fun than any Six Flags roller coaster – until it breaks.

Jan: Thanks a lot Cliff, that’s all the time we have.