Sustainability and Software - Designing a Framework for Software Sustainability

Tony [00:00:04] Welcome to Code Together, a podcast for developers by developers, where we discuss technology and trends in industry. I'm your host Tony Mongkolsmai.  

Tony [00:00:17] Earlier this month, I had Marlow Weston and Niki Manoledaki on the podcast, two contributors to the CNCF environmental Sustainability Technical Advisory Group to discuss a variety of topics around sustainability. We discussed why sustainability should be a significant consideration when designing data centers, software processes, or when choosing a software stack. Today, we're going to continue that conversation around sustainability and how developers can focus on and contribute to sustainability efforts as part of their work. And we are fortunate today to be joined by Asim Hussain. Asim is a developer, author and speaker with over 20 years of experience working for organizations such as the European Space Agency, Google, Microsoft and now Intel, where he is the director of Green Software. He is also the executive director and chairperson of the Green Software Foundation, a global industry consortium with a mission to create a trusted ecosystem of people, standards, tooling and best practices for building green software. Welcome to the podcast Asim. 

Asim [00:01:11]  Thank you very much. I'm glad to be here.  

Tony [00:01:15] So yeah, we have a lot of different ways that we could start. Maybe you should... I don't think my listeners will know about the Green Software Foundation, and you have your own podcast, so we'll have to plug that later. But why don't you tell us about the Green Software Foundation and how you ended up where you are now focusing on sustainability?  

Asim [00:01:33] Yeah. So on a on a personal level, I used to work at Microsoft and I used to be in what's called the Developer Relations Team. Well, a team that you're probably quite familiar with inside organizations. So now a large part of my job was really just connecting with developer communities out there and just really start just talking. We have a saying help first, sell last. Just like helping, talking, what are you issues? Let's talk about them. At the same time. I was somebody who personally was very interested in sustainability and asking myself the question one day, How come, you know, like we have this whole story. We just, our first son was just born and we decided to use cloth nappies. And with cloth nappies you have to deal with honestly, a lot of poop. It's just everywhere. Like you just got to face to face of it and just deal with it. And like, I used to joke to people like, like literally, like for the parents out there, you all know what I'm talking about like, your hands are covered in poop. Like sometimes you got poop on your on your clothes and everything. But like, I was willing to deal with poop on a day to day basis for the environment because I cared about wanting to have a, you know, be a good custodian, but never I realized I had a epiphany one day, which I never actually asked the question. The thing that I do 8 hours a day, 10 hours a day, 12 hours a day. How can I do that more sustainably? And that kind of started me on like quite a long journey. I first joined a community out there which is still very active and actually recommend a lot of people join in. It's free to join. It's called Climate Action Dot Tech. Honestly, anybody in the in the tech industry who's very interested in sustainability and not just making that software more sustainable, but just interested in general. So I joined that community. I then became a co-organizer of that community. I was also a time leading kind of EMEAs, Microsoft's kind of EMEA developer relations team, and I was really starting to ask questions like, Well, I had a team that was flying all over Europe and the Middle East constantly, and I started asking questions like, What can we do to make that even that more sustainable? And then honestly, then an opportunity arose. There was a reorg at Microsoft, and I just I was I said, look, there's a there's a, I saw it. I was part of it. I was part of this growth. I saw that there was a large number of engineers out there who cared as passionately about this area as I did, who were honestly asking themselves the question, Well, what should I what should we be doing? Well, what should we be doing? And Microsoft should really have like a, you know, a role which is really talking to and advocating for on behalf of that team. So I got the approval to kind of incubate like a and a team was just two of us, a team at Microsoft. And that really grew. I started writing a lot. I always say like one of the most powerful things you can do in this world is write. I really recommend if I if I meant to anybody, the first thing I say is just start writing blog posts. You know, in ten years time, that's going to be the most valuable thing you can ever do in your career. So start writing and talking and creating material and actually start pulling people together like towards me and kind of other organizations start reaching out. Like large, Accenture reached out, the big organizations reached out and said, look Asim, we love what you're saying. We're saying the same things. Let's sit and collaborate. And that's when the idea of forming the Green Software Foundation came along, because that's like a vehicle I describe as a vehicle for collaboration as we launched just over two years ago now with about eight members, and we're now almost 50 members. So we've grown phenomenally. It's been just a real challenge. Like I've never dealt with anything this grown this fast in my entire life. Like every six months I have to change all of our processes because we can't handle, you know, the load that we have right now. And yeah, and I think that's two years later now. Now there's a large like the what the foundation does is we focus on four areas. One is around standards or of the standards you need in order to well build greener is software. We have a community working group which does like education training, like what is green software? Like, how do we even define it? Like, what are the things you need to work about, what the training you need to create? We have a policy working group and they're really trying to figure out we're quite growth focused inside, they're looking outside.  They're asking the question like what regulations are coming down the pipeline or even just what order you mentioned the CNCF. So they're like, how do we how do we work with the CNCF? Like, how do we collaborate? Things like that? And then we have like our open source working group, which is, as you can imagine, focusing on creating an open source tooling and things like that to further, make it easier really to build green software. And that's kind of like I'll probably leave it there for, for a second as an intro, but that's like, that's like basically my journey and kind of where the green stuff. In the meantime, in about a year ago I joined Intel because like, it's interesting, like Intel. Like Intel's like only customer really is people who build software. I might go through some resellers or something, but, you know, people that build software is kind of like their primary customer. So like a question, it kind of makes sense for Intel to be really kind of driving and thinking through, like, how do we make it strong so that when people build software on our chips that build them the building them as sustainably as possible? Yeah.  

Tony [00:07:12] Yeah, and that's pretty interesting. One of the nice things for Intel is we were recently named, I don't know, the most sustainable company, something like that, some, some poll which is, which is nice. Yeah. But then there's still the, the I'll say conflict, I don't know conflict... We build parts that use a lot of power and you have to plug them into a wall. And other companies, large companies building GPUs, accelerators also have the same issue. And one of the things that we talk about a lot is that accelerators can theoretically make you more sustainable because even though they draw a lot of power, they might give you better power to work ratio. So that that's one thing that we talk about on this podcast in a way that you might be able to make things more sustainable by understanding how to leverage the accelerators. It is interesting, though, that obviously this is all workload dependent. And so while we say that, the question is, you know, how often is that actually true? And that's one of the things that we wanted to talk about today, which was how do I measure if that's true? And as a developer, how do I understand what tradeoffs I need to make in order to make good, efficient software that solves a problem while at the same time making good, efficient software in the sense of that it is helping with the sustainability issue. It's not using more power than it necessarily should. And I know that's one of the things that you wanted to talk about today. So let's dive into that a little bit and let's talk about measurement.  

Asim [00:08:44] It's interesting, as you mention, because there's actually a term I had here for the first time, because I've actually this I've been an application developer my whole career. So like Intel has been quite a journey of getting to grips with kind of the much lower level aspects of my of my world as a wonderful term. I had to have the first time, but I always think it should be like a a sci fi series or a novel dark silicon. I love that term. Dark silicon. I don't know if I'm quoting like, what dark silicon is? I'm just quoting like, what I think is but it's like like, as you mentioned, like stuff on a chip, which if you're not using if I'm telling you if I'm using it incorrectly, if you're not using it at all, it's dark silicon. It just doesn't light up at all. So like, I think that's kind of quite interesting that, you know, when you think you mentioned like the use of accelerators, but only really makes sense if you actually are using them. Otherwise they're just like you emitted carbon, creating it, putting on the chip, you're not using it and it's just a waste. It's actually it would have been better not even to have them on the chip if you're not going to use them. So that's kind of like, well, what we have in the in the foundation, we have like we've well, generally broadly, the the world is kind of narrowed down into thinking of green software in three buckets, while one primary kind of principle, which is what's called carbon efficiency, which is like everything we do emits carbon, which is kind of like a shocking thing for something somebody in the sustainability space to kind of admit to. But, you know, like I'm breathing right now and it's emitting carbon into the atmosphere and we just have to accept that and everything we do emits carbon. And so what our goal is to just make it so that there's just a few molecules emitted into the atmosphere as possible because...Here's the another thing, it's actually quite systemic, Climate change is quite simple. It's just a function. Simple function, ok it's not simple, it's quite a complex function, but it's a function of the amount of carbon there is in the atmosphere, and that's the literal number of carbon molecules that are in the atmosphere. More there are the greater the the temperature and the more the kind of like issues that that that that comes as a result of that. So that's what you should really be thinking of is like literal carbon molecules floating on the atmosphere and is what I'm doing making fewer of them go into atmosphere or more of them go into the atmosphere. And we say there's only like three ways that that that you can really tweak those levers. One of them is energy efficiency, which is use the least amount of energy possible to do a job. So like if you coded something so it doesn't use an accelerator, you're doing the same job, you just using more energy or if it lets frankly, coating something poorly, you're just using more energy to the same job. Or if you're like using an arch... It's much more less coding these days. The low hanging fruit is architecting, like if you've like massively distributed systems, if they're just architected poorly, then you're just using up to more energy than you need to. So just using less energy is one of the best... Oh, because energy if you're audience don't know, is that still the single biggest measure of carbon emissions in the world.  About 80% of electricity is still created through burning of coal. And what most people don't understand, which I think is fascinating, is it's actually it's actually the really it's a it's usually the very low quality coal which gets burnt because if you...Do you remember acid rain? Remember the whole thing about acid rain in the in the noughties there was like a whole thing like whole forests were dying out because of acid rain. That's because we used to be we used to be burning a different type of coal that actually emitted fewer carbon, less carbon and was more dense but created acid rain. So that all just went away. And we start burning this really low quality coal that doesn't cause acid rain but causes carbon emission. So we've got this whole like complex balancing act of kind of environmental issues and carbon issues. But anyway, most actually most electricity created burning of coal. So just using less electricity period is is makes makes your application greener. The other thing we call it harder efficiency like the fact of the you write like every Intel chip out there emitted carbon when it was made. It's a, we are either we are per like meter squared off fabs are some of the most highly, how I'm going to say this right, like most densely like energy consumed per square meter is like the highest in our fabs versus like most of the facilities in the rest of the world. Like very, very energy because you have a lot of energy. So use a lot of energy to make our chips and a lot of other things, you know, ram metals, all this other stuff that's called what's called the the embodied or embedded carbon of a chip. And so, you know, if you if you had to kind of like, buy an intel chip and then just not use it, even though you're not using any electricity. It's still really bad because that's like carbon emissions which were emitted and somebody else could be using that chip for something else. But they're just not just like lying there, like for anything. So one of the most important things you can do is because we call it hardware efficiency, which is basically using the least amount of what we call embodied carbon possible, which for like in the cloud space, what that means is like densify your servers running as much compute as you can on as few hardware as you can. If there's accelerators on the chip like using them, like use everything you possibly can and utilize everything to the maximum capability. And the last one is carbon awareness, which is a dig into what's actually really interesting, which is you're going to be using electricity in the first place. So can you... Can you engineer your software so it uses the clean, so it does more when the electricity is clean and does less when it's dirty. This is like a very interesting space, actually requires a lot less investment to kind of get started. Microsoft did some great work here, like Windows became carbon aware like two years ago on Windows 11. I think it was it became carbon awareness. Updates for Windows 11 happen at more a times when electricity's cleaner. Xbox is just like six months ago, Xbox implemented something similar like six months ago. So it's kind of like becoming this functionality is becoming much more prevalent. Google did some great work in the space, like a couple of years ago with the data center work. And it's just yeah, it's just a really interesting. Oh, UBS did some really great work here as well with the machine learning models. So that's kind of the third area. So this is basically called we call it the energy, the embodied carbon, and the third kind of factor you need to think about is like how clean up does your electricity actually is, because that also factors in like overall like what your what your emissions are. But a pause there for a second because I can usually ramble on for ages.  

Tony [00:15:56]  That's okay. I mean, you're giving us a lot of good information and a lot of it I don't know, or at least I haven't. I haven't probably I mean, categorize things like that in my space, in my head. So it's good that you give us that kind of insight. So then other people can also kind of have a framework to think about carbon emissions, as I was, as you're talking about it, I was thinking about the one nice thing that again, from a oneAPI perspective, as my job is partially as a oneAPI evangelist, I'm talking about Intel software. Like when you write something in SYCL, the nice thing is it's going to use the accelerator that's there and if it's not there, it's going to use the CPU. So that way I am hopefully utilizing the hardware more efficiently just by changing my programing paradigm. Now, as you think about it, there's also going to be tradeoffs because now I have to have some type of runtime that figures out where that energy goes. And if I always run it on the CPU and I never can offload, I'm just using that extra energy to then point myself right to the CPU anyway. So maybe that's not more efficient. 

Asim [00:16:59] This is, like that. It's, it's is everything is always a tradeoff. Everything is always a tradeoff. You trading of one thing for another thing for another thing for another thing. And so like that kind of like evaluation you've had, you've just made in your head like this is never going to be easy. It's the same way like performance optimization is never just going to be like this really easy thing that you do. You just really have to like, think it through. Like, like there's always like a time space trade off or something along those lines. And so there's really like a time space carbon like sustainability tradeoff, like thing going on. Like one of the things I always say we just wanted to we just want to we just want to be like one of the tradeoffs that you're thinking in your head. We just want to be one of those tradeoffs because you might actually have like a really valid reason for saying, you know what, in this use case, I'm just going to make a lot of carbon because the other tradeoffs are just too important and there's knock on benefits. But but right now there's none. And I think also it's like just going back to measurement as well. I think one of the other issues is the measurement like. There's very few context in which, as an engineer, you're like even asked to measure the energy that application. There's very few contexts. Like if you're a mobile app developer, you are going to measure the hell out of the energy of the  application, because it's like it's not even going to get accepted into the apps, into the Apple App Store, at least if it kind of like, you know, detect is like consuming, you know, inordinate amounts of energy. Because your apps never going to get blamed, they're going to blame Apple aren't they?  They're going to blame those, you know you don't blame like the app, you blame the manufacturer. So like, there's already like this pressure there to kind of make things more energy efficient. But when you're building for like desktops, or in the cloud there's just like no like almost no pressure at all to make things more energy efficient. And that's kind of like one of the challenges that we have. And one of the things we want to make, but that's actually the area that we're really, truly trying to work hard and we haven't got anything to release yet. But that's what we're really trying to work hard on inside Intel is really making kind of energy measurement far more ubiquitous and just far easier for engineers across the board to develop. And I'm just not make it to be like. Like, Oh, if you have a battery, you care about energy. No you should care about energy regardless of whether you're plugged into a wall socket, you care about energy. And more like one of the things that when we go back to Microsoft, one of the Windows did have like an energy measurement like feature internally, but it was just always switched off if you plugged into the wall. So if you literally plugged into a wall, it wouldn't even bother measuring your energy. So one of the things that that was done by my, well Scott's my current leader at Intel and he was at Microsoft as well. Like one of the things he drove was like making sure that that measurement was available. Like even maybe little things, like even if you're plugged into the wall. So, like, we really need to start measuring energy and just making a lot easier to measure energy. And this is why I'm excited to be at Intel because I think one of the we're one of the best companies out there in order to, you know, to really, like, facilitate that kind of measurement.  

Tony [00:20:17] So interestingly, when I built a PC last year, I have see which power supply to have. I think you have a Corsair power supply. And the nice thing is all the rails. I can see all the power draw for that power supply, pseudo real time. So that's a really nice thing. 

Asim [00:20:36] Where could you see it?  You could see via software or via like a simple hardware interface.  

Tony [00:20:39] Or via, via software. So I can use their IQ software and real time. You can see all the power actually to each rail and the total power pull for that power supply. It's wow, yeah, it's really cool. And then my next thought was, this is probably something that we should standardize so then everybody could pull that information and potentially use it. And I'm sure it's not standardized...maybe it is, but I have to use their software. I actually that might there might be another way to do it or it might be standardized because maybe there's other pieces of software that also know how to to view that. But. It's not something that developers think about. It's that common enough that developers think about it. And it's also that's on a consumer system and it's a very high end power supply because I have three GPUs in my system, so I'm actually sucking a lot of carbon as I do all my testing. But how do we how do we think about this either on a consumer system or in a data center? Because in a data center, I assume and maybe that's incorrect. I don't know if you know off the top of your head, do data centers consume a lot more power than consumer systems? I would think so.  

Asim [00:21:49] I think. Well, it's an interesting question. I've always I've always wondered about because there are like, you know, by the P the PUE, power utilization effectiveness it's like a metric for data centers. It's like a measure for how like efficient, like a data center is. If a PUE is like 1.1, it means that for data center, it means that if you're if you measure on chip that you've consumed one kilowatt hour, you actually sucked up 1.1 kilowatt hours from the grid. And that point one is the is the waste, The overhead, like the power supply is like all that. It's like all of that stuff is waste. I, I don't think I don't know if anybody's really done that for your consumer system like how much like for every kilowatt hour that's reporting to you. How much actually has gone into waste kind of going from like AC to DC and all of the transformations across the across the way to chip. I don't know if anybody really calculates that because that PUE metric is, the only the pressure to calculate that is from a data center perspective. So they all actually quite efficient. Like the air also depends kind of like organization to organization. So I think Google reports the best score, which is 1... I think I'm going to probably correct wrong, I think it's 1.08, but most of the kind of Hyperscalers report like, you know, 1.1, 1.2, something like that. But like, if you're like more of like a, you know, bespoke I don't know the word bespoke, but like non hyperscale DC. I don't want to over generalize but there are DCs out there which I like around 2. And if you're like in a really hot country, you're hovering around 3, you know, like you just really is a law and there's actually regulation. Now in certain countries like Netherlands say you can't build a DC anymore if the PUE is less than one greater than 1.3. So there's actually like pressures across that board. There's also like, you know, from a DC perspective, there's actually like, like you're probably not bothered, like, I don't think your computer I don't think your computer's electricity draw is driving enough cost for you for you to really start thinking about optimizing. It's almost not worth your time in a way, whereas a DC level like it's worth like having a team of people, like focusing in on this because that's like millions and millions of dollars savings. So that aspect of it as well. Um.  

Tony [00:24:16] But it's interesting though, right? Because if, if a developer who is writing some type of, say, content creation software that is going to run on a workstation and not a data center has the ability to measure that, they can again choose to make that tradeoff, like you said, right? They can decide, is my software more efficient? Is it a metric that I care about? But again, I need that metric in order to make that tradeoff. And I don't think that that's readily available on a workstation system, much less in the data center.  

Asim [00:24:47] I mean, you could be you could be building software which, you know, behaves differently depending upon the context upon which it's run. Like, there's actually really great examples of this. There's like quite a few websites. One I usually quote is like Branch magazine so branch.climateaction.tech, and that like magazine that's not based on energy consumption, that's based on like how clean the dirty the electricity is. But it literally shows you like it's like a grayscale is the images. If the electricity is really dirty right now and other things like that, they'd be like, wouldn't it be great if like your browser detected that like. It's maybe it's like maybe it's running on a really terrible, like, home, like something I would build, which is, like, so incompatible, like the is so inefficient and your computer's so inefficient. I'm going to, like, go into eco mode because you are you're not being a good custodian of electricity like, I think. But this is what this what I always say like. Give engineers the information. If you give people the information, they will usually make the right decisions. There's usually enough people in every single team that they will like engineer a solution around this, but they just don't have that. The data is not there. One of the projects that we're working on in the foundation is kind of like an incubation projects with even the name changes quite frequently until we like graduate, but it's is currently called Carbon QL and it's like trying to get a system where you could kind of build a model around that and you could say like, I'm your audience is not going to see this. I'm going to use my hands. But like I'm an Intel chip that's running on this motherboard, that's running on this power supply, that's running on this grid, and you can maybe layer it all together and say, well, actually this is what your energy consumption really looks like. And if you like have that information, you can maybe make different choices versus like, I'm an Intel chip running on like a server data center and a PUE, and it's like a really good one, it's like PUE 1.1. I might make different decisions and choose a different way. I think it's all about like modeling. And I'm very I'm very, very, very I'm a good engineer. I believe in engineers. Like I believe if you just give us like information, like you don't need to tell us. They just give us to give us data. Like, like if you if you, you know, if you create like a pull request, if you integrate into your like pull request and you just start adding in like so sort of performance, like imagine just one person team adds in like performance measurement to pull requests, and pull request just start like listing out like oh your code change like increased the latency by 0.01%. I don't even think you need to even instill a culture inside a team. I just think engineers will just go, all right, okay, fine. Game on. Let's get let's get that down. And I'm very much in that in that mindset. We just give them the data and then then everything will just fix itself.  

Tony [00:27:59] Yeah. So then usually. So I worked on a team as an architect where we measured the power through software and I guess estimated the power through software for Intel chips, which gives you part of the story. And I talked about this with Marlow and Niki a little bit, we can get some information about power when we were talking about it. The cool thing that I really liked about my power supply is I'm actually getting the power drawn from the power supply, so I don't even need to worry about the rest of the components. I can. I care what they are to make that tradeoff, as you mentioned, potentially. But I, I just know the power draw at the wall.  In a data center. Is there a way to measure that efficiently for an entire system? Can we look at like a BMC or something like that and give us that information for an entire system? Or how do we measure?  

Asim [00:28:47] So the challenge that we have is in order to get information, the more the the kind of actions that you need to take require extremely granular like information like you mentioned, like, like, you know, the power at the socket, the wall socket level, you don't know which of your applications you're actually running on your computer is the one that's drawing. Now, you can make some fairly decent guesses, but you don't really know. So you can know like the socket level and things like that and the power supply level kind of where like how much is being drawn. But it's really challenging from a data center perspective in order to kind of like attribute that down to a process level. And then then you then you add like orchestration layers and virtual machines in between all of that and that whole thing gets even more complicated. And so, yeah, it's really challenging. I think. I think at the end of the day, this is only really going to be... There's two pressures right now, let me put it...there's two pressures right now. One is that there's measurement that's done from a required reporting like almost regulatory, sometimes reporting requirements. And then there's measurement, which as an engineer you need in order to make decisions. We call it measurement for reporting and measurement for action. A lot of the pressure on data centers to measure right now is from a regulatory reporting perspective, which means that you I mean, I think there's an EU regulation right now and it's coming into effect for DC like larger than like 50, I think it's 50 megawatts, I can't remember what is it is, but it's the only requirement is building by building. You need to be measuring building by building. There's not even rack by rack requirements, just building by building. And so like from a regulatory perspective, that's where the pressures and that's kind of where the efforts going to measure. Whereas we need to know like, what is that process that's running on that virtual machine that's running inside that Kubernetes container that's running on that bare metal host. Like, what is the energy that that's that's using? And that's really, really that's why you need to start using like, like statistical models which go like. No, I know how much energy is coming in. I know your utilization on that VM. I know that VMs utilization versus other VMs and I can maybe like create graph that kind of takes some of those values and puts it back to a process. But we're not there yet at all. But that's all we need to get to. I would say if you want to get like action.  

Tony [00:31:22] Yeah, I mean, our software used heuristics as well, right? Yeah, We write software estimates.  

Asim [00:31:27] Do you know what you were using? 

Tony [00:31:29] Yeah, yeah. We were we're basically looking at the, the different states of the CPU and then we kind of had an idea of how much power each state is supposed to draw so we could kind of map that back. But the problem with that is, like you said, it's still a heuristic and it's only as good as our, our, our initial estimates of what the power draw was or the our mapping and our mapping could be wrong in the real world. And especially when you factor in, like you said, location, the amount of cooling needed, air conditioning, if I'm in a hot country versus a cold country, you know. But then the question I would I have in my head is why don't we just install something that measures the power? I don't know if it's a rack by rack system, everything plugs into a wall. Why don't we just have something that legally has to be in between the two and then you can just measure, I guess there's there's cost, obviously, which people really care about, especially when you're talking about thousands of systems.  

Asim [00:32:22] Yeah.  

Tony [00:32:23] I wonder if that's worth the tradeoff. I mean, obviously we care about sustainability. I would say it's worth the tradeoff.  

Asim [00:32:29] I was yeah, that's the problem.  I would say, you know, get the here's my money, you know, just go I want it. But the pressure's not there. And for the people, what we're doing, for instance, what what a lot of people want lot almost everybody is doing these days is just going by using models effectively. And they not even that is almost exactly what you just did using. So he restricts use use of utilization or some other proxy. Multiply one by the other look go to lookup table and your energy is probably this. More often than not, the more times you look at that people are always like, you know the squinting Oh but is it, is it, is it good enough? But more often than not, it kind of is. Like you're not going to get like it's not telling you even if it's like 70% of what the real value is is good enough. Especially since the alternative is like so much more complicated. One of the things I always say is like, it's almost not, and this is this is why people get a little confused with what I'm saying as well, because it's almost not even important knowing how much electricity you're consuming. What more, if you think about it from like all it is is a signal to you as an engineer to make decisions.  

Tony [00:33:40] Right.  

Asim [00:33:41] So as long as it's giving you the right signals, so you're implementing the right changes to your software, that's all that matters. So I always say like the actual energy value you could be getting could be like half what the reality is, as long as it goes in the right direction. Like when you optimize some code, as long as that number goes down, that's all that matters. It could be a gro, we should call them grots or something like that. Like it's just like it's not really what the electricity really is. It's just a signal.  

Tony [00:34:07] Accuracy doesn't matter. It's the relative signaling to the developer, which I think is true. So like, I guess there's two things that popped into my head that this one, I think that we do have that at least for CPU. So we have different power states. We can see those things for an AMD CPU, an Intel CPU. I don't know whether we have that analogy, that analogous measurement for GPUs, but there's so many other parts of the system too: motherboard, memory, hard drive where I may not have those things. I might have proxies like whether the hard drive spins up or not or the SSD is active. I might have like maybe IOPS is a good measurement there, but then the next problem is even if I have all these measurements, which I probably do when I am doing my analysis of my application, we typically will do things like performance analysis, but our performance analysis tools don't naturally integrate the energy consumed from different parts of the system. If, for instance, if on Linux when I ran perf, it always dumped me. Here's the amount of time in each state on the CPU. Then you had to actually make a decision if it was just there by default and just always dumped it for all CPU's you'd be like, Oh well, I use more power there or GPU's or memory. I wonder if even just a change like that either in our open source tooling or in Windows for instance, and Windows does the energy reported stuff. If we were able to get those metrics, not just collect them, because I know we can get them, we can report them, but if they were there it by default. So it was just in the developer face, then that might actually force some kind of change, Some type of change.  

Asim [00:35:52] Yeah, I think it is. I think I think I think that if you to I do agree. I mean, I. I don't know what the overhead of that would be. The the sound like those like it debug overhead. But like if there's, if if you know there's ways of like now you can always like go into a debug state or something like that if you needed it. Um I think the Firefox browser recently, like there's another colleague foundation called the Green Web Foundation, which is run by Chris Adams, someone I've known for for quite a while, and what they've released recently, something called C02.js, which is a library that helps you to and it's very web focused, very, very targeted towards a web space. Right? But what Firefox has recently integrated that library into that developer tooling is actually if you look in the developer, if you like, open up a webpage in Firefox, they actually will give you like a flame graph of energy consumption of carbon. I believe it's I can't remember if it's energy or directly carbon. I forgot exactly, but it lets you give you like function call by function, call it how much energy a carbon that's consuming. So you actually have that can be called like the inner loop, like it gives you like. Once you've narrowed down, like to like the process or maybe like the script, what you know, which one is the one that's consuming the energy. You then need that kind of like inner loop information to go, Well, let's put your function like, why is it consuming that much energy? So yeah, that is kind of like available there. But I would say right now the the bigger challenge is much more like architectural rather than code. If your software system is being optimized to the point where the only efficiencies now can be through code efficiency, you're doing a great job because most architectures are, have a lot of room to become more optimized. Like, there's that whole like statistic that the on premise they're now kind of non hyperscale environment is kind of still hovering I think about 11-12% utilization because most people are still like have got to make sure they've got enough overhead to handle like peak load capacity and stuff like that. So there's like we could architecture systems to be scalable and kind of like use, you know, other architecture. And still there's a lot of room that a lot of room for. For optimizations at that level still. 

Tony [00:38:23] And it's easier for us to affect the data center just in the sense of there's fewer data centers. Right? I mean, even fewer people doing data center design.  

Asim [00:38:32] Yeah, there's probably a few doing data center design.  There's like a paper recently, I'm still like flabbergasted by it, it was like 7.5 million data centers in the world. We're still like questioning like what they consider a data center. Like, I'm not sure whether like, you know, some sort of IOT tower somewhere is like we'd like to, you know, two services instead of a data center. But like, yeah, there's a lot of kind of interconnected like nodes out there in the world. But you're right, there's a lot of people, there's a lot of pressure on data center space as well just to be as efficient as possible. I wonder I should I should actually make sure I mention it. So one of the things we have in the foundation is something called a software carbon intensity specification, SCI. And what that is, is a way of measuring software, which is very much more focused on what we've just been describing. So it's like we're using a tool called Riverside right now to like record this podcast episode, like the, the standards for like carbon measurement or the price of carbon measurement these days, like Riverside. The only pressure for Riverside would be what is your total carbon footprint in the entirety of your entire product? But from a software, from a dev perspective, you're like, That's a very challenging place to be because like Riverside and the Riverside is a great platform. We use it as well. It's like Riverside is probably growing like year on year, like massively, right. So their total carbon footprint is growing. So imagine you're a software developer whose job is to make Riverside more carbon efficient. And yet, every single month your total carbon footprint is increasing because the product is growing. It's a terrible signal for you to know what it is you need to do, right? It's like, Well, dammit, the did everything I did last month, did it have any impact? So we developing SCI and SCI is actually a rate. So for Riverside it would be like perhaps carbon per minute. And then you could imagine, like, you know, the growth of your organization and your product could be growing or shrinking, but your carbon, if you're doing a good job, your carbon per minute should be going down month by month by month as a signal to an engineer is much more useful like measurement signal. And it also is like it only has three things it takes input: energy in entirety, hardware, like what the hardware Riverside needs to like service all of its customers, divided by, you know, customers in minutes or something like that and how clean or dirty electricity the Riverside's using. And you know, if it does stuff that makes it use more cleaner electricity, then you know what its carbon per minute should go down. And so like that's kind of this measurement that we've got. And again, one of the things I believe is like if you just gave engineers this way of measuring, if you just if I told you your SCI score for your product is 1.5 and I want it to 1.4 in six months time, you'll just figure out what to do. You'll just figure out what to do. Yeah.  

Tony [00:41:29] So when that factors in... The energy types. So for instance, I'm really excited. In two weeks, after a very long time, my house is going to have 75 solar panels. My old house in Texas had 75 solar panels. My one in Florida is going to have 75 solar panels. I'm going to have three batteries, lithium ion batteries to try to get off the grid. So it's going to be really cool.  

Asim [00:41:50] Nice, wow!  

Tony [00:41:51] How would that affect my SCI score? Because I theoretically, I hopefully am only using solar power, but I've got a massive computer with three GPUs, I've got TVs, I've got, you know, Apple products, Apple TVs, iPads, WiFi. Does that...how much does that affect the carbon output? Because I'm still outputting carbon, right? I'm still using electricity, even though it comes from a cleaner source.  

Asim [00:42:15] You've touched on what is like the edge case that the most complicated edge case to to deal with. And I believe the way the team has resolved that is it fortunately won't make an impact. We will still you still go by what's called location base, which is what your local grid will be doing because of, if I remember rightly like, you would actually have more impact... I don't think you could do it in the U.S.. Because you do in the U.K., I can actually feed electricity directly to the grid and make money on my house like I can feed solar to the grid to make money.  

Tony [00:42:49] You could do it in Texas, I don't know if I could do it here. I got like a $20 check after living in that house for three years, because the first the first year and a half I wasn't on solar. The last year and a half I was so like, I made back all the energy that I had spent the first year and a half after another year and a half. So the person that bought the house is doing great! 

Asim [00:43:12] Like Texas and Florida, you actually get what you want. Like we get four weeks of sunshine a year where I am.  It's going to get wonky, but it's actually like arguably if you put your electricity back on the grid would have like a more positive output. And just to avoid if we if we allow that in it, it opens the door to gaming the algorithm in a way where we'd be constantly going, putting adding things to spec like except, except, except, except, except. So we just like had like this really hard rule, which is like, just use what's on the grid. Like, even if you have like a, like a wind farm, although we had like a talk the other day... I'm still thinking of it Tony, like nuclear power datacenters like, it's like a really interesting idea. It's like because you could man that's going to be an interesting one like if it literally was a nuclear powered data center, man that would be so interesting. I think you have to argue, if I'm just thinking out loud right now, if it's a nuclear powered data center and the only grid was the data center, you could argue that's the location because there's nobody else can access the energy. And therefore you could argue that data center is true zero emissions.  

Tony [00:44:27] Because it's off grid, essentially you have no wires going in.  

Asim [00:44:30] Yeah, no wires going in and therefore probably even have to be like treated as its own grid. I don't know. I think that's a really interesting idea. Somebody, somebody out there should go Bill Gates Breakthrough Ventures should go away and like take one of these nuclear power plants and stick into a data center and see what happens.  

Tony [00:44:47] That would be interesting. I wonder how much, I saw the thing about how they're how they've got the kind of more modern nuclear reactors and they're what, they're building them, I think they're building them in China, right?  

Asim [00:44:56] Yeah, because they're the only ones that are amenable to it. I just I won't go into the nuclear debate, but I'm a big fan.  

Tony [00:45:02] Yeah. And I guess, I mean, solar just doesn't solar, wind farm, maybe wind farms, can they generate enough energy to to support a data center?  

Asim [00:45:10] Sure, I'm sure. As long as you got a large enough area, they absolutely could. The problem with solar and wind is that whole idea of variability, because if the wind stops blowing, like, what are you going to do with that? That is datacenters need like constant...there's things you can do. Like there's definitely like an amount of variability and I think data centers need to have like a certain amount of flexibility. And if you can take advantage of that, you can do like carbon aware computing. But by and large, the scale of data centers you couldn't like, if the wind stopped blowing, you just they had to turn off a data center. That's an outage. Like it just, you know, you need to, like, have like a constant source as well. Yeah.  

Tony [00:45:47] Yeah. And I don't I don't even know the number off the top of my head. But I do remember that the hardware architect of the data center that we had for, for AI, he used to joke that the data center itself could power, I think 150 houses. We were trying to do the math. How many houses could our one data center power?  

Asim [00:46:05] Well, it's interesting. The top cloud providers in the world, I think there's a top I'll have to make sure I get the right. It was it was from a talk from Adrian Cockcroft, he used to be the VP of Amazon Tech Sustainability, I don't remember exactly what his title was, he gave a talk recently and in it he said that the top cloud providers at least Microsoft, Amazon, Google, I think Alibaba and a couple of the others, if you counted all of their renewable energy purchases, they would be bigger than the 10th biggest country in the world, if that makes sense. So like the country level is country level purchases.  

Tony [00:46:38] So when you say renewable energy purchase, you mean they're actually deploying renewable energy or they are... 

Asim [00:46:46] No  everybody is just buying... It depends on organization by organization. There's these things called RECs, which is renewable energy contracts, I can't remember exactly what the, they might be credit or contracts. They are like literally like market based instrument you can like buy the like there's like financial markets as trade as a trade in them. You know, just the way trade and swaps and like options and things like that. And so you could just you would let you just buy them and then the renewable energy providers and selling them onto the onto the secondary marketplace. But what most of the bigger names, what they what they tend to do is they actually like do something called a PPA, which is somebody like wanting to build a wind farm and then looking for an investment. And they'll say like we have a power purchase agreement and then like Intel might go along and go like, we'll pay for your PPA, PPA, we'll buy it will buy will guarantee to buy all your electricity for 20 years. Then that windfarm can then go away to a bank and go Intel's going to buy electricity for 20 years and then bank goes here's money here's money go build your wind farm. Then they go build a wind farm. And that is actually a much a much more preferable way to like lay the claim of electricity, of renewable energy. But then you can you can more like directly link what you're doing to like building a wind farm, right?  

Tony [00:48:03] So renewable energy certificates.  

Asim [00:48:06] Certificates, yeah there you go. 

Tony [00:48:07] But yeah, that's still kind of I still find that not satisfying, I want to say frustrating, but not satisfying in the sense of I don't want.  

Asim [00:48:18] Are you? You want... 

Tony [00:48:20] Yeah. I want you to actually go build it, make sure that it happens. It's one thing to to say I'm I'm generating renewable. I'm being a good steward of the environment because I'm throwing money and telling somebody else to go solve the problem versus actually doing the investment directly and solving the problem. Right. That's I think though, at the end of the last podcast, I kind of had a little rant about making sure that companies really can measure the amount of energy they use versus the like the actual amount of energy they use, the carbon that they're emitting versus the, "here's the number that I have once I've paid X amount of money to therefore subtract off all these things. It's like what you were talking about with when I'm measuring stuff in software, when you're doing your SCI measurement, it's not fair to say, hey, I've, I.  

Asim [00:49:11] It's like columns of a spreadsheet.  

Tony [00:49:12] Yeah, I don't want columns on a spreadsheet. I want actual numbers to see what the impact to the environment, especially because my data center, so the data center that we had, for instance, if I buy offsets somewhere else or somewhere else in the world, it's still emitting carbon. It that actually matters where this emit happens.  

Asim [00:49:29] So what, so one of the things about what you just said is that's actually quite interesting is the way the way renewable energy credits is kind of accounted for right now is you do have to purchase them, I don't know if you have to purchase them or whether they are whether the...there's a group called RE100, which kind of all the companies that kind of agreed to to be 100% renewable power. They're kind of agreed I should remember if if if if it's an agreement as a member of RE100 or if it's actually part of the greenhouse gas protocol, but you have to actually purchase the RECs on the same grid where you're consuming it from, if that makes sense. However, however, that's actually not a good thing to do because if you were to purchase those RECs in India where there's almost no like a much, much fewer renewables, you would be doing more good. You would be you would be forcing fewer carbon molecules into the atmosphere. So now, like a movement to kind of like start asking those questions, like it's basically you basically ask the question, I've got a pot cash, where do I put that pot of cash has the best impact?  Not where the accounting me insists I put the pot of cash. It just goes to show that this is an extremely complicated space. It's just an extremely complicated space. Yeah.  

Tony [00:50:47] When I think about it, I think that it makes sense to say, let's like I said, it makes sense to say, let's offset the emissions where I make them. But you're right, if I'm actually if the way that I'm actually doing the offset is throwing money at the problem, let me put my money where it matters the most. Well, so we're almost out of time and I always end with the, where do you hope technology goes in in a few years? And obviously it will be something around sustainability. So what's the what's the the near-term goal? Or maybe, what's the long term goal? Obviously, the long term goal is to help us reduce emissions somehow. But what's on your mind? What do you think that we can a ffect?  

Asim [00:51:27] We started to call it like impact optimization and impact measurement. I think that we need to start measuring the impacts of stuff that we do and we build and we use beyond. How much money is it going to make? You know, it's going to be like what the carbon emissions actually after carbon emissions is. You know, we're heading into a world where water tension and scarcity is coming to the fore. Like one of the interesting questions, it was it was raised, it was about carbon taxes. A lot of these especially coal power plants, they they emit kind of, you know, very health damaging chemicals to the atmosphere. And there's lots of studies that can now show the, you know, deaths you see per thousand like increases significantly around like coal power plants and trade's off. So like what if an impact was deaths? What if, like, you're deciding to build a website and you could use Times New Roman and it's like zero deaths? Or you if you want to really like, go for it and put like, you know, Verdana in or something, but that's going to be like four deaths per million views. Like impact measurement I think is going to be I really do think it's going to be the future. We really need to get better at it. But like, could you imagine being part of that dev meeting? Like, I really want Verdana. It's worth it. It's worth 4 lives, you know, this is kind of like, there's a, there's a real like cost to this, you know, but we don't really think about it. And I think that's what's going to come. That's what I want to happen. That's I hope that happens is a lot of these non obvious kind of financial costs and risks. And these are the things this is just hopefully just going to surface and become a lot more prevalent and we're going to be having a lot more of these trade off discussions, really serious trade off discussions.  

Tony [00:53:18] Yeah. So being able to measure things and then importantly being able to have the people who can do something about them see those measurements.  

Asim [00:53:25] Exactly. Yeah.  

Tony [00:53:27] All right. Well, thank you all for joining us. It's a wonderful conversation. It was extra long. And the nice thing is, I'm not going to cut hardly any of it because it was just fantastic conversation. It's going to be extra long. So I'm sorry, listeners, I hope you're still listening, but I hope you learned a lot about green software. And hopefully if you're a developer, you think about different things that you might be able to do to impact software sustainability. And if you are, check out Green Software Foundation, check out Asim's, he has his own podcast. What's what's your podcast called?  

Asim [00:53:58] Oh, this the foundation's podcast. And I would say like Chris Adams from the Green Web Foundation has been the main host on it for a while, but it's called Environment Variables, which I love the name.  And also if you want to, if you want to just just a last plug, if you if you go to the Green Software Foundation's website, if you just go to the landing page, we've got a newsletter If you want to find out anything as a first point of call, just sign up for the newsletter, we have a weekly newsletter and we'll just keep you abreast of everything that's going on in this space.  

Tony [00:54:27] All right. Well, thanks for joining us Asim and thank you, listeners, for joining us.  

Asim [00:54:31] Awesome. Thanks, Tony.