Article
Nexus Pro
min read
James Dice

Episode #29 reaction: why Google created their own ontology

December 3, 2020

Happy Thursday!

Welcome to this week’s deep dive exclusively for Nexus Pro members. It’s an honor to have you here. This deep dive is a follow up to my recent podcast conversation with Trevor Sodorff of DB Engineering and Keith Berkoben of Google. I learned a lot from this conversation and want to share my takeaways and the full transcript with you below.

In case you missed it in your inbox, you can find the audio or video here:

Nexus site | Apple Podcasts | Spotify | YouTube | Add to other podcast apps

Enjoy!

—James


Outline

  • My reaction, including highlights
  • Full transcript

My reaction

This episode on how Google thinks about smart buildings is a good one to pair with Emmanuel Daniel’s episode from a few weeks ago on how Microsoft thinks about them. My take is that Microsoft is thinking about solving the problems on their campus in a way that can be productized, whereas it seems like Google is solving them for themselves, while open-sourcing some of the methods and not thinking about products much.

I enjoyed hearing that the impetus for the DBO was in Google’s effort to do advanced supervisory control—at this scale, you can’t help but try and centralize and standardize control sequences, setpoints, and schedules. This gets at how the data model is only as good as the use cases it enables. Keith said that Haystack and Brick are not “opinionated” enough, don’t contain enough “composable functionality”, and don’t have the tooling to allow the supervisory control sequences they want to enable at their scale.

Next, I think Trevor (and perhaps DB Engineering) and I are in a solid and healthy disagreement about analytics that I plan to follow up on at some point. As Trevor said, he sees FDD as physics and you can’t put a patent on it, so therefore it’s not a product. The list of rules and the math behind the rules are not where the value is. Taking action is the key. I agree that action is important, but I think he’s missing that getting to the #1 best action, the highest priority, is easiest with the best FDD products and that’s where the true value lies. Otherwise you’re left with, as he said, 5,000 faults per day to sift through.

Finally, I’m sort of left with more questions around ontology convergence than answers at this point. Keith said probably none of these standards are going away, but they should all be able to be represented using RDF, which is the master set of primitives that allows you to represent anything.

My highlights:

  • James’ favorite question - Keith’s straightforward take; Trevor’s frustrating experience with vendor lock-in / interoperability - tied together with Keith’s analogy to the airline industry and electronic medical records (6:03)
  • Impetus for Google’s digital buildings project and where it fits with the company’s broader initiatives (10:31)
  • Walking through the progression of their rollout (21:12) and why they chose not to use other industry standards (24:19)
  • given creation of own ontology, vision for industry convergence (45:15); and response from the industry at this point (49:16)
  • diving into the applications, and DB Engineering’s unique perspective on FDD (51:11)
  • Lessons learned from rolling out this project at scale (58:47)

Full transcript

Note: transcript was created using an imperfect machine learning tool and lightly edited by a human (so you can get the gist). Please forgive errors!

James Dice: [00:00:00] Hello, friends. Welcome to Nexus, a smart buildings technology podcast for smart humans. I'm your host, James Dice. If we haven't met before, I write a weekly newsletter on the same topic. It's also called Nexus. Each week I share what I've learned, my opinions, and what I'm excited about in the quickly evolving world of intelligent buildings. Readers have called Nexus the best way to stay up to date on the future of this industry without all the marketing fluff. You can check it out and subscribe at nexus.substack.com or click the link in the show notes.

Since starting the Nexus newsletter, many of you have reached out to me wanting to talk shop, and we have. After a few weeks of those wonderful conversations, I realized I needed to record and share them with our growing community. So here we are. The Nexus podcast is born. This is our chance to explore and learn with the brightest in our industry together.

Episode 29 is a conversation with Trevor Sodorff of DB engineering and Keith Berkoben software engineer at Google. These are two of the people responsible for rolling out the smart buildings program worldwide at Google. We talked about many of the topics we've been covering for a while here on the podcast, like analytics. Data modeling and supervisory control but this conversation was about doing all of that at scale doing it worldwide doing it on new buildings so how's Google doing that and it's pretty impressive. Of course, we covered the digital buildings, ontology effort as well, where the team is trying to make a lot of their data modeling we're public so others can take advantage of it All right, Keith and Trevor, welcome to the nexus podcast. I'll start with you, Keith, can you introduce yourself? for us?

Keith Berkoben: [00:01:51] Sure.

Uh, my name's Keith , I'm a staff software engineer at Google, and I'm responsible for building out our internal, smart buildings platform, and doing a lot of the work, related to, modeling our buildings so that we can, write software that's portable, across our fleet.

All right.

James Dice: [00:02:10] We're going to nerd out about that topic, Trevor. How about you? Can you introduce yourself now?

Trevor Sodorff: [00:02:15] Sure. My name is Debra . I'm the director of engineering analytics for a company called . DB engineering We're located in Redmond, Washington, and we do smart building consulting for fortune 50 companies and governments around the world. Google happens to be one of our clients and my specialty is in the acquisition organization. Utilization of data at scale. And that's my primary focus.

James Dice: [00:02:39] Cool, cool. All right, Keith. So back to you, um, I wanna hear how you got to smart buildings. So you're working in smart buildings. Now. I bet you didn't start out thinking, man, I'm going to make buildings smarter.

Oh. And you started out your career. So how'd you, get here.

Keith Berkoben: [00:02:52] You know, think the thing that was really interesting about how I got here is that, I did this like junior year group engineering project, in college that was like optimizing a, uh, HVAC system for, for one of the buildings on campus.

And I was like, Oh, this is kind of neat, but never like, thought anything of it and then realized, you know, 10 years later, Holy crap, this is my job. So, so that was sort of odd. Um, I, I fell into this, I think. because, I've always done, a lot of different types of engineering. So, you know, building stuff, software engineering, mechanical things.

And, so I. I immediately felt comfortable bridging this gap between these mechanical systems and all these things in the building and the building trades and things like that and the software. And so it was just kind of a natural fit.

James Dice: [00:03:50] Great. I love that phrase, bridging this gap. that's kind of what Lexus is all about.

How about you Trevor?

Trevor Sodorff: [00:03:56] I studied mechanical engineering in college. I went to Washington state university where they specialize in manufacturing processes, which I really didn't like I got on with a firm in Redmond called DB engineering. And that's where I currently am.

And I was doing a lot of energy optimization for buildings in the kind of conventional retrocommissioning process where.

And you step into a building. You look at the BMS, you look for a fan coils that have failed chill, water valves, or air handlers that are running too often. And one of my primary responsibilities was quantifying those, right? As, as you pass through, if you're doing like utility, right? And so I spend a lot of time doing that, pulling data, organizing it.

And I found that, you know, a lot of these things are, are something that you could replicate multiple times. I ended up doing the same analysis over and over again, and it was kind of, you know, it came a pain after a while at the same time I was working. for one, I'm doing this for one of my clients who happened to be Microsoft.

And they were really interested back in 2011, 2012, and in getting into the smart building space, because it wasn't really a thing at the time, but, you know, you wanted to be able to integrate all that data. They understood the value of the data. And so

I was involved with helping them to implement that platform, organize their data and then write fault detection around it.

And so I was able to kind of dip my toes into the mechanical waters. And into the software side. And I mean, it was, I found out going through the mechanical,

systems, they kind of suck the whole industry is, you know, especially at that time, it was a little bit backward, right. Everything's proprietary, locked down in its box and that was just drove me insane.

And so. As Microsoft started to expand that spent a lot of time working on, you know, advanced fault detection and other analytics that we wanted to do at scale, help them do a bunch of projects around the world for themselves. And for other customers, uh, got involved at Google when one of the directors at Microsoft,

moved down there and started working on that campus.

And I met Keith. I saw that they were trying to go down the same sort of path, but using more of a homegrown technology, as opposed to like an off the shelf product, like you might get with like a switch of sky spark and Iconix or whatever. Okay. I

thought that was interesting.

Okay.

James Dice: [00:05:59] It sounds about like something Google would do. All right. before we get into that, the last 10 episodes, or so I've been asking this question, which is my favorite question of the podcast, which is why is technology in our buildings? You know, decades behind other industries. so Keith, I'll start with you.

What's your perspective? Uh,

Keith Berkoben: [00:06:18] can I just defer that question to Trevor? Um, though I think the, Physical buildings just last for a long time. Right. And there is, you know, all of the incentives around how you finance buildings and how you build them and maintain them, uh, discourages you from replacing stuff that's already there.

And so, as a result, you built a building 25 years ago and the stuff you put in there, it's still there. Totally. Yeah. That's

James Dice: [00:06:47] a simple answer. What about you, Charlie?

Trevor Sodorff: [00:06:49] Yeah, I mean,

they're starting to do this more and more today, but you know, back when I started, the

idea is that you're kind of, as a building owner, you're

locked in with a manufacturer group of manufacturers, basically for the life of the building or for the life cycle of the BMS.

And you were kind of stuck with their shortcomings, the concept of interoperability really wasn't there

and

We're starting to see that progress pretty heavily in the, in the tech space,

but because of, I think the business

models of a lot of the, legacy.

you know, controls providers, mechanical providers and stuff like that.

It's like they want to lock you into their box.

And I,

I mean, it's not, the direction that we want to go over the next 10, 20 years. And, um,

yeah,

it's a big shift to try and

steer right.

James Dice: [00:07:31] I can feel your frustration because you and I have a very similar career path and basically starting with energy, becoming mechanical systems kind of experts, and then getting into software and then realizing that basically the data is not free.

And that's the main piece of it. When it comes to energy engineering, you need the data to optimize the system. to analyze it and the data is not free. And the word that Andrew Rogers used is probably like seven or eight episodes ago on the podcast was he said building owners are not able to self-actualize and I love that phrase as far as like I'm trying to do stuff in my existing systems and existing vendors, usually because of their business models, like you're saying are, are sort of not letting us progress the way we want to progress.

Trevor Sodorff: [00:08:14] And I wouldn't say

that it's all like malevolent or Like someone's just technology is old, right? Like I've been working with one client trying to integrate systems that are 20, 30 years old and, you know, Just how, how can you

possibly do it?

The available data that even the underlying systems has, like a lot of it's, it's a technology problem. As

much as it is like a.

James Dice: [00:08:34] A business model, problem

Keith Berkoben: [00:08:35] and motivation. Yeah. you know, the thing you made me think of actually is, um, EMR, electronic medical records, or even like airline reservation systems is another great example. And, when you have a huge ecosystem with embedded technology or entrenched technology of significant complexity, right.

Yeah. It always ends up being easier to be backwards compatible than it is to try to migrate the thing that already exists. And so what you end up is you have like these core aspects of the way these systems work back then being a great example, right. That are, you know, I don't know exactly how old backnet is.

Maybe it's 40 years old. It's not a young protocol, right. Is still endemic to the industry because it's. Just been backwards compatible, you know, systems that have been added for years and years and years. and that is just the nature of something where you have physical infrastructure that, you have to deal with.

James Dice: [00:09:38] And how does that show up with like, just for the people that are new to technology, kind of like me, how does that show up with electronic medical records?

Keith Berkoben: [00:09:46] Oh, well, it's that you have, all of these hospitals that have huge systems to deal with all these, records and, You know, if do, as a, technology vendor, come in with something new and it's not backwards compatible with what they already have, it's immediately a non-starter because no one wants to invest in upgrading the system that they spent a bunch of money on, that they already have.

And so you see that the airline industry is just now moving past systems that are, you know, from the seventies or maybe even earlier, right. And similarly with EMR is it's very hard to get onto modern platforms because no one wants to invest in migrating.

James Dice: [00:10:25] Totally. It's fascinating how these parallels happen.

a news industry. So I let us all right. Let's, start talking about Google then. so we're talking about smart buildings today, but let's start by kind of putting that in context and this overall, like what is Google trying to do? And specifically I want to ask about, um, I had this really cool article in the newsletter.

I don't know, four months ago where it was talking about how Google is shifting data center loads across the world and real time to basically, Match up the loads with the cleanest power at that time as a strategy to get to, some sort of carbon neutrality or, you know, positivity at some point.

Right. So can you kind of paint us where this fits in the context of, of Google strategy, Keith.

Keith Berkoben: [00:11:11] so there's, there's a couple aspects to it. And actually the carbon one, is, you know, kind of new in a way. the bigger one is that, Google operates at such a scale that we're often, the largest customer of any given system in any domain, whether it's, you know, a physical security system or, you know, uh, HVAC head-end or, you know, et cetera, et cetera, et cetera.

Right. And to the point where we start breaking these things and, scale really is. with traditional approaches, oftentimes a barrier, and the whole goal of what we're trying to do in the smart buildings world is to turn that on its head and use the scale to our advantage rather than have it.

Hold us back. And then of course, recently with our, you know, newly announced initiative to be, you know, zero carbon by I forgotten the year now, right now, not only is this kind of a philosophical thing where like, we would like to be able to build applications that scale for our buildings, but now we really have a very strong incentive.

you know, to deliver something very specific with those applications, which is energy savings. So

James Dice: [00:12:27] let's talk about this breaking systems. tell me a story about this. so how many buildings do you guys have, on a given campus?

Keith Berkoben: [00:12:33] on a given camp.

Well, I mean, so I think in the Bay area we have over a hundred buildings. I don't know what the exact number is. And I probably couldn't tell you if I did, Siemens

James Dice: [00:12:45] to Sega or like a Johnson Medicis and you plug in a hundred buildings to that. And you're saying just the scale like these.

Keith Berkoben: [00:12:51] Well, I mean, so, so the fact is that we don't even try.

Right. I think that, You know, for the Bay area we are running. I don't know, Trevor probably knows the exact number, but certainly more than a dozen separate, web control, instances for HVAC. Um, Because simply they would blow up if you tried to do anything else. Right. And so that kind of thing gets really bothersome when you have hundreds of buildings globally.

James Dice: [00:13:20] Got it.

All right.

Trevor Sodorff: [00:13:22] Plus new construction

and that's

that's. The other thing is that Google is growing at the same time. This is happening. We're not talking about like, company that's kind of at status and it's just the same campuses being maintained. It's they're, they're doing a lot of growth and so.

You know, we're dealing not just with existing legacy buildings and legacy systems, but we're also dealing with net new buildings and processes around those with BIM and with, you know, different IOT style approaches. Right. I don't know if you want to talk at all about IOT light versus IOT, heavy Keith, and as part of what you guys are doing, but like just the different philosophies and they're all kind of there at the same time.

James Dice: [00:13:59] Hmm. Okay. Before we dive into that piece, just help me understand how Google. Operates buildings, like, do you guys have your own separate facility management group? You hire that out or how does that

Keith Berkoben: [00:14:10] show up? uh, it depends where regionally, we are big enough that there isn't one answer.

in, the Bay area, we try to centralize it. So we have a central operations center that. fields, all of the issues and, you know, has big screens on the wall and, and such like, we contract out a lot of the, actual operations work to, various companies. Okay. Um, but it varies cause we have sites, some places in the world where you have, you know, one building in a country, and then you have other places like the Bay area where we have.

You know, over a hundred buildings in a region. and so part of what we're thinking about as well, how do you, homogenized, the way you deal with that, even though you have such big structural differences in that, not sure we exactly have the answer yet, but it's something that we're thinking about.

James Dice: [00:14:59] Yeah. So this was basically the impetus for. This do you call it the digital buildings project? Is that what you call it?

Keith Berkoben: [00:15:07] Yes. So the input is that's the GitHub project anyway. Okay.

James Dice: [00:15:12] All right. Well, that's good enough for me. So the impetus for this is basically we've realized we have all this diversity of systems.

All of these systems can't really handle our scale. We have different ways of operating worldwide. So what's the solution. Like what, where are you guys headed? Uh, given all of that and the given your goals.

Keith Berkoben: [00:15:31] so my opinion, and I think Trevor probably shares this opinion is that, there are few, high value things that, that you want to do in your building and that you can build, you know, a supervisory control apparatus that allows you to do those things the same way.

In every site, and that captures a huge amount of the value of. your building systems and all of this data that you're getting without requiring you to deal with all of the complexity and all the differences involved the things across all of the buildings. And so that.

Really in my mind is the strategy is basically how do you make it so that this high value thing that you care about, whether it's, you're, you know, air handler, supplier temperature reset, or, occupancy based control of whether or not you're running a system in a particular place, um, can be done the same way across the entire fleet.

And when. you turn on a new building, all of those features and all of those, you know, analytics that you have for all the other buildings just work in this new building that that's the Holy grail. I love it.

James Dice: [00:16:53] an Trevor. The reason I'm so excited about what he just said is like our history, as energy engineers turned analytics professionals, right?

The progression that everyone gets to when they head in that direction, Is that, you know, I'm doing a bunch of stuff that could be automated. Like you said earlier, I'm doing it over and over again and spreadsheets and hobo meters and things like that. Right. And then you, go into, okay, now I'm going to put analytics on top.

I'm going to do some analytics and then. very next question then is, okay. Analytics discovered like Keith just said a supplier temperature that is not resetting and it should be okay. Well now I have to go talk to that vendor and it convinced them why they need to do the reset and, basically pay them or figure out if this.

Service contract covers their time, and then they say it's done and then it's not done. And then I go through the entire thing again. So, so how are you seeing this in terms of helping this sort of supervisory control get implemented, you know, throughout the,

Trevor Sodorff: [00:17:50] well, so I think we want to, talk about two pieces of this problem.

One is the end use, which is a kind of a problem everywhere. And one is how you get to that end use. So like with, Google, what they've been doing, they have a lot of control over their own systems. It's not like they're at the behest of like manufacturers for a lot of these things. They have their own controls group in-house and then they partially subcontract.

My team will, uh, with the data that comes out of the Carson system, which is part of the boss system, which is the overarching thing that Keith is kind of referring to with their strategy. Um, we take that data, we'll build like dashboards or analytics on top of that using their base data. And then those are just automatically updated refresh as the data

that comes

in.

And we'll have engineers both with my team or with other people around the world that are using it. Or with, you know, HVAC techs or whoever's looking at this, there'll be able to use and interact with it and make the changes. And we do training to try and help them understand why they should care about these things.

Google has a pretty good handle on how they deal with that. And, they're really good about like, not making everyone's responsibility, like fault rules, right? Like one of the, one of the things I've seen done elsewhere is, you know, Oh, all the HVAC techs are responsible for their quota of faults that they have to get fixed in

Keith Berkoben: [00:19:04] a month.

And

Trevor Sodorff: [00:19:05] that tends to not be very efficacious because like they don't care. They have other motivations pothole called one of them. Right. And so if it doesn't reduce the amount of hot cold calls they have, they don't really care. It's kind of a waste of their time. You're better off having a strategic group that is responsible for these things that has the tools available to be monitoring these things, or in an ideal case, a system that can automatically respond and posture the system or adjust, which is something that we're moving towards with Google.

But the piece that underlies all of this is the structure and content of the data. And I don't mean just in terms of what data is available. Like, Oh, the air handler has a static pressure and a static pressure set point in a VFD speed. Right. It's also in. I've got this air handler that has a fan wall, and it's got effectively two or four fans versus this one with one fan.

Are those analogous? And how do I teach a machine or encode into a machine that these two things are analogous and that I can treat The analysis or the response to the same, that piece of it is something that requires some rigor, right? You can't just like supply fan one supply fan.

Like there, there has to be some underlying meaning to supply fan versus discharged fan versus exhaust fan, if it's some sort of contextual meaning. And so what we realized as we started rolling this out is that the use cases are there and the scalability. In terms of connection is there, but the scalability in terms of organization and the robustness of that organization is the thing that is really at the heart of the problem that the industry sees today, in my opinion, right?

Like I hate organizing data. I really hate it, but I know that if we don't organize it properly using best practices, then you're going to regret it. And you're not going to get the use out of the system that you thought you would

James Dice: [00:20:51] enable. What. Keith just described, which is, um, I'm trying to set a new control standard throughout my entire portfolio.

And you can't enable that unless all of the systems are modeled properly. So, you know, which sequences apply it to which systems and that kind of thing. So I want to pause on the modeling question, cause I know we're about to get into that. Just pause on that real quick. So when you're, when you're thinking about rolling this out, though, paint the roadmap for me. So what I'm picturing is we're getting all the data together. and then what, and I think what I'm hearing though, is you're going from, the data's together. I'm gonna model it somehow. We'll get to that, and detail on a second, then you're adding visualization analytics onto it.

And at some point you're, you're adding supervisory control as well. So paint to me like how you guys are like, rolling that out. And if that's not the correct progression, correct me

Trevor Sodorff: [00:21:41] well, so the, the analytics and that piece are really somewhat independent of the rollout because like the use cases are going to vary from region to region. And we've got a lot of use cases built and in production in the Bay area. And as we start to roll this out globally, like,

They tend to be a little bit

different. And so, that of it is I think, a little more straightforward because you know what you'd want to do with the data, if you're a facility manager in the UK or in Singapore or wherever, if you had it.

And so the real question is how do we get it? And that is the piece that Keith and I have really been

focusing on.

because we know that's the hardest part. Like that's the most manual human reliant part, everything else after that is gravy, right?

Like you can roll out a new application or a new analytics component once you have that. So how do we enable people to apply a model? That's. good rigorous and in the format that we want and they don't have to have a lot of information about the underlying ontology in order to do it again.

I have to have as much familiarity. And, uh, so the way that we're approaching it, I think globally, Keith can talk to this is we're trying to enable MSIs around the world that are working with, Google to perform their new construction, energy projects or whatever. Uh, we're trying to enable them with tooling and with this project and with processes defined as part of that project to ease the project so that they can.

They understand up front, like these are the types of systems that we want to integrate. They have whatever they do to integrate them and if it's Jace or, or whatever it is. And we give them a format, like a template for what we want the ultimate model to be in. And we provide tooling through our project that validates that any extensions that they want to make to the model or any instances of the model that they want to give us as part of like the final delivery handoff for that project.

James Dice: [00:23:22] Okay. And

Trevor Sodorff: [00:23:23] we can talk about that in more detail, but in my mind, like the analytics piece is everyone in the world knows about analytics, kind of how to do analytics, how to scale analytics. it's the piece in front. Like how do you get a system to have that organized data? How do you have a model that gives you something that you can then extend on top of.

James Dice: [00:23:43] Yeah. Yeah, totally. Let's dig into that piece. so it sounds like major thrust of the project, if we could call it that it's a bunch of projects that are happening together is just getting systems online, getting the data flowing, freeing it up, using local MSIs to do that in a standardized way.

Yeah. Okay.

Trevor Sodorff: [00:24:01] Yeah. That's that is the goal. and we can talk about this a little bit because it's not been all like, you know, flowery goodness, it's hang ups to it. And part of what we want to be extending our project to do in the near future is to help reduce the friction associated with this more rigorous model.

Okay. And, you know, a lot of the, flack that we get for this project is like, well, there's brick and there's haystack. And, you know, I think Siemens has one, like, why aren't we just using one of those? Well, part of that is because there are things that we want to be able to do in support that aren't fully supported within those models.

And there's, in our opinion, not adequate tooling around the enforcement of extension of those models. And I mean, this is your area. Maybe there's something you want to,

James Dice: [00:24:49] we kind of just set the stage for someone that doesn't have that context, of why you, said what you just said, which is you realized, okay, I have all this data worldwide.

I'm going to. Get it into some centralized location. I know I want to be able to do cool stuff with it later on, but right now this data has no meaning. And what I heard you just say, as you looked at all the other efforts to produce meaning in our industry, which is haystack, and brick, and there's probably Several other worldwide that people are like, well, why didn't you use our standard? Right. So, so Keith when you looked at that, what did you find what did you do? I guess it'd be a good time to save what the project is for the first time.

Keith Berkoben: [00:25:29] Okay. Well, yeah, so, actually I think first thing that's kind of relevant is what my context was when I came in.

Right. So I'm a software engineer, and I. Up until I started working on this while I had some ghetto kind of general experience and like what building systems are and roughly the things that they do. I had no experience as like an MSI or, actually working in the building. Industry. And so I came into all this stuff completely naive and I was like, okay, we need to model this stuff.

Cause I understand, you know, generally how you write software, you need an abstraction layer to do stuff. And I was like, okay, I got to find the same distraction layer. And I looked at haystack and brick and at the time, and the gap for me was usability. Right. Like, as somebody who didn't know anything about this, who didn't have any available tooling who, needed to do something quickly with rigor, those platforms just didn't offer what I needed.

Like they were. either, you know, they didn't have coverage in the right places or had too much coverage in another place and they weren't opinionated enough that I could do something with them that was functional quickly. there's

James Dice: [00:26:49] a lot of good and a lot of good terms here and I'll let you keep going, but I want to kind

Keith Berkoben: [00:26:52] of slowly and so, right.

And so, coming into us, I was like, okay, well, I have this. very specific use case, which is that I want to enable supervisory control and analytics at scale. And I need something that makes that easy for me to do. part of the art of modeling. Is being opinionated about something in a way that makes it easier to get to your ultimate application goal.

because if you try to model the entire world and there, Places you know, companies who have tried this, I think there was this really great, IBM effort to like create an ontology of everything at one point. And it's just like very quickly becomes so unwieldy that you can't do anything with it.

And so you have to be very careful about,

shaping your model

to really map to your application and what you need. and so we. Decided to build something because we were like, we want something that helps the user to get to that thing that enables the supervisory control analysis quickly. You know, and make it so it's hard for them to make mistakes.

And so that, kinda how we got to where we are today. I

James Dice: [00:28:14] feel like this is a missing piece of the sort of ontology or data modeling conversation in our industry. So Nick, I asked you from KTS buildings and I talked about this on, I think it was episode three, but his perspective is. That he's seen a lot of people say like quote unquote, tag data.

And then the, the myth, I guess, in the industry is that now that the data is tagged, I can do anything I want with it. Right. But the thing that people find out is then when I go try to do a slightly more advanced use case, right? Like supervisory control, which is an extremely advanced use case, you're then finding, Oh, I didn't quite tag it in a way that enabled that to happen.

Right. So is that, is that what you mean by opinionated or. W what do you mean by that?

Keith Berkoben: [00:29:01] well,

sure. So opinionated, in the sense that you have to decide, What things you care about representing from the real world in your abstract model? You know, so as an example, maybe if I only care about high-level supervisory control, cause that seems to be our meme in this podcast, right.

Um, maybe I don't need to model a PID loop for a VAV damper. Because I just don't care. Right. The only thing I'm ever going to change is the set point. And so, building a model that lets me do that, it doesn't necessarily have any value. I mean, I don't think we've had a decision about whether or not we're going to model PID loops, but the point being that, if the thing you're trying to do, doesn't need a certain piece of complexity, then the opinionation Of the model is, to basically, erase that complexity from the world so that you just don't have to deal with it. And you have to be careful about what you erase and what you don't.

Trevor Sodorff: [00:30:01] I would like to talk explicitly about some of the things that were missing from brick or from haystack, because I want to give concrete examples real quick.

Keith and I started collaborating on this. I want to say like, Year and a half, two years ago on this thing. And at the time there were a couple of drawbacks with both haystack and brick that we didn't see a workaround for without kind of doing some hodgepodge mixture between the two.

The thing that was missing from haystack, we thought at the time was a little bit of the rigor around definition. And the lack of relationships, most of that solved in haystack felt like they're doing a good job with, with improving the platform. But at the time, like that was a big piece that was missing.

But the ability to natively within that, tagging structure build in relationships, because that's a core component of what we need in order to be able to do, any sort of, system control. Right? The, on the commerce side brick at the time was sort of a new thing. And it

had the concepts

of, you know, relationships and entities and these things built into it.

Natively that was missing around. It was any sort of extension of available. They're not really tagged sets, but you could imagine like, standard field names. Okay. No, it's like, well, if we're going to adopt this one, we're basically going to build it all for them. Plus, and Keith, you might give some additional content here on why brick, like we model ours after brick, but we, extended brick

Shout about that a little bit, or,

Keith Berkoben: [00:31:33] yeah, I mean, so I think the thing that, we, did a, I don't want to say better, but that we invested more effort in. Then these other aspects was the idea of modeling above the level of. Points, and giving real structure to equipment and, composable functionality.

Um,

James Dice: [00:31:57] opinionation and tooling and composable functionality. Like we're going to be a glossary for this episode, but yeah. So what do you mean by that?

Keith Berkoben: [00:32:06] So, so as an example, And I'm a little bit fuzzy on the details now. Cause I haven't looked in awhile, but you know, brick, as an example was really atomic down to like the point it was a pure graph, right?

Like your piece of equipment. In the model was, you know, a bunch of connections between these points to entities that made up a piece of equipment. because it basically was RDF. Right. and For our purposes, that was too low level. cause it, just the amount of complexity you needed to like build up a biological piece of equipment that you cared about was, was really high.

And so we kind of. Jumped up one, step higher and said, okay, well we want this concept of a piece of equipment. We logically understand, you know, an air handler, a fan quote, a year, a VAV, et cetera. and we want to make it, easy to build one of those and model it. and so we sort of focus more around this idea of equipment and equipment with functionalities, where you could define a piece of functionality.

So like, you know, damper control, based on a temperature or something. and you could then say, I have a piece of equipment and it has this functionality and this functionality then. requires that you define these points for that functionality, because that's what you need to analyze that functionality.

Okay. And so then through composition of these things and inheritance, you can now, very rigorously construct things that, you can deal with programmatically.

James Dice: [00:33:49] So if I build on our meme here and I want to say. Implement, uh, static pressure control on all of the fans that can possibly do that worldwide.

That's kind of the approach you're saying is like anywhere I have VFD on a supply fan VAV boxes downstream, and a discharge pressure sensor. Now that component, and that is just an example that might not be quite accurate, but that component is now able to be pushed out. And now I can. Standardize that sequence as long as those components are there.

Trevor Sodorff: [00:34:24] Yeah. Yeah, exactly. And so our idea was that a think, well it is not equivalent to the sum total of its available points to its sum, total of composed functionality. Yeah. So say that I've got like fan control, it's going to have a start stop. And it's going to have like a status feedback, maybe some, feedback from the Ambridge or for something like that, that becomes one chunk, one composable piece.

Then I've got it. Does zone temperature control with dual set points, a heating and a cool access point. And it uses that to control the chilled water helm. That becomes one piece, the dual set point control can also be its own piece and you can compose it

of these kind of overlapping

chunks. such that you end up with the full list of required data fields whether they're optional or required or whatever, but you can essentially compose devices more quickly by just having a basic understanding of chunks of functionality that are reusable across fans, fan coils, air handlers, you know, shellers whatever.

That was, kind of our idea. And we've implemented that. The other thing that we've kind of found was missing through some of these other projects was a large body of precedent. So, you know this, right? Like with, haystack,

you know, to MSIs in two different places can have two different tags sets for the same thing.

Keith Berkoben: [00:35:39] Right. It's

Trevor Sodorff: [00:35:40] very common. So one of the things that we wanted to do as part of our project was all of the things that we're actually applying to existing pieces of equipment. We're going to make that part of the, ontology part of the model documents. So that like you have these tempers, one thing that we've seen before so that you can get an idea for like, Maybe I have this, fan coil with, you know, the discharge VFP speed control and whatever else associated with it.

And I can just take, or extended So that was kind of our core concept. There was no body of precedent that we can give to people and that they can kind of get an idea for what we're doing. And we have names associated with tags. Like you can imagine tag sets and, like what we call standard field names.

It's being equivalent And we just define those and we have different contexts where we apply them differently. And it's all just embedded there.

James Dice: [00:36:29] Got it. Yeah. This is something that I think it was episode 11, Corey, most men who works at NRL. is working on these similar concepts, uh, day in and day out.

We talked about some of these limitations. If anyone wants to go dig back into that episode, hasn't listened to it yet. we talked about a lot of these, sort of aspects. What did you guys mean by? I think I understand that concept, which is just like, let's give people examples of if they come across this fan coil, how should we do that next time?

Right. what are some examples of similar examples around this tooling concept? So you mentioned I have an MSI in, the UK, right. And I want to make sure that they're basically setting this building up in the right way, according to the standard. What do you mean by tooling and how do you sort of support that MSI setting up that building and caveat to that question though?

Is it, it just struck me as like, how massive you guys are in terms of scale is a really a great thing. If you're thinking about this in this way for our industry, right? So you're, you're basically saying we have 10 millionaire handlers and we're gonna just basically model out all of those, right. In a standardized way for everyone to sort of benefit from, which is a difference with like at least haystack, which is the one I'm most familiar with.

Right. You just said it, Trevor is like, What MSI is going to do it their way. And that's their sort of version of haystack. And that's probably going to be their little thing that no one else gets to benefit from in most cases.

Keith Berkoben: [00:37:56] Right? Yeah. so yeah, I'm gonna kind of discuss some of the more, conceptual aspects of it.

And then Trevor can talk a lot about, you know, the actual process, cause he's actually done hundreds of buildings now. so a couple of the things that were missing, I think, in the other ontologies, that we have, both methods and tooling for. are both dimensional units, which was a really big thing.

Uh, if you're going to actually do this, especially across the world, we have multiple units. And also the idea that, standardization, needs to happen, in the data path. Right. Because one of the things that doesn't scale is Enforcing that every piece of equipment, exactly models your current, data model, on the device.

And so a lot of the tooling that we built was, you know, structure around things like dementia units and validation to make sure that when you, bring that stuff into the model, you do it consistently, but also all the tooling to get. From your native device and its payload into the model.

So we have, the ability to effectively configure a building and, define all of the translations between, what we're actually getting out of the building, what we want it to look like and have that all happen in stream as the data is flowing.

Trevor Sodorff: [00:39:25] And so, let me break that down a little bit.

Cause that's pretty high level.

there, there are three things that we kind of anticipate wanting Q1 for and recurrent lead doing two of them. Uh, first thing, the first thing is how do I apply the model and know that it's like a correct implementation of a model. So I have an in Europe that is trying to build me this.

Effectively, it's an RDF model, right? it's in Yammel but we have the ability to convert it. So trivial difference. Right. But like this human readable model, I want them to construct it. They need to be able to construct it based on things that are already defined in the ontology today, once they're done with it, it gives me a translation between air handler one, two, three, with all of its different goofy available data points that map correctly by dimensional and units and by a direct one, one napping to something that exists in our ontology into a type that exists in our ontology.

When you start to do that, like how do you make sure that what they gave you was valid or all the relationships between the air handlers and the VAVs correctly defined? Are they using the relationships consistently when you need tooling to help you with that? Because no human is going to be, so we built that to the second peak because we have not seen everything there is to see in the world, obviously, woman doing this for a couple of years and it really

focused on,

the U S and part of Europe.

we know that they're going to need to be extensions to our ontology. We're gonna need to add new concepts, like, You know, certain types of alarms that are wonky in Europe or different types of control strategies. I don't think we looked at a chill beam yet. We haven't looked at commercial refrigeration and we want to add, adjust things as we go.

How do I ensure that when I do an update to the ontology, that doesn't break something that already exists or is in Congress for something we've already done.

So again,

you need some tooling and so. What we've done is we built this,

tooling that helps us to enforce the rigor of the model when you go to extend the third piece, and this is the thing that we have not developed yet, but that Keith and I have been arguing about and trying to figure out how we're going to tackle this one.

We've got some POC in place, but this is the place where we really want to focus. I think in the upcoming years is how do I automate the application of the model? Because I've got this air handler say I've got this CSR dumped from the, system. How do I convert that into this, into this Yammel file that thankfully gets converted into my, translation hierarchy or whatever.

today there's a series of things that are out there like brick has, plastering that Rico. I don't know if you seen that to do some like automated NL based, alignment, a lot of different people have their

own.

proprietary things, but an MSIs will tend to have some mix of manual and automated, like tagging or whatever neglecting has this built in, but there's no overarching like available tool set.

That's robust enough that MSIs can use it at that's also free. this is kind of like part of the secret sauce that MSIs have. And.

James Dice: [00:42:21] Labor integration labor.

Trevor Sodorff: [00:42:23] Yeah, exactly. And frankly, hate that. That's not something that we, as an industry had tried to impact more openly because like, you can imagine that when you are doing mechanical designs, these systems are coming about, you know, due to the heterogeneity of engineering conditions and.

So the engineers aren't limited in terms of the wonky stupid stuff that they can be putting in, in any building anywhere in the world. And so

I have to be flexible

and interpret it. Right. And, frankly, the whole idea of mapping the state is a waste of time to me. Like it should be done by a machine.

And my goal

is

particularly with regard to this and also with FTD and some of these other things that we kind of sometimes do by hand. Is to automate it away. I want no in the world in 10 years from now to have to build a load sheet or a spreadsheet that maps from one-to-one. I just, I hate it.

And so that's, that's one of the things that we're gonna be focusing on in the near future, because that's a missing piece.

James Dice: [00:43:18] Right. Love it. Okay. So I sort of have the picture for how this ontology came to be. Right. and so I have to kind of. I think they're related questions in that. Okay. So now you guys have open source, all the work you've done, right?

It's all on get hub. so the questions I have around. Okay. We've been using our use case of, of supervisory control as an example. What happens if there's some other use case that you guys don't use? That someone else that wants to use this new standard, they want to use it. Right. But the model doesn't enable that.

how were you thinking about that? And if you're not worrying about it, that's certainly okay, too. I'm just wondering how, if I'm a new organization and wants to just take advantage of the work you've done, but I have this use case over here that you guys aren't doing. W how does that work?

Keith Berkoben: [00:44:10] Um, so I guess it depends what your use case requires that doesn't exist already. you know, we definitely are, Interested in suggestions for, improvements and extensions. So, uh, you know, I'm sure we haven't thought of everything. and there are definitely some things that we've thought of that we haven't had time to implement.

So there's kind of that route it's like, maybe there's a core aspect to what we're doing that like, It needs to be added and, you know, there's a collaboration that needs to happen there. Um, the other thing that, could easily done if we're just talking about a model

coverage,

right. cause I'm sure that there's a good Jillian pieces of equipment out there that we're never going to see in a Google building that somebody might want to use this for.

those can either be contributed or can be extended locally. Right. it's very easy to adjust. Add new, namespaces or add new, entities to the model for things you know, that are relevant to your individual use case

James Dice: [00:45:08] flood people will contribute. If you're listening to this and you're going to add your own, why don't you just contribute?

so, okay, so second question, like follow up to that is. Now we have, the Google digital buildings technology that we have haystack at brick, and we have the other ones. How do we, as an industry get converged at some point, and is that, in the cards?

Trevor Sodorff: [00:45:31] It is, it is pre chronic youth.

Keith Berkoben: [00:45:36] So, I fundamentally, and if my, teammate, was here, he would be very adamant about this is there are, open standards for interoperability that are applicable in this domain. So RDF being the most obvious one and all of these things. Can and should be cross mappable, you know, between each other.

so that's kind of the first step is if you have a haystack model, you know, it is possible to make a transformation that makes it a brick model or digital buildings model for the most part. Right. Um, I think each one of these different ontologies is natively good at different things.

and so it isn't necessarily the case that you need to, use one. For everything, right? Because they are cross mappable. You could imagine that, in the future, all of the tools are, you know, RDF compliant, let's say so everything's natively using RD. Yeah. F as, As this model of the data and you use digital buildings because digital buildings is a really easy way to rigorously configure your model. Right. And then you spit out some RDF and then the schools use it. or maybe you want to use say sky spark or something.

Right. And so you can then go through RDF to haystack and now whatever tooling you get natively on sky spark, you can use it. and you know, you could think of a bunch of other use cases. So, well, convergence. would be great. You know, we have one thing that rules the mall. practically speaking, probably none of these things are really ever gonna go away, but we definitely can inter operate and should inter operate so that we have tools that can deal with, all of the things.

James Dice: [00:47:22] Got it. All right. I like that answer. So, so what is the thing that would be like the universal translator? In the future, I'm just imagining something that says, well, I can talk to Google and I can talk to pay stack and I could talk to brick. Right. is that how it would work or is it like, is it some other implementation?

Like the middleman middleware?

Keith Berkoben: [00:47:44] I mean, so RDF is, um, if you think about it in terms of language, it's like the, um, what is it? The IPA international phonetic alphabet. I think  I'm getting that right? Like RDF is that kind of master set of primitives that allows you to represent anything.

And so that's ultimately, you know, something that has to work with everything needs to be able to understand that. set of primitives and then you can represent all of these different models with those primitives. I see. but in terms of the, the interoperability to use language as a concrete example, um, you know, native, People who live in snowy places have lots of words for snow, right.

Because it's something that they know a lot about. And there's no way to like, literally translate all 22 words for snow into English, because we just, we don't have that concept. Right. And so there is. Always going to be some kind of manual interpretation between, you know, your model and haystack and your model and digital buildings or brick or whatever, because.

The concepts will not always a hundred percent map one-to-one. I found

James Dice: [00:48:59] that

every good, like ontology nerd has all of these analogies, just like in their back pocket in case they need them. Uh, and I, I love that Corey will love that too. And he's listening to this, uh, shout out to Corey and all his analogies as well.

Cool. Um, so what's the status of the project, right? So, I'm sure people are reaching out to you all the time. First of all, like, why didn't you use haystack or why didn't you just Brit, like they're reaching out about that, but then what are you guys getting from the community as far as contributions and that sort of thing, with the project right now,

Trevor Sodorff: [00:49:31] I would say the most part from the community, from what I've seen is, questions about concrete utilization.

Because like I said, one of the things that we don't actively support right now is like tooling, that makes, application the model very expressed or easy to, interpret. I think that's one of the shortcomings that we're still kind of dealing with. Like we believe the models

is very good, but that

it still requires a little bit of tooling to make it to the point where like the lowest common denominator of, user can, effectively take it and run with it.

some of the comments that I've been seeing are just trying to understand some of our core concepts. I don't know, Keith, what would you say.

Keith Berkoben: [00:50:08] so definitely seen, some interest from the other, ontology purveyors in terms of, uh, you know, sort of seeing which concepts that we focused on that were different.

and, I think that, that's been interesting, and. there's also, um, Brian Turner and buildings, a T has been actually working on kind of a universal translator. so that's, been sort of neat. His shop, I think is, very heavily invested in haystack and, his company also does fair amount of work for sure.

For Google. And so, uh, he's been, sort of trying to figure out the tooling that does this cross interpretation. So that's been pretty interesting. Got

James Dice: [00:50:54] it. I didn't know that that's what, Brian's project was before I asked that earlier question. Uh, so good to know. Okay. Very good. All right.

So a couple more minutes left here as we kind of close things out. I want to head away from that. So if you have anything else to say about the project, let me know. Okay. so one of the main things that we haven't quite hit on is. you, you mentioned supervisor control and you mentioned visualization the data.

We've mentioned FDD. are those the main sort of applications that you guys are developing today? and specifically, Trevor, I want to ask you something I heard from Jeremy, the other day, who's one of your coworkers. I was in a meeting with him and he said something along the lines of, FDD specifically.

You guys have a little bit of a different philosophy around it and how it's not a product. And I don't know what it is if it's not a product. So can you kind of explain that and how you guys are implementing analytics in FTD? once you do get the data model properly

Trevor Sodorff: [00:51:55] in my mind,

FDD is physics. Parents simple.

And you can't put a patent on physics.

And so like you can come up with a fancy way of doing valve vault detection or doing some, whatever you want to do. Right. But like anyone with a core understanding of first principles should be able to replicate it. I just, I hate the idea similar to what we're talking about earlier with, You know, somebody's making a career out of mapping data when a machine should be doing it. I feel like a lot of these things that we're doing in FTD, I happily gives all the FDA that we're working on to our clients. And even sometimes to our competitors, because like, that's not where the value is. I can detect for you, anything that you want to detect.

And frankly you can too. the thing that, my company specifically is trying to. provide as part of the value is utilization of that insight. an action, which is always the hardest part. Like I remember at Microsoft when we started.

back in 2012, we were flagging what 5,000 fall today across a hundred buildings, like far too many for anyone to actually action. It was like you maybe hit five or 10 bolts a week if you're good. And so what was the purpose of detecting all that other stuff? So in my mind, fall detection is somewhat of a red herring because everyone thinks that this is the big be all end all, but you know, 90% of it probably isn't because it's detecting things you don't care about.

And number two, it's fairly easy to detect, like even the advanced stuff. and I know that people out there are going to argue with me about like, Oh, well, you're, you're on doing it this cool way or whatever. Well, I don't believe that the value property position for people that use fault the texture is in the creation of the fall detection.

Keith Berkoben: [00:53:33] Hmm.

James Dice: [00:53:34] Yeah, there's going to be some people that I reached out. I'll tell you that.

Keith Berkoben: [00:53:39] Go ahead. I was going to say to jump on that and one of our, uh, ML engineers,  open source the, uh, the project that obsoleted a bunch of our FDD rules, fairly recently. So we actually built a tool and this is a great example of like, why standardization matters, right.

Is, so we built an ML engine that took the standardized data. And then, without any labeling. So this is unsupervised ML was able to, pick out anomalous devices, and, you know, rank them by, how anomalous they were and also attempt to attribute, To, which points where we're likely responsible for the anomalous Snus, and, basically

covered,

all of the manual FDD rules for the piece of equipment.

We tried it on, with this one model. And so we now have in our, operation centers, just this one dashboard, that just pulls up these anomalies, You know, ranked by severity and that has completely changed the way that they deal with that whole area. And the nice thing about it as, as Trevor were saying is a, it, first of all, it doesn't give you anything.

That's. not anomalous. Right? And so already you've brought down the number of things that you have to look at a whole lot, and then you can very easily just turn the knob on, like how bad do they have to be before you show me. And you can bring that number now down to a number that's actionable.

and so your facilities operators are happy and you end up dealing with the things that are actually the most problematic. So

James Dice: [00:55:19] this is a different perspective. on FTD that I'm used to. So I have two questions on it. one is, you've mentioned like getting to a root point or root sensor that's causing the anomaly, but what I'm not hearing is how do you in, that approach?

So, so my opinion on FDD software is that the best FTD software, it does the best job of. Producing a root cause for you and the fault. Right? So the worst FDD software says, Hey, I have like 5,000 faults Like you said, and I don't know why you should do about it. Someone should probably go, uh, go out to the field and check out all 5,000 of those.

And then hopefully, you know, you can figure out what the highest value group causes the best ones though. They say, okay, here's your one, right? And here's your one. And I've ranked it by. Whatever metric you want to rank it by I've ranked it by annual savings. I've ranked it by the impact it has on the most amount of people, you know, those sorts of metrics.

So the reason I think it's a product is because I can do all this in Excel. by going out to the building, like we talked about Trevor, like we used to do this without the software. So. It's doable, but the product is what's automating all of that. And the product to me is what is. Just a huge difference between what I can do with Excel versus what I can do with the best FDD software out

there.

Trevor Sodorff: [00:56:46] Absolutely. Absolutely. And don't take me

the

wrong way. I'm not saying that FTD platforms, you know, aren't of value in, and of themselves because I think, I think they are, if you can figure them, right. I just don't like the idea of talking with one manufacturer the other day and I was asking him, Oh, what kind of fault rules do you have?

And it's like, Oh, we have a whole library of proprietary fault rules and it's like, can I see them? Are they in? Oh yeah, we'll get you on the NDA. And then we'll be able to send you some of this back information. It's like, Screw that, I can rewrite all of these, of just get them out of an Excel file from 22.

I love

that. Yeah. I had this client a couple of years ago and it was with one of the big four and the one of the big forwards doing FTD and what we're trying to do. And I've told this on one of my friendly rant episodes, we were trying to figure out what the existing rules did so that we could just program new ones to supplement them.

That's all we were trying to do. And we had to go through. a lawsuit to get these existing rules. And it was just like the easiest, like, basically the NIST air handler rules. This is what we were asking for. Right. And while we got back was so you're going to have to sign an NDA and anyone who sees this, like, spreadsheet with these rules is going to have to like sign their life away.

I totally agree with you. It's crazy out there. It's not

proprietary. people treat this, like it's like it's proprietary. And frankly, I would love to start an online Rebo maybe as part of this project than we ever did there, Keith of all the different it's available and just have it use it. I mean, you're still going to have people out there that are rock stars and that can help you to utilize it and take advantage of it.

And there's still going to be people out there that kind of stuck with it. And I mean, that's always the state of every industry. I think my goal would be a seat. All boats rise as this technology advances and the idea of these little pockets of like proprietary knowledge is being orthogonal to that motivation.

James Dice: [00:58:42] Brilliant. I love the rants. so, okay. So I guess my last question is around. don't know that I'm fully grasping the scale here. Right. So tell me about like lessons you've learned around doing this at such such scale. And, what would I not understand about that?

Trevor Sodorff: [00:59:01] Jeez. You want to take the first step? Maybe it'll take a second step.

Keith Berkoben: [00:59:03] Well, so I guess I'm gonna sort of skirt the question in some ways and say, one of the things that we haven't had to worry about, Is actually the scaling of the infrastructure that runs this stuff. And that's been really great because effectively what we've done is pushed all of this up through, you know, various cloud technologies.

Obviously there are cloud technologies cause we're an internal Google team, but like, the fact that. We have a, you know, big table and that we have IOT core and, that we have, um, Apache beam, you know, Basically make the

infrastructure part

of this trivial. You could do this for a massive, massive fleet of buildings, um, sending your data all the time and like it just scales horizontally forever.

The hard part is the part that we make Trevor do, which is actually, getting the data, you know, cleaned up from all these buildings and getting the coverage of all the equipment.

Trevor Sodorff: [01:00:10] Yeah. So the initial modeling of the core concepts that we can use, replicate them from place to place, like

novel concept requires

some deliberation about how you might model, which is kind of a hard part for this.

And it doesn't exist for the other ontologies because I think they just sort of sidestep the issue. It's a you know, choose your own adventure kind of thing. And we fundamentally don't want ours to be like that.

Keith Berkoben: [01:00:32] Okay. Um, but

Trevor Sodorff: [01:00:33] then the second piece is like, how do I rinse some bladder, repeat, take like a spreadsheet or an initial discovery from a backend device and convert it.

And so I've pilot  some software that I wrote

that

hopefully Keith and his team will be taking and running with from a workflow perspective that we'll be able to add to our open source project to help them. Aid the exploration of the ontology, like say that I got this device with a set of fields. I'd love to be able to just ask an application, Hey, give me the thing in the model that is closest maps to this.

so we've got some software that currently does that, but like, it needs to get added to the, platform. And so I would say that that's probably the piece that certainly we have not solved yet. But we're going to be focusing on it and trying to solve at least a piece of it. And that might also don't think anyone else in the industry has bothered to grapple with, for reasons that we've kind of already discussed.

James Dice: [01:01:22] Okay.

Trevor Sodorff: [01:01:22] I mean, Yeah, that's the first hurdle. If you can get through that, you can get through that integration layer, like Kate's platform does, you know, quality assessments of data that come through and most good platforms do this. I've worked with clients that have built their own and obviously in Google, but also in Azure.

Uh, I've worked with clients that have done this using third-party softwares. And frankly, we're not like for Google specifically, like, yeah, use that, but from my perspective, as an outside consultant,

I don't really care what people

use as long as they follow some of these core principles

James Dice: [01:01:52] So last question, I always do this where I said, it's the last one. And then I come with another one, sorry. Um, to go was building a new building tomorrow, right? And it's in some country. How do you guys enforce this? So like, if I'm an organization listening to this and I'm going to build a new building tomorrow and I want to use this, how do you guys make the contractors and make the output of a new building?

meet the standard.

Keith Berkoben: [01:02:15] Required in the contract, actually. I'm not kidding about that. Um, so there's the requirements part, which is literally like you put it in the contract, and then there's the, making it feasible for them to do it part. And the second part's a lot harder. So, conceptually the goal is to bring the model information as high up the design stack as possible, because what we've found and, and basically, so you think about the evolution of this.

We started with nothing and then like we started building this ontology and. Now all of a sudden we have this thing that we want to implement in buildings, but these buildings have already been in construction and we're like, okay, we want to attack this on at the end. And what we've learned basically is that the closer and closer you get to FTO B, the.

less time people have and the more stressful it is and the harder it is to do anything. And so what we've learned as we're doing this is the more mature we get the farther up the design stack, we bring this stuff. And so the ultimate goal. Is that when you have your mechanical designer doing something in BIM, they can say, okay, I'm putting in this piece of mechanical equipment or this piece of fire system or whatever I can select in my BIM plugin, which type from the ontology it is.

And now the information on like what fields are required to be mapped, you know, just kind of. Comes out of the design process and you could just hand it to your integrator and say like, this is what it needs to look like.

And so that's what

we're trying to get to. but right now we're kind of in that intermediate stage where like, we've now gotten the MSIs on board, early in the process of commissioning, but we're not.

so mature yet that we can literally just design the building with this built in. Got it.

James Dice: [01:04:08] Yeah. And I would imagine, like, there's still this aspect of just like, we're commissioning an actual building system. We then need to still commission the digital side of it as well. Like, so is there lot that pops up around enforcing it after the fact?

Cause I can just picture a contractor being like it's done.

Keith Berkoben: [01:04:26] yeah, Trevor said it earlier. no human should do this. And we do have automation that basically uh, we require that they give us what we call building config. So it's a Yammel configuration that tells us all about, what equipment's there, what its relationships are, what types they are, et cetera.

we ingest that. We validate it. We make sure it's Self-consistent and then we actually will validate the telemetry coming up to the cloud to make sure that it matches the model automatically. And then just hand you your error report, please go fix. Got it.

Trevor Sodorff: [01:05:03] you can. Train them in how to apply and build the building config model.

We have tooling in place to confirm that it's correct. And like I've said, one of the things

we want to kind of

extend this to is number one up the design stack. So getting into BIM into the design phase earlier, but number two, like you know, creating this model after the BMS has already been implemented, it's still going to be kind

of a

common business as usual process.

And so making it easier for an MSI, with minimal training and with, their kind of the current workflow is working with spreadsheets or whatever it is they do. how do we make it so that, that process sucks the least that it can.

Keith Berkoben: [01:05:42] Yeah. and you know, one other thing that was really interesting, that I didn't even think of initially, with this modeling stuff, which is that, When you define a model, you're making some statements about actually how you want something to be built.

And what we found, is that. we do actually have strong opinions about how we want things to be built, you know, in mechanical systems and his example that weren't really getting, followed in practice. And when you have a model with sort of some pre-made entities that represent your expectations, you can actually.

catch these, design anomalies early on because your designer says, hi, um, there's nothing in your model to represent this thing that I want to make. And then we say, no, don't make that thing right. Make this other thing follows our standards. And so that was really interesting. I didn't even think about.

That when we started doing this whole modeling exercise, and then we found this one building that we built where, you know, it was not at all to how we expected, you know, the mechanical system to be built. and we were like, Oh, if we had just given the model earlier, we would have, we would have caught this and not had a building that was weird.

James Dice: [01:06:58] Right. Fascinating. So

I I've been preaching the power of like control standards, which include. Profiles and, pieces of equipment and points lists for those profiles. And then obviously the data model on top of that. But when the data model already includes those things, you can just start with the data model.

That's really interesting.

Trevor Sodorff: [01:07:16] Yeah. And just to give a concrete understanding of what was happening there, we were seeing on these brand new buildings, pressure dependent, VAVs

being

specified without PID control or anything like that. And it was just like, Don't do that.

Like that, that was the sort of stuff where it's like, if I saw a dual duct VAV,

our

model,

do I want to

extend them all? Or did I want to just shame them?

Yeah,

I'm

sure somebody, again, like you know, I've done some mechanical design and stuff like that. I'm sure that somebody out there has a great, you know, the reason for

James Dice: [01:07:51] pressure dependent view boxes.

I'm not sure about that. Yeah. I'm not sure about that. Okay.

anyway, that's a, that's a wrap. Thanks so much guys for coming on the show. we'll have to do another to get an update sometime, but thanks for doing what you do.

Trevor Sodorff: [01:08:05] Yeah. Thanks for having some fun.

James Dice: [01:08:07] Alright, friends. Thanks for listening to this episode of the Nexus podcast. For more episodes like this and to get the weekly Nexus newsletter, please subscribe at nexus.substack.com. You can find the show notes for this conversation there as well. As always, please reach out on LinkedIn with any thoughts on this episode.

I'd love to hear from you. Have a great day

Upgrade to Nexus Pro to continue reading

Upgrade

Upgrade to Nexus Pro to continue reading

Upgrade

Happy Thursday!

Welcome to this week’s deep dive exclusively for Nexus Pro members. It’s an honor to have you here. This deep dive is a follow up to my recent podcast conversation with Trevor Sodorff of DB Engineering and Keith Berkoben of Google. I learned a lot from this conversation and want to share my takeaways and the full transcript with you below.

In case you missed it in your inbox, you can find the audio or video here:

Nexus site | Apple Podcasts | Spotify | YouTube | Add to other podcast apps

Enjoy!

—James


Outline

  • My reaction, including highlights
  • Full transcript

My reaction

This episode on how Google thinks about smart buildings is a good one to pair with Emmanuel Daniel’s episode from a few weeks ago on how Microsoft thinks about them. My take is that Microsoft is thinking about solving the problems on their campus in a way that can be productized, whereas it seems like Google is solving them for themselves, while open-sourcing some of the methods and not thinking about products much.

I enjoyed hearing that the impetus for the DBO was in Google’s effort to do advanced supervisory control—at this scale, you can’t help but try and centralize and standardize control sequences, setpoints, and schedules. This gets at how the data model is only as good as the use cases it enables. Keith said that Haystack and Brick are not “opinionated” enough, don’t contain enough “composable functionality”, and don’t have the tooling to allow the supervisory control sequences they want to enable at their scale.

Next, I think Trevor (and perhaps DB Engineering) and I are in a solid and healthy disagreement about analytics that I plan to follow up on at some point. As Trevor said, he sees FDD as physics and you can’t put a patent on it, so therefore it’s not a product. The list of rules and the math behind the rules are not where the value is. Taking action is the key. I agree that action is important, but I think he’s missing that getting to the #1 best action, the highest priority, is easiest with the best FDD products and that’s where the true value lies. Otherwise you’re left with, as he said, 5,000 faults per day to sift through.

Finally, I’m sort of left with more questions around ontology convergence than answers at this point. Keith said probably none of these standards are going away, but they should all be able to be represented using RDF, which is the master set of primitives that allows you to represent anything.

My highlights:

  • James’ favorite question - Keith’s straightforward take; Trevor’s frustrating experience with vendor lock-in / interoperability - tied together with Keith’s analogy to the airline industry and electronic medical records (6:03)
  • Impetus for Google’s digital buildings project and where it fits with the company’s broader initiatives (10:31)
  • Walking through the progression of their rollout (21:12) and why they chose not to use other industry standards (24:19)
  • given creation of own ontology, vision for industry convergence (45:15); and response from the industry at this point (49:16)
  • diving into the applications, and DB Engineering’s unique perspective on FDD (51:11)
  • Lessons learned from rolling out this project at scale (58:47)

Full transcript

Note: transcript was created using an imperfect machine learning tool and lightly edited by a human (so you can get the gist). Please forgive errors!

James Dice: [00:00:00] Hello, friends. Welcome to Nexus, a smart buildings technology podcast for smart humans. I'm your host, James Dice. If we haven't met before, I write a weekly newsletter on the same topic. It's also called Nexus. Each week I share what I've learned, my opinions, and what I'm excited about in the quickly evolving world of intelligent buildings. Readers have called Nexus the best way to stay up to date on the future of this industry without all the marketing fluff. You can check it out and subscribe at nexus.substack.com or click the link in the show notes.

Since starting the Nexus newsletter, many of you have reached out to me wanting to talk shop, and we have. After a few weeks of those wonderful conversations, I realized I needed to record and share them with our growing community. So here we are. The Nexus podcast is born. This is our chance to explore and learn with the brightest in our industry together.

Episode 29 is a conversation with Trevor Sodorff of DB engineering and Keith Berkoben software engineer at Google. These are two of the people responsible for rolling out the smart buildings program worldwide at Google. We talked about many of the topics we've been covering for a while here on the podcast, like analytics. Data modeling and supervisory control but this conversation was about doing all of that at scale doing it worldwide doing it on new buildings so how's Google doing that and it's pretty impressive. Of course, we covered the digital buildings, ontology effort as well, where the team is trying to make a lot of their data modeling we're public so others can take advantage of it All right, Keith and Trevor, welcome to the nexus podcast. I'll start with you, Keith, can you introduce yourself? for us?

Keith Berkoben: [00:01:51] Sure.

Uh, my name's Keith , I'm a staff software engineer at Google, and I'm responsible for building out our internal, smart buildings platform, and doing a lot of the work, related to, modeling our buildings so that we can, write software that's portable, across our fleet.

All right.

James Dice: [00:02:10] We're going to nerd out about that topic, Trevor. How about you? Can you introduce yourself now?

Trevor Sodorff: [00:02:15] Sure. My name is Debra . I'm the director of engineering analytics for a company called . DB engineering We're located in Redmond, Washington, and we do smart building consulting for fortune 50 companies and governments around the world. Google happens to be one of our clients and my specialty is in the acquisition organization. Utilization of data at scale. And that's my primary focus.

James Dice: [00:02:39] Cool, cool. All right, Keith. So back to you, um, I wanna hear how you got to smart buildings. So you're working in smart buildings. Now. I bet you didn't start out thinking, man, I'm going to make buildings smarter.

Oh. And you started out your career. So how'd you, get here.

Keith Berkoben: [00:02:52] You know, think the thing that was really interesting about how I got here is that, I did this like junior year group engineering project, in college that was like optimizing a, uh, HVAC system for, for one of the buildings on campus.

And I was like, Oh, this is kind of neat, but never like, thought anything of it and then realized, you know, 10 years later, Holy crap, this is my job. So, so that was sort of odd. Um, I, I fell into this, I think. because, I've always done, a lot of different types of engineering. So, you know, building stuff, software engineering, mechanical things.

And, so I. I immediately felt comfortable bridging this gap between these mechanical systems and all these things in the building and the building trades and things like that and the software. And so it was just kind of a natural fit.

James Dice: [00:03:50] Great. I love that phrase, bridging this gap. that's kind of what Lexus is all about.

How about you Trevor?

Trevor Sodorff: [00:03:56] I studied mechanical engineering in college. I went to Washington state university where they specialize in manufacturing processes, which I really didn't like I got on with a firm in Redmond called DB engineering. And that's where I currently am.

And I was doing a lot of energy optimization for buildings in the kind of conventional retrocommissioning process where.

And you step into a building. You look at the BMS, you look for a fan coils that have failed chill, water valves, or air handlers that are running too often. And one of my primary responsibilities was quantifying those, right? As, as you pass through, if you're doing like utility, right? And so I spend a lot of time doing that, pulling data, organizing it.

And I found that, you know, a lot of these things are, are something that you could replicate multiple times. I ended up doing the same analysis over and over again, and it was kind of, you know, it came a pain after a while at the same time I was working. for one, I'm doing this for one of my clients who happened to be Microsoft.

And they were really interested back in 2011, 2012, and in getting into the smart building space, because it wasn't really a thing at the time, but, you know, you wanted to be able to integrate all that data. They understood the value of the data. And so

I was involved with helping them to implement that platform, organize their data and then write fault detection around it.

And so I was able to kind of dip my toes into the mechanical waters. And into the software side. And I mean, it was, I found out going through the mechanical,

systems, they kind of suck the whole industry is, you know, especially at that time, it was a little bit backward, right. Everything's proprietary, locked down in its box and that was just drove me insane.

And so. As Microsoft started to expand that spent a lot of time working on, you know, advanced fault detection and other analytics that we wanted to do at scale, help them do a bunch of projects around the world for themselves. And for other customers, uh, got involved at Google when one of the directors at Microsoft,

moved down there and started working on that campus.

And I met Keith. I saw that they were trying to go down the same sort of path, but using more of a homegrown technology, as opposed to like an off the shelf product, like you might get with like a switch of sky spark and Iconix or whatever. Okay. I

thought that was interesting.

Okay.

James Dice: [00:05:59] It sounds about like something Google would do. All right. before we get into that, the last 10 episodes, or so I've been asking this question, which is my favorite question of the podcast, which is why is technology in our buildings? You know, decades behind other industries. so Keith, I'll start with you.

What's your perspective? Uh,

Keith Berkoben: [00:06:18] can I just defer that question to Trevor? Um, though I think the, Physical buildings just last for a long time. Right. And there is, you know, all of the incentives around how you finance buildings and how you build them and maintain them, uh, discourages you from replacing stuff that's already there.

And so, as a result, you built a building 25 years ago and the stuff you put in there, it's still there. Totally. Yeah. That's

James Dice: [00:06:47] a simple answer. What about you, Charlie?

Trevor Sodorff: [00:06:49] Yeah, I mean,

they're starting to do this more and more today, but you know, back when I started, the

idea is that you're kind of, as a building owner, you're

locked in with a manufacturer group of manufacturers, basically for the life of the building or for the life cycle of the BMS.

And you were kind of stuck with their shortcomings, the concept of interoperability really wasn't there

and

We're starting to see that progress pretty heavily in the, in the tech space,

but because of, I think the business

models of a lot of the, legacy.

you know, controls providers, mechanical providers and stuff like that.

It's like they want to lock you into their box.

And I,

I mean, it's not, the direction that we want to go over the next 10, 20 years. And, um,

yeah,

it's a big shift to try and

steer right.

James Dice: [00:07:31] I can feel your frustration because you and I have a very similar career path and basically starting with energy, becoming mechanical systems kind of experts, and then getting into software and then realizing that basically the data is not free.

And that's the main piece of it. When it comes to energy engineering, you need the data to optimize the system. to analyze it and the data is not free. And the word that Andrew Rogers used is probably like seven or eight episodes ago on the podcast was he said building owners are not able to self-actualize and I love that phrase as far as like I'm trying to do stuff in my existing systems and existing vendors, usually because of their business models, like you're saying are, are sort of not letting us progress the way we want to progress.

Trevor Sodorff: [00:08:14] And I wouldn't say

that it's all like malevolent or Like someone's just technology is old, right? Like I've been working with one client trying to integrate systems that are 20, 30 years old and, you know, Just how, how can you

possibly do it?

The available data that even the underlying systems has, like a lot of it's, it's a technology problem. As

much as it is like a.

James Dice: [00:08:34] A business model, problem

Keith Berkoben: [00:08:35] and motivation. Yeah. you know, the thing you made me think of actually is, um, EMR, electronic medical records, or even like airline reservation systems is another great example. And, when you have a huge ecosystem with embedded technology or entrenched technology of significant complexity, right.

Yeah. It always ends up being easier to be backwards compatible than it is to try to migrate the thing that already exists. And so what you end up is you have like these core aspects of the way these systems work back then being a great example, right. That are, you know, I don't know exactly how old backnet is.

Maybe it's 40 years old. It's not a young protocol, right. Is still endemic to the industry because it's. Just been backwards compatible, you know, systems that have been added for years and years and years. and that is just the nature of something where you have physical infrastructure that, you have to deal with.

James Dice: [00:09:38] And how does that show up with like, just for the people that are new to technology, kind of like me, how does that show up with electronic medical records?

Keith Berkoben: [00:09:46] Oh, well, it's that you have, all of these hospitals that have huge systems to deal with all these, records and, You know, if do, as a, technology vendor, come in with something new and it's not backwards compatible with what they already have, it's immediately a non-starter because no one wants to invest in upgrading the system that they spent a bunch of money on, that they already have.

And so you see that the airline industry is just now moving past systems that are, you know, from the seventies or maybe even earlier, right. And similarly with EMR is it's very hard to get onto modern platforms because no one wants to invest in migrating.

James Dice: [00:10:25] Totally. It's fascinating how these parallels happen.

a news industry. So I let us all right. Let's, start talking about Google then. so we're talking about smart buildings today, but let's start by kind of putting that in context and this overall, like what is Google trying to do? And specifically I want to ask about, um, I had this really cool article in the newsletter.

I don't know, four months ago where it was talking about how Google is shifting data center loads across the world and real time to basically, Match up the loads with the cleanest power at that time as a strategy to get to, some sort of carbon neutrality or, you know, positivity at some point.

Right. So can you kind of paint us where this fits in the context of, of Google strategy, Keith.

Keith Berkoben: [00:11:11] so there's, there's a couple aspects to it. And actually the carbon one, is, you know, kind of new in a way. the bigger one is that, Google operates at such a scale that we're often, the largest customer of any given system in any domain, whether it's, you know, a physical security system or, you know, uh, HVAC head-end or, you know, et cetera, et cetera, et cetera.

Right. And to the point where we start breaking these things and, scale really is. with traditional approaches, oftentimes a barrier, and the whole goal of what we're trying to do in the smart buildings world is to turn that on its head and use the scale to our advantage rather than have it.

Hold us back. And then of course, recently with our, you know, newly announced initiative to be, you know, zero carbon by I forgotten the year now, right now, not only is this kind of a philosophical thing where like, we would like to be able to build applications that scale for our buildings, but now we really have a very strong incentive.

you know, to deliver something very specific with those applications, which is energy savings. So

James Dice: [00:12:27] let's talk about this breaking systems. tell me a story about this. so how many buildings do you guys have, on a given campus?

Keith Berkoben: [00:12:33] on a given camp.

Well, I mean, so I think in the Bay area we have over a hundred buildings. I don't know what the exact number is. And I probably couldn't tell you if I did, Siemens

James Dice: [00:12:45] to Sega or like a Johnson Medicis and you plug in a hundred buildings to that. And you're saying just the scale like these.

Keith Berkoben: [00:12:51] Well, I mean, so, so the fact is that we don't even try.

Right. I think that, You know, for the Bay area we are running. I don't know, Trevor probably knows the exact number, but certainly more than a dozen separate, web control, instances for HVAC. Um, Because simply they would blow up if you tried to do anything else. Right. And so that kind of thing gets really bothersome when you have hundreds of buildings globally.

James Dice: [00:13:20] Got it.

All right.

Trevor Sodorff: [00:13:22] Plus new construction

and that's

that's. The other thing is that Google is growing at the same time. This is happening. We're not talking about like, company that's kind of at status and it's just the same campuses being maintained. It's they're, they're doing a lot of growth and so.

You know, we're dealing not just with existing legacy buildings and legacy systems, but we're also dealing with net new buildings and processes around those with BIM and with, you know, different IOT style approaches. Right. I don't know if you want to talk at all about IOT light versus IOT, heavy Keith, and as part of what you guys are doing, but like just the different philosophies and they're all kind of there at the same time.

James Dice: [00:13:59] Hmm. Okay. Before we dive into that piece, just help me understand how Google. Operates buildings, like, do you guys have your own separate facility management group? You hire that out or how does that

Keith Berkoben: [00:14:10] show up? uh, it depends where regionally, we are big enough that there isn't one answer.

in, the Bay area, we try to centralize it. So we have a central operations center that. fields, all of the issues and, you know, has big screens on the wall and, and such like, we contract out a lot of the, actual operations work to, various companies. Okay. Um, but it varies cause we have sites, some places in the world where you have, you know, one building in a country, and then you have other places like the Bay area where we have.

You know, over a hundred buildings in a region. and so part of what we're thinking about as well, how do you, homogenized, the way you deal with that, even though you have such big structural differences in that, not sure we exactly have the answer yet, but it's something that we're thinking about.

James Dice: [00:14:59] Yeah. So this was basically the impetus for. This do you call it the digital buildings project? Is that what you call it?

Keith Berkoben: [00:15:07] Yes. So the input is that's the GitHub project anyway. Okay.

James Dice: [00:15:12] All right. Well, that's good enough for me. So the impetus for this is basically we've realized we have all this diversity of systems.

All of these systems can't really handle our scale. We have different ways of operating worldwide. So what's the solution. Like what, where are you guys headed? Uh, given all of that and the given your goals.

Keith Berkoben: [00:15:31] so my opinion, and I think Trevor probably shares this opinion is that, there are few, high value things that, that you want to do in your building and that you can build, you know, a supervisory control apparatus that allows you to do those things the same way.

In every site, and that captures a huge amount of the value of. your building systems and all of this data that you're getting without requiring you to deal with all of the complexity and all the differences involved the things across all of the buildings. And so that.

Really in my mind is the strategy is basically how do you make it so that this high value thing that you care about, whether it's, you're, you know, air handler, supplier temperature reset, or, occupancy based control of whether or not you're running a system in a particular place, um, can be done the same way across the entire fleet.

And when. you turn on a new building, all of those features and all of those, you know, analytics that you have for all the other buildings just work in this new building that that's the Holy grail. I love it.

James Dice: [00:16:53] an Trevor. The reason I'm so excited about what he just said is like our history, as energy engineers turned analytics professionals, right?

The progression that everyone gets to when they head in that direction, Is that, you know, I'm doing a bunch of stuff that could be automated. Like you said earlier, I'm doing it over and over again and spreadsheets and hobo meters and things like that. Right. And then you, go into, okay, now I'm going to put analytics on top.

I'm going to do some analytics and then. very next question then is, okay. Analytics discovered like Keith just said a supplier temperature that is not resetting and it should be okay. Well now I have to go talk to that vendor and it convinced them why they need to do the reset and, basically pay them or figure out if this.

Service contract covers their time, and then they say it's done and then it's not done. And then I go through the entire thing again. So, so how are you seeing this in terms of helping this sort of supervisory control get implemented, you know, throughout the,

Trevor Sodorff: [00:17:50] well, so I think we want to, talk about two pieces of this problem.

One is the end use, which is a kind of a problem everywhere. And one is how you get to that end use. So like with, Google, what they've been doing, they have a lot of control over their own systems. It's not like they're at the behest of like manufacturers for a lot of these things. They have their own controls group in-house and then they partially subcontract.

My team will, uh, with the data that comes out of the Carson system, which is part of the boss system, which is the overarching thing that Keith is kind of referring to with their strategy. Um, we take that data, we'll build like dashboards or analytics on top of that using their base data. And then those are just automatically updated refresh as the data

that comes

in.

And we'll have engineers both with my team or with other people around the world that are using it. Or with, you know, HVAC techs or whoever's looking at this, there'll be able to use and interact with it and make the changes. And we do training to try and help them understand why they should care about these things.

Google has a pretty good handle on how they deal with that. And, they're really good about like, not making everyone's responsibility, like fault rules, right? Like one of the, one of the things I've seen done elsewhere is, you know, Oh, all the HVAC techs are responsible for their quota of faults that they have to get fixed in

Keith Berkoben: [00:19:04] a month.

And

Trevor Sodorff: [00:19:05] that tends to not be very efficacious because like they don't care. They have other motivations pothole called one of them. Right. And so if it doesn't reduce the amount of hot cold calls they have, they don't really care. It's kind of a waste of their time. You're better off having a strategic group that is responsible for these things that has the tools available to be monitoring these things, or in an ideal case, a system that can automatically respond and posture the system or adjust, which is something that we're moving towards with Google.

But the piece that underlies all of this is the structure and content of the data. And I don't mean just in terms of what data is available. Like, Oh, the air handler has a static pressure and a static pressure set point in a VFD speed. Right. It's also in. I've got this air handler that has a fan wall, and it's got effectively two or four fans versus this one with one fan.

Are those analogous? And how do I teach a machine or encode into a machine that these two things are analogous and that I can treat The analysis or the response to the same, that piece of it is something that requires some rigor, right? You can't just like supply fan one supply fan.

Like there, there has to be some underlying meaning to supply fan versus discharged fan versus exhaust fan, if it's some sort of contextual meaning. And so what we realized as we started rolling this out is that the use cases are there and the scalability. In terms of connection is there, but the scalability in terms of organization and the robustness of that organization is the thing that is really at the heart of the problem that the industry sees today, in my opinion, right?

Like I hate organizing data. I really hate it, but I know that if we don't organize it properly using best practices, then you're going to regret it. And you're not going to get the use out of the system that you thought you would

James Dice: [00:20:51] enable. What. Keith just described, which is, um, I'm trying to set a new control standard throughout my entire portfolio.

And you can't enable that unless all of the systems are modeled properly. So, you know, which sequences apply it to which systems and that kind of thing. So I want to pause on the modeling question, cause I know we're about to get into that. Just pause on that real quick. So when you're, when you're thinking about rolling this out, though, paint the roadmap for me. So what I'm picturing is we're getting all the data together. and then what, and I think what I'm hearing though, is you're going from, the data's together. I'm gonna model it somehow. We'll get to that, and detail on a second, then you're adding visualization analytics onto it.

And at some point you're, you're adding supervisory control as well. So paint to me like how you guys are like, rolling that out. And if that's not the correct progression, correct me

Trevor Sodorff: [00:21:41] well, so the, the analytics and that piece are really somewhat independent of the rollout because like the use cases are going to vary from region to region. And we've got a lot of use cases built and in production in the Bay area. And as we start to roll this out globally, like,

They tend to be a little bit

different. And so, that of it is I think, a little more straightforward because you know what you'd want to do with the data, if you're a facility manager in the UK or in Singapore or wherever, if you had it.

And so the real question is how do we get it? And that is the piece that Keith and I have really been

focusing on.

because we know that's the hardest part. Like that's the most manual human reliant part, everything else after that is gravy, right?

Like you can roll out a new application or a new analytics component once you have that. So how do we enable people to apply a model? That's. good rigorous and in the format that we want and they don't have to have a lot of information about the underlying ontology in order to do it again.

I have to have as much familiarity. And, uh, so the way that we're approaching it, I think globally, Keith can talk to this is we're trying to enable MSIs around the world that are working with, Google to perform their new construction, energy projects or whatever. Uh, we're trying to enable them with tooling and with this project and with processes defined as part of that project to ease the project so that they can.

They understand up front, like these are the types of systems that we want to integrate. They have whatever they do to integrate them and if it's Jace or, or whatever it is. And we give them a format, like a template for what we want the ultimate model to be in. And we provide tooling through our project that validates that any extensions that they want to make to the model or any instances of the model that they want to give us as part of like the final delivery handoff for that project.

James Dice: [00:23:22] Okay. And

Trevor Sodorff: [00:23:23] we can talk about that in more detail, but in my mind, like the analytics piece is everyone in the world knows about analytics, kind of how to do analytics, how to scale analytics. it's the piece in front. Like how do you get a system to have that organized data? How do you have a model that gives you something that you can then extend on top of.

James Dice: [00:23:43] Yeah. Yeah, totally. Let's dig into that piece. so it sounds like major thrust of the project, if we could call it that it's a bunch of projects that are happening together is just getting systems online, getting the data flowing, freeing it up, using local MSIs to do that in a standardized way.

Yeah. Okay.

Trevor Sodorff: [00:24:01] Yeah. That's that is the goal. and we can talk about this a little bit because it's not been all like, you know, flowery goodness, it's hang ups to it. And part of what we want to be extending our project to do in the near future is to help reduce the friction associated with this more rigorous model.

Okay. And, you know, a lot of the, flack that we get for this project is like, well, there's brick and there's haystack. And, you know, I think Siemens has one, like, why aren't we just using one of those? Well, part of that is because there are things that we want to be able to do in support that aren't fully supported within those models.

And there's, in our opinion, not adequate tooling around the enforcement of extension of those models. And I mean, this is your area. Maybe there's something you want to,

James Dice: [00:24:49] we kind of just set the stage for someone that doesn't have that context, of why you, said what you just said, which is you realized, okay, I have all this data worldwide.

I'm going to. Get it into some centralized location. I know I want to be able to do cool stuff with it later on, but right now this data has no meaning. And what I heard you just say, as you looked at all the other efforts to produce meaning in our industry, which is haystack, and brick, and there's probably Several other worldwide that people are like, well, why didn't you use our standard? Right. So, so Keith when you looked at that, what did you find what did you do? I guess it'd be a good time to save what the project is for the first time.

Keith Berkoben: [00:25:29] Okay. Well, yeah, so, actually I think first thing that's kind of relevant is what my context was when I came in.

Right. So I'm a software engineer, and I. Up until I started working on this while I had some ghetto kind of general experience and like what building systems are and roughly the things that they do. I had no experience as like an MSI or, actually working in the building. Industry. And so I came into all this stuff completely naive and I was like, okay, we need to model this stuff.

Cause I understand, you know, generally how you write software, you need an abstraction layer to do stuff. And I was like, okay, I got to find the same distraction layer. And I looked at haystack and brick and at the time, and the gap for me was usability. Right. Like, as somebody who didn't know anything about this, who didn't have any available tooling who, needed to do something quickly with rigor, those platforms just didn't offer what I needed.

Like they were. either, you know, they didn't have coverage in the right places or had too much coverage in another place and they weren't opinionated enough that I could do something with them that was functional quickly. there's

James Dice: [00:26:49] a lot of good and a lot of good terms here and I'll let you keep going, but I want to kind

Keith Berkoben: [00:26:52] of slowly and so, right.

And so, coming into us, I was like, okay, well, I have this. very specific use case, which is that I want to enable supervisory control and analytics at scale. And I need something that makes that easy for me to do. part of the art of modeling. Is being opinionated about something in a way that makes it easier to get to your ultimate application goal.

because if you try to model the entire world and there, Places you know, companies who have tried this, I think there was this really great, IBM effort to like create an ontology of everything at one point. And it's just like very quickly becomes so unwieldy that you can't do anything with it.

And so you have to be very careful about,

shaping your model

to really map to your application and what you need. and so we. Decided to build something because we were like, we want something that helps the user to get to that thing that enables the supervisory control analysis quickly. You know, and make it so it's hard for them to make mistakes.

And so that, kinda how we got to where we are today. I

James Dice: [00:28:14] feel like this is a missing piece of the sort of ontology or data modeling conversation in our industry. So Nick, I asked you from KTS buildings and I talked about this on, I think it was episode three, but his perspective is. That he's seen a lot of people say like quote unquote, tag data.

And then the, the myth, I guess, in the industry is that now that the data is tagged, I can do anything I want with it. Right. But the thing that people find out is then when I go try to do a slightly more advanced use case, right? Like supervisory control, which is an extremely advanced use case, you're then finding, Oh, I didn't quite tag it in a way that enabled that to happen.

Right. So is that, is that what you mean by opinionated or. W what do you mean by that?

Keith Berkoben: [00:29:01] well,

sure. So opinionated, in the sense that you have to decide, What things you care about representing from the real world in your abstract model? You know, so as an example, maybe if I only care about high-level supervisory control, cause that seems to be our meme in this podcast, right.

Um, maybe I don't need to model a PID loop for a VAV damper. Because I just don't care. Right. The only thing I'm ever going to change is the set point. And so, building a model that lets me do that, it doesn't necessarily have any value. I mean, I don't think we've had a decision about whether or not we're going to model PID loops, but the point being that, if the thing you're trying to do, doesn't need a certain piece of complexity, then the opinionation Of the model is, to basically, erase that complexity from the world so that you just don't have to deal with it. And you have to be careful about what you erase and what you don't.

Trevor Sodorff: [00:30:01] I would like to talk explicitly about some of the things that were missing from brick or from haystack, because I want to give concrete examples real quick.

Keith and I started collaborating on this. I want to say like, Year and a half, two years ago on this thing. And at the time there were a couple of drawbacks with both haystack and brick that we didn't see a workaround for without kind of doing some hodgepodge mixture between the two.

The thing that was missing from haystack, we thought at the time was a little bit of the rigor around definition. And the lack of relationships, most of that solved in haystack felt like they're doing a good job with, with improving the platform. But at the time, like that was a big piece that was missing.

But the ability to natively within that, tagging structure build in relationships, because that's a core component of what we need in order to be able to do, any sort of, system control. Right? The, on the commerce side brick at the time was sort of a new thing. And it

had the concepts

of, you know, relationships and entities and these things built into it.

Natively that was missing around. It was any sort of extension of available. They're not really tagged sets, but you could imagine like, standard field names. Okay. No, it's like, well, if we're going to adopt this one, we're basically going to build it all for them. Plus, and Keith, you might give some additional content here on why brick, like we model ours after brick, but we, extended brick

Shout about that a little bit, or,

Keith Berkoben: [00:31:33] yeah, I mean, so I think the thing that, we, did a, I don't want to say better, but that we invested more effort in. Then these other aspects was the idea of modeling above the level of. Points, and giving real structure to equipment and, composable functionality.

Um,

James Dice: [00:31:57] opinionation and tooling and composable functionality. Like we're going to be a glossary for this episode, but yeah. So what do you mean by that?

Keith Berkoben: [00:32:06] So, so as an example, And I'm a little bit fuzzy on the details now. Cause I haven't looked in awhile, but you know, brick, as an example was really atomic down to like the point it was a pure graph, right?

Like your piece of equipment. In the model was, you know, a bunch of connections between these points to entities that made up a piece of equipment. because it basically was RDF. Right. and For our purposes, that was too low level. cause it, just the amount of complexity you needed to like build up a biological piece of equipment that you cared about was, was really high.

And so we kind of. Jumped up one, step higher and said, okay, well we want this concept of a piece of equipment. We logically understand, you know, an air handler, a fan quote, a year, a VAV, et cetera. and we want to make it, easy to build one of those and model it. and so we sort of focus more around this idea of equipment and equipment with functionalities, where you could define a piece of functionality.

So like, you know, damper control, based on a temperature or something. and you could then say, I have a piece of equipment and it has this functionality and this functionality then. requires that you define these points for that functionality, because that's what you need to analyze that functionality.

Okay. And so then through composition of these things and inheritance, you can now, very rigorously construct things that, you can deal with programmatically.

James Dice: [00:33:49] So if I build on our meme here and I want to say. Implement, uh, static pressure control on all of the fans that can possibly do that worldwide.

That's kind of the approach you're saying is like anywhere I have VFD on a supply fan VAV boxes downstream, and a discharge pressure sensor. Now that component, and that is just an example that might not be quite accurate, but that component is now able to be pushed out. And now I can. Standardize that sequence as long as those components are there.

Trevor Sodorff: [00:34:24] Yeah. Yeah, exactly. And so our idea was that a think, well it is not equivalent to the sum total of its available points to its sum, total of composed functionality. Yeah. So say that I've got like fan control, it's going to have a start stop. And it's going to have like a status feedback, maybe some, feedback from the Ambridge or for something like that, that becomes one chunk, one composable piece.

Then I've got it. Does zone temperature control with dual set points, a heating and a cool access point. And it uses that to control the chilled water helm. That becomes one piece, the dual set point control can also be its own piece and you can compose it

of these kind of overlapping

chunks. such that you end up with the full list of required data fields whether they're optional or required or whatever, but you can essentially compose devices more quickly by just having a basic understanding of chunks of functionality that are reusable across fans, fan coils, air handlers, you know, shellers whatever.

That was, kind of our idea. And we've implemented that. The other thing that we've kind of found was missing through some of these other projects was a large body of precedent. So, you know this, right? Like with, haystack,

you know, to MSIs in two different places can have two different tags sets for the same thing.

Keith Berkoben: [00:35:39] Right. It's

Trevor Sodorff: [00:35:40] very common. So one of the things that we wanted to do as part of our project was all of the things that we're actually applying to existing pieces of equipment. We're going to make that part of the, ontology part of the model documents. So that like you have these tempers, one thing that we've seen before so that you can get an idea for like, Maybe I have this, fan coil with, you know, the discharge VFP speed control and whatever else associated with it.

And I can just take, or extended So that was kind of our core concept. There was no body of precedent that we can give to people and that they can kind of get an idea for what we're doing. And we have names associated with tags. Like you can imagine tag sets and, like what we call standard field names.

It's being equivalent And we just define those and we have different contexts where we apply them differently. And it's all just embedded there.

James Dice: [00:36:29] Got it. Yeah. This is something that I think it was episode 11, Corey, most men who works at NRL. is working on these similar concepts, uh, day in and day out.

We talked about some of these limitations. If anyone wants to go dig back into that episode, hasn't listened to it yet. we talked about a lot of these, sort of aspects. What did you guys mean by? I think I understand that concept, which is just like, let's give people examples of if they come across this fan coil, how should we do that next time?

Right. what are some examples of similar examples around this tooling concept? So you mentioned I have an MSI in, the UK, right. And I want to make sure that they're basically setting this building up in the right way, according to the standard. What do you mean by tooling and how do you sort of support that MSI setting up that building and caveat to that question though?

Is it, it just struck me as like, how massive you guys are in terms of scale is a really a great thing. If you're thinking about this in this way for our industry, right? So you're, you're basically saying we have 10 millionaire handlers and we're gonna just basically model out all of those, right. In a standardized way for everyone to sort of benefit from, which is a difference with like at least haystack, which is the one I'm most familiar with.

Right. You just said it, Trevor is like, What MSI is going to do it their way. And that's their sort of version of haystack. And that's probably going to be their little thing that no one else gets to benefit from in most cases.

Keith Berkoben: [00:37:56] Right? Yeah. so yeah, I'm gonna kind of discuss some of the more, conceptual aspects of it.

And then Trevor can talk a lot about, you know, the actual process, cause he's actually done hundreds of buildings now. so a couple of the things that were missing, I think, in the other ontologies, that we have, both methods and tooling for. are both dimensional units, which was a really big thing.

Uh, if you're going to actually do this, especially across the world, we have multiple units. And also the idea that, standardization, needs to happen, in the data path. Right. Because one of the things that doesn't scale is Enforcing that every piece of equipment, exactly models your current, data model, on the device.

And so a lot of the tooling that we built was, you know, structure around things like dementia units and validation to make sure that when you, bring that stuff into the model, you do it consistently, but also all the tooling to get. From your native device and its payload into the model.

So we have, the ability to effectively configure a building and, define all of the translations between, what we're actually getting out of the building, what we want it to look like and have that all happen in stream as the data is flowing.

Trevor Sodorff: [00:39:25] And so, let me break that down a little bit.

Cause that's pretty high level.

there, there are three things that we kind of anticipate wanting Q1 for and recurrent lead doing two of them. Uh, first thing, the first thing is how do I apply the model and know that it's like a correct implementation of a model. So I have an in Europe that is trying to build me this.

Effectively, it's an RDF model, right? it's in Yammel but we have the ability to convert it. So trivial difference. Right. But like this human readable model, I want them to construct it. They need to be able to construct it based on things that are already defined in the ontology today, once they're done with it, it gives me a translation between air handler one, two, three, with all of its different goofy available data points that map correctly by dimensional and units and by a direct one, one napping to something that exists in our ontology into a type that exists in our ontology.

When you start to do that, like how do you make sure that what they gave you was valid or all the relationships between the air handlers and the VAVs correctly defined? Are they using the relationships consistently when you need tooling to help you with that? Because no human is going to be, so we built that to the second peak because we have not seen everything there is to see in the world, obviously, woman doing this for a couple of years and it really

focused on,

the U S and part of Europe.

we know that they're going to need to be extensions to our ontology. We're gonna need to add new concepts, like, You know, certain types of alarms that are wonky in Europe or different types of control strategies. I don't think we looked at a chill beam yet. We haven't looked at commercial refrigeration and we want to add, adjust things as we go.

How do I ensure that when I do an update to the ontology, that doesn't break something that already exists or is in Congress for something we've already done.

So again,

you need some tooling and so. What we've done is we built this,

tooling that helps us to enforce the rigor of the model when you go to extend the third piece, and this is the thing that we have not developed yet, but that Keith and I have been arguing about and trying to figure out how we're going to tackle this one.

We've got some POC in place, but this is the place where we really want to focus. I think in the upcoming years is how do I automate the application of the model? Because I've got this air handler say I've got this CSR dumped from the, system. How do I convert that into this, into this Yammel file that thankfully gets converted into my, translation hierarchy or whatever.

today there's a series of things that are out there like brick has, plastering that Rico. I don't know if you seen that to do some like automated NL based, alignment, a lot of different people have their

own.

proprietary things, but an MSIs will tend to have some mix of manual and automated, like tagging or whatever neglecting has this built in, but there's no overarching like available tool set.

That's robust enough that MSIs can use it at that's also free. this is kind of like part of the secret sauce that MSIs have. And.

James Dice: [00:42:21] Labor integration labor.

Trevor Sodorff: [00:42:23] Yeah, exactly. And frankly, hate that. That's not something that we, as an industry had tried to impact more openly because like, you can imagine that when you are doing mechanical designs, these systems are coming about, you know, due to the heterogeneity of engineering conditions and.

So the engineers aren't limited in terms of the wonky stupid stuff that they can be putting in, in any building anywhere in the world. And so

I have to be flexible

and interpret it. Right. And, frankly, the whole idea of mapping the state is a waste of time to me. Like it should be done by a machine.

And my goal

is

particularly with regard to this and also with FTD and some of these other things that we kind of sometimes do by hand. Is to automate it away. I want no in the world in 10 years from now to have to build a load sheet or a spreadsheet that maps from one-to-one. I just, I hate it.

And so that's, that's one of the things that we're gonna be focusing on in the near future, because that's a missing piece.

James Dice: [00:43:18] Right. Love it. Okay. So I sort of have the picture for how this ontology came to be. Right. and so I have to kind of. I think they're related questions in that. Okay. So now you guys have open source, all the work you've done, right?

It's all on get hub. so the questions I have around. Okay. We've been using our use case of, of supervisory control as an example. What happens if there's some other use case that you guys don't use? That someone else that wants to use this new standard, they want to use it. Right. But the model doesn't enable that.

how were you thinking about that? And if you're not worrying about it, that's certainly okay, too. I'm just wondering how, if I'm a new organization and wants to just take advantage of the work you've done, but I have this use case over here that you guys aren't doing. W how does that work?

Keith Berkoben: [00:44:10] Um, so I guess it depends what your use case requires that doesn't exist already. you know, we definitely are, Interested in suggestions for, improvements and extensions. So, uh, you know, I'm sure we haven't thought of everything. and there are definitely some things that we've thought of that we haven't had time to implement.

So there's kind of that route it's like, maybe there's a core aspect to what we're doing that like, It needs to be added and, you know, there's a collaboration that needs to happen there. Um, the other thing that, could easily done if we're just talking about a model

coverage,

right. cause I'm sure that there's a good Jillian pieces of equipment out there that we're never going to see in a Google building that somebody might want to use this for.

those can either be contributed or can be extended locally. Right. it's very easy to adjust. Add new, namespaces or add new, entities to the model for things you know, that are relevant to your individual use case

James Dice: [00:45:08] flood people will contribute. If you're listening to this and you're going to add your own, why don't you just contribute?

so, okay, so second question, like follow up to that is. Now we have, the Google digital buildings technology that we have haystack at brick, and we have the other ones. How do we, as an industry get converged at some point, and is that, in the cards?

Trevor Sodorff: [00:45:31] It is, it is pre chronic youth.

Keith Berkoben: [00:45:36] So, I fundamentally, and if my, teammate, was here, he would be very adamant about this is there are, open standards for interoperability that are applicable in this domain. So RDF being the most obvious one and all of these things. Can and should be cross mappable, you know, between each other.

so that's kind of the first step is if you have a haystack model, you know, it is possible to make a transformation that makes it a brick model or digital buildings model for the most part. Right. Um, I think each one of these different ontologies is natively good at different things.

and so it isn't necessarily the case that you need to, use one. For everything, right? Because they are cross mappable. You could imagine that, in the future, all of the tools are, you know, RDF compliant, let's say so everything's natively using RD. Yeah. F as, As this model of the data and you use digital buildings because digital buildings is a really easy way to rigorously configure your model. Right. And then you spit out some RDF and then the schools use it. or maybe you want to use say sky spark or something.

Right. And so you can then go through RDF to haystack and now whatever tooling you get natively on sky spark, you can use it. and you know, you could think of a bunch of other use cases. So, well, convergence. would be great. You know, we have one thing that rules the mall. practically speaking, probably none of these things are really ever gonna go away, but we definitely can inter operate and should inter operate so that we have tools that can deal with, all of the things.

James Dice: [00:47:22] Got it. All right. I like that answer. So, so what is the thing that would be like the universal translator? In the future, I'm just imagining something that says, well, I can talk to Google and I can talk to pay stack and I could talk to brick. Right. is that how it would work or is it like, is it some other implementation?

Like the middleman middleware?

Keith Berkoben: [00:47:44] I mean, so RDF is, um, if you think about it in terms of language, it's like the, um, what is it? The IPA international phonetic alphabet. I think  I'm getting that right? Like RDF is that kind of master set of primitives that allows you to represent anything.

And so that's ultimately, you know, something that has to work with everything needs to be able to understand that. set of primitives and then you can represent all of these different models with those primitives. I see. but in terms of the, the interoperability to use language as a concrete example, um, you know, native, People who live in snowy places have lots of words for snow, right.

Because it's something that they know a lot about. And there's no way to like, literally translate all 22 words for snow into English, because we just, we don't have that concept. Right. And so there is. Always going to be some kind of manual interpretation between, you know, your model and haystack and your model and digital buildings or brick or whatever, because.

The concepts will not always a hundred percent map one-to-one. I found

James Dice: [00:48:59] that

every good, like ontology nerd has all of these analogies, just like in their back pocket in case they need them. Uh, and I, I love that Corey will love that too. And he's listening to this, uh, shout out to Corey and all his analogies as well.

Cool. Um, so what's the status of the project, right? So, I'm sure people are reaching out to you all the time. First of all, like, why didn't you use haystack or why didn't you just Brit, like they're reaching out about that, but then what are you guys getting from the community as far as contributions and that sort of thing, with the project right now,

Trevor Sodorff: [00:49:31] I would say the most part from the community, from what I've seen is, questions about concrete utilization.

Because like I said, one of the things that we don't actively support right now is like tooling, that makes, application the model very expressed or easy to, interpret. I think that's one of the shortcomings that we're still kind of dealing with. Like we believe the models

is very good, but that

it still requires a little bit of tooling to make it to the point where like the lowest common denominator of, user can, effectively take it and run with it.

some of the comments that I've been seeing are just trying to understand some of our core concepts. I don't know, Keith, what would you say.

Keith Berkoben: [00:50:08] so definitely seen, some interest from the other, ontology purveyors in terms of, uh, you know, sort of seeing which concepts that we focused on that were different.

and, I think that, that's been interesting, and. there's also, um, Brian Turner and buildings, a T has been actually working on kind of a universal translator. so that's, been sort of neat. His shop, I think is, very heavily invested in haystack and, his company also does fair amount of work for sure.

For Google. And so, uh, he's been, sort of trying to figure out the tooling that does this cross interpretation. So that's been pretty interesting. Got

James Dice: [00:50:54] it. I didn't know that that's what, Brian's project was before I asked that earlier question. Uh, so good to know. Okay. Very good. All right.

So a couple more minutes left here as we kind of close things out. I want to head away from that. So if you have anything else to say about the project, let me know. Okay. so one of the main things that we haven't quite hit on is. you, you mentioned supervisor control and you mentioned visualization the data.

We've mentioned FDD. are those the main sort of applications that you guys are developing today? and specifically, Trevor, I want to ask you something I heard from Jeremy, the other day, who's one of your coworkers. I was in a meeting with him and he said something along the lines of, FDD specifically.

You guys have a little bit of a different philosophy around it and how it's not a product. And I don't know what it is if it's not a product. So can you kind of explain that and how you guys are implementing analytics in FTD? once you do get the data model properly

Trevor Sodorff: [00:51:55] in my mind,

FDD is physics. Parents simple.

And you can't put a patent on physics.

And so like you can come up with a fancy way of doing valve vault detection or doing some, whatever you want to do. Right. But like anyone with a core understanding of first principles should be able to replicate it. I just, I hate the idea similar to what we're talking about earlier with, You know, somebody's making a career out of mapping data when a machine should be doing it. I feel like a lot of these things that we're doing in FTD, I happily gives all the FDA that we're working on to our clients. And even sometimes to our competitors, because like, that's not where the value is. I can detect for you, anything that you want to detect.

And frankly you can too. the thing that, my company specifically is trying to. provide as part of the value is utilization of that insight. an action, which is always the hardest part. Like I remember at Microsoft when we started.

back in 2012, we were flagging what 5,000 fall today across a hundred buildings, like far too many for anyone to actually action. It was like you maybe hit five or 10 bolts a week if you're good. And so what was the purpose of detecting all that other stuff? So in my mind, fall detection is somewhat of a red herring because everyone thinks that this is the big be all end all, but you know, 90% of it probably isn't because it's detecting things you don't care about.

And number two, it's fairly easy to detect, like even the advanced stuff. and I know that people out there are going to argue with me about like, Oh, well, you're, you're on doing it this cool way or whatever. Well, I don't believe that the value property position for people that use fault the texture is in the creation of the fall detection.

Keith Berkoben: [00:53:33] Hmm.

James Dice: [00:53:34] Yeah, there's going to be some people that I reached out. I'll tell you that.

Keith Berkoben: [00:53:39] Go ahead. I was going to say to jump on that and one of our, uh, ML engineers,  open source the, uh, the project that obsoleted a bunch of our FDD rules, fairly recently. So we actually built a tool and this is a great example of like, why standardization matters, right.

Is, so we built an ML engine that took the standardized data. And then, without any labeling. So this is unsupervised ML was able to, pick out anomalous devices, and, you know, rank them by, how anomalous they were and also attempt to attribute, To, which points where we're likely responsible for the anomalous Snus, and, basically

covered,

all of the manual FDD rules for the piece of equipment.

We tried it on, with this one model. And so we now have in our, operation centers, just this one dashboard, that just pulls up these anomalies, You know, ranked by severity and that has completely changed the way that they deal with that whole area. And the nice thing about it as, as Trevor were saying is a, it, first of all, it doesn't give you anything.

That's. not anomalous. Right? And so already you've brought down the number of things that you have to look at a whole lot, and then you can very easily just turn the knob on, like how bad do they have to be before you show me. And you can bring that number now down to a number that's actionable.

and so your facilities operators are happy and you end up dealing with the things that are actually the most problematic. So

James Dice: [00:55:19] this is a different perspective. on FTD that I'm used to. So I have two questions on it. one is, you've mentioned like getting to a root point or root sensor that's causing the anomaly, but what I'm not hearing is how do you in, that approach?

So, so my opinion on FDD software is that the best FTD software, it does the best job of. Producing a root cause for you and the fault. Right? So the worst FDD software says, Hey, I have like 5,000 faults Like you said, and I don't know why you should do about it. Someone should probably go, uh, go out to the field and check out all 5,000 of those.

And then hopefully, you know, you can figure out what the highest value group causes the best ones though. They say, okay, here's your one, right? And here's your one. And I've ranked it by. Whatever metric you want to rank it by I've ranked it by annual savings. I've ranked it by the impact it has on the most amount of people, you know, those sorts of metrics.

So the reason I think it's a product is because I can do all this in Excel. by going out to the building, like we talked about Trevor, like we used to do this without the software. So. It's doable, but the product is what's automating all of that. And the product to me is what is. Just a huge difference between what I can do with Excel versus what I can do with the best FDD software out

there.

Trevor Sodorff: [00:56:46] Absolutely. Absolutely. And don't take me

the

wrong way. I'm not saying that FTD platforms, you know, aren't of value in, and of themselves because I think, I think they are, if you can figure them, right. I just don't like the idea of talking with one manufacturer the other day and I was asking him, Oh, what kind of fault rules do you have?

And it's like, Oh, we have a whole library of proprietary fault rules and it's like, can I see them? Are they in? Oh yeah, we'll get you on the NDA. And then we'll be able to send you some of this back information. It's like, Screw that, I can rewrite all of these, of just get them out of an Excel file from 22.

I love

that. Yeah. I had this client a couple of years ago and it was with one of the big four and the one of the big forwards doing FTD and what we're trying to do. And I've told this on one of my friendly rant episodes, we were trying to figure out what the existing rules did so that we could just program new ones to supplement them.

That's all we were trying to do. And we had to go through. a lawsuit to get these existing rules. And it was just like the easiest, like, basically the NIST air handler rules. This is what we were asking for. Right. And while we got back was so you're going to have to sign an NDA and anyone who sees this, like, spreadsheet with these rules is going to have to like sign their life away.

I totally agree with you. It's crazy out there. It's not

proprietary. people treat this, like it's like it's proprietary. And frankly, I would love to start an online Rebo maybe as part of this project than we ever did there, Keith of all the different it's available and just have it use it. I mean, you're still going to have people out there that are rock stars and that can help you to utilize it and take advantage of it.

And there's still going to be people out there that kind of stuck with it. And I mean, that's always the state of every industry. I think my goal would be a seat. All boats rise as this technology advances and the idea of these little pockets of like proprietary knowledge is being orthogonal to that motivation.

James Dice: [00:58:42] Brilliant. I love the rants. so, okay. So I guess my last question is around. don't know that I'm fully grasping the scale here. Right. So tell me about like lessons you've learned around doing this at such such scale. And, what would I not understand about that?

Trevor Sodorff: [00:59:01] Jeez. You want to take the first step? Maybe it'll take a second step.

Keith Berkoben: [00:59:03] Well, so I guess I'm gonna sort of skirt the question in some ways and say, one of the things that we haven't had to worry about, Is actually the scaling of the infrastructure that runs this stuff. And that's been really great because effectively what we've done is pushed all of this up through, you know, various cloud technologies.

Obviously there are cloud technologies cause we're an internal Google team, but like, the fact that. We have a, you know, big table and that we have IOT core and, that we have, um, Apache beam, you know, Basically make the

infrastructure part

of this trivial. You could do this for a massive, massive fleet of buildings, um, sending your data all the time and like it just scales horizontally forever.

The hard part is the part that we make Trevor do, which is actually, getting the data, you know, cleaned up from all these buildings and getting the coverage of all the equipment.

Trevor Sodorff: [01:00:10] Yeah. So the initial modeling of the core concepts that we can use, replicate them from place to place, like

novel concept requires

some deliberation about how you might model, which is kind of a hard part for this.

And it doesn't exist for the other ontologies because I think they just sort of sidestep the issue. It's a you know, choose your own adventure kind of thing. And we fundamentally don't want ours to be like that.

Keith Berkoben: [01:00:32] Okay. Um, but

Trevor Sodorff: [01:00:33] then the second piece is like, how do I rinse some bladder, repeat, take like a spreadsheet or an initial discovery from a backend device and convert it.

And so I've pilot  some software that I wrote

that

hopefully Keith and his team will be taking and running with from a workflow perspective that we'll be able to add to our open source project to help them. Aid the exploration of the ontology, like say that I got this device with a set of fields. I'd love to be able to just ask an application, Hey, give me the thing in the model that is closest maps to this.

so we've got some software that currently does that, but like, it needs to get added to the, platform. And so I would say that that's probably the piece that certainly we have not solved yet. But we're going to be focusing on it and trying to solve at least a piece of it. And that might also don't think anyone else in the industry has bothered to grapple with, for reasons that we've kind of already discussed.

James Dice: [01:01:22] Okay.

Trevor Sodorff: [01:01:22] I mean, Yeah, that's the first hurdle. If you can get through that, you can get through that integration layer, like Kate's platform does, you know, quality assessments of data that come through and most good platforms do this. I've worked with clients that have built their own and obviously in Google, but also in Azure.

Uh, I've worked with clients that have done this using third-party softwares. And frankly, we're not like for Google specifically, like, yeah, use that, but from my perspective, as an outside consultant,

I don't really care what people

use as long as they follow some of these core principles

James Dice: [01:01:52] So last question, I always do this where I said, it's the last one. And then I come with another one, sorry. Um, to go was building a new building tomorrow, right? And it's in some country. How do you guys enforce this? So like, if I'm an organization listening to this and I'm going to build a new building tomorrow and I want to use this, how do you guys make the contractors and make the output of a new building?

meet the standard.

Keith Berkoben: [01:02:15] Required in the contract, actually. I'm not kidding about that. Um, so there's the requirements part, which is literally like you put it in the contract, and then there's the, making it feasible for them to do it part. And the second part's a lot harder. So, conceptually the goal is to bring the model information as high up the design stack as possible, because what we've found and, and basically, so you think about the evolution of this.

We started with nothing and then like we started building this ontology and. Now all of a sudden we have this thing that we want to implement in buildings, but these buildings have already been in construction and we're like, okay, we want to attack this on at the end. And what we've learned basically is that the closer and closer you get to FTO B, the.

less time people have and the more stressful it is and the harder it is to do anything. And so what we've learned as we're doing this is the more mature we get the farther up the design stack, we bring this stuff. And so the ultimate goal. Is that when you have your mechanical designer doing something in BIM, they can say, okay, I'm putting in this piece of mechanical equipment or this piece of fire system or whatever I can select in my BIM plugin, which type from the ontology it is.

And now the information on like what fields are required to be mapped, you know, just kind of. Comes out of the design process and you could just hand it to your integrator and say like, this is what it needs to look like.

And so that's what

we're trying to get to. but right now we're kind of in that intermediate stage where like, we've now gotten the MSIs on board, early in the process of commissioning, but we're not.

so mature yet that we can literally just design the building with this built in. Got it.

James Dice: [01:04:08] Yeah. And I would imagine, like, there's still this aspect of just like, we're commissioning an actual building system. We then need to still commission the digital side of it as well. Like, so is there lot that pops up around enforcing it after the fact?

Cause I can just picture a contractor being like it's done.

Keith Berkoben: [01:04:26] yeah, Trevor said it earlier. no human should do this. And we do have automation that basically uh, we require that they give us what we call building config. So it's a Yammel configuration that tells us all about, what equipment's there, what its relationships are, what types they are, et cetera.

we ingest that. We validate it. We make sure it's Self-consistent and then we actually will validate the telemetry coming up to the cloud to make sure that it matches the model automatically. And then just hand you your error report, please go fix. Got it.

Trevor Sodorff: [01:05:03] you can. Train them in how to apply and build the building config model.

We have tooling in place to confirm that it's correct. And like I've said, one of the things

we want to kind of

extend this to is number one up the design stack. So getting into BIM into the design phase earlier, but number two, like you know, creating this model after the BMS has already been implemented, it's still going to be kind

of a

common business as usual process.

And so making it easier for an MSI, with minimal training and with, their kind of the current workflow is working with spreadsheets or whatever it is they do. how do we make it so that, that process sucks the least that it can.

Keith Berkoben: [01:05:42] Yeah. and you know, one other thing that was really interesting, that I didn't even think of initially, with this modeling stuff, which is that, When you define a model, you're making some statements about actually how you want something to be built.

And what we found, is that. we do actually have strong opinions about how we want things to be built, you know, in mechanical systems and his example that weren't really getting, followed in practice. And when you have a model with sort of some pre-made entities that represent your expectations, you can actually.

catch these, design anomalies early on because your designer says, hi, um, there's nothing in your model to represent this thing that I want to make. And then we say, no, don't make that thing right. Make this other thing follows our standards. And so that was really interesting. I didn't even think about.

That when we started doing this whole modeling exercise, and then we found this one building that we built where, you know, it was not at all to how we expected, you know, the mechanical system to be built. and we were like, Oh, if we had just given the model earlier, we would have, we would have caught this and not had a building that was weird.

James Dice: [01:06:58] Right. Fascinating. So

I I've been preaching the power of like control standards, which include. Profiles and, pieces of equipment and points lists for those profiles. And then obviously the data model on top of that. But when the data model already includes those things, you can just start with the data model.

That's really interesting.

Trevor Sodorff: [01:07:16] Yeah. And just to give a concrete understanding of what was happening there, we were seeing on these brand new buildings, pressure dependent, VAVs

being

specified without PID control or anything like that. And it was just like, Don't do that.

Like that, that was the sort of stuff where it's like, if I saw a dual duct VAV,

our

model,

do I want to

extend them all? Or did I want to just shame them?

Yeah,

I'm

sure somebody, again, like you know, I've done some mechanical design and stuff like that. I'm sure that somebody out there has a great, you know, the reason for

James Dice: [01:07:51] pressure dependent view boxes.

I'm not sure about that. Yeah. I'm not sure about that. Okay.

anyway, that's a, that's a wrap. Thanks so much guys for coming on the show. we'll have to do another to get an update sometime, but thanks for doing what you do.

Trevor Sodorff: [01:08:05] Yeah. Thanks for having some fun.

James Dice: [01:08:07] Alright, friends. Thanks for listening to this episode of the Nexus podcast. For more episodes like this and to get the weekly Nexus newsletter, please subscribe at nexus.substack.com. You can find the show notes for this conversation there as well. As always, please reach out on LinkedIn with any thoughts on this episode.

I'd love to hear from you. Have a great day

⭐️ Pro Article

This article is for Nexus Pro members only

Upgrade to Nexus Pro
⭐️ Pro Article

This article is for Nexus Pro members only

Upgrade to Nexus Pro

Are you a Nexus Pro member yet? Join now to get access to our community of 600+ members.

Join Today

Have you taken our Smart Building Strategist Course yet? Sign up to get access to our courses platform.

Enroll Now

Get the renowned Nexus Newsletter

Access the Nexus Community

Head over to Nexus Connect and see what’s new in the community. Don’t forget to check out the latest member-only events.

Go to Nexus Connect

Upgrade to Nexus Pro

Join Nexus Pro and get full access including invite-only member gatherings, access to the community chatroom Nexus Connect, networking opportunities, and deep dive essays.

Sign Up