Learn about NexusCon!
"Benefits to the university from this type of project and the work that we've done on this are twofold. The first is energy efficiency by having more standardized infrastructure and bringing data into SkySpark for fault detection diagnostics. We've been able to continue the energy savings improvements that the University of California, Irvine, has really been known for. In addition, there's also through fault detection diagnostics, we are able to... Better identify issues with the buildings before the users do. So one of the, one of the advantages obviously is hot cold calls, being able to diagnose those more effectively, or even identify them before the users have an opportunity to call and complain..ā
ā
āJoseph Fleshman
Welcome to Nexus, a newsletter and podcast for smart people applying smart building technologyāhosted by James Dice. If youāre new to Nexus, you might want to start here.
The Nexus podcast (Apple | Spotify | YouTube | Other apps) is our chance to explore and learn with the brightest in our industryātogether. The project is directly funded by listeners like you who have joined the Nexus Pro membership community.
You can join Nexus Pro to get a weekly-ish deep dive, access to the Nexus Vendor Landscape, and invites to exclusive events with a community of smart buildings nerds.
Episode 154 is a conversation with Joseph Fleshman from University of California Irvine and Jim Meacham from Altura Associates and Altaire Systems.
Episode 154 features Joseph Fleshman from University of California, Irvine and Jim Meacham from Altura Associates and Altaire Systems and is our fourth episode in the Case Study series looking at real-life, large-scale deployments of smart building technologies. These are not marketing fluff stories, these are lessons from leaders that others can put into use in their smart buildings programs. Joe and Jim talk in depth about the systematic overhaul of University of California Irvineās building automation system, operational technology networking, and deploying fault detection and diagnostics software. Enjoy!
You can find Joe and Jim on LinkedIn.
Project overview (01:49)
Project origins (05:34)
Building automation controls as a enterprise system (11:42)
Whatās next as the project continues (18:05)
Network infrastructure before and after (19:51)
Setting up building automation (22:18)
Fault detection and diagnostics (29:26)
Project challenges and lessons learned (35:37)
Playbook for other universities (48:13)
ā
Music credits: There Is A Reality by Common Tigerālicensed under an Music Vine Limited Pro Standard License ID: SS478748-15083.
Note: transcript was created using an imperfect machine learning tool and lightly edited by a human (so you can get the gist). Please forgive errors!
[00:00:00] James Dice: Hey friends. If you like the Nexus podcast, the best way to continue the learning is to join our community. There are three ways to do that. First, you can join the Nexus Pro Membership. It's our global community of smart billing professionals. We have monthly events, paywall, deep dive content, and a private chat room, and it's just 35 a month.
[00:00:23] Second, you can upgrade from the Pro Membership to our courses offering. It's headlined by our flagship course, the smart building strategist. And we're building a catalog of courses taught by world leading experts on each topic under the smart buildings umbrella third. And finally, our marketplace is how we connect leading vendors with buyers, looking for their solutions.
[00:00:42] The links are below in the show notes, and now let's go on to the podcast.
[00:00:49] Alright, welcome to the Nexus podcast. This is the fourth episode in a new series diving into case studies of real life, large scale deployments of smart building technologies. [00:01:00] And I emphasize real life again because we're not here to create a fluff story. We're here to share lessons from leaders that others can put to use in their smart building programs.
[00:01:10] And I also emphasize large scale because we're not here to talk about pilot projects. We hear a lot about pilot projects, but this is, uh, large scale deployments, deeper commitments to integrating technology into operations and actually changing the way that our buildings are operated. And today we have a story coming out of the University of California, Irvine on their journey, sort of overhauling their building automation system.
[00:01:33] They're operational technology networking and deploying fault detection and diagnostic software. So first I want to introduce Joe Fleshman. He's the interim director of energy engineering at the university. Welcome, Joe.
[00:01:48] Joseph Fleshman: Thank you.
[00:01:49] James Dice: And we're going to start with some sort of rapid fire questions to sort of frame this case study. So can you talk about who your vendor team is?
[00:01:57] Joseph Fleshman: So, we've, we've worked with Altura, uh, [00:02:00] to identify multiple manufacturers for equipment, um, so we have multiple, uh, vendors that can provide the hardware, um, but on our, as far as our vendor team for implementation, um, Altura Associates is our, is our commissioning agent that implements SkySpark integration.
[00:02:15] But we have, uh, selected multiple vendors through a competitive solicitation for implementation in the Niagara system. Um, so currently that's MCOR and Climatech. Um, but that's a competitive solicitation that we do every couple of years to bring new or return vendors on board, um, that can work on our system.
[00:02:32] And the goal there being, uh, you can have multiple vendors that can provide a seamless integration, um, with a consistent look and feel regardless of which vendor is actually doing the implementation.
[00:02:42] James Dice: Right. So you're not locked into any, any one vendor. How many buildings are we talking about here?
[00:02:47] Joseph Fleshman: Uh, right now on the Niagara system, we have 23 buildings, which is about a quarter of our portfolio.
[00:02:52] When did this project start? In 2016, the University of California, Irvine, uh, wanted to proceed with enhanced commissioning, [00:03:00] uh, via the SkySmart platform and do, uh, fault detection and diagnostics.
[00:03:04] James Dice: If you were to go talk to the university president, what would you say to them in terms of. him or her in terms of how this impacts the university's like big picture goals, whether it be decarbonization or student experience or whatever.
[00:03:21] How does this impact, impact the, the university at a big picture level?
[00:03:25] Joseph Fleshman: Benefits to the university, uh, from, from this type of project and the work that we've done on this are, are twofold. The first is energy efficiency, um, by having Uh, more standardized infrastructure and bringing data into SkySpark for fault detection diagnostics.
[00:03:39] We've been able to continue the energy savings improvements that the University of California, Irvine, has really been known for. Um, in addition, there's also, uh, through fault detection diagnostics, we are able to... Better identify issues with the buildings before the users do. So one of the, one of the advantages obviously is, is if, [00:04:00] um, hot cold calls, being able to diagnose those more effectively, or even identify them before the users have an opportunity to call and complain.
[00:04:08] Because one of the, uh, main things about the university is both the student experience but also, uh, research facilities, right? So we need to make sure that research facilities are operating in tip top shape. In order for us to continue doing the important work that, that, that is required, um, as well as, you know, student spaces, making sure that they're operational so that we can provide the educational experience that they expect.
[00:04:27] James Dice: Cool. And what have been the total results since then? And that could be qualitative or quantitative, like, energy savings per year or things like that.
[00:04:35] Joseph Fleshman: Yeah, so we've, we've, uh, About 750, 000 a year in annual energy savings, but that's across, I think, multiple projects. I think over 20 buildings we've done, uh, smart commissioning through the SkySpark platform.
[00:04:48] Um, we've done, you know, regular retro commissioning type projects, um, or what we call monitoring based commissioning projects. We've done compressed air commissioning. Um, we've done some building automation system upgrades, [00:05:00] and when we do those building automation system upgrades, we take that opportunity to implement resets in buildings that may not already have them.
[00:05:07] Um, so we, we, we're replacing the hardware, we're replacing all the programming, we take the opportunity to implement supplier temperature resets, static pressure resets, those sorts of
[00:05:14] things.
[00:05:15] James Dice: All right, great. That's a great overview of the project. We're going to dive into that in detail here. Um, I also want to introduce Jim Meacham of Altura.
[00:05:22] Thanks for coming, Jim.
[00:05:23] Jim Meacham: Absolutely. Great to be here. Thanks, James.
[00:05:25] James Dice: So let's just have a conversation about this. I want to get into both of your perspectives as we, um, sort of unpack each piece of this. Um, but Joe, let's start with you sort of, why did you start down this path and what were the sort of the goals of this project?
[00:05:39] Joseph Fleshman: The trigger originally for starting the whole project was energy savings. Um, we've, we've, we've done commissioning and energy projects. Going back for at least 2007, um, University of California Irvine has spent a lot of money and saved a lot of energy. Um, that's why we're known for, for instance, our smart labs program and other, and other [00:06:00] things, um, you know, nationwide and worldwide.
[00:06:02] Um, when we started... Uh, the commissioning projects in 2016, uh, we wanted to use the SkySpark platform and expand our ability to, you know, really dial in buildings. Um, once we started on that project, though, we found things like network instability, um, from converting. So the first step in doing that project was we need to get the data out to SkySpark.
[00:06:24] So, uh, what we did was we went through and, and used the deferred maintenance. obsolete controllers from these buildings and upgrade them to be BACnet compatible. Once we did that, though, we identified that the network congestion became, uh, really difficult to manage, and controllers were getting knocked offline.
[00:06:43] We were having communication stability issues. It became a real big problem. So then, um, as we, as we started, Experiencing those issues, we worked with Altura to start isolating buildings onto their own networks, right, to limit how much traffic, because you had a bunch of buildings basically shouting at each other via [00:07:00] BACnet on a flat network, so we started segregating those buildings into their own silos so that they could talk to each other as well as our network.
[00:07:07] server, but not to anybody else, and that really cleaned up a lot of our automation, um, traffic, and that became sort of the new standard for going forward. So every new building that we, that we build or that we upgrade now goes into that sort of, uh, network, uh, network. Topology. Topology, yeah. So that's, that's where, um, that's where every new building and every new upgrade just follows that same format that's now been established for seven years.
[00:07:32] Um, the other thing that we realized was that, um, implementing these energy efficiency projects required us a lot of times to go to a proprietary vendor. In order to implement sequences of operations. So, you know, I, I want to re, replay at, at a supplier temperature reset and static pressure reset in a laboratory building.
[00:07:49] I have to go to that proprietary vendor that happens to be for that particular building. Um, and that locks us into having a one stop shop, which. You know, like, uh, you know, one [00:08:00] of the things that's important to the university is competitive solicitation and having options, um, so that what we did was we then said, Hey, you know what?
[00:08:06] Let's look at a open source platform that we can integrate those buildings into that would allow us to have more flexibility, both in the hardware side, but also in the implementation side. And that's how we ended up moving towards a Niagara framework. We worked with Altura to establish a Niagara server.
[00:08:22] Um, worked, worked with them to establish, uh, standards, point naming protocols, develop standardized graphics, working with our automation shop to make sure that they were satisfied with what the graphics would look like. And then we did a competitive solicitation to get vendors in that can do that work.
[00:08:37] And so that way we, we have multiple vendors that we can go to to implement those reset sequences, to implement those graphics. Um, and we're not dialed into one proprietary vendor per building that, you know, we, we don't have a choice but to go to them.
[00:08:51] Jim Meacham: Yeah. And what that really, what we were experiencing in the early days of those energy efficiency projects that were driven by SkySpark was [00:09:00] it's, it can be a cost problem, right?
[00:09:01] If you, if you don't have the ability to bid, but it's also just, you, you lose control of the timing. You know, we wanted to get those projects done now, and that didn't always fit with the resource capability of, of certain vendors who had different projects going on on the campus or otherwise. And so as much as Cost was an issue, but it was also largely, um, being able to control the destiny of the timing because time is of the essence when you're doing energy efficiency projects, you need those savings.
[00:09:32] James Dice: Yeah. And we get a lot of people that come into the Nexus community and start to, you know, take our courses and read our content. And they say, I thought I was going to learn about decarbonization. I thought I was going to learn about energy efficiency and we're, we're talking about networks and the different layers of the stack and the data layer and all of this.
[00:09:48] Right. And I have to like ask people to take a step back. Uh, a lot or zoom in a little bit and realize that your energy efficiency goals are a product of your tech stack, [00:10:00] right? A lot of times That's right. So I think pe-people, you know, if they don't listen to the details of this conversation, hopefully one takeaway is this, all this it stuff matters.
[00:10:10] when it comes to your decarbonization. Um, And another piece of this that I think is interesting is, um, there's a theme here that we're unpacking, right? Which is the old way to implement controls was to have each building have a full stack with one vendor. And you're talking about supervisory layer, um, sequences where You know, you had to go to that building's vendor to implement a control sequence.
[00:10:34] And now I think what you guys are talking about here, correct me if I'm wrong, is let's take those supervisory control sequences and let's move them higher in the stack and have them be centralized for the whole campus. Is that right?
[00:10:46] Joseph Fleshman: Yeah, that's, that's exactly right. Um, local field controllers shouldn't be doing any more work than they absolutely have to, and then what we do is we, we use the Niagara optimization server.
[00:10:55] We have two, two Niagara servers. We have the graphics servers that's your sort of front end, but then we have the [00:11:00] optimization server, and the optimization server does all those calculations for the building and then pushes those set points down to the local field level controllers or air handler controllers and, and, and, and tells them what to do.
[00:11:11] Um, which, which means that that's, That's centralized, it's sequences that can be reused, um, in the, by any vendor that's working in the, in the Niagara system, um, and it can be accessed remotely. You don't have to use either proprietary vendor hardware to get down into the field controllers or physically go out to the field controllers to do that programming.
[00:11:31] It's all high level, anyone with, you know, you know, in the automation shop with remote web access is able to access those sequences and can, and can modify or, or,
[00:11:42] James Dice: Absolutely. And, and Jim, you guys are helping other customers implement this sort of approach. Can you just talk about this trend? Um, and I say trend because this isn't the way that controls were done always, but I feel like it's being more and more accepted in terms of this is the best practice [00:12:00] moving forward.
[00:12:00] Jim Meacham: That's right. Yeah, we're early days in that transformation, but I think what we're realizing, particularly with campus clients or, you know, large corporate institutional clients is just like any other enterprise system, they need to treat building automation controls like a truly enterprise system. And so you wouldn't have, for example, you wouldn't have an SAP system that was like per building, right?
[00:12:25] You have an enterprise SAP system. You have an, you have a lot of these enterprise systems and you build standards around those and you have maintenance and you have centralized agreements. And so it really helps from a scalability perspective. And, you know, we're talking about like UC Irvine or it's 150 buildings on campus or more, right?
[00:12:47] So really have to bring a new level of thinking. Historically, The traditional building automation approach is, is just building by building. It's a, it's a total vertical silo. Um, and, and a [00:13:00] largely that's driven by procurement, um, because you have a capital project and it wants to procure everything in those boundaries and that's what happens and it gets handed over.
[00:13:10] So what we're, a lot of our clients are really starting to understand that. We need to own these systems as opposed to just allowing them to happen to us. And so that takes this new level of thinking of, Oh, I need to drive the standards. I need to have an, you know, enterprise administration. And then you can realize the benefits of that, but it takes a totally new way of procuring and thinking about how you're going to get these projects done to get the scale benefit.
[00:13:40] James Dice: And like Joe said, you're doing this while you're maintaining the ability to have choice at the field controller layer as well. Um, let's talk about the main phases of this. So if someone's thinking about like, how do I implement this in my building? I imagine this wasn't like a very Like a totally smooth process for you guys, but [00:14:00] can you talk about the sort of the main phases of this going back to, you know, 2016, like Joe said, when we started this.
[00:14:06] Joseph Fleshman: I think I'll like, I can tell you how we did it and then how we should have done it.
[00:14:10] Um, because I think that those are two different things. So, um, what, what we Decided to do, um, when we wanted to go with the SkySpark, we said, okay, well, we need BACnet to go in, to get into SkySpark. Um, we started upgrading field controllers, and that's when we ran into those networking issues. Um, that's, that is, I say, if, if I could do it again, we would have...
[00:14:32] Knowing what we know now, we would have started with the network portion first. What does the network regime look like? What are those IP addresses? How do we want to silo these buildings? How much, how much space, like IP addressable space, do we want to give to each building? Um, before we go through this conversion.
[00:14:47] Because some buildings, you know, especially new buildings, everything's IP addressable. I might need like a slash 23 network to fit all those laboratory... You know, devices on there with 512 IP addresses, or, you know, it's an older building that doesn't really [00:15:00] have a whole lot of zone control. I need a much smaller network domain, right?
[00:15:03] So we could plan ahead and have that information planned out and set up that network topology, but we kind of did it the other way around, and it was only Thank you. after suffering through months of controllers dropping offline and losing connections and throwing alarms that we realized that that was the problem and that we needed to redo the network topology.
[00:15:23] But regardless, we started upgrading field devices. We figured out that there were some network issues. We started, uh, you know, we did start pulling data into SkySpark so that we could proceed with our, um, with our commissioning projects. By 2018, We were putting major renovation projects also, um, so it's further expanding it.
[00:15:43] So initially it was just commissioning projects and then we did some major HVAC upgrades in some major laboratory buildings, um, about I think 60, 000 square feet of pot that we did four of them in a row. Um, and so we brought those into Niagara and SkySpark as part [00:16:00] of the project. So when I'm budgeting for those projects, I'm building that in.
[00:16:03] Um, that's how we make the change to get more and more stuff into. Uh, Niagara and SkySpark is building it into these major renovation projects or new building construction projects. So, by 2018, we're putting large renovation projects in there. By 2019, we set up a whole nother set of commissioning, uh, building, building commissioning projects.
[00:16:22] So, it's like another round from the 2016 cycle ended, 2019 cycle started. And then, um, you know, by In 2022, uh, we, like last year and then this year, we've started migrating additional buildings, uh, into the, um, into the, the Niagara and SkySpark platform. You know, one of the things I have right now is a project to replace air handlers across a half a dozen different buildings.
[00:16:45] As part of that, we've built into the budget that those buildings will be converted to Niagara when they get their new controllers and pulled into SkySpark. So that's, that's how it's... It's an incremental process, right? So if we talk about 2016, we brought the first group of like 10 buildings [00:17:00] into SkySpark.
[00:17:00] We're now at 23, you know, we'll be at 30 by the end of next year or more. So we, we, we, uh, over time are biting off pieces and pieces to, to get that ball rolling, that snowball rolling downhill to get us off of the proprietary, uh, systems and onto. The open platform.
[00:17:19] Jim Meacham: I think that's an important point just to highlight that I always recommend folks take the all of the above approach.
[00:17:27] If you have the standards and specifications that define what needs to be delivered, then you, you can. Have all your minor, minor capital projects, your major capital projects, your deferred maintenance projects. Every time you're really touching these equipment and doing any kind of upgrade is defined how we move that over and it can be strategically completed and.
[00:17:50] Especially for, I mean, for any client, really, like that's how you optimize the budget to actually get the work done. Because it's not like you can go in to an automation system like this and do it all [00:18:00] at once. It's not practical in any way.
[00:18:05] James Dice: And so I'd imagine that, like, if we look forward and say, okay, those are the phases that got you guys here.
[00:18:10] What are you guys doing next? It sounds like it's just go from 25 to 100 percent of the campus covered with this. Anything else to add? Besides that, in terms of what's next,
[00:18:21] Joseph Fleshman: as, as we have the opportunity, um, we're, we're trying to move as much stuff as we can into, into both, um, both Niagara and SkySpark.
[00:18:30] And, um, I will say that we actually have more than like 20 or 30 buildings in, in our SkySpark platform, because we also use the SkySpark platform for energy metering, uh, BTU metering, kilowatt metering. We actually use it for, um, for like internal recharge. For charging in internal campus customers, energy consumption.
[00:18:49] So we have, we have probably at least a hundred buildings worth of some sort of data in the SkySpark platform. But when we're talking about like heavy duty HVAC data for every zone in the building, [00:19:00] that's, that's something that's going to take, that's going to take time.
[00:19:03] Jim Meacham: Yeah. Yeah. The deep, deep data is about 30. All the way down to the zone level, and it's 146, um, that we touch in some way for energy metering or otherwise.
[00:19:15] Joseph Fleshman: Yeah, and when we, when we, uh, do things like air handler replacements, a lot of these are older buildings with pneumatics, so what we're doing is we're really just bringing like air handlers and exhaust fans and pumps.
[00:19:25] into the Niagara system, but what we've now done is establish that this is a Niagara building, right? This is a SkySpark building, and so if at some point in the future we go through and we do a major VAV retrofit and install DDC controls, we've already decided what system it's being integrated into.
[00:19:41] It's not, like, do we put it in the old system or the new system? Like, it's already on Niagara, it's already been moved over, that's the de facto standard of what you're going to do in the building.
[00:19:49] James Dice: Got it. Got it. All right. Let's talk like a little bit more and a little bit more detail around this approach.
[00:19:56] The first piece of this it sounds like is the network [00:20:00] piece. Can you guys talk about what, what is that piece? What are you doing there and how is that different from like what the networks were like before?
[00:20:08] Joseph Fleshman: Yeah, so, before we basically had, um, four flat networks, where every device lived on, you know, like, every Siemens device lived on one network, every Johnson device lived on another network, you know, and so, they were very limited in, as far as being segregated, it was, you'd have You know, a hundred devices on a single, on a single flat network, which when they're talking proprietary protocols, they weren't really conflicting with each other.
[00:20:35] Um, but once we migrated to, you know, BACnet and flash panels to BACnet or upgraded them to BACnet, it's basically you had a bunch of people in an arena all shouting at each other and none of the devices could hear themselves. And so they would drop offline. So what we, what we did was we said, okay, instead of being like, uh, uh, a single first two octets, and then everything else just follows after, um, what we did was we said, [00:21:00] okay, we're going to, we're going to make each building under a first and second octet, and then the third octet.
[00:21:06] In the IP address range is going to determine the building. So like, you know, it'd be x. x. 68 is this building, x. x. 70 is this building, and x. x. 72 is this building. So what we did was we established these individual networks. And then we had to work with, in some cases, a proprietary vendor. But if we were upgrading hardware, we would just...
[00:21:25] Start off with that IP range and move those, those, those, it's, it's kind of like taking everybody in that arena and putting them in their own little soundproof rooms. And then what we did was we established, um, on the, the, on the virtual machine that runs Niagara, virtual network interface cards that, Let that computer believe that it was on each one of those rooms, or in each one of those rooms, so basically the rooms are shouting at each other, everybody in that building is shouting at each other, and then they crack open the door and talk to the server, and then they close the door, um, is, is effectively how we set the system up, so it's peace and quiet [00:22:00] for the, for the vast majority of the network, unless they need to talk out, and then they can only talk to one outside entity, um, because you were having like, is, Uh, laboratory controllers in one building would be knocking, uh, legacy controllers offline in a building across campus and that was not sustainable.
[00:22:17] James Dice: Got it. Okay. All right. Let's talk about the building automation system piece. Um, can you talk about like, what are the, how are you setting that up? We've talked about it a little bit, but feel free to go into a little bit more detail. And then how's that different from what, what it was set up as before?
[00:22:31] Jim Meacham: So I'll start with before, right? We talked a little bit about this earlier in terms of the, the typical kind of, um, building by building approach, right? So historically it would be each building has supervisory controllers inside the building, talking down to all the field level networks. Um, you know, UCI being the scale that it is, did have centralized, already virtualized proprietary, [00:23:00] um, Servers for those networks, talking to those, those controllers.
[00:23:05] But, you know, it was very much a building by building process. Um, and you know, copy and paste these graphics. So they kind of look the same, but maybe not if it was a different controls tech who had his own version of graphics that they liked from their last project, right? So you got, it was a hodgepodge approach.
[00:23:24] And, and, and, and I think. You know, a little bit of, um, the UCI story too that, that, um, a lot of folks experience is, you know, Joe on the facility side doesn't control all of the, the, um, new construction projects, right? Heavily influenced and in collaboration, but. Those are kind of get delivered and then hand it over to a large part.
[00:23:48] That's just how it works at most universities. So there's even more of a kind of firewall of what's coming down the pike and, and how is it actually being handled? So again, just kind of building by building [00:24:00] and those silos. Um, and so the new infrastructure, uh, we spent a lot of time with OIT, the Office of Information Technology with UCI after we got into it.
[00:24:13] I mean, that was a big part of it was really breaking down the communications barriers and, and luckily Joe can speak this language and our team can speak this language, but a lot of folks, this is a fundamental barrier. If you can't sit down with the IT folks and the InfoSec, you know, information security folks.
[00:24:32] And talk their language, be able to build that confidence of how we can manage this on your network. You're generally going to get a no in terms of moving forward. So you have to be able to, to understand what does security mean for OT on IT and how do we navigate that? We spent. A year, probably really defining the parameters of how we would interface with them.
[00:24:57] And it was a lot of new, [00:25:00] uh, personalities and new interactions. Now it's great. Like we built all that understanding it's endemic to the organization. So you want a new building, you want to, you know, control it. Like it's all understood the devices are understood. So we've been through that, but that's, you don't want to underestimate that phase.
[00:25:17] You really have to invest in that, that it phase, which enables that. new BAS infrastructure. So now we have, as Joe mentioned, for the Niagara system, um, we have a centralized optimization server and servers that will grow over time. Multiple servers, you know, for different, as it scales, because you can't have everything in one.
[00:25:40] Um, and Then, um, the single graphical user interface. And we like to decouple that from a resource perspective. So you're not impacting that experience as a user. If you have a lot of users on, on the system. And so you can kind of decouple the horsepower there, um, and have people being able to work in both of [00:26:00] those simultaneously without having big impacts to the experience.
[00:26:04] Um, and so now, you know, we've got, uh, The ability to to work at both layers and so the new at the building layer now what's happening is we've removed all of those kind of supervisory controllers, things that you might expect to see like a Jace controller or something. Those aren't in, in the, in the standard.
[00:26:28] It's actually interesting because we started back in 20, probably 18 with Jace's as part of the infrastructure and quickly learned this isn't really scalable or manageable for UCI to have all of these, these Jace's from a cost perspective and just like management and license management and all these things.
[00:26:48] And then we kind of realized like. Wait, this network is actually bulletproof. Like, why do we, why do we need these JACEs? And so that was some of the learning, early learnings to get away from even, [00:27:00] even in Niagara environment. Um, we were realizing that those JACEs kind of locked you in too. If we, if we wanted, uh, to be able to rip the head, you know, maybe Niagara 10 years from now, isn't the best, uh, supervisory software.
[00:27:14] Maybe it's. You know, whatever's next, we want to have the, oh, the IT infrastructure and that this automation infrastructure that doesn't create any reliance or dependence on that hardware. Right. So we got those, we had, we did like, I think one or two buildings that had Jaysus and we realized, nope. Like those have to come off.
[00:27:32] We can do this all in that centralized environment. I've been working down that path ever since. So that was one of the good early lessons learned. And we still had, I think we got one building snuck through with using the wrong specification. So it was a lesson learned. We had developed the new specification.
[00:27:51] They still had the old spec and it got missed in the project. And we got to the end and said, there's Jace's in here. What happened? So you have to really [00:28:00] pay attention to what's, uh, what's being used when you're updating your standards.
[00:28:05] Joseph Fleshman: And I think your point on the network security is really important for how we use the SkySpark system.
[00:28:11] So we have, uh, Altura hosts a SkySpark server out in the cloud, which was a whole nother layer of communications about network security because we had to get, working with Altura, get our IT folks comfortable with a business to business VPN out to a Rackspace or Amazon Web Service cloud server that's, that's attached to our network, right?
[00:28:34] So those, those are the kinds of things that we had to work out to make sure that, that the appropriate security was in place, um, that, that they were satisfied with any firewall rules, that we, that we tighten it up as much as possible while still allowing them to the, to collect the data. And so now. Um, like Jim said, we've established this protocol.
[00:28:51] So if I need a new build, if a new building is being constructed, I email a guy and say, give me my, my new IP address. And he goes, okay, here's your range. And it's, it's a [00:29:00] very seamless back and forth. Like, cause all the bugs have already been worked out. It just took us kind of years to get there. So that's where we would encourage people to start those conversations early before you start converting hardware.
[00:29:11] Jim Meacham: That's right.
[00:29:12] James Dice: Yeah. And it sounds like they might want to call you guys. We probably don't have enough time on this podcast to go into all the lessons learned related to talking to the IT folks and sort of explain to them all the things that you guys are trying to accomplish and, you know, the best practices around that.
[00:29:26] The last piece of this was the FDD software. So it sounds like from, from, you know, talking to you guys offline and seeing your notes around this project, it seems like there's You're doing a lot of stuff. So you're not only doing FDD, which is like what SkySpark is, you know, traditionally thought of as, but you're also doing the energy, um, metering, uh, analysis and build back, like you said, and sort of trending around that.
[00:29:53] And then you're. It sounds like you're also using it in the commissioning process. So maybe using it to fire commands down to the systems to do functional [00:30:00] testing and things like that. Um, can you talk about how all these different ways you're using it? And then like, who is using it on campus? It's probably not just you, Joe, it sounds like this is a tool that's being used by many different departments. Can you talk about that a little bit?
[00:30:14] Joseph Fleshman: Yeah. So. I would say that the, we, we use SkySpark for multiple things, but trending and, and fault detection have been the two that it's been most valuable for up until recently. Um, so one of the things that we, you know, to be honest, we're, we, we don't have a lot of manpower to be able to comb through.
[00:30:34] Sparks and fault detection and go through trend analysis. And so that's where, you know, what we do is we hire a consultant to do the commissioning like Altura. So Altura comes in, sets up all the sparks, is able to, you know, make specific recommendations that we may not have the, the, the, the physical manpower to be able to comb through.
[00:30:51] Um, that's something that we're working on. Like, how do we, how do we better. Manage the sparks so that we can get those into our work order and system and stuff like that. That's that's some of the activities to [00:31:00] come, but you know, in the 2016 cycle, 2016 to 18, and then 2019 to 22, and then, you know, when we do future Niagara conversions, like the air handlers that I was mentioning before, Altura is there to support us to help augment our interiors.
[00:31:16] Um, as far as the other activities, uh, we do do a billing through it. Um, we also collect, for instance, like, uh, central plan data. Um, so central, we have central plan, like, what are all our chillers doing? What's our cogen doing? What's our electrical import on a one minute basis, right? Because that helps us, uh, sometimes when we need to transmit information to the utility or to, you know, hire up the, the, the university chain.
[00:31:40] Um, I would say the, the people who use the system, um, is really the automation group, as well as, you know, my team and the business office who does the, the recharge. So, um, we have, uh, an automation technician who's usually in front of the computer and he'll have multiple, you know, because we have multiple automation systems still, he'll have multiple windows [00:32:00] up plus the SkySpark because he gets a hot cold call.
[00:32:03] He can dive into the SkySpark trend data, look at that piece of equipment and help diagnose so that when he calls the HVAC tech out in the field, he can point him in the right direction. Like, hey, You know, check the strainer on your hot water coil because your valve's open, but we're not getting, you know, a change in temperature.
[00:32:18] And so he can do the initial troubleshooting that helps the field technicians be more cost effective and time effective, um, and, and, and resolve those, those customer issues.
[00:32:29] Jim Meacham: Yeah, one of my observations, it's kind of fun to hear how it gets used because some, you know, you set it up for certain uses and you, you never know, but some of the fun things that have happened over time.
[00:32:40] I mean, one is just the flight data recorder aspect where. Something happens, even like water leaks, like a water leak happens. Could, could, can we see that in the data? We've, we've explored that. Um, but, but for sure. And some of these high profile, like [00:33:00] lab incidents or things that happen. Where as opposed to, it used to kind of be pointing fingers, like what, what really happened, we can go back to the data, Joe can pull up the data and say, Nope, like everything was working fine.
[00:33:14] And, and we see, you know, the user did this or whatever happened, like the system was fine. And we actually have like the flight data recorder to say, Okay. Everything was okay or wasn't it? And we had a, this failure and we know what it is and we can do root cause analysis and go, go address the issue. So I think foundationally that's kind of, you take it for granted after a while to have that type of access, cause you think, Oh, I've got an automation system and it can do trends.
[00:33:42] But even if you have the trends, the lift to go get those data from these automation systems is, is really 10x what you can do with a platform like SkySpark, it's like three clicks and you can get whatever data you want.
[00:33:55] Joseph Fleshman: Well, and you have to have the foresight to set up the appropriate trends [00:34:00] for that particular, like, so an example of like what Jim's saying is we had, we had a laboratory where.
[00:34:05] Um, they were doing really, uh, costly experiments with a particular sensitive piece of equipment and they lost a run because the, the, um, temperature and humidity changed too quickly in the room. Well, what we were able to do is go back in SkySpark and identify exactly when the air handler went down and exactly when their temperature went out of spec.
[00:34:24] And we were able to hand them that trend data so that they can then go, for instance, to the insurance company, right? So they could, they could use that as, as like, here's exactly what happened. It wasn't our fault. It was caused by, you know, this failure of this piece of equipment. And we have, because I would have never thought, Oh, I should trend the relative humidity and temperature of this one particular lab and the status of the air handler.
[00:34:44] Like, that's not the kind of thing that you would have the foresight. I'm sure people run into that where something happens and they're like, Man, I really wish I had that data. And you never know when you're going to need it until it comes up. And so that's what having that flight data recorder is really valuable for us.
[00:34:59] Um, [00:35:00] for either identifying what went wrong or what didn't go wrong.
[00:35:03] Jim Meacham: Yeah. I think another really impactful use case has been measurement verification. The tools that we have. To have real time M& V on these buildings, even post project, um, you know, Joe mentioned still really working on how do we optimize the ongoing kind of commissioning internally to UCI, but it's all there and we have, um, you know, you can very easily go look at like, how was this building performing?
[00:35:32] How should it be performing? And how is it performing? Very easily.
[00:35:37] James Dice: Totally. All right, let's dive into the sort of lessons learned here. Um, can you guys talk about, I think there's probably, I think you guys talked about three big challenges here. Can we talk about what's challenge number one? And then how did you sort of Resolve that.
[00:35:53] Joseph Fleshman: Yeah. So I think challenge number one, we've talked about a little bit is the, is the flat network, um, and not having foresight into [00:36:00] what that network configuration should look like. We charged ahead with BACnet conversions, not knowing that it was going to cause the problems that it did. Um, so working with, working early with your OIT, uh, enterprise networking team to establish what that virtual LAN scheme looks like, setting up the subdomains that are set aside for building automation.
[00:36:18] Um, and then that way, when you go to do your hardware available Or you go to convert the protocol that they're, that those devices are talking, you've already sort of set the stage that you're not going to have a disruptions.
[00:36:31] James Dice: Yeah, yeah, I really focus on that network infrastructure.
[00:36:33] Jim Meacham: And I would say a follow on to that from a lesson learned perspective is when you move to any type of enterprise, particularly enterprise BACnet network.
[00:36:43] It has to be actively managed when you have all of these different projects going on and all of these different systems that are jumping on and talking back net lighting systems, metering systems, HVA system, lab control systems. I mean, we've got all kinds of stuff out there and. [00:37:00] It's happening all the time.
[00:37:01] There's lots of vendors in the system, right? Because now it's truly multi vendor and the network, if someone puts the wrong network number or duplicate, you know, instance IDs or things like that, all of a sudden devices over in this building start going offline that have nothing to do with where that work was going on.
[00:37:20] So we've actually now got a process where Where our team meets with the automation shop every week. We do, um, BACnet captures. We do health scores using visual BACnet, uh, and we're looking proactively looking for is something is, is the health being maintained? Is there anything that's making this more brittle and we're back checking other vendors work so that we don't.
[00:37:44] get to an issue where it's taking two minutes for the network to update or something like that. We've been there and it's been painful. And if you let that go too long, it takes a long time to reel it back and really trying to, you know, stay up to date with how the network [00:38:00] health is looking. Totally. All right, what's challenge number two and how is that resolved?
[00:38:04] Joseph Fleshman: So I think challenge number two, uh, is more related to the SkySpark integration, which would be establishing that, that business to business VPN network connection, working with the Office of Information Technology network security people to make sure because you're not going to accomplish anything if they're not comfortable with what you're asking them to do.
[00:38:23] And so, you know, working, working with them, getting them comfortable, having People who can talk their language, um, come in and that's where, you know, Jim and I and his team can kind of talk that language with the IT people about networking and IP addresses and firewalls and ports and all that sort of stuff.
[00:38:38] Um, and so we were able to gain their confidence and work with them to establish something that they were comfortable with. Um, one of the advantages that that's provided for us is, is we don't just send data up to SkySpark for trending. Um, there actually have been cases where it made sense for us to do...
[00:38:54] What I used to call, when I was a commissioning agent, midnight testing. Um, they were able to send BACnet [00:39:00] commands to a building that, one of those major renovations in a laboratory building that I was talking about. And so what they did, what, what we did was at two in the morning, they sent BACnet commands from SkySpark to turn, uh, half the building into heating, and then into cooling, and then the other half of the building into heating and into cooling, and then they looked at the SkySpark data.
[00:39:20] To identify which zones weren't behaving the way that they were supposed to, whether that was a PID loop tuning, whether that was a hot water coil that wasn't, that was, you know, strainer needed to be flushed or, um, you know, air valves that were hunting, we were able to identify all of that for 100 percent of the building, which is different than like the way that commissioning used to be.
[00:39:39] I was a commissioning agent back in 2011, 2013, you'd walk around with like a radar gun and hit registers to make sure they were reheating, you know, like it was, It was, it's, it's so much more technologically advanced compared to the boots on the ground way of doing things in the past, where we could hit every single zone in the building digitally, um, and it would also flag [00:40:00] things like, hey, your temperature sensor is reading 32 degrees there.
[00:40:02] That's a bad, that's a bad stat. You got to go replace it. Um, all of this was able to be progressing quickly. Automated and then Altura combed through the data and made specific recommendations for the contractor on the project to repair as a punch list item.
[00:40:15] Jim Meacham: One of the interesting, yeah, that's, that's a good point.
[00:40:17] Yeah, the automated testing. It was actually one of the first projects, we did one of the first projects that we did automated testing on was at UCI back in the Must've been 2018. That was 17. 17. Yeah. So a long time ago now, like we've, we've been talking about automated testing for a long time. I guess we started that at Caltech actually, and then brought it to UCI.
[00:40:38] Um, but the contractor, the, the controls contractor, I remember in particular in that case had never been through such a rigorous process and it was shocking. So I think a lesson learned is really that we didn't do a good enough job of. Probably describing to them what the level of expectation was going to be, that we were going to be [00:41:00] doing a hundred percent like testing of everything, every single point in the system.
[00:41:06] And they really weren't prepared for that. And it, so the commissioning process took a long time. We got there because the data don't lie. There's nowhere to hide. The building performs awesome now, but. That was one of the longer tails, I would say, of a functional testing process that we've ever had because just took the bar, raised the bar quite a lot.
[00:41:27] Joseph Fleshman: Well and part of the reason why it ended up being successful was eventually we're showing data and they're like, can you just give us access to your SkySpark system so we can see this stuff ourselves? And then they could go in and they could check because they knew that Altura was going to come behind them and say this is hunting, this isn't working, this isn't working, so they went in and when they were addressing punch list items could actually make sure that things were working the way, looking at the exact same data that my commissioning agent would be looking at to make sure that it was actually fixed so that they didn't get called on the carpet for, you know, failing to do what the punch list item required them to do.
[00:41:58] James Dice: Minimize the back and [00:42:00] forth. Exactly. What's the, what's, yes. What's challenge number three and then the resolution?
[00:42:07] Joseph Fleshman: Um, I think managing multiple vendors. Um, One of the things that we, we actually did a pilot with Niagara with one of the local vendors. And it was originally before we sort of set up what our standards and graphics and, and everything would be.
[00:42:22] And it had their logo all over it. And it would didn't, it was emblazoned with every, you know, every page had their logo all over it. And, and there was no interface with like the automation system, the automation guys of what, what they want to see in the system. Um, and so, you know, we, We realized that like, if we're going to have different vendors in here, we need everything to look the same.
[00:42:42] Um, and we need everything to be, all the point naming has to be consistent. All the graphics have to be consistent. All the programming for the resets have to be consistent. And so that's, that is something that once we established those standards with Altera's support, it's enforcing that, right? [00:43:00] So, you know, one of the things that I tell the new vendor, the vendors when they come in, uh, to do the graphics, it's like, if you don't have a graphic, Tell me, and we'll get you a graphic.
[00:43:09] Don't build your own, because I want it to look exactly the same regardless of who's doing the graphic. And so we, we're just up front with them of like, you're basically taking cookie cutter modules, and you're putting them together, and that's how you're building the graphics. We don't want creativity, we don't want, you know, we don't want your special flair or touch or logo on it.
[00:43:27] We want you to be consistent because... I want to, if I have four different vendors working in the automation system, I want to be able to look at any building and not know which vendor did it, because it should look exactly the same regardless.
[00:43:39] Jim Meacham: That's, that was, we don't need your creativity. Yeah, that's right.
[00:43:43] Um, we, uh, one of my, the early wins, I would say when we knew we were finally like powering out of, you know, the, the, the initial phases of piloting some stuff and we, we built those standards and, and we did, we had three. different [00:44:00] projects that completed around the same time, and we're able to grab those graphics that, you know, all the way down to the zones and everything.
[00:44:08] And you couldn't tell, and we put them side by side and said, this is a win. You know, we, with three different vendors, same outcome for the automation shop, um, and you know, it really, that's, that was one, a good example.
[00:44:24] James Dice: One of the pieces I feel like we haven't talked enough about here is the actual, um, creation and sort of the, uh, adoption of the actual standards and specs.
[00:44:34] Can you guys talk about, I'm sure you got a lot of pushback from the incumbent vendors, like this isn't the right way to do it, nobody does it like this, kind of like, uh, sort of ingrained in the past, right? So can you talk about like that piece and how you sort of overcame this is a new way of doing it and this is the way we're doing it moving forward?
[00:44:56] Jim Meacham: Do you want to start on that? Yeah, I can start on it. That was a key part [00:45:00] of the process early on. Um, it's, it's, it was a couple fold, I would say. Um, one, we did a lot of outreach and engagement of the vendors. We didn't just go off into a silo and say, you know, we're cooking up this special thing and then we're going to like, just knock you over the head with it.
[00:45:20] It, we, we engaged them. They understood why, you know, a lot of the, the, the, the people who are associated with all of the vendor landscape at UCI are. They wanted UCI to succeed. They understood what was why UCI was pushing for a new way. Um, now we of course got pushed back and there were some different levels of technical understanding.
[00:45:48] Of things, you know, in proprietary vendors, we're, of course, trying to protect their, their systems, but eventually everyone got on board. Um, and [00:46:00] you know, those proprietary vendors are still doing work and doing Niagara work and have done Niagara work on the campus. So it's not just about saying like, Oh, we don't like you as a vendor.
[00:46:11] It's no, like now you're just doing it this way. And. That has worked. I mean, it's, it's a very similar vendor landscape. We've got some new ones, which is a big win. Cause we, that's one of the things we were looking for. It was more horsepower, uh, for the campus and more flexibility, but those same vendors are still there doing the work as well.
[00:46:30] Um,
[00:46:32] Joseph Fleshman: Yeah, and I would say that, um, it's been a stepwise process, especially as new technology has come out. So when we did our first building, um, that was in, that was one of those major renovations, um, that started to, like, okay, by default we're going to be missioning in SkySpark. That first project was still field controllers of the MSTP networks, because the BACnet, You know, Ethernet addressable individual terminal units hadn't really come out yet, right?
[00:46:59] So when we went and did our [00:47:00] project in 2018, where we did two additional buildings of similar scope, every one of those devices was a BACnet over Ethernet IP addressable device. What we've tried to do is, is, is move, move the ball as the technology also became available saying, yeah, just because we've done MSTP networks for the last 20 years, we're not doing that anymore.
[00:47:20] We're moving to BACnet. And so now our current specifications that's handed out to every contractor on any substantial sized job or new building construction says. Serial networks are prohibited. Everything has to be BACnet over Ethernet IP. Now, there are certain situations where you have a, you know, a minor renovation in like a couple rooms in an existing building that is on one of the proprietary systems and to migrate the entire 60, 000 square foot building because of a 2, 000 square foot space that you're renovating doesn't make sense and so we, we, what we do on the engineering side, um, the engineering group supporting the project managers is work with them to identify Which specification, the legacy spec or the new spec, is [00:48:00] appropriate for their job, but if they're doing substantial things like air handler replacements or major renovations of multiple floors of a building, we're pushing to have the new specification and then once that goes out to bid, that's the responsibility of the contractor to comply with that specification.
[00:48:13] James Dice: Love it. Okay, let's go. Our last four sort of five minutes or so here. I want you guys to walk us through if you were going driving down the road to the next university and they hadn't started any of this, what would be the playbook that you would walk them through? Um, or that you would want them to follow to sort of minimize these challenges, minimize these lessons learned, but sort of get this implemented as soon as possible.
[00:48:37] Um, and I'll just let you guys go. So, and I I don't think we need to, like, go into fine detail on each step because I feel like we've hit all of them at this point, but, like, what's the playbook, um, in sort of short, manageable statements here?
[00:48:51] Joseph Fleshman: Sure. So, um, one is get your internal stakeholders on board. That includes your automation people.
[00:48:56] That includes your renovation people. That includes your new building [00:49:00] construction people. All of them need to be on board, um, because if you're, if you're, for instance, your new buildings aren't being constructed with your new standards, then you're getting stuff turned over to you that. that is, is behind where you want to be.
[00:49:11] Um, the next one would be, uh, developing those clear standards and specifications, um, that you can put out to bid on, on, on jobs. Um, and then that's obviously working with outside vendors to identify what products might be appropriate for inclusion in that specification. Like, what we tend to do is like a technical specification that has performance requirements with a list of, with a list of vendors plus and or equal, where if someone wants to submit an additional You know, product that meets all of those technical requirements.
[00:49:38] We're happy to take a look at it. We want to bolster the amount of options that we have that comply with those, those open back net requirements. Um, also, uh, working with the information technology people for an it, an enterprise administrator to treat the system like an enterprise application, like, like, like Jim was talking about before, um, especially when you have a multi vendor environment.
[00:49:58] Um, Getting them access, getting them, [00:50:00] you know, to be consistent, um, and not have conflicts with things like instance IDs or IP addresses. Um, Developing a phasing plan for integration and hardware upgrades. So one of the things that I've heard from other universities is that I have a whole bunch of legacy stuff that talks to proprietary application.
[00:50:18] It's obsolete and I can't upgrade it to BACnet unless I rip the hardware out and put new hardware in. That's a substantial capital investment that you need to be aware of when you, when you go into it. Now, there's some vendors that are already BACnet, like natively, and there's some vendors that have legacy hardware that's not.
[00:50:37] Um, and so one of the ways that you could do that, for instance, is I'm going to target the ones that are already natively BACnet for my first phase of integration while I line up funding to replace the hardware that's just a proprietary network to upgrade it to BACnet, right? So you can get started once you establish those first couple things we talked about with the ones that are already natively BACnet because you've, you've prepped the hardware [00:51:00] already.
[00:51:00] It's just, it's natively ready while you sort of line up the funding. I don't know how other universities funding cycles are, but you know, on an annual basis you get state funding. So you, you're like. Planning for which, which ones can I upgrade next so that I can do an energy project on them 12 or 18 months from now.
[00:51:16] Um, so, so developing that phasing plan, uh, knowing what your hardware is and where it needs to get to, um, and then just monitoring your performance along the way and that's what we use the SkySpark platform for.
[00:51:27] Jim Meacham: I would add training. I think that's like continuous engagement. This is it's new. The topology is new.
[00:51:34] So everyone has to be trained. The internal automation shop folks. Um, and it's continuous. It's not like, oh, here's your half day of training. That was we we learned that lesson. It's it needs to be a much longer touch. Um, and and continuous touch. Uh, the vendors need a lot of training. We didn't really Kind of appreciate that.
[00:51:55] I think early on to say, Hey, these vendors also like are [00:52:00] living in their own little dynamic ecosystem, right? That their people change. And so do they really know the standards? So making sure there is that kind of continuous touch of training and oversight and feedback to the vendors that are servicing the network, like this is how it's done here.
[00:52:16] These are, these are the standards. Um, so that's, I think, been another important. Kind of part of the overall playbook.
[00:52:24] James Dice: Jim, can you add just real quick, the pieces when we say standards and specifications, can you, can you talk about the decisions that need to be made for that and, and the pieces of that that need to be figured out for people?
[00:52:36] Jim Meacham: Sure. Yeah. I mean, I, it really starts with the network architecture standard, as you know, Joe was saying about the IT standards. Um, a lot of folks just don't even have one. From our experience, there's a lot of controls drawings out there. There's not even a network architecture that's meaningful. Like what is the network actually going to look like when you're done?
[00:52:58] So having a standard of [00:53:00] what is that going to look like and getting the buy in from. That, that can work. That's step one. And then, uh, obviously Division 23 standards. If your, if your control specification lives in a Division 23, sometimes it's Division 25, um, is where that standard is, is living. So those, those need to be updated.
[00:53:23] And then, You have to close the loop with all the, all the folks that that impacts is what Joe is bringing up, like the stakeholder piece. There's, there's even within UCI, there's multiple organizations that implement that standard. And, and so while it's managed by Joe shop, it gets. Implemented by other folks.
[00:53:46] And, and so there's, there's kind of that continuous touch too. And then there's all the other divisional standards that, that get, um, impacted by this decision around architecture and integration path. [00:54:00] Um, so division 26, for example, electrical for metering and the related things on emergency power, anything else that might be on there, um, any of the specialties, you know, division 15.
[00:54:11] So. Division 27 is the network side, right? So there's a lot of, and then you just have to look at what is everything I expect to be integrated into my system. And then you have to go to every single one of those specifications and the folks that own those specifications and make sure the right language is in there that is going to result in the right topology, which then becomes to free up this kind of.
[00:54:35] Data layer, um, component that, that, that kind of breaks down the, um, the siloing of the systems.
[00:54:43] Joseph Fleshman: Even some of those, some of those things that you don't even think about. So you mentioned 23, 25, 26. What about 11 for cold rooms? Right? You want to pull your cold room data out there. Like, you may not think like, why would I look at environmental rooms?
[00:54:54] But, but we in, on our campus, we implement BACnet controllers [00:55:00] in our... Um, in our cold rooms, so that we can run an Ethernet cable and get that data out to the cloud, um, and, and monitor it, and see it, also see it on our graphics, rather than it sort of being the stand alone that something goes wrong with it, we don't hear about it until the alarm's going off and someone calls into the service desk.
[00:55:15] James Dice: Amazing. One of the things I love about this case study is, is all the different ways in which you guys have knocked down organizational silos through this process, which sounds, you guys are making it sound easy, but I know it hasn't been easy, even just the internal departments at the university. But now you're talking about the construction process and all those silos.
[00:55:35] And then all the people that are helping out on each renovation project and upgrade. Um, so bravo and thanks for telling the story on the podcast. I'm sure people are going to love, love hearing the story.
[00:55:47] Jim Meacham: Great. Thanks for having us.
[00:55:53] Okay, friends. Thank you for listening to this episode. As we continue to grow our global community of change makers, we need your help [00:56:00] for the next couple of months. We're challenging our listeners to share a link to their favorite nexus episode on LinkedIn with a short post about why you listen. It would really, really help us out.
[00:56:10] Make sure to tag us in the post so we can see it. Have a good one.
ā
"Benefits to the university from this type of project and the work that we've done on this are twofold. The first is energy efficiency by having more standardized infrastructure and bringing data into SkySpark for fault detection diagnostics. We've been able to continue the energy savings improvements that the University of California, Irvine, has really been known for. In addition, there's also through fault detection diagnostics, we are able to... Better identify issues with the buildings before the users do. So one of the, one of the advantages obviously is hot cold calls, being able to diagnose those more effectively, or even identify them before the users have an opportunity to call and complain..ā
ā
āJoseph Fleshman
Welcome to Nexus, a newsletter and podcast for smart people applying smart building technologyāhosted by James Dice. If youāre new to Nexus, you might want to start here.
The Nexus podcast (Apple | Spotify | YouTube | Other apps) is our chance to explore and learn with the brightest in our industryātogether. The project is directly funded by listeners like you who have joined the Nexus Pro membership community.
You can join Nexus Pro to get a weekly-ish deep dive, access to the Nexus Vendor Landscape, and invites to exclusive events with a community of smart buildings nerds.
Episode 154 is a conversation with Joseph Fleshman from University of California Irvine and Jim Meacham from Altura Associates and Altaire Systems.
Episode 154 features Joseph Fleshman from University of California, Irvine and Jim Meacham from Altura Associates and Altaire Systems and is our fourth episode in the Case Study series looking at real-life, large-scale deployments of smart building technologies. These are not marketing fluff stories, these are lessons from leaders that others can put into use in their smart buildings programs. Joe and Jim talk in depth about the systematic overhaul of University of California Irvineās building automation system, operational technology networking, and deploying fault detection and diagnostics software. Enjoy!
You can find Joe and Jim on LinkedIn.
Project overview (01:49)
Project origins (05:34)
Building automation controls as a enterprise system (11:42)
Whatās next as the project continues (18:05)
Network infrastructure before and after (19:51)
Setting up building automation (22:18)
Fault detection and diagnostics (29:26)
Project challenges and lessons learned (35:37)
Playbook for other universities (48:13)
ā
Music credits: There Is A Reality by Common Tigerālicensed under an Music Vine Limited Pro Standard License ID: SS478748-15083.
Note: transcript was created using an imperfect machine learning tool and lightly edited by a human (so you can get the gist). Please forgive errors!
[00:00:00] James Dice: Hey friends. If you like the Nexus podcast, the best way to continue the learning is to join our community. There are three ways to do that. First, you can join the Nexus Pro Membership. It's our global community of smart billing professionals. We have monthly events, paywall, deep dive content, and a private chat room, and it's just 35 a month.
[00:00:23] Second, you can upgrade from the Pro Membership to our courses offering. It's headlined by our flagship course, the smart building strategist. And we're building a catalog of courses taught by world leading experts on each topic under the smart buildings umbrella third. And finally, our marketplace is how we connect leading vendors with buyers, looking for their solutions.
[00:00:42] The links are below in the show notes, and now let's go on to the podcast.
[00:00:49] Alright, welcome to the Nexus podcast. This is the fourth episode in a new series diving into case studies of real life, large scale deployments of smart building technologies. [00:01:00] And I emphasize real life again because we're not here to create a fluff story. We're here to share lessons from leaders that others can put to use in their smart building programs.
[00:01:10] And I also emphasize large scale because we're not here to talk about pilot projects. We hear a lot about pilot projects, but this is, uh, large scale deployments, deeper commitments to integrating technology into operations and actually changing the way that our buildings are operated. And today we have a story coming out of the University of California, Irvine on their journey, sort of overhauling their building automation system.
[00:01:33] They're operational technology networking and deploying fault detection and diagnostic software. So first I want to introduce Joe Fleshman. He's the interim director of energy engineering at the university. Welcome, Joe.
[00:01:48] Joseph Fleshman: Thank you.
[00:01:49] James Dice: And we're going to start with some sort of rapid fire questions to sort of frame this case study. So can you talk about who your vendor team is?
[00:01:57] Joseph Fleshman: So, we've, we've worked with Altura, uh, [00:02:00] to identify multiple manufacturers for equipment, um, so we have multiple, uh, vendors that can provide the hardware, um, but on our, as far as our vendor team for implementation, um, Altura Associates is our, is our commissioning agent that implements SkySpark integration.
[00:02:15] But we have, uh, selected multiple vendors through a competitive solicitation for implementation in the Niagara system. Um, so currently that's MCOR and Climatech. Um, but that's a competitive solicitation that we do every couple of years to bring new or return vendors on board, um, that can work on our system.
[00:02:32] And the goal there being, uh, you can have multiple vendors that can provide a seamless integration, um, with a consistent look and feel regardless of which vendor is actually doing the implementation.
[00:02:42] James Dice: Right. So you're not locked into any, any one vendor. How many buildings are we talking about here?
[00:02:47] Joseph Fleshman: Uh, right now on the Niagara system, we have 23 buildings, which is about a quarter of our portfolio.
[00:02:52] When did this project start? In 2016, the University of California, Irvine, uh, wanted to proceed with enhanced commissioning, [00:03:00] uh, via the SkySmart platform and do, uh, fault detection and diagnostics.
[00:03:04] James Dice: If you were to go talk to the university president, what would you say to them in terms of. him or her in terms of how this impacts the university's like big picture goals, whether it be decarbonization or student experience or whatever.
[00:03:21] How does this impact, impact the, the university at a big picture level?
[00:03:25] Joseph Fleshman: Benefits to the university, uh, from, from this type of project and the work that we've done on this are, are twofold. The first is energy efficiency, um, by having Uh, more standardized infrastructure and bringing data into SkySpark for fault detection diagnostics.
[00:03:39] We've been able to continue the energy savings improvements that the University of California, Irvine, has really been known for. Um, in addition, there's also, uh, through fault detection diagnostics, we are able to... Better identify issues with the buildings before the users do. So one of the, one of the advantages obviously is, is if, [00:04:00] um, hot cold calls, being able to diagnose those more effectively, or even identify them before the users have an opportunity to call and complain.
[00:04:08] Because one of the, uh, main things about the university is both the student experience but also, uh, research facilities, right? So we need to make sure that research facilities are operating in tip top shape. In order for us to continue doing the important work that, that, that is required, um, as well as, you know, student spaces, making sure that they're operational so that we can provide the educational experience that they expect.
[00:04:27] James Dice: Cool. And what have been the total results since then? And that could be qualitative or quantitative, like, energy savings per year or things like that.
[00:04:35] Joseph Fleshman: Yeah, so we've, we've, uh, About 750, 000 a year in annual energy savings, but that's across, I think, multiple projects. I think over 20 buildings we've done, uh, smart commissioning through the SkySpark platform.
[00:04:48] Um, we've done, you know, regular retro commissioning type projects, um, or what we call monitoring based commissioning projects. We've done compressed air commissioning. Um, we've done some building automation system upgrades, [00:05:00] and when we do those building automation system upgrades, we take that opportunity to implement resets in buildings that may not already have them.
[00:05:07] Um, so we, we, we're replacing the hardware, we're replacing all the programming, we take the opportunity to implement supplier temperature resets, static pressure resets, those sorts of
[00:05:14] things.
[00:05:15] James Dice: All right, great. That's a great overview of the project. We're going to dive into that in detail here. Um, I also want to introduce Jim Meacham of Altura.
[00:05:22] Thanks for coming, Jim.
[00:05:23] Jim Meacham: Absolutely. Great to be here. Thanks, James.
[00:05:25] James Dice: So let's just have a conversation about this. I want to get into both of your perspectives as we, um, sort of unpack each piece of this. Um, but Joe, let's start with you sort of, why did you start down this path and what were the sort of the goals of this project?
[00:05:39] Joseph Fleshman: The trigger originally for starting the whole project was energy savings. Um, we've, we've, we've done commissioning and energy projects. Going back for at least 2007, um, University of California Irvine has spent a lot of money and saved a lot of energy. Um, that's why we're known for, for instance, our smart labs program and other, and other [00:06:00] things, um, you know, nationwide and worldwide.
[00:06:02] Um, when we started... Uh, the commissioning projects in 2016, uh, we wanted to use the SkySpark platform and expand our ability to, you know, really dial in buildings. Um, once we started on that project, though, we found things like network instability, um, from converting. So the first step in doing that project was we need to get the data out to SkySpark.
[00:06:24] So, uh, what we did was we went through and, and used the deferred maintenance. obsolete controllers from these buildings and upgrade them to be BACnet compatible. Once we did that, though, we identified that the network congestion became, uh, really difficult to manage, and controllers were getting knocked offline.
[00:06:43] We were having communication stability issues. It became a real big problem. So then, um, as we, as we started, Experiencing those issues, we worked with Altura to start isolating buildings onto their own networks, right, to limit how much traffic, because you had a bunch of buildings basically shouting at each other via [00:07:00] BACnet on a flat network, so we started segregating those buildings into their own silos so that they could talk to each other as well as our network.
[00:07:07] server, but not to anybody else, and that really cleaned up a lot of our automation, um, traffic, and that became sort of the new standard for going forward. So every new building that we, that we build or that we upgrade now goes into that sort of, uh, network, uh, network. Topology. Topology, yeah. So that's, that's where, um, that's where every new building and every new upgrade just follows that same format that's now been established for seven years.
[00:07:32] Um, the other thing that we realized was that, um, implementing these energy efficiency projects required us a lot of times to go to a proprietary vendor. In order to implement sequences of operations. So, you know, I, I want to re, replay at, at a supplier temperature reset and static pressure reset in a laboratory building.
[00:07:49] I have to go to that proprietary vendor that happens to be for that particular building. Um, and that locks us into having a one stop shop, which. You know, like, uh, you know, one [00:08:00] of the things that's important to the university is competitive solicitation and having options, um, so that what we did was we then said, Hey, you know what?
[00:08:06] Let's look at a open source platform that we can integrate those buildings into that would allow us to have more flexibility, both in the hardware side, but also in the implementation side. And that's how we ended up moving towards a Niagara framework. We worked with Altura to establish a Niagara server.
[00:08:22] Um, worked, worked with them to establish, uh, standards, point naming protocols, develop standardized graphics, working with our automation shop to make sure that they were satisfied with what the graphics would look like. And then we did a competitive solicitation to get vendors in that can do that work.
[00:08:37] And so that way we, we have multiple vendors that we can go to to implement those reset sequences, to implement those graphics. Um, and we're not dialed into one proprietary vendor per building that, you know, we, we don't have a choice but to go to them.
[00:08:51] Jim Meacham: Yeah. And what that really, what we were experiencing in the early days of those energy efficiency projects that were driven by SkySpark was [00:09:00] it's, it can be a cost problem, right?
[00:09:01] If you, if you don't have the ability to bid, but it's also just, you, you lose control of the timing. You know, we wanted to get those projects done now, and that didn't always fit with the resource capability of, of certain vendors who had different projects going on on the campus or otherwise. And so as much as Cost was an issue, but it was also largely, um, being able to control the destiny of the timing because time is of the essence when you're doing energy efficiency projects, you need those savings.
[00:09:32] James Dice: Yeah. And we get a lot of people that come into the Nexus community and start to, you know, take our courses and read our content. And they say, I thought I was going to learn about decarbonization. I thought I was going to learn about energy efficiency and we're, we're talking about networks and the different layers of the stack and the data layer and all of this.
[00:09:48] Right. And I have to like ask people to take a step back. Uh, a lot or zoom in a little bit and realize that your energy efficiency goals are a product of your tech stack, [00:10:00] right? A lot of times That's right. So I think pe-people, you know, if they don't listen to the details of this conversation, hopefully one takeaway is this, all this it stuff matters.
[00:10:10] when it comes to your decarbonization. Um, And another piece of this that I think is interesting is, um, there's a theme here that we're unpacking, right? Which is the old way to implement controls was to have each building have a full stack with one vendor. And you're talking about supervisory layer, um, sequences where You know, you had to go to that building's vendor to implement a control sequence.
[00:10:34] And now I think what you guys are talking about here, correct me if I'm wrong, is let's take those supervisory control sequences and let's move them higher in the stack and have them be centralized for the whole campus. Is that right?
[00:10:46] Joseph Fleshman: Yeah, that's, that's exactly right. Um, local field controllers shouldn't be doing any more work than they absolutely have to, and then what we do is we, we use the Niagara optimization server.
[00:10:55] We have two, two Niagara servers. We have the graphics servers that's your sort of front end, but then we have the [00:11:00] optimization server, and the optimization server does all those calculations for the building and then pushes those set points down to the local field level controllers or air handler controllers and, and, and, and tells them what to do.
[00:11:11] Um, which, which means that that's, That's centralized, it's sequences that can be reused, um, in the, by any vendor that's working in the, in the Niagara system, um, and it can be accessed remotely. You don't have to use either proprietary vendor hardware to get down into the field controllers or physically go out to the field controllers to do that programming.
[00:11:31] It's all high level, anyone with, you know, you know, in the automation shop with remote web access is able to access those sequences and can, and can modify or, or,
[00:11:42] James Dice: Absolutely. And, and Jim, you guys are helping other customers implement this sort of approach. Can you just talk about this trend? Um, and I say trend because this isn't the way that controls were done always, but I feel like it's being more and more accepted in terms of this is the best practice [00:12:00] moving forward.
[00:12:00] Jim Meacham: That's right. Yeah, we're early days in that transformation, but I think what we're realizing, particularly with campus clients or, you know, large corporate institutional clients is just like any other enterprise system, they need to treat building automation controls like a truly enterprise system. And so you wouldn't have, for example, you wouldn't have an SAP system that was like per building, right?
[00:12:25] You have an enterprise SAP system. You have an, you have a lot of these enterprise systems and you build standards around those and you have maintenance and you have centralized agreements. And so it really helps from a scalability perspective. And, you know, we're talking about like UC Irvine or it's 150 buildings on campus or more, right?
[00:12:47] So really have to bring a new level of thinking. Historically, The traditional building automation approach is, is just building by building. It's a, it's a total vertical silo. Um, and, and a [00:13:00] largely that's driven by procurement, um, because you have a capital project and it wants to procure everything in those boundaries and that's what happens and it gets handed over.
[00:13:10] So what we're, a lot of our clients are really starting to understand that. We need to own these systems as opposed to just allowing them to happen to us. And so that takes this new level of thinking of, Oh, I need to drive the standards. I need to have an, you know, enterprise administration. And then you can realize the benefits of that, but it takes a totally new way of procuring and thinking about how you're going to get these projects done to get the scale benefit.
[00:13:40] James Dice: And like Joe said, you're doing this while you're maintaining the ability to have choice at the field controller layer as well. Um, let's talk about the main phases of this. So if someone's thinking about like, how do I implement this in my building? I imagine this wasn't like a very Like a totally smooth process for you guys, but [00:14:00] can you talk about the sort of the main phases of this going back to, you know, 2016, like Joe said, when we started this.
[00:14:06] Joseph Fleshman: I think I'll like, I can tell you how we did it and then how we should have done it.
[00:14:10] Um, because I think that those are two different things. So, um, what, what we Decided to do, um, when we wanted to go with the SkySpark, we said, okay, well, we need BACnet to go in, to get into SkySpark. Um, we started upgrading field controllers, and that's when we ran into those networking issues. Um, that's, that is, I say, if, if I could do it again, we would have...
[00:14:32] Knowing what we know now, we would have started with the network portion first. What does the network regime look like? What are those IP addresses? How do we want to silo these buildings? How much, how much space, like IP addressable space, do we want to give to each building? Um, before we go through this conversion.
[00:14:47] Because some buildings, you know, especially new buildings, everything's IP addressable. I might need like a slash 23 network to fit all those laboratory... You know, devices on there with 512 IP addresses, or, you know, it's an older building that doesn't really [00:15:00] have a whole lot of zone control. I need a much smaller network domain, right?
[00:15:03] So we could plan ahead and have that information planned out and set up that network topology, but we kind of did it the other way around, and it was only Thank you. after suffering through months of controllers dropping offline and losing connections and throwing alarms that we realized that that was the problem and that we needed to redo the network topology.
[00:15:23] But regardless, we started upgrading field devices. We figured out that there were some network issues. We started, uh, you know, we did start pulling data into SkySpark so that we could proceed with our, um, with our commissioning projects. By 2018, We were putting major renovation projects also, um, so it's further expanding it.
[00:15:43] So initially it was just commissioning projects and then we did some major HVAC upgrades in some major laboratory buildings, um, about I think 60, 000 square feet of pot that we did four of them in a row. Um, and so we brought those into Niagara and SkySpark as part [00:16:00] of the project. So when I'm budgeting for those projects, I'm building that in.
[00:16:03] Um, that's how we make the change to get more and more stuff into. Uh, Niagara and SkySpark is building it into these major renovation projects or new building construction projects. So, by 2018, we're putting large renovation projects in there. By 2019, we set up a whole nother set of commissioning, uh, building, building commissioning projects.
[00:16:22] So, it's like another round from the 2016 cycle ended, 2019 cycle started. And then, um, you know, by In 2022, uh, we, like last year and then this year, we've started migrating additional buildings, uh, into the, um, into the, the Niagara and SkySpark platform. You know, one of the things I have right now is a project to replace air handlers across a half a dozen different buildings.
[00:16:45] As part of that, we've built into the budget that those buildings will be converted to Niagara when they get their new controllers and pulled into SkySpark. So that's, that's how it's... It's an incremental process, right? So if we talk about 2016, we brought the first group of like 10 buildings [00:17:00] into SkySpark.
[00:17:00] We're now at 23, you know, we'll be at 30 by the end of next year or more. So we, we, we, uh, over time are biting off pieces and pieces to, to get that ball rolling, that snowball rolling downhill to get us off of the proprietary, uh, systems and onto. The open platform.
[00:17:19] Jim Meacham: I think that's an important point just to highlight that I always recommend folks take the all of the above approach.
[00:17:27] If you have the standards and specifications that define what needs to be delivered, then you, you can. Have all your minor, minor capital projects, your major capital projects, your deferred maintenance projects. Every time you're really touching these equipment and doing any kind of upgrade is defined how we move that over and it can be strategically completed and.
[00:17:50] Especially for, I mean, for any client, really, like that's how you optimize the budget to actually get the work done. Because it's not like you can go in to an automation system like this and do it all [00:18:00] at once. It's not practical in any way.
[00:18:05] James Dice: And so I'd imagine that, like, if we look forward and say, okay, those are the phases that got you guys here.
[00:18:10] What are you guys doing next? It sounds like it's just go from 25 to 100 percent of the campus covered with this. Anything else to add? Besides that, in terms of what's next,
[00:18:21] Joseph Fleshman: as, as we have the opportunity, um, we're, we're trying to move as much stuff as we can into, into both, um, both Niagara and SkySpark.
[00:18:30] And, um, I will say that we actually have more than like 20 or 30 buildings in, in our SkySpark platform, because we also use the SkySpark platform for energy metering, uh, BTU metering, kilowatt metering. We actually use it for, um, for like internal recharge. For charging in internal campus customers, energy consumption.
[00:18:49] So we have, we have probably at least a hundred buildings worth of some sort of data in the SkySpark platform. But when we're talking about like heavy duty HVAC data for every zone in the building, [00:19:00] that's, that's something that's going to take, that's going to take time.
[00:19:03] Jim Meacham: Yeah. Yeah. The deep, deep data is about 30. All the way down to the zone level, and it's 146, um, that we touch in some way for energy metering or otherwise.
[00:19:15] Joseph Fleshman: Yeah, and when we, when we, uh, do things like air handler replacements, a lot of these are older buildings with pneumatics, so what we're doing is we're really just bringing like air handlers and exhaust fans and pumps.
[00:19:25] into the Niagara system, but what we've now done is establish that this is a Niagara building, right? This is a SkySpark building, and so if at some point in the future we go through and we do a major VAV retrofit and install DDC controls, we've already decided what system it's being integrated into.
[00:19:41] It's not, like, do we put it in the old system or the new system? Like, it's already on Niagara, it's already been moved over, that's the de facto standard of what you're going to do in the building.
[00:19:49] James Dice: Got it. Got it. All right. Let's talk like a little bit more and a little bit more detail around this approach.
[00:19:56] The first piece of this it sounds like is the network [00:20:00] piece. Can you guys talk about what, what is that piece? What are you doing there and how is that different from like what the networks were like before?
[00:20:08] Joseph Fleshman: Yeah, so, before we basically had, um, four flat networks, where every device lived on, you know, like, every Siemens device lived on one network, every Johnson device lived on another network, you know, and so, they were very limited in, as far as being segregated, it was, you'd have You know, a hundred devices on a single, on a single flat network, which when they're talking proprietary protocols, they weren't really conflicting with each other.
[00:20:35] Um, but once we migrated to, you know, BACnet and flash panels to BACnet or upgraded them to BACnet, it's basically you had a bunch of people in an arena all shouting at each other and none of the devices could hear themselves. And so they would drop offline. So what we, what we did was we said, okay, instead of being like, uh, uh, a single first two octets, and then everything else just follows after, um, what we did was we said, [00:21:00] okay, we're going to, we're going to make each building under a first and second octet, and then the third octet.
[00:21:06] In the IP address range is going to determine the building. So like, you know, it'd be x. x. 68 is this building, x. x. 70 is this building, and x. x. 72 is this building. So what we did was we established these individual networks. And then we had to work with, in some cases, a proprietary vendor. But if we were upgrading hardware, we would just...
[00:21:25] Start off with that IP range and move those, those, those, it's, it's kind of like taking everybody in that arena and putting them in their own little soundproof rooms. And then what we did was we established, um, on the, the, on the virtual machine that runs Niagara, virtual network interface cards that, Let that computer believe that it was on each one of those rooms, or in each one of those rooms, so basically the rooms are shouting at each other, everybody in that building is shouting at each other, and then they crack open the door and talk to the server, and then they close the door, um, is, is effectively how we set the system up, so it's peace and quiet [00:22:00] for the, for the vast majority of the network, unless they need to talk out, and then they can only talk to one outside entity, um, because you were having like, is, Uh, laboratory controllers in one building would be knocking, uh, legacy controllers offline in a building across campus and that was not sustainable.
[00:22:17] James Dice: Got it. Okay. All right. Let's talk about the building automation system piece. Um, can you talk about like, what are the, how are you setting that up? We've talked about it a little bit, but feel free to go into a little bit more detail. And then how's that different from what, what it was set up as before?
[00:22:31] Jim Meacham: So I'll start with before, right? We talked a little bit about this earlier in terms of the, the typical kind of, um, building by building approach, right? So historically it would be each building has supervisory controllers inside the building, talking down to all the field level networks. Um, you know, UCI being the scale that it is, did have centralized, already virtualized proprietary, [00:23:00] um, Servers for those networks, talking to those, those controllers.
[00:23:05] But, you know, it was very much a building by building process. Um, and you know, copy and paste these graphics. So they kind of look the same, but maybe not if it was a different controls tech who had his own version of graphics that they liked from their last project, right? So you got, it was a hodgepodge approach.
[00:23:24] And, and, and, and I think. You know, a little bit of, um, the UCI story too that, that, um, a lot of folks experience is, you know, Joe on the facility side doesn't control all of the, the, um, new construction projects, right? Heavily influenced and in collaboration, but. Those are kind of get delivered and then hand it over to a large part.
[00:23:48] That's just how it works at most universities. So there's even more of a kind of firewall of what's coming down the pike and, and how is it actually being handled? So again, just kind of building by building [00:24:00] and those silos. Um, and so the new infrastructure, uh, we spent a lot of time with OIT, the Office of Information Technology with UCI after we got into it.
[00:24:13] I mean, that was a big part of it was really breaking down the communications barriers and, and luckily Joe can speak this language and our team can speak this language, but a lot of folks, this is a fundamental barrier. If you can't sit down with the IT folks and the InfoSec, you know, information security folks.
[00:24:32] And talk their language, be able to build that confidence of how we can manage this on your network. You're generally going to get a no in terms of moving forward. So you have to be able to, to understand what does security mean for OT on IT and how do we navigate that? We spent. A year, probably really defining the parameters of how we would interface with them.
[00:24:57] And it was a lot of new, [00:25:00] uh, personalities and new interactions. Now it's great. Like we built all that understanding it's endemic to the organization. So you want a new building, you want to, you know, control it. Like it's all understood the devices are understood. So we've been through that, but that's, you don't want to underestimate that phase.
[00:25:17] You really have to invest in that, that it phase, which enables that. new BAS infrastructure. So now we have, as Joe mentioned, for the Niagara system, um, we have a centralized optimization server and servers that will grow over time. Multiple servers, you know, for different, as it scales, because you can't have everything in one.
[00:25:40] Um, and Then, um, the single graphical user interface. And we like to decouple that from a resource perspective. So you're not impacting that experience as a user. If you have a lot of users on, on the system. And so you can kind of decouple the horsepower there, um, and have people being able to work in both of [00:26:00] those simultaneously without having big impacts to the experience.
[00:26:04] Um, and so now, you know, we've got, uh, The ability to to work at both layers and so the new at the building layer now what's happening is we've removed all of those kind of supervisory controllers, things that you might expect to see like a Jace controller or something. Those aren't in, in the, in the standard.
[00:26:28] It's actually interesting because we started back in 20, probably 18 with Jace's as part of the infrastructure and quickly learned this isn't really scalable or manageable for UCI to have all of these, these Jace's from a cost perspective and just like management and license management and all these things.
[00:26:48] And then we kind of realized like. Wait, this network is actually bulletproof. Like, why do we, why do we need these JACEs? And so that was some of the learning, early learnings to get away from even, [00:27:00] even in Niagara environment. Um, we were realizing that those JACEs kind of locked you in too. If we, if we wanted, uh, to be able to rip the head, you know, maybe Niagara 10 years from now, isn't the best, uh, supervisory software.
[00:27:14] Maybe it's. You know, whatever's next, we want to have the, oh, the IT infrastructure and that this automation infrastructure that doesn't create any reliance or dependence on that hardware. Right. So we got those, we had, we did like, I think one or two buildings that had Jaysus and we realized, nope. Like those have to come off.
[00:27:32] We can do this all in that centralized environment. I've been working down that path ever since. So that was one of the good early lessons learned. And we still had, I think we got one building snuck through with using the wrong specification. So it was a lesson learned. We had developed the new specification.
[00:27:51] They still had the old spec and it got missed in the project. And we got to the end and said, there's Jace's in here. What happened? So you have to really [00:28:00] pay attention to what's, uh, what's being used when you're updating your standards.
[00:28:05] Joseph Fleshman: And I think your point on the network security is really important for how we use the SkySpark system.
[00:28:11] So we have, uh, Altura hosts a SkySpark server out in the cloud, which was a whole nother layer of communications about network security because we had to get, working with Altura, get our IT folks comfortable with a business to business VPN out to a Rackspace or Amazon Web Service cloud server that's, that's attached to our network, right?
[00:28:34] So those, those are the kinds of things that we had to work out to make sure that, that the appropriate security was in place, um, that, that they were satisfied with any firewall rules, that we, that we tighten it up as much as possible while still allowing them to the, to collect the data. And so now. Um, like Jim said, we've established this protocol.
[00:28:51] So if I need a new build, if a new building is being constructed, I email a guy and say, give me my, my new IP address. And he goes, okay, here's your range. And it's, it's a [00:29:00] very seamless back and forth. Like, cause all the bugs have already been worked out. It just took us kind of years to get there. So that's where we would encourage people to start those conversations early before you start converting hardware.
[00:29:11] Jim Meacham: That's right.
[00:29:12] James Dice: Yeah. And it sounds like they might want to call you guys. We probably don't have enough time on this podcast to go into all the lessons learned related to talking to the IT folks and sort of explain to them all the things that you guys are trying to accomplish and, you know, the best practices around that.
[00:29:26] The last piece of this was the FDD software. So it sounds like from, from, you know, talking to you guys offline and seeing your notes around this project, it seems like there's You're doing a lot of stuff. So you're not only doing FDD, which is like what SkySpark is, you know, traditionally thought of as, but you're also doing the energy, um, metering, uh, analysis and build back, like you said, and sort of trending around that.
[00:29:53] And then you're. It sounds like you're also using it in the commissioning process. So maybe using it to fire commands down to the systems to do functional [00:30:00] testing and things like that. Um, can you talk about how all these different ways you're using it? And then like, who is using it on campus? It's probably not just you, Joe, it sounds like this is a tool that's being used by many different departments. Can you talk about that a little bit?
[00:30:14] Joseph Fleshman: Yeah. So. I would say that the, we, we use SkySpark for multiple things, but trending and, and fault detection have been the two that it's been most valuable for up until recently. Um, so one of the things that we, you know, to be honest, we're, we, we don't have a lot of manpower to be able to comb through.
[00:30:34] Sparks and fault detection and go through trend analysis. And so that's where, you know, what we do is we hire a consultant to do the commissioning like Altura. So Altura comes in, sets up all the sparks, is able to, you know, make specific recommendations that we may not have the, the, the, the physical manpower to be able to comb through.
[00:30:51] Um, that's something that we're working on. Like, how do we, how do we better. Manage the sparks so that we can get those into our work order and system and stuff like that. That's that's some of the activities to [00:31:00] come, but you know, in the 2016 cycle, 2016 to 18, and then 2019 to 22, and then, you know, when we do future Niagara conversions, like the air handlers that I was mentioning before, Altura is there to support us to help augment our interiors.
[00:31:16] Um, as far as the other activities, uh, we do do a billing through it. Um, we also collect, for instance, like, uh, central plan data. Um, so central, we have central plan, like, what are all our chillers doing? What's our cogen doing? What's our electrical import on a one minute basis, right? Because that helps us, uh, sometimes when we need to transmit information to the utility or to, you know, hire up the, the, the university chain.
[00:31:40] Um, I would say the, the people who use the system, um, is really the automation group, as well as, you know, my team and the business office who does the, the recharge. So, um, we have, uh, an automation technician who's usually in front of the computer and he'll have multiple, you know, because we have multiple automation systems still, he'll have multiple windows [00:32:00] up plus the SkySpark because he gets a hot cold call.
[00:32:03] He can dive into the SkySpark trend data, look at that piece of equipment and help diagnose so that when he calls the HVAC tech out in the field, he can point him in the right direction. Like, hey, You know, check the strainer on your hot water coil because your valve's open, but we're not getting, you know, a change in temperature.
[00:32:18] And so he can do the initial troubleshooting that helps the field technicians be more cost effective and time effective, um, and, and, and resolve those, those customer issues.
[00:32:29] Jim Meacham: Yeah, one of my observations, it's kind of fun to hear how it gets used because some, you know, you set it up for certain uses and you, you never know, but some of the fun things that have happened over time.
[00:32:40] I mean, one is just the flight data recorder aspect where. Something happens, even like water leaks, like a water leak happens. Could, could, can we see that in the data? We've, we've explored that. Um, but, but for sure. And some of these high profile, like [00:33:00] lab incidents or things that happen. Where as opposed to, it used to kind of be pointing fingers, like what, what really happened, we can go back to the data, Joe can pull up the data and say, Nope, like everything was working fine.
[00:33:14] And, and we see, you know, the user did this or whatever happened, like the system was fine. And we actually have like the flight data recorder to say, Okay. Everything was okay or wasn't it? And we had a, this failure and we know what it is and we can do root cause analysis and go, go address the issue. So I think foundationally that's kind of, you take it for granted after a while to have that type of access, cause you think, Oh, I've got an automation system and it can do trends.
[00:33:42] But even if you have the trends, the lift to go get those data from these automation systems is, is really 10x what you can do with a platform like SkySpark, it's like three clicks and you can get whatever data you want.
[00:33:55] Joseph Fleshman: Well, and you have to have the foresight to set up the appropriate trends [00:34:00] for that particular, like, so an example of like what Jim's saying is we had, we had a laboratory where.
[00:34:05] Um, they were doing really, uh, costly experiments with a particular sensitive piece of equipment and they lost a run because the, the, um, temperature and humidity changed too quickly in the room. Well, what we were able to do is go back in SkySpark and identify exactly when the air handler went down and exactly when their temperature went out of spec.
[00:34:24] And we were able to hand them that trend data so that they can then go, for instance, to the insurance company, right? So they could, they could use that as, as like, here's exactly what happened. It wasn't our fault. It was caused by, you know, this failure of this piece of equipment. And we have, because I would have never thought, Oh, I should trend the relative humidity and temperature of this one particular lab and the status of the air handler.
[00:34:44] Like, that's not the kind of thing that you would have the foresight. I'm sure people run into that where something happens and they're like, Man, I really wish I had that data. And you never know when you're going to need it until it comes up. And so that's what having that flight data recorder is really valuable for us.
[00:34:59] Um, [00:35:00] for either identifying what went wrong or what didn't go wrong.
[00:35:03] Jim Meacham: Yeah. I think another really impactful use case has been measurement verification. The tools that we have. To have real time M& V on these buildings, even post project, um, you know, Joe mentioned still really working on how do we optimize the ongoing kind of commissioning internally to UCI, but it's all there and we have, um, you know, you can very easily go look at like, how was this building performing?
[00:35:32] How should it be performing? And how is it performing? Very easily.
[00:35:37] James Dice: Totally. All right, let's dive into the sort of lessons learned here. Um, can you guys talk about, I think there's probably, I think you guys talked about three big challenges here. Can we talk about what's challenge number one? And then how did you sort of Resolve that.
[00:35:53] Joseph Fleshman: Yeah. So I think challenge number one, we've talked about a little bit is the, is the flat network, um, and not having foresight into [00:36:00] what that network configuration should look like. We charged ahead with BACnet conversions, not knowing that it was going to cause the problems that it did. Um, so working with, working early with your OIT, uh, enterprise networking team to establish what that virtual LAN scheme looks like, setting up the subdomains that are set aside for building automation.
[00:36:18] Um, and then that way, when you go to do your hardware available Or you go to convert the protocol that they're, that those devices are talking, you've already sort of set the stage that you're not going to have a disruptions.
[00:36:31] James Dice: Yeah, yeah, I really focus on that network infrastructure.
[00:36:33] Jim Meacham: And I would say a follow on to that from a lesson learned perspective is when you move to any type of enterprise, particularly enterprise BACnet network.
[00:36:43] It has to be actively managed when you have all of these different projects going on and all of these different systems that are jumping on and talking back net lighting systems, metering systems, HVA system, lab control systems. I mean, we've got all kinds of stuff out there and. [00:37:00] It's happening all the time.
[00:37:01] There's lots of vendors in the system, right? Because now it's truly multi vendor and the network, if someone puts the wrong network number or duplicate, you know, instance IDs or things like that, all of a sudden devices over in this building start going offline that have nothing to do with where that work was going on.
[00:37:20] So we've actually now got a process where Where our team meets with the automation shop every week. We do, um, BACnet captures. We do health scores using visual BACnet, uh, and we're looking proactively looking for is something is, is the health being maintained? Is there anything that's making this more brittle and we're back checking other vendors work so that we don't.
[00:37:44] get to an issue where it's taking two minutes for the network to update or something like that. We've been there and it's been painful. And if you let that go too long, it takes a long time to reel it back and really trying to, you know, stay up to date with how the network [00:38:00] health is looking. Totally. All right, what's challenge number two and how is that resolved?
[00:38:04] Joseph Fleshman: So I think challenge number two, uh, is more related to the SkySpark integration, which would be establishing that, that business to business VPN network connection, working with the Office of Information Technology network security people to make sure because you're not going to accomplish anything if they're not comfortable with what you're asking them to do.
[00:38:23] And so, you know, working, working with them, getting them comfortable, having People who can talk their language, um, come in and that's where, you know, Jim and I and his team can kind of talk that language with the IT people about networking and IP addresses and firewalls and ports and all that sort of stuff.
[00:38:38] Um, and so we were able to gain their confidence and work with them to establish something that they were comfortable with. Um, one of the advantages that that's provided for us is, is we don't just send data up to SkySpark for trending. Um, there actually have been cases where it made sense for us to do...
[00:38:54] What I used to call, when I was a commissioning agent, midnight testing. Um, they were able to send BACnet [00:39:00] commands to a building that, one of those major renovations in a laboratory building that I was talking about. And so what they did, what, what we did was at two in the morning, they sent BACnet commands from SkySpark to turn, uh, half the building into heating, and then into cooling, and then the other half of the building into heating and into cooling, and then they looked at the SkySpark data.
[00:39:20] To identify which zones weren't behaving the way that they were supposed to, whether that was a PID loop tuning, whether that was a hot water coil that wasn't, that was, you know, strainer needed to be flushed or, um, you know, air valves that were hunting, we were able to identify all of that for 100 percent of the building, which is different than like the way that commissioning used to be.
[00:39:39] I was a commissioning agent back in 2011, 2013, you'd walk around with like a radar gun and hit registers to make sure they were reheating, you know, like it was, It was, it's, it's so much more technologically advanced compared to the boots on the ground way of doing things in the past, where we could hit every single zone in the building digitally, um, and it would also flag [00:40:00] things like, hey, your temperature sensor is reading 32 degrees there.
[00:40:02] That's a bad, that's a bad stat. You got to go replace it. Um, all of this was able to be progressing quickly. Automated and then Altura combed through the data and made specific recommendations for the contractor on the project to repair as a punch list item.
[00:40:15] Jim Meacham: One of the interesting, yeah, that's, that's a good point.
[00:40:17] Yeah, the automated testing. It was actually one of the first projects, we did one of the first projects that we did automated testing on was at UCI back in the Must've been 2018. That was 17. 17. Yeah. So a long time ago now, like we've, we've been talking about automated testing for a long time. I guess we started that at Caltech actually, and then brought it to UCI.
[00:40:38] Um, but the contractor, the, the controls contractor, I remember in particular in that case had never been through such a rigorous process and it was shocking. So I think a lesson learned is really that we didn't do a good enough job of. Probably describing to them what the level of expectation was going to be, that we were going to be [00:41:00] doing a hundred percent like testing of everything, every single point in the system.
[00:41:06] And they really weren't prepared for that. And it, so the commissioning process took a long time. We got there because the data don't lie. There's nowhere to hide. The building performs awesome now, but. That was one of the longer tails, I would say, of a functional testing process that we've ever had because just took the bar, raised the bar quite a lot.
[00:41:27] Joseph Fleshman: Well and part of the reason why it ended up being successful was eventually we're showing data and they're like, can you just give us access to your SkySpark system so we can see this stuff ourselves? And then they could go in and they could check because they knew that Altura was going to come behind them and say this is hunting, this isn't working, this isn't working, so they went in and when they were addressing punch list items could actually make sure that things were working the way, looking at the exact same data that my commissioning agent would be looking at to make sure that it was actually fixed so that they didn't get called on the carpet for, you know, failing to do what the punch list item required them to do.
[00:41:58] James Dice: Minimize the back and [00:42:00] forth. Exactly. What's the, what's, yes. What's challenge number three and then the resolution?
[00:42:07] Joseph Fleshman: Um, I think managing multiple vendors. Um, One of the things that we, we actually did a pilot with Niagara with one of the local vendors. And it was originally before we sort of set up what our standards and graphics and, and everything would be.
[00:42:22] And it had their logo all over it. And it would didn't, it was emblazoned with every, you know, every page had their logo all over it. And, and there was no interface with like the automation system, the automation guys of what, what they want to see in the system. Um, and so, you know, we, We realized that like, if we're going to have different vendors in here, we need everything to look the same.
[00:42:42] Um, and we need everything to be, all the point naming has to be consistent. All the graphics have to be consistent. All the programming for the resets have to be consistent. And so that's, that is something that once we established those standards with Altera's support, it's enforcing that, right? [00:43:00] So, you know, one of the things that I tell the new vendor, the vendors when they come in, uh, to do the graphics, it's like, if you don't have a graphic, Tell me, and we'll get you a graphic.
[00:43:09] Don't build your own, because I want it to look exactly the same regardless of who's doing the graphic. And so we, we're just up front with them of like, you're basically taking cookie cutter modules, and you're putting them together, and that's how you're building the graphics. We don't want creativity, we don't want, you know, we don't want your special flair or touch or logo on it.
[00:43:27] We want you to be consistent because... I want to, if I have four different vendors working in the automation system, I want to be able to look at any building and not know which vendor did it, because it should look exactly the same regardless.
[00:43:39] Jim Meacham: That's, that was, we don't need your creativity. Yeah, that's right.
[00:43:43] Um, we, uh, one of my, the early wins, I would say when we knew we were finally like powering out of, you know, the, the, the initial phases of piloting some stuff and we, we built those standards and, and we did, we had three. different [00:44:00] projects that completed around the same time, and we're able to grab those graphics that, you know, all the way down to the zones and everything.
[00:44:08] And you couldn't tell, and we put them side by side and said, this is a win. You know, we, with three different vendors, same outcome for the automation shop, um, and you know, it really, that's, that was one, a good example.
[00:44:24] James Dice: One of the pieces I feel like we haven't talked enough about here is the actual, um, creation and sort of the, uh, adoption of the actual standards and specs.
[00:44:34] Can you guys talk about, I'm sure you got a lot of pushback from the incumbent vendors, like this isn't the right way to do it, nobody does it like this, kind of like, uh, sort of ingrained in the past, right? So can you talk about like that piece and how you sort of overcame this is a new way of doing it and this is the way we're doing it moving forward?
[00:44:56] Jim Meacham: Do you want to start on that? Yeah, I can start on it. That was a key part [00:45:00] of the process early on. Um, it's, it's, it was a couple fold, I would say. Um, one, we did a lot of outreach and engagement of the vendors. We didn't just go off into a silo and say, you know, we're cooking up this special thing and then we're going to like, just knock you over the head with it.
[00:45:20] It, we, we engaged them. They understood why, you know, a lot of the, the, the, the people who are associated with all of the vendor landscape at UCI are. They wanted UCI to succeed. They understood what was why UCI was pushing for a new way. Um, now we of course got pushed back and there were some different levels of technical understanding.
[00:45:48] Of things, you know, in proprietary vendors, we're, of course, trying to protect their, their systems, but eventually everyone got on board. Um, and [00:46:00] you know, those proprietary vendors are still doing work and doing Niagara work and have done Niagara work on the campus. So it's not just about saying like, Oh, we don't like you as a vendor.
[00:46:11] It's no, like now you're just doing it this way. And. That has worked. I mean, it's, it's a very similar vendor landscape. We've got some new ones, which is a big win. Cause we, that's one of the things we were looking for. It was more horsepower, uh, for the campus and more flexibility, but those same vendors are still there doing the work as well.
[00:46:30] Um,
[00:46:32] Joseph Fleshman: Yeah, and I would say that, um, it's been a stepwise process, especially as new technology has come out. So when we did our first building, um, that was in, that was one of those major renovations, um, that started to, like, okay, by default we're going to be missioning in SkySpark. That first project was still field controllers of the MSTP networks, because the BACnet, You know, Ethernet addressable individual terminal units hadn't really come out yet, right?
[00:46:59] So when we went and did our [00:47:00] project in 2018, where we did two additional buildings of similar scope, every one of those devices was a BACnet over Ethernet IP addressable device. What we've tried to do is, is, is move, move the ball as the technology also became available saying, yeah, just because we've done MSTP networks for the last 20 years, we're not doing that anymore.
[00:47:20] We're moving to BACnet. And so now our current specifications that's handed out to every contractor on any substantial sized job or new building construction says. Serial networks are prohibited. Everything has to be BACnet over Ethernet IP. Now, there are certain situations where you have a, you know, a minor renovation in like a couple rooms in an existing building that is on one of the proprietary systems and to migrate the entire 60, 000 square foot building because of a 2, 000 square foot space that you're renovating doesn't make sense and so we, we, what we do on the engineering side, um, the engineering group supporting the project managers is work with them to identify Which specification, the legacy spec or the new spec, is [00:48:00] appropriate for their job, but if they're doing substantial things like air handler replacements or major renovations of multiple floors of a building, we're pushing to have the new specification and then once that goes out to bid, that's the responsibility of the contractor to comply with that specification.
[00:48:13] James Dice: Love it. Okay, let's go. Our last four sort of five minutes or so here. I want you guys to walk us through if you were going driving down the road to the next university and they hadn't started any of this, what would be the playbook that you would walk them through? Um, or that you would want them to follow to sort of minimize these challenges, minimize these lessons learned, but sort of get this implemented as soon as possible.
[00:48:37] Um, and I'll just let you guys go. So, and I I don't think we need to, like, go into fine detail on each step because I feel like we've hit all of them at this point, but, like, what's the playbook, um, in sort of short, manageable statements here?
[00:48:51] Joseph Fleshman: Sure. So, um, one is get your internal stakeholders on board. That includes your automation people.
[00:48:56] That includes your renovation people. That includes your new building [00:49:00] construction people. All of them need to be on board, um, because if you're, if you're, for instance, your new buildings aren't being constructed with your new standards, then you're getting stuff turned over to you that. that is, is behind where you want to be.
[00:49:11] Um, the next one would be, uh, developing those clear standards and specifications, um, that you can put out to bid on, on, on jobs. Um, and then that's obviously working with outside vendors to identify what products might be appropriate for inclusion in that specification. Like, what we tend to do is like a technical specification that has performance requirements with a list of, with a list of vendors plus and or equal, where if someone wants to submit an additional You know, product that meets all of those technical requirements.
[00:49:38] We're happy to take a look at it. We want to bolster the amount of options that we have that comply with those, those open back net requirements. Um, also, uh, working with the information technology people for an it, an enterprise administrator to treat the system like an enterprise application, like, like, like Jim was talking about before, um, especially when you have a multi vendor environment.
[00:49:58] Um, Getting them access, getting them, [00:50:00] you know, to be consistent, um, and not have conflicts with things like instance IDs or IP addresses. Um, Developing a phasing plan for integration and hardware upgrades. So one of the things that I've heard from other universities is that I have a whole bunch of legacy stuff that talks to proprietary application.
[00:50:18] It's obsolete and I can't upgrade it to BACnet unless I rip the hardware out and put new hardware in. That's a substantial capital investment that you need to be aware of when you, when you go into it. Now, there's some vendors that are already BACnet, like natively, and there's some vendors that have legacy hardware that's not.
[00:50:37] Um, and so one of the ways that you could do that, for instance, is I'm going to target the ones that are already natively BACnet for my first phase of integration while I line up funding to replace the hardware that's just a proprietary network to upgrade it to BACnet, right? So you can get started once you establish those first couple things we talked about with the ones that are already natively BACnet because you've, you've prepped the hardware [00:51:00] already.
[00:51:00] It's just, it's natively ready while you sort of line up the funding. I don't know how other universities funding cycles are, but you know, on an annual basis you get state funding. So you, you're like. Planning for which, which ones can I upgrade next so that I can do an energy project on them 12 or 18 months from now.
[00:51:16] Um, so, so developing that phasing plan, uh, knowing what your hardware is and where it needs to get to, um, and then just monitoring your performance along the way and that's what we use the SkySpark platform for.
[00:51:27] Jim Meacham: I would add training. I think that's like continuous engagement. This is it's new. The topology is new.
[00:51:34] So everyone has to be trained. The internal automation shop folks. Um, and it's continuous. It's not like, oh, here's your half day of training. That was we we learned that lesson. It's it needs to be a much longer touch. Um, and and continuous touch. Uh, the vendors need a lot of training. We didn't really Kind of appreciate that.
[00:51:55] I think early on to say, Hey, these vendors also like are [00:52:00] living in their own little dynamic ecosystem, right? That their people change. And so do they really know the standards? So making sure there is that kind of continuous touch of training and oversight and feedback to the vendors that are servicing the network, like this is how it's done here.
[00:52:16] These are, these are the standards. Um, so that's, I think, been another important. Kind of part of the overall playbook.
[00:52:24] James Dice: Jim, can you add just real quick, the pieces when we say standards and specifications, can you, can you talk about the decisions that need to be made for that and, and the pieces of that that need to be figured out for people?
[00:52:36] Jim Meacham: Sure. Yeah. I mean, I, it really starts with the network architecture standard, as you know, Joe was saying about the IT standards. Um, a lot of folks just don't even have one. From our experience, there's a lot of controls drawings out there. There's not even a network architecture that's meaningful. Like what is the network actually going to look like when you're done?
[00:52:58] So having a standard of [00:53:00] what is that going to look like and getting the buy in from. That, that can work. That's step one. And then, uh, obviously Division 23 standards. If your, if your control specification lives in a Division 23, sometimes it's Division 25, um, is where that standard is, is living. So those, those need to be updated.
[00:53:23] And then, You have to close the loop with all the, all the folks that that impacts is what Joe is bringing up, like the stakeholder piece. There's, there's even within UCI, there's multiple organizations that implement that standard. And, and so while it's managed by Joe shop, it gets. Implemented by other folks.
[00:53:46] And, and so there's, there's kind of that continuous touch too. And then there's all the other divisional standards that, that get, um, impacted by this decision around architecture and integration path. [00:54:00] Um, so division 26, for example, electrical for metering and the related things on emergency power, anything else that might be on there, um, any of the specialties, you know, division 15.
[00:54:11] So. Division 27 is the network side, right? So there's a lot of, and then you just have to look at what is everything I expect to be integrated into my system. And then you have to go to every single one of those specifications and the folks that own those specifications and make sure the right language is in there that is going to result in the right topology, which then becomes to free up this kind of.
[00:54:35] Data layer, um, component that, that, that kind of breaks down the, um, the siloing of the systems.
[00:54:43] Joseph Fleshman: Even some of those, some of those things that you don't even think about. So you mentioned 23, 25, 26. What about 11 for cold rooms? Right? You want to pull your cold room data out there. Like, you may not think like, why would I look at environmental rooms?
[00:54:54] But, but we in, on our campus, we implement BACnet controllers [00:55:00] in our... Um, in our cold rooms, so that we can run an Ethernet cable and get that data out to the cloud, um, and, and monitor it, and see it, also see it on our graphics, rather than it sort of being the stand alone that something goes wrong with it, we don't hear about it until the alarm's going off and someone calls into the service desk.
[00:55:15] James Dice: Amazing. One of the things I love about this case study is, is all the different ways in which you guys have knocked down organizational silos through this process, which sounds, you guys are making it sound easy, but I know it hasn't been easy, even just the internal departments at the university. But now you're talking about the construction process and all those silos.
[00:55:35] And then all the people that are helping out on each renovation project and upgrade. Um, so bravo and thanks for telling the story on the podcast. I'm sure people are going to love, love hearing the story.
[00:55:47] Jim Meacham: Great. Thanks for having us.
[00:55:53] Okay, friends. Thank you for listening to this episode. As we continue to grow our global community of change makers, we need your help [00:56:00] for the next couple of months. We're challenging our listeners to share a link to their favorite nexus episode on LinkedIn with a short post about why you listen. It would really, really help us out.
[00:56:10] Make sure to tag us in the post so we can see it. Have a good one.
ā
Head over to Nexus Connect and see whatās new in the community. Donāt forget to check out the latest member-only events.
Go to Nexus ConnectJoin Nexus Pro and get full access including invite-only member gatherings, access to the community chatroom Nexus Connect, networking opportunities, and deep dive essays.
Sign Up
Nexus Labs exists to move the smart buildings industry forward, without the fluff.
On Wednesday, we send out the renowned (and free) Nexus Newsletter.