YHK - yet another hacker news clone

Virtual Oxide Labs: Emulating a Full Oxide Rack

148 points tosh 2 days ago 14 comments

swherdman 2 days ago | parent

If you just want to try emulating the virtualization stack i build this about 6-9 months ago https://github.com/swherdman/whoah-cli

Steps to build if you prefer it manually: https://swherdman.com/blog/We-Have-Oxide-at-Home-P1/ https://swherdman.com/blog/We-Have-Oxide-at-Home-P2/

stevekaram 2 days ago | parent

Hey that's really interesting, thanks for the share! It looks like this is using the standalone Omicron build, but on a Proxmox VM, right? Great stuff for running the CP in easy mode!

aliasxneo 2 days ago | parent

Just out of curiosity, did you do this for the kick of it or did you find the stack to actually be nice to use? Not that it takes much to convince me to nuke my homelab for the 10th time, but given the nature of Oxide, I'd assume they are running pretty reliable software (very willing to be wrong here).

stevekaram 2 days ago | parent

I don't want to speak for swherdman, but I don't know that you'd want to nuke anything just to use this. If you want to learn about the Oxide stack, go for it and it'll be a great teacher. But if you're just looking for a better way to run your homelab, I don't know that this would be it.

Of course, I say this as someone running a Nucleo STM32H753ZI board with Hubris as an SP for my home NAS....

qchris 2 days ago | parent

> When I joined Oxide last year, one of the first things I was introduced to was our racklettes; that is, smaller than average racks of four sleds and two switches each. These racklettes can be initialized and run just like a real rack, precisely because they are a real rack. Just not in any form factor it would be economical to sell.

Probably shouldn't have mentioned this publicly ;) Every time an Oxide post comes up, there's people in the comments expressing fervent wishes for just this sort of thing. Now that they know it exists, I imagine the the voices asking to make it available for developers and homelabs will grow even louder.

stevekaram 2 days ago | parent

Ha! Even a racklette might be a smidge too big for the homelab. But you know, given these Voxel 'sleds' are ~6GB RAM each, we could probably get a solid mini-PC farm going now....

zer00eyz 2 days ago | parent

FTA: At home, I run Voxel labs on a Minisforum MS-A2 mini-PC with a 32-thread AMD Ryzen 9 9955HX and 96GB RAM.

I have 128 gb dd4 system sitting at home - Emulation (mostly of clients nonsense) has been massively insightful over the last few years. And as a dev, being able to do all my work "not on my laptop" has been a major blessing.

When prices normalize build a couple bulky machines and a have a stack of small and cheap (I have a bunch of 8th gen i5's I got used). You can do a lot (and give yourself room to grow/upgrade) on a pretty tight budget.

duttish 2 days ago | parent

They mentioned this on the podcast, the problem is that since they're development tools the networking stack is still scaled for a full rack with pretty beefy switches.

Either they need to support a whole second network stack with firmware etc which they don't want to do now, or a racklette would cost way too much and almost nobody would pay the price.

Melatonic 1 day ago | parent

I court see a lot of small businesses wanting one. Or as an addon to their existing Colo setup.

Could also be great for DR sites and smaller remote offices. Power is a big factor in deployments and I can see a smaller little turnkey setup that uses less be very popular

pbnjay 2 days ago | parent

Fun to see the emulation heading up the stack, I wish my employer could afford some real oxide racks!

About 4 months ago, I booted helios + omicron on bare metal on a server in my homelab (following the YouTube here: https://www.youtube.com/watch?v=qwXtS0-r1uQ). Performance wasn't great on my old hardware (32cores, dual Xeon 2.6ghz, 256gb DDR4), and it used like 30% CPU while idle, but it was fun poking around and playing with the software while I could.

duttish 2 days ago | parent

30% is surprising, did you dig into it at all? Sounds like a bunch of active polling or something?

happyPersonR 2 days ago | parent

Like 3/4 of my career was building stuff like this along with the software that ran in it for basically this exact use case. We even had it integrated into our CI. Spent upwards of a decade building things like this.

One thing we realized was that we needed to be able to simulate upgrades like this.

Lead to a lot of complexity but it was worth it.

This testing was some of the most valuable and because we had it automated it really let us build better things faster and find issues and regressions and verify new functionality :)

UltraSane 2 days ago | parent

Oxide racks would be very cool with those 245TB ssds and VAST data style software.

dabbz 2 days ago | parent

Oxide is doing such fun stuff, really wish I had the skill set needed for their job postings.