Yet another Hacker News

One-minute frequency disturbances from synchronization in power systems

36 points croes 1 day ago 10 comments

geerlingguy 1 hour ago | parent

I know my older APC UPSes do weekly self tests, but I guess newer equipment does more frequent self testing? And of course they're all internet-connected meaning clocks are within a few hundred ms of each other.

It also made me think about how many GitHub Actions I have that kick off on Sunday morning at a specific time. I try to vary the time/day for those jobs but I'm guessing there are enough things that kick off at specific time intervals to make an impact at the top of an hour, or at :15, :30, :45...

rmunn 1 hour ago | parent

This is why systemd gives you the RandomizedDelaySec option. (Other cron-like facilities may have similar features, I'm not discussing them here). Set that to N and each time the timer reaches its trigger (say, once per day), its actual start will be up to N seconds after the trigger. E.g. if a once-per-day trigger is at midnight but you have RandomizedDelaySec=300 set, then it will run sometime between 0:00:00 and 0:05:00, with a different actual start time each day.

If enough people used that, with a high enough value of N (say 1800 for jobs that really don't have to start at any particular time), it would greatly spread out the spikes in accidentally-coordinated usage.

schiffern 1 hour ago | parent

The corresponding concept in gears is to avoid meshing two gears unless the number of teeth are co-prime, because this will cause uneven wear patterns. If the tooth counts are coprime, then each tooth will contact every tooth on the other gear the same number of times, effectively "wear leveling" the teeth.

In this instance, one fix is to fire the timer every 59 seconds, so it will naturally drift out of phase with the minute hand of the clock. Or, if work needs to be done exactly every 60 seconds, then WAIT(RAND(60)) seconds for the timer initialization, so if 10,000 units start simultaneously they don't try all at once.

This is why many systems will do automated updates every 17 hours: 17 is coprime with 24.

Classic "thundering herd" mitigation tactics.

Edit: Tempting to blame IoT, but (since this is news) I don't think anyone knew there was something to fix here. We have the tools however.

HPsquared 36 minutes ago | parent

Coprime meaning "no common factors".

vlovich123 29 minutes ago | parent

Small nit: Wait(50+rand(10)) is probably closer to what you want to get about a minute between executions. As stated your proposed execution has an average about 30s execution interval with a variance of about 17s. If you’re talking about a random sleep on a 1 minute interval, this actually results in a possibility of 120s between tasks, also not what you want.

But yeah, as soon as you start using Rand most people make all sorts of mistakes. And if you’re using C Rand, don’t. It has other statistical problems.

jagged-chisel 24 minutes ago | parent

He’s still allowing for exactly 60s between executions. He’s saying stagger the startup by a random amount up to 60s. Then we get what we want every 60s, but they’re all just staggered a random amount.

vlovich123 14 minutes ago | parent

My bad. I misread it. Delaying the start like that does divide the thundering herd peak by 60. The point about c Rand stands.

The other thing to watch out for is you’ve still got 1/60 of the thundering herd at the second level. If the tasks are <1s and you have enough of them (as you probably do at grid scale), then doing it finer grained is warranted like offsetting the timer by a millisecond sleep of 10 * uniform_distribution<int> between 0 and 6000. That then gives you 10ms granularity, distributing the workload by two more orders of magnitude.

mkj 24 minutes ago | parent

59 seconds will still have second alignment. Maybe it needs to be prime milliseconds or microseconds.

bob1029 1 hour ago | parent

I'm not sure I agree with the conclusion that this is merely caused by UPS system control logic. That is an extreme amount of coordination across a lot of makes, models and configurations.

The FFT indicates this is a periodic signal, not a once a day or week thing. It's happening continuously all day every day. I don't know of any UPS systems that perform tests with this kind of regularity.

In any case, I refuse to believe an external sync signal is causing this. The more likely explanation is that the active elements are pulling each other to a common mode due to being physically coupled on the same grid. This is like how clocks on the same beam will synchronize.

https://en.wikipedia.org/wiki/Injection_locking

the8472 52 minutes ago | parent

The external sync here is NTP. Synchronized clocks + timer-scheduled actions → globally synchronized behavior

From the article:

> The origin of these current peaks was independently confirmed by the manufacturer to be periodic self-tests executed by the battery EMS, confirming our interpretation. Similar minute-scale diagnostic routines are also documented for other commercial UPS systems (cf. Supplementary Note 8).

https://www.snaptec.com.au/akkutec-4801-din-user-manual.pdf

> battery circuit testing 1 x per minute for 1 sec

https://adelsystem.com/hubfs/PDF%20DC-UPS/Instruction%20Manu...

> The battery testing is carried out automatically every 60 sec.