130 points surprisetalk 3 days ago 28 comments
tinco 1 day ago | parent
orlp 1 day ago | parent
In my mental model `shift` would move the entire window, it would be one push + one pop.
tinco 23 hours ago | parent
orlp 23 hours ago | parent
That's the case for a LIFO queue, but not a FIFO queue. Both commonly use the push/pop nomenclature.
tinco 23 hours ago | parent
dataflow 20 hours ago | parent
The terminology I've seen people use for FIFO instead is enqueue() and dequeue(). Or push_front(), pop_back(). Etc.
teiferer 23 hours ago | parent
Maybe use popleft?
armcat 1 day ago | parent
- Amortized O(1) work per window update
- O(1) aggregate queries, assuming a fixed cost merge operation
- O(W) memory for W buckets, with sketch size treated as fixed
- No need to recompute the entire window every second
orlp 1 day ago | parent
- The aggregate only depends on the value currently in the window.
Most problems caused by NaNs/infinities/catastrophic cancellation/etc in sliding windows are covered by this property.
Sesse__ 20 hours ago | parent
MawKKe 23 hours ago | parent
somat 18 hours ago | parent
def example_sum():
def sum_empty():
return 0
def sum_unit(value):
return value
def sum_combine(agg_a, agg_b):
return agg_a + agg_b
def sum_finalize(agg):
return agg
sum_agg = two_stack_agg(sum_empty, sum_unit, sum_combine, sum_finalize)
w_size = 4
entry_values = list(range(1, 11) )
trim_flag = False
i = 0
print(agg_sum.eval(), agg_sum.cum_aggs, agg_sum.values)
agg_sum.push(entry_values[i])
i += 1
while agg_sum.len() > 0:
if i < len(entry_values):
agg_sum.push(entry_values[i])
i += 1
if trim_flag:
agg_sum.pop()
elif agg_sum.len() >= w_size:
trim_flag = True
print(agg_sum.eval(), agg_sum.cum_aggs, agg_sum.values)
result: 0 [] []
3 [] [1, 2]
6 [] [1, 2, 3]
10 [] [1, 2, 3, 4]
14 [5, 9, 12, 14] []
18 [5, 9, 12] [6]
22 [5, 9] [6, 7]
26 [5] [6, 7, 8]
30 [] [6, 7, 8, 9]
34 [10, 19, 27, 34] []
27 [10, 19, 27] []
19 [10, 19] []
10 [10] []
0 [] []
So a four window aggregate you push values and pop it when greater than the window size. I used the length(being the len of values + len cum_agg ) to measure the window size, but you could just count.Most of the clunk was to get the same full fill/drain behavior as the example, probably unneeded on real data. The rest of the clunk is because I tend to write simple clunky python. and I stored the provided agg functions in the agg class.
teiferer 23 hours ago | parent
Is it maybe for repeated finalize calls? To essentially re-use those computations?
orlp 22 hours ago | parent
Note that there isn't a `finalize` call, only an `eval`, which gives the current aggregate over the window. You typically use these sliding window functions in a way where you want to get the current value every time the window moves.
ktimespi 22 hours ago | parent
ktimespi 23 hours ago | parent
mlochbaum 22 hours ago | parent
I don't know who first described this as a streaming version but I like the presentation at https://signalsmith-audio.co.uk/writing/2022/constant-time-p... . It is indeed too obscure considering how useful it is!
aslushnikov 22 hours ago | parent
Our current algorithm is suboptimal, so we had to limit the "lookback": a test failure is marked as a regression iff it had a perfect history for the last 10 observed commits.
While the current approach works reasonably well, we had a few requests to make lookback larger/configurable, which we now might be able to do with this algorithm.
Thank you for sharing!
ncruces 22 hours ago | parent
SQLite uses the "streaming" push/pop/eval API for windowed aggregates.
DuckDB implements "segment tree aggregation" by swapping "combine" for "pop".
This means I can offer an adapter that turns combinable aggregates into invertible aggregates, in amortized O(1).
bradrn 22 hours ago | parent
repiret 21 hours ago | parent
My present Rust implementation can keep a sliding window sum of f32 at 300M/s.
Lichtso 20 hours ago | parent
While the algorithm presented here is sequential, this smells like it could be done in parallel.
In fact the "cum_aggs" stack is a prefix scan [2].
For the sum of values (and by extension also the average), if you have the input prefix scanned (which can be done in parallel) then you can answer any (yes any, dynamic size, not fixed size window) subslice reduction in constant time.
[0]: https://en.wikipedia.org/wiki/Fold_(higher-order_function) [1]: https://en.wikipedia.org/wiki/Convolution [2]: https://en.wikipedia.org/wiki/Prefix_sum
ashkankiani 18 hours ago | parent
teiferer 15 hours ago | parent
ashkankiani 3 hours ago | parent
- I should stop posting late at night when my ability to think is severely impacted
- I'm remembering this is not the first time I've made this mistake before with this particular algorithm, lol. I wonder what it is about it that leaves a hole in my reasoning/imagination.
montefischer 10 hours ago | parent
krackers 5 hours ago | parent
And then unlike the usual monotone queue for sliding max/min, this works for any monoid since you're never comparing two elements.
bmacho 2 hours ago | parent
This gives you C*n 'amortized' runtime, where 'amortized' means that it is C*n for a whole, big dataset independent of W. But there is an additional O(W) work needed to be done after every W-th element when you build up all the backwards sums.
I like it!