Seems like just an extension of Short-conv in the linear attention methods where we instead carry the EMA of the entire sequence instead of nearest 4 tokens.
It would make sense in my opinion to do the evals based on the reason for needing Short-conv in the architectures that use it in the first place, where it was added to support exact retrieval/needle-in-haystack when we compress the attention state.
It's odd to make this arch extension and not test beyond a 2k context window, or with the needle-in-haystack benchmark.
In my head this should fail because the EMA cannot carry this granular information
royal__ 7 hours ago [-]
How significant is a .98 decrease in perplexity. I'm a researcher in this field and I'm having trouble interpreting the implication of this.
tkslucas 7 hours ago [-]
Interesting approach to scale compute, never thought of using the per layer state like that
It would make sense in my opinion to do the evals based on the reason for needing Short-conv in the architectures that use it in the first place, where it was added to support exact retrieval/needle-in-haystack when we compress the attention state.
It's odd to make this arch extension and not test beyond a 2k context window, or with the needle-in-haystack benchmark.
In my head this should fail because the EMA cannot carry this granular information