Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

>Inverted indexes are stored on top of Redis strings using the modules DMA method.

What does 'DMA method' mean here?



I'll expand on it on the repo, but the idea is this: In redis modules, you can take a redis string key, allocate as much space as needed, and get Direct Memory Access to that key's data, with zero copy overhead. You can then resize that string and grow it.

So basically you can model data structures on top of this, provided that there is an easy way to convert offsets to pointers, or that your structs are scalar only.

In the case of the inverted index, it uses delta and varint encoding of the index records, so no real mapping to structs was needed, I just traverse the data and deserialize each entry on the fly in a pretty fast manner.

Another example was the use of this for bloom filters, that directly manipulate bits on the key.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: