If you have a high selectivity query, your effort should be small. But with interval coding, your effort is always proportional to the number of rows in the entire table. That is not good for high selectivity queries. And interval coding does not even fare all that well for low selectivity queries where I prefer projection and bit-sliced indexes. For one thing, it is far simpler to implement a projection index to interval coding!

As for approximate bitmap(AB) index, I implemented it and improved it into a accurate scheme with a small space and time overhead. But experiments show that the AB scheme is hard to be pratical for it only suits for small query region(narrow row and col selectivity). This technique essentially doesn’t work for bitmap index. The problem is the heavy iterative cost for making (row+col) keys for hashing, which is far more time consuming compared to CPU bitwise operations as WAH exploits.

It is unfortunately easy to underestimate the CPU bottleneck when working on paper. Researchers often write that since we have 4 cores or more per CPU, CPU cycles are cheap. But that is not so simple, is it?

As for approximate bitmap(AB) index, I implemented it and improved it into a accurate scheme with a small space and time overhead. But experiments show that the AB scheme is hard to be pratical for it only suits for small query region(narrow row and col selectivity). This technique essentially doesn’t work for bitmap index. The problem is the heavy iterative cost for making (row+col) keys for hashing, which is far more time consuming compared to CPU bitwise operations as WAH exploits.

According to Wu’s paper, the WAH space complexity is 2N words for most datasets, 4N for the worst case. Note it is 2N words,that is 64N bits. My m*b/2*n is in bits!m and b are both very small, isn’t it? I can’t image m L^(m/2)/2, since L can be a very big number. Chan would not be so foolish to present such a solution.

Chan is far from foolish. That is not my point. Interval coding is a nice idea, but it fails for non-trivial reasons. You only realize it is a bad idea after playing with real-world implementations of the idea.

The formula is something like m L^(1/m)/2 and not m L^(m/2)/2. To make L^(m/2)/2 into a small constant, you need for m to be rather large (say m=5).

Eventually, you will make interval coding to be just as small as a regular BBC, WAH or EWAH bitmap index, but what will happen of your query performance? Remember that every time you use the multicomponent trick, your performance degrades… and it does so faster than you might think! To see why, try to implement it! You will see that it gets nasty, and you end up having rather complicated boolean functions.

So you will need to load many incompressible bitmaps. Hence, for any range query, the amount of data you will load is in Omega(n) because your bitmaps are incompressible. The only way this is ok is if your selectivity is always very low. But then, at that point, why not use a Bit-Sliced or projection indexes?

There is absolutely no way you can get good practical performance, and nice fundamental properties, with incompressible bitmaps. Multicomponent is not a good substitute to compression because it trades off space for speed. You want to get better speed and less storage, both together! RLE compression does exactly this!

BTW, how do you comment on Hakan’s Approximate bitmap index? I have developed this scheme into an 100% accurate one with 0 false positives.
Do you think this is promising?

Hard to tell. Send me a draft and I’ll give you an opinion. But a good sanity test is to implement your scheme and see how fast it is!

I’m guessing you have a scheme that allows random access? Is it random access in constant time? Then what is the constant?

According to Wu’s paper, the WAH space complexity is 2N words for most datasets, 4N for the worst case. Note it is 2N words,that is 64N bits. My m*b/2*n is in bits!m and b are both very small, isn’t it? I can’t image m L^(m/2)/2, since L can be a very big number. Chan would not be so foolish to present such a solution.

BTW, how do you comment on Hakan’s Approximate bitmap index? I have developed this scheme into an 100% accurate one with 0 false positives.
Do you think this is promising?

EWAH is not an important contribution, nevertheless… see this paper:

Sorting improves word-aligned bitmap indexes

See also my slides:

http://www.slideshare.net/lemire/all-about-bitmap-indexes-and-sorting-them

What is hierarchical coding? where can I find it?

R. R. Sinha, M. Winslett, Multi-resolution bitmap indexes for scientific
data, ACM Trans. Database Syst. 32 (3) (2007)

Yes, results of interval encoding and such like will be hard to compress further, but they are not supposed to do that. The n*L bits with L distinct values and n rows are not interval encoding, they are just simple(basic) bitmap index. For interval encoding, with m components, let m = log_b_(L),it will produce m*(b/2)*n bits, far less than n*L bits.

With RLE-based compression (including BBC, WAH, EWAH), compression accelerates query as well as reducing storage. The total storage will be O(n) with a small constant.

Note that Chan and Ionnadis did not invent multicomponent indexing. It goes back to the seventies. What they stated is that if you don’t compress your bitmaps using RLE compression, then one level of multicomponent indexing was ideal. The problem with this analysis is that there is no reason not to compress the bitmap indexes with RLE.

Then, they invented interval coding. The problem is that interval coding precludes compression!

Yes, you can use multicomponents indexes… but you can use multicomponents with anything, not just interval coding. Is your factor m*b/2 small? It is actually m L^(1/m)/2. It is pretty hard to make it into a small constant.

Think about the fact that if the index is ten times the size of the original table, this may be prohibitive. Indexes are not supposed to be much larger than the original data set… Very large indexes tend to be slow in practice due to buffering problems.

But read what Sinha and Winslett have written on this topic. Wu has also written on why interval coding is not a good idea.

Yes, results of interval encoding and such like will be hard to compress further, but they are not supposed to do that. The n*L bits with L distinct values and n rows are not interval encoding, they are just simple(basic) bitmap index. For interval encoding, with m components, let m = log_b_(L),it will produce m*(b/2)*n bits, far less than n*L bits.

Chan’s interval encoding is a bit of a problem because it will generate an incompressible index. If you have L different attribute values in your column, and n rows, you will end up with nL bits that are incompressible. This can be quite a pain. It will be slow to index, it will use a lot of storage, it will be hard to keep things in memory, and so on.

I would not recommend interval coding. Rather, look at hierarchical coding with maybe 2-3 levels.

I quote their abstract:

” In this paper, we present a new compression scheme called Word-Aligned Hybrid (WAH) code that makes com-
pressed bitmap indices efficient even for high cardinality attributes. ”

(WAH is based on BBC which is used by Oracle.)

http://sc.tri-bit.com/outgoing/NeverTrustWikipediaTechnicalArticles.png

Also, your anti-spam mechanism is inappropriately case sensitive.

As I point out in my post, you typically use a b-tree to get the location of the bitmap. You need some sort of b-tree-like data structure because compressed bitmaps have different sizes so you can’t just have them in a fix-length-record flat file.

It is not very exciting to index a field where all values are distinct because, yes, the bitmaps would all be very short. Ah! But you rarely have a single dimension. Bitmap indexes are typically used in a DSS context where you “always” have numerous dimensions.

I’m installing the software today, and don’t have the GNU seq command installed; so the /testequalityqueries.sh failed.

Let me suggest you replace `seq 0 10` with
0 1 2 3 4 5 6 7 8 9 10 in the two places seq is called in the script. This was then successful.