Posts tagged compression
How I made a gzip encoder faster than memcpy
In the compression world it’s usual to compare the time spent compressing and decompressing data with the time difference in transmitting the compressed or uncompressed data over a given network. In this experiment I managed to make the compression faster than the bandwidth to RAM. Sort of. Under special circumstances and with no apologies for the egregious clickbait headline.
Poisoning badly-behaved AI crawlers
Regardless of how you feel about generative AI stealing your job or filling creative spaces with slop or whatever, one thing I hope is less debatable is that their crawlers should still obey the rules of the road.
Entropy coding as a text generator
The compression gains of entropy coding revolves around making likely symbols short and unlikely symbols long. In principle you have some pre-arranged model of the string you want to compress and decompress, and you use that model to tell you the likelihoods that different symbols will turn up next and then you distribute the coding space accordingly, so it takes very few bits to land in the likeliest buckets and many more bits to force the outcome to land in an improbable bucket.
Automatic wiki
A project I’ve wanted to do for a long while now, is a dynamically generated wiki. Use generative AI to produce an encyclopedic-looking page of text with a topic deduced from the URL and the random seed determined by a hash of the URL.