Alternative Backends (“tusks”)

In addition to Redis, I’ve been experimenting with adding support for alternative redis-like backends. These alternative backends are referred to as tusks, and currently Walrus supports the following:

  • RLite, a self-contained and serverless Redis-compatible database engine. Use rlite if you want all the features of Redis, without the separate server process..

  • Vedis, an embeddable data-store written in C with over 70 commands similar in concept to Redis. Vedis is built on a fast key/value store and supports writing custom commands in Python. Use vedis if you are OK working with a smaller subset of commands out-of-the-box or are interested in writing your own commands.

  • ledisdb, Redis-like database written in Golang. Supports almost all the Redis commands. Requires ledis-py.

rlite

rlite is an embedded Redis-compatible database.

According to the project’s README,

rlite is to Redis what SQLite is to Postgres.

The project’s features are:

  • Supports virtually every Redis command.

  • Self-contained embedded data-store.

  • Serverless / zero-configuration.

  • Transactions.

  • Databases can be in-memory or stored in a single file on-disk.

Use-cases for rlite:

  • Mobile environments, where it is not practical to run a database server.

  • Development or testing environments. Database fixtures can be distributed as a simple binary file.

  • Slave of Redis for additional durability.

  • Application file format, alternative to a proprietary format or SQLite.

Python bindings

rlite-py allows rlite to be embedded in your Python apps. To install rlite-py, you can use pip:

$ pip install hirlite

Using with Walrus

To use rlite instead of Redis in your walrus application, simply use the WalrusLite in place of the usual Walrus object:

from walrus.tusks.rlite import WalrusLite

walrus = WalrusLite('/path/to/database.db')

WalrusLite can also be used as an in-memory database by omitting a path to a database file when instantiating, or by passing the special string ':memory:':

from walrus.tusks.rlite import WalrusLite

walrus_mem_db = WalrusLite(':memory:')

Vedis

Vedis is an embedded Redis-like database with over 70 commands. Vedis, like rlite, does not have a separate server process. And like rlite, Vedis supports both file-backed databases and transient in-memory databases.

According to the project’s README,

Vedis is a self-contained C library without dependency. It requires very minimal support from external libraries or from the operating system. This makes it well suited for use in embedded devices that lack the support infrastructure of a desktop computer. This also makes Vedis appropriate for use within applications that need to run without modification on a wide variety of computers of varying configurations.

The project’s features are:

  • Serverless / zero-configuration.

  • Transactional (ACID) datastore.

  • Databases can be in-memory or stored in a single file on-disk.

  • Over 70 commands covering many Redis features.

  • Cross-platform file format.

  • Includes fast low-level key/value store.

  • Thread-safe and fully re-entrant.

  • Support for Terabyte-sized databases.

  • Python bindings allow you to write your own Vedis commands in Python.

Use-cases for Vedis:

  • Mobile environments, where it is not practical to run a database server.

  • Development or testing environments. Database fixtures can be distributed as a simple binary file.

  • Application file format, alternative to a proprietary format or SQLite.

  • Extremely large databases that do not fit in RAM.

  • Embedded platforms with limited resources.

Note

Unlike rlite, which supports virtually all the Redis commands, Vedis supports a more limited subset. Notably lacking are sorted-set operations and many of the list operations. Hashes, Sets and key/value operations are very well supported, though.

Warning

The authors of Vedis have indicated that they are not actively working on new features for Vedis right now.

Python bindings

vedis-python allows Vedis to be embedded in your Python apps. To install vedis-python, you can use pip:

$ pip install vedis

Using with Walrus

To use Vedis instead of Redis in your walrus application, simply use the WalrusVedis in place of the usual Walrus object:

from walrus.tusks.vedisdb import WalrusVedis

walrus = WalrusVedis('/path/to/database.db')

WalrusVedis can also be used as an in-memory database by omitting a path to a database file when instantiating, or by passing the special string ':memory:':

from walrus.tusks.vedisdb import WalrusVedis

walrus_mem_db = WalrusVedis(':memory:')

Writing a custom command

One of the neat features of Vedis is the ease with which you can write your own commands. Here are a couple examples:

from walrus.tusks.vedisdb import WalrusVedis

db = WalrusVedis()  # Create an in-memory database.

@db.command('SUNION')  # Vedis supports SDIFF and SINTER, but not SUNION.
def sunion(context, key1, key2):
    return list(db.smembers(key1) | db.smembers(key2))

@db.command('KTITLE')  # Access the low-level key/value store via the context.
def ktitle(context, source, dest_key):
    source_val = context[source]
    if source_val:
        context[dest_key] = source_val.title()
        return True
    return False

We can use these commands like so:

>>> s1 = db.Set('s1')
>>> s1.add(*range(3))
3
>>> s2.add(*range(1, 5))
4
>>> db.SUNION('s1', 's2')
['1', '0', '3', '2', '4']

>>> db['user.1.username'] = 'charles'
>>> db.KTITLE('user.1.username', 'user.1.display_name')
1
>>> print(db['user.1.display_name'])
Charles

Ledis

ledis is a Redis-like database written in Golang.

The project’s features are:

  • Supports virtually every Redis command.

  • Supports multiple backends, including LevelDB, RocksDB, LMDB, BoltDB and in-memory databases.

  • Data storage is not limited by RAM, since the databases are disk-based.

  • Transactions.

  • Supports the Redis protocol for communication, so most Redis clients work with Ledis.

  • Written in golang, easy to deploy.

Use-cases for ledisdb:

  • Store data-sets that exceed RAM.

  • Use with LevelDB, RocksDB, etc.

Python bindings

ledis-py allows you to connect to ledisdb. To install ledis-py, you can use pip:

$ pip install ledis

Using with Walrus

To use ledisdb instead of Redis in your walrus application, simply use the WalrusLedis in place of the usual Walrus object:

from walrus.tusks.ledisdb import WalrusLedis

walrus = WalrusLedis()