vitess

We will be presenting Vitess at the upcoming MySQL conference: http://www.percona.com/live/mysql-conference-2012/sessions/scaling-mysql-databases-web.

If you are planning to go, the following promotional code should give you a 10% discount: YTV-PL12.

Introduction

The main goal of the vitess project is to provide servers and tools to facilitate scaling of MySQL databases for the web. The Project Goals page has more details on this.

Vtocc is the first usable product of vitess. It acts as a front-end to MySQL providing an RPC interface that accepts and transmits SQL commands. It is capable of efficiently multiplexing a large number of incoming connections (10K+) over a small number of db connections at reasonable throughput (~10kqps). It also has an SQL parser which gives the server the ability to understand and intelligently reshape the queries it receives.

Vtocc is already being used in a large scale production environment. It is the core of YouTube's new MySQL serving infrastructure.

Feature List

• A Python DBAPI 2.0 compliant client interface (vt_occ2.py).
• A go database/sql compliant client interface.
• Understands a variety of protocols over HTTP or direct TCP sockets.
• Bind variables for queries, backed by a query cache: To avoid repeated parsing and enable efficient reuse of query plans.
• Connection pooling.
• Transaction management: Ability to limit the number of concurrent transactions and manage deadlines.
• DML annotation: Every DML is rewritten to include a comment field at the end of a query identifying the primary key of the rows it changed.
• Zero downtime restarts.
• Built-in fail-safes
  • Query consolidation: The ability to reuse the results of an in-flight query to any subsequent requests that were received while the query was still executing.
  • Limit the max number of rows that can be returned by a query and fail if limit is exceeded.
  • Transaction killer for transactions that are open for too long.
  • Integrated query killer for queries that take too long to return data.
  • Discard idle backend connections to avoid offline db errors.

Vtocc is still under active development. Here is the list of upcoming features:

• A consistent row cache and the ability to rewrite queries to maximize utilization of the row cache.
• A binlog interpreter that can parse the DML documentation injected by vtocc to provide a stream of updates for rows that have changed.
• Persistent (reserved) connections.

 

Resources

• Installation instructions
• Detailed feature list
• Operational instructions
• vtocc internals