fallenpegasus: (Default)
Drizzle. You can integrate it into your architecture.

MySQL / Postgres / Oracle. You have to integrate your architecture to fit them.
fallenpegasus: (Default)
The Drizzle Developers,
hard at work

DSC_0129, originally uploaded by krow.

The Drizzle Developers,
hard at work,
thinking about Burning Man

DSC_0130, originally uploaded by krow.

fallenpegasus: (Default)
While at the MySQL Conference last week, I stopped at the booth for HeidiSQL, and spoke with Bill Culp, who is the developer of jHeidi, about Drizzle.

A few days ago, he emailed me, saying that he had spent the time, and now jHeidi supports Drizzle.

So now we have a multiplatform GUI for monitoring, browsing, and querying a Drizzle database server.
fallenpegasus: (Default)
While sitting in [livejournal.com profile] krow's talk on Drizzle, I was hacking on my project to allow query logging to Gearman from Drizzle, and I found a couple of annoying bugs in GearmanD, and a critical bug in Drizzle. With MontyT's help, I found and fixed the Drizzle bug, and then showed the GearmanD bugs to EricD, who has gone off to fix them.

Maybe there will be an actual running demo of it all at my BOF this evening.
fallenpegasus: (Default)
Here I am again, at a conference away from home, waking up to learn that my employer has been acquired. MySQL to Sun. Sun to Oracle.

This time I have fewer opinions, and more things to think about.

I have many options and opportunties, and fortunately I am in a place where I can decide without rushing.

As for MySQL and Drizzle and Gearman and MemCacheD? They are all open source, they are all GPLed, so they are all remarkably immune to the threats and risks of corporate games. For as long as people use them, there will always be support and services and fixes and features for those projects.

Sun Java and JVM are open source and GPL now, so they also are safe.

For other stuff? I have no idea. I hope for the best.
fallenpegasus: (Default)
Working on Drizzle. A while ago the "log to gearman" module I wrote got incorporated into the main tree. The current task is to finish "get gearman via INFORMATION_SCHEMA", and then Drizzle will be able to push its query stream out into a chain of gearman based map reduce system.

So here is a question for the DBAs big system architects out there. If you had an array of database server instances, generating a compiled query log stream of many thousands of statements per second, and you could map, reduce, and analyze that query stream for "interesting stuff"...

What sorts of "interesting stuff" would you want to look for?

There is going to be a BOF on this at the MySQL Users Conference next week.
fallenpegasus: (Default)
I just got back from the monthly MySQL Meetup. It's been moved from the Elysium to downtown, to the offices of Blue Gecko.

So the food wasn't as good, but it was easier to talk and listen.

The conversation orbited a lot around Drizzle, but also to practical issues of types of sharding, using VIEWs to access denormalized tables, data warehousing solutions, and using snapshots vs online backup tools for backups.

There was also a great deal of discussion about AWS; running databases in EC2 nodes, backing up stuff into S3, building scaling web services, and using CloudFront.

It was fun.

I was "on" most of the time; bright, cheerful, helpful, and outgoing.

Which means, of course, that as soon as I got home, I was exhausted. An introvert am I, still.
fallenpegasus: (Default)
The PrimeBase PBMS storage engine is now running on Drizzle.

It uses blobcontainer plugin hooks that I was asked to add to Drizzle at the OpenSQL camp in Virginia a few months ago.

It's always very rewarding to see something I built enable someone else to do something cool
fallenpegasus: (Default)
Today I spent a couple of hours at Amazon, because I have a friend who works there who invited me, giving some completely unstructured presentations.

The first I call "All these MySQL forks. WTF?!", giving some history and background and direction and impressions on MySQL mainline development, the Percona tree, the OurDelta effort, and of course Drizzle.

The second I call "Why distributed source control rocks!", talking about the workflow of git/bzr/hg compared and contrasted with p4/svn.

It was fun. And I had some ideas for bzr (that can also work for git and hg) that I will mention in a later post.
fallenpegasus: (Default)
A feature of the MySQL server that is used a lot, and yet is a source of much user confusion, code complexity, and multiprocessor lock contention, is logging. Query logging, slow query logging, and the new 5.1 feature, "log to table".

I've removed most all of that stuff from Drizzle (and removed two or three sets of now-no-longer-necessary mutex locks in the process), and replaced it with hooks into a logging plugin subsystem, and have implemented two plugins for it. One logs to a file, and the other logs to syslog.

The output looks almost completely unlike the current MySQL logging. There are no hash-prefixed pseudocomments, for one thing. And there is no distinction between the query log and the slow query log. Queries get logged, and the amount of time each query takes gets logged with it. This subsumes the "micro-slow patch" that is spreading around in the MySQL legacy world.

The current format is pretty "programmer-centric", but concrete advice and patches are welcome.

snprintf(msgbuf, MAX_MSG_LEN,
"thread_id=%ld query_id=%ld"
" t_connect=%lld t_start=%lld t_lock=%lld"
" command=%.*s"
" rows_sent=%ld rows_examined=%u\n"
" db=\"%.*s\" query=\"%.*s\"\n",
(unsigned long) session->thread_id,
(unsigned long) session->query_id,
(unsigned long long)(t_mark - session->connect_utime),
(unsigned long long)(t_mark - session->start_utime),
(unsigned long long)(t_mark - session->utime_after_lock),
(unsigned long) session->sent_row_count,
(uint32_t) session->examined_row_count,
session->db_length, session->db,
session->query_length, session->query);

How to interpret this output, and how to turn on, control, and use logging, will be described in additional posts.
fallenpegasus: (Default)
I've created a Twitter for Drizzle at http://twitter.com/drizzlebugs

Follow it for announcements, bugs reports, and other Drizzle related stuff
fallenpegasus: (Default)
I've been involved with the Drizzle project since very soon after it began, working on it on nights and weekends.

That has just changed. As of today, I'm no longer a MySQL Professional Services consultant, instead I'm part of a new division of Sun

Much of my time is to be spent working on Drizzle, with a focus on plugin interfaces and making it work well in Extremely Large distributed environments.

I will be blogging heavily about what I am doing. How I sort that blogging out between my personal LiveJournal, my (mostly unused) Sun employee blog, and maybe some other blog system, remains TBD.

This is going to be fun.
fallenpegasus: (Default)
[livejournal.com profile] krow just asked me to add another plugin type to Drizzle, for pluggable replication, and so I have, and it's pushing up to Launchpad now to merge with the mainline.

The plugin type is actually called "replicator", to avoid conflicts with the existing source files called "replication.*"

The existing replication system in MySQL 5 is... well. Like the old joke about if you like laws or sausages, you probably shouldn't watch them get made, if you need to think that MySQL 5 replication is rock solid, you probably shouldn't read the source code. Heck, a serious bug was discovered in it at OpenSQLCamp a few weekends ago, just by projecting a random page of code on the wall and having a dozen pairs of eyes read it.
fallenpegasus: (Default)

OpenSQLCamp - hackfest
Originally uploaded by datacharmer
I need a haircut, tho not as short as Jay Pipes, seated next to me.

I think at that moment I was either working on more improvements to the errmsg plugin interface to Drizzle, or else I was adding a new plugin type to enable PBXT's blob streaming protocol.
fallenpegasus: (Default)
Open SQL Camp was a success.

It was hosted by Baron "Xaprb" Schwartz, who I learned is one of the few people I don't have to look down to look in the eye.

There were maybe 100 people over the whole the weekend, including the upper echalon of MySQL and other open source database hackers, as well as technical people from Infobright and Tokutek and PBXT.

There were people who quite literally flew in from the other side of the planet and from Europe.

It was good to see Monty Widenius, and to introduce him to the pleasure that is well made matcha.

Vadim Tkachenko's and Peter Zaitsev's presentation on the Percona patches was interesting and eye opening. The following random roundtable discussion between them, Brian Aker of Drizzle, and Arjen Lentz about the open source future of InnoDB was productive, in that it let to Percona moving their patch set development to Launchpad here: https://code.launchpad.net/percona-patches.

At that meeting, Oracle/Innobase were very notable in their absence.

I got to meet Richard Hipp, the author of SQLite, and then to introduce him to Jim Starkey. Richard's presentation on how much you can and can't trust the operating system, and how to make a database durable in the face of the real world was very illuminating, and more than a little bit scary. I wonder if InnoDB and MyISAM (and PBXT and Maria and Falcon) test as rigiously against the edge conditions of write failure as SQLite does.

Saturday night I was asked to make a custom tree of Drizzle for PBXT, for the blob streaming protocol work, and so I did. The drizzle-blobcontainer patch is now up on my Launchpad account.

Sunday was a hackathon, which I spent plumbing up the Drizzle pluggable error handlers. Almost done, tho hard to debug, since when it doesn't work, Drizzle doesnt output any error messages! :) When it's done, another one or two locks will be removed from the main execution path, plus a whole pile of spagghetti and hardened lava, and Drizzle will be even faster!

Monday morning I caught the train up to New York Penn Station, riding along with Sheeri Cabral of Pythian and Ronald Bradford of 42SQL. Taking the train was cheaper and faster than driving or flying from Charlottesville to NYC.
fallenpegasus: (Default)
I've ported my AWS S3 storage engine to Drizzle.

The source is at bzr branch lp:~fallenpegasus/drizzle/awss3. Pull it and build it like you would the main Drizzle. The engine is built-in, no need to plugin load it.

Example use:

   CREATE TABLE colors
        CONNECTION='awss3 bucket_name aws_id aws_secret';
   SELECT val FROM colors WHERE nam='BlueViolet';

I will try to keep it tracking the main Drizzle dev tree.
fallenpegasus: (Default)
[livejournal.com profile] krow asked for two more plugin types for Drizzle: Scheduler (which allocates and controls threads, and assigns them to sessions), and (drumroll), Parser.

People have been asking for plugin parsing for MySQL for years.

Drizzle is about to get it.

fallenpegasus: (Default)
I just pushed up a new Drizzle branch at lp:~fallenpegasus/drizzle/newplugins

It contains two new plugin types, one for configuration interface, and one for query cache. Those two are not "plumbed in", and are in fact just templates, containing two dummy entry points with two dummy parameters. But they follow the evolving pattern for plugin types.

It also contains fixes and improvements for the logging and errmsg plugins. The logging engine implementation has parameters for filtering for slow queries and for "big queries", both ones that return a lot of rows, and ones that just examine a lot of rows.

It's all also been internationalized.

There is still lots of work to be done, but it's fun to get this foundation stuff going.

I just read that Toru has a branch he's been working on a query cache plugin interface as well, so we need to work together to stitch all our work together.
fallenpegasus: (Default)
The errmsg plugin interface is almost done.

The next plugin interface will be fairly more fraught, but useful. Making the configuration interface as a plugin.

This means that instead of keeping the configuration in the /etc/my.cnf and/or /etc/drizzle.cnf, the configuration variables could be kept in the Gnome registry, or in LDAP, or via lookups in some online configuration database server.

This would make managing and configuring huge arrays of Drizzle servers much much easier.
fallenpegasus: (Default)
MySQL, and thus currently Drizzle, logs stuff to it's error log. "Stuff" being errors, warnings, and info messages. Inside the code, they ultimately get sent to stderr, which is directed to whatever file is specified by the "log-error" option. It's all in the MySQL documentation at http://dev.mysql.com/doc/refman/6.0/en/error-log.html

I'm changing that in Drizzle, replacing the error logging stuff with a plugin interface.

The first implementation will, just like the current builtin system, write to a file, or to stderr. I expect soon after, plugins that send the messages to syslog, as SNMP traps, and to a CSV table, will be written.

Just like the query logging work, getting this going will shink the drizzle core even more, and likely remove still more unneeded locks.


fallenpegasus: (Default)
Mark Atwood

November 2016

202122 23242526


RSS Atom

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Feb. 19th, 2017 11:30 pm
Powered by Dreamwidth Studios