MySQL Utilities: The New .frm Reader Utility

Have you ever wondered what was in those .frm files littered throughout your data directory? Better still, have you encountered a situation where your data is either missing (was deleted) or damaged and all you have is the .frm files but don't know the structure of the table? Well, wonder no more!

The MySQL Utilities Team is pleased to announce the newest utility - the .frm reader (mysqlfrm). This utility is designed to read .frm files and produce a facsimile of the CREATE statement for the table or view.

That's Impossible! How Can That Work?

It works by making a copy of the .frm file(s) and launching a new, read-only instance of your existing server. The server need not be running but you are required to provide an open port for the new instance with the --port option.

The utility will launch the cloned server without reading your configuration file (--no-defaults). The utility also makes some minor modifications to the cloned server instance configuration to allow reading of the .frm file without data. Yes, that's right - you don't need your data files to use this utility!

Using a new cloned instance and copying the .frm files means your original server is not altered in any way. The .frm reader also cleansup after itself by removing all temporary files and shutting down the cloned server.

Two Modes for Reading .frm Files

The .frm reader has two modes of operation. The default is intended for use in the normal process of discovering the CREATE statement in a .frm file. There is also a diagnostic mode for cases where the .frm file contains complex table settings or is damaged in some way.

Default Mode

The default mode, as described above, reads most .frm files and produces the CREATE statement for each. In this case, you need provide only the connection to the server via the --server option or the path to the server installation (for a downed server) with the --basedir option. You also need to provide a port with the --port option and, of course, a list of the .frm files you want to read or a list of directories to scan for .frm files.

If you do not want to (or cannot) use an existing server to clone, you can use the --diagnostic mode instead.

Diagnostic Mode

The diagnostic mode reads the .frm files byte-by-byte and makes a best effort to read the data in the file. We say best effort because there are many nuances to the .frm file that have been introduced over the years. Suffice to say that without the servers code to assist, deciphering the data is non-trivial.

We built the diagnostic mode as a feature to make it possible to get something useful from the files in the event the file is damaged or unreadable by the server or has complex table settings that the default mode cannot read or gives an error while reading. Thus, the CREATE statements produced in this mode may not be completely accurate and may be missing some parts. To get the most out of the diagnostic mode, provide a server connection to allow the reading of the character set information. This will improve the accuracy of column definitions.

While this may sound like the diagnostic mode isn't as useful, remember that it is designed to be a tool for diagnosing problems (hence the name) rather than a duplication of the server code. If you think about it in that light, the diagnostic mode is a very important tool that you may need in certain situations where no other tool will work.

While its accuracy is limited today, we plan to improve the feature in the future.

What's the Catch?

If you're thinking this is too good to be true, you're right - there are some limitations. Fortunately, these limitations are, er limited, for the default mode.

The default mode currently cannot handle storage engines marked as PARTITION and PERFORMANCE_SCHEMA. Also, some elements of the table structure are not stored in the .frm file so these are not included in the CREATE statement. Again, fortunately this is a short list that currently includes foreign keys and autoincrement values.

If you find the utility reporting it cannot read a .frm file in the default mode, try it again with the diagnostic mode.

Is That It?

The utility has a few features to help make it more versatile. For example, you can see the statistics for each file (dates, size, etc.) using the --show-stats option and you can substitute a new storage engine to be printed in the CREATE statement with the --new-storage-engine option (applies to all files read for each run of the utility).

There is also a debug mode that prints more information. When used with the diagnostic mode, and you can see the actual values read from the file.

Skip the Hype and Show Me How it Works!

Suppose you find yourself in a situation where your server has gone wonky in such a way as to make your data inaccessible. Suppose that you do have access to your .frm files but no record of the latest changes to the schema. Now suppose you have a backup of the raw data. How do you know if the .frm files in your existing server match those in the backup? Simple: just run the .frm reader on your existing server .frm files then compare the results to your backup.

But wait, what if the data directory is protected (your datadir is protected, isn't it)? Well, the .frm reader provides the --user option to allow you to launch the utility with elevated privileges to read the .frm files but execute the cloned server with a different user account.

Observe the command:

$ sudo env PYTHONPATH=$PYTHONPATH mysqlfrm --server=root:pass@localhost --port=3310 --user=mysql /usr/local/mysql/data/employees
In this command, we clone the server with the mysql user account and tell the .frm reader to read all of the .frm files for the employees database.

Here's the output.

# Source on localhost: ... connected.
# Starting the spawned server on port 3310 ... done.
# Reading .frm files
#
# Reading the departments.frm file.
#
# CREATE statement for /usr/local/mysql/data/employees/departments.frm:
#

CREATE TABLE `employees`.`departments` (
  `dept_no` char(4) NOT NULL,
  `dept_name` varchar(40) NOT NULL,
  PRIMARY KEY (`dept_no`),
  UNIQUE KEY `dept_name` (`dept_name`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1

#
# Reading the dept_emp.frm file.
#
# CREATE statement for /usr/local/mysql/data/employees/dept_emp.frm:
#

CREATE TABLE `employees`.`dept_emp` (
  `emp_no` int(11) NOT NULL,
  `dept_no` char(4) NOT NULL,
  `from_date` date NOT NULL,
  `to_date` date NOT NULL,
  PRIMARY KEY (`emp_no`,`dept_no`),
  KEY `emp_no` (`emp_no`),
  KEY `dept_no` (`dept_no`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1

#
# Reading the dept_manager.frm file.
#
# CREATE statement for /usr/local/mysql/data/employees/dept_manager.frm:
#

CREATE TABLE `employees`.`dept_manager` (
  `dept_no` char(4) NOT NULL,
  `emp_no` int(11) NOT NULL,
  `from_date` date NOT NULL,
  `to_date` date NOT NULL,
  PRIMARY KEY (`emp_no`,`dept_no`),
  KEY `emp_no` (`emp_no`),
  KEY `dept_no` (`dept_no`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1

#
# Reading the employees.frm file.
#
# CREATE statement for /usr/local/mysql/data/employees/employees.frm:
#

CREATE TABLE `employees`.`employees` (
  `emp_no` int(11) NOT NULL,
  `birth_date` date NOT NULL,
  `first_name` varchar(14) NOT NULL,
  `last_name` varchar(16) NOT NULL,
  `gender` enum('M','F') NOT NULL,
  `hire_date` date NOT NULL,
  PRIMARY KEY (`emp_no`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1

#
# Reading the salaries.frm file.
#
# CREATE statement for /usr/local/mysql/data/employees/salaries.frm:
#

CREATE TABLE `employees`.`salaries` (
  `emp_no` int(11) NOT NULL,
  `salary` int(11) NOT NULL,
  `from_date` date NOT NULL,
  `to_date` date NOT NULL,
  PRIMARY KEY (`emp_no`,`from_date`),
  KEY `emp_no` (`emp_no`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1

#
# Reading the titles.frm file.
#
# CREATE statement for /usr/local/mysql/data/employees/titles.frm:
#

CREATE TABLE `employees`.`titles` (
  `emp_no` int(11) NOT NULL,
  `title` varchar(50) NOT NULL,
  `from_date` date NOT NULL,
  `to_date` date DEFAULT NULL,
  PRIMARY KEY (`emp_no`,`title`,`from_date`),
  KEY `emp_no` (`emp_no`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1

#
# Reading the view1.frm file.
#
# CREATE statement for /usr/local/mysql/data/employees/view1.frm:
#

CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `employees_temp`.`view1` AS select `employees`.`t1`.`a` AS `a` from `employees`.`t1`

#...done.

Is that cool or what? Notice it also reads .frm files for views. Very cool.

What About Diagnostic Mode?

Let's take one of the tables in the employees database and run it with the diagnostic mode. For fun, lets change the storage engine and show the file statistics. We specify the server (this is optional) so that if there are multiple byte character sets the diagnostic mode will correctly translate the field lengths. For example, if your table has a 3-byte character set and you do not provide a server connection, there is no way to know that a value of 30 read from the file is actually a field length of size 10 thus char(10) would appear char(30) without the server connection.

Observe the command and output:

$ sudo env PYTHONPATH=$PYTHONPATH mysqlfrm --server=root:pass@localhost --diagnostic --show-stats --new-storage-engine=MEMORY /usr/local/mysql/data/employees/titles.frm
# Source on localhost: ... connected.
# CAUTION: The diagnostic mode is a best-effort parse of the .frm file. As such, it may not identify all of the components of the table correctly. This is especially true for damaged files. It will also not read the default values for the columns and the resulting statement may not be syntactically correct.
# Reading .frm file for /usr/local/mysql/data/employees/titles.frm:
# The .frm file is a TABLE.
# CREATE TABLE Statement:

CREATE TABLE `employees`.`titles` (
  `emp_no` int(11) NOT NULL,
  `title` varchar(50) COLLATE `latin1_swedish_ci` NOT NULL,
  `from_date` date NOT NULL,
  `to_date` date DEFAULT NULL,
PRIMARY KEY `PRIMARY` (`emp_no`,`title`,`from_date`),
KEY `emp_no` (`emp_no`)
) ENGINE=MEMORY DEFAULT CHARSET=latin1;

# File Statistics:
#         Last Modified : Thu Jan 26 14:23:14 2012
#         Creation Time : Wed Jan  4 12:15:21 2012
#         Last Accessed : Fri Mar 22 15:18:33 2013
#                  Mode : 33200
#                  Size : 8672

# Table Statistics:
#                Engine : INNODB
#           frm Version : 10
#         MySQL Version : 5.1.50
#      frm File_Version : 5
#               IO_SIZE : 4096
#  Def Partition Engine : None

#...done.

Interesting, eh? If the disclaimer is scary, it is intended to be a reminder that the output may not be as accurate as the default mode. In this case, we see some very minor differences but none of which are showstoppers nor do they demure the usefulness of the output. Lastly, notice we did indeed change the storage engine.

Notice also the file stats show the modification and creation date as well as the original storage engine and version of the server when the .frm file was created.

Ok, I'm Hooked! Where Can I Get It?

The .frm reader utility is part of the new release-1.3.0 Alpha available as a separate download at http://dev.mysql.com/downloads/tools/utilities/. Simply choose the platform repository or source repository and download it.

We welcome your comments and hope that this utility will help expand your diagnosis and recovery toolkit.

Postlude 

Many thanks to Giuseppe Maxia for creating the surprisingly useful employees test database! You can download it from: https://launchpad.net/test-db/

Introducing MySQL Utilities release-1.3.0

The MySQL Utilities Team is pleased to announce a major advancement of MySQL Utilities. It is now available as a separate download!

That's right. If you want to use MySQL Utilities without installing MySQL Workbench, you can do that now.

The Utilities release-1.3.0 has been built for Windows Installer, RPM archive, and .tar/.zip. We have also made downloads for source only if you want to use Utilities to develop your own utilities or install the product in custom location. We plan to add other repositories in the future.

Is that it? Well, not quite. We have also included a new utility - the .frm Reader. See the blog, "New Utility: .frm Reader" for more information.

If you'd like to try out the new download, visit the MySQL Workbench download page:

http://dev.mysql.com/downloads/tools/utilities/

Note: this is an alpha release and will follow the alpha-beta-GA development path. As such, it is not included in MySQL Workbench. However, it does include all of the improvements from the release-1.2.X tree.

For more information about switchover, failover, replication and MySQL high availability, download the "Guide to Building a Self-healing Replication Topology", which discusses how Global Transaction Identifiers and the MySQL replication utilities can be used for failover and switchover with slave promotion from:

http://www.mysql.com/why-mysql/white-papers/mysql-replication-high-availability/

Introducing MySQL Utilities release-1.2.1

The MySQL Utilities Team is pleased to announce our latest release, version 1.2.1. This release contains many quality improvements and enhancements to the HA and Replication utilities. The following lists some of the most significant improvements.

• Improved transaction gathering algorithm for failover
• Skips slaves that are already caught up
• Ensures all transactions in the relay logs on the slaves are executed first
• External scripts in mysqlfailover and mysqlrpladmin now receive the old and new master information
• Improved demote master handling for switchover
• Improved connection error handling
• Quoting of tables and database names has been improved
• Login-path feature now reads port and socket

The utilities team continues to focus on improving usability, making features easier to use, and improving overall quality of error handling.

You can try the new release by downloading it from LaunchPad at:

https://launchpad.net/mysql-utilities

For more information about switchover, failover, replication and MySQL high availability, download the "Guide to Building a Self-healing Replication Topology", which discusses how Global Transaction Identifiers and the MySQL replication utilities can be used for failover and switchover with slave promotion from:

http://www.mysql.com/why-mysql/white-papers/mysql-replication-high-availability/

Note: this is an interim release. These features will be included in a future release of MySQL Workbench.

New in MySQL Utilities release-1.1.0

New in MySQL Workbench 5.2.44 is the latest release of MySQL Utilities 1.1.0. This release contains a new utility called the MySQL Utilities Users' Console (mysqluc).

The MySQL Utilities Users' Console is designed to make using the utilities easier. While it is not a new utility in the sense it doesn't provide any new functionlity for managing MySQL servers or data, it does provide a unique shell environment with command completion, help for each utility, user defined variables, and type completion for options.

That's right, you no longer have to type out the entire name of the utility. For example, you can type mysqldbe and press the TAB key and it will complete the command as mysqldbexport. Don't remember the name of a database utility you want to use? That's no problem either - just type mysqldb and press TAB twice. The console will list all of the utilities that start with mysqldb.

The same is true for options. Type a few characters of the option name and press TAB to complete it or press TAB again if nothing returns and you will see a list of the options that match the prefix.

If that wasn't powerful enough, the best feature of the console allows you to define variables and reuse them in your commands.

For example, if you are planning to use a number of utilities or run the same utility many times, you can set a variable to store the connection information and just reuse it for each command. Here is how you could do that:

mysqluc> set SERVER=root:root@some.host.far.away:13000
mysqluc> mysqlrplshow --master==$SERVER ...
Note: user defined variables have a lifetime of the console run time.

There are a number of console commands as well. To see a list, enter 'help'.

mysqluc> help
Command                 Description                                       
----------------------  ---------------------------------------------------
help utilities          Display list of all utilities supported.          
help           Display help for a specific utility.              
help | help commands    Show this list.                                   
exit | quit             Exit the console.                                 
set =  Store a variable for recall in commands.          
show options            Display list of options specified by the user on  
                        launch.                                           
show variables          Display list of variables.                        
                 Press ENTER to execute command.                   
                Press ESCAPE to clear the command entry.          
                  Press DOWN to retrieve the previous command.      
                    Press UP to retrieve the next command in history. 
                   Press TAB for type completion of utility, option, 
                        or variable names.                                
              Press TAB twice for list of matching type         
                        completion (context sensitive).                   
If you've been using MySQL Utilities or want to try them out, the users' console makes it easy to get started. Go to http://dev.mysql.com and download the latest version of MySQL Workbench and try out MySQL Utilties. You will find MySQL Utilities listed as a plugin to MySQL Workbench.

MySQL Utilities Frequently Asked Questions

Momentum for MySQL Utilities continues to build.  I hosted a webinar recently about MySQL Utilities (available on-demand from the link below), which generated a lot of interest and some good questions.

http://event.on24.com/eventRegistration/EventLobbyServlet?target=lobby.jsp&eventid=448952&sessionid=1&key=7E741ED049DFBF49D10C90A2B62E410F&eventuserid=63530507

With so many questions and ideas coming in I decided to create a blog of FAQs. I plan to add these to the MySQL documentation as well.  Keep your ideas and questions coming!

I hope you find these questions enlightening. I have grouped them for easier reading. You can find the MySQL Utilities documentation using the link below. 

http://dev.mysql.com/doc/workbench/en/mysql-utilities.html

General

Are these utilities present in the community version of MySQL?

They are included in the community edition of the MySQL Workbench product, which can be downloaded from the following link.

http://www.mysql.com/downloads/workbench/

Should/can we run this on live data?

Yes. Naturally, you would want to test some operations before jumping directly into a production environment. For example, you may want to test any database migration, transformation, or similar massive change in a test environment.

Can we use the utilities in a production environment under the GPL license?

Yes. MySQL Utilities is part of MySQL Workbench so all such licenses apply accordingly.

Storage Engines

Can the utilities be used with MyISAM or CSV?

Yes. There are no storage engine specific limitations in using the utilities. There are some features written specifically for InnoDB so those may not apply but in general no utility is storage engine specific. For example, the mysqldiskusage utility shows exact sizes for MyISAM and InnoDB files but uses estimated sizes for any other storage engine based on number of rows and row size.

Platforms

Can I use MySQL Utilities on Linux?

Yes. MySQL Utilities runs on all platforms supported by MySQL Workbench.

Can the utilities be used on Windows?

Yes!

Do the utilities work both for window-based and linux-based servers?

Yes! They work for any server hosting MySQL.

Installation

Do we have to install the utilities with rpm or can we use the tar ball extract and run from there?

MySQL Utilities is installed as part of MySQL Workbench. You can download and install Workbench using several platform-specific installers.

You can also branch and download MySQL Utilities from Launchpad. You can also build and install it from the source code you’ve downloaded using typical Python install steps (python ./setup.py install).

https://launchpad.net/mysql-utilities

What's the link to download these utilities?

MySQL Utilities is part of MySQL Workbench. You can download MySQL Workbench from the following link.

http://www.mysql.com/downloads/workbench/

Locking

Do the utilities lock tables while running?

Yes, but only for situations that require locks. The mysqldbexport utility also allows you to specify what type of lock to use:

no-locks = do not use any table locks

lock-all = use table locks but no transaction and no consistent read

snaphot (default) = consistent read using a single transaction.

Are there any utilities that can show DB locks (like which query is blocking which one)?

No, not currently but that is an excellent suggestion!

mysqldbcompare

How fast is mysqldbcompare? Say a table with 10 million rows?

It is difficult to predict a precise estimate of run time based on number of rows. However, it is generally such that the more rows there are the longer the utility will run. The mysqldbcompare utility is used to produce a difference of two databases. It creates a difference between objects of the same name for either object definitions, data, or both. When comparing object definitions, the performance is very fast because there isn’t a lot of processing involved. When comparing data, the utility uses an algorithm to create checksums for each row in the table. During this phase, the tables are locked. Once that stage is done, the tables are unlocked and the algorithm begins to compact the checksums into chunks, which are later compared between the servers. If the checksums differ, the chunks are expanded and the differences calculated. Thus, for tables containing millions of rows the utility will take some time to complete. The best time to run this utility is during low usage periods such as times reserved for upgrades, backups, and similar operations.

How will running mysqldbcompare effect a production database?

If generating a difference for data, the utility will lock the tables long enough to calculate a checksum for each row. Depending on the number of rows this could be for a long time and in those cases you should run mysqldbcompare during low usage periods. The utility will use a consistent read to lock InnoDB tables but will issue table locks for non-InnoDB tables.

Will mysqldbcompare cause table locking on MyISAM table?

Yes. A table lock is issued during checksum creation.

What is the load going to be on the servers when mysqldbcompare runs?

The load on the server itself is minimal. There is moderate CPU usage during checksum creation but nothing that should cause a problem. The longest period of activity is when the table scans are executed for creating a checksum for each row.

mysqldbexport

Is mysqldbexport similar to mysqldump?

Yes, the mysqldbexport is designed to export data in a row-by-row or logical fashion. However, you can export data in CSV, TAB, Vertical formats as well as SQL statements using CREATE TABLE, INSERT, etc. making mysqldbexport more versatile than mysqldump. You would use mysqldbexport in situations where you need special machine or human readable output for operations like transforming the data or examining the structures in more detail – especially if you need a format other than SQL statements.

Replication Utilities

Are the high availability features available for version 5.5 or 5.6?

The general replication utilities such as mysqlrplcheck and mysqlreplicate will work with servers version 5.0 and later. The newest high availability feature, failover, in mysqlrpladmin and mysqlfailover work only for servers that support global transaction identifiers (GTIDs) which were added in version 5.6.5.

You can discover more about GTIDs from the following blog by Luis Soares.

http://d2-systems.blogspot.co.uk/2012/04/global-transaction-identifiers-are-in.html

Where do I get more info about mysqlrpladmin?

The online MySQL Workbench Manual has information about each utility. You can also use the --help option to show all options and their descriptions.

http://dev.mysql.com/doc/workbench/en/mysql-utilities.html

How can I use the utilities to test replication on a single host?

You can use mysqlserverclone to clone an existing, running instance of MySQL or clone from an installation (basedir), then mysqlreplicate to create the replication topology.

Is the replication failover feature only for version 5.6?

Yes. It requires support for global transaction identifiers, which were added in version 5.6.5. A developer milestone release of 5.6 is available for download.

http://dev.mysql.com/downloads/mysql/#downloads

(Select the Development Releases tab)

What features of mysqlrpladmin will work on version 5.5?

All of the features except slave election and failover.

Do you need to create a replication user on the slave site other than the master?

The mysqlreplicate utility provides an option to use a specific user on the master for replication or it will create a user by default. You can also request that a new user be created during the operation.

Compare and Synchronize Databases with MySQL Utilities

The mysqldiff and mysqldbcompare utilities were designed to produce a difference report for objects and in the case of mysqldbcompare the data. Thus, you can compare two databases and produce a report of the differences in both object definitions and data rows. 

While that may be very useful, would it not be much more useful to have the ability to produce SQL commands to transform databases? Wait no longer! The latest release of MySQL Utilities has added the ability to generate SQL transformation statements by both the mysqldiff and mysqldbcompare utilities. 

To generate SQL transformations in either utility, simply use the --sql option to tell the utility to produce the statements.

Object Transformations with mysqldiff

If you would like to compare the schema of two databases (the objects and their definitions), mysqldiff can do that for you and produce a difference report in a number of formats including CSV, TAB, GRID, and Vertical (like the mysql client’s \G option).

However, its greatest feature is the ability to generate transformation statements to alter the objects so that they conform. Best of all, mysqldiff works on all object types including the ability to recognize renames so you can get a true transformation path for all objects. For even greater flexibility, you can generate the difference in both directions. This means you can generate transformations for db1-to-db2 as well as db2-to-db1 in the same pass. Cool.

The following shows an example of running mysqldiff on two servers where some of the objects have diverged. It also shows how you can generate the reverse transformation statements.


$ mysqldiff --server1=root@localhost --server2=root@otherhost \
--changes-for=server1 --show-reverse util_test:util_test \
--force --difftype=SQL
# server1 on localhost: ... connected.
# server2 on localhost: ... connected.
# WARNING: Objects in server1.util_test but not in server2.util_test:
# EVENT: e1
# Comparing util_test to util_test [PASS]
# Comparing util_test.f1 to util_test.f1 [PASS]
# Comparing util_test.p1 to util_test.p1 [PASS]
# Comparing util_test.t1 to util_test.t1 [PASS]
# Comparing util_test.t2 to util_test.t2 [PASS]
# Comparing util_test.t3 to util_test.t3 [FAIL]
# Transformation for --changes-for=server1:
#
ALTER TABLE util_test.t3
DROP COLUMN b,
ADD COLUMN d char(30) NULL AFTER a
ENGINE=MyISAM;
#
# Transformation for reverse changes (--changes-for=server2):
#
# ALTER TABLE util_test.t3
# DROP COLUMN d,
# ADD COLUMN b char(30) NULL AFTER a,
# ENGINE=InnoDB;
#
# Comparing util_test.trg to util_test.trg [FAIL]
# Transformation for --changes-for=server1:
#
DROP TRIGGER IF EXISTS `util_test`.`trg`;
CREATE DEFINER=root@localhost TRIGGER util_test.trg BEFORE UPDATE ON util_test.t1
FOR EACH ROW INSERT INTO util_test.t1 VALUES('Wax on, wax off');
#
# Transformation for reverse changes (--changes-for=server2):
#
# DROP TRIGGER IF EXISTS `util_test`.`trg`;
# CREATE DEFINER=root@localhost TRIGGER util_test.trg AFTER INSERT ON util_test.t1
# FOR EACH ROW INSERT INTO util_test.t2 VALUES('Test objects count');
#
# Comparing util_test.v1 to util_test.v1 [FAIL]
# Transformation for --changes-for=server1:
#
ALTER VIEW util_test.v1 AS
select `util_test`.`t2`.`a` AS `a` from `util_test`.`t2`;
#
# Transformation for reverse changes (--changes-for=server2):
#
# ALTER VIEW util_test.v1 AS
# select `util_test`.`t1`.`a` AS `a` from `util_test`.`t1`;
#
Compare failed. One or more differences found.

Generating Data Transformation with mysqldbcompare

The mysqldbcompare utility provides all of the object difference functionality included in mysqldiff along with the ability to generate transformation SQL statements for data. This means you can make sure your test or development databases are similar to your production databases or perhaps even your offline, read only databases match your online databases. Like mysqldiff, you can also get the reverse transformations at the same time. Very cool, eh?

The following shows an example of running mysqldbcompare to generate differences in data.

$ mysqldbcompare --server1=root@localhost --server2=root@otherhost \

inventory:inventory -a --difftype=sql --changes-for=server1 \

--show-reverse

# server1 on localhost: ... connected.

# server2 on localhost: ... connected.

# Checking databases inventory on server1 and inventory on server2

#

# WARNING: Objects in server1.inventory but not in server2.inventory:

# VIEW: finishing_up

# VIEW: cleaning

#

[...]

# TABLE supplier pass FAIL FAIL

#

# Row counts are not the same among inventory.supplier and inventory.supplier.

#

# Transformation for --changes-for=server1:

#

# Data differences found among rows:

UPDATE inventory.supplier SET name = 'Wesayso Corporation' WHERE code = '2';

INSERT INTO inventory.supplier (code, name) VALUES('3', 'Never Enough Inc.');

#

# Transformation for reverse changes (--changes-for=server2):

#

# # Data differences found among rows:

# UPDATE inventory.supplier SET name = 'Never Enough Inc.' WHERE code = '2';

# DELETE FROM inventory.supplier WHERE code = '3';

#

# Database consistency check failed.

#

# ...done

MySQL Utilities and Global Transaction Identifiers

The new MySQL 5.6 Development Milestone Release (DMR) includes many new enhancements. One of the most impressive is the use of Global Transaction Identifiers (GTIDs) for replication. With GTIDs enabled, administrators no longer need to keep track of binary log files and positions. In a nutshell, GTIDs simplify the setup and maintenance of replication.

MySQL Utilities has taken this a step further by providing two new utilities that automate two of the most complex replication administration tasks - switchover and failover. Switchover is changing the role of an active, healthy master to one of its slaves whereas failover is the act of promoting a candidate slave to become the new master. Clearly, switchover is an elective operation and failover is performed when there are issues with the master.

The GTID utilities are included in release-1.0.5 of MySQL Utilities. They are included as a plugin for MySQL Workbench or via source download from launchpad (see below). The new GTID utilities are included in Workbench version 5.2.39.

Automatic Failover Utility
The most impressive utility is mysqlfailover, It is an interactive tool used to report replication health, report GTIDs in use, and perform automatic failover. Yes, that’s right - you can setup mysqlfailover to automatically failover to one of a specific set of slaves whenever the master goes offline. It is designed to work with the MySQL 5.6.5 and later versions of the server.



As you can see in the screenshot, a list is presented including the host, port, role, state, and replication health for each server in the topology. The utility connects to a single master and its slaves. When used in multiple tier environments, users can run an instance for each master. The utility provides the ability to run a failover check and report health at specific intervals in seconds from five seconds and up

To start the utility, users can specify a list of slaves or provide a default user and password to be used in discovering the slaves connected to the master. Discovery of slaves requires the slaves to report the correct host and port when connecting to the master. Along with the list of slaves, the user can specify a list of servers to be used as candidates for selecting a new master when a failover event is detected.

Finally, the user can control how failover occurs with the failover mode. The auto mode tells the utility to failover to the list of candidates first and if none are viable, search the list of slaves for a candidate. The elect mode limits election to the candidate slave list and if none are viable, failover does not occur. The fail mode tells the utility to not perform failover and instead stop execution.

Along with these options are four extension points permitting users to interact with the utility during failover. These extension points permit users to specify a script to run at each of the following events.

• exec-fail-check - execute a script to determine if failover is needed. This replaces the default downed master detection and allows users to perform application-specific detection for failover.
• exec-before - execute a script before failover is performed. This can be used to tell the application to cease write attempts while a new master is setup.
• exec-after - execute a script immediately after failover to a new master. This permits users to inform the application that the new master is ready.
• exec-post-fail - execute a script after failover is complete and all slaves have been attached to the new master. This can be used to inform applications that use read-level scale out that it is safe to resume reads from the slaves.

The combination of options to control failover, the option to perform automatic failover, and the ability to inform applications of the failover event are powerful features that enable unattended automatic failover for critical replication-based applications.

Replication Administration Utility
The other utility, mysqlrpladmin, is used to perform switchover and failover operations and more on-demand permitting administrators to execute these tasks with a single command. You can use the command to perform one of the following commands.

• elect - This command is available to only those servers supporting global transaction identifiers (GTIDs), perform best slave election and report best slave to use in the event a switchover or failover is required. Best slave election is simply the first slave to meet the prerequisites.
• failover - This command is available to only those servers supporting GTIDs. Conduct failover to the best slave. The command will test each candidate slave listed for the prerequisites. Once a candidate slave is elected, it is made a slave of each of the other slaves thereby collecting any transactions executed on other slaves but not the candidate. In this way, the candidate becomes the most up-to-date slave.
• gtid - This command is available to only those servers supporting GTIDs. It displays the contents of the GTID variables used to report GTIDs in replication. The command also displays universally unique identifiers (UUIDs) for all servers.
• health - Display the replication health of the topology.
• reset - Execute the STOP SLAVE and RESET SLAVE commands on all slaves.
• start - Execute the START SLAVE command on all slaves.
• stop - Execute the STOP SLAVE command on all slaves.
• switchover - Perform slave promotion to a specified candidate slave as designated by the --new-master option. This command is available for both gtid-enabled servers and non-gtid-enabled scenarios.

These two utilities raise the bar for replication ease of use making the administrator’s job easier.

How Can I Get MySQL Utilities?
You can check out these new utilities and the entire suite of utilities by either downloading the source code from launchpad or by downloading and installing MySQL Workbench.

You can download MySQL Workbench from:

http://www.mysql.com/downloads/workbench/

You can also download the latest development source code tree for the MySQL Workbench Utilities from:

https://launchpad.net/mysql-utilities

To learn more about all of the great new replication features in MySQL 5.6, check out the developer zone article at:

http://dev.mysql.com/tech-resources/articles/mysql-5.6-replication.html