Showing posts with label utilities. Show all posts
Showing posts with label utilities. Show all posts

Friday, August 5, 2016

Announcing: MySQL Utilities release-1.6.4 GA

The MySQL Utilities Team is pleased to announce a new GA release of MySQL Utilities. This release includes a number of improvements for usability, stability, and a few enhancements for better compatibility with MySQL 5.7. A complete list of all improvements can be found in our release notes.

Many Improvements!


This release represents a stable release of the product and is a significant improvement from the 1.5 release. Along with defect patches, we also include the following enhancements.

  • Support for MySQL 5.7 
  • New! Binlog Utilities:
    • mysqlbinlogpurge - purge old binary logs
    • mysqlbinlogrotate - rotate the binary log
    • mysqlbinlogmove - relocate binary log files
  • New! Replication Utilities: 
    • mysqlslavetrx - skip transactions on one or more slaves to solve errant transactions and consistency error
  • New! General Utilities: 
    • mysqlgrants - show grants for users
  • Improved functionality of --exclude option with SQL wildcards
  • Improved packaging in RPM and Windows distributions
  • Improved accuracy of calculated disk usage for mysqldiskusage
  • Improved packaging in RPM and Windows distributions
  • Improved output for mysqldbcompare and mysqldiff
  • Improved SSL support
  • The Replication utilities have undergone a number of improvements for new 5.7 features as well as quality and usability improvements.
  • The documentation has been improved including more descriptions of SSL connections and clarification of replication commands.
  • Finally, this release contains a host of minor improvements for quality and robustness

How Can I Download MySQL Utilities?


You can download MySQL Utilities 1.6.4 GA from the following link using one of the pre-built installation repositories including a source download.

http://dev.mysql.com/downloads/tools/utilities/

Where is the Documentation?


You can find online documentation for MySQL Utilities version 1.6 at:

http://dev.mysql.com/doc/index-utils-fabric.html

Thursday, March 31, 2016

Announcing MySQL Utilities 1.6.3 Beta!

The MySQL Utilities Team is pleased to announce a new beta release of MySQL Utilities. This release includes a number of improvements for usability, stability, and a few enhancements. A complete list of all improvements can be found in our release notes.

New Enhancements!


This release represents a stable release of the product. Along with several defect patches, we also include the following enhancements.

Improved support for MySQL 5.7
Improved functionality of --exclude option with SQL wildcards
Improved packaging in RPM and Windows distributions
Improved accuracy of calculated disk usage for mysqldiskusage
...and a host of minor improvements for quality and robustness


How Can I Download MySQL Utilities?


You can download MySQL Utilities 1.6.3 Beta from the following link using one of the pre-built installation repositories including a source download. Click on "Development Releases" to see the 1.6.3 release downloads.

http://dev.mysql.com/downloads/tools/utilities/

Where is the Documentation?


You can find online documentation for MySQL Utilities version 1.6 at:

http://dev.mysql.com/doc/index-utils-fabric.html

Wednesday, March 4, 2015

Announcing MySQL Utilities release-1.6.1 Alpha!

The MySQL Utilities Team is pleased to announce a new alpha release of MySQL Utilities. This release includes a number of improvements for useabilty, stability, and a few enhancements. A complete list of all improvements can be found in our release notes.

New Enhancements!


This release represents a stable release of the product. Along with several defect patches, we also include the following enhancements.

Improved support for MySQL 5.7 early releases
Improved output for mysqldbcompare
Improved SSL support

New Utilities!


This release also has three new utilities for you to try out:

mysqlslavetrx - skip transactions on one or more slaves to solve errant transactions and consistency errors - docs
mysqlbinlogpurge - purge old binary logs - docs
mysqlbinlogrotate - rotate the binary logs - docs

How Can I Download MySQL Utilities?


You can download MySQL Utilities 1.6.1 Alpha from the following link using one of the pre-built installation repositories including a source download. Click on "Development Releases" to see the 1.6.1 release downloads.

http://dev.mysql.com/downloads/tools/utilities/

Where is the Documentation?


You can find online documentation for MySQL Utilities version 1.6 at:

http://dev.mysql.com/doc/index-utils-fabric.html

Announcing MySQL Utilities release-1.5.4 GA!

The MySQL Utilities Team is pleased to announce the general availability (GA) release of MySQL Utilities. This release includes a number of improvements for useabilty, stability, and a few enhancements. A complete list of all improvements can be found in our release notes.

New Enhancements!

 

This release represents a stable release of the product. Along with several defect patches, we also include the following enhancements.

  • Improved support for MySQL 5.7 early releases
  • Improved output for mysqldbcompare
  • Improved SSL support

How Can I Download MySQL Utilities?


You can download MySQL Utilities 1.5.2 GA from the following link using one of the pre-built installation repositories including a source download.

http://dev.mysql.com/downloads/tools/utilities/

Where is the Documentation?


You can find online documentation for MySQL Utilities version 1.5 at:

http://dev.mysql.com/doc/index-utils-fabric.html

Sunday, September 28, 2014

Announcing MySQL Utilities release-1.5.2!

The MySQL Utilities Team is pleased to announce the general availability (GA) release of MySQL Utilities. This release includes a number of improvements for useabilty, stability, and a few enhancements. A complete list of all improvements can be found in our release_notes.

New Enhancements!

This release represents a stable release of the product. Along with several defect patches, we also include the following enhancements.
  • The mysqlserverclone utility now checks diskspace as a prerequisite to cloning the server.
  • The --force option was removed from the mysqlfailover utility to remove confusion on its purpose.
  • A warning is issued if the mysqld --skip-innodb option is used because this option is ignored in MySQL 5.7 and later.
  • Utilities can correctly parse and display the PROXY privilege for GRANT statements.
 

How Can I Download MySQL Utilities?

You can download MySQL Utilities 1.5.2 GA from the following link using one of the pre-built installation repositories including a source download. http://dev.mysql.com/downloads/tools/utilities/

Where is the Documentation?

You can find online documentation for MySQL Utilities version 1.5 at: http://dev.mysql.com/doc/index-utils-fabric.html

Announcing MySQL Utilities release-1.6.0-alpha

The MySQL Utilities Team is pleased to announce the alpha release of MySQL Utilities. This release represents a new evolution of the product including the following enhancements.
  • The mysqlprocgrep utility now displays the processes and connections killed during a --kill operation. The displayed rows come from the appropriate SHOW PROCESSLIST entries.
  • The mysqlbinlogmove utility was added. It relocates binary log files, and moves files based on their sequence number or modified date.
  • The mysqlgrants utility was added. It displays grants per object, and produces reports by user, user with grants, and GRANT statement.
  • Health reports can now be generated for a list of slaves without a master specified. In this case, "No master specified" appears for connection status for each slave, instead of an error.
 

How Can I Download MySQL Utilities?

You can download MySQL Utilities 1.6.0 Alpha from the following link using one of the pre-built installation repositories including a source download. Click on the Development Releases tab. http://dev.mysql.com/downloads/tools/utilities/

Do you want to learn more about DevOps and MySQL?

Are you attending Oracle Open World? Would you like to know more about Oracle DevOps products for managing your MySQL servers? If so, stop by Tuesday, September 30 at 10:15 in the Hotel Nikko - Bay View and join us for a hands on lab (HOL9234) for MySQL Utilities and Connector/Python.

We will present the basics for getting started using Python to manage your MySQL servers. MySQL Utilities includes utilities for a wide variety of operations from displaying GRANT statements to setting up automatic failover for replication.

Check out the abstract for the HOL here.

Thursday, April 3, 2014

New! MySQL Utilities release-1.4.2-RC

The MySQL Utilities Team is pleased to announce the latest release candidate (RC) release of MySQL Utilities. This release includes a number of improvements for useabilty, stability, and a few enhancements. A complete list of all improvements can be found in our release_notes.

New Utilities!


We have also included two new utilities.
  • The mysqlrplsync utility was added, which checks data consistency between servers in a replicated setup. 
  • The mysqlrplms utility was added, which provides round-robin multi-source replication (a slave server continually cycles through multiple masters in order to store a consolidated data set).

How Can I Download MySQL Utilities?


You can download MySQL Utilities 1.4.2 from the following link using one of the pre-built installation repositories including a source download. Click on the Development Releases tab.

http://dev.mysql.com/downloads/tools/utilities/

Where is the Documentation?


You can find online documentation for MySQL Utilities version 1.4 at:

http://dev.mysql.com/doc/index-gui.html

Thursday, December 19, 2013

New! MySQL Utilities release-1.3.6 GA

The MySQL Utilities Team is pleased to announce the latest GA release of MySQL Utilities. This release includes a number of improvements for usability, stability, and a few enhancements. We have also included a performance upgrade for exporting, importing, and copying databases.

Improvements


The following highlights a few of the more significant improvements.

* mysqldbexport, mysqldbimport, and mysqldbcopy have multiprocessing support that allows for much improved performance
* mysqlfrm can now generate a .frm file with storage engine substitution
* Mac OS X packages added!
* mysqlserverinfo now includes the log files (error, general, slow)
* mysqlprocgrep can now search and kill processes by id
* mysqlmetagrep can now search the body of routines with the new --body option
* all utilities report license type with --version and --help
* all utilities have the new --license option to view the license text
* the mysqluc now reports errors with clearer text and tags the message with the name of the utility that returned the error
* mysqlindexcheck now warns user if there is not enough information to calculate best/worst indexes
* rpm, debian, and msi packages will update/remove old versions automatically when installing a newer version
* the documentation is now a separate reference manual (see link below)

The following spotlight some of the more important enhancements.

Multiprocessing with mysqldbexport, mysqldbimport, and mysqldbcopy


The performance of the mysqldbcopy, mysqldbexport and mysqldbimport utilities has been significantly improved. Moreover, a new --multiprocess option was added to allow concurrent execution making the most of the available CPU resources (the number of CPU cores).

Note: the --thread option in mysqldbcopy was replaced by the --multiprocess option.

Multiprocessing is applied at different levels according to the operating system. The utilities mysqldbcopy and mysqldbexport allow multiprocessing at the table-level for non-Windows systems and at the database-level for Windows system. The mysqldbimport utility allows multiprocessing at the file-level independently from the OS.

Other more specific options were also added for performance reasons for the other utility. A new --output-file option was added to mysqldbexport to specify a file to store the generated output which allows faster output than sending messages to the terminal.

Two additional additional options are now available in mysqldbimport: 1) --autocommit to enable autocommit for each operation because now by default a single commit is performed at the end of importing each file which is much faster, and 2) --max-bulk-insert to adjust the maximum number of inserts in a bulk, following the improved bulk insert support that is now provided.

Create New .frm Files with New Storage Engine


The mysqlfrm utility allows you to use the --new-storage-engine and the new --frmdir option to provide a directory to store the new .frm files. This feature is useful for those who want to recover the CREATE statement from existing .frm files and change the storage engine without having to launch the server. Try it out!

Mac OS X Installer


Yes, we now have a Mac OS X package installer. If you install Utilities with this installer, you will need to either use Connector/Python version 1.1.4 or later (which has a Mac OS X installer too) or use the Connector/Python 1.0.8 or later source code package and install manually.

How Can I Download MySQL Utilities?


You can download MySQL Utilities 1.3.6 from the following link using one of the pre-built installation repositories including a source download.

http://dev.mysql.com/downloads/tools/utilities/

If you are a commercial customer, you can download MySQL Utilities from the following link:

https://edelivery.oracle.com/

MySQL Utilities is also available on Lauchpad as a source download at:

https://code.launchpad.net/mysql-utilities

Where is the Documentation?


You can find online documentation for MySQL Utilities version 1.3 at:

http://dev.mysql.com/doc/index-gui.html

Thursday, August 22, 2013

Announcing MySQL Utilities release-1.3.4 GA

The MySQL Utilities Team is pleased to announce the latest GA release of
MySQL Utilities. This release marks a milestone of concentrated effort to
expand the use of utilities in more diverse installations through improved
robustness, error handling, and quality.

Many Improvements


There are number such enhancements in this release. In this post we will
highlight a few of the more significant improvements.
  • (new utility) MySQL .frm Reader (mysqlfrm) - read .frm files and generate CREATE statements with or without a server connection.
  • (revised) improved documentation including a section on example administrative tasks - see http://dev.mysql.com/doc/workbench/en/mysql-utilities.html
  • MySQL Utilities is packaged for .msi, .rpl, .deb platforms and source .tar/.zip
  • You can run mysqlfailover as a daemon on POSIX systems
  • Improved accuracy redundant index checking algorithm for mysqlindexcheck
  • Improved accuracy for comparing databases with mysqldbcompare
  • The --exclude option for mysqldbcopy and mysqldbexport now accept database patterns
  • Improved quoting of database, table, index names in SQL generation
  • External script return code checking for mysqlfailover, mysqlrpladmin
  • Slave thread state included in verbosity output of mysqlrplshow
  • The mysqldbimport utility can now read raw CSV files with headers
...and that's just a few of the many improvements in this release. While many
of these improvements have been filtering into the 1.2.X release over the last
year, this release marks the first GA in that time frame.

How Can I Download MySQL Utilities?


You can download MySQL Utilities 1.3.4 from
http://dev.mysql.com/downloads/tools/utilities/ using one of the pre-built
installation repositories including a source download. MySQL Utilities is also
available on Lauchpad as a source download at
https://code.launchpad.net/mysql-utilities.

Friday, April 5, 2013

MySQL Utilities: The New .frm Reader Utility

Have you ever wondered what was in those .frm files littered throughout your data directory? Better still, have you encountered a situation where your data is either missing (was deleted) or damaged and all you have is the .frm files but don't know the structure of the table? Well, wonder no more!

The MySQL Utilities Team is pleased to announce the newest utility - the .frm reader (mysqlfrm). This utility is designed to read .frm files and produce a facsimile of the CREATE statement for the table or view.

That's Impossible! How Can That Work?


It works by making a copy of the .frm file(s) and launching a new, read-only instance of your existing server. The server need not be running but you are required to provide an open port for the new instance with the --port option.

The utility will launch the cloned server without reading your configuration file (--no-defaults). The utility also makes some minor modifications to the cloned server instance configuration to allow reading of the .frm file without data. Yes, that's right - you don't need your data files to use this utility!

Using a new cloned instance and copying the .frm files means your original server is not altered in any way. The .frm reader also cleansup after itself by removing all temporary files and shutting down the cloned server.

Two Modes for Reading .frm Files


The .frm reader has two modes of operation. The default is intended for use in the normal process of discovering the CREATE statement in a .frm file. There is also a diagnostic mode for cases where the .frm file contains complex table settings or is damaged in some way.

Default Mode


The default mode, as described above, reads most .frm files and produces the CREATE statement for each. In this case, you need provide only the connection to the server via the --server option or the path to the server installation (for a downed server) with the --basedir option. You also need to provide a port with the --port option and, of course, a list of the .frm files you want to read or a list of directories to scan for .frm files.

If you do not want to (or cannot) use an existing server to clone, you can use the --diagnostic mode instead.

Diagnostic Mode


The diagnostic mode reads the .frm files byte-by-byte and makes a best effort to read the data in the file. We say best effort because there are many nuances to the .frm file that have been introduced over the years. Suffice to say that without the servers code to assist, deciphering the data is non-trivial.

We built the diagnostic mode as a feature to make it possible to get something useful from the files in the event the file is damaged or unreadable by the server or has complex table settings that the default mode cannot read or gives an error while reading. Thus, the CREATE statements produced in this mode may not be completely accurate and may be missing some parts. To get the most out of the diagnostic mode, provide a server connection to allow the reading of the character set information. This will improve the accuracy of column definitions.

While this may sound like the diagnostic mode isn't as useful, remember that it is designed to be a tool for diagnosing problems (hence the name) rather than a duplication of the server code. If you think about it in that light, the diagnostic mode is a very important tool that you may need in certain situations where no other tool will work.

While its accuracy is limited today, we plan to improve the feature in the future.

What's the Catch?


If you're thinking this is too good to be true, you're right - there are some limitations. Fortunately, these limitations are, er limited, for the default mode.

The default mode currently cannot handle storage engines marked as PARTITION and PERFORMANCE_SCHEMA. Also, some elements of the table structure are not stored in the .frm file so these are not included in the CREATE statement. Again, fortunately this is a short list that currently includes foreign keys and autoincrement values.

If you find the utility reporting it cannot read a .frm file in the default mode, try it again with the diagnostic mode.

Is That It?


The utility has a few features to help make it more versatile. For example, you can see the statistics for each file (dates, size, etc.) using the --show-stats option and you can substitute a new storage engine to be printed in the CREATE statement with the --new-storage-engine option (applies to all files read for each run of the utility).

There is also a debug mode that prints more information. When used with the diagnostic mode, and you can see the actual values read from the file.

Skip the Hype and Show Me How it Works!


Suppose you find yourself in a situation where your server has gone wonky in such a way as to make your data inaccessible. Suppose that you do have access to your .frm files but no record of the latest changes to the schema. Now suppose you have a backup of the raw data. How do you know if the .frm files in your existing server match those in the backup? Simple: just run the .frm reader on your existing server .frm files then compare the results to your backup.

But wait, what if the data directory is protected (your datadir is protected, isn't it)? Well, the .frm reader provides the --user option to allow you to launch the utility with elevated privileges to read the .frm files but execute the cloned server with a different user account.

Observe the command:

$ sudo env PYTHONPATH=$PYTHONPATH mysqlfrm --server=root:pass@localhost --port=3310 --user=mysql /usr/local/mysql/data/employees
In this command, we clone the server with the mysql user account and tell the .frm reader to read all of the .frm files for the employees database.

Here's the output.

# Source on localhost: ... connected.
# Starting the spawned server on port 3310 ... done.
# Reading .frm files
#
# Reading the departments.frm file.
#
# CREATE statement for /usr/local/mysql/data/employees/departments.frm:
#

CREATE TABLE `employees`.`departments` (
  `dept_no` char(4) NOT NULL,
  `dept_name` varchar(40) NOT NULL,
  PRIMARY KEY (`dept_no`),
  UNIQUE KEY `dept_name` (`dept_name`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1

#
# Reading the dept_emp.frm file.
#
# CREATE statement for /usr/local/mysql/data/employees/dept_emp.frm:
#

CREATE TABLE `employees`.`dept_emp` (
  `emp_no` int(11) NOT NULL,
  `dept_no` char(4) NOT NULL,
  `from_date` date NOT NULL,
  `to_date` date NOT NULL,
  PRIMARY KEY (`emp_no`,`dept_no`),
  KEY `emp_no` (`emp_no`),
  KEY `dept_no` (`dept_no`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1

#
# Reading the dept_manager.frm file.
#
# CREATE statement for /usr/local/mysql/data/employees/dept_manager.frm:
#

CREATE TABLE `employees`.`dept_manager` (
  `dept_no` char(4) NOT NULL,
  `emp_no` int(11) NOT NULL,
  `from_date` date NOT NULL,
  `to_date` date NOT NULL,
  PRIMARY KEY (`emp_no`,`dept_no`),
  KEY `emp_no` (`emp_no`),
  KEY `dept_no` (`dept_no`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1

#
# Reading the employees.frm file.
#
# CREATE statement for /usr/local/mysql/data/employees/employees.frm:
#

CREATE TABLE `employees`.`employees` (
  `emp_no` int(11) NOT NULL,
  `birth_date` date NOT NULL,
  `first_name` varchar(14) NOT NULL,
  `last_name` varchar(16) NOT NULL,
  `gender` enum('M','F') NOT NULL,
  `hire_date` date NOT NULL,
  PRIMARY KEY (`emp_no`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1

#
# Reading the salaries.frm file.
#
# CREATE statement for /usr/local/mysql/data/employees/salaries.frm:
#

CREATE TABLE `employees`.`salaries` (
  `emp_no` int(11) NOT NULL,
  `salary` int(11) NOT NULL,
  `from_date` date NOT NULL,
  `to_date` date NOT NULL,
  PRIMARY KEY (`emp_no`,`from_date`),
  KEY `emp_no` (`emp_no`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1

#
# Reading the titles.frm file.
#
# CREATE statement for /usr/local/mysql/data/employees/titles.frm:
#

CREATE TABLE `employees`.`titles` (
  `emp_no` int(11) NOT NULL,
  `title` varchar(50) NOT NULL,
  `from_date` date NOT NULL,
  `to_date` date DEFAULT NULL,
  PRIMARY KEY (`emp_no`,`title`,`from_date`),
  KEY `emp_no` (`emp_no`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1

#
# Reading the view1.frm file.
#
# CREATE statement for /usr/local/mysql/data/employees/view1.frm:
#

CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `employees_temp`.`view1` AS select `employees`.`t1`.`a` AS `a` from `employees`.`t1`

#...done.


Is that cool or what? Notice it also reads .frm files for views. Very cool.

What About Diagnostic Mode?


Let's take one of the tables in the employees database and run it with the diagnostic mode. For fun, lets change the storage engine and show the file statistics. We specify the server (this is optional) so that if there are multiple byte character sets the diagnostic mode will correctly translate the field lengths. For example, if your table has a 3-byte character set and you do not provide a server connection, there is no way to know that a value of 30 read from the file is actually a field length of size 10 thus char(10) would appear char(30) without the server connection.

Observe the command and output:

$ sudo env PYTHONPATH=$PYTHONPATH mysqlfrm --server=root:pass@localhost --diagnostic --show-stats --new-storage-engine=MEMORY /usr/local/mysql/data/employees/titles.frm
# Source on localhost: ... connected.
# CAUTION: The diagnostic mode is a best-effort parse of the .frm file. As such, it may not identify all of the components of the table correctly. This is especially true for damaged files. It will also not read the default values for the columns and the resulting statement may not be syntactically correct.
# Reading .frm file for /usr/local/mysql/data/employees/titles.frm:
# The .frm file is a TABLE.
# CREATE TABLE Statement:

CREATE TABLE `employees`.`titles` (
  `emp_no` int(11) NOT NULL,
  `title` varchar(50) COLLATE `latin1_swedish_ci` NOT NULL,
  `from_date` date NOT NULL,
  `to_date` date DEFAULT NULL,
PRIMARY KEY `PRIMARY` (`emp_no`,`title`,`from_date`),
KEY `emp_no` (`emp_no`)
) ENGINE=MEMORY DEFAULT CHARSET=latin1;

# File Statistics:
#         Last Modified : Thu Jan 26 14:23:14 2012
#         Creation Time : Wed Jan  4 12:15:21 2012
#         Last Accessed : Fri Mar 22 15:18:33 2013
#                  Mode : 33200
#                  Size : 8672

# Table Statistics:
#                Engine : INNODB
#           frm Version : 10
#         MySQL Version : 5.1.50
#      frm File_Version : 5
#               IO_SIZE : 4096
#  Def Partition Engine : None

#...done.


Interesting, eh? If the disclaimer is scary, it is intended to be a reminder that the output may not be as accurate as the default mode. In this case, we see some very minor differences but none of which are showstoppers nor do they demure the usefulness of the output. Lastly, notice we did indeed change the storage engine.

Notice also the file stats show the modification and creation date as well as the original storage engine and version of the server when the .frm file was created.

Ok, I'm Hooked! Where Can I Get It?


The .frm reader utility is part of the new release-1.3.0 Alpha available as a separate download at http://dev.mysql.com/downloads/tools/utilities/. Simply choose the platform repository or source repository and download it.

We welcome your comments and hope that this utility will help expand your diagnosis and recovery toolkit.

Postlude 


Many thanks to Giuseppe Maxia for creating the surprisingly useful employees test database! You can download it from: https://launchpad.net/test-db/

Introducing MySQL Utilities release-1.3.0

The MySQL Utilities Team is pleased to announce a major advancement of MySQL Utilities. It is now available as a separate download!

That's right. If you want to use MySQL Utilities without installing MySQL Workbench, you can do that now.

The Utilities release-1.3.0 has been built for Windows Installer, RPM archive, and .tar/.zip. We have also made downloads for source only if you want to use Utilities to develop your own utilities or install the product in custom location. We plan to add other repositories in the future.

Is that it? Well, not quite. We have also included a new utility - the .frm Reader. See the blog, "New Utility: .frm Reader" for more information.

If you'd like to try out the new download, visit the MySQL Workbench download page:

http://dev.mysql.com/downloads/tools/utilities/

Note: this is an alpha release and will follow the alpha-beta-GA development path. As such, it is not included in MySQL Workbench. However, it does include all of the improvements from the release-1.2.X tree.

For more information about switchover, failover, replication and MySQL high availability, download the "Guide to Building a Self-healing Replication Topology", which discusses how Global Transaction Identifiers and the MySQL replication utilities can be used for failover and switchover with slave promotion from:

http://www.mysql.com/why-mysql/white-papers/mysql-replication-high-availability/

Introducing MySQL Utilities release-1.2.1

The MySQL Utilities Team is pleased to announce our latest release, version 1.2.1. This release contains many quality improvements and enhancements to the HA and Replication utilities. The following lists some of the most significant improvements.
  • Improved transaction gathering algorithm for failover
    • Skips slaves that are already caught up
    • Ensures all transactions in the relay logs on the slaves are executed first
  • External scripts in mysqlfailover and mysqlrpladmin now receive the old and new master information
  • Improved demote master handling for switchover
  • Improved connection error handling
  • Quoting of tables and database names has been improved
  • Login-path feature now reads port and socket
The utilities team continues to focus on improving usability, making features easier to use, and improving overall quality of error handling.

You can try the new release by downloading it from LaunchPad at:

https://launchpad.net/mysql-utilities

For more information about switchover, failover, replication and MySQL high availability, download the "Guide to Building a Self-healing Replication Topology", which discusses how Global Transaction Identifiers and the MySQL replication utilities can be used for failover and switchover with slave promotion from:

http://www.mysql.com/why-mysql/white-papers/mysql-replication-high-availability/

Note:
this is an interim release. These features will be included in a future release of MySQL Workbench.

Saturday, September 29, 2012

New in MySQL Utilities release-1.1.0

New in MySQL Workbench 5.2.44 is the latest release of MySQL Utilities 1.1.0. This release contains a new utility called the MySQL Utilities Users' Console (mysqluc).

The MySQL Utilities Users' Console is designed to make using the utilities easier. While it is not a new utility in the sense it doesn't provide any new functionlity for managing MySQL servers or data, it does provide a unique shell environment with command completion, help for each utility, user defined variables, and type completion for options.

That's right, you no longer have to type out the entire name of the utility. For example, you can type mysqldbe and press the TAB key and it will complete the command as mysqldbexport. Don't remember the name of a database utility you want to use? That's no problem either - just type mysqldb and press TAB twice. The console will list all of the utilities that start with mysqldb.

The same is true for options. Type a few characters of the option name and press TAB to complete it or press TAB again if nothing returns and you will see a list of the options that match the prefix.

If that wasn't powerful enough, the best feature of the console allows you to define variables and reuse them in your commands.

For example, if you are planning to use a number of utilities or run the same utility many times, you can set a variable to store the connection information and just reuse it for each command. Here is how you could do that:

mysqluc> set SERVER=root:root@some.host.far.away:13000
mysqluc> mysqlrplshow --master==$SERVER ...

Note: user defined variables have a lifetime of the console run time.

There are a number of console commands as well. To see a list, enter 'help'.

mysqluc> help
Command                 Description                                       
----------------------  ---------------------------------------------------
help utilities          Display list of all utilities supported.          
help           Display help for a specific utility.              
help | help commands    Show this list.                                   
exit | quit             Exit the console.                                 
set =  Store a variable for recall in commands.          
show options            Display list of options specified by the user on  
                        launch.                                           
show variables          Display list of variables.                        
                 Press ENTER to execute command.                   
                Press ESCAPE to clear the command entry.          
                  Press DOWN to retrieve the previous command.      
                    Press UP to retrieve the next command in history. 
                   Press TAB for type completion of utility, option, 
                        or variable names.                                
              Press TAB twice for list of matching type         
                        completion (context sensitive).   
                

If you've been using MySQL Utilities or want to try them out, the users' console makes it easy to get started. Go to http://dev.mysql.com and download the latest version of MySQL Workbench and try out MySQL Utilties. You will find MySQL Utilities listed as a plugin to MySQL Workbench.

Wednesday, May 9, 2012

MySQL Utilities Frequently Asked Questions

Momentum for MySQL Utilities continues to build.  I hosted a webinar recently about MySQL Utilities (available on-demand from the link below), which generated a lot of interest and some good questions.

http://event.on24.com/eventRegistration/EventLobbyServlet?target=lobby.jsp&eventid=448952&sessionid=1&key=7E741ED049DFBF49D10C90A2B62E410F&eventuserid=63530507

With so many questions and ideas coming in I decided to create a blog of FAQs. I plan to add these to the MySQL documentation as well.  Keep your ideas and questions coming!

I hope you find these questions enlightening. I have grouped them for easier reading. You can find the MySQL Utilities documentation using the link below. 

http://dev.mysql.com/doc/workbench/en/mysql-utilities.html

General

Are these utilities present in the community version of MySQL?

They are included in the community edition of the MySQL Workbench product, which can be downloaded from the following link.

http://www.mysql.com/downloads/workbench/

Should/can we run this on live data?

Yes. Naturally, you would want to test some operations before jumping directly into a production environment. For example, you may want to test any database migration, transformation, or similar massive change in a test environment.

Can we use the utilities in a production environment under the GPL license?

Yes. MySQL Utilities is part of MySQL Workbench so all such licenses apply accordingly.

Storage Engines

Can the utilities be used with MyISAM or CSV?

Yes. There are no storage engine specific limitations in using the utilities. There are some features written specifically for InnoDB so those may not apply but in general no utility is storage engine specific. For example, the mysqldiskusage utility shows exact sizes for MyISAM and InnoDB files but uses estimated sizes for any other storage engine based on number of rows and row size.

Platforms

Can I use MySQL Utilities on Linux?

Yes. MySQL Utilities runs on all platforms supported by MySQL Workbench.

Can the utilities be used on Windows?

Yes!

Do the utilities work both for window-based and linux-based servers?

Yes! They work for any server hosting MySQL.

Installation

Do we have to install the utilities with rpm or can we use the tar ball extract and run from there?

MySQL Utilities is installed as part of MySQL Workbench. You can download and install Workbench using several platform-specific installers.

You can also branch and download MySQL Utilities from Launchpad. You can also build and install it from the source code you’ve downloaded using typical Python install steps (python ./setup.py install).

https://launchpad.net/mysql-utilities

What's the link to download these utilities?

MySQL Utilities is part of MySQL Workbench. You can download MySQL Workbench from the following link.

http://www.mysql.com/downloads/workbench/

Locking

Do the utilities lock tables while running?

Yes, but only for situations that require locks. The mysqldbexport utility also allows you to specify what type of lock to use:

no-locks = do not use any table locks

lock-all = use table locks but no transaction and no consistent read

snaphot (default) = consistent read using a single transaction.

Are there any utilities that can show DB locks (like which query is blocking which one)?

No, not currently but that is an excellent suggestion!

mysqldbcompare

How fast is mysqldbcompare? Say a table with 10 million rows?

It is difficult to predict a precise estimate of run time based on number of rows. However, it is generally such that the more rows there are the longer the utility will run. The mysqldbcompare utility is used to produce a difference of two databases. It creates a difference between objects of the same name for either object definitions, data, or both. When comparing object definitions, the performance is very fast because there isn’t a lot of processing involved. When comparing data, the utility uses an algorithm to create checksums for each row in the table. During this phase, the tables are locked. Once that stage is done, the tables are unlocked and the algorithm begins to compact the checksums into chunks, which are later compared between the servers. If the checksums differ, the chunks are expanded and the differences calculated. Thus, for tables containing millions of rows the utility will take some time to complete. The best time to run this utility is during low usage periods such as times reserved for upgrades, backups, and similar operations.

How will running mysqldbcompare effect a production database?

If generating a difference for data, the utility will lock the tables long enough to calculate a checksum for each row. Depending on the number of rows this could be for a long time and in those cases you should run mysqldbcompare during low usage periods. The utility will use a consistent read to lock InnoDB tables but will issue table locks for non-InnoDB tables.

Will mysqldbcompare cause table locking on MyISAM table?

Yes. A table lock is issued during checksum creation.

What is the load going to be on the servers when mysqldbcompare runs?

The load on the server itself is minimal. There is moderate CPU usage during checksum creation but nothing that should cause a problem. The longest period of activity is when the table scans are executed for creating a checksum for each row.

mysqldbexport

Is mysqldbexport similar to mysqldump?

Yes, the mysqldbexport is designed to export data in a row-by-row or logical fashion. However, you can export data in CSV, TAB, Vertical formats as well as SQL statements using CREATE TABLE, INSERT, etc. making mysqldbexport more versatile than mysqldump. You would use mysqldbexport in situations where you need special machine or human readable output for operations like transforming the data or examining the structures in more detail – especially if you need a format other than SQL statements.

Replication Utilities

Are the high availability features available for version 5.5 or 5.6?

The general replication utilities such as mysqlrplcheck and mysqlreplicate will work with servers version 5.0 and later. The newest high availability feature, failover, in mysqlrpladmin and mysqlfailover work only for servers that support global transaction identifiers (GTIDs) which were added in version 5.6.5.

You can discover more about GTIDs from the following blog by Luis Soares.

http://d2-systems.blogspot.co.uk/2012/04/global-transaction-identifiers-are-in.html

Where do I get more info about mysqlrpladmin?

The online MySQL Workbench Manual has information about each utility. You can also use the --help option to show all options and their descriptions.

http://dev.mysql.com/doc/workbench/en/mysql-utilities.html

How can I use the utilities to test replication on a single host?

You can use mysqlserverclone to clone an existing, running instance of MySQL or clone from an installation (basedir), then mysqlreplicate to create the replication topology.

Is the replication failover feature only for version 5.6?

Yes. It requires support for global transaction identifiers, which were added in version 5.6.5. A developer milestone release of 5.6 is available for download.

http://dev.mysql.com/downloads/mysql/#downloads

(Select the Development Releases tab)

What features of mysqlrpladmin will work on version 5.5?

All of the features except slave election and failover.

Do you need to create a replication user on the slave site other than the master?

The mysqlreplicate utility provides an option to use a specific user on the master for replication or it will create a user by default. You can also request that a new user be created during the operation.

Monday, October 10, 2011

MySQL Utilities Release 1.0.3

The MySQL Utilities project continues to evolve with key new features for
replication and export. The latest release, 1.0.3, is no exception.

MySQL Utilities is included in the MySQL Workbench product which can be
downloaded from HTTP://dev.mysql.com/downloads/workbench/5.2.html

If you want the latest developments for MySQL Utilities, you can create a
bazaar branch using the following command:

bzr branch lp:~mysql/mysql-utilities/trunk

New Utility - mysqlrplshow

You can now view a list of the slaves attached to your master with
mysqlrplshow. The utility displays a graph of the master and its slaves y
default but you can also get a list of the slaves in GRID, CSV, TAB, or
VERTICAL format as follows.

  • GRID - Displays output formatted like that of the mysql monitor in a grid or table layout. 
  • CSV - Displays the output in a comma-separated list. 
  • TAB - Displays the output in a tab-separated list.
  • VERTICAL - Displays the output in a single column similar to the '\G' option for the mysql monitor commands.

You can use the GRID format to output the data and store it away for a later
run then compare or diff against the recent output to see what has changed. In
this case, what new slaves have been changed, added or removed from your
topology.

# Replication Topology Graph
localhost:3311 (MASTER)
   |
   +--- localhost:3310 - (SLAVE)
   |
   +--- localhost:3312 - (SLAVE + MASTER)
       |
       +--- localhost:3313 - (SLAVE)


If you want to see your replication topology, you can use the --recurse option
to instruct mysqlrplshow to traverse each slave found to locate downstream
slaves. Circular topologies are detected and displayed using a special
notation. You can even limit the depth of the recursive search if you want to
see only a portion of your topology.

The utility currently works by reading slaves with the SHOW SLAVE HOSTS
command. For most versions of MySQL, this requires the use of --report-host and
--report-port options when setting up your slaves. For example, version 5.5.3
and later always show the slaves regardless if --report-host is specified.
Versions older than 5.5.3 will not show the slaves unless --report-host is
specified.

You should specify the --report-host option when starting your slaves but take
care to ensure the correct host or IP is specified. If the slave is using a NAT
derived IP, be sure to use the IP address that can reach the slave from the
master.

You do not normally need to specify --report-port unless the slave is using a
custom port (something other than the default of 3306).

Improvements

There have been a number of minor enhancements and there are two major feature
additions that will improve usage in many scenarios.

Enhanced binlog control for mysqlreplicate - you can now start replication in
one of four ways:

  1. Start from the first event recorded in the binary log on the master. This is typically used to setup a new master and slave replication topology.
  2. Start from the first event in a specific binary log file This is used to restart replication from a specific point or to start replication after a slave has had data loaded (for example via a restore or file copy from the master).
  3. Start from a specific event (position) in a specific binary log file. This is a common operation for point-in-time recovery.
  4. Start from the current position of the master. This is typically used to start a slave following a recovery of the slave, fail-back to a recovered master, or restarting replication after an error event has been resolved.

We added --all option to mysqldbcopy, mysqldbexport. These utilities allow you
to include all of the databases on your server in the operation except the
system databases (mysql, information_schema, and performance_schema).

Try it out and let us know what you think!

Thursday, April 7, 2011

MySQL Workbench Utilities

Introducing MySQL Workbench Utilities

One of the many new things that is being introduced during the Collaborate 2011 and 2011 MySQL Users’ Conference are some great new additions to some of the external tools for managing MySQL servers. One of those tools receiving updates is the MySQL Workbench.

One of the jewels in a long list of new features is the addition of new command-line utilities to help you administer your servers. The new feature is called MySQL Workbench Utilities. It is a package of easy-to-use utilities for maintenance and administration of MySQL servers. These utilities incapsulate a set of primitive commands bundling them so that you can perform macro operations with a single command. Some of the key features in MySQL Workbench Utilities include:

  • Plugin for MySQL Workbench 5.2.31
  • Available under the GPLv2 license
  • Written in Python
  • Easily to extend using the supplied library
How Does It Work?

There are two ways to access the utilities from within the MySQL Workbench.

You can click on the drop down arrow icon to the right of the MySQL Workbench main window. This will display a list of the plugins available. You can scroll through the screens and find the MySQL Utilities icon. The image below shows what the window looks like.




Once you hover on the MySQL Utilities icon, you will see an Action button appear. Click Action then Start Plugin to launch the MySQL Utilities shell in a new window. The image below shows what the window would look like.



You can launch any of the utilities listed by typing the name of the command. To find out what options are available, use the --help option.

Note: you can also add the MySQL Utilities icon to your home screen. Highlight Add to Home Screen and then choose a position from the menu.

You can also launch the MySQL Utilities command window by clicking on the Plugins menu item and selecting Start Shell for MySQL Utilities.

What Utilities are Available?

The MySQL Workbench and MySQL Workbench Utilities developers are adding new features and utilities with every release. If you don’t see a utility you want, check back with each release of Workbench or send the developers an email and let them know what you want!

You should try the --help option for each tool to see what options it supports. Some utilities have lots of options for controlling the operation (for example the mysqldbexport can produce one of several formats with object definitions, data, or both).

There are several options that are common to the utilities available (for instance --version, --help, --verbose) but the one that is most important is the server specification option used to connect to a specific server. This manifests as --server, --source, --destination, or --server1, etc. and all require the same format as follows:

--server=user_name:password@hostname:port_num:socket_file.

The parameter includes the user name followed by the user’s password (if available) separated by a colon (colon is omitted if no password) followed by the @ symbol and the host machine name. Optionally, you can provide the port number and the socket file each preceded by a colon. That sounds like a lot of crazy typing but it is really simple. The following are some examples using this format.

--server=root@localhost:3306
--server=joe:pass@192.168.1.101
--server=dolly:sassyshoes@localhost:3310:/tmp/mysql.sock

The following list describes the utilities available as well as some examples of use. For more details, please see the manuals for each utility.

mysqldbcopy - Permits a database administrator to copy a database from one server either to another server as the same name or a different name or to the same server as the same or as a different name.

Example 1: Copy a database named ‘util_test’ to a new name ‘util_test_copy’ on the same server.

mysqldbcopy util_test:util_test_copy
--source=root:pass@host1:3306
--destination=root:pass@host1:3306

Example 2: Copy a database named ‘util_test’ to another server.

mysqldbcopy util_test:util_test
--source=root:pass@host1:3306
--destination=root:pwd@host2:33010

mysqldbexport - Permits a database administrator to export the metadata (object definitions, hence definitions) or data or both from one or more databases. By default, the utility will export only definitions.

Example 1: Export the definitions of the database ‘dev’ from a MySQL server on localhost via port 3306 producing CREATE statements.

mysqldbexport --skip=GRANTS
--server=root:pass@localhost
--export=DEFINITIONS util_test

Example 2: Export the data of the database ‘util_test’ producing bulk insert statements.

mysqldbexport --export=DATA
--bulk-insert util_test
--server=root:pass@localhost

mysqldbimport - Permits a database administrator to import the metadata (objects) or data for one or more databases from one or more files in either SQL or a text format such as CSV, TAB, GRID, or VERTICAL.

Example 1: Import the metadata of the database ‘util_test’ to server1 on port 3306 using a file in CSV format.

mysqldbimport --import=definitions
--server=root@localhost --format=csv
data.csv

Example 2: Import both the data and definitions of the database ‘util_test’ to server1 on port 3306 producing bulk insert statements from a file that contains SQL statements.

mysqldbimport data.sql --import=both
--bulk-insert --format=sql
--server=root@localhost

mysqldiff - Reads the definitions of objects and compares them using a diff-like method to determine if two objects are the same based on the definition of the object.

Example 1: Find the differences among objects in database employees on one server and emp1 on the same server.

mysqldiff --server1=root@localhost
employees:emp1

Example 2: Find differences between the salaries table on one server and the same table on another server.

mysqldiff --server1=root@localhost
--server2=root@host2:3306
employees.salaries:emp1.salaries
--differ

mysqldiskusage - Permits a database administrator to see the disk space usage for one or more databases in either CSV, TAB, GRID, or VERTICAL text formats. The utility will also allow the user to examine the disk usage for the binary logs, slow, error, and general log; and InnoDB tablespace usage.

Example 1: Show only the disk space usage for the employees and test databases in grid format.

mysqldiskusage --server=root@localhost
employees test

Example 2: Show all disk usage for the server in CSV format.

mysqldiskusage --server=root@localhost
--format=csv -a

mysqlindexcheck - Eeads the indexes for one or more tables and identifies duplicate and potentially redundant indexes.

Example 1: Scan all of the tables in the employees database to see the possible redundant and duplicate indexes as well as the DROP statements for the indexes.

mysqlindexcheck --server=root@localhost 
 --show-drops employees

Example 2: Scan all of the tables in the employees database and display the indexes. Display the output in a tabular format.

mysqlindexcheck --server=root@localhost
--show-indexes --format=TAB employees

mysqlmetagrep - Searches for objects matching a given pattern and shows a table of the objects that match the pattern `’t\_’.

mysqlmetagrep --pattern=“t_”
--server=mats@localhost

Example 2: Find all objects that contain ‘t2’ in the name or the body (for routines, triggers, and events).

mysqlmetagrep -b --pattern=“%t2%”
--server=mats@localhost:3306

mysqlprocgrep - Scans the process lists for processes that match the search criteria specified and will either print the result (the default) or execute certain actions on it.

Example 1: Kill all connections created by user “mats” that are younger than 1 minute.

mysqlprocgrep --server=root@localhost
--match-user=mats --age=1m
--kill-query

Example 2: Kill all connections with queries that have been idle for more than 1 hour.

mysqlprocgrep --server=root@localhost
--match-command=sleep --age=1h
--kill

mysqlreplicate - Permits an administrator to start replication among two servers. The user provides login information to the slave and provides connection information for connecting to the master.

Example 1: Setup replication between a MySQL instance on two different hosts using the default settings.

mysqlreplicate --rpl-user=rpl:rpl
--master=root@localhost:3306
--slave=root@localhost:3307

Example 2: Ensure the replication between the master and slave is successful if and only if the InnoDB storage engines are the same and both servers have the same storage engines with the same default specified.

mysqlreplicate --pedantic
--master=root@localhost:3306
--slave=root@localhost:3307
--rpl-user=rpl:rpl -vv

mysqlserverclone - Permits an administrator to start a new instance of a running server.

Example 1: Create a new instance of a running server.

mysqlserverclone --new-port=3310
--server=root:pass@localhost
--new-data=/source/test123

Example 2: Create a new instance of a running server set the root password and turn binary logging on.

mysqlserverclone --new-port=3310
--server=root:pass@localhost
--new-data=/source/test123
--root-password=pass
--mysqld=--log-bin=mysql-bin

mysqluserclone - Permits a database administrator to use an existing user account on one server as a template and clone a MySQL user such that one or more new user accounts are created on another (or the same) server with the same privileges as the original user.

Example 1: Clone ‘joe’ as ‘sam’ and ‘sally’ with passwords and logging in as root on the local machine.

mysqluserclone --source=root@localhost
--destination=root@localhost
joe@localhost sam:secret1@localhost
sally:secret2@localhost

Example 2: Show all of the users on the localhost server in the most verbose output in CSV format.

mysqluserclone --source=root@localhost
--list --format=CSV -vvv

mut - Designed to execute predefined tests to test the MySQL Utilities.

Example 1: Run all tests.

mut --server=root@localhost

Example 2: Run all tests that start with ‘clone_user’ and set the width of the display to 70 characters.

mut --server=root@localhost
--do-tests=clone_user –width=70

How Can I Get It?

You can download MySQL Workbench from:

http://www.mysql.com/downloads/workbench/

You can also download the latest development source code tree for the MySQL Workbench Utilities from:

https://launchpad/net/mysql-utilities