kopano-dbadm(8)

Name
kopano-dbadm — database administration utility

Synopsis
kopano-dbadm [-c server.cfg] [-j jobs] action...

Description

This tool is for special surgical tasks on the Kopano SQL database. kopano-server ought to be stopped when running this utility, unless otherwise noted. You should make a database backup first.

Options

-c server.cfg
    By default, kopano-dbadm will read /etc/kopano/server.cfg for MySQL connection parameters, but an alternate file may be specified.

-j jobs
    Specifies an upper limit on the number of SQL queries running simultaneously. This may improve or worsen the overall time to process ALTER statements, depending on the underlying storage of the SQL database. The default is 1.

Actions

  index-tags

Create helper indices for the "tag" columns. This action can be executed while kopano-server is active.

  k-1216

This is an alias for the combination of: index-tags, np-remove-highid, np-remove-unused, np-repair-dups, np-defrag, rm-helper-index. It may be used to address the K-1216 issue that kopano-server(8) may report when performing a schema update. If the tag indices already exist, harmless warnings will be shown by kopano-dbadm, but the indices will be used and left as-is after the k-1216 action is done.

  kc-1375

Deletes all cached IMAP Envelopes that show signs of having address lists with excess parentheses. This action can be executed while kopano-server is active. If you do so, also execute `kopano-srvadm --clear-cache=cell` afterwards.

  kc-1444

Synchronizes all PR_MESSAGE_FLAGS values from the "properties" to the "tproperties" table. This action can be executed while kopano-server is active. If you do so, execute `kopano-srvadm --clear-cache=cell` afterwards.

  kf-2888

Fixes corruption in the stores.guid column by copying back by re-extracting the store GUIDs from the PR_ENTRYID of the store object.

  np-defrag

Compact the ID space in the namedproperties table and lowers the AUTO_INCREMENT value, thereby making IDs actually usable after a remove operation (such as np-remove-xh).

  np-remove-highid

Remove all named property mappings with internal ID higher than 31485. As Kopano/Zarafa MAPI clients cannot handle these IDs anyway, the IDs are never used to store any data and only occupy the ID space needlessly.

  np-remove-unused

Remove all named property mappings that have zero use counts in other tables.

  np-remove-xh

kopano-dagent(8) used to index all X-Headers from Internet e-mails by default, but administrators may not have needed this information in the first place. By using np-remove-xh, all named properties whose name starts with "X-" and have GUID=PS_INTERNET_HEADERS are removed. Afterwards, `kopano-dbadm np-defrag` should be run.

  np-repair-dups

Repairs duplicate named properties (only IDs below or equal to 31485).

  np-stat

Show statistics for the namedproperties table. This action can be executed while kopano-server is active.

  populate

Create and populate the server database as specified in the configuration file. kopano-server normally does this itself if it detects an empty database. When, however, an SQL application layer gateway is used that prevents DDL statements, calling dbadm populate (with a separate config file specifying not the gateway, but rather the origin server) can be used.

  rm-helper-index

Remove helper indices again. This action can be executed while kopano-server is active.

  usmp

This is an alias for the combination of: usmp-shrink-columns, usmp-charset.

  usmp-charset

Convert the database to utf8mb4 to support the Unicode Supplemental Plane, better known for Emoji characters. This action can be executed while kopano-server is active, though the SQL server may queue/block database write operations, rendering the server practically unavailable.

  usmp-shrink-columns

A number of table columns will be shrunk to accommodate certain MySQL index limits. This action can be executed while kopano-server is active, though the SQL server may queue/block database write operations, rendering the server practically unavailable.

Performance

Totally depends on the amount of data that needs to be moved. MySQL also seems to process the requests single-threadedly.

An approximate row count of a table can be retrieved with SHOW TABLE STATUS LIKE "properties". For an exact count, use SELECT COUNT(*) FROM `properties`.

  index-tags

System: 16-thr Ryzen 1700X, 64 GB memory, 2-disk RAID-1 Toshiba XG5 NVMe, Linux 5.0, mariadb 10.2, filesystem yields 430 MiB/s sequential read speed, 130 MiB/s sequential write speed.

Database: 8.9 GiB, properties: 17.78M rows, tproperties: 13.22M, mvproperties: 10.3K, indexedproperties: 445K, singleinstances: 228K rows.

dbadm index-tags -j1: completed in 48s. top(1) reported system load 1.25 with mysqld using 100% CPU (ceiling 1600%). This operation was CPU-bound.

dbadm index-tags -j6: completed in 27s.

  k-1216

System: 12-thr i7-3930K, 64 GB memory, 2-disk RAID-1 SATA SSD, Linux 4.18ish, mariadb 10.x.

Database: 4.96 GiB, properties: 12.94M rows, tproperties: 9.65M, mvproperties: 9.9K, indexedproperties: 3.24M, singleinstances: 188K, names: 2988 rows.

dbadm k-1216: 2 minutes indexing

dbadm np-remove-xh np-defrag: 5 1/2 minutes.

  usmp-charset

Same 1700X/XG5/8.9GiB/17.78M.

dbadm usmp-charset -j1: completed in 5m42s. top(1) reported system load 1.56, with mysqld using 55-70% CPU use. So this operation was waiting on I/O.

dbadm usmp-charset -j2: completed in 6m38s, as there was nothing more to be gained from the I/O.

usmp-charset -j2 on tmpfs: 3m13s.

See also

kopano-server.cfg(5)