1.1. Copyright Information

This document is Copyright © 2001 Vivek Venugopalan. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be found in Appendix A.

This document may be reproduced and distributed in whole or in part, in any medium physical or electronic, as long as this copyright notice is retained on all copies. Commercial redistribution is allowed and encouraged; however, the author would like to be notified of any such distributions.

All translations, derivative works, or aggregate works incorporating this document must be covered under this copyright notice. That is, you may not produce a derivative work from this document and impose additional restrictions on its distribution. Exceptions to these rules may be granted under certain conditions; please contact the author at the address given below.

In short, we wish to promote dissemination of this information through as many channels as possible. However, we do wish to retain copyright on the document, and would like to be notified of any plans to redistribute the same.

1.2. Disclaimer

No liability for the contents of this document can be accepted. Use the concepts, examples and other content at your own risk. As this is a new edition of this document, there may be errors and inaccuracies that may of course be damaging to your system. Proceed with caution, and although this is highly unlikely, the author(s) do not take any responsibility whatsoever.

All copyrights are held by their respective owners, unless specifically noted otherwise. Use of a term in this document should not be regarded as affecting the validity of any trademark or service mark.

Naming of particular products or brands should not be seen as endorsements.

You are strongly recommended to take a backup of your system before major installation and backups at regular intervals.

1.3. New Versions

This document is Version : 0.7.

The latest version of this document can be obtained from (In the order of latest version availability)

1. My website

2. The linux documentation project

1.4. Credits

The list of people who have provided information and correction for this paper in no particular order are.

1. Jens-Uwe Mager

2. Jorgen Grahn

3. Thomas S. Urban

4. Cam Mayor

5. Sally Miller

6. Niels Jakob Darger

1.5. Feedback

Feedback is most certainly welcome for this document. Without your submissions and input, this document wouldn't exist. Please send your additions, comments and criticisms to the following email address : <vivekv at yahoo dot com>.

3.1. Use GUI CVS client

This paper recommends using such GUI clients during the initial deployment of CVS in an organization.

Developers typically use integrated development environments that have the CM tools integrated into them. These tools minimize the learning for the developers about the intricacies of CVS usage and instead allow them to be productive from day one. Developers who are accustomed to other CM tools will find the CVS command-line interface daunting. The adoption and usage of CVS can be improved by using GUI tools for CVS clients.

GUI tools for CVS are available at http://cvsgui.sourceforge.net/. GUI interfaces are available for most of the popular platforms (Windows, Mac and Linux). In addition, on the Windows platform there is an SCC extension that allows integration of CVS as the configuration control tool with popular IDE.

4.1. Keep System clocks in Sync

CVS tracks change to source files by using the timestamp on the file. If each client system date and time is not in sync, there is a definite possibility of CVS getting confused. Thus system clocks must be kept in sync by use of a central time server or similar mechanism.

CVS is designed from ground up to handle multiple timezones. As long as the host operating system has been setup and configured correctly, CVS will be able to track changes correctly.

4.2. Do not share the sandbox

Sandboxes have to be unique for each developer or purpose. They should not be used for multiple things at the same time. A sandbox can be a working area for a developer or the build area for the final release. If such sandboxes are shared, then the owner of the sandbox will not be aware of the changes made to the files resulting in confusion.

In CVS, the sandbox is created automatically when a working copy is checked out for a CVS project using the cvs checkout {project-name} command.

In very large projects, it does not make sense for the developers to check-out the entire source into the local sandbox. In such cases, they can take the binaries generated by the build team on a regular basis for all those components of the application that is not changed by them and only check-out the parts that are built by the developer.

For example, in a Java project, the build team can keep the results of their last successful build in a standard location in the form of JAR files on the network file servers. Individual developers will use a standard classpath setup that has the network drives mounted on standard paths. Thus, the developers will automatically get the latest version of the files as required by them.

4.3. Stay in sync with the repository

To gain the benefits of working within a sandbox as mentioned above, the developer must keep his or her sandbox in sync with the main repository. A regular cvs update with the appropriate tag or branch name will ensure that the sandboxes are kept up to date.

4.4. Do not work outside the sandbox

The sandbox can be thought of as a controlled area within which CVS can track for changes made to the various source files. Files belonging to other developers will be automatically updated by CVS in the developer's sandbox. Thus the developer who lives within the sandbox will stand to gain a lot of benefits of concurrent development.

4.5. Cleanup after Completion

Make sure that the sandbox is cleaned up after completion of work on the files. Cleanup can be done in CVS by using the cvs release command. This ensures that no old version of the files exists in the development sandbox. As explained previously, pre-built binaries from the build team can be used to ensure that all the parts of the application are available to the developer without the need for a complete compilation in the sandbox.

4.6. Check-in Often

To help other developers keep their code in sync with your code, you must check-in your code often into the CVS repository. The best practice would be to check-in soon as a piece of code is completed, reviewed and tested, check-in the changes with cvs commit to ensure that your changes are committed to the CVS repository.

CVS promotes concurrent development. Concurrent development is possible only if all the other developers are aware of the ongoing changes on a regular basis. This awareness can be termed as "situation awareness"

One of the "bad" practices that commonly occur is the sharing of files between developers by email. This works against most of the best practices mentioned above. To share updates between two developers, CVS must be used as the communication medium. This will ensure that CVS is "aware" of the changes and can track them. Thus, audit trail can be established if necessary.

5.1. CVS access control

One of the important questions that I have been asked time and again is the ability to have access control for files/folders/branches etc., within the CVS repository for various users. Unfortunately CVS does not come with a built in Access control capability but it does support a rudimentary form of access control through the readers/writers files in the CVSROOT repository. I have put together a set of scripts that use the readers/writers files to provide a slightly useable version of access control. This is available at http://cvspermissions.sarovar.org as an Open Source project. Feel free to use it and let me know how it works for you.

5.2. Server side scripting

Server side scripting refers to the ability to make CVS server execute certain scripts when an event occurs. A common script that helps is to verify that all cvs commits contain acomment entered by the developer. The process involves setting up the CVSROOT/verifymsg file to run a script when a file is checked-in.

------CVSROOT/verifymsg---------

#Set the verifymsg file to fire a script
DEFAULT /usr/local/bin/validate-cvs-log.sh


------/usr/local/bin/validate-cvs-log.sh ---------

#!/bin/sh
#
# validate-cvs-log.sh logfile
 
# test that log message has some characters in it
if [ `cat $1 | wc -c ` -lt 10 ] ; then
echo "log message too short; please enter a description for the changes"
    exit 1
else
    exit 0
fi

5.3. Server Notification

The CVS server can be configured to notify through e-mails in case of a commit happening. This can be used to verify whether commits are occurring during the course of a daily/release build. If such commits occur, based on the project policy, the commits can be ignored or the entire build automatically restarted.

6.1. Assign ownership to Trunk and Branches

The main trunk of the source tree and the various branches should have a owner assigned who will be responsible for.

1. Keep the list of configurable items for the branch or trunk.

   The owner will be the maintainer of the contents list for the branch or trunk. This list should contain the item name and a brief description about the item. This list is essential since new artifacts are always added to or removed from the repository on an ongoing basis. This list will be able to track the new additions/deletions to the repository for the respective branch.

2. Establish a working policy for the branch or trunk.

   The owner will establish policies for check-in and check-out. The policy will define when the code can be checked in (after coding or after review etc.,). Who is responsible to merge changes on the same file and resolve conflicts (the author or the person who recently changed the file).

3. Identify and document policy deviations

   Policies once established tend to have exceptions. The owner will be responsible for identifying the workaround and tracking/documenting the same for future use.

4. Responsible for merge with the trunk

   The branch owner will be responsible for ensuring that the changes in the branch can be successfully merged with the main trunk at a reasonable point in time.

6.2. Tag each release

As part of the release process, the entire code base must be tagged with an identifier that can help in uniquely identifying the release. A tag gives a label to the collection of revisions represented by one developer's working copy (usually, that working copy is completely up to date so the tag name is attached to the "latest and greatest" revisions in the repository).

The identifier for the tag should provide enough information to identify the release at any point in time in the future. One suggested tag identifier is of the form.

release_{major version #}_{minor version #}

As one reader pointed out to me, a good practice here is to tag the release first. Checkout the entire codebase using the tag, and then proceed to go through a build / deploy / test process before making the actual release. This will absolutely ensure that what "leaves the door " is a verified and tested codebase.

6.3. Create a branch after each release

After each software release, once the CVS repository is tagged, a branch has to be immediately created. This branch will serve as the bug fix baseline for that release. This branch is created only if the release is not a bug fix or patch release in the first place. Patches that have to be made for this release at any point in time in the future will be developed on this branch. The main trunk will be used for ongoing product development.

With this arrangement, the changes in the code for the ongoing development will be on the main trunk and the branch will provide a separate partition for hot fixes and bug fix releases.

The identifier for the branch name can be of the form.

release_{major version #}_{minor version #}_patches

6.4. Make bug fixes to branches only

This practice extends from the previous practice of creating a separate branch after a major release. The branch will serve as the code base for all bug fixes and patch release that have to be made. Thus, there is a separate repository "sandbox" where the hot fixes and patches can be developed apart from the mainstream development.

This practice also ensures that bug fixes done to previous releases do not mysteriously affect the mainstream version. In addition, new features added to the mainstream version do not creep into the patch release accidentally.

6.5. Make patch releases from branches only

Since all the bug fixes for a given release are done on its corresponding branch, the patch releases are made from the branch. This ensures that there is no confusion on the feature set that is released as part of the patch release.

After the patch release is made, the branch has to be tagged using the release tagging practice (see Tag each release).

7.1. Merge branch with the trunk after release

After each release from a branch, the changes made to the branch should be merged with the trunk. This ensures that all the bug fixes made to the patch release are properly incorporated into future releases of the application.

This merge could potentially be time consuming depending on the amount of changes made to the trunk and the branch being merged. In fact, it will probably result in a lot of conflicts in CVS resulting in manual merges. After the merge, the trunk code base must be tested to verify that the application is in proper working order. This must be kept in mind while preparing the project schedule.

In the case of changes occurring on branches for a long period, these changes can be merged to the main branch on a regular basis even before the release is made. The frequency of merge is done based on certain logical points in the branch's evolution. To ensure that duplicate merging does not occur, the following practice can be adopted.

In addition to the branch tag, a tag called {branch_name}_MERGED should be created. This is initially at the same level as the last release tag for the branch. This tag is then "moved" after each intermediate merge by using the -F option. This eliminates duplicate merging issues during intermediate merges.

8.1. Build Early and Build Often (BEBO)

A variation of this adage has been around in the Open Source community called "Release Early and Release Often" for quite some time albeit for a different reason. BEBO helps a development team identify issues that can arise from checking in the wrong files. BEBO will address integration issues at the application level that might have slipped passed individual developer builds. It will also improve the team morale when they see a working version of the application.

Builds must be done on a regular basis. There should be a dedicated resource(s) assigned to do the same. The entire project team must be trained to view the daily build as an important activity and not as a chore. Builds must be completed without any failures on a regular basis. Build failures must be a rare event and should be treated with utmost seriousness. The project team should ensure that successful builds are top priority on their agenda. The seriousness can be emphasised by setting up a penalty for breaking the build.

Each build can be tagged in CVS using a standard naming convention. This can help developers checkout a working version of the entire system from daily builds for local development.

8.2. Automate build Process completely

Another key practice for software builds is to automate the build process completely. The automation process must also include automatic retrieval of the right source files from the CVS repository. This ensures that the build process is completely repeatable and consistent. In addition, the chances of a build with the wrong version of the application source files are reduced to a large degree.

By automating the build process, the task of building often becomes less burdensome.

8.3. All necessary files must be checked-in before build

This adage sounds trivial at first but this problem is very common even with experienced development teams due to oversight. The problem of oversight cannot be easily addressed since the onus is on the individual developer to ensure that his or her file has been checked in. This practice should be drummed into the team in the form of training and pre-build announcements to ensure that the right version of source code is available in the repository.

Automated build process as explained above will help in catching this problem to a certain degree since they will automatically take the source code from the CVS repository and perform the software build. Any missed items will surface during the build process itself (makefiles etc.,) or during the regression testing of the product (older version of the file checked in).

A penalty based system can be setup to handle wrong check-in. Having a kitty for a post project party to which each person who makes a wrong check-in will contribute a fixed amount will act a good penalty system.

9.1. Implement Change Management Process

All organizations must implement a good Change management process (CMP). A good CMP will define how changes are received, recorded, tracked, executed and delivered. CVS provides version control for your project. Change management addresses the "bigger picture" of how enhancements and bugs are received, tracked and closed. CVS will play a smaller but a very important part in this entire picture. With a formal change management process in place in the organization, tools such as CVS will be looked at as aiding this process instead of acting as a general development overhead.

Change management is quite a vast topic that cannot be done justice here. Please look up other sources of information on change management.

9.2. Make CVS Usage part of Objectives

To institutionalize CVS, it can be made as part of the performance objectives for the developer to use CVS in the project. In addition, it can also be made as part of the objective for the project manager to deploy CVS in his or her project.

Compliance of this can then be reviewed as part of the appraisal cycle for the employee.

9.3. Collect metrics on CVS usage

CVS usage metrics can be collected in terms of percentage of deployment in the organization, project size handled etc., This information will spur other line managers and program managers to look at CVS as a tool that will aid them in their daily operations.

10.1. Inception

Consider a software project where version 1.0 has just been put into production and everyone is done celebrating. The next step is to start working on the new features of the subsequent release. Also, the users of the system have started to use it full-time and bug reports of various levels have started to come in.

Before jumping into new enhancements or bug fixes, the best practices for Branching and Merging should be followed. Few of the important practices are Tag each release and Create a branch after each release. These practices will effectively established two "development environments", one for regular enhancements and the other for bug fixes and minor enhancements on the last release.

Let us assume that the release was tagged as

release_1_0

Then the branch was created with the branch name

release_1_0_patches

10.2. Development and Delivery

Now, we are ready for business. Let us examine the bug fixes and enhancements track. Assume that there are three bugs of which two are of a high priority that should be fixed right away (possibly within a week) and the third can be delivered after some time (say after 4 weeks). In the middle of this schedule there is a regular release scheduled in three weeks. Considering that we have a busy month ahead, let us see how exactly we can use the Best practices to ease the days ahead.

The timeline for the various release in the next month looks like this.

             Fix       Enhancement  Fix 
     Today   Release 1  Release    Release 2
	 |_______|______________|_________|
		    Time -->

We have two teams, one working on the bug fix branch and another team working on the features for the next release on the main trunk. These teams must make sure that they start out with the right version in their sandbox.

1. The bug fix team will check out using the command line

   cvs checkout -R -r release_1_0_patches {project name}

2. The team that is working on the next release will use the command line

   cvs checkout -R {project name}

As soon as the bug fix team completes the two top priority bugs, they will update, verify a successful build and commit their changes to the bug fix branch using the command line

The team should perform a build at this point to verify that the update did not break any code on the branch. Once the build is successful, the branch should be committed back into the repository.

Build Early and Build Often : On a daily basis, each developer will check in code to CVS and to ensure sanity of code, daily builds on the bug fixed branch will be undertaken by checking out from CVS on a clean environment and completely rebuilt. These daily builds can be tagged in CVS using the following naming convention

build_1_1_yyyymmdd : for the branch
build_2_0_yyyymmdd : for the trunk

The regular process of build-test-fix is followed to make a version ready for delivery. The tag will help developers checkout a working copy of the latest build as and when necessary.

When the source code is released to the outside world, two practices have to be followed.

1. Tag each release : This ensures that the bug fix release is tagged correctly and so can be traced out at a later point in time if necessary.

2. Merge branch with the trunk after release : This ensures that the bug fix is merged back into the main trunk ensuring that all future releases is a truly cumulative delivery.