Login | Register
My pages Projects Community openCollabNet

cabie
Wiki: Cabie Usage

Wiki: Cabie Usage

Edit this page | Links to this page | Page information | Attachments | Refresh page

 

What is CABIE?

Cabie (Continuous Automated Build and Integration Environment) is an open source, client / server based build system using a centralized collection point for gathering all build information which can then be used for easy access to builds, their logs, file changes, bug/issue associations, build statistics and more. Cabie utilizes RPC/XML as the underlying communications protocol. On POSIX based systems such as linux and OS X, Cabie runs as a daemon. On Win32 systems Cabie operates as standard windows service.

Cabie provides rapid email notification of completed builds (pass or fail) to individual users as well as groups, it also provides a watchdog facility used to notify build administrators and IT personnel of problems such as late builds, low available disk space and service interruption. The UI supports a broad range of assignable user levels limiting access to functionality. There is also a common SQL editing engine for use with internal tables for editing of job parameters, custom fields, users, promotion states, job groupings and file exclusion lists.

Communications between the collector and various build servers is handled by wrapping up supported Cabie server calls using RPC/XML, rendering them in a standard Apache instance which supports PHP. Access to each server can also be handled via the command line interface. Each instance of the Cabie server can redirect calls to other known Cabie servers. The command line interface has record, playback, save and load features which can be utilized for scripting.

For each job defined, there are standard sets of available build triggers:

  • prebuild
  • onfail
  • testbuild
  • postbuild
  • postpostbuild

The triggers are executed only when they are defined as part of the server configuration and exist as scripts.

Build scripts can be defined as retail (release) or debug scripts. Functionality may be combined, or each script run separately. If the script exits with a non-zero code, the build fails and the onfail trigger is executed. If the build completes successfully, the postbuild trigger is called. If the build has a testbuild trigger defined and the trigger exits with a non-zero code, the build is flagged as having failed test, not failing the build, the postbuild script is then executed.

For each build there may be a <buildname>.errors file that is parsed separately upon completion the build to determine where (if possible) errors may exist in the build logs. It is possible to have a return code of 0 and still trap an error via the <buildname>.errors file thus resulting in a failed build. The <buildname>.errors also have an exclusionary identifier in order to avoid having expected unit test failures (or other strings) treated as errors.

Job Partitioning

In a typical environment, jobs will be partitioned by server, based upon the tool set available on the server.

General Usage of the Cabie UI

Please note that hovering over any icon will give an indication as to its use.

Job Groupings

On some of the Cabie servers, jobs may be grouped together in order to allow displaying selected job types. Moving between defined sets of jobs just takes selecting the defined job set using the pull down menu 'group by type' under 'view options' displayed below the last defined job:

The resulting display will only list build jobs for the selected group, in this case 'Player'. The default grouping displayed when the UI is initially rendered is called 'All'.

Job Identification

The job as displayed on the job menu may look like this for an accurev based job:

Along with menu functions, and the job title is, in parentheses, the parent stream of the accurev client associated with the job, and the feature number of the build. The feature number is part of Cabie's ability to set custom values, then use them internally and while executing builds.

A job which uses bazaar as it's source code control system may look like this:

Along with it's menu functions and the job title is, in parentheses, a white space delimited list of bazaar modules associated with the job. The exclamation at the start of the module tells Cabie to not fire off a build if it finds changes to those modules. If multiple build jobs share a common directory such as 'rpms' and all output from those jobs write to 'rpms' there would be nothing but continuous builds triggered by check-ins to the rpms module.

There is another way to keep Cabie from triggering builds, an 'exclusion table' used to identify files that should be ignored while Cabie is polling a job for changes. Editing the exclusion table will be covered in [[Cabie Administration]].

Job Expansion

Each job may have multiple associated builds that have been run or completed, the use of the icon displayed on left side of the job menu :

when clicked, expands the display to include a number of builds executed for the defined job. The number of jobs displayed depends on what the value 'expanded jobs' is set to under 'view options'. Here is an example of an expanded display:

In the above expanded view, there are a few different functions available to anyone with access (unauthenticated access). The first is in the left hand column titled 'build id'. Each number displayed in that column represents a unique identifier for each job. Depending on the source code control system, it may be a date, a transaction id, a change number, a unique identifier for each module defined for a job. In the above example, accurev is being used so the build number is the highest transaction in the workspace (after an update) defined for the job, followed by a dot, then the ascending build number. The second is the and will take the user to the build output page. Clicking on the build number results in a display which will show the list of changes that were pulled between the last job and the selected job. The display will look like this:

For the columns 'job', 'user' and 'filename', there are additional hypertext links that go to build output, compose email, and the defined source browser respectively. The source browser in this case, requires an accurev account.

Cabie Server Navigation

To move between Cabie servers, click on the server name displayed in the server selection tabs:

The resulting display will change the highlighted tab to that of selected Cabie server, and the resulting display will include the available, active jobs defined for that server.

Cabie UI Authentication

Under the server selection tabs is the server menu:

Each developer at Move Networks has an account for additional access to Cabie. In order to login for additional access, click on the . Upon selecting the icon, a dialog box will appear which looks like this:

The default password is the same as your example domain login, so if your name is Bob Smith, your example domain login is bsmith, and your Cabie assigned password is bsmith. After logging into the Cabie UI, you may change your password by clicking on the icon. This will bring up a list of all available users with a hypertext link on the user name that was used to log in with, in this case bugs:

By clicking on your user ID after it's been displayed in the user selection list, an editor window will appear:

Unless you have administrative rights, the only fields you will be able to edit is the password and password verification fields. Since some of the login permissions include the ability to make comments about a job, and promote a job, it is highly recommended that you change your password. Once the password has been changed click on the 'Submit' button at the bottom of the form, then click on 'Ok' in the confirmation dialog.

To un-authenticate from the Cabie UI, click in the icon.

Build Job Progress

While a job is running a progress bar will be displayed in the job menu to the right side of the job name:

Along with displaying remaining estimated time for the running job, clicking on the 'percentage complete' will bring up the running build log, which will be refreshed at the same rate set for 'refresh rate' in the 'view options' box:

Everything within the build log will be hypertext linked, and can be examined while the build job is running.

If for some reason anyone wishes to see what steps the build server has executed under the covers, an icon, , has been provided. Clicking on the icon will display the steps executed internally by the build:

This debug log is useful for determining where a job may be hung up...

Viewing Server Processes

Another useful function, this one from the 'server menu' is available by clicking on this icon and results in this view:

The hypertext links in this view allow an administrator to kill processes without having to directly login to the Cabie server. Anyone with less than [[Cabie Administration]] rights will be denied access when trying to kill a process.

Disabling and Enabling Build Jobs

There may be an occasion when a developer is working on a particular job and does not want a job to fire off. This can be for various reasons. There is a disable/enable facility built in to the UI. The icon displayed for an enabled job is , the icon displayed for a disabled job is . Clicking on the 'lock' icon for a particular job will result in the associated job menu looking like this:

Clicking on the 'unlock' icon for a locked job will result in the associated job menu looking like this:

Viewing Job Characteristics

If the need arises for information about job characteristics that would normally only be available to a Cabie administrator, such as the number of builds to be saved before disk reclamation (deletion of old builds), the source server, port, the client or modules being accessed from the source server, along with any additional information defined by the Cabie administrator, the icon can be clicked on resulting in a display similar to this:

Build Job Notification

For the team lead, manager, tester etc. who may be interested in build notification for all builds run for a defined job, there is a build subscription mechanism. To subscribe for build output the user needs to have an account in Cabie with a valid email address and be [[cabie#cabie_ui_authentication|authenticated to the Cabie UI]]. Clicking on the icon will subscribe the authenticated user to all build output for the selected job. Clicking on the icon will stop email notification. The email icons are toggled...

Developers who submit changes that result in triggering a build are automatically notified upon completion of a job. Sequential build failures will result in notifications to all developers going back to the first build failure detected in the job sequence!

The defined Cabie administrator is notified about all builds for all jobs, it is possible to have different administrators defined for each Cabie server!

Build Job Statistics

For simplified build statistics, there's the icon. It's part of the job menu, and when clicked on, will display simplified statistics for the selected job:

Currently only the production jobs have smoketests that report back pass / failure stats for tests. All jobs will display pass / fail for compilation, syntax or generation. Additional statistics are available via command-line utilities installed on the Cabie servers.

Cabie UI Help

For both server and job menus there is help available. The help is context sensitive based upon access level to the Cabie UI. The icon to click on for help is . From the job menu the resulting display will look like this (for any job):

From the server menu the resulting display will look like this (for any server):

Viewing the Last Changes Made to a Build Job

In order to view just the last set of changes that were applied to a job, just click on the icon. If the job uses accurev as it's source code control system, the resulting display may look like so:

Note that hypertext links exist and point off to the transaction id, the issue number and email address of the developer for each file.

If the job uses bazaar as it's source code control system, the resulting display is slightly different:

Hypertext links for bazaar are slightly different. They just include a link to the source browser and email address for the developer who checked in to the tree.

If the Cabie server for the selected job has been rebooted, then the last set of changes may not show up since it's only stored in /tmp. There are other ways to find the last set of changes, this is provided as a matter of convienence

Smoke Test Status

If a smoke test trigger has been defined for a job, it is possible that additional icons may show up on the job menu. The icon would be displayed if the last build executed successfully completed smoke testing, or a icon would be displayed if smoke testing failed. Here's an example of jobs whose most recent builds passed smoke testing:

Build Failure Status

If the icon appears on the job menu, then the last build failed to compile. Clicking on the icon will expand the job display allowing easy access to additional build information as [[cabie#job_expansion|discussed earlier]]. Here's an example of the job menu when the most recent job failed compilation:

Pending Changes

The Cabie UI also has the ability to detect and display pending changes which would result in firing off a build. When this icon is present on the job menu, then there are changes pending for that particular job. When clicked, the resulting display would look similar to this:

Depending on the state of the Cabie server, and the source code control system being used, it may take a while to render the results, just be patient.

Current Unused Functionality

One icon not talked about is . The underlying functions for the 'cog' icon rely on an elaborate test harness with individual test states dynamically defined in a separate and currently unused table. Currently for all defined build jobs on all Cabie servers, clicking on will result in no records being displayed. This may change in the future.

Cabie UI Server Specific Functions

**Where some of these functions have been covered in other sections, hypertext links will be provided. Other functions may require additional permissions.**

Authenticated Cabie UI Server Menu

Upon authenticating to the Cabie UI, the server menu will look like this:

The above menu is for a user who's authentication level is 2. Typically developers will have level 1 authentication, but for team leads, managers, and QA, authentication is set to 2 so that they have the ability to 'promote' a job, and notify a set of developers over a range of builds where errors may have occurred.

Server Home

Clicking on this icon , will always take the user back to the default view of the Cabie server with the highlighted tab. If the default view previously included [[cabie#job_expansion|expanded jobs]], the expansion will be included in the default view.

Server Help

Clicking on the in the Cabie server menu will expand the [[cabie#cabie_ui_help_access|help menu]].

Server Information

By clicking on this icon in the Cabie server menu will display the system information for the server whose tab is highlighted.

Here's system information for the current systems running the Cabie server process:

linuxbuild1.example.com

macbuild1.example.com

macbuild2.example.com

winbuild1.example.com

Another way to retrieve similar information would be to click on the icon. The resulting display will list all known servers along with their request port, run states and information except for the highlighted server:

The server names are hypertext linked back to [[cabie#viewing_server_processes|process information]] without having to click on the icon.

Finding Changed Files

There will be occasion where a user may want to view files changed over a range of builds. This can be accomplished from the Cabie UI server menu by clicking on the icon. When selected it will display all jobs, active or not, that are associated with the highlighted server. By clicking on the job name, a list of all builds executed for that job will automatically be populated:

In the box labeled 'The starting job number to include in the display', select the starting job, that is the lowest build number in the search range.

In the box labeled 'The ending job number to include in the display', select the ending job, that is the one with the highest number in the search range.

In this example, the search will include all files that have changed starting with build 68098.13 all the way through 69572.22. Click on submit, then Ok in the pop-up dialog. The resulting page will look like so:

This above view has been truncated, but would display all changed in the selected range. The columns job, user and file name will all be hypertext linked to build logs, email author, and file browser link respectively.

Searching for Bugs or Issues

The Cabie UI has a feature allowing any authenticated user to search for generated builds by bug or issue numbers. Clicking on the icon in the [[cabie#authenticated_cabie_ui_server_menu|Cabie UI server]] menu will generate a form which looks like this:

Entering 5231 into the text field below the 'Enter Defect ID', clicking on Submit, then Ok in the popup dialog box will search the Cabie database for any issues or defects logged containing 5231 and will result in the following output:

The information produced does not necessarily indicate what build the defect was fixed in, the defect could have been addressed multiple times, but it does indicate which jobs, their build numbers, and the servers the builds were executed on for the searched defect. The hypertext links in the columns 'job' and 'bug number[s]' are linked back to build output and to the appropriate bug tracking system. In the case of accuwork, Cabie directly accesses accuwork to produce a summary view of an issue like so:

For other defect tracking systems such as Jira, a new browser window would be generated with the appropriate links.

Viewing the Cabie Execution Environment

Just like any other executable script, or binary, the Cabie server and scripts launched from build requests require a 'correct' environment. Variables in shell environments and Windows environments need to be set prior to starting up the Cabie server or service. In a POSIX environment, Cabie is launched via a startup script (rc script), or by hand (never as root). An execution environment for starting the Cabie daemon needs to be set. When running Cabie as a Windows service, the start-up of the service requires 'login as user' in order to access things such as network shares. These will be covered in [[Cabie Administration]].

When debugging build errors such as 'command not found', 'no such command or file name', or dynamic library errors etc., examination of the environment Cabie is executing under is invaluable. To view the execution environment click on the icon. The resulting display:

will show what is set in the execution environment allowing further examination of possible execution problems.

Job Retirement and Restoration

Cabie has the ability for an administrator to retire a job from active service. Retiring the job means nothing more than putting the job out of view for normal Cabie operations. Once retired, the job will no longer be polled, triggered, or displayed via the Cabie UI, the generated output for builds associated with the job will not be removed and will remain available as long as they're not physically removed from the disk. The characteristics of the job will remain in SQL in case the job needs to be restored for use once again (think late delivery). Job removal and restoration will be covered in [[Cabie Administration]] so for now the focus will be on viewing retired jobs.

The icon when clicked on, will bring up a list of jobs (alphabetically), which have been retired from use on the highlighted server:

The only hypertext link is the job name, and can only be completed by a Cabie administrator. The general description of the retired job is displayed as well.

Viewing the Cabie Schema

For those interested, the Cabie UI provides a view of the underlying schema it uses. All an authenticated user needs to do is click on the icon from the Cabie UI server menu which renders this view:

All tables are hypertext linked to their description, including their usage. Tables that do not have a description are tables that Cabie does not use internally, but are accessed by other means. Here's a view of Cabie's buildserver table:

The summary displays what the table is used for, also displayed is the field name, the data type for the field and what kind of information the field contains.

Cabie Usage (last edited 2009-02-13 06:11:34 -0700 by ?getsw)