found this section very useful:
Modifying Jenkins configuration from the command line
You may be wondering about the XML files at the top level of the Jenkins workspace. These are configuration files.
config.xml is the main one, dealing with the default server values, but there are also specific ones for any plugins that have values set through the GUI.
There is also a jobs sub-directory underneath the workspace. Each individual Job configuration is contained in a sub-directory with the same name as the Job. The Job-specific configuration is then stored in config.xml within the sub-directory. There is a similar situation for the user's directory with one sub-directory per user, with the user information stored in its own config.xml file.
Under a controlled situation, where all the Jenkins servers in your infrastructure have the same plugins and version levels, it is possible for you to test on one sacrificial machine and then push the configuration files to all the other machines. You can then restart the servers with the
Command-Line Interface (CLI).
This recipe familiarizes you with the main XML configuration structure and provides hints about the plugin API, based on the details of the XML.
You will need a fresh install of Jenkins with security enabled.
- In the top-level directory of Jenkins, look for the file named
config.xml. Go to the line that has the<numExecutor> tag, and edit it by changing the number from 2 to 3, as follows:
- Restart the server. You will see that the number of executors has increased from a default of
2 to 3.
- Look for the file named
thinBackup.xml. You will not find it unless you have installed the thinBackup plugin.
- Replay the recipe Back up and Restoring, and look again. You will now find the following XML file.
Jenkins uses
Xstream (
http://xstream.codehaus.org/) to persist its configuration into a readable XML format. The XML files in the workspace are configuration files for plugins, tasks, and an assortment of other persisted information.
config.xml is the main configuration file. Security settings and global configuration are set here and reflect changes made through the GUI. Plugins use the same structure, and the XML values correspond to member values in underlying plugin classes. The GUI itself is created from XML through the
Jelly framework (
http://commons.apache.org/jelly/).
By restarting the server, you should be certain that any configuration changes are picked up during the initialization phase.
Here are a few things for you to consider.
When you are testing new security features, it is easy to lock yourself out of Jenkins. You will not be able to log in again. To get around this problem, modify
useSecurity to
false in
config.xml, and restart Jenkins. The security features are now turned off.
Finding JavaDoc for custom plugin extensions
The following line of code is the first line of the thin plugin configuration file
thinBackup.xml, mentioning the class from which the information is persisted. The class name is a great Google search term. Plugins can extend the functionality of Jenkins, and there may be useful methods exposed for administrative Groovy scripts.
The effects of adding garbage
Jenkins is great at recognizing rubbish configuration as long as it is recognizable as a valid XML fragment. For example, add the following line of code to
config.xml:
When you reload the configuration, you will see the following error at the top of the manage Jenkins screen:
Pressing the Manage button will return you to a detailed page of the debug information, including the opportunity to reconcile the data.
From this, you can notice that Jenkins is developer-friendly when reading corrupted configuration that it does not understand.
- Using a sacrificial Jenkins instance
- Participating in the community - Maven archetypes and plugins, Chapter 7,Exploring Plugins
Reporting overall disc usage
Organizations have their own way of dealing with increasing disc usage. Policy ranges from no policy, depending on ad-hoc human interactions, to the most state of the art software with central reporting facilities. Most organizations sit between these two extremes with mostly ad-hoc intervention, with some automatic reporting for the more crucial systems. With minimal effort, you can make Jenkins report disc usage from the GUI, and periodically run Groovy scripts that trigger helpful events.
This recipe highlights the disk usage plugin and uses the recipe as a vehicle to discuss the cost of keeping archives stored within the Jenkins workspace.
The disc usage plugin is the strongest in combination with an early warning system that notifies you when soft or hard disc limits are reached. The recipe:
A Job to warn of disc usage violations through log parsing details a solution. Both the recipes show that configuring Jenkins requires little effort. Each step might even seem trivial. The power of Jenkins is that you can build complex responses out of a series of simple steps and scripts.
You will need to install the disc usage plugin.
- Press the Disk usage link under the Manage Jenkins page.
- After clicking on the Disk Usage link, Jenkins displays a page with each project and the builds and Workspace disc usage summary. Click on the top of the table to sort the workspace by file usage.
Adding a plugin in Jenkins is very simple. The question is what are you going to do with the information.
It is easy for you to forget a tick box in a build, perhaps an advanced option is enabled where it should not be. Advanced options can at times be problematic, as they are not displayed directly in the GUI. You will need to hit the advanced button first, before reviewing. On a Friday afternoon, this might be one step too far.
Advanced options include artifact retention choices, which you will need to correctly configure to avoid overwhelming disc usage. In the previous example, the workspace for the
Sakai CLE is
2GB. The size is to do with the Job having its own local Maven repository, as defined by the advanced option Use a private Maven repository. The option is easy for you to miss. In this case, there is nothing to be done, as trunk pulls in snapshot JARs, which might cause instability for other projects.
The simple act of being able to sort disc usage points the offending Jobs out to you, ready for further inspection of their advanced configuration.
If you are keeping a large set of artifacts, it is an indicator of a failure of purpose of your use of Jenkins. Jenkins is the engine that pushes a product through its life cycle. For example, when a job builds snapshots every day, then you should be pushing the snapshots out to where developers find them most useful. That is not Jenkins but a Maven repository or a repository manager such as
Artifactory (
http://www.jfrog.com/products.php),
Apache Archiva (
http://archiva.apache.org/) or
Nexus (
http://nexus.sonatype.org/). These repository managers have significant advantages over dumping to disc. They have the following advantages:
- Speed builds by acting as a cache: Development teams tend to work on similar or the same code. If you build and use the repository manager as a mirror, then the repository manager will cache the dependencies, and when
Job Y asks for the same artifact, the download will be local.
- Acts as a mechanism to share snapshots locally: Perhaps some of your snapshots are only for local consumption. The repository manager has facilities to limit the access.
- GUI interface for ease of artifact management: All the three repository managers have intuitive GUIs, making your management tasks as easy as possible.
With these considerations in mind, if you are seeing a build-up of artifacts in Jenkins, where they are less accessible and beneficial than deployed to a repository, consider this a signal for the need to upgrade your infrastructure.
Retention policy
Jenkins can be a significant consumer of disk space. In the Job configuration, you can decide to either keep artifacts or remove them automatically after a given period of time. The issue with removing artifacts is that you will also remove the results from any automatic testing. Luckily, there is a simple trick for you to avoid this. When configuring a Job, click on Discard Old Builds, and then the Advanced checkbox, define the Max # of builds to keep with the artifacts. The artifacts are then removed after the number of builds specified, but the logs and results are kept. This has one important consequence; you have now allowed the reporting plugins to keep displaying a history of tests even though you have removed the other more disc consuming artifacts.
Deliberately failing builds through log parsing
Scenario: You have been asked to clean up the code removing depreciated Java methods across all the source contained under a Jenkins Jobs; that is a lot of code. If you miss some residue defects, then you will want the Jenkins build to fail.
What you need is a flexible log parser that can fail or warn about issues found in the build output. To the rescue: This recipe describes how you can configure a log parsing plugin that spots unwanted patterns in the console output and fails Jobs.
You will need to install the
Log Parser Plugin as mentioned at:
- Create the
log_rules directory owned by Jenkins, under the Jenkins workspace.
- Add the file named
depreciated.rule to the log_rules directory with one line:
- Create a Job with a source code that gives deprecated warnings on compilation. In the following example, you are using the Roster tool from the
Sakai project:
- Run the build. It should not fail.
- As shown in the next screenshot, visit the Manage configuration page for Jenkins, and to the Console Output section, add a description and location of the parsing rules file that was mentioned in step 2.
- Check the Console output (build log) parsing box in the Post-build Actions section of your Job.
- Check the Mark build Failed on Error checkbox.
- Select Kill Deprecated from the Select Parsing Rules list box.
- Build the Job; it should now fail.
- Click on the Parsed Console Output link in the left-hand menu. You will now be able to see the parsed errors.
The rule file you used is composed of one line: error /DEPRECATED/.
If the pattern DEPRECATED (a case-sensitive test) is found in the console output, then the plugin considers this as an error, and the build fails. More lines to test can be added to the file. The first rule found wins. Other levels include warn and ok.
The rules file has the distinct .rules extension in case you want to write an exclude rule during backups.
Once the plugin is installed, you can choose a Job between the rule files previously created.
This plugin empowers you to periodically scan for the obvious link and adapt to the new circumstances. You should consider sweeping systematically through a series of rule files failing suspect builds, until a full clean-up to in-house style has taken place.
Two other examples of the common log patterns that are an issue, but do not normally fail a build are:
- MD5 check sums: If a Maven repository has an artifact, but not its associated MD5 checksum file, then the build will download the artifact even if it already has a copy. Luckily, the process will leave a warning in the console output.
- Failure to start up custom integration services: These failures might be logged at the
warn or info level when you really want them to fail the build.
- A Job to warn about the disc usage violations through log parsing
A Job to warn about the disc usage violations through log parsing
The disk usage plugin is unlikely to fulfill all of your disc maintenance requirements. This recipe will show how you can strengthen disc monitoring by adding a custom
Perl script to warn about the disc usage violations.
The script will generate two alerts: a hard error when the disc usage is above an acceptable level, and a softer warning when the disc is getting near to that limit. The log parser plugin will then react appropriately.
Using Perl is typical for a Jenkins Job, as Jenkins plays well and adapts to most environments. You can expect Perl, Bash, Ant, Maven, and a full range of scripts and binding code to be used in the battle to get the work done.
If you have not already done so, create a directory owned by Jenkins under the
Jenkins workspace named
log_rules. Also, make sure that the
Perl scripting language is installed on your computer and is accessible by Jenkins.
Perl is installed by default on Linux distributions.
Activestate provides a decent
Perl distribution for MAC and Windows (
http://www.activestate.com/downloads).
- Add a file to the
log_rules directory named disc.rule with the following two lines:
- Visit the Manage configuration page for Jenkins, and add a description as
DISC_USAGE to the Console Output section. Point to the location of the Parsing Rules file.
- Add the following
Perl script to a location of choice named disc_limits.pl, making sure that the Jenkins user can read the file.
- Modify the
$content variable to point to the Jenkins workspace.
- Create a free-style software project Job.
- Under the build section, add the build
Step / Execute Shell. For the command, add perl disc_limits.pl 9000000 2000000.
- Feel free to change the hard and soft limits (
9000000 2000000).
- Check the Console output (build log) parsing in the Post-build Actions section.
- Check the Mark build Unstable on Warning checkbox.
- Check the Mark build Failed on Error checkbox.
- Select DISC_USAGE from the Select Parsing Rules combo box.
- Run the build a number of times.
- Under build history on the left-hand, select the trend link. You can now view trend reports and see a timeline of success and failure.
The
Perl script expects two command-line inputs: hard and soft limits. The hard limit is the value in bytes that the disc utilization under the
$content directory should not exceed. The soft limit is a smaller value in bytes that triggers a warning rather than an error. The warning gives the administrators time to clean up before the hard limit is reached.
The
Perl script transverses the Jenkins workspace and counts the size of all the files. The script calls the method
size_summary for each file or directory underneath the workspace.
If the hard limit is less than the content size, then the script generates the log output [HARD_LIMIT ERROR]. The parsing rules will pick this up and fail the build. If the soft limit is reached, then the script will generate the output [SOFT_LIMIT WARN]. The plugin will spot this due to the rule warn/SOFT_LIMIT/, and then signal a Job warn.
Welcome to the wonderful world of Jenkins. You can now utilize all of the installed features at your disposal. The Job can be scheduled, e-mails can be sent out on failure. You can also tweet, add entries to Google calendar, trigger extra events, for example disc-cleaning builds, and so on. You are mostly limited by your imagination and 21st century technologies.
Keeping in contact with Jenkins through Firefox
If you are a Jenkins administrator, then it is your role to keep an eye on the ebb and flow of the build activity within your infrastructure. Builds can occasionally freeze or break due to non-coding reasons. If a build fails, and this is related to infrastructural issues, then you will need to be warned quickly. Jenkins can do this in numerous ways.
Chapter 5,
Communicating Through Jenkins is dedicated to the different approaches for different audiences. From e-mail, Twitter, and speaking servers, you can choose a wide range of prods, kicks, shouts, and pings. I could even imagine a Google summer of code project with a remotely controlled buggy moving to the sleeping administrator and then toting.
This recipe is one of the more pleasant ways for you to be reached. You will pull in the Jenkins RSS feeds using a Firefox add-on. This allows you to view the build process, while going about your everyday business.
You will need Firefox 5 or later installed on your computer and an account on at least one Jenkins instance, with a history of running Jobs.
A plug for the developers
If you like the add-on and want more features in the future, then it is enlightened in the self-interest to donate a few bucks at the add-on author's website.
- Select the Firefox tab at the top-left hand side of the browser.
- In the Search box (top-right) with the title Search all add-ons, search for Jenkins.
- Click on the Install button for the Jenkins Build monitor.
- Restart Firefox.
- Select the Firefox tab at the top left-hand side of the browser.
- Enable the Add-On Bar by selecting Options, and then Add-On Bar. Now, at the bottom right-hand side of Firefox, you will see a small Jenkins icon.
- Right-click on the icon.
- Select the preferences, and the Feeds screen appears.
- Add a recognizable, but short, name for your Jenkins instance. For example,
Plugin test server.
- Add a URL using the following structure for Feed URL:
http://host:port/rssAll e.g.: http://localhost:8080/rssAll.
- Check Enable executor monitoring.
- Click on the OK button. An area in the Add-On tool bar will appear with the name Plugin test Server of the Feed URL(s) displayed, and a health icon. If you hover your mouse over the name, then a more detailed status information will be displayed.
Jenkins provides RSS feeds to make its status information accessible to a wide variety of tools. The Firefox add-on polls the configured feed and displays the information in a digestible format.
To configure for a specific crucial Job, you will need to use the following structure: http://host:port/job/job name/rssAll.
To view only the build failures, replace rssAll with rssFailed. To view only the last build, replace rssAll with rssLatest.
If security is enabled on your Jenkins instances, then most of your RSS feeds will be password-protected. To add a password, you will need to modify the Feed URL to the following structure:
http://username:password@host:port/path
Warning
The negative aspect of using this add-on is that any Feed URL password is displayed in plain text during editing.
- Visualizing schedules the — Google calendar
- Shouting at developers through Twitter
Monitoring through JavaMelody
JavaMelody (
http://code.google.com/p/javamelody/) is an open source project that provides comprehensive monitoring. The Jenkins plugin monitors both the
Master instance of Jenkins and also its
nodes. The plugin provides a detailed wealth of the important information. You can view the evolution charts ranging from a day or weeks to months of the main quantities, such as the CPU or the memory.
Evolution charts are very good at pinpointing the scheduled Jobs that are resource-hungry. JavaMelody allows you to keep a pulse on the incremental degradation of resources. It eases the writing of reports by exporting statistics in a PDF format. Containing over 25 years of human effort, JavaMelody is feature-rich.
Community partnership
If you find this plugin useful, consider contributing back to either the plugin or the core JavaMelody project.
You will need to have installed the JavaMelody plugin.
- Click on the Monitoring Hudson/Jenkins master link on the Manage Jenkins page. You will now see the detailed monitoring information.
- Read the online help at the URL
http://host:port/monitoring?resource=help/help.html, where the host and port point to your server.
- Review the monitoring of the node processes directly, by visiting
http://host:port/monitoring/nodes.
JavaMelody has the advantage of running as the Jenkins user and can gain access to all the relevant metrics. Its main disadvantage is that it runs as part of the server and will stop monitoring as soon as there is a failure. Because of this disadvantage, you should consider JavaMelody as part of the monitoring solution and not the whole.
Monitoring is the foundation for comprehensive testing and troubleshooting. This section explores the relationship between these issues and the measurements exposed in the plugin.
Troubleshooting with JavaMelody - memory
Your Jenkins server can at times have memory issues due to greedy builds, leaky plugins, or some hidden complexity in the infrastructure.
JavaMelody has a comprehensive range of memory measurements, including a heap dump and a memory histogram.
The Java virtual machine divides the memory into various areas, and to clean up, it removes objects that have no references to other objects. Garbage collection can be CPU-intensive when it is busy, and the nearer you get to full memory, the busier the garbage collection becomes. To an external monitoring agent ,this looks like a CPU spike that is often difficult to track down. Just because the garbage collector manages memory, it is also a fallacy to believe that there is no potential for memory leakage in Java. Memory can be held too long by many common practices, such as custom caches or calls to native libraries.
Slow-burning memory leaks will show up as gentle slopes on the memory-related evolution graphs. If you suspect that you have a memory leak, then you can get the plugin to force a full garbage collection through the link Execute the garbage collector. If it is not a memory leak, then the gentle slope will abruptly fall.
Memory issues can also express themselves as large CPU spikes as the garbage collector frantically tries to clean up, but can barely clean enough space. The garbage collector can also pause the application while comprehensively looking for no longer referenced objects, and cause large response times for web browser requests. This can be seen through the mean and max times under the Statistics labeled http - 1 day.
Troubleshooting with JavaMelody - painful Jobs
You should consider the following points:
- Offload work: For a stable infrastructure, offload as much work as possible from the master instance. If you have scheduled the tasks, keep the heaviest ones separate in time. Time separation not only evens out load, but also makes finding the problematic build easier through the observation of the evolution charts of JavaMelody. Also consider spatial separation; if a given node or a labeled set of nodes show problematic issues, then start switching around machine location of Jobs, and view their individual performance characteristics through
http://host:port/monitoring/nodes.
- Hardware is cheap: Compared to paying for human hours, buying an extra 8GB is cheap.
A common gotcha is to add the memory to the server, but forget to update the init scripts to allow Jenkins to use more memory.
- Review the build scripts: Javadoc generation, custom Ant scripts can fork JVMs, and reserve memory are defined within their own configuration. Programming errors can also be the cause of the frustration. Don't forget to review JavaMelody's report on the Statistic system error log and Statistic http system errors.
- Don't forget external factors: Factors include backups, cron Jobs, updating the locate database, and network maintenance. These will show up as periodic patterns in the evolution charts.
- Strength in numbers: Use the JavaMelody in combination with the disk usage plugin and others to keep a comprehensive overview of the vital statistics. Each plugin is simple to configure, but their usefulness to you will grow quicker than the maintenance costs of adding extra plugins.
- Running a script to obtain the monitoring information
Keeping a track of the script glue
There are negative implications for backing up and especially restoring if maintenance scripts are scattered across the infrastructure. It is better to keep your scripts in one place, and then run them remotely through the nodes. Consider placing your scripts under the Master Jenkins home directory. It would be even better for the community if you can share the less-sensitive scripts online. Your organization can reap the benefits; the scripts will then get some significant peer review and improvements.
In this recipe, we explore the use of the
Scriptler plugin to manage your scripts locally and download useful scripts from an online catalog.
- Click on the Scriptler link under the Manage Jenkins page. You will notice the text in bold: Currently you do not have any scripts available, you can import scripts from a remote catalog or create your own.
- Click on the link on the left-hand side of Remote Script catalogs.
- Click on the icon of the floppy disk for
getThreadDump. If the script is not available, then choose another script of your choice.
- You have now returned to the Scriptler main page. You will see three icons. Choose the furthest right to execute the script.
- You are now in the Run a script page. Select a node and hit the Run button.
If the script fails with a message startup failed, then please add a new line between entry.key and for, and the script will then function correctly.
- To write a new Groovy script or to upload the one that you have on your local system, click on the Add a new Script link on the left-hand side.
This plugin allows you to easily manage your Groovy scripts, and enforces a standard place for all Jenkins administrators to keep their code, making it easier for you to plan backups and indirectly share knowledge.
The plugin creates a directory named scriptler under the Jenkins workspace and persists the meta information about the files that you have created in the scriptler.xml file. A second file, scriptlerweb-catalog.xml, mentions the list of online files that you can download.
All the local scripts are contained in the sub-directory scripts.
If enough people use this plugin, then the list of online scripts will radically increase the process of generating a significant library of reusable code. Therefore, if you have interesting Groovy scripts, then upload them. You will need to create a new account the first time to log in at:
http://scriptlerweb.appspot.com/login.gtpl.
Uploading your scripts allows people to vote on them and to send you feedback. The free peer review can only improve your scripting skills and increase your recognition in a wider community.
- Scripting the Jenkins command-line interface
- Global modifications of Jobs with Groovy
- Scripting the global build reports
Scripting the Jenkins command-line interface
The Jenkins
Command-Line Interface (CLI),
https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+CLI, allows you to perform a number of maintenance tasks on remote servers. Tasks include moving the Jenkins instances on and offline, triggering builds and running Groovy scripts. This makes for easy scripting of the most common chores.
In this recipe, you will log on to a Jenkins instance and run a Groovy script that looks for files greater than a certain size, and log off. The script represents a typical maintenance task. You can imagine chaining a second script to the first, to remove the large files found.
Download the CLI JAR file from http://host/jnlpJars/jenkins-cli.jar.
Add the following script to a directory under the control of Jenkins and call it large_files.groovy.
- Run the next command from a terminal, replacing
http://host with the real address of your server, for examplehttp://localhost:8080.
- Input your password.
- Look at the online help:
- Run the Groovy script. The command-line output will now mention all the oversize files.
- Log out.
The CLI allows you to work from the command line and perform standard tasks. Wrapping the CLI in a shell script, such as bash, allows you to script maintenance tasks a large number of Jenkins instances at the same time. This recipe performs a lot of drudgework. In this case, it reviews X thousand files for oversized artifacts, saving you time that you can better spend on more interesting tasks.
Before performing any commands, you need to first authenticate through the login command.
Reviewing the script root jenkins.model.Jenkins.instance.getRootDir() uses the Jenkins framework to obtain a java.io.File file, which points to the Jenkins workspace.
The maximum file size is set to 32MB through maxsize=1024*1024*32.
The script visits every file under the Jenkins workspace, using the standard Groovy method root.eachFileRecurse(){ file ->.
You can find the current JavaDoc for Jenkins at:
The authentication used in this recipe can be improved. You can add your SSH public key under
http://localhost:8080/user/{username}/configure (where
username is your username), by cutting and pasting into the SSH Public Keys section. You can find detailed instructions at:
https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+CLI.
The CLI is easily-extendable, and therefore over time, the CLI's command list increases. It is therefore important that you occasionally check the in-built help.
- Global modifications of Jobs with Groovy
- Scripting global build reports
Global modifications of Jobs with Groovy
Jenkins is not only a continuous integration server but also a rich framework with an exposed internal structure available from within the script console. You can programmatically iterate through the Jobs, plugins, node configuration, and a variety of rich objects. As the number of Jobs increase, you will notice that scripting becomes more valuable. For example, imagine that you need to increase custom memory settings across 100 Jobs. A Groovy script can do that in seconds.
This recipe is a representative example: You will run a script that iterates through all Jobs. The script then finds one specific Job by its name, and then updates the description of that Job with a random number.
Log in to Jenkins with an administrative account.
- Create an empty Job named
MyTest.
- Within the Manage Jenkins page, click on the Script console link.
- Cut and paste the following script into the text area input.
- Click on the run button. The results should be similar to the following screenshot:
- Run the script a second time, and you will notice that the random number in the description has now changed.
- Copy and run the following script:
- If you have no slave instances on your Jenkins master, then no results are returned. Otherwise, the output will look similar to the following screenshot:
- Scripting the Jenkins command-line interface
- Scripting the global build reports
Signaling the need to archive
Each development team is unique. Teams have their own way of doing business. In many organizations, there are one-off tasks that need to be done periodically, for example at the end of each year.
This recipe details a script that checks for the last successful run of any Job, and if the year is different to the current year, then a warning is set at the beginning of the Jobs description. Thus, hinting to you it is time to perform some action, such as archiving and then deleting. You can, of course, programmatically do the archiving. However, for high value actions, it is worth forcing interceding, letting the Groovy scripts focus your attention.
Log in to Jenkins with an administrative account.
Within the Manage Jenkins page, click on the Script console link, and run the following script:
Any project that had its last successful build in another year than this will have the word
[ARCHIVE] in red, added at the start of its description.
Reviewing the code listing:
A warning string is defined, and the current date is stored in now. Each Job in Jenkins is programmatically iterated through the for statement.
Jenkins has a class to store information about the running of builds. The runtime information is retrieved through
job.getLastSuccessfulBuild(), and is stored in the
lastSuccessfulBuild instance. If no successful build has occurred, then
lastSuccessfulBuild is set to
null, otherwise it has the runtime information.
The time of the last successful build is retrieved, and then stored in the time instance throughlastSuccessfulBuild.getTimestamp().getTime().
The current year is compared with the year of the last successful build, and if they are different and the warning string has not already been added to the front of the Job description, then the description is updated.
If you find any defects while reviewing the plugin code base, please contribute to the community through patches and bug reports.
- Scripting the Jenkins command-line interface
- Global modifications of Jobs with Groovy
