Dawid Weiss added a comment - 30/May/12 21:22
Yeah, but what is ${FAILED_TESTS}? I mean – can we edit it to detect "<<< FAILURES!" and report 100 lines before this string?
Steven Rowe added a comment - 30/May/12 21:47
Yeah, but what is ${FAILED_TESTS}? I mean – can we edit it to detect "<<< FAILURES!" and report 100 lines before this string?
Jenkins's Content Token Reference
All arguments are optional. Arguments may be given for each token in the form name="value" for strings and in the form name=value for booleans and numbers. The {'s and }'s may be omitted if there are no arguments.
Examples: $TOKEN, ${TOKEN}, ${TOKEN, count=100}, ${ENV, var="PATH"}
Available Tokens
- ${DEFAULT_SUBJECT} - This is the default email subject that is configured in Jenkins's system configuration page.
- ${DEFAULT_CONTENT} - This is the default email content that is configured in Jenkins's system configuration page.
- ${PROJECT_DEFAULT_SUBJECT} - This is the default email subject for this project. The result of using this token in the advanced configuration is what is in the Default Subject field above. WARNING: Do not use this token in the Default Subject or Content fields. Doing this has an undefined result.
- ${PROJECT_DEFAULT_CONTENT} - This is the default email content for this project. The result of using this token in the advanced configuration is what is in the Default Content field above. WARNING: Do not use this token in the Default Subject or Content fields. Doing this has an undefined result.
- ${BUILD_LOG, maxLines, escapeHtml} - Displays the end of the build log.
- maxLines - display at most this many lines of the log.
Defaults to 250. - escapeHtml - If true, HTML is escaped.
Defaults to false.
- maxLines - display at most this many lines of the log.
- ${BUILD_LOG_REGEX, regex, linesBefore, linesAfter, maxMatches, showTruncatedLines, substText, escapeHtml, matchedLineHtmlStyle} - Displays lines from the build log that match the regular expression.
- regex - Lines that match this regular expression are included. See also java.util.regex.Pattern
Defaults to "(?i)\b(error|exception|fatal|fail(ed|ure)|un(defined|resolved))\b". - linesBefore - The number of lines to include before the matching line. Lines that overlap with another match or linesAfter are only included once.
Defaults to 0. - linesAfter - The number of lines to include after the matching line. Lines that overlap with another match or linesBefore are only included once.
Defaults to 0. - maxMatches - The maximum number of matches to include. If 0, all matches will be included.
Defaults to 0. - showTruncatedLines - If true, include ...truncated ### lines... lines.
Defaults to true. - substText - If non-null, insert this text into the email rather than the entire line.
Defaults to null. - escapeHtml - If true, escape HTML.
Defaults to false. - matchedLineHtmlStyle - If non-null, output HTML. matched lines will become <b style="your-style-value">html escaped matched line</b>.
Defaults to null.
- regex - Lines that match this regular expression are included. See also java.util.regex.Pattern
- ${BUILD_LOG_EXCERPT, start, end} - Displays an excerpt from the build log.
- start - Regular expression to match the excerpt starting line to be included (exluded). See java.util.regex.Pattern
- end - Regular expression to match the excerpt ending line to be included (exluded)
- ${BUILD_NUMBER} - Displays the number of the current build.
- ${BUILD_STATUS} - Displays the status of the current build. (failing, success, etc...)
- ${BUILD_URL} - Displays the URL to the current build.
- ${CHANGES, showPaths, showDependencies, format, pathFormat} - Displays the changes since the last build.
- showDependencies - if true, changes to projects this build depends on are shown.
Defaults to false. - showPaths - if true, the paths modified by a commit are shown.
Defaults to false. - format - for each commit listed, a string containing %X, where %X is one of %a for author, %d for date, %m for message, %p for paths, or %r for revision. Not all revision systems support %d and %r. If specified, showPaths is ignored.
Defaults to "[%a] %m\n". - pathFormat - a string containing %p to indicate how to print paths.
Defaults to "\t%p\n".
- showDependencies - if true, changes to projects this build depends on are shown.
- ${CHANGES_SINCE_LAST_SUCCESS, reverse, format, showPaths, changesFormat, pathFormat} - Displays the changes since the last successful build.
<ul>- reverse - indicates that most recent builds should be at the top.
Defaults to false. - format - for each build listed, a string containing %X, where %X is one of %c for changes, or %n for build number.
Defaults to "Changes for Build #%n\n%c\n". - showPaths, changesFormat, pathFormat - defined as showPaths, format, and pathFormat from ${CHANGES}, respectively.
- reverse - indicates that most recent builds should be at the top.
- ${CHANGES_SINCE_LAST_UNSTABLE, reverse, format, showPaths, changesFormat, pathFormat} - Displays the changes since the last unstable or successful build.
- reverse - indicates that most recent builds should be at the top.
Defaults to false. - format - for each build listed, a string containing %X, where %X is one of %c for changes, or %n for build number.
Defaults to "Changes for Build #%n\n%c\n". - showPaths, changesFormat, pathFormat - defined as showPaths, format, and pathFormat from ${CHANGES}, respectively.
- reverse - indicates that most recent builds should be at the top.
- ${ENV, var} - Displays an environment variable.
- var - the name of the environment variable to display. If "", show all.
Defaults to "".
- var - the name of the environment variable to display. If "", show all.
- ${FAILED_TESTS, showStack, maxTests} - Displays failing unit test information, if any tests have failed.
- showStack - indicates that most recent builds should be at the top.
Defaults to true. - maxTests - display at most this many failing tests.
No limit is set by default.
- showStack - indicates that most recent builds should be at the top.
- ${JENKINS_URL} - Displays the URL to the Jenkins server. (You can change this on the system configuration page.)
- ${HUDSON_URL} - deprecated, please use $JENKINS_URL
- ${PROJECT_NAME} - Displays the project's name.
- ${PROJECT_URL} - Displays a URL to the project's page.
- ${SVN_REVISION} - Displays the subversion revision number.
- ${CAUSE} - Displays the cause of the build.
- ${JELLY_SCRIPT, template} - Custom message content generated from a Jelly script template. There are two templates provided: "html" and "text". Custom Jelly templates should be placed in $JENKINS_HOME/email-templates. When using custom templates, the template filename without ".jelly" should be used for the "template" argument.
- template - the template name.
Defaults to "html".
- template - the template name.
- ${FILE, path} - Includes the content of a specified file.
- path - The path to the file. Relative to the workspace root.
- ${TEST_COUNTS, var} - Displays the number of tests.
- var - Defaults to "total".
- total - the number of all tests.
- fail - the number of failed tests.
- skip - the number of skipped tests.
- var - Defaults to "total".
- ${SCRIPT, script, template, init} - Custom message content generated from a script using JSR 223. Custom scripts should be placed in $JENKINS_HOME/email-templates. When using custom scripts, the script filename WITH .py/.rb/etc should be used for the "script" argument.
templates and other items may be loaded using the host.readFile(String fileName) function
the function will look in the resources for the email-ext plugin first, and then in the $JENKINS_HOME/email-templates directory. No other directories will be searched.- script - the script name.
Defaults to "email-ext.groovy". - template - the template filename.
Defaults to "groovy-html.template" - init - true to run the language's init script.
Defaults to true - Available Script Engines
- ECMAScript - 1.8 (js)
- Groovy - 1.8.5 (groovy)
- script - the script name.
Dawid Weiss added a comment - 30/May/12 21:59
Ok, so BUILD_LOG_REGEX should do! I don't know Jenkins but any failed test (suite) will have the "<<< FAILURES!" marker attached – feel free to experiment 
Steven Rowe added a comment - 30/May/12 22:13
I'll switch it to the following - hopefully it will capture everything (any length multiline) between the suite header and "<<< FAILURES!"
Build: ${BUILD_URL}
${FAILED_TESTS}
Build Log (for compile errors):
${BUILD_LOG_REGEX,regex="(?s:\\[java4\\]\\s*Suite:.*?<<<\\s*FAILURES!)"}
Steven Rowe added a comment - 30/May/12 22:19
regex="(?s:\\[java4\\]\\s*Suite:.*?<<<
s*FAILURES!)"
Hmm, that won't work - it'll grab everything from the first "Suite:" to "<<< FAILURES!", including any number of (non-failing) Suite mentions inbetween.
I guess "<<< FAILURES!" and 100 previous lines will work, but I'd rather get the exact region. I'll work on it.
Robert Muir added a comment - 30/May/12 22:33
Is there also a way we could improve the output of other checks (e.g. the javadocs warnings task, two javadocs checkers in javadocs-lint, and the rat-checker)
so that if it causes a build failure its included as well, rather than just "No tests ran" or "All tests passed" or whatever it does today?
so that if it causes a build failure its included as well, rather than just "No tests ran" or "All tests passed" or whatever it does today?
Robert Muir added a comment - 30/May/12 22:34
By the way: I'm not suggesting to make the regex even more hairy, i'm just wondering if we can modify these tasks
so that when they fail, they can include certain symbols/stuff to ensure they are included in the summary...
so that when they fail, they can include certain symbols/stuff to ensure they are included in the summary...
Steven Rowe added a comment - 30/May/12 22:54
This one seems to work (from Perl against a recent Jenkins log with a failure):
[^\r\n]*\[junit4\]\s*Suite:.*[\r\n]+[^\r\n]*\[junit4\]\s*(?!Completed)(?!IGNOR)\S(?s:.*?)<<<\s*FAILURES!
I'll change the Lucene-Solr-tests-only-trunk job configuration to use it (after escaping backslashes).
Steven Rowe added a comment - 30/May/12 23:04
Is there also a way we could improve the output of other checks (e.g. the javadocs warnings task, two javadocs checkers in javadocs-lint, and the rat-checker)
so that if it causes a build failure its included as well, rather than just "No tests ran" or "All tests passed" or whatever it does today?
It may be that alternates can be added to the BUILD_LOG_REGEX - I'll look at the output and see if any changes need to be made to enable that.
Steven Rowe added a comment - 31/May/12 18:00
I plan on adding the following (as suggested by Robert) as alternations to the BUILD_LOG_REGEX for all non-Maven Jenkins jobs (some of these things don't run under the Maven jobs, and Maven's output is different enough that it'll require separate treatment):
the javadocs warnings task
(?:[^\r\n]*\[javadoc\].*\r?\n)*[^\r\n]*\[javadoc\]\s*[1-9]\d*\s+warnings.*\r?\n
two javadocs checkers in javadocs-lint
Output from javadocs-lint seems to show up only when there's a problem, so any output from it will always be extracted by the following regex:
[^\r\n]*javadocs-lint:.*\r?\n(?:[^\r\n]*\[echo\].*\r?\n)*
and the rat-checker
[^\r\n]*rat-sources:\s+\[echo\].*(?:\r?\n[^\r\n]*\[echo\].*)*\s*[1-9]\d*\s+Unknown\s+Licenses.*\r?\n(?:[^\r\n]*\[echo\].*\r?\n)*
Along with two others:
- Compilation failures:
(?:[^\r\n]*\[javac\].*\r?\n)*[^\r\n]*\[javac\]\s*[1-9]\d*\s*error.*\r?\n
- Jenkins FATAL errors:
[^\r\n]*FATAL:(?s:.*)
Steven Rowe added a comment - 31/May/12 18:18
I plan on adding the following (as suggested by Robert) as alternations to the BUILD_LOG_REGEX for all non-Maven Jenkins jobs
Done. Here's the full Editable Email Notification/Default Content configuration:
Build: ${BUILD_URL}
${FAILED_TESTS}
Build Log:
${BUILD_LOG_REGEX,regex="(?x:
# Compilation failures
(?:[^\\r\\n]*\\[javac\\].*\\r?\\n)*[^\\r\\n]*\\[javac\\]\\s*[1-9]\\d*\\s*error.*\\r?\\n
# Test failures
|[^\\r\\n]*\\[junit4\\]\\s*Suite:.*[\\r\\n]+[^\\r\\n]*\\[junit4\\]\\s*(?!Completed)(?!IGNOR)\\S(?s:.*?)<<<\\s*FAILURES!
# License problems
|[^\\r\\n]*rat-sources:\\s+\\[echo\\].*(?:\\r?\\n[^\\r\\n]*\\[echo\\].*)*\\s*[1-9]\\d*\\s+Unknown\\s+Licenses.*\\r?\\n(?:[^\\r\\n]*\\[echo\\].*\\r?\\n)*
# Javadocs warnings
|(?:[^\\r\\n]*\\[javadoc\\].*\\r?\\n)*[^\\r\\n]*\\[javadoc\\]\\s*[1-9]\\d*\\s+warnings.*\\r?\\n
# Other javadocs problems (broken links and missing javadocs)
|[^\\r\\n]*javadocs-lint:.*\\r?\\n(?:[^\\r\\n]*\\[echo\\].*\\r?\\n)*
# Jenkins problems
|[^\\r\\n]*FATAL:(?s:.*)
)"}
Steven Rowe added a comment - 31/May/12 18:44
I'm going to add one more to the regex:
# Third-party dependency license/notice problems |[^\\r\\n]*validate:.*\\r?\\n[^\\r\\n]*\\[echo\\].*\\r?\\n(?:[^\\r\\n]*\\[licenses\\].*\\r?\\n)*[^\\r\\n]*\\[licenses\\].*[1-9]\\d*\\s+error.*\\r?\\n
Steven Rowe added a comment - 31/May/12 18:48
I'm going to add one more to the regex
Done - added to the configuration on all non-Maven Jenkins jobs
Dawid Weiss added a comment - 31/May/12 21:09
Thanks for working on this, Steve. It'll really be useful.
Steven Rowe added a comment - 31/May/12 23:08
Two problems:
- Spreading the BUILD_LOG_REGEX regex value over multiple lines is not supported by Jenkins's email templating functionality, which is provided by the Jenkins Email Extension Plugin (email-ext) https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin. See the configuration token parsing regexes in ContentBuilder.Tokenizer, in particular the comment over thestringRegex field:
// Sequence of (1) not \ " CR LF and (2) \ followed by non line terminator private static final String stringRegex = "\"([^\\\\\"\\r\\n]|(\\\\.))*\"";
This could be fixed by allowing line terminators to be escaped:// Sequence of (1) not \ " CR LF and (2) \ followed by any non-CR/LF character or (CR)LF private static final String stringRegex = "\"([^\\\\\"\\r\\n]|(\\\\(?:.|\r?\n)))*\"";
I submitted a Jenkins JIRA issue for this: https://issues.jenkins-ci.org/browse/JENKINS-13976. - BuildLogRegexContent, the content parser for BUILD_LOG_REGEX, matches line-by-line, so regexes targeting multiple lines will fail. I can see two possible routes to address this:
- The BUILD_LOG_EXCERPT token allows specification of begin/end line regexes, and includes everything inbetween matches. I'm doubtful this will enable capture of the stuff we want, though.
- I'll try to add an argument to BUILD_LOG_REGEX to enable multi-line content matching, and make a pull request/JIRA issue to get it included in the next release of the plugin.
In the mean time, I'll switch the configuration in our Jenkins jobs to the following:
Build: ${BUILD_URL}
${FAILED_TESTS}
Build Log:
${BUILD_LOG_REGEX,regex="[ \\t]*(?:\\[javac\\]\\s+[1-9]\\d*\\s+error|\\[junit4\\].*<<<\\s+FAILURES!|\\[javadoc\\]\\s+[1-9]\\d*\\s+warning).*",linesBefore=100}
${BUILD_LOG_REGEX,regex="[ \\t]*\\[echo\\].*)*\\s*[1-9]\\d*\\s+Unknown\\s+Licenses.*",linesBefore=17,linesAfter=20}
${BUILD_LOG_REGEX,regex="[ \\t]*javadocs-lint:.*",linesBefore=0,linesAfter=75}
${BUILD_LOG_REGEX,regex=".*FATAL:.*",linesBefore=0,linesAfter=100}
Steven Rowe added a comment - 01/Jun/12 03:02
I'll switch the configuration in our Jenkins jobs to the following ...
Done.
Steven Rowe added a comment - 01/Jun/12 20:50
This one had syntax problems (a recent test failure notification email complained about it):
${BUILD_LOG_REGEX,regex="[ \\t]*\\[echo\\].*)*\\s*[1-9]\\d*\\s+Unknown\\s+Licenses.*",linesBefore=17,linesAfter=20}
I switched it to the following on all non-Maven Jenkins job configuratinos:
${BUILD_LOG_REGEX,regex="[ \\t]*\\[echo\\]\\s+[1-9]\\d*\\s+Unknown\\s+Licenses.*",linesBefore=17,linesAfter=20}
Spreading the BUILD_LOG_REGEX regex value over multiple lines is not supported by Jenkins's email templating functionality
[...]
This could be fixed by allowing line terminators to be escaped:
[...]
I submitted a Jenkins JIRA issue for this: https://issues.jenkins-ci.org/browse/JENKINS-13976.
I forked the email-ext project on github and made a pull request, which has now been incorporated into the upstream project.
BuildLogRegexContent, the content parser for BUILD_LOG_REGEX, matches line-by-line, so regexes targeting multiple lines will fail. [...] I'll try to add an argument to BUILD_LOG_REGEX to enable multi-line content matching, and make a pull request/JIRA issue to get it included in the next release of the plugin.
I made a new content token named BUILD_LOG_MULTILINE_REGEX in my fork of the email-ext-plugin - see JENKINS-14000. My pull request (#40) was merged into the upstream project a few days ago, so the next release will include this change. (Looks like this plugin releases once a month or so, so the wait before we can use it shouldn't be too long, assuming the plugins on the ASF Jenkins instance are kept up-to-date.)
Dawid Weiss added a comment - 07/Jun/12 13:47
Assigning to you, Steven, since you do the heavy lifting here anyway. Thanks!
| Field | Original Value | New Value |
|---|---|---|
| Assignee | Steven Rowe [ steve_rowe ] |
Steven Rowe added a comment - 16/Jun/12 16:39
v2.22 of the Jenkins Email-ext plugin was released today, incorporating the new BUILD_LOG_MULTILINE_REGEX content token functionality. Olivier Lamy kindly upgraded the plugin in ASF Jenkins and restarted it.
I'm going to re-try using the full-log multiline regex I posted in comment-13286757, with modification I've since added to Lucene jobs' configurations (using multiple BUILD_LOG_REGEX content tokens, one per context window a.k.a. linesBefore/linesAfter).
Hopefully now we'll be able to get exactly the right snippet from the logs for each failure case.
Here is the Editable Email Notification/Default Content configuration I applied to all non-Maven ASF Jenkins jobs (modulo whitespace in front of the trailing line-continuation backslashes):
edit: BUILD_LOG_REGEX -> BUILD_LOG_MULTILINE_REGEX
Build: ${BUILD_URL}
${FAILED_TESTS}
Build Log:
${BUILD_LOG_MULTILINE_REGEX,regex="(?x: \
# Compilation failures \
(?:.*\\[javac\\].*\\r?\\n)*.*\\[javac\\]\\s+[1-9]\\d*\\s+error.*\\r?\\n \
# Test failures \
|.*\\[junit4\\]\\s*Suite:.*[\\r\\n]+.*\\[junit4\\]\\s*(?!Completed)(?!IGNOR)\\S(?s:.*?)<<<\\s*FAILURES! \
# Source file license problems \
|.*rat-sources:.*(?:\\r?\\n.*\\[echo\\].*)*\\s+[1-9]\\d*\\s+Unknown\\s+Licenses.*\\r?\\n(?:.*\\[echo\\].*\\r?\\n)* \
# Third-party dependency license problems - include 2 preceding lines and 1 following line \
|(?:.*\\r?\\n){2}.*\\[licenses\\]\\s+MISSING\\s+sha1(?:.*\\r?\\n){2} \
# Javadoc warnings \
|(?:.*\\[javadoc\\].*\\r?\\n)*.*\\[javadoc\\]\\s*[1-9]\\d*\\s+warnings.*\\r?\\n \
# Other javadocs problems: broken links and missing javadocs \
|.*javadocs-lint:.*\\r?\\n(?:.*\\[echo\\].*\\r?\\n)* \
# Thread dumps - include 1 preceding line and the remainder of the log \
|.*\\r?\\n.*Full thread dump(?s:.*) \
# Jenkins problems - include the remainder of the log \
|.*(?:FATAL|ERROR):(?s:.*) \
# Include the Ant call stack - include the remainder of the log \
|.*BUILD\\s+FAILED(?s:.*) \
)"}
Steven Rowe added a comment - 16/Jun/12 18:14
${BUILD_LOG_REGEX,regex="(?x: [...]
Ack! I forgot to switch the previous version to use the new BUILD_LOG_MULTILINE_REGEX content token!
Fixed now...
| Parent | ||
| Issue Type | Sub-task [ 7 ] | Task [ 3 ] |
Steven Rowe added a comment - 30/Jul/12 13:34
A bug effectively prevented use of the new BUILD_LOG_MULTILINE_REGEX config: JENKINS-14132. The fix on that issue has been incorporated in v2.24.1 of the Email-ext plugin, and Olivier Lamy has kindly installed this version on ASF Jenkins. Uwe helped work out the kinks, and has been running the patched version for a while on his Jenkins instance - upgraded to the release version now.
I will close as fixed later today once it's clear the ASF configuration is functional.
Steven Rowe added a comment - 30/Jul/12 13:41
Here's the current configuration regex (edited so that the trailing backslashes are lined up):
Build Log:
${BUILD_LOG_MULTILINE_REGEX,regex="(?x: \
\
(?:.*\\[javac\\]\\s++(?![1-9]\\d*\\s+error).*\\r?\\n)*+.*\\[javac\\]\\s+[1-9]\\d*\\s+error.*\\r?\\n \
\
|.*\\[(?:junit4:)?junit4\\]\\s+Suite:.*+\\s++.*\\[(?:junit4:)?junit4\\]\\s++(?!Completed)(?!IGNOR).*\\r?\\n \
(?:.*\\[(?:junit4:)?junit4\\]\\s++(?!Completed\\s+.*<<<\\s*FAILURES!).*\\r?\\n)*+ \
.*\\[(?:junit4:)?junit4\\]\\s++Completed\\s+.*<<<\\s*FAILURES!\\r?\\n \
\
|.*rat-sources:.*\\r?\\n \
(?:\\s*+\\[echo\\]\\s*\\r?\\n|\\s*+\\[echo\\]\\s++(?![1-9]\\d*\\s+Unknown\\s+License)\\S.*\\r?\\n)*+ \
\\s*+\\[echo\\]\\s+[1-9]\\d*\\s+Unknown\\s+License.*\\r?\\n \
(?:\\s*+\\[echo\\].*\\r?\\n)*+ \
\
|(?:.*\\r?\\n){2}.*\\[licenses\\]\\s+MISSING\\s+sha1(?:.*\\r?\\n){2} \
\
|.*check-licenses:.*\\r?\\n\\s*\\[echo\\].*\\r?\\n\\s*\\[licenses\\]\\s+MISSING\\s+LICENSE.*\\r?\\n \
(?:\\s*+\\[licenses\\].*\\r?\\n)++ \
\
|(?:.*\\[javadoc\\]\\s++(?![1-9]\\d*\\s+warning).+\\r?\\n)*+.*\\[javadoc\\]\\s+[1-9]\\d*\\s+warning.*\\r?\\n \
\
|.*javadocs-lint:.*\\r?\\n(?:.*\\[exec\\].*\\r?\\n)*+ \
\
|.*check-forbidden-apis:.*\\r?\\n \
(?:\\s*+\\[forbidden-apis\\]\\s*\\r?\\n \
|\\s*+\\[forbidden-apis\\]\\s++(?!Scanned\\s+\\d+\\s+class\\s+file\\(s\\))\\S.*\\r?\\n)*+ \
\\s*+\\[forbidden-apis\\]\\s++Scanned\\s+\\d+\\s+class\\s+file\\(s\\).*[1-9]\\d*\\s+error\\(s\\)\\.\\r?\\n \
\
|.*\\r?\\n.*Full\\s+thread\\s+dump(?s:.*+) \
\
|.*(?:FATAL|ERROR):(?s:.*+) \
\
|.*BUILD\\s+FAILED(?s:.*+) \
)"}
Steven Rowe added a comment - 30/Jul/12 22:55
ASF Jenkins build log excerpts in notification emails are looking good.
| Status | Open [ 1 ] | Resolved [ 5 ] |
| Resolution | Fixed [ 1 ] |
Build: ${BUILD_URL} ${FAILED_TESTS} Build Log (for compile errors): ${BUILD_LOG_REGEX,regex="\\[javac\\]\\s+\\d+ error(s*)\\b",linesBefore=100,linesAfter=0}