jenkins Pipeline | fy'博客

# Pipeline

The Continuous Delivery Pipeline 持续交付流水线

DSL Domain-specific_language 某些特定领域设计的专用语言，比如sql

Pipeline as Code 流程即代码

Jenkins Pipeline is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins. Pipeline provides an extensible set of tools for modeling simple-to-complex delivery pipelines "as code" via the Pipeline DSL.

翻译：

Jenkins Pipeline 是一系列插件，它支持在jenkins中实现和集成持续交付流水线。Pipeline通过Pipeline DSL为从简单到复杂的交付场景，以代码化的方式为交付管道提供了一套可扩展工具。

Pipeline就是一套运行于Jenkins上的工作流框架，将原本独立运行于单个或者多个节点的任务连接起来，实现单个任务难以完成的复杂发布流程。Pipeline的实现方式是一套Groovy DSL，任何发布流程都可以表述为一段Groovy脚本，并且Jenkins支持从代码库直接读取脚本，从而实现了Pipeline as Code的理念。

# Pipeline key concepts--基本概念

交付流水线：

Jenkins Pipeline docker运行jenkins：

docker run \
--name blueocean \
-u root \
--rm \
-d \
-p 8080:8080 \
-p 50000:50000 \
-v ~/.jenkinshome:/var/jenkins_home \
-v /var/run/docker.sock:/var/run/docker.sock \
jenkinsci/blueocean

1
2
3
4
5
6
7
8
9
10

webUI方式运行Scripted Pipeline

Jenkinsfile (Scripted Pipeline)
node {  
    stage('Build') { 
        // 编译
        steps.echo "Scripted Pipeline:Build"
    }
    stage('Test') { 
        // 测试 
        steps.echo "Scripted Pipeline:Test"

    }
    stage('Deploy') { 
        // 部署 
        steps.echo "Scripted Pipeline:Deploy"
    }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

webUI方式运行Declarative Pipeline

Jenkinsfile (Declarative Pipeline)
pipeline {
    agent any
    stages {
        stage('Build') { 
            steps {
                // 编译
                echo "Declarative Pipeline:Build"
            }
        }
        stage('Test') { 
            steps {
                // 测试
                echo "Declarative Pipeline:Test"
            }
        }
        stage('Deploy') { 
            steps {
                // 部署
                echo "Declarative Pipeline:Deploy"
            }
        }
    }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

Blue Ocean Pipeline Editor

通过Jenkinsfile方式(Pipeline script from SCM)运行Scripted Pipeline和Declarative Pipeline

Jenkinsfile方式指的是在源码仓库根目录下（或其他路径），指定Jenkinsfile文件作为Pipeline的入口，可以直接在文件中书写Scripted Pipeline和Declarative Pipeline的脚本。

一些名词：

Jenkinsfile

Pipeline模式中定义脚本，被调用的入口
node

一个Node就是一个Jenkins节点，或者是Master，或者是Agent，是执行Step的具体运行期环境
agent

为pipeline分配执行器和工作空间，类似Scripted Pipeline中的node
stage

一个Pipeline可以划分为若干个Stage，每个Stage代表一组操作。注意，Stage是一个逻辑分组的概念，可以跨多个Node
steps

Step是最基本的操作单元，小到创建一个目录，大到构建一个Docker镜像，由各类Jenkins Plugin提供，并动态扩充的

摘要：

Pipeline supports two syntaxes, Declarative (introduced in Pipeline 2.5) and Scripted Pipeline. Both of which support building continuous delivery pipelines. Both may be used to define a Pipeline in either the web UI or with a Jenkinsfile, though it’s generally considered a best practice to create a Jenkinsfile and check the file into the source control repository.

Pipeline支持两种语法，陈述式（Pipeline2.5引入）和脚本式。两种语法都支持编译的持续集成。尽管通常最佳实践方式是创建一个Jenkins文件，并将其纳入源码控制，但是两种语法都可以通过 web UI 或者一个 Jenkinsfile文件的方式来使用。

The agent directive, which is required, instructs Jenkins to allocate an executor and workspace for the Pipeline

（在陈述式Pipeline中，）agent代码块是必须的，它指示Jenkins为指定Pipeline分配执行器和工作空间。

The Jenkinsfile is not a replacement for an existing build tool such as GNU/Make, Maven, Gradle, etc, but rather can be viewed as a glue layer to bind the multiple phases of a project’s development lifecycle (build, test, deploy, etc) together.

Jenkinsfile并不是现存编译工具比如GNU/Make, Maven, Gradle等的替代品，而是可以被视为一个工程研发生命周期（编译，测试，部署等）中多个阶段的粘合层。

# Advanced Syntax for Pipeline--高级语法

String Interpolation--字符串插值

Pipeline中字符串插值语法与groovy相同。groovy中可以用单引号、双引号来定义字符串。
```
def singlyQuoted = 'Hello'
def doublyQuoted = "World"
```
1
2
只有双引号定义的字符串才可以以${变量名}的方式进行插值。linux shell编程中也是遵循这个规则。
```
def username = 'Jenkins'
echo 'Hello Mr. ${username}'
echo "I said, Hello Mr. ${username}"
```
1
2
3
结果：
```
Hello Mr. ${username}
I said, Hello Mr. Jenkins
```
1
2
理解怎么运用字符串插值对于运用Pipeline其它的一些高级特性是至关重要的。

Working with the Environment--运用环境变量

内置环境变量：http://localhost:8080/pipeline-syntax/globals#env

   // Declarative //
   pipeline {
       agent any
       stages {
           stage('Example') {
               steps {
                   echo "Running ${env.BUILD_ID} on ${env.JENKINS_URL}"
               }
           }
       }
   }
   // Script //
   node {
       echo "Running ${env.BUILD_ID} on ${env.JENKINS_URL}"
   }

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

设置环境变量

// Declarative //
pipeline {
    agent any 
    environment {
        CC = 'clang'
    }
    stages {
        stage('Example') {
            environment {
                DEBUG_FLAGS = '-g'
            }
            steps {
                sh 'printenv'
            }
        }
    }
}
// Script //
node {
    /* .. snip .. */
    withEnv(["PATH+MAVEN=${tool 'M3'}/bin"]) {
        sh 'mvn -B verify'
    }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

environment表达式：

定义在顶层的环境变量全局生效，定义在stage下面的仅在该stage下生效。

steps.withEnv:

http://localhost:8080/job/pipeline-declarative-test/pipeline-syntax/

withEnv(['变量=值', 'PATH+WHATEVER=/something']) {
    // some block
}

1
2
3
4

自定义环境变量

环境变量存储在env的静态变量中，可以直接操作env来达到修改环境变量值的目的。env本质上是一个map。可以通过env.环境变量名来修改或者取值。
```
echo "${env.PATH}"
env.ABC = "abc"
echo "${env.ABC}"
echo env.ABC
```
1
2
3
4
这里的环境变量是全局的，在整个Jenkinsfile作用范围内也就是一个Pipeline内都可以生效。环境变量名通常大写，单词间以下划线分割。

Parameter--参数化构建

// Declarative //
pipeline {
  agent any
  parameters {
      string(name: 'Greeting', defaultValue: 'Hello', description: 'How should I greet the world?')
  }
  stages {
    stage('Example') {
      steps {
          echo "${params.Greeting} World!"
      }
    }
  }
}
// Script //
properties([
  parameters([
    string(defaultValue: 'Hello', description: 'How should I greet the world?', name: 'Greeting')
  ])
])
node {
    echo "${params.Greeting} World!"
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

通过${params.Greeting}可以取到实际输入的参数值。其他的参数类型还有booleanParam, choice, file, text, password, run, or string：

properties([
  parameters([
    string(name: 'PERSON', defaultValue: 'Mr Jenkins', description: 'Who should I say hello to?'),
    text(name: 'BIOGRAPHY', defaultValue: '', description: 'Enter some information about the person'),
    booleanParam(name: 'TOGGLE', defaultValue: true, description: 'Toggle this value'),
    choice(name: 'CHOICE', choices: ['One', 'Two', 'Three'], description: 'Pick something'),
    password(name: 'PASSWORD', defaultValue: 'SECRET', description: 'Enter a password'),
    file(name: "FILE", description: "Choose a file to upload"),
    run(description: 'Defines the job from which the user can pick runs. The last run will be the default. These parameters are exposed to the build as environment variables:', filter: 'ALL', name: 'jobName', projectName: 'pipeline-scripted-test')
  ])
])

1
2
3
4
5
6
7
8
9
10
11

修改默认值的写法：

properties([
    parameters([
        string(defaultValue: 'Hello', description: 'How should I greetthe world?', name: 'Greeting')
    ])
])

properties([
    parameters([
        string(defaultValue: "${params.Greeting}", description: 'How should I greetthe world?', name: 'Greeting')
    ])
])

1
2
3
4
5
6
7
8
9
10
11

通过Snippet Generator (opens new window)中的properties:Set job properties来生成。

Handling Failures--失败处理
- try/catch/finally
- currentBuild (opens new window)通过currentResult获取当前job状态并处理

Using multiple agents—集群模式下指定运行node

指定node上的执行器及分配工作空间，也可以指定docker或者dockerfile来执行构建。这部分没有实践，不细讲。

// Declarative //
pipeline {
    agent none
    stages {
        stage('Build') {
            agent any
            steps {
                checkout scm
                sh 'make'
                stash includes: '**/target/*.jar', name: 'app'
            }
        }
        stage('Test on Linux') {
            agent {
                label 'linux'
            }
            steps {
                unstash 'app'
                sh 'make check'
            }
            post {
                always {
                    junit '**/target/*.xml'
                }
            }
        }
        stage('Test on Windows') {
            agent {
                label 'windows'
            }
            steps {
                unstash 'app'
                bat 'make check'
            }
            post {
                always {
                    junit '**/target/*.xml'
                }
            }
        }
    }
}
// Script //
stage('Build') {
    node {
        checkout scm
        sh 'make'
        stash includes: '**/target/*.jar', name: 'app'
    }
}
stage('Test') {
    node('linux') {
        checkout scm
        try {
            unstash 'app'
            sh 'make check'
        } finally {
            junit '**/target/*.xml'
        }
    }
    node('windows') {
        checkout scm
        try {
            unstash 'app'
            bat 'make check'
        } finally {
            junit '**/target/*.xml'
        }
    }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70

Optional step arguments--step附加参数

step传入附加参数，完成特殊功能。
```
String jsonStr = steps.sh returnStdout: true, script: "echo abc"
```
1
http://localhost:8080/job/pipeline-declarative-test/pipeline-syntax/html

sh: Shell Script

trigge--触发job

checkout scm

// SCMTrigger
properties([
    pipelineTriggers([
        scm('H/5 * * * *')
    ])
])
// TimerTrigger
properties([
    pipelineTriggers([
        cron('H/5 * * * *')
    ])
])

// 触发其他job
steps.build job: 'pipeline-scripted-test', parameters: [string(name: 'Greeting', value: 'Hello')]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

api方式调用

curl -X POST http://10.3.208.42:8080/jenkins/job/test-serviceApiTest/job/xhqb-svrTest/buildWithParameters --user fengyu:密码

flow control--流程控制

这部分遵循groovy语法，是Scripted Pipeline特有的，有if else，case when，while，.each等

// Scripted //
node {
    stage('Example') {
        if (env.BRANCH_NAME == 'master') {
            echo 'I only execute on the master branch'
        } else {
            echo 'I execute elsewhere'
        }
    }
}

1
2
3
4
5
6
7
8
9
10

// Scripted //
node {
    stage('Example') {
        try {
            sh 'exit 1'
        } catch (exc) {
            echo 'Something failed, I should sound the klaxons!'
            throw
        }
    }
}

1
2
3
4
5
6
7
8
9
10
11

parallel--并行

简单并行

step并行

node {
    stage("Build/Test") {
        parallel(
          "b1": {
            sh 'date'
            sh 'sleep 5'
          },
          "b2": {
            sh 'date'
            sh 'sleep 5'
          },
          "b3": {
            sh 'date'
            sh 'sleep 5'
          }
        )
    }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

stage并行

node {
    parallel(
        "b1": {
            stage("b1") {
                sh 'date'
                sh 'sleep 5'
            }
        },
        "b2": {
            stage("b2") {
                sh 'date'
                sh 'sleep 5'
            }
        },
        "b3": {
            stage("b3") {
                sh 'date'
                sh 'sleep 5'
            }
        }
    )
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

Parallel From Grep

import jenkins.model.*

// While you can't use Groovy's .collect or similar methods currently, you can
// still transform a list into a set of actual build steps to be executed in
// parallel.

def stepsForParallel = [:]

// Since this method uses grep/collect it needs to be annotated with @NonCPS
// It returns a simple string map so the workflow can be serialized
@NonCPS
def jobs(jobRegexp) {
  Jenkins.instance.getAllItems()
         .grep { it.name ==~ ~"${jobRegexp}"  }
         .collect { [ name : it.name.toString(),
                      fullName : it.fullName.toString() ] }
}

j = jobs('test-(dev|stage)-(unit|integration)')
for (int i=0; i < j.size(); i++) {
    stepsForParallel["${j[i].name}"] = transformIntoStep(j[i].fullName)
}

// Actually run the steps in parallel - parallel takes a map as an argument,
// hence the above.
parallel stepsForParallel

// Take the string and echo it.
def transformIntoStep(jobFullName) {
    // We need to wrap what we return in a Groovy closure, or else it's invoked
    // when this method is called, not when we pass it to parallel.
    // To do this, you need to wrap the code below in { }, and either return
    // that explicitly, or use { -> } syntax.
    return {
       // Job parameters can be added to this step
       build jobFullName
    }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

Parallel From List

// While you can't use Groovy's .collect or similar methods currently, you can
// still transform a list into a set of actual build steps to be executed in
// parallel.

// Our initial list of strings we want to echo in parallel
def stringsToEcho = ["a", "b", "c", "d"]

// The map we'll store the parallel steps in before executing them.
def stepsForParallel = stringsToEcho.collectEntries {
    ["echoing ${it}" : transformIntoStep(it)]
}

// Actually run the steps in parallel - parallel takes a map as an argument,
// hence the above.
parallel stepsForParallel

// Take the string and echo it.
def transformIntoStep(inputString) {
    // We need to wrap what we return in a Groovy closure, or else it's invoked
    // when this method is called, not when we pass it to parallel.
    // To do this, you need to wrap the code below in { }, and either return
    // that explicitly, or use { -> } syntax.
    return {
        node {
            echo inputString
        }
    }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

Parallel Multiple Nodes

def labels = ['precise', 'trusty'] // labels for Jenkins node types we will build on
def builders = [:]
for (x in labels) {
    def label = x // Need to bind the label variable before the closure - can't do 'for (label in labels)'

    // Create a map to pass in to the 'parallel' step so we can fire all the builds at once
    builders[label] = {
      node(label) {
        // build steps that should happen on all nodes go here
      }
    }
}

parallel builders

1
2
3
4
5
6
7
8
9
10
11
12
13
14

#pipeline-syntax工具

获取pipeline代码的快捷方式，通过jenkins自带的工具来生成。

http://localhost:8080/pipeline-syntax/

Snippet Generator

step切片生成器
Declarative Directive Generator

陈述式语法生成器
Steps Reference

根据已安装的插件，动态生成的step文档
Global Variable Reference

全局变量相关：env环境变量，params参数化构建时用户输入参数，currentBuild
- env 环境变量
- params 参数化构建时用户输入参数
- currentBuild 本次构建相关参数
  
  currentBuild.description 二维码展示
- docker

# Multibranch Pipeline--多分支流水线

Multibranch Pipeline就是只需要配置一个git代码仓库，系统会根据git仓库自动创建Pipeline分支。能自动被创建的分支需要有两个条件：

1、满足Multibranch Pipeline设置的过滤条件；

2、分支根目录有Jenkinsfile文件；

# Shared Libraries--公共库

在jenkins中创建公共库环境变量

一个标准的公共库代码组成：

文件结构

$ tree jenkins-pipeline2
jenkins-pipeline2
├── README.md
├── src
│   ├── com
│   │   └── xhqb
│   │       └── pipeline
│   │           └── frontendWithSubmodule
│   │               └── BaseSteps.groovy
│   └── jenkins.gdsl
└── vars
    └── frontendWithSubmodule.groovy

1
2
3
4
5
6
7
8
9
10
11
12
13

import com.xhqb.pipeline.frontendWithSubmodule.BaseSteps

// vars/buildPlugin.groovy
def call(body) {
    // evaluate the body block, and collect configuration into the object
    def config = [:]
    body.resolveStrategy = Closure.DELEGATE_FIRST
    body.delegate = config
    body()


    node {
        agent any
        stage("scm checkout") {
            checkout scm
        }
        def buildSteps = stepsOfBranch(config)
//        循环调用BaseSteps中stage
        for (def name : buildSteps.stageNames()) {
            stage(name) {
                buildSteps.stepOfStage(name)()
            }
        }
    }
}

BaseSteps stepsOfBranch(config) {
    return new BaseSteps(this, config)
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

//BaseSteps.groovy
package com.xhqb.pipeline.frontendWithSubmodule


class BaseSteps implements Serializable {

    final INIT_STAGE = "Init"
    final COMPILE_STAGE = "Build"


    def steps
    def env
    def config = [:]
    def currentBuild


    BaseSteps(scripts, config) {
        this.steps = scripts.steps
        this.env = scripts.env
        this.config = config
        this.currentBuild = scripts.currentBuild
    }


    def stageNames() {
        return [INIT_STAGE, COMPILE_STAGE]
    }

    def stepOfStage(String name) {
        switch (name) {
            case INIT_STAGE: return this.&initStep
            case COMPILE_STAGE: return this.&compile
            default: this.&noop
        }
    }

    def noop() {
        steps.echo("noop")
    }

    def initStep() {

    }

    void compile() {
        
    }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

jenkinsfile

@Library('xh-build@h5build') _

frontendWithSubmodule{
	deployEnv = [
			"master": "test"
	]
	buildConfig = [
			"master": "./build.sh"
	]
	baseVersion = [
			"base_version":  "1.0.0"
	]
	modulesVersion = [
		"apply": "1.0.0",
		"login": "1.0.0"
	]
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

groovy闭包的代理模式：

OWNER_FIRST：所有者（如封闭的定义）首先检查，则委托
OWNER_ONLY：所有者检查，代表仅在明确引用时才被检查
DELEGATE_FIRST：代理首先被检查，然后所有者
DELEGATE_ONLY：委托第一次检查，如果引用明确
TO_SELF主人，才检查：既不是代表也不是所有人都检查