Kubernetes Secrets: A Secure Credential Store for Jenkins

栏目: IT技术 · 发布时间: 4年前

Cloud inResearch

Kubernetes Secrets: A Secure Credential Store for Jenkins

By: Vasumathy Seenuvasan

At eBay, we containerized Jenkins to provide a continuous build infrastructure on Kubernetes Clusters to power the ecommerce marketplace experience. Our goal was to leverage the capability of Kubernetes secrets, for managing the Jenkins credentials.

Kubernetes Secrets: A Secure Credential Store for Jenkins

Background

eBay.com is powered by applications that are built with Jenkins as the opinionated choice of build-machinery. We run Jenkins instances as a cloud native workload on the Kubernetes Clusters with server-agent mode, all of which are managed by the eBay cloud team.

The Jenkins servers provided to eBay development teams have a default credential plugin installed. The credentials stored on these Jenkins instances are used for various purposes, such as connecting to GitHub and third-party APIs. The credential plugin stores the user credentials, as well as the encryption key, on the disk in an encrypted format.

These credentials are very sensitive in nature and need to be stored safely and securely. However, storing the key and the credential on the same disk poses a risk. Jenkins instances are also generally shared across a team with ADMINISTER privileges , which increases the chances of team members knowing each other’s credentials.

Initially, we considered setting up a Vault 1 store, which is generally used in the community to secure the credentials in Jenkins. However, that would be a vendor lock-in. To remain vendor-neutral, we chose a different strategy. 

However, eBay has its own proprietary key management platform, which integrates with Kubernetes cluster’s control-plane and mounts secrets in a secure manner to the containers leveraging Kubernetes API standards. Since we are running Jenkins as workloads on Kubernetes, a simple solve was to standardize by using Kubernetes Secrets. Another benefit of this approach is that the same secrets can also be backed up to the in-house key management solution.

The challenge here was to integrate the Jenkins application with this proprietary key management platform. To solve the challenge, we developed a new plugin - Jenkins Credentials Secrets Plugin - which replaces the default credential plugin and is now available as an eBay open source project.

This article explains the design and process of a more secure way of storing Jenkins credentials as secrets on Kubernetes Clusters.

Scale of Operation

We have enabled this plugin to store credentials as Kubernetes Secrets on ~6000 Jenkins instances at eBay. We have also migrated all the credentials currently stored in these instances to Kubernetes secrets with ZERO down time for the end users - eBay application developers.

Assumptions

1.     The Jenkins master runs as a container in Kubernetes Cluster.

2.     The Jenkins master has a Kubernetes service account mounted to it.

Limitations

Only credentials under the Jenkins Global domain are supported by the plugin at this moment.

Design

When a credential is created by the user from a Jenkins UI or API call, a Kubernetes Secret is generated in a namespace (as specified by the plugin user) in the cluster with the required information from the credentials. The Secret specification’s “data” will hold the credential’s “sensitive” information. The Secret will have labels and annotations in the spec to store the below details.

Kubernetes Secrets: A Secure Credential Store for Jenkins

The secret name is generated as “sec”  + “-” + “UUID” + “-” + “JENKINS_INSTANCE_NAME”

Example : sec-<>-myCI.

We use UUID because the Kubernetes Secret object’s naming convention does not support all the characters allowed by Jenkins credentials (e.g. whitespaces and underscores). The credentials ID stored by the user is captured in the Secret’s labels, as shown in the above chart. We also capture other credential information, like description and type, and this list can be extended to attributes added in the future as well.

Secrets-To-Credentials Converter

The core component of this approach is the “Secrets-to-credentials converter” module. On Jenkins start up, the plugin gets the list of secrets that have the Jenkins name label selector. This is necessary so only the secrets belonging to the Jenkins in question are pulled. Converters for all of the below Jenkins supported credentials are implemented.

  • Username with password
  • Docker Host Certificate authentication
  • Kubernetes Service Account
  • OpenShift OAuth token
  • OpenShift Username and Password
  • SSH Username with Private Key
  • SSH Username with Private Key and Passphrase
  • Secret file
  • Secret text
  • Certificate

The credential type of a secret is identified from the label (jenkins.io/credentials-type), and the corresponding converter implementation is invoked to convert a secret to a credential. This credential is added to the Jenkins credentials Map offered by the Jenkins credentials plugin. This makes the approach transparent to the users, as they just see the same credential information on Jenkins UI, and all the secret conversion happens in the background.

Lifecycle Management of Secrets in Jenkins

Create: When users add credentials in the Jenkins UI, secrets will be created on the corresponding Kubernetes cluster in the namespace provided, and credentials will no longer be stored in `credentials.xml` on the disk.

Read: On Jenkins startup, credentials of the particular CI are loaded from the Kubernetes cluster (using label selector “jenkins.io/ci-name”).

Update: When a credential is updated, the corresponding secret is updated (using label selectors “jenkins.io/ci-name” and “jenkins.io/credential-id”).

Delete: When a credential is deleted, the corresponding secret is deleted (using label selectors “jenkins.io/ci-name” and “jenkins.io/credential-id”).

Secrets YAML

Username Password credential Type:

Kubernetes Secrets: A Secure Credential Store for Jenkins

The rest of the credential type YAML specs are available in the source code repository here:

https://github.com/eBay/kube-credentials-plugin/tree/master/credentialspecsamples

  • Base Credentials plugin - 2.1.19
  • Kubernetes plugin (dependency) - 1.1.4

We have open sourced this project, and the git repo can be found here:

https://github.com/eBay/kube-credentials-plugin

We welcome any PullRequests (PR) or github-issues on this repo. We have listed to-do items on the repo and welcome PRs to address them.

Vault is a product from HashiCorp:  https://www.vaultproject.io/

Tag: Cloud

Previous Article : Relation Embedding with Dihedral Group in Knowledge Graph


以上就是本文的全部内容,希望本文的内容对大家的学习或者工作能带来一定的帮助,也希望大家多多支持 码农网

查看所有标签

猜你喜欢:

本站部分资源来源于网络,本站转载出于传递更多信息之目的,版权归原作者或者来源机构所有,如转载稿涉及版权问题,请联系我们

Linux内核完全剖析

Linux内核完全剖析

赵炯 / 机械工业出版社 / 2006-1 / 79.00元

本书对早期Linux操作系统内核全部代友文件进行了详细的剖析,旨在让读者在尽量短的时间内对Linux的工作机理获得全面而深刻的理解,为进一步学习和研究Linux系统打下坚实的基础。虽然选择的版本较低,但该内核已能够正常编译运行,并且其中已包括了Linux工作原理的精髓。书中首先以Linux源代码版本的变迁为主线,简要介绍了Lin-ux系统的发展历史,同时着重说明了各个内核版本之间的主要区别和改进方......一起来看看 《Linux内核完全剖析》 这本书的介绍吧!

随机密码生成器
随机密码生成器

多种字符组合密码

RGB HSV 转换
RGB HSV 转换

RGB HSV 互转工具

RGB CMYK 转换工具
RGB CMYK 转换工具

RGB CMYK 互转工具