Back to home

Kubecon Copenhagen 2018

Kubecon/CloudNativeCon Europe CPH from an OpenStack/TripleO point of view

I recently had the opportunity to attend Kubecon/CloudNativeCon Europe in Copenhagen. Although the event was very Kubernetes oriented, I chose to focus on the general security bits of the conference, as well as the service-mesh related topics. This was with the following reasoning:

• Given that we’re aiming to containerize as much as we can from OpenStack, we really need to catch up and take more container security practices into use.

• A lot of problems that we’re tackling on OpenStack are also being tackled in the kubernetes/cloud native community. We should try to converge whenever possible instead of trying to brew our own solutions.

• The service mesh use-cases resonate very well with a lot of the security user-stories that we’ve been working on lately.

With this in mind, what I gathered from the different projects.

Service mesh

Lately I’ve been quite interested in the service-mesh topic since it’s brought up by folks tackling the same issues we’ve been facing lately in OpenStack & TripleO.

Background

The concept of a “service mesh” is the concept of offloading all the network interaction from the service itself to a layer that sits somewhere in the host. This layer usually takes the form of a proxy. It can be in the form of a side-car container that runs with the application (so you’ll have a proxy per-service), or it can run as a singleton in the host and catch all traffic that goes towards the applications. The proxy will then be aware of all the applications that are part of the “mesh”, and handle load-balancing, service discovery, monitoring, policy, circuit breaking and it can even be your TLS proxy for the service(s).

One important concept to understand is the separation between the control plane and the data plane. The control plane is how you configure the mesh itself (probably in the form of a service that has an API or a set of APIs for this purpose), and a data plane, which is handled by the proxy, and it’s where all your application traffic flows. So, with this in mind, there are some mesh solutions that will already have a control plane implemented, whereas for other solutions, you have to brew your own.

For more info on the service mesh, I recommend these blog posts:

• Introduction to modern network load balancing and proxying
• Service mesh data plane vs. control plane
• Pattern: Service Mesh

Tying it up with OpenStack/TripleO

One of the features we did recently was enabling TLS everywhere for TripleO, and if I may say so… It was a big pain. First off we had the issue of every service doing TLS in their own way and having to configure (or even enable the configuration) them for each technology with all their own knobs and handles. Some services were even hard-coding ‘http’ in their endpoints, or were limited to just using IPs (not FQDNs). These are details and nits, but still stuff that you have to do and takes up time.

The service mesh addresses this issue by allowing you to offload that to a proxy, which is where you configure TLS. So there is ONE way to set things up. Yes, it has its own knobs and handles, but at least there is only one set of knobs and handles to worry about.

There’s also the issue of getting an acceptable PKI with all the necessary features, as opposed to copying in a bunch of certificates and forgetting the rest. For this, in TripleO, we used FreeIPA (which I still think was a good choice).

The way this is addressed by service mesh solutions depends on the implementation. Some solutions, such as Istio and Conduit, provide their own PKI solution, so you’ll get TLS by default. In other implementations you have to provide your own. Given that we already have a chosen PKI, it shouldn’t be too hard to take it into use for this purpose; although, Istio’s PKI (the one that I checked out in the conference) is not pluggable yet.

The proxy will also take care of metrics for you, so we could replace the OpenStack-specific OSProfiler and take that into use instead. This would give us more visibility on the overall OpenStack service performance, and help us identify bottle necks.

Finally, a potential benefit would be to take the service-discovery and loadbalancing capabilities into use:

• We could reduce the amount of hops through the network, since service-to-service communication would no longer need to go through HAProxy in all cases (Which is what happens when a service needs to talk to keystone, for instance).

• We could potentially deploy several versions of the same service at the same time, and do a rolling-upgrade with relative ease thanks to the proxy (Although this would only be possible for services using Oslo.Versioned objects and with zero-downtime upgrades figured out).

While this is not something we have in the timeline at the moment, I would like to investigate this approach further, as it seems to provide quite a lot of benefits.

Istio

The first project that I had in mind tracking was Istio, which is a service mesh implementation that’s been getting a lot of momentum lately. Istio uses Envoy as the default underlying proxy that does the actual heavy lifting, and provides several services for configuring, monitoring and securing your service mesh. They have their own CA component (that’s is now called Citadel) which uses the SPIFFE specification to identify workloads and effectively give them certificates (more on SPIFFE later).

There seems to be a lot of work on-going to make Istio’s configuration easier, these enhancements include an API to self-configure Istio (instead of using Custom Resource Definitions). This API will hopefully also address the update issues they’ve been seeing, and also enable configuration rollbacks.

Istio also handles policy for the mesh traffic. For this, there are two built-in policy adaptors: One based on Kubernetes’ RBAC, and the other one based on Open Policy Agent (more on OPA later).

Unfortunately, it seems that all the efforts are right now being placed on Kubernetes. So, if we would like to take Istio into there are several things we would need to work on first in order to have it address our needs:

• Make the CA component extendible

  • Currently the CA component is not extendible, so either we ditch our FreeIPA solution, or we code the pluggability into “Citadel”

• Make the CA auth customizable

  • The built-in CA auth only take Kubernetes into account, so not only would we need to get customizable auth policies into Istio, we would also need to come up with relevant policies to assure that we’re giving out certs to the correct workloads/components.

  • Right now we have some assurance that only specific hosts can request certificates, and they can only do so for specific services. This all thanks to FreeIPA’s kerberos instance. We would need to add similar support to Istio.

• Make Istio run in environments that are not Kubernetes

  • The current configuration and deployment scripts are only meant for kubernetes. Making it run in other environments would require a significant effort, as well as re-packaging of all the components.

  • There are options to make Istio add VMs to the service mesh, but they still require Kubernetes to be running somewhere, and is limited to Google Cloud VMs. So this doesn’t necessarily help us.

Even though we could run Istio without some components to have an easier setup, It seems to me that adopting it would take a significant amount of effort.

If we want to take the benefits of the service mesh into use in TripleO, it might be better to take a slower approach, and just take fewer components into use, instead of going with the full-blown solution. This lead me to take a deeper look into just using Envoy. This conclusion was also inspired by the recommendations in other talks in the conference that mentioned how other companies started adopting the service mesh: going for a cautious and thoughtful approach instead of a full-on migration.

Envoy

Envoy is a magical proxy that folks have been using to enable service mesh architectures in their deployments. While it’s not the only proxy out there, it seems to be getting a lot of momentum in the community.

It supports proxying several protocols, which include: HTTP, HTTP/2, gRPC, MongoDB, Redis, among others. It’ll also provide L3/L4 filtering, health checking and TLS termination. It also provides statistics (with statsd integration) and distributed tracing via plugins.

For a long time it was configured with static files, but it now has support for dynamic configuration via gRPC. This is called xDS API, and is what control planes need to implement in order to take Envoy into use (this is what Istio does, for instance). So, in order for us to take it into use in OpenStack, we would need to either expand and run their reference xDS API or implement our own (which is what other folks seem to be doing for some reason).

A nice feature of Envoy is it’s support for hot-restarts which is envoy’s ability to reload and even do a binary update without dropping connections. I can see this feature to be very useful for our updates/upgrades in TripleO.

Currently, it seems to me that if we want to experiment with proxies to try to bring service mesh benefits into TripleO, starting with Envoy and some services would be a good start.

SPIFFE

There were a lot of talks about SPIFFE/SPIRE in Kubecon. Given that it’s a security component and it seems to be taken into use by several other projects, I decided to take a look at it.

SPIFFE’s main goal is to establish trust between workloads and have the workload answer the question “who am I?”. So basically it aims to establish the workload’s identity. SPIFFE is an open specification which defines the overall mechanisms and how the API should look like. It introduces a couple of concepts:

• SPIFFE ID

  • It’s how we identify the given workload

  • Takes the form of a URL such as the following: spiffe://acme.com/billing/payments

  • acme.com would be the trust domain (could be a kerberos realm?)

  • billing/payments would identify the workload.

• SVID (SPIFFE Verifiable Identity Document)

  • The current proposal is an x509 certificate that contains your SPIFFE ID as part of the certificate’s SAN.

  • It’s meant to be a specification of the minimum requirements for a verifiable document, x509 is one implementation, but there could be more.

  • The advantage of the SVID being an x509 certificate, is that it could effectively be used for TLS, which is what Istio is doing.

SPIFFE then dictates ways that the workload could get its SVID, which is via customizable and deployer-defined attestation policies. The policies are executed by an agent that runs on every node, and there should be a central server that ultimately verifies the claims and provides the agents with the actual SVIDs.

The team provided a reference implementation called SPIRE. This implements the SPIFFE specification and contains code for the agents and the server. There is already Vault support, and work on-going to hook up the server to HSMs.

It communicates with the workloads via a unix domain socket, so there’s no explicit authentication to communicate with the API. It is then the responsibility of the node-agent to check if the workload’s request is actually valid. Which is where the attestation policies kick in.

SPIRE already has customizable attestation policies which we could extend, and it seems that some folks from VMWare already implemented a Kerberos-based attestation plugin which ended up in very interesting conversations. I can definitely see that attestor being used with the kerberos instance provided by FreeIPA.

Currently SPIRE uses short-lived certificates, but support for actual revocation is coming in later this year. There is still no decision as to what mechanism will be used (could be CRLs or OCSP).

There is also another limitation, which is HA support. Currently SPIRE is implemented with SQLite and assumes one server node. How to get SPIRE to use a distributed database and make it scale is still an active problem the team is tackling.

There will also be a public security audit of the code in the near future.

Unfortuntely, even though Istio uses the SPIFFE specification for it’s CA (Citadel), it doesn’t use SPIRE, and instead contains its own implementation in the code-base. Asking around, it seems that the reason is to not lock in both project’s speeds together; so this allows Istio to move at it’s own pace.

Open Policy Agent

Open Policy Agent aims to unify policy configurations. It takes the form of an agent that your service communicates with in order to enforce the policy decision. It has it’s own format to configure policies which the agent interprets.

It seems to be getting some adoption, with integration with Istio and Kubernetes.

The main reason I took a look into it, was because I thought it would be a nice way to get the centralized policy policy story going for OpenStack. This would require us to implement an OPA driver in oslo in order to do policy and have an agent running with each OpenStack service. Unfortunately it seems we would need to implement a control plane for OPA (centralized configuration), since currently only the agent is open source. So we would have an OpenStack specific control plane that can read the already-existing policy.json files and transforms them into OPA documents; these would then be taken into use by the agents. The separate agent approach would give us the advantage that we wouldn’t need to reload OpenStack services in order to apply new policies (although this is not a very common operation). Another benefit would be to have the same policy format for Kubernetes and OpenStack.

Conclusion

The conference was great and gave me a lot of ideas that we could take into use in OpenStack. Although we might not use all of them (or even none of them, time will tell), I still learned a lot and got me very excited about the future in the cloud industry. I wish to see more interaction between the Cloud Native and the OpenStack communities, as I think there are a lot of convergence points where we could use the wisdom of both communities. I really hope I can attend again.

tags: tripleo - openstack - kubernetes

Back to home