Category: Innovation

Simplifying Storage in Kubernetes

On December 9th, 2020, I had the pleasure of speaking with two great minds on the topic of simplifying storage in Kubernetes. Alex Chircop is the Founder and CEO at StorageOS (https://storageos.com), and Bruno Andrade is the Founder and CEO at Shipa (https://www.shipa.io). The full recording of this discussion is below. Here is a summary of what we discussed:

Simplify Deploying and Operating Stateful Applications within Kubernetes

All applications are stateful in some form, in the sense that all applications need to store state somewhere, whether it’s in a database or message queue, or just files or an object store for example. The reality is that as applications become portable, more and more organizations are looking to Kubernetes as the leading orchestrator. The challenge has always been, what do you need to do when it comes to storage? What is the key to simplifying storage in Kubernetes? A cloud native storage environment, such as StorageOS can provide developers that declarative and composable way of defining their storage and the attributes that their storage needs. 

Everything has become very dynamic in the world of cloud compute, and traditional system storage concepts don’t necessarily apply. Newer concepts tend to bring new complexity from a developer perspective, and it is important minimize these complexities so that the development team is not burdened with managing an infrastructure. The benefits of having a  declarative format that applies to every layer of the infrastructure effectively means that now the developer is in control of the attributes for their application, and it means that they can deploy their application in exactly the same way they always have.

kubernetes storage simplified

Storage Challenges at Scale

There is a scale of the data plane, and that is more of a traditional “speeds and feeds” kind of discussion where we’re talking about IOPS and megabytes per second, etc. But then there’s also the speeds of the control plane and this is really important because in a Kubernetes environment, apart from the dynamic structure of the cluster, in terms of nodes and scaling, you also have the concept of volumes coming and going much more dynamically.

StorageOS provides a disaggregated consensus concept in the control plane where every single volume effectively has its own mini brain and can manage placement decisions and failover decisions independently, so you don’t get a serialization. And this allows lots of activity to happen at the same time. Alex shares a quick demo of this in the video.

kubernetes storage simplified

“Portability is one of those things that’s still really, really difficult”

As people get more and more comfortable with Kubernetes, cloud native apps, and microservices, one tendency is that they’re scaling across multiple clusters. These multiple clusters can be on premises, or in the cloud, and in hybrid environments. Moving between environments, changing cluster versions, etc brings an entirely new set of complexity.

Shipa enables the concept of volume plans. Volume plans are directly attached to your storage class, and with a higher level integration into StorageOS, you can easily bind your storage to the application, regardless of the cluster you are using. Bruno shares a quick demo of this in the video.

Check out the full video for more details, and follow these links for additional documentation:

https://learn.shipa.io

https://docs.storageos.com

Monolith to Microservices – A Thought Leadership Talk

On November 11, 2020, I had the pleasure of speaking with two incredible thought leaders in the DevOps space: Vivek Pandey, VP of Engineering at Shipa (https://www.shipa.io), and Patrick Deuley, Sr Product Manager at GitLab (https://www.gitlab.com). The topic of the discussion was “Monolith to Microservices,” and we covered three key areas: Starting out, Scaling and Developer Experience.

GETTING STARTED WITHOUT A COOKIE CUTTER:

By 2022, IDC predicts that 90% of all new apps will feature a microservices architecture. Some of the drivers behind that are the improved ability to design, debug, update and leverage third-party code, and more. However, transitioning an existing application from a traditional monolith architecture to microservices architecture can be very challenging. How do we get started down the microservices path?

SCALING, AKA “BLOWING THINGS UP”

Often times, teams get really excited once they start to see benefits from a microservices architecture, and they start migrating everything. But scaling brings in an entirely new set of challenges. Gone are the days of simply throwing more load balancers at the problem. What are some additional challenges that teams are going to encounter when they start to scale and what are some strategies that can help them address some of these things?

THE DEVELOPER EXPERIENCE - “IT’S SO COMPLICATED”

What changes for a developer in an object-oriented architecture? Are there additional complexities? And if so, what are some strategies to improve their experience and allow them to focus on writing code, instead of thinking about clusters and YAML files and all the other things that come along with microservices and Kubernetes?

See how Ship and GitLab integrate to provide a powerful combination for tackling many of these challenges HERE

Shipa Framework For Developers

...the central focus of these services is still Kubernetes itself, and now a new breed of abstraction has begun to appear which goes one step above these managed Kubernetes offerings to bring the focus back to the application itself. One such recently-launched company, Shipa, delivers a cloud native application management framework built to manage the full application lifecycle in an application-centric fashion.

The New Stack has recently featured Shipa in an article that provides excellent insight on Shipa’s product launch and how it can change how cloud-native applications are deployed and managed.

Read the full article here.



Rethinking Ops

Looking back at my years working with infrastructure and going through it’s changes, I believe its time we start to rethink Operations because clearly this model of Ops as cluster or infrastructure admins does not scale. Developers will always out-demand their capacity to supply. Either your headcount is out of control or your ability to innovate and deliver is severely hamstrung. Operations becomes this interrupt-driven thing where we’re just fighting fires as they happen. Ops as masters of production usually devolves to Ops becoming human incident routers, trying to figure out what team or person can help resolve problems because, being responsible for everything, they don’t have the insight to fix it themselves.

The idea of “Ops lock-in” can be a major problem, where your own Ops team who just isn’t able to support the kind of innovation that you’re trying to do slows down innovation.

My thought and vision for the future of Operations is taking combined engineering to its logical conclusion. Just like with QA, Ops capabilities should be embedded within development teams. The reality is you can’t be an effective software engineer today without some Ops skills, and I think every role should be working towards automating itself out of a job. Specifically, my thought and vision is that we should look at enabling developers to self-service through a continuous operation platform and empowering them to deploy and operate their services…with minimal Ops intervention..

With this, Ops become force multipliers. We move away from the reactive, interrupt-driven model where Ops are masters of production responsible for everything. Instead, we make dev teams responsible for their services but provide the tools they need to actually own their systems end-to-end — from the code on their laptops to operating it in production.

Enabling developers to self-service means treating Ops as a product team. The infrastructure automation, deployment automation, configuration management, logging, monitoring, and production tools — these are all products and it’s these products that allow teams to fully own their services. This leads to empowerment.

Products enable ownership. We move away from Ops as masters of production responsible for everything and push that responsibility onto dev teams. They are the experts for their services. They are best equipped to deal with problems that arise but we provide the tools they need to diagnose and resolve those problems on their own.

I believe the near future is exciting and I’m excited to see how we bridge more and more the gap between Devs and Ops while helping organizations to transition to a more effective model, that delivers value faster while reducing toil