Day two of an awesome conference. https://www.continuouslifecycle.de/2019/material/ concon:Con_2019

Embedded raus aus dem Bett - Harald Göttlicher (Bosch)

Interesting presnetation from a Bosch colleague trying to change the way Bosch develops software. Builds tools for CarDealers. They managed to develop a fully automated release pipeline including the Bosch QM gateways.

Challenges - Why do we have slow releases in Embedded

  • rigid processes
  • standards spice
  • legacy code
  • complex QA processes
  • Hardware

Vision

Butcher the monolith use the Microservice concept in Embedded.

CICD

  • Jenkins, Conan, Gradle, Buckaroo

Atomare Updates

  • Wechsel zu Pull Updates

Atomares Testen

  • HIL Tests in Container
  • USB Ethernet Hub to connect

Atomare Auslieferung

  • We deliver what is finished!

Prozessänderungen

  • schnelle Freigabe und QA ist entscheidend
  • Ende-zu-Ende-Verantwortlichkeit für Komponenten fördern
  • Effiziente Kooperation der Teams
  • Metriken auf DevOps Paradigmen umstellen (Lead-Cycle-Time)
  • BizDevOps

Tooling

  • nixOs, RPM, deb for package management

Grafana Loki

Interesting alternative to the ELK stack, showing logs and metrics in the same UI (Grafana) is really tempting.

  • founded 03/2018
  • v-1.0 Beta Release

Simple to scale

  • Existing solutions do full text indexing and support complex queries
  • Loki doesnt’t index the text of the logs instead grouping entries into “streams” and indexing those with labels.

1 Integrated with existing observablilty

  • storing 70 TB of log data a week!
  • Workflow: Alert -> Dashboards -> Prometheus -> Log Aggregation -> Distributed Tracing
  • Prometheus:
    • Timeserises and labels are key:value pairs
  • promtail collects the logs with the same labels as Prometheus
  • fluentd patched for Loki or Grafana

Airplane friendly and cloud native

  • single binary
  • horizontally scalable
  • Store data in Minio or any S3 storage S3, GCS (5Cent a GB)
  • Use PromQL to Query logs and display graphs

Keynote “Trajectory Chaos Engineering” - Casey Rosenthal

What is Chaos engineering

  • A complex architecture could not be modeled by a human
  • Test chaos engineering in staging environment
  • All components could be 100% correct in a complex systems, and it can show unwanted behaviour!

“Magic Triangle” Economics - Safety - Workload

Economic Pillars of complexity (Kent Beck)

  • States (Features increase states)
  • Relationships (Dependencies with layers of abstractions)
  • Evinronments ()
  • Reversability (Software excels)

Software Engineering: doing it wrong since 1913.

Myths

  1. Remove People who cause accidents
  2. Document best practices and runbooks
  3. Defend against prior root causes, in a complex system there are root causes
  4. Enforce procedures (a lot of data to prove)
  5. Avoid risk, if you put up guardrails you are putting blinders on the people and prevent them to use the tools they need to use
  6. Simplify
  7. Add redundancy

Don’t fight complexity, navigate it!

Continuous Verification

Casey Rosenthal “Chaos Engineering” https://www.amazon.de/Chaos-Engineering-System-Resiliency-Practice/dp/1492043869

Effiency and Safety are at odds. Portugese captains, two groups risky and risk averse. Risky captains had a lot less mortality than risk avers captains.

Introduce latency to test complex systems

Cloud Native Transformation - Pini Reznik

Disrupting the indurstry in 3-4 years. Cloud Native Transformation Practical Patterns for Innovation

Great example of a cloud native maturity matrix https://blog.container-solutions.com/cloud-native-maturity-matrix

Patterns

  • Split the delivery team in two, one for legacy and one for the new platform

Why is it so difficult

Because - Cloud Native is new complex and needs a new way of thinking.

https://github.com/cncf/landscape https://github.com/cncf/toc/blob/master/DEFINITION.md

Cloud native technologies empower organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach.

These techniques enable loosely coupled systems that are resilient, manageable, and observable. Combined with robust automation, they allow engineers to make high-impact changes frequently and predictably with minimal toil.

The Cloud Native Computing Foundation seeks to drive adoption of this paradigm by fostering and sustaining an ecosystem of open source, vendor-neutral projects. We democratize state-of-the-art patterns to make these innovations accessible for everyone.

Cloud native maturity matrix https://blog.container-solutions.com/cloud-native-maturity-matrix

Design thinking

https://de.wikipedia.org/wiki/Design_Thinking

Cloud native transformation Patterns (book)

  • Dynamic strategy
  • Measure what matters
  • No Regret moves -> options & hedges -> big bet
  • ongoing eduction.
  • open source internal projects, if they do not deliver to the value of the

The propblem of enterprises is that they think everything is predictable!

Cognitive biases

“Thinking fast and slow” Daniel Kahnmann

  • confirmation Bias
  • Illusion of control The tendency to oversestimate influence
  • Status quo bias
  • Ikea effect

Complexity requires creative thinking and linking. You cannot learn new things with deadlines.

How to drive change and transformation

  • Impossible to deliver anything innovative with to large teams (use 6-8 team)

Helm - Reinhard Nägele (Codecentric)

Helm3 Released yesterday!!

Helm 3

  • Helm 2 client bekam Rechte vom Tiller, daher nur ein Tiller pro Namespace
  • Helm client ist wie anderer Server zum K8s API Server
  • Helm installiert release informationen im entsprechenden Namespace
    • commands: search, status, create, list, get, uninstall
    • Kein init von helm mehr!!
  • Helm ist in Go geschrieben, daher go templating sprache
    • function library sprig to work with go templates
    • Flow Control: if/elses/else if; with; range
    • built-in objects:
      • capabilities.APIVersions
  • Hooks
    • z.B. pre-install, post-rollback…

Files:

  • NOTES.txt wird auf sysput gerändert
  • Chart.yml
    • contain dependencies
  • _helpers.tlp: contains best practices from kubernetes for e.g. labels

Usage

helm upgrade demo charts/hello-world --install -f hello-world_values.yaml

https://github.com/unguiculus/hello-world

Helm Ökosystem

Helm Hub

ct - the chart testing tool

  • Lint install and test helm charts in a ci pipeline
  • Test chart upgrades

Kind - Kubernetes in Docker

  • Run local K8s clusters in docker

cr - the chart releaser tool

  • use github pages and releases tp host your helm chart

Openshift

Helm supports openshift, routes have to be stored in helm chart

TODO