Day two of an awesome conference. https://www.continuouslifecycle.de/2019/material/ concon:Con_2019
Embedded raus aus dem Bett - Harald Göttlicher (Bosch)
Interesting presnetation from a Bosch colleague trying to change the way Bosch develops software. Builds tools for CarDealers. They managed to develop a fully automated release pipeline including the Bosch QM gateways.
Challenges - Why do we have slow releases in Embedded
- rigid processes
- standards spice
- legacy code
- complex QA processes
Butcher the monolith use the Microservice concept in Embedded.
- Jenkins, Conan, Gradle, Buckaroo
- Wechsel zu Pull Updates
- HIL Tests in Container
- USB Ethernet Hub to connect
- We deliver what is finished!
- schnelle Freigabe und QA ist entscheidend
- Ende-zu-Ende-Verantwortlichkeit für Komponenten fördern
- Effiziente Kooperation der Teams
- Metriken auf DevOps Paradigmen umstellen (Lead-Cycle-Time)
- nixOs, RPM, deb for package management
Interesting alternative to the ELK stack, showing logs and metrics in the same UI (Grafana) is really tempting.
- founded 03/2018
- v-1.0 Beta Release
Simple to scale
- Existing solutions do full text indexing and support complex queries
- Loki doesnt’t index the text of the logs instead grouping entries into “streams” and indexing those with labels.
1 Integrated with existing observablilty
- storing 70 TB of log data a week!
- Workflow: Alert -> Dashboards -> Prometheus -> Log Aggregation -> Distributed Tracing
- Timeserises and labels are key:value pairs
- promtail collects the logs with the same labels as Prometheus
- fluentd patched for Loki or Grafana
Airplane friendly and cloud native
- single binary
- horizontally scalable
- Store data in Minio or any S3 storage S3, GCS (5Cent a GB)
- Use PromQL to Query logs and display graphs
Keynote “Trajectory Chaos Engineering” - Casey Rosenthal
What is Chaos engineering
- A complex architecture could not be modeled by a human
- Test chaos engineering in staging environment
- All components could be 100% correct in a complex systems, and it can show unwanted behaviour!
“Magic Triangle” Economics - Safety - Workload
- It’s hard to have an implicit intuiton about safety of a system
Economic Pillars of complexity (Kent Beck)
- States (Features increase states)
- Relationships (Dependencies with layers of abstractions)
- Evinronments ()
- Reversability (Software excels)
Software Engineering: doing it wrong since 1913.
- Remove People who cause accidents
- Document best practices and runbooks
- Defend against prior root causes, in a complex system there are root causes
- Enforce procedures (a lot of data to prove)
- Avoid risk, if you put up guardrails you are putting blinders on the people and prevent them to use the tools they need to use
- Add redundancy
Don’t fight complexity, navigate it!
Casey Rosenthal “Chaos Engineering” https://www.amazon.de/Chaos-Engineering-System-Resiliency-Practice/dp/1492043869
Effiency and Safety are at odds. Portugese captains, two groups risky and risk averse. Risky captains had a lot less mortality than risk avers captains.
Introduce latency to test complex systems
Cloud Native Transformation - Pini Reznik
Disrupting the indurstry in 3-4 years. Cloud Native Transformation Practical Patterns for Innovation
Great example of a cloud native maturity matrix https://blog.container-solutions.com/cloud-native-maturity-matrix
- Split the delivery team in two, one for legacy and one for the new platform
Why is it so difficult
Because - Cloud Native is new complex and needs a new way of thinking.
Cloud native technologies empower organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach.
These techniques enable loosely coupled systems that are resilient, manageable, and observable. Combined with robust automation, they allow engineers to make high-impact changes frequently and predictably with minimal toil.
The Cloud Native Computing Foundation seeks to drive adoption of this paradigm by fostering and sustaining an ecosystem of open source, vendor-neutral projects. We democratize state-of-the-art patterns to make these innovations accessible for everyone.
Cloud native maturity matrix https://blog.container-solutions.com/cloud-native-maturity-matrix
Cloud native transformation Patterns (book)
- Dynamic strategy
- Measure what matters
- No Regret moves -> options & hedges -> big bet
- ongoing eduction.
- open source internal projects, if they do not deliver to the value of the
The propblem of enterprises is that they think everything is predictable!
“Thinking fast and slow” Daniel Kahnmann
- confirmation Bias
- Illusion of control The tendency to oversestimate influence
- Status quo bias
- Ikea effect
Complexity requires creative thinking and linking. You cannot learn new things with deadlines.
How to drive change and transformation
- Impossible to deliver anything innovative with to large teams (use 6-8 team)
Helm - Reinhard Nägele (Codecentric)
Helm3 Released yesterday!!
- Kubernetes Deployments haben eine Revision History https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-back-a-deployment
- JSON Schemas support
- Support for Library Chart
- Helm 2 client bekam Rechte vom Tiller, daher nur ein Tiller pro Namespace
- Helm client ist wie anderer Server zum K8s API Server
- Helm installiert release informationen im entsprechenden Namespace
search, status, create, list, get, uninstall
- Kein init von helm mehr!!
- Helm ist in Go geschrieben, daher go templating sprache
- function library sprig to work with go templates
- Flow Control:
if/elses/else if; with; range
- built-in objects:
- NOTES.txt wird auf sysput gerändert
- contain dependencies
- _helpers.tlp: contains best practices from kubernetes for e.g. labels
helm upgrade demo charts/hello-world --install -f hello-world_values.yaml
- https://hub.helm.sh analog zur Ansible galaxy seite,
- helm charts repo wird deprecated
ct - the chart testing tool
- Lint install and test helm charts in a ci pipeline
- Test chart upgrades
Kind - Kubernetes in Docker
- Run local K8s clusters in docker
cr - the chart releaser tool
- use github pages and releases tp host your helm chart
Helm supports openshift, routes have to be stored in helm chart
- harbor for docker hub
- learn sed and grep
- Kubernetes Jobs ??
- READ: “Thinking fast and slow” Daniel Kahnmann
- Experiment more
- Automatisiertes Testen der Buildlandschaft
- Visitenkarten beantragen
- AWS Device Farm => OSS source project zum testen von iPhones https://openstf.io => use for embedded devices
- watch for similiar problems in other areas topic
- collect status_code values with promethes
- Openshift and Helm Operators: https://blog.openshift.com/build-kubernetes-operators-from-helm-charts-in-5-steps/
- howto write for and while loops with bash.
- Check for ansible role to deploy a kubernetes on MacOS
- Use Grafana Ansible annotations during ansible Jobs: https://docs.ansible.com/ansible/latest/plugins/callback/grafana_annotations.html