orchestration (computing) automcatic management of them
manifests system robust
common question why use k8s instead of docker swarm
number 1 ad: docker swarm is built in K8S: far richer, deploy do different cloud platforms K8S is just far more popular is a orchestration of choice. K8S is the most in demand container orchestration system out there.
A Java-based with angular on the front-end.
Terminology
Cluster: A group of physical or virtual machines
Node: Ap hysical or virtual machine that is part of a cluster
Control Plane: The set of processes that run the core k8s services (e.g. APi, scheduler, etcd …)
Pod: The lowest compute unit in Kubernetes, a group of containers that run on a Node.
Architecture
Head Node That’s the brain of Kubernetes
API server
Scheduler: to place the containers where they need to go
Controller manager: makes sure that the state of the system is what it should be
Etcd: data store, used to store the state of the system
Sometimes:
kubelet: a process manage all of this
docker: container engine
Worker node
kubelet That’s the Kubernetes agent that runs on all the Kubernetes cluster nodes. Kubelet talks with the Kubernetes API server and then talks to the local Docker daemon to be able to manage the Docker containers running on the node.
kube-proxy: a system that allos you to manage the IP tables in that node so that the traffic between the pods and the nodes is what it should be
You might have an incompatibility between your distribution and the one that Minikube is expecting. two command line tools: kubectl is the controller pogramme for k8s. and Minikube
Enable the Hyper-V role throught settings when enable don’t use oracle virtual box
goo.gl/4yEFbF for win 10 Professional
minikube start hangs forever on mac #2765
If you’re completely stuck then do ask me
Docker Overview
Difference between Docker Images and Container: A container is an instance of docker images. Docker container is the run time instance of images.
1 2 3 4 5 6 7
docker image pull richardchesterwood/k8s-fleetman-webapp-angular:release0-5 docker image ls docker container run -p 8080:80 -d richardchesterwood/k8s-fleetman-webapp-angular:release0-5 docker container ls minikube ip docker container stop 2c5 docker container rm 2c5
Pods
A pod is a group of one or more containers, with shared storage/network, and a specification for how to run the containers.
basic concept is Pod. A pod and a container are in a one to one relationship.
writing a Pod
kubectl get all show everything we have defined in our Kubernetes cluster.
kubectl apply -f first-pod.yaml
kubectl describe pod webapp
kubectl exec webapp ls
kubectl -it exec webapp sh it means get in interactively with teletype emulation. interactive
Services
Pods are not visible outside the cluster Pods are designed to be very throw away things. Pods have short lifetimes. Pods regularly die.
Service has stable port, with a service we can connect to kubernetes cluster.
cat webapp-service.yaml apiVersion: v1 kind: Service metadata: name: fleetman-webapp
spec: # This defines which pods are going to be represented by this Service # The service becomes a network endpoint for either other services # or maybe external users to connect to (eg browser) selector: app: webapp release: "0-5"
kubectl get po NAME READY STATUS RESTARTS AGE webapp 1/1 Running 2 2h webapp-release-0-5 1/1 Running 0 8m
kubectl get po --show-labels NAME READY STATUS RESTARTS AGE LABELS webapp 1/1 Running 2 2h app=webapp,release=0 webapp-release-0-5 1/1 Running 0 8m app=webapp,release=0-5
kubectl get po --show-labels -l release=0 NAME READY STATUS RESTARTS AGE LABELS webapp 1/1 Running 2 2h app=webapp,release=0
kubectl get po --show-labels -l release=1 No resources found.
REPLICASETS
ReplicaSets
When Pod die, it will never come back.
1 2 3 4 5
kubectl get all
kubectl describe svc fleetman-webapp
kubectl delete po webapp-release-0-5
ReplicaSets specify how many instances of this pod do we want k8s running on time
kubectl apply -f pods.yaml replicaset.apps "webapp" created pod "queue" created
kubectl get all NAME READY STATUS RESTARTS AGE pod/queue 1/1 Running 0 58s pod/webapp-hzpcp 1/1 Running 0 58s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/fleetman-queue NodePort 10.106.99.143 <none> 8161:30010/TCP 3h service/fleetman-webapp NodePort 10.108.217.186 <none> 80:30080/TCP 23h service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 1d
NAME DESIRED CURRENT READY AGE replicaset.apps/webapp 1 1 1 59s
The difference between current and ready is, current is the number of containers that are running, and ready is the number of containers that are responding to requests.
kubectl apply -f pods.yaml deployment.apps "webapp" created pod "queue" unchanged
kubectl get all NAME READY STATUS RESTARTS AGE pod/queue 1/1 Running 0 24m pod/webapp-7469fb7fd6-4mcth 1/1 Running 0 12s pod/webapp-7469fb7fd6-sv4rw 1/1 Running 0 12s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/fleetman-queue NodePort 10.106.99.143 <none> 8161:30010/TCP 3h service/fleetman-webapp NodePort 10.108.217.186 <none> 80:30080/TCP 23h service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 1d
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE deployment.apps/webapp 2 2 2 2 12s
NAME DESIRED CURRENT READY AGE replicaset.apps/webapp-7469fb7fd6 2 2 2 12s
kubectl rollout status deploy webapp deployment "webapp" successfully rolled out
kubectl rollout status deploy webapp Waiting for rollout to finish: 1 out of 2 new replicas have been updated... Waiting for rollout to finish: 1 out of 2 new replicas have been updated... Waiting for rollout to finish: 2 old replicas are pending termination... Waiting for rollout to finish: 1 old replicas are pending termination... Waiting for rollout to finish: 1 old replicas are pending termination... Waiting for rollout to finish: 1 old replicas are pending termination... deployment "webapp" successfully rolled out
cat networking-tests.yaml apiVersion: v1 kind: Pod metadata: name: mysql labels: app: mysql spec: containers: - name: mysql image: mysql:5 env: # Use secret in real life - name: MYSQL_ROOT_PASSWORD value: password - name: MYSQL_DATABASE value: fleetman
--- kind: Service apiVersion: v1 metadata: name: database spec: selector: app: mysql ports: - port: 3306 type: ClusterIP
kga NAME READY STATUS RESTARTS AGE pod/mysql 1/1 Running 0 3m pod/queue 1/1 Running 0 18h pod/webapp-7469fb7fd6-sg87f 1/1 Running 0 17h pod/webapp-7469fb7fd6-znbxx 1/1 Running 0 17h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/database ClusterIP 10.101.3.159 <none> 3306/TCP 3m service/fleetman-queue NodePort 10.106.99.143 <none> 8161:30010/TCP 21h service/fleetman-webapp NodePort 10.108.217.186 <none> 80:30080/TCP 1d service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 1d
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE deployment.apps/webapp 2 2 2 2 17h
NAME DESIRED CURRENT READY AGE replicaset.apps/webapp-7469fb7fd6 2 2 2 17h replicaset.apps/webapp-74bd9697b4 0 0 0 17h replicaset.apps/webapp-8f948b66c 0 0 0 17h
kubectl exec -it webapp-7469fb7fd6-sg87f sh / # ls bin etc lib mnt root sbin sys usr dev home media proc run srv tmp var
# mysql -h database -uroot -ppassword fleetman Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 3 Server version: 5.7.26 MySQL Community Server (GPL)
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [fleetman]> show tables; +--------------------+ | Tables_in_fleetman | +--------------------+ | testable | +--------------------+ 1 row in set (0.01 sec)
Can find the ip address of any service that we like just by its name. And that’s called service discovery.
Fully Qualified Domain Names (FQDN)
1 2 3 4 5
# nslookup database nslookup: can't resolve '(null)': Name does not resolve
Each microservice should be Highly Cohesive and Lossely Coupled.
highly cohesive: each microservice should handlng one business requirement. Cohesive means that a microservice should have a single set of reponsibilities.
Each microservice will maintain its own data store. And that microservice will be really the only poart of the system that can read or write that data.
Fleetman Microservices- setting the scene
The logic in the API gateway is typically some kind of a mapping. So it would be something like if the incoming request ends with /vehicles, then delegate the call to, in this case, the position tracker.
API gateway: a web front end which is implemented in Java script
Position tracker: back end, calcuating the speeds of vehicles and storing the positions of all the vehicles.
Queue: which is going to store the messages that are received from the vehicles as they move around the country.
Positon simulator: a testing microservice which is going to generate some positions of vehicles.
Delete all the Pods:
1 2 3 4 5 6 7
kubectl delete -f . pod "mysql" deleted service "database" deleted deployment.apps "webapp" deleted pod "queue" deleted service "fleetman-webapp" deleted service "fleetman-queue" deleted
2019-06-05 06:09:06.044 INFO 1 --- [ main] c.v.s.PositionsimulatorApplication : Starting PositionsimulatorApplication v0.0.1-SNAPSHOT on position-simulator-6f97fd485f-gplr8 with PID 1 (/webapp.jar started by root in /) 2019-06-05 06:09:06.056 INFO 1 --- [ main] c.v.s.PositionsimulatorApplication : The following profiles are active: producadskfjsjfsislfslsj 2019-06-05 06:09:06.151 INFO 1 --- [ main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@5f4da5c3: startup date [Wed Jun 05 06:09:06 UTC 2019]; root of context hierarchy 2019-06-05 06:09:07.265 WARN 1 --- [ main] s.c.a.AnnotationConfigApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'journeySimulator': Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'fleetman.position.queue' in string value "${fleetman.position.queue}" 2019-06-05 06:09:07.273 INFO 1 --- [ main] utoConfigurationReportLoggingInitializer :
Error starting ApplicationContext. To display the auto-configuration report enable debug logging (start with --debug)
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'journeySimulator': Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'fleetman.position.queue' in string value "${fleetman.position.queue}" at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:355) ~[spring-beans-4.3.2.RELEASE.jar!/:4.3.2.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1214) ~[spring-beans-4.3.2.RELEASE.jar!/:4.3.2.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543) ~[spring-beans-4.3.2.RELEASE.jar!/:4.3.2.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) ~[spring-beans-4.3.2.RELEASE.jar!/:4.3.2.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) ~[spring-beans-4.3.2.RELEASE.jar!/:4.3.2.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.3.2.RELEASE.jar!/:4.3.2.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) ~[spring-beans-4.3.2.RELEASE.jar!/:4.3.2.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) ~[spring-beans-4.3.2.RELEASE.jar!/:4.3.2.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:776) ~[spring-beans-4.3.2.RELEASE.jar!/:4.3.2.RELEASE] at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:861) ~[spring-context-4.3.2.RELEASE.jar!/:4.3.2.RELEASE] at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:541) ~[spring-context-4.3.2.RELEASE.jar!/:4.3.2.RELEASE] at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759) [spring-boot-1.4.0.RELEASE.jar!/:1.4.0.RELEASE] at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:369) [spring-boot-1.4.0.RELEASE.jar!/:1.4.0.RELEASE] at org.springframework.boot.SpringApplication.run(SpringApplication.java:313) [spring-boot-1.4.0.RELEASE.jar!/:1.4.0.RELEASE] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1185) [spring-boot-1.4.0.RELEASE.jar!/:1.4.0.RELEASE] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1174) [spring-boot-1.4.0.RELEASE.jar!/:1.4.0.RELEASE] at com.virtualpairprogrammers.simulator.PositionsimulatorApplication.main(PositionsimulatorApplication.java:28) [classes!/:0.0.1-SNAPSHOT] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131] at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [webapp.jar:0.0.1-SNAPSHOT] at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [webapp.jar:0.0.1-SNAPSHOT] at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [webapp.jar:0.0.1-SNAPSHOT] at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:58) [webapp.jar:0.0.1-SNAPSHOT] Caused by: java.lang.IllegalArgumentException: Could not resolve placeholder 'fleetman.position.queue' in string value "${fleetman.position.queue}" at org.springframework.util.PropertyPlaceholderHelper.parseStringValue(PropertyPlaceholderHelper.java:174) ~[spring-core-4.3.2.RELEASE.jar!/:4.3.2.RELEASE] at org.springframework.util.PropertyPlaceholderHelper.replacePlaceholders(PropertyPlaceholderHelper.java:126) ~[spring-core-4.3.2.RELEASE.jar!/:4.3.2.RELEASE] at org.springframework.core.env.AbstractPropertyResolver.doResolvePlaceholders(AbstractPropertyResolver.java:219) ~[spring-core-4.3.2.RELEASE.jar!/:4.3.2.RELEASE] at org.springframework.core.env.AbstractPropertyResolver.resolveRequiredPlaceholders(AbstractPropertyResolver.java:193) ~[spring-core-4.3.2.RELEASE.jar!/:4.3.2.RELEASE] at org.springframework.context.support.PropertySourcesPlaceholderConfigurer$2.resolveStringValue(PropertySourcesPlaceholderConfigurer.java:172) ~[spring-context-4.3.2.RELEASE.jar!/:4.3.2.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.resolveEmbeddedValue(AbstractBeanFactory.java:813) ~[spring-beans-4.3.2.RELEASE.jar!/:4.3.2.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1039) ~[spring-beans-4.3.2.RELEASE.jar!/:4.3.2.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1019) ~[spring-beans-4.3.2.RELEASE.jar!/:4.3.2.RELEASE] at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:566) ~[spring-beans-4.3.2.RELEASE.jar!/:4.3.2.RELEASE] at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88) ~[spring-beans-4.3.2.RELEASE.jar!/:4.3.2.RELEASE] at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:349) ~[spring-beans-4.3.2.RELEASE.jar!/:4.3.2.RELEASE] ... 24 common frames omitted
kubectl logs -f position-simulator-6f97fd485f-gplr8 follow the log
2019-06-05 06:13:36.205 INFO 1 --- [ main] c.v.s.PositionsimulatorApplication : Starting PositionsimulatorApplication v0.0.1-SNAPSHOT on position-simulator-6d8769d8-ghtmw with PID 1 (/webapp.jar started by root in /) 2019-06-05 06:13:36.213 INFO 1 --- [ main] c.v.s.PositionsimulatorApplication : The following profiles are active: production-microservice 2019-06-05 06:13:36.361 INFO 1 --- [ main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@443b7951: startup date [Wed Jun 05 06:13:36 UTC 2019]; root of context hierarchy 2019-06-05 06:13:38.011 INFO 1 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on startup 2019-06-05 06:13:38.016 INFO 1 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Starting beans in phase 2147483647 2019-06-05 06:13:38.041 INFO 1 --- [ main] c.v.s.PositionsimulatorApplication : Started PositionsimulatorApplication in 2.487 seconds (JVM running for 3.201) 2019-06-05 06:13:38.046 INFO 1 --- [ main] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@443b7951: startup date [Wed Jun 05 06:13:36 UTC 2019]; root of context hierarchy 2019-06-05 06:13:38.048 INFO 1 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Stopping beans in phase 2147483647 2019-06-05 06:13:38.049 INFO 1 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Unregistering JMX-exposed beans on shutdown ca^H^H^C
cat services.yaml apiVersion: v1 kind: Service metadata: name: fleetman-webapp
spec: # This defines which pods are going to be represented by this Service # The service becomes a network endpoint for either other services # or maybe external users to connect to (eg browser) selector: app: webapp
ports: - name: http port: 80 nodePort: 30080
type: NodePort
--- apiVersion: v1 kind: Service metadata: name: fleetman-queue
spec: # This defines which pods are going to be represented by this Service # The service becomes a network endpoint for either other services # or maybe external users to connect to (eg browser) selector: app: queue
ports: - name: http port: 8161 nodePort: 30010
- name: endpoint port: 61616
type: NodePort
--- apiVersion: v1 kind: Service metadata: name: fleetman-position-tracker
spec: # This defines which pods are going to be represented by this Service # The service becomes a network endpoint for either other services # or maybe external users to connect to (eg browser) selector: app: position-tracker
ports: - name: http port: 8080
type: ClusterIP
kubectl apply -f services.yaml service "fleetman-webapp" unchanged service "fleetman-queue" unchanged service "fleetman-position-tracker" configured
kga NAME READY STATUS RESTARTS AGE pod/position-simulator-589c64887f-lhl8g 1/1 Running 0 38m pod/position-tracker-86d694f997-5j6fm 1/1 Running 0 45m pod/queue-9668b9bb4-4pqxr 1/1 Running 0 45m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/fleetman-position-tracker ClusterIP 10.104.177.133 <none> 8080/TCP 3m service/fleetman-queue NodePort 10.110.95.121 <none> 8161:30010/TCP,61616:30536/TCP 1h service/fleetman-webapp NodePort 10.103.224.156 <none> 80:30080/TCP 1h service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 2d
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE deployment.apps/position-simulator 1 1 1 1 1h deployment.apps/position-tracker 1 1 1 1 45m deployment.apps/queue 1 1 1 1 1h
--- apiVersion: v1 kind: Service metadata: name: fleetman-api-gateway
spec: # This defines which pods are going to be represented by this Service # The service becomes a network endpoint for either other services # or maybe external users to connect to (eg browser) selector: app: api-gateway
$ kops create cluster --zones ap-southeast-2a,ap-southeast-2b,ap-southeast-2c ${NAME} I0607 06:10:02.189636 3468 create_cluster.go:519] Inferred --cloud=aws from zone "ap-southeast-2a" I0607 06:10:02.243690 3468 subnets.go:184] Assigned CIDR 172.20.32.0/19 to subnet ap-southeast-2a I0607 06:10:02.243802 3468 subnets.go:184] Assigned CIDR 172.20.64.0/19 to subnet ap-southeast-2b I0607 06:10:02.243857 3468 subnets.go:184] Assigned CIDR 172.20.96.0/19 to subnet ap-southeast-2c Previewing changes that will be made:
SSH public key must be specified when running with AWS (create with `kops create secret --name fleetman.k8s.local sshpublickey admin -i ~/.ssh/id_rsa.pub`)
[ec2-user@foobar ~]$ ssh-keygen -b 2048 -t rsa -f ~/.ssh/id_rsa [ec2-user@foobar ~]$ kops create secret --name ${NAME} sshpublickey admin -i ~/.ssh/id_rsa.pub [ec2-user@foobar ~]$ kops edit ig nodes --name ${NAME} [ec2-user@foobar ~]$ kops get ig --name ${NAME} NAME ROLE MACHINETYPE MIN MAX ZONES master-ap-southeast-2a Master m3.medium 1 1 ap-southeast-2a nodes Node t2.medium 3 5 ap-southeast-2a,ap-southeast-2b,ap-southeast-2c [ec2-user@foobar ~]$ kops edit ig master-ap-southeast-2a --name ${NAME} Edit cancelled, no changes made.
[ec2-user@i ~]$ kops update cluster ${NAME} --yes I0607 06:28:38.011102 32239 apply_cluster.go:559] Gossip DNS: skipping DNS validation I0607 06:28:38.263244 32239 executor.go:103] Tasks: 0 done / 94 total; 42 can run I0607 06:28:39.702035 32239 vfs_castore.go:729] Issuing new certificate: "apiserver-aggregator-ca" I0607 06:28:40.216189 32239 vfs_castore.go:729] Issuing new certificate: "etcd-clients-ca" I0607 06:28:40.356654 32239 vfs_castore.go:729] Issuing new certificate: "etcd-peers-ca-main" I0607 06:28:40.743191 32239 vfs_castore.go:729] Issuing new certificate: "etcd-peers-ca-events" I0607 06:28:40.824760 32239 vfs_castore.go:729] Issuing new certificate: "etcd-manager-ca-events" I0607 06:28:41.265388 32239 vfs_castore.go:729] Issuing new certificate: "etcd-manager-ca-main" I0607 06:28:41.373174 32239 vfs_castore.go:729] Issuing new certificate: "ca" I0607 06:28:41.551597 32239 executor.go:103] Tasks: 42 done / 94 total; 26 can run I0607 06:28:42.539134 32239 vfs_castore.go:729] Issuing new certificate: "kube-scheduler" I0607 06:28:42.891972 32239 vfs_castore.go:729] Issuing new certificate: "kubecfg" I0607 06:28:43.157916 32239 vfs_castore.go:729] Issuing new certificate: "apiserver-proxy-client" I0607 06:28:43.556052 32239 vfs_castore.go:729] Issuing new certificate: "kubelet" I0607 06:28:43.677894 32239 vfs_castore.go:729] Issuing new certificate: "apiserver-aggregator" I0607 06:28:43.748079 32239 vfs_castore.go:729] Issuing new certificate: "kube-proxy" I0607 06:28:44.025132 32239 vfs_castore.go:729] Issuing new certificate: "kubelet-api" I0607 06:28:44.589696 32239 vfs_castore.go:729] Issuing new certificate: "kube-controller-manager" I0607 06:28:44.730038 32239 vfs_castore.go:729] Issuing new certificate: "kops" I0607 06:28:44.864527 32239 executor.go:103] Tasks: 68 done / 94 total; 22 can run I0607 06:28:45.089177 32239 launchconfiguration.go:364] waiting for IAM instance profile "masters.fleetman.k8s.local" to be ready I0607 06:28:45.101954 32239 launchconfiguration.go:364] waiting for IAM instance profile "nodes.fleetman.k8s.local" to be ready I0607 06:28:55.483430 32239 executor.go:103] Tasks: 90 done / 94 total; 3 can run I0607 06:28:55.974524 32239 vfs_castore.go:729] Issuing new certificate: "master" I0607 06:28:56.119668 32239 executor.go:103] Tasks: 93 done / 94 total; 1 can run I0607 06:28:56.336766 32239 executor.go:103] Tasks: 94 done / 94 total; 0 can run I0607 06:28:56.407976 32239 update_cluster.go:291] Exporting kubecfg for cluster kops has set your kubectl context to fleetman.k8s.local
Cluster is starting. It should be ready in a few minutes.
Suggestions: * validate cluster: kops validate cluster * list nodes: kubectl get nodes --show-labels * ssh to the master: ssh -i ~/.ssh/id_rsa admin@api.fleetman.k8s.local * the admin user is specific to Debian. If not using Debian please use the appropriate user based on your OS. * read about installing addons at: https://github.com/kubernetes/kops/blob/master/docs/addons.md.
[ec2-user@i ~]$ kops validate cluster Using cluster from kubectl context: fleetman.k8s.local
Validating cluster fleetman.k8s.local
INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS master-ap-southeast-2a Master m3.medium 1 1 ap-southeast-2a nodes Node t2.medium 3 5 ap-southeast-2a,ap-southeast-2b,ap-southeast-2c
NODE STATUS NAME ROLE READY ip-172-20-115-253.ap-southeast-2.compute.internal node True ip-172-20-39-212.ap-southeast-2.compute.internal node True ip-172-20-45-219.ap-southeast-2.compute.internal master True ip-172-20-89-8.ap-southeast-2.compute.internal node True
Your cluster fleetman.k8s.local is ready
[ec2-user@i ~]$ kubectl get all NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 100.64.0.1 <none> 443/TCP 4m50s
Provisioning SSD drives with a StorageClass
We have a workloads yaml file, where we’d be finding the pods that we want to deploy to our cluster. We have mongostack which is a specialist file, just for the mongo database. We have storage.yaml, which is currently defining that we want to store the mongo data in a local directory on the host machine. And we have a yaml file for the services.
--- # How do we want it implemented apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: cloud-ssd provisioner: kubernetes.io/aws-ebs parameters: type: gp2
[ec2-user@ip-1 ~]$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE mongo-pvc Bound pvc-b2ed286e-88f3-11e9-b509-02985f983814 7Gi RWO cloud-ssd 3m45s
[ec2-user@ip-1 ~]$ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-b2ed286e-88f3-11e9-b509-02985f983814 7Gi RWO Delete Bound default/mongo-pvc cloud-ssd 18s
spec: # This defines which pods are going to be represented by this Service # The service becomes a network endpoint for either other services # or maybe external users to connect to (eg browser) selector: app: webapp
ports: - name: http port: 80 type: LoadBalancer
--- apiVersion: v1 kind: Service metadata: name: fleetman-queue
spec: # This defines which pods are going to be represented by this Service # The service becomes a network endpoint for either other services # or maybe external users to connect to (eg browser) selector: app: queue
ports: - name: http port: 8161
- name: endpoint port: 61616
type: ClusterIP
--- apiVersion: v1 kind: Service metadata: name: fleetman-position-tracker
spec: # This defines which pods are going to be represented by this Service # The service becomes a network endpoint for either other services # or maybe external users to connect to (eg browser) selector: app: position-tracker
ports: - name: http port: 8080
type: ClusterIP
--- apiVersion: v1 kind: Service metadata: name: fleetman-api-gateway
spec: # This defines which pods are going to be represented by this Service # The service becomes a network endpoint for either other services # or maybe external users to connect to (eg browser) selector: app: api-gateway
NAME DESIRED CURRENT READY AGE replicaset.apps/api-gateway-5d445d6f69 1 1 1 82s replicaset.apps/api-gateway-6d7dccc464 0 0 0 11m replicaset.apps/mongodb-5559556bf 1 1 1 20m replicaset.apps/position-simulator-549554f4d9 0 0 0 11m replicaset.apps/position-simulator-7ffd4f8f68 1 1 1 82s replicaset.apps/position-tracker-5ff4fb7479 1 1 1 11m replicaset.apps/queue-75f4ddd795 1 1 1 82s replicaset.apps/queue-b46577b46 0 0 0 11m replicaset.apps/webapp-689dd9b4f4 1 1 1 82s replicaset.apps/webapp-6cdd565c5 0 0 0 11m [ec2-user@ip-172-31-4-9 ~]$ kubectl log -f pod/position-tracker-5ff4fb7479-jjj9f log is DEPRECATED and will be removed in a future version. Use logs instead. 2019-06-07 07:42:40.878 ERROR 1 --- [enerContainer-1] o.s.j.l.DefaultMessageListenerContainer : Could not refresh JMS Connection for destination 'positionQueue' - retrying using FixedBackOff{interval=5000, currentAttempts=15, maxAttempts=unlimited}. Cause: Could not connect to broker URL: tcp://fleetman-queue.default.svc.cluster.local:61616. Reason: java.net.SocketException: Socket closed 2019-06-07 07:42:46.002 INFO 1 --- [enerContainer-1] o.s.j.l.DefaultMessageListenerContainer : Successfully refreshed JMS Connection 2019-06-07 07:42:47.440 INFO 1 --- [nio-8080-exec-8] org.mongodb.driver.connection : Opened connection [connectionId{localValue:3, serverValue:3}] to fleetman-mongodb.default.svc.cluster.local:27017 illljffkkkkerror: unexpected EOF
Docker swarm uses a concept called a rooting, or routing, mesh to find the node that your web application is running on. None of that is used here, it uses a standard AWS load balancer to find the correct node.
Setting up a real Domain Name
Add a CNAME record in your own domain to ELB address.
Surviving Node Failure
Requirement
Even in the vent of a Node (or Availability Zone) failure, the web site must be accessible
It doesn’t matter if reports from vehicles stop coming in, as long as service is restored within a few minutes
For our example, take the queue pod, give it two replicas and therefore, in the event of a node failure, one of the nodes will always survive. Unfortunately you can’t do that because this particular pod, the queue pod is stateful. In other words, it contains data. And because it contains data, if you replicate it, you’re going to end up with a kind of a split brain situation, where half the data is in one part, half the data is in the other part. And all kinds of chaos will follow on from that. Really what you’re aiming for with any pod is to make it stateless, so it’s not holding data.
State Store: Required value: Please set the --state flag or export KOPS_STATE_STORE. For example, a valid value follows the format s3://<bucket>. You can find the supported stores in https://github.com/kubernetes/kops/blob/master/docs/state.md. [ec2-user@ip-172-31-4-9 ~]$ export KOPS_STATE_STORE=s3://stanzhou-state-storage [ec2-user@ip-172-31-4-9 ~]$ kops delete cluster --name ${NAME} --yes TYPE NAME ID autoscaling-config master-ap-southeast-2a.masters.fleetman.k8s.local-20190607062844 master-ap-southeast-2a.masters.fleetman.k8s.local-20190607062844 autoscaling-config nodes.fleetman.k8s.local-20190607062844 nodes.fleetman.k8s.local-20190607062844 autoscaling-group master-ap-southeast-2a.masters.fleetman.k8s.local master-ap-southeast-2a.masters.fleetman.k8s.local autoscaling-group nodes.fleetman.k8s.local nodes.fleetman.k8s.local dhcp-options fleetman.k8s.local dopt-0a0be88814a0c83a9 iam-instance-profile masters.fleetman.k8s.local masters.fleetman.k8s.local iam-instance-profile nodes.fleetman.k8s.local nodes.fleetman.k8s.local iam-role masters.fleetman.k8s.local masters.fleetman.k8s.local iam-role nodes.fleetman.k8s.local nodes.fleetman.k8s.local instance master-ap-southeast-2a.masters.fleetman.k8s.local i-012c7c446b65343d4 instance nodes.fleetman.k8s.local i-07583ee103342be9a instance nodes.fleetman.k8s.local i-079cea61e4a7736b9 instance nodes.fleetman.k8s.local i-0bf949dbd290d81c3 internet-gateway fleetman.k8s.local igw-0a939d66d6e93e0d5 keypair kubernetes.fleetman.k8s.local-fc:11:5b:a8:1d:16:4a:36:36:15:2d:9f:f3:69:d2:0a kubernetes.fleetman.k8s.local-fc:11:5b:a8:1d:16:4a:36:36:15:2d:9f:f3:69:d2:0a load-balancer a660e0c5e88f611e9b50902985f98381 load-balancer api.fleetman.k8s.local api-fleetman-k8s-local-tkmafs route-table fleetman.k8s.local rtb-06b591f24a01973f6 security-group sg-07b79756088cf753c security-group api-elb.fleetman.k8s.local sg-005c9b49b63793004 security-group masters.fleetman.k8s.local sg-07ef00367ce1a7b62 security-group nodes.fleetman.k8s.local sg-01f81918cdbaba212 subnet ap-southeast-2a.fleetman.k8s.local subnet-060ec2db19027cf6a subnet ap-southeast-2b.fleetman.k8s.local subnet-0def003bdbfd97915 subnet ap-southeast-2c.fleetman.k8s.local subnet-0016862a30fe5f443 volume a.etcd-events.fleetman.k8s.local vol-0d21c97044fcc06dd volume a.etcd-main.fleetman.k8s.local vol-0f1c9f3e983c5848a volume fleetman.k8s.local-dynamic-pvc-b2ed286e-88f3-11e9-b509-02985f983814 vol-074914e494e4b656d vpc fleetman.k8s.local vpc-0775d4b463932d2f7
load-balancer:api-fleetman-k8s-local-tkmafs ok load-balancer:a660e0c5e88f611e9b50902985f98381 ok autoscaling-group:nodes.fleetman.k8s.local ok keypair:kubernetes.fleetman.k8s.local-fc:11:5b:a8:1d:16:4a:36:36:15:2d:9f:f3:69:d2:0a ok internet-gateway:igw-0a939d66d6e93e0d5 still has dependencies, will retry autoscaling-group:master-ap-southeast-2a.masters.fleetman.k8s.local ok instance:i-012c7c446b65343d4 ok instance:i-07583ee103342be9a ok instance:i-0bf949dbd290d81c3 ok instance:i-079cea61e4a7736b9 ok iam-instance-profile:nodes.fleetman.k8s.local ok iam-instance-profile:masters.fleetman.k8s.local ok iam-role:nodes.fleetman.k8s.local ok iam-role:masters.fleetman.k8s.local ok subnet:subnet-0def003bdbfd97915 still has dependencies, will retry subnet:subnet-0016862a30fe5f443 still has dependencies, will retry autoscaling-config:nodes.fleetman.k8s.local-20190607062844 ok volume:vol-0d21c97044fcc06dd still has dependencies, will retry autoscaling-config:master-ap-southeast-2a.masters.fleetman.k8s.local-20190607062844 ok volume:vol-0f1c9f3e983c5848a still has dependencies, will retry subnet:subnet-060ec2db19027cf6a still has dependencies, will retry volume:vol-074914e494e4b656d still has dependencies, will retry security-group:sg-005c9b49b63793004 still has dependencies, will retry security-group:sg-01f81918cdbaba212 still has dependencies, will retry security-group:sg-07b79756088cf753c still has dependencies, will retry security-group:sg-07ef00367ce1a7b62 still has dependencies, will retry Not all resources deleted; waiting before reattempting deletion route-table:rtb-06b591f24a01973f6 internet-gateway:igw-0a939d66d6e93e0d5 security-group:sg-005c9b49b63793004 subnet:subnet-060ec2db19027cf6a security-group:sg-07b79756088cf753c subnet:subnet-0016862a30fe5f443 subnet:subnet-0def003bdbfd97915 security-group:sg-07ef00367ce1a7b62 volume:vol-0d21c97044fcc06dd dhcp-options:dopt-0a0be88814a0c83a9 volume:vol-074914e494e4b656d security-group:sg-01f81918cdbaba212 volume:vol-0f1c9f3e983c5848a vpc:vpc-0775d4b463932d2f7 subnet:subnet-060ec2db19027cf6a still has dependencies, will retry subnet:subnet-0def003bdbfd97915 still has dependencies, will retry subnet:subnet-0016862a30fe5f443 still has dependencies, will retry volume:vol-074914e494e4b656d still has dependencies, will retry volume:vol-0f1c9f3e983c5848a still has dependencies, will retry internet-gateway:igw-0a939d66d6e93e0d5 still has dependencies, will retry volume:vol-0d21c97044fcc06dd still has dependencies, will retry security-group:sg-01f81918cdbaba212 still has dependencies, will retry security-group:sg-005c9b49b63793004 still has dependencies, will retry security-group:sg-07ef00367ce1a7b62 still has dependencies, will retry security-group:sg-07b79756088cf753c still has dependencies, will retry Not all resources deleted; waiting before reattempting deletion volume:vol-074914e494e4b656d security-group:sg-01f81918cdbaba212 volume:vol-0f1c9f3e983c5848a vpc:vpc-0775d4b463932d2f7 route-table:rtb-06b591f24a01973f6 internet-gateway:igw-0a939d66d6e93e0d5 security-group:sg-005c9b49b63793004 subnet:subnet-060ec2db19027cf6a subnet:subnet-0016862a30fe5f443 security-group:sg-07b79756088cf753c volume:vol-0d21c97044fcc06dd subnet:subnet-0def003bdbfd97915 security-group:sg-07ef00367ce1a7b62 dhcp-options:dopt-0a0be88814a0c83a9 subnet:subnet-060ec2db19027cf6a still has dependencies, will retry subnet:subnet-0def003bdbfd97915 still has dependencies, will retry volume:vol-0f1c9f3e983c5848a still has dependencies, will retry volume:vol-0d21c97044fcc06dd still has dependencies, will retry internet-gateway:igw-0a939d66d6e93e0d5 still has dependencies, will retry volume:vol-074914e494e4b656d still has dependencies, will retry subnet:subnet-0016862a30fe5f443 still has dependencies, will retry security-group:sg-07b79756088cf753c still has dependencies, will retry security-group:sg-01f81918cdbaba212 still has dependencies, will retry security-group:sg-07ef00367ce1a7b62 still has dependencies, will retry security-group:sg-005c9b49b63793004 still has dependencies, will retry Not all resources deleted; waiting before reattempting deletion route-table:rtb-06b591f24a01973f6 internet-gateway:igw-0a939d66d6e93e0d5 security-group:sg-005c9b49b63793004 subnet:subnet-060ec2db19027cf6a subnet:subnet-0016862a30fe5f443 security-group:sg-07b79756088cf753c volume:vol-0d21c97044fcc06dd subnet:subnet-0def003bdbfd97915 security-group:sg-07ef00367ce1a7b62 dhcp-options:dopt-0a0be88814a0c83a9 volume:vol-074914e494e4b656d security-group:sg-01f81918cdbaba212 volume:vol-0f1c9f3e983c5848a vpc:vpc-0775d4b463932d2f7 subnet:subnet-0def003bdbfd97915 still has dependencies, will retry subnet:subnet-060ec2db19027cf6a still has dependencies, will retry internet-gateway:igw-0a939d66d6e93e0d5 still has dependencies, will retry volume:vol-074914e494e4b656d still has dependencies, will retry volume:vol-0d21c97044fcc06dd ok volume:vol-0f1c9f3e983c5848a ok security-group:sg-01f81918cdbaba212 still has dependencies, will retry subnet:subnet-0016862a30fe5f443 ok security-group:sg-07b79756088cf753c still has dependencies, will retry security-group:sg-07ef00367ce1a7b62 still has dependencies, will retry security-group:sg-005c9b49b63793004 still has dependencies, will retry Not all resources deleted; waiting before reattempting deletion dhcp-options:dopt-0a0be88814a0c83a9 volume:vol-074914e494e4b656d security-group:sg-01f81918cdbaba212 vpc:vpc-0775d4b463932d2f7 route-table:rtb-06b591f24a01973f6 internet-gateway:igw-0a939d66d6e93e0d5 security-group:sg-005c9b49b63793004 subnet:subnet-060ec2db19027cf6a security-group:sg-07b79756088cf753c subnet:subnet-0def003bdbfd97915 security-group:sg-07ef00367ce1a7b62 internet-gateway:igw-0a939d66d6e93e0d5 still has dependencies, will retry volume:vol-074914e494e4b656d ok subnet:subnet-060ec2db19027cf6a still has dependencies, will retry security-group:sg-01f81918cdbaba212 still has dependencies, will retry security-group:sg-005c9b49b63793004 still has dependencies, will retry security-group:sg-07b79756088cf753c still has dependencies, will retry subnet:subnet-0def003bdbfd97915 ok security-group:sg-07ef00367ce1a7b62 ok Not all resources deleted; waiting before reattempting deletion security-group:sg-01f81918cdbaba212 vpc:vpc-0775d4b463932d2f7 route-table:rtb-06b591f24a01973f6 internet-gateway:igw-0a939d66d6e93e0d5 security-group:sg-005c9b49b63793004 subnet:subnet-060ec2db19027cf6a security-group:sg-07b79756088cf753c dhcp-options:dopt-0a0be88814a0c83a9 subnet:subnet-060ec2db19027cf6a still has dependencies, will retry security-group:sg-005c9b49b63793004 still has dependencies, will retry security-group:sg-07b79756088cf753c still has dependencies, will retry internet-gateway:igw-0a939d66d6e93e0d5 ok security-group:sg-01f81918cdbaba212 ok Not all resources deleted; waiting before reattempting deletion dhcp-options:dopt-0a0be88814a0c83a9 vpc:vpc-0775d4b463932d2f7 route-table:rtb-06b591f24a01973f6 security-group:sg-005c9b49b63793004 subnet:subnet-060ec2db19027cf6a security-group:sg-07b79756088cf753c subnet:subnet-060ec2db19027cf6a ok security-group:sg-005c9b49b63793004 ok security-group:sg-07b79756088cf753c ok route-table:rtb-06b591f24a01973f6 ok vpc:vpc-0775d4b463932d2f7 ok dhcp-options:dopt-0a0be88814a0c83a9 ok Deleted kubectl config for fleetman.k8s.local
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/monitoring-prometheus-node-exporter 4 4 4 4 4 <none> 4m27s
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE deployment.apps/monitoring-grafana 1 1 1 1 4m27s deployment.apps/monitoring-kube-state-metrics 1 1 1 1 4m27s deployment.apps/monitoring-prometheus-oper-operator 1 1 1 1 4m27s
NAME DESIRED CURRENT READY AGE replicaset.apps/monitoring-grafana-c768bb86f 1 1 1 4m27s replicaset.apps/monitoring-kube-state-metrics-6488587c6 1 1 1 4m27s replicaset.apps/monitoring-prometheus-oper-operator-7b54f56766 1 1 1 4m27s
NAME DESIRED CURRENT AGE statefulset.apps/alertmanager-monitoring-prometheus-oper-alertmanager 1 1 4m statefulset.apps/prometheus-monitoring-prometheus-oper-prometheus 1 1 3m53s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/monitoring-prometheus-node-exporter 4 4 4 4 4 <none> 17m
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE deployment.apps/monitoring-grafana 1 1 1 1 17m deployment.apps/monitoring-kube-state-metrics 1 1 1 1 17m deployment.apps/monitoring-prometheus-oper-operator 1 1 1 1 17m
NAME DESIRED CURRENT READY AGE replicaset.apps/monitoring-grafana-c768bb86f 1 1 1 17m replicaset.apps/monitoring-kube-state-metrics-6488587c6 1 1 1 17m replicaset.apps/monitoring-prometheus-oper-operator-7b54f56766 1 1 1 17m
NAME DESIRED CURRENT AGE statefulset.apps/alertmanager-monitoring-prometheus-oper-alertmanager 1 1 16m statefulset.apps/prometheus-monitoring-prometheus-oper-prometheus 1 1 16m
Working with Grafana
change back from LoadBalancer to ClusterIP on service/monitoring-prometheus-oper-prometheus change from ClusterIP to LoadBalancer on service/monitoring-grafana
