Kubernetes, Local to Production with Django: 4 - Celery with Redis and Flower

$ minikube start
$ eval $(minikube docker-env)
$ minikube dashboard
$ kubernetes_django/deploy/..
# Setup project
$ git clone git@github.com:gitumarkk/kubernetes_django.git
$ cd kubernetes_django/deploy/kubernetes/
$ git checkout part_4-redis-celery
# Configure minikube
$ minikube start
$ eval $(minikube docker-env)
$ minikube dashboard # Open dashboard in new browser
# Apply Manifests
$ kubectl apply -f postgres/ # See dashboard in browser
$ kubectl apply -f redis/
$ kubectl apply -f django/
$ kubectl apply -f celery/
$ kubectl apply -f flower/
# Show services in browser
$ minikube service django-service # Wait if not ready
$ minikube service flower-service
# Delete cluster when done
$ minikube delete

1. Background

2. Deploying Redis

$ kubectl apply -f redis/deployment.yaml
$ kubectl apply -f redis/service.yaml

3. Updating the Code base

celery==4.4.7
Django==3.1.2
django-health-check==3.14.3
django-redis==4.12.1
flower==0.9.5
gunicorn==20.0.4
kombu==4.6.10
psycopg2-binary==2.9.6
redis==3.5.3
# REDIS
REDIS_URL = "redis://{host}:{port}/1".format(
host=os.getenv('REDIS_HOST', 'localhost'),
port=os.getenv('REDIS_PORT', '6379')
)
# CELERY
CELERY_BROKER_URL = REDIS_URL
CELERY_RESULT_BACKEND = REDIS_URL
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
# CACHE
CACHES = {
"default": {
"BACKEND": "django_redis.cache.RedisCache",
"LOCATION": REDIS_URL,
"OPTIONS": {
"CLIENT_CLASS": "django_redis.client.DefaultClient"
},
"KEY_PREFIX": "example"
}
}
# http://docs.celeryproject.org/en/latest/django/first-steps-with-django.htmlfrom __future__ import absolute_import, unicode_literals
import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', '<mysite>.settings')app = Celery('<mysite>')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
app.conf.beat_schedule = {
'display_time-20-seconds': {
'task': 'demoapp.tasks.display_time',
'schedule': 20.0
},
}
@app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
from __future__ import absolute_import, unicode_literals# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app
__all__ = ['celery_app']
from datetime import datetime
from celery import shared_task
@shared_task
def display_time():
print("The time is %s :" % str(datetime.now()))
return True
$ eval $(minikube docker-env)
$ docker build -t <IMAGE_NAME>:<TAG>

4. Deploying Django with Celery

$ kubectl apply -f django/deployment.yaml
$ kubectl apply -f django/worker-deployment.yaml
$ kubectl apply -f django/celery-beat-deployment.yaml
$ kubectl apply -f flower/worker-deployment.yaml
$ kubectl apply -f flower/service.yaml

5. Results

$ kubectl get podsNAME                             READY     STATUS    RESTARTS   AGE
celery-beat-fbd55d667-8qczf 1/1 Running 0 5m
celery-worker-7b9849b5d6-ndfjd 1/1 Running 0 5m
django-799d45fb77-7cn48 1/1 Running 0 22m
flower-f7b5479f5-84dhc 1/1 Running 0 19m
postgres-68fdbc869b-rmljm 1/1 Running 3 4d
redis-847499c948-jfvb6 1/1 Running 0 8m
$ kubectl logs celery-beat-fbd55d667-8qczf[2018-01-22 16:51:41,132: INFO/MainProcess] beat: Starting...[2018-01-22 17:21:17,481: INFO/MainProcess] Scheduler: Sending due task display_time-20-seconds (demoapp.tasks.display_time)[2018-01-22 17:21:17,492: DEBUG/MainProcess] demoapp.tasks.display_time sent. id->4f9ea7fa-066d-4cc8-b84a-0231e4357de5[2018-01-22 17:21:17,493: DEBUG/MainProcess] beat: Waking up in 19.97 seconds.
$ kubectl logs celery-worker-7b9849b5d6-ndfjd[2018-01-22 16:51:41,250: INFO/MainProcess] Connected to redis://redis-service:6379/1[2018-01-22 17:21:37,477: INFO/MainProcess] Received task: demoapp.tasks.display_time[4f9ea7fa-066d-4cc8-b84a-0231e4357de5][2018-01-22 17:21:37,478: WARNING/ForkPoolWorker-1] The time is 2018-01-22 17:21:37.478215 :[2018-01-22 17:21:37,478: INFO/ForkPoolWorker-1] Task demoapp.tasks.display_time[4f9ea7fa-066d-4cc8-b84a-0231e4357de5] succeeded in 0.0007850109977880493s: True
$ minikube service django-service
$ minikube service flower-service

6. Conclusion

7. Credits

8. Tutorial Links

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store