Trouble Shooting

Reports Failing to Generate?

This is normally due to the celery workers processes not running in the background on the linux server. You can restart them on both the development and production server by running the DEV/PROD - Start Celery Beat job on Jenkins. If jenkins is down, you’ll have to run the following manually on the api.dectechsports.com server.

Development

sudo su portalws-dev;
cd /mtn/portalws-drive/portalws-dev/portal/code/backend;
workon portalws-dev;
pkill -f celery -u portalws-dev;
celery -A fb_api worker -Q dev-reports -l info --concurrency=2 --detach -n dev-worker@%h --pidfile='celery.pid' --autoscale=2,1 -E
celery -A fb_api worker -Q dev-reports-delayed -l info --concurrency 2 --detach -n dev-worker2@%h --pidfile='celery_delayed.pid' -E  --autoscale=2,1
celery beat -A fb_api --detach

Production

sudo su portalws-prod;
cd /mtn/portalws-drive/portalws-prod/portal/code/backend;
workon portalws-prod;
pkill -f celery -u portalws-prod;
celery -A fb_api worker -Q prod-reports -l info --concurrency=6 --detach -n prod-worker@%h --autoscale=6,1 -E --pidfile='celery.pid'
celery -A fb_api worker -Q prod-reports-delayed -l info --concurrency 6 --detach -n prod-worker2@%h --pidfile='celery_delayed.pid' -E  --autoscale=6,1
celery beat -A fb_api --detach

Celery Doesn’t Restart?

If there is a problem with restarting the celery workers, this could be due to the linux server running into memory problems which causes the rabbitmq or redis server to be killed. You will need to restart these processes by doing the following on the api.dectechsports.com server.

sudo service redis start
sudo rabbitmqctl start

Credits Failing to be Generated?

If this is happening, its probably due to either the check_contract_daily or check_credit_dailys scheduled failing to run as these are the tasks responsible for replenishing the credits. For scheduled tasks to work with celery, you need to have the celery beat working in the background. To start this process, you need to the following on the api.dectechsports.com server:

Development

sudo su portalws-dev;
cd /mtn/portalws-drive/portalws-dev/portal/code/backend;
workon portalws-dev;
celery beat -A fb_api --detach

Production

sudo su portalws-prod;
cd /mtn/portalws-drive/portalws-prod/portal/code/backend;
workon portalws-prod;
celery beat -A fb_api --detach

Portal Directory Disappearing?

This normally happens when the football server restarts and fails to map any external EBS drives. The portal directory (/mtn/portalws-drive) is stored on a EBS drive mapped to the EC2 at the api.dectechsports.com domain. This problem can be solved by re mapping the drive on the AWS console.