In the first part of this series, we explored how AppSignal can significantly enhance the robustness of Open edX platforms. We saw the challenges that Open edX faces as it scales and how AppSignal's features — including real-time performance monitoring and automated error tracking — provide essential tools for DevOps teams. Our walkthrough covered the initial setup and integration of AppSignal with Open edX, highlighting the immediate benefits of this powerful observability framework.
In this second post, we'll dive deeper into the advanced monitoring capabilities that AppSignal offers. This includes streaming logs from Open edX to AppSignal, monitoring background workers with Celery, and tracking Redis queries. We will demonstrate how these features can be leveraged to address specific operational challenges, ensuring that our learning platform remains fail-safe under varying circumstances.
By the end of this article, you will know how to utilize AppSignal to its full potential in maintaining and improving the performance and reliability of your Open edX platform.
One of AppSignal's strongest features is centralized log management.
Commonly at Open edX, the support team reports an issue with the site, and an engineer can SSH into the server right away to check for Nginx, Mongo, MySQL, and Open edX Application logs.
A centralized storage place that houses logs without the need for you to SSH into the server is a really powerful feature. We can also set up notifications based on an issue's severity.
Now let's see how we can stream our logs from Open edX to AppSignal.
Under the Logging section, click on Manage sources and create a new source, with HTTP as the platform and JSON as the format. After creating the source, AppSignal provides an endpoint and API KEY that we can POST our logs to.
To have more control over log transmission, we can write a simple Python script that reads logs from our local Open edX, pre-processes them, and moves the important ones to AppSignal. For example, I wrote the following script to move only ERROR logs to AppSignal (skipping INFO and WARNING logs):
import requests
import json
from datetime import datetime
import logging
# Setup logging configuration
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
# File to keep track of the last processed line
log_pointer_file = '/root/.local/share/tutor/data/lms/logs/processed.log'
log_file = '/root/.local/share/tutor/data/lms/logs/all.log'
# APpSignal API KEY
api_key = "MY-API-KEY" # Replace with your actual API key
# URL to post the logs
url = f'https://appsignal-endpoint.net/logs?api_key={api_key}'
def read_last_processed():
try:
with open(log_pointer_file, 'r') as file:
content = file.read().strip()
last_processed = int(content) if content else 0
logging.info(f"Last processed line number read: {last_processed}")
return last_processed
except (FileNotFoundError, ValueError) as e:
logging.error(f"Could not read from log pointer file: {e}")
return 0
def update_last_processed(line_number):
try:
with open(log_pointer_file, 'w') as file:
file.write(str(line_number))
logging.info(f"Updated last processed to line number: {line_number}")
except Exception as e:
logging.error(f"Could not update log pointer file: {e}")
def parse_log_line(line):
if 'ERROR' in line:
parts = line.split('ERROR', 1)
timestamp = parts[0].strip()
message_parts = parts[1].strip().split(' - ', 1)
message = message_parts[1] if len(message_parts) > 1 else ''
attributes_part = message_parts[0].strip('[]').split('] [')
# Flatten attributes into a dictionary with string keys and values
attributes = {}
for attr in attributes_part:
key_value = attr.split(None, 1)
if len(key_value) == 2:
key, value = key_value
key = key.rstrip(']:').replace(' ', '_').replace('.', '_') # Replace spaces and dots in keys
if len(key) last_processed:
json_data = parse_log_line(line)
if json_data:
response_code = post_logs(json_data)
if response_code == 200:
update_last_processed(i)
else:
logging.warning(f"Failed to post log, HTTP status code: {response_code}")
if __name__ == '__main__':
logging.info("Starting log processing script.")
process_logs()
logging.info("Finished log processing.")
Here's how the script works:
Important: Please make sure you don't send any personally identifiable information to the endpoint.
Now run this script and it should move ERROR logs to AppSignal:
You can also create a new trigger to notify you as soon as a specific event like ERROR happens:
Celery (a distributed task queue) is a vital component of Open edX, responsible for managing background tasks such as grading, certificate generation, and bulk email dispatch. Redis often acts as the broker for Celery, managing task queues. Both systems are essential for asynchronous processing and can become bottlenecks during periods of high usage. Monitoring these services with AppSignal provides valuable insights into task execution and queue health, helping you preemptively address potential issues. Let's see how we can monitor Celery and Redis.
First, install the necessary packages. Add the following to the OPENEDX_EXTRA_PIP_REQUIREMENTS variable in the .local/share/tutor/config.yml file:
import requests
import json
from datetime import datetime
import logging
# Setup logging configuration
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
# File to keep track of the last processed line
log_pointer_file = '/root/.local/share/tutor/data/lms/logs/processed.log'
log_file = '/root/.local/share/tutor/data/lms/logs/all.log'
# APpSignal API KEY
api_key = "MY-API-KEY" # Replace with your actual API key
# URL to post the logs
url = f'https://appsignal-endpoint.net/logs?api_key={api_key}'
def read_last_processed():
try:
with open(log_pointer_file, 'r') as file:
content = file.read().strip()
last_processed = int(content) if content else 0
logging.info(f"Last processed line number read: {last_processed}")
return last_processed
except (FileNotFoundError, ValueError) as e:
logging.error(f"Could not read from log pointer file: {e}")
return 0
def update_last_processed(line_number):
try:
with open(log_pointer_file, 'w') as file:
file.write(str(line_number))
logging.info(f"Updated last processed to line number: {line_number}")
except Exception as e:
logging.error(f"Could not update log pointer file: {e}")
def parse_log_line(line):
if 'ERROR' in line:
parts = line.split('ERROR', 1)
timestamp = parts[0].strip()
message_parts = parts[1].strip().split(' - ', 1)
message = message_parts[1] if len(message_parts) > 1 else ''
attributes_part = message_parts[0].strip('[]').split('] [')
# Flatten attributes into a dictionary with string keys and values
attributes = {}
for attr in attributes_part:
key_value = attr.split(None, 1)
if len(key_value) == 2:
key, value = key_value
key = key.rstrip(']:').replace(' ', '_').replace('.', '_') # Replace spaces and dots in keys
if len(key) last_processed:
json_data = parse_log_line(line)
if json_data:
response_code = post_logs(json_data)
if response_code == 200:
update_last_processed(i)
else:
logging.warning(f"Failed to post log, HTTP status code: {response_code}")
if __name__ == '__main__':
logging.info("Starting log processing script.")
process_logs()
logging.info("Finished log processing.")
It should look like the following:
- opentelemetry-instrumentation-celery==0.45b0 - opentelemetry-instrumentation-redis==0.45b0
As you can see, we are installing opentelemetry packages for Celery and Redis.
Now, we can instrument Celery with worker_process_init to report its metrics to AppSignal.
Heading back to our dashboard in AppSignal, we should see Celery and Redis reports in the Performance section, with background as the namespace.
For Redis queries, you can click on Slow queries:
In this section, we'll revisit the initial issues outlined in part one of this series and apply practical AppSignal monitoring solutions to ensure our Open edX platform stays robust and reliable. Here’s a breakdown.
Let's begin by assessing overall site performance. In the Performance section, under the Issue list, we can see key metrics for all visited URLs:
Now let's order all the actions based on the mean. Any item higher than 1 second should be considered a red flag:
As we see, Celery tasks to rescore and reset student attempts, LMS requests to show course content, and some APIs are taking more than 1 second. Also, we should note that this is only for one active user. If we have more concurrent users, this response time will go up. Our first solution is to add more resources to the server (CPU and memory) and do another performance test.
After identifying actions with mean response times exceeding 1 second, consider performance optimization strategies such as:
We talked about anomaly detection and host monitoring in the previous article. Let's add triggers for the following items:
Two really important metrics for our platform are our number of active users and enrollments. Let's see how we can measure these metrics using AppSignal.
First, add increment_counter to common/djangoapps/student/views/management.py and openedx/core/djangoapps/user_authn/views/login.py to track and increment the number of logins and enrollments when there is a new event.
Now let's log in to Open edX and enroll in a course. Next, let's head to our dashboard in AppSignal. Click on Add dashboard, then Create dashboard, and give it a name and description.
Click on Add graph, enter Active Users as the title, select Add Metric and use login_count:
Your dashboard should look like the following:
You can follow the same steps to add a graph for enrollments using an enrollment_count metric.
To make sure our site's styling stays consistent, let's add a new uptime check for static/tailwind/css/lms-main-v1.css and get notified when a URL is broken:
In the Error section of the dashboard, we can view all errors, set up notifications for them, and work on fixes as soon as possible to prevent users from being negatively impacted.
In the Monitor Celery and Redis section of this article, we saw how to instrument Celery and Redis using AppSignal. Let's follow the same steps to enable AppSignal so we can see graded tasks. In the lms/djangoapps/grades/tasks.py file, add the following lines:
We should now see a couple of items to grade under Performance -> Issue list.
As you can see, recalculate_subsection_grade_v3 (our main grading Celery task) takes 212 milliseconds. For regrading, lms.djangoapps.instructor_task.tasks.reset_problem_attempts and lms.djangoapps.instructor_task.tasks.rescore_problem take 1.77 seconds.
In this two-part series, we integrated AppSignal with Open edX to fortify its monitoring capabilities. We started with the basics — setting up and understanding the fundamental offerings of AppSignal, including error tracking and performance monitoring.
In this article, we tackled how to efficiently stream logs from various Open edX services to AppSignal, ensuring all relevant information was centralized and readily accessible. We also monitored crucial asynchronous tasks handled by Celery and Redis.
Finally, we addressed some real-world challenges, such as slow site responses, resource bottlenecks during high enrollment periods, and unexpected issues like broken styling.
By now, you should have a comprehensive understanding of how to leverage AppSignal to not just monitor, but also significantly improve, the performance and reliability of your Open edX platform.
If you have any questions about Open edX or need further assistance, feel free to visit cubite.io or reach out to me directly at amir@cubite.io.
P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!
The above is the detailed content of Advanced Open edX Monitoring with AppSignal for Python. For more information, please follow other related articles on the PHP Chinese website!
![[Web front-end] Node.js quick start](https://img.php.cn/upload/course/000/000/067/662b5d34ba7c0227.png)
