Sentry 監控 - Snuba 資料中臺本地開發環境配置實戰
克隆倉庫
分別克隆 getsentry/sentry
與 getsentry/snuba
:
git clone http://github.com/getsentry/sentry.git git clone http://github.com/getsentry/snuba.git
安裝系統依賴(以 Mac 為例)
Xcode CLI tools
xcode-select --install
Brewfile
進入 sentry
資料夾,你會看到一個 Brewfile
檔案:
cd sentry
Brewfile
# required to run devservices cask 'docker' brew 'pyenv' # required for pyenv's python-build brew 'openssl' brew 'readline' # required for yarn test -u brew 'watchman' # required to build some of sentry's dependencies brew 'pkgconfig' brew 'libxslt' brew 'libxmlsec1' brew 'geoip' # Currently needed because on Big Sur there's no wheel for it brew 'librdkafka' # direnv isn't defined here, because we have it configured to check for a bootstrapped environment. # If it's installed in the early steps of the setup process, it just leads to confusion. # brew 'direnv' tap 'homebrew/cask' # required for acceptance testing cask 'chromedriver'
如果你本地已經安裝了 Docker Desktop
並且已經啟動,可以把 cask 'docker'
註釋掉。
接下來,執行:
brew bundle --verbose
如果你之前本地沒有 Docker Desktop
,則還需要手動啟動一下它:
open -g -a Docker.app
構建工具鏈
Sentry
依賴於 Python Wheels
(包含二進位制擴充套件模組的包),官方為以下平臺分發:
-
Linux
相容PEP-513
(manylinux1
) -
macOS 10.15
或更高版本
如果您的開發機器沒有執行上述系統之一,則需要安裝 Rust
工具鏈。按照 http://www.rust-lang.org/tools/install
上的說明安裝編譯器和相關工具。安裝後, Sentry
安裝程式將自動使用 Rust
構建所有二進位制模組,無需額外配置。
官方通常會跟蹤最新的穩定 Rust
版本,該版本每六週更新一次。因此,請確保通過偶爾執行來使您的 Rust
工具鏈保持最新:
rustup update stable
Python
Sentry 使用 pyenv 來安裝和管理 Python
版本。它是在您執行 brew bundle
時安裝的。
要安裝所需版本的 Python
,您需要執行以下命令。這將需要一段時間,因為您的計算機實際上正在編譯 Python
!
make setup-pyenv
這裡假設你是 Zsh
使用者。
如果您鍵入 which python
,您應該看到類似 $HOME/.pyenv/shims/python
而不是 /usr/bin/python
的內容。這是因為以下內容已新增到您的啟動指令碼中:
cat ~/.zprofile
,你會看到如下內容:
# MacPorts Installer addition on 2021-10-20_at_11:48:22: adding an appropriate PATH variable for use with MacPorts. export PATH="/opt/local/bin:/opt/local/sbin:$PATH" # Finished adapting your PATH environment variable for use with MacPorts. # It is assumed that pyenv is installed via Brew, so this is all we need to do. eval "$(pyenv init --path)"
虛擬環境
您現在已準備好建立 Python
虛擬環境。執行:
python -m venv .venv
並激活虛擬環境:
source .venv/bin/activate
如果一切正常,執行 which python
現在應該會導致類似 /Users/you/sentry/.venv/bin/python
的結果。
Snuba 配置實戰
啟動 Snuba 相關依賴項容器
cd ../sentry git checkout master git pull source .venv/bin/activate sentry devservices up --exclude=snuba # 11:17:59 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured. # 11:18:01 [INFO] sentry.plugins.github: apps-not-configured # > Pulling image 'postgres:9.6-alpine' # > Pulling image 'yandex/clickhouse-server:20.3.9.70' # > Not starting container 'sentry_relay' because it should be started on-demand with devserver. # > Creating 'sentry_redis' volume # > Creating 'sentry_zookeeper_6' volume # > Creating 'sentry_kafka_6' volume # > Creating container 'sentry_redis' # > Creating container 'sentry_zookeeper' # > Creating container 'sentry_kafka' # > Starting container 'sentry_redis' (listening: ('127.0.0.1', 6379)) # > Starting container 'sentry_kafka' (listening: ('127.0.0.1', 9092)) # > Starting container 'sentry_zookeeper' # > Creating 'sentry_clickhouse' volume # > Creating container 'sentry_clickhouse' # > Creating 'sentry_postgres' volume # > Creating 'sentry_wal2json' volume # > Starting container 'sentry_clickhouse' (listening: ('127.0.0.1', 9000), ('127.0.0.1', 9009), ('127.0.0.1', 8123)) # > Creating container 'sentry_postgres' # > Starting container 'sentry_postgres' (listening: ('127.0.0.1', 5432))
這將在 master
上獲取最新版本的 Sentry
,並調出所有 snuba
的依賴項。
Snuba
主要依賴 clickhouse
, zookeeper
, kafka
, redis
相關容器。
docker ps
檢視一下:
1149a6f6ff23 postgres:9.6-alpine "docker-entrypoint.s…" 3 minutes ago Up 3 minutes 127.0.0.1:5432->5432/tcp sentry_postgres a7f3af7d52bb yandex/clickhouse-server:20.3.9.70 "/entrypoint.sh" 3 minutes ago Up 3 minutes 127.0.0.1:8123->8123/tcp, 127.0.0.1:9000->9000/tcp, 127.0.0.1:9009->9009/tcp sentry_clickhouse 68913ee15c43 confluentinc/cp-zookeeper:6.2.0 "/etc/confluent/dock…" 3 minutes ago Up 3 minutes 2181/tcp, 2888/tcp, 3888/tcp sentry_zookeeper 5a248eb26ed3 confluentinc/cp-kafka:6.2.0 "/etc/confluent/dock…" 3 minutes ago Up 3 minutes 127.0.0.1:9092->9092/tcp sentry_kafka 0573aff7b5af redis:5.0-alpine "docker-entrypoint.s…" 3 minutes ago Up 3 minutes 127.0.0.1:6379->6379/tcp sentry_redis
設定 Snuba 虛擬環境
cd snuba make pyenv-setup python -m venv .venv source .venv/bin/activate pip install --upgrade pip==21.1.3 make develop
檢視遷移列表
snuba migrations list # system # [ ] 0001_migrations # # events # [ ] 0001_events_initial # [ ] 0002_events_onpremise_compatibility # [ ] 0003_errors # [ ] 0004_errors_onpremise_compatibility # [ ] 0005_events_tags_hash_map (blocking) # [ ] 0006_errors_tags_hash_map (blocking) # [ ] 0007_groupedmessages # [ ] 0008_groupassignees # [ ] 0009_errors_add_http_fields # [ ] 0010_groupedmessages_onpremise_compatibility (blocking) # [ ] 0011_rebuild_errors # [ ] 0012_errors_make_level_nullable # [ ] 0013_errors_add_hierarchical_hashes # [ ] 0014_backfill_errors (blocking) # [ ] 0015_truncate_events # # transactions # [ ] 0001_transactions # [ ] 0002_transactions_onpremise_fix_orderby_and_partitionby (blocking) # [ ] 0003_transactions_onpremise_fix_columns (blocking) # [ ] 0004_transactions_add_tags_hash_map (blocking) # [ ] 0005_transactions_add_measurements # [ ] 0006_transactions_add_http_fields # [ ] 0007_transactions_add_discover_cols # [ ] 0008_transactions_add_timestamp_index # [ ] 0009_transactions_fix_title_and_message # [ ] 0010_transactions_nullable_trace_id # [ ] 0011_transactions_add_span_op_breakdowns # [ ] 0012_transactions_add_spans # # discover # [ ] 0001_discover_merge_table # [ ] 0002_discover_add_deleted_tags_hash_map # [ ] 0003_discover_fix_user_column # [ ] 0004_discover_fix_title_and_message # [ ] 0005_discover_fix_transaction_name # [ ] 0006_discover_add_trace_id # [ ] 0007_discover_add_span_id # # outcomes # [ ] 0001_outcomes # [ ] 0002_outcomes_remove_size_and_bytes # [ ] 0003_outcomes_add_category_and_quantity # [ ] 0004_outcomes_matview_additions (blocking) # # metrics # [ ] 0001_metrics_buckets # [ ] 0002_metrics_sets # [ ] 0003_counters_to_buckets # [ ] 0004_metrics_counters # [ ] 0005_metrics_distributions_buckets # [ ] 0006_metrics_distributions # [ ] 0007_metrics_sets_granularity_10 # [ ] 0008_metrics_counters_granularity_10 # [ ] 0009_metrics_distributions_granularity_10 # [ ] 0010_metrics_sets_granularity_1h # [ ] 0011_metrics_counters_granularity_1h # [ ] 0012_metrics_distributions_granularity_1h # [ ] 0013_metrics_sets_granularity_1d # [ ] 0014_metrics_counters_granularity_1d # [ ] 0015_metrics_distributions_granularity_1d # # sessions # [ ] 0001_sessions # [ ] 0002_sessions_aggregates # [ ] 0003_sessions_matview
執行遷移
snuba migrations migrate --force # ...... # 2021-12-01 19:45:57,557 Running migration: 0014_metrics_counters_granularity_1d # 2021-12-01 19:45:57,575 Finished: 0014_metrics_counters_granularity_1d # 2021-12-01 19:45:57,589 Running migration: 0015_metrics_distributions_granularity_1d # 2021-12-01 19:45:57,610 Finished: 0015_metrics_distributions_granularity_1d # 2021-12-01 19:45:57,623 Running migration: 0001_sessions # 2021-12-01 19:45:57,656 Finished: 0001_sessions # 2021-12-01 19:45:57,669 Running migration: 0002_sessions_aggregates # 2021-12-01 19:45:57,770 Finished: 0002_sessions_aggregates # 2021-12-01 19:45:57,792 Running migration: 0003_sessions_matview # 2021-12-01 19:45:57,849 Finished: 0003_sessions_matview # Finished running migrations
檢查遷移
進入 Clickhouse
容器:
docker exec -it sentry_clickhouse clickhouse-client # 執行如下 `sql` 語句: select count() from sentry_local # ClickHouse client version 20.3.9.70 (official build). # Connecting to localhost:9000 as user default. # Connected to ClickHouse server version 20.3.9 revision 54433. # a7f3af7d52bb :) select count() from sentry_local # SELECT count() # FROM sentry_local # ┌─count()─┐ # │ 0 │ # └─────────┘ # 1 rows in set. Elapsed: 0.008 sec. # a7f3af7d52bb :)
檢視相關實體資料集
snuba entities list # Declared Entities: # discover # events # groups # groupassignee # groupedmessage # metrics_sets # metrics_counters # metrics_distributions # outcomes # outcomes_raw # sessions # org_sessions # spans # transactions # discover_transactions # discover_events
啟動開發伺服器
此命令將啟動 api
和所有 Snuba
消費者以從 Kafka
攝取資料:
snuba devserver
轉到 http://localhost:1218/events/snql
,你將會看到一個簡易的查詢 UI
。
- Undermoon - 基於 Redis Cluster Protocol 的自管理 Redis 集群系統
- Rust 寫的 Undermoon Redis 叢集 - Redis Cluster Protocol與Server Proxy
- Sentry SDK 開銷測量(Benchmark)
- Sentry 開發者貢獻指南 - 什麼是 Scope, 什麼是 Hub?
- Sentry 開發者貢獻指南 - 瀏覽器 SDK 整合測試
- Sentry 開發者貢獻指南 - 測試技巧
- Sentry 開發者貢獻指南 - Web API
- Sentry 開發者貢獻指南 - 資料庫遷移
- Sentry 開發者貢獻指南 - 配置 PyCharm
- Sentry 產品指南文件(內附 17 張詳細腦圖)
- Sentry 開發者貢獻指南 - Django Rest Framework(Serializers)
- Sentry 企業級資料安全解決方案 - Relay PII 和資料清理
- Sentry 開發者貢獻指南 - Feature Flag
- Sentry 企業級資料安全解決方案 - Relay 執行模式
- Sentry 企業級資料安全解決方案 - Relay 入門
- Sentry 開發者貢獻指南 - SDK 開發(資料處理)
- Sentry 開發者貢獻指南 - SDK 開發(會話)
- Sentry 開發者貢獻指南 - SDK 開發(客戶端報告)
- Sentry 開發者貢獻指南 - SDK 開發(效能監控:Sentry SDK API 演進)
- Sentry 開發者貢獻指南 - SDK 開發(事件負載)