Linux C Crash Reporting with Sentry
Preparation
sudo apt install docker.io
sudo apt install docker-compose-
curl -SL https://github.com/docker/compose/releases/download/v2.24.6/docker-compose-linux-x86_64 -o $HOME/.local/bin/docker-compose
chmod +x $HOME/.local/bin/docker-compose
sentry server
Install behind proxy
Comment out following two lines in cron/Dockerfile:
RUN if [ -n "${http_proxy}" ]; then echo "Acquire::http::proxy \"${http_proxy}\";" >> /etc/apt/apt.conf; fi
RUN if [ -n "${https_proxy}" ]; then echo "Acquire::https::proxy \"${https_proxy}\";" >> /etc/apt/apt.conf; fi
Pull and create docker images
unset http_proxy HTTP_PROXY https_proxy HTTPS_PROXY
VERSION="24.2.0"
git clone https://github.com/getsentry/self-hosted
cd self-hosted
git checkout ${VERSION}
./install.sh --no-report-self-hosted-issues
-----------------------------------------------------------------
You're all done! Run the following command to get Sentry running:
docker-compose up -d
Configuration
Change system.url-prefix to URL to ip.add.re.ss:9000 in sentry/config.yml:
system.url-prefix: http://192.168.1.89:9000
docker-compose up
This may take several minutes.
Upload debug information files
pip install sentry-cli
sentry-cli debug-files find 8184c616-c413-c0cb-2168-5a9dcbbca27e
8184c616-c413-c0cb-2168-5a9dcbbca27e /[...]/projects/sentry/sentry-native/build/sentry_example [elf]
export SENTRY_ORG=sentry
export SENTRY_PROJECT=helloworld
export SENTRY_AUTH_TOKEN=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
sentry-cli debug-files upload --wait build/sentry_example
> Found 1 debug information file
> Prepared debug information file for upload
> Uploading completed in 0.066s
> Uploaded 1 missing debug information file
> File processing complete:
OK 8184c616-c413-c0cb-2168-5a9dcbbca27e (sentry_example; arm64 library)
Cross compile sentry native sdk for arm64
Build sentry sdk
git clone https://github.com/getsentry/sentry-native
cd sentry-native
git checkout 0.7.0
git submodule update --init --recursive
source /opt/poky/4.0.15/environment-setup-cortexa53-crypto-poky-linux
cmake -B build -DCMAKE_C_COMPILER=aarch64-poky-linux-gcc
cmake --build build --parallel
cmake --install build --prefix install
exa --tree install
install
├── bin
│ └── crashpad_handler
├── include
│ └── sentry.h
└── lib
├── cmake
│ └── sentry
│ ├── sentry-config-version.cmake
│ ├── sentry-config.cmake
│ ├── sentry-targets-relwithdebinfo.cmake
│ ├── sentry-targets.cmake
│ ├── sentry_crashpad-targets-relwithdebinfo.cmake
│ └── sentry_crashpad-targets.cmake
└── libsentry.so
scp install/bin/crashpad_handler root@192.168.1.81:/usr/bin/
scp install/lib/libsentry.so root@192.168.1.81:/usr/lib/
Install required libraries
Add the following to kas-roc-rk3328-cc.yml:
IMAGE_INSTALL:append = "\
libcurl \
libstdc++ \
libunistring \
"
Copy libraries to target board:
export KAS_BUILD_DIR=$(pwd)/build-roc-rk3328-cc
kas build kas-roc-rk3328-cc.yml
scp build-roc-rk3328-cc/tmp/work/roc_rk3328_cc-poky-linux/core-image-minimal/1.0-r0/rootfs/usr/lib/libcurl.so.4 root@192.168.1.81:/usr/lib/
scp build-roc-rk3328-cc/tmp/work/roc_rk3328_cc-poky-linux/core-image-minimal/1.0-r0/rootfs/usr/lib/libstdc++.so.6 root@192.168.1.81:/usr/lib/
scp build-roc-rk3328-cc/tmp/work/roc_rk3328_cc-poky-linux/core-image-minimal/1.0-r0/rootfs/lib/libgcc_s.so.1 root@192.168.1.81:/usr/lib/
scp build-roc-rk3328-cc/tmp/work/roc_rk3328_cc-poky-linux/core-image-minimal/1.0-r0/rootfs/usr/lib/libidn2.so.0 root@192.168.1.81:/usr/lib/
scp build-roc-rk3328-cc/tmp/work/roc_rk3328_cc-poky-linux/core-image-minimal/1.0-r0/rootfs/usr/lib/libssl.so.3 root@192.168.1.81:/usr/lib/
scp build-roc-rk3328-cc/tmp/work/roc_rk3328_cc-poky-linux/core-image-minimal/1.0-r0/rootfs/usr/lib/libunistring.so.2 root@192.168.1.81:/usr/lib/
helloworld
root@roc-rk3328-cc:/usr/bin# SENTRY_DSN=http://f1e3d59bc01b1e5b0c88a95410a2dd43@192.168.1.78:9000/2 sentry_example log stdout capture-event capture-exception raise
Segmentation fault
sentry-cli
Install sentry-cli
Ubuntu:
pip install sentry-cli
macOS:
brew install getsentry/tools/sentry-cli
Configuration
sentry-cli --url http://192.168.1.78:9000 login -g --auth-token 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
This helps you signing in your sentry-cli with an authentication token.
If you do not yet have a token ready we can bring up a browser for you
to create a token now.
Sentry server: 192.168.1.78
Valid org token
Stored token in /Users/fdbai/.sentryclirc
cat $HOME/.sentryclirc
[auth]
token=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
Upload Source Bundles
sentry-cli debug-files bundle-sources build/sentry_example
/[...]/projects/sentry/sentry-native/build/sentry_example.src.zip
sentry-cli debug-files upload build/sentry_example.src.zip
> Found 1 debug information file (1 with embedded sources)
> Prepared debug information file for upload
> Uploading completed in 0.158s
> Uploaded 1 missing debug information file
> File upload complete:
UPLOADED 8184c616-c413-c0cb-2168-5a9dcbbca27e (sentry_example.src.zip; arm64 sources)
Or use --include-sources to skip creating bundle files:
sentry-cli debug-files upload --include-sources build/sentry_example
Examples
sentry-cli debug-files check build/sentry_example
Debug Info File Check
Type: elf library
Contained debug identifiers:
> Debug ID: 8184c616-c413-c0cb-2168-5a9dcbbca27e
Code ID: 16c6848113c4cbc021685a9dcbbca27e28130539
Arch: arm64
Contained debug information:
> symtab, debug, unwind
Usable: yes
sentry-cli releases list
+--------------+-------------------------+------------+--------------+
| Released | Version | New Events | Last Event |
+--------------+-------------------------+------------+--------------+
| (unreleased) | test-example-release | 7 | 11 hours ago |
| (unreleased) | frontend@24.2.0+unknown | 1 | - |
+--------------+-------------------------+------------+--------------+
sentry-cli releases -o sentry -p helloworld list
+--------------+----------------------+------------+--------------+
| Released | Version | New Events | Last Event |
+--------------+----------------------+------------+--------------+
| (unreleased) | test-example-release | 7 | 11 hours ago |
+--------------+----------------------+------------+--------------+
sentry-cli releases info test-example-release
+----------------------+--------------------------------+-------------------------+
| Version | Date created | Last event |
+----------------------+--------------------------------+-------------------------+
| test-example-release | 2024-03-01 13:39:07.728837 UTC | 2024-03-01 14:06:45 UTC |
+----------------------+--------------------------------+-------------------------+
If $HOME/.sentryclirc does not exists, export auth token also works:
export SENTRY_AUTH_TOKEN=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
sentry-cli releases -o sentry -p helloworld list
Relay Server
docker run --rm -it \
-v $(pwd)/config/:/work/.relay/:z \
--entrypoint bash \
getsentry/relay:24.3.0 \
-c 'chown -R relay:relay /work/.relay'
# Generate the configuration
docker run --rm -it \
-v $(pwd)/config/:/work/.relay/:z \
getsentry/relay:24.3.0 \
config init
Initializing relay in /work/.relay
✔ Do you want to create a new config? · Yes, create default config
Generated new credentials
relay id: 2444abd0-7779-4887-9dce-444c56f8d141
public key: hIoJVkTE8_GDBC6ORmPTxuF-nDAQ7Qi2OPChVZiRJhQ
All done!
cat config/config.yml
relay:
mode: managed
upstream: http://192.168.1.62:9000/
host: 0.0.0.0
port: 3000
logging:
level: TRACE
cat docker-compose.yml
services:
relay:
image: getsentry/relay:24.3.0
network_mode: host
ports:
- 3000:3000
volumes:
- type: bind
read_only: true
source: ./config
target: /work/.relay
docker-compose up
[+] Running 2/0
✔ Container relay-relay-1 Created 0.1s
! relay Published ports are discarded when using host network mode 0.0s
Attaching to relay-relay-1
relay-relay-1 | 2024-03-30T13:50:22.162697Z INFO relay::setup: launching relay from config folder /work/.relay
relay-relay-1 | 2024-03-30T13:50:22.162724Z INFO relay::setup: relay mode: managed
relay-relay-1 | 2024-03-30T13:50:22.162727Z INFO relay::setup: relay id: 2444abd0-7779-4887-9dce-444c56f8d141
relay-relay-1 | 2024-03-30T13:50:22.162732Z INFO relay::setup: public key: hIoJVkTE8_GDBC6ORmPTxuF-nDAQ7Qi2OPChVZiRJhQ
relay-relay-1 | 2024-03-30T13:50:22.162735Z INFO relay::setup: log level: trace
relay-relay-1 | 2024-03-30T13:50:22.162744Z INFO relay_server: relay server starting
relay-relay-1 | 2024-03-30T13:50:22.172599Z INFO relay_server::services::outcome: Configured to emit outcomes as client reports
relay-relay-1 | 2024-03-30T13:50:22.172676Z INFO relay_server::services::global_config: global config service starting
relay-relay-1 | 2024-03-30T13:50:22.172717Z INFO relay_server::services::global_config: requesting global config from upstream
relay-relay-1 | 2024-03-30T13:50:22.172745Z INFO relay_server::services::processor: starting 8 envelope processing workers
relay-relay-1 | 2024-03-30T13:50:22.172636Z INFO relay_server::services::upstream: registering with upstream descriptor=http://192.168.1.62:9000/
relay-relay-1 | 2024-03-30T13:50:22.172833Z INFO relay_metrics::router: metrics router started
relay-relay-1 | 2024-03-30T13:50:22.172923Z INFO relay_server::services::relays: key cache started
relay-relay-1 | 2024-03-30T13:50:22.173083Z INFO relay_server::services::outcome: OutcomeProducer started.
relay-relay-1 | 2024-03-30T13:50:22.173096Z INFO relay_server::services::project_cache: project cache started
relay-relay-1 | 2024-03-30T13:50:22.173125Z INFO relay_server::services::outcome_aggregator: outcome aggregator started
relay-relay-1 | 2024-03-30T13:50:22.173187Z INFO relay_server::services::project_cache: waiting for global config
relay-relay-1 | 2024-03-30T13:50:22.173281Z INFO relay_server::services::project_upstream: project upstream cache started
relay-relay-1 | 2024-03-30T13:50:22.173280Z INFO relay_server::services::project_local: project local cache started
relay-relay-1 | 2024-03-30T13:50:22.173412Z INFO relay_server::services::server: spawning http server
relay-relay-1 | 2024-03-30T13:50:22.173419Z INFO relay_server::services::server: listening on http://0.0.0.0:3000/
relay-relay-1 | 2024-03-30T13:50:22.173354Z DEBUG relay_server::utils::garbage: Start garbage collection thread
relay-relay-1 | 2024-03-30T13:50:22.546831Z DEBUG relay_server::services::upstream: got register challenge token="eyJ0aW1lc3RhbXAiOjE3MTE4MDY2MjIsInJlbGF5X2lkIjoiMjQ0NGFiZDAtNzc3OS00ODg3LTlkY2UtNDQ0YzU2ZjhkMTQxIiwicHVibGljX2tleSI6ImhJb0pWa1RFOF9HREJDNk9SbVBUeHVGLW5EQVE3UWkyT1BDaFZaaVJKaFEiLCJyYW5kIjoiMDB3ZnJ5cHZJemJ2bXJ5ZjZWM2VzVjVHdkF3a1hCV2hQWnN6Z29NVnlsWWZUZkdIaVFSWkt6bkUxR0tfd2locmtiOXBrUUtyaDM0LVc4ZFBBSm00M1EifQ:_F7AGcZZtp18cskC-rjPY86Q9vzzGUzx7cy86a_KUw413u5fQILLN-XucezwAGh2ETcwgE1OH8HEqlKZj-1qiA"
relay-relay-1 | 2024-03-30T13:50:22.546882Z DEBUG relay_server::services::upstream: sending register challenge response
relay-relay-1 | 2024-03-30T13:50:23.430135Z INFO relay_server::services::upstream: relay successfully registered with upstream
relay-relay-1 | 2024-03-30T13:50:23.747339Z INFO relay_server::services::global_config: received global config from upstream
Copy the public key of relay server and register it to sentry server:
docker run --rm -it \
-v $(pwd)/config/:/work/.relay/ \
getsentry/relay:24.3.0 \
credentials show
Credentials:
relay id: 2444abd0-7779-4887-9dce-444c56f8d141
public key: hIoJVkTE8_GDBC6ORmPTxuF-nDAQ7Qi2OPChVZiRJhQ
Report crash event to relay server
root@roc-rk3328-cc:/usr/bin# SENTRY_DSN=http://d876ef1d51a5b762eaaffa49cbc9b526@192.168.1.78:3000/2 ./sentry_example log abort
[sentry] INFO using database path "/usr/bin/.sentry-native"
[sentry] DEBUG starting transport
[sentry] DEBUG starting background worker thread
[sentry] DEBUG starting backend
[sentry] DEBUG starting crashpad backend with handler "/usr/bin/crashpad_handler"
[sentry] DEBUG background worker thread started
[sentry] DEBUG using minidump URL "http://192.168.1.78:3000/api/2/minidump/?sentry_client=sentry.native/0.7.0&sentry_key=d876ef1d51a5b762eaaffa49cbc9b526"
[sentry] INFO started crashpad client handler
[sentry] DEBUG processing and pruning old runs
[sentry] INFO flushing session and queue before crashpad handler
[sentry] INFO handing control over to crashpad
Aborted
Troubleshooting
CSRF Verification Failed
CSRF Verification Failed
A required security token was not found or was invalid
sentry-self-hosted-web-1 | 02:28:08 [WARNING] django.security.csrf: Forbidden (Origin checking failed - http://192.168.1.89:9000 does not match any trusted origins.): /auth/login/sentry/ (status_code=403 request=<WSGIRequest: POST '/auth/login/sentry/'>)
Answer sentry/config.yml
system.url-prefix: http://192.168.1.89:9000
No Organization Access
All the users create with command line has this issue, and are not listed in webUI:
docker compose run --rm web createuser --email fudongbai@gmail.com --password 123456 --org-id sentry --superuser
I create superuser with above command because user create was skipped in the installation process:
./install.sh --skip-user-creation --no-report-self-hosted-issues
Answer I think this is a bug, but didn’t find a solution, so I use this command for sentry install:
./install.sh --no-report-self-hosted-issues
A required debug information file was missing
Answer Upload debug info before report crashes to sentry.
authentication encountered error error
docker-compose up
[...]
relay-relay-1 | 2024-03-29T21:39:51.351818Z INFO relay_server::services::server: spawning http server
relay-relay-1 | 2024-03-29T21:39:51.351826Z INFO relay_server::services::server: listening on http://0.0.0.0:3000/
relay-relay-1 | 2024-03-29T21:39:51.351751Z DEBUG relay_server::utils::garbage: Start garbage collection thread
relay-relay-1 | 2024-03-29T21:39:54.352990Z ERROR relay_server::services::upstream: authentication encountered error error=could not send request to upstream error.sources=[error sending request for url (http://192.168.1.62:9000/api/0/relays/register/challenge/): error trying to connect: operation timed out, error trying to connect: operation timed out, operation timed out]
relay-relay-1 | 2024-03-29T21:39:54.353224Z DEBUG relay_server::services::upstream: scheduling authentication retry in 0 seconds
relay-relay-1 | 2024-03-29T21:39:54.354327Z INFO relay_server::services::upstream: registering with upstream descriptor=http://192.168.1.62:9000/
relay-relay-1 | 2024-03-29T21:39:57.355775Z ERROR relay_server::services::upstream: authentication encountered error error=could not send request to upstream error.sources=[error sending request for url (http://192.168.1.62:9000/api/0/relays/register/challenge/): error trying to connect: operation timed out, error trying to connect: operation timed out, operation timed out]
Answer
Add network_mode: host to docker-compose.yml.