MonitoRSSをセルフホストしてRSS FeedをDiscordに通知する
Table of Contents
SaaS版MonitoRSSのFeed制限がきついので、セルフホストしてRSS FeedをDiscordに通知する話
MonitoRSSとは
MonitoRSSとはwebベースで複数のRSSフィードを管理しつつ、Discord botで通知を行える便利なツール。
今まではSaas版MonitoRSSを使っていたが、こちらは無料枠では3つまでしかFeedを追加できないので、少し不便に思っていた。
セルフホストすれば制限がなくなるので、重い腰を上げてマニフェストを書いた。
各Imageの役割
インフラまわり
redis:7-alpine- フィード取得やユーザーフィード周りのキャッシュ・一時データ保存に使う Redis。
rabbitmq:3-management-alpine- 各マイクロサービス間でイベントやジョブを受け渡しするメッセージブローカー。
mongo:8.0/mongo:7.0.2/mongo:latest- RSS、アプリ設定などのMonitoRSSのメインデータを保持するMongoDB。
- 環境毎に違うバージョンが使われているようだ。
postgres:17-alpine/postgres:14.1-alpine- フィード取得ジョブ、ユーザーフィード情報などを保持するPostgreSQL。
- 本番系は
17-alpineが使われている。
chrislusf/seaweedfs- 開発環境で使う S3 互換ストレージ。フィードコンテンツなどのファイルを擬似S3に保存するためのコンテナ。本番では不要。
アプリケーションまわり
ghcr.io/synzen/monitorss-bot-presence:${MONITORSS_VERSION}- Discord上のBotのプレゼンス維持するサービス。
ghcr.io/synzen/monitorss-discord-rest-listener:${MONITORSS_VERSION}- DiscordのREST APIへのリクエストを行うサービス。
ghcr.io/synzen/monitorss-feed-requests:${MONITORSS_VERSION}- RSSフィードの取得リクエストを受け付け・キューイングし、実際のフィード取得処理・結果保存を行うバックエンドサービス。
- 同じイメージを
feed-requests-postgres-migrationでも使い、PostgreSQLのマイグレーションを行っている
ghcr.io/synzen/monitorss-user-feeds:${MONITORSS_VERSION}- ユーザーごとのフィード設定 (どのフィードをどのチャンネルへ投稿するか等)を管理するAPIサービス。
- 同じイメージを
user-feeds-postgres-migrationでも使い、ユーザーフィード用の Postgres スキーマをマイグレーションしている。
ghcr.io/synzen/monitorss-monolith:${MONITORSS_VERSION}monolithサービス本体。Web/APIのエントリーポイントとなり、Mongo/Feed Requests/User Feeds/RabbitMQ など各サービスを束ねるバックエンド。schedule-emitter-serviceとしても同じイメージを利用し、定期的なスケジューラ処理(フィード更新などのトリガー発行)を専用コマンドで実行する。
実作業
Discord Developer Portalでsecret取得
Discord Developer PortalでNew Applicationからbotを作成し、 OAuth2のRedirect URLを設定する。
また、OAuth2のタブからClient IDとClient Secretを控えておく。(後から環境変数として入れる)
Botタブからはbot tokenを生成し、こちらも控えておく。
Manifest
元repoのDockerfileをもとに必要なサービスを定義していく。
MongoDBでストアされる情報がどうやらNASで利用している古いCPUに対応していないみたいだったので、MongoDBは4.4.18を使っている。
Secretには先程控えておいたキーを入力する。
apiVersion: v1
kind: Namespace
metadata:
name: monitorss
---
apiVersion: v1
kind: Secret
metadata:
name: monitorss-discord-secret
namespace: monitorss
type: Opaque
stringData:
discord-bot-token: "xxx"
discord-client-id: "xxx"
discord-client-secret: "xxx"
---
apiVersion: v1
kind: ConfigMap
metadata:
name: monitorss-env
namespace: monitorss
data:
BOT_PRESENCE_STATUS: "online"
BOT_PRESENCE_ACTIVITY_TYPE: ""
BOT_PRESENCE_ACTIVITY_NAME: ""
BOT_PRESENCE_RABBITMQ_URL: "amqp://guest:guest@rabbitmq-broker:5672/"
BACKEND_API_MONGODB_URI: "mongodb://mongo:27017/rss"
BACKEND_API_DEFAULT_REFRESH_RATE_MINUTES: "10"
BACKEND_API_DEFAULT_MAX_USER_FEEDS: "10000"
BACKEND_API_MAX_DAILY_ARTICLES_DEFAULT: "100000"
BACKEND_API_SESSION_SECRET: "xxx"
BACKEND_API_SESSION_SALT: "xxx"
BACKEND_API_LOGIN_REDIRECT_URI: "http://monitorss.4nm1tsu.com"
BACKEND_API_DISCORD_REDIRECT_URI: "http://monitorss.4nm1tsu.com/api/v1/discord/callback-v2"
BACKEND_API_ALLOW_LEGACY_REVERSION: "true"
USER_FEEDS_DELIVERY_RECORD_PERSISTENCE_MONTHS: "1"
USER_FEEDS_ARTICLE_PERSISTENCE_MONTHS: "12"
USER_FEEDS_FEED_REQUESTS_API_URL: "http://feed-requests-service:5000/v1/feed-requests"
USER_FEEDS_FEED_REQUESTS_API_KEY: "feed-requests-api-key"
USER_FEEDS_API_KEY: "user-feeds-api-key"
USER_FEEDS_POSTGRES_URI: "postgres://monitorss:monitorss@feed-requests-postgres-db:5432/userfeeds"
USER_FEEDS_POSTGRES_DATABASE: "userfeeds"
USER_FEEDS_REDIS_URI: "redis://feed-requests-redis-cache:6379"
USER_FEEDS_REDIS_DISABLE_CLUSTER: "true"
USER_FEEDS_API_PORT: "5000"
USER_FEEDS_RABBITMQ_BROKER_URL: "amqp://guest:guest@rabbitmq-broker:5672"
DISCORD_REST_LISTENER_MONGO_URI: "mongodb://mongo:27017/rss?replicaSet=dbrs&directConnection=true"
DISCORD_REST_LISTENER_MAX_REQ_PER_SEC: "40"
DISCORD_REST_LISTENER_RABBITMQ_URI: "amqp://rabbitmq-broker:5672"
FEED_REQUESTS_FEEDS_MONGODB_URI: "mongodb://mongo:27017/rss?replicaSet=dbrs&directConnection=true"
FEED_REQUESTS_FEED_REQUEST_DEFAULT_USER_AGENT: "MonitoRSS [Self-Hosted]/1.0"
FEED_REQUESTS_HISTORY_PERSISTENCE_MONTHS: "1"
FEED_REQUESTS_MAX_FAIL_ATTEMPTS: "11"
FEED_REQUESTS_REQUEST_TIMEOUT_MS: "15000"
FEED_REQUESTS_API_KEY: "feed-requests-api-key"
FEED_REQUESTS_POSTGRES_URI: "postgres://monitorss:monitorss@feed-requests-postgres-db:5432/feed_requests"
FEED_REQUESTS_POSTGRES_SCHEMA: "feedrequests"
FEED_REQUESTS_API_PORT: "5000"
FEED_REQUESTS_REDIS_URI: "redis://feed-requests-redis-cache:6379/0"
FEED_REQUESTS_REDIS_DISABLE_CLUSTER: "true"
FEED_REQUESTS_RABBITMQ_BROKER_URL: "amqp://rabbitmq-broker:5672"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-monitorss-mongo
spec:
storageClassName: monitorss-mongo
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
nfs:
server: xxx.xxx.xxx.xxx
path: /k8s/monitorss/mongo
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-monitorss-mongo
namespace: monitorss
spec:
storageClassName: monitorss-mongo
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-monitorss-postgres
spec:
storageClassName: monitorss-postgres
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
nfs:
server: xxx.xxx.xxx.xxx
path: /k8s/monitorss/postgres
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-monitorss-postgres
namespace: monitorss
spec:
storageClassName: monitorss-postgres
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-monitorss-rabbitmq
spec:
storageClassName: monitorss-rabbitmq
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
nfs:
server: xxx.xxx.xxx.xxx
path: /k8s/monitorss/rabbitmq
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-monitorss-rabbitmq
namespace: monitorss
spec:
storageClassName: monitorss-rabbitmq
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-monitorss-redis
spec:
storageClassName: monitorss-redis
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
nfs:
server: xxx.xxx.xxx.xxx
path: /k8s/monitorss/redis
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-monitorss-redis
namespace: monitorss
spec:
storageClassName: monitorss-redis
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mongo
namespace: monitorss
spec:
replicas: 1
selector:
matchLabels:
app: mongo
template:
metadata:
labels:
app: mongo
spec:
containers:
- name: mongo
image: mongo:4.4.18
ports:
- containerPort: 27017
volumeMounts:
- name: mongo-data
mountPath: /data/db
volumes:
- name: mongo-data
persistentVolumeClaim:
claimName: pvc-monitorss-mongo
---
apiVersion: v1
kind: Service
metadata:
name: mongo
namespace: monitorss
spec:
selector:
app: mongo
ports:
- name: mongo
port: 27017
targetPort: 27017
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: feed-requests-postgres-db
namespace: monitorss
spec:
replicas: 1
selector:
matchLabels:
app: feed-requests-postgres-db
template:
metadata:
labels:
app: feed-requests-postgres-db
spec:
containers:
- name: postgres
image: postgres:17
env:
- name: POSTGRES_DB
value: "feed_requests"
- name: POSTGRES_USER
value: "monitorss"
- name: POSTGRES_PASSWORD
value: "monitorss"
ports:
- containerPort: 5432
volumeMounts:
- name: postgres-data
mountPath: /var/lib/postgresql/data
volumes:
- name: postgres-data
persistentVolumeClaim:
claimName: pvc-monitorss-postgres
---
apiVersion: v1
kind: Service
metadata:
name: feed-requests-postgres-db
namespace: monitorss
spec:
selector:
app: feed-requests-postgres-db
ports:
- name: postgres
port: 5432
targetPort: 5432
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: feed-requests-redis-cache
namespace: monitorss
spec:
replicas: 1
selector:
matchLabels:
app: feed-requests-redis-cache
template:
metadata:
labels:
app: feed-requests-redis-cache
spec:
containers:
- name: redis
image: redis:7
ports:
- containerPort: 6379
volumeMounts:
- name: redis-data
mountPath: /data
volumes:
- name: redis-data
persistentVolumeClaim:
claimName: pvc-monitorss-redis
---
apiVersion: v1
kind: Service
metadata:
name: feed-requests-redis-cache
namespace: monitorss
spec:
selector:
app: feed-requests-redis-cache
ports:
- name: redis
port: 6379
targetPort: 6379
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: rabbitmq-broker
namespace: monitorss
spec:
replicas: 1
selector:
matchLabels:
app: rabbitmq-broker
template:
metadata:
labels:
app: rabbitmq-broker
spec:
containers:
- name: rabbitmq
image: rabbitmq:3-management
env:
- name: RABBITMQ_DEFAULT_USER
value: "guest"
- name: RABBITMQ_DEFAULT_PASS
value: "guest"
ports:
- containerPort: 5672
- containerPort: 15672
volumeMounts:
- name: rabbitmq-data
mountPath: /var/lib/rabbitmq
volumes:
- name: rabbitmq-data
persistentVolumeClaim:
claimName: pvc-monitorss-rabbitmq
---
apiVersion: v1
kind: Service
metadata:
name: rabbitmq-broker
namespace: monitorss
spec:
selector:
app: rabbitmq-broker
ports:
- name: amqp
port: 5672
targetPort: 5672
- name: http
port: 15672
targetPort: 15672
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: bot-presence-service
namespace: monitorss
spec:
replicas: 1
selector:
matchLabels:
app: bot-presence-service
template:
metadata:
labels:
app: bot-presence-service
spec:
containers:
- name: bot-presence-service
image: ghcr.io/synzen/monitorss-bot-presence:main
command: ["node", "dist/main.js"]
envFrom:
- configMapRef:
name: monitorss-env
env:
- name: BOT_PRESENCE_DISCORD_BOT_TOKEN
valueFrom:
secretKeyRef:
name: monitorss-discord-secret
key: discord-bot-token
- name: NODE_ENV
value: "production"
---
apiVersion: v1
kind: Service
metadata:
name: bot-presence-service
namespace: monitorss
spec:
selector:
app: bot-presence-service
ports:
- name: http
port: 3000
targetPort: 3000
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: discord-rest-listener-service
namespace: monitorss
spec:
replicas: 1
selector:
matchLabels:
app: discord-rest-listener-service
template:
metadata:
labels:
app: discord-rest-listener-service
spec:
containers:
- name: discord-rest-listener-service
image: ghcr.io/synzen/monitorss-discord-rest-listener:main
command: ["node", "build/app.js"]
envFrom:
- configMapRef:
name: monitorss-env
env:
- name: DISCORD_REST_LISTENER_BOT_TOKEN
valueFrom:
secretKeyRef:
name: monitorss-discord-secret
key: discord-bot-token
- name: DISCORD_REST_LISTENER_BOT_CLIENT_ID
valueFrom:
secretKeyRef:
name: monitorss-discord-secret
key: discord-client-id
- name: NODE_ENV
value: "production"
---
apiVersion: v1
kind: Service
metadata:
name: discord-rest-listener-service
namespace: monitorss
spec:
selector:
app: discord-rest-listener-service
ports:
- name: http
port: 5000
targetPort: 5000
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: feed-requests-service
namespace: monitorss
spec:
replicas: 1
selector:
matchLabels:
app: feed-requests-service
template:
metadata:
labels:
app: feed-requests-service
spec:
containers:
- name: feed-requests-service
image: ghcr.io/synzen/monitorss-feed-requests:main
command: ["sh", "-c", "npm run migration:up && node dist/main.js"]
envFrom:
- configMapRef:
name: monitorss-env
env:
- name: NODE_ENV
value: "production"
---
apiVersion: v1
kind: Service
metadata:
name: feed-requests-service
namespace: monitorss
spec:
selector:
app: feed-requests-service
ports:
- name: http
port: 5000
targetPort: 5000
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: user-feeds-service
namespace: monitorss
spec:
replicas: 1
selector:
matchLabels:
app: user-feeds-service
template:
metadata:
labels:
app: user-feeds-service
spec:
containers:
- name: user-feeds-service
image: ghcr.io/synzen/monitorss-user-feeds:main
command: ["sh", "-c", "node dist/src/scripts/run-migrations.js && node dist/index.js"]
envFrom:
- configMapRef:
name: monitorss-env
env:
- name: USER_FEEDS_DISCORD_CLIENT_ID
valueFrom:
secretKeyRef:
name: monitorss-discord-secret
key: discord-client-id
- name: USER_FEEDS_DISCORD_API_TOKEN
valueFrom:
secretKeyRef:
name: monitorss-discord-secret
key: discord-bot-token
- name: NODE_ENV
value: "production"
---
apiVersion: v1
kind: Service
metadata:
name: user-feeds-service
namespace: monitorss
spec:
selector:
app: user-feeds-service
ports:
- name: http
port: 5000
targetPort: 5000
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: legacy-feed-bulk-converter-service
namespace: monitorss
spec:
replicas: 1
selector:
matchLabels:
app: legacy-feed-bulk-converter-service
template:
metadata:
labels:
app: legacy-feed-bulk-converter-service
spec:
containers:
- name: legacy-feed-bulk-converter-service
image: ghcr.io/synzen/monitorss-monolith:main
command: ["node", "dist/scripts/legacy-feed-bulk-converter.js"]
envFrom:
- configMapRef:
name: monitorss-env
env:
- name: BACKEND_API_DISCORD_BOT_TOKEN
valueFrom:
secretKeyRef:
name: monitorss-discord-secret
key: discord-bot-token
- name: BACKEND_API_DISCORD_CLIENT_ID
valueFrom:
secretKeyRef:
name: monitorss-discord-secret
key: discord-client-id
- name: BACKEND_API_DISCORD_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: monitorss-discord-secret
key: discord-client-secret
- name: BACKEND_API_USER_FEEDS_API_HOST
value: "http://user-feeds-service:5000"
- name: BACKEND_API_USER_FEEDS_API_KEY
value: "user-feeds-api-key"
- name: BACKEND_API_FEED_REQUESTS_API_HOST
value: "http://feed-requests-service:5000"
- name: BACKEND_API_FEED_REQUESTS_API_KEY
value: "feed-requests-api-key"
- name: BACKEND_API_LOGIN_REDIRECT_URI
value: "http://monitorss.4nm1tsu.com"
- name: BACKEND_API_DISCORD_REDIRECT_URI
value: "http://monitorss.4nm1tsu.com/api/v1/discord/callback-v2"
- name: BACKEND_API_DEFAULT_MAX_FEEDS
value: "10"
- name: BACKEND_API_FEED_USER_AGENT
value: "MonitoRSS"
- name: BACKEND_API_RABBITMQ_BROKER_URL
value: "amqp://guest:guest@rabbitmq-broker:5672/"
- name: NODE_ENV
value: "production"
---
apiVersion: v1
kind: Service
metadata:
name: legacy-feed-bulk-converter-service
namespace: monitorss
spec:
selector:
app: legacy-feed-bulk-converter-service
ports:
- name: http
port: 5001
targetPort: 5001
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: schedule-emitter-service
namespace: monitorss
spec:
replicas: 1
selector:
matchLabels:
app: schedule-emitter-service
template:
metadata:
labels:
app: schedule-emitter-service
spec:
containers:
- name: schedule-emitter-service
image: ghcr.io/synzen/monitorss-monolith:main
command: ["node", "dist/scripts/schedule-emitter.js"]
envFrom:
- configMapRef:
name: monitorss-env
env:
- name: BACKEND_API_DISCORD_BOT_TOKEN
valueFrom:
secretKeyRef:
name: monitorss-discord-secret
key: discord-bot-token
- name: BACKEND_API_DISCORD_CLIENT_ID
valueFrom:
secretKeyRef:
name: monitorss-discord-secret
key: discord-client-id
- name: BACKEND_API_DISCORD_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: monitorss-discord-secret
key: discord-client-secret
- name: BACKEND_API_USER_FEEDS_API_HOST
value: "http://user-feeds-service:5000"
- name: BACKEND_API_USER_FEEDS_API_KEY
value: "user-feeds-api-key"
- name: BACKEND_API_FEED_REQUESTS_API_HOST
value: "http://feed-requests-service:5000"
- name: BACKEND_API_FEED_REQUESTS_API_KEY
value: "feed-requests-api-key"
- name: BACKEND_API_DEFAULT_MAX_FEEDS
value: "10"
- name: BACKEND_API_FEED_USER_AGENT
value: "MonitoRSS"
- name: BACKEND_API_RABBITMQ_BROKER_URL
value: "amqp://guest:guest@rabbitmq-broker:5672/"
- name: NODE_ENV
value: "production"
---
apiVersion: v1
kind: Service
metadata:
name: schedule-emitter-service
namespace: monitorss
spec:
selector:
app: schedule-emitter-service
ports:
- name: http
port: 5002
targetPort: 5002
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: monitorss-monolith
namespace: monitorss
spec:
replicas: 1
selector:
matchLabels:
app: monitorss-monolith
template:
metadata:
labels:
app: monitorss-monolith
spec:
containers:
- name: monitorss-monolith
image: ghcr.io/synzen/monitorss-monolith:main
command: ["node", "dist/main.js"]
envFrom:
- configMapRef:
name: monitorss-env
env:
- name: BACKEND_API_DISCORD_BOT_TOKEN
valueFrom:
secretKeyRef:
name: monitorss-discord-secret
key: discord-bot-token
- name: BACKEND_API_DISCORD_CLIENT_ID
valueFrom:
secretKeyRef:
name: monitorss-discord-secret
key: discord-client-id
- name: BACKEND_API_DISCORD_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: monitorss-discord-secret
key: discord-client-secret
- name: USER_FEEDS_DISCORD_CLIENT_ID
valueFrom:
secretKeyRef:
name: monitorss-discord-secret
key: discord-client-id
- name: USER_FEEDS_DISCORD_API_TOKEN
valueFrom:
secretKeyRef:
name: monitorss-discord-secret
key: discord-bot-token
- name: BACKEND_API_NODE_ENV
value: "local"
- name: BACKEND_API_PORT
value: "8000"
- name: BACKEND_API_DEFAULT_MAX_FEEDS
value: "999999"
- name: BACKEND_API_USER_FEEDS_API_HOST
value: "http://user-feeds-service:5000"
- name: BACKEND_API_FEED_REQUESTS_API_HOST
value: "http://feed-requests-service:5000"
- name: BACKEND_API_FEED_USER_AGENT
value: "MonitoRSS"
- name: BACKEND_API_RABBITMQ_BROKER_URL
value: "amqp://guest:guest@rabbitmq-broker:5672/"
- name: BACKEND_API_USER_FEEDS_API_KEY
value: "user-feeds-api-key"
- name: BACKEND_API_FEED_REQUESTS_API_KEY
value: "feed-requests-api-key"
- name: LOG_LEVEL
value: "info"
- name: NODE_ENV
value: "production"
ports:
- containerPort: 8000
---
apiVersion: v1
kind: Service
metadata:
name: monitorss-monolith
namespace: monitorss
spec:
type: NodePort
selector:
app: monitorss-monolith
ports:
- name: http
port: 8000
targetPort: 8000botをサーバに追加
Discord Developer PortalのOAuth2タブから、 必要なbot permissionをつけて生成されたリンクを踏むとbotをサーバに追加できる。

一生feedが通知されない問題
デプロイ後アクセスすると、いくら待ってもfeedの通知が行われなかった。 feedの追加はエラーなく完了し、手動での通知テストもできるのにだ。
デバッグ中RabbitMQのコンソールにアクセスしたら、ジョブキューが空っぽだった。

RabbitMQ関連の環境変数が怪しいなと思い、調査したら以下の環境変数を設定し忘れて居ることに気づいた。
USER_FEEDS_DISCORD_RABBITMQ_URI: "amqp://rabbitmq-broker:5672"
FEED_REQUESTS_RABBITMQ_BROKER_URL: "amqp://rabbitmq-broker:5672"上記をマニフェストに追加したら無事動き出した。

おわりに
今回はMonitoRSSをセルフホストすることで、feed数制限を解除することができた。
正直やっていることに対して、MonitoRSSの機能はtoo muchな気もするが、まあ使い慣れてるツールなのでヨシ。