Pulsar Standalone服务搭建

Pulsar Standalone服务搭建

说明

简介

Apache Pulsar 是 Apache 软件基金会顶级项目，是下一代云原生分布式消息流平台，集消息、存储、轻量化函数式计算为一体。该系统源于 Yahoo，最初在 Yahoo 内部开发和部署，支持 Yahoo 应用服务平台 140 万个主题，日处理超过 1000 亿条消息。Pulsar 于 2016 年由 Yahoo 开源并捐赠给 Apache 软件基金会进行孵化，2018 年成为 Apache 软件基金会顶级项目。Pulsar 采用计算与存储分离的分层架构设计，支持多租户、持久化存储、多机房跨区域数据复制，具有强一致性、高吞吐、低延时以及高可扩展流数据存储特性。

Topic

Topic是Pulsar的核心概念，表示一个“channel”，Producer可以写入数据，Consumer从中消费数据（Kafka、RocketMQ都是这样）。

Topic名称的URL类似如下的结构：

{persistent|non-persistent}://tenant/namespace/topic

• persistent|non-persistent表示数据是否持久化（Pulsar支持消息持久化和非持久化两种模式）
• Tenant为租户
• Namespace一般聚合一系列相关的Topic，一个租户下可以有多个Namespace

租户和Namespace

上图中Property即为租户，每个租户下可以有多个Namespace，每个Namespace下有多个Topic。

Namespace是Pulsar中的操作单元，包括Topic是配置在Namespace级别的，包括多地域复制，消息过期策略等都是配置在Namespace上的。

订阅模型

Pulsar提供了灵活的消息模型，支持三种订阅类型：

• Exclusive subscription：排他的，只能有一个Consumer，接收一个Topic所有的消息
• Shared subscription：共享的，可以同时存在多个Consumer，每个Consumer处理Topic中一部消息（Shared模型是不保证消息顺序的，Consumer数量可以超过分区的数量）
• Failover subscription：Failover模式，同一时刻只有一个有效的Consumer，其余的Consumer作为备用节点，在Master Consumer不可用后进行替代（看起来适用于数据量小，且解决单点故障的场景）

启动服务

将以下内容保存到docker-compose文件，可启动对应服务。

version: "3"
services:
    portainer:
        image: reg.ji.com/portainer/portainer:1.21.0
        restart: always
        ports:
            - 19000:9000
        volumes:
            - "/var/run/docker.sock:/var/run/docker.sock"
            - "$VOLUME_PATH/portainer/data:/data"

    postgres:
        image: reg.ji.com/postgres-timescaledb-postgis:pg12-2.0.0
        restart: always
        environment:
           #- POSTGRES_DB=prest
            - POSTGRES_PASSWORD=123456
            - TZ=PRC
            - PGTZ=PRC
        ports:
            - 5434:5432
        volumes:
            - "$VOLUME_PATH/postgresql_timescaledb_postgis/data:/var/lib/postgresql/data"
            - "$VOLUME_PATH/postgresql_timescaledb_postgis/init//:/docker-entrypoint-initdb.d/"
        labels:
            - "traefik.enable=false"

    pulsar_manager:
        image: reg.ji.com/apache_pulsar_manager:0.2.0
        restart: always
        environment:
            - SPRING_CONFIGURATION_FILE=/pulsar-manager/pulsar-manager/application.properties
            - REDIRECT_HOST=http://10.10.11.190
            - REDIRECT_PORT=9527
            - DRIVER_CLASS_NAME=org.postgresql.Driver
            - URL=jdbc:postgresql://postgres:5432/pulsar_manager
            - USERNAME=postgres
            - PASSWORD=123456
            - LOG_LEVEL=INFO
            - JWT_TOKEN=eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJteS10ZXN0In0.hE_uDehkv92VS2b1PPkDvkSK48NUZhv2jTG7MiKqdpY
            - SECRET_KEY=data:base64,67r9KG04WWEpzIJ3pxFO3gO76/fzaOnh8ywADBDCcIs=
        ports:
            - 7750:7750
            - 9527:9527
        depends_on:
            - pulsar
            - postgres
        volumes:
            - "$VOLUME_PATH/pulsar-manager/application.properties:/pulsar-manager/pulsar-manager/application.properties"
            - "$VOLUME_PATH/pulsar-manager/my-secret.key:/pulsar-manager/pulsar-manager/my-secret.key"
            - "$VOLUME_PATH/pulsar-manager/data:/data"

    pulsar:
        image: reg.ji.com/apache_pulsar:2.7.2
        restart: always
        command: bin/pulsar standalone
        ports:
            - 6650:6650
            - 8080:8080
        volumes:
            - "$VOLUME_PATH/pulsar/data:/pulsar/data"
            - "$VOLUME_PATH/pulsar/conf:/pulsar/conf"

pulsar-manager数据库初始化文件

--
-- Licensed under the Apache License, Version 2.0 (the "License");
-- you may not use this file except in compliance with the License.
-- You may obtain a copy of the License at
--
--     http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
--

CREATE DATABASE pulsar_manager;

\c pulsar_manager;

CREATE TABLE IF NOT EXISTS environments (
  name varchar(256) NOT NULL,
  broker varchar(1024) NOT NULL,
  CONSTRAINT PK_name PRIMARY KEY (name),
  UNIQUE (broker)
);

CREATE TABLE IF NOT EXISTS topics_stats (
  topic_stats_id BIGSERIAL PRIMARY KEY,
  environment varchar(255) NOT NULL,
  cluster varchar(255) NOT NULL,
  broker varchar(255) NOT NULL,
  tenant varchar(255) NOT NULL,
  namespace varchar(255) NOT NULL,
  bundle varchar(255) NOT NULL,
  persistent varchar(36) NOT NULL,
  topic varchar(255) NOT NULL,
  producer_count BIGINT,
  subscription_count BIGINT,
  msg_rate_in double precision	,
  msg_throughput_in double precision	,
  msg_rate_out double precision	,
  msg_throughput_out double precision	,
  average_msg_size double precision	,
  storage_size double precision	,
  time_stamp BIGINT
);

CREATE TABLE IF NOT EXISTS publishers_stats (
  publisher_stats_id BIGSERIAL PRIMARY KEY,
  producer_id BIGINT,
  topic_stats_id BIGINT NOT NULL,
  producer_name varchar(255) NOT NULL,
  msg_rate_in double precision	,
  msg_throughput_in double precision	,
  average_msg_size double precision	,
  address varchar(255),
  connected_since varchar(128),
  client_version varchar(36),
  metadata text,
  time_stamp BIGINT,
  CONSTRAINT fk_publishers_stats_topic_stats_id FOREIGN KEY (topic_stats_id) References topics_stats(topic_stats_id)
);

CREATE TABLE IF NOT EXISTS replications_stats (
  replication_stats_id BIGSERIAL PRIMARY KEY,
  topic_stats_id BIGINT NOT NULL,
  cluster varchar(255) NOT NULL,
  connected BOOLEAN,
  msg_rate_in double precision	,
  msg_rate_out double precision	,
  msg_rate_expired double precision	,
  msg_throughput_in double precision	,
  msg_throughput_out double precision	,
  msg_rate_redeliver double precision	,
  replication_backlog BIGINT,
  replication_delay_in_seconds BIGINT,
  inbound_connection varchar(255),
  inbound_connected_since varchar(255),
  outbound_connection varchar(255),
  outbound_connected_since varchar(255),
  time_stamp BIGINT,
  CONSTRAINT FK_replications_stats_topic_stats_id FOREIGN KEY (topic_stats_id) References topics_stats(topic_stats_id)
);

CREATE TABLE IF NOT EXISTS subscriptions_stats (
  subscription_stats_id BIGSERIAL PRIMARY KEY,
  topic_stats_id BIGINT NOT NULL,
  subscription varchar(255) NULL,
  msg_backlog BIGINT,
  msg_rate_expired double precision	,
  msg_rate_out double precision	,
  msg_throughput_out double precision	,
  msg_rate_redeliver double precision	,
  number_of_entries_since_first_not_acked_message BIGINT,
  total_non_contiguous_deleted_messages_range BIGINT,
  subscription_type varchar(16),
  blocked_subscription_on_unacked_msgs BOOLEAN,
  time_stamp BIGINT,
  UNIQUE (topic_stats_id, subscription),
  CONSTRAINT FK_subscriptions_stats_topic_stats_id FOREIGN KEY (topic_stats_id) References topics_stats(topic_stats_id)
);

CREATE TABLE IF NOT EXISTS consumers_stats (
  consumer_stats_id BIGSERIAL PRIMARY KEY,
  consumer varchar(255) NOT NULL,
  topic_stats_id BIGINT NOT NUll,
  replication_stats_id BIGINT,
  subscription_stats_id BIGINT,
  address varchar(255),
  available_permits BIGINT,
  connected_since varchar(255),
  msg_rate_out double precision	,
  msg_throughput_out double precision	,
  msg_rate_redeliver double precision	,
  client_version varchar(36),
  time_stamp BIGINT,
  metadata text
);

CREATE TABLE IF NOT EXISTS tokens (
  token_id BIGSERIAL PRIMARY KEY,
  role varchar(256) NOT NULL,
  description varchar(128),
  token varchar(1024) NOT NUll,
  UNIQUE (role)
);

CREATE TABLE IF NOT EXISTS users (
  user_id BIGSERIAL PRIMARY KEY,
  access_token varchar(256),
  name varchar(256) NOT NULL,
  description varchar(128),
  email varchar(256),
  phone_number varchar(48),
  location varchar(256),
  company varchar(256),
  expire BIGINT NOT NULL,
  password varchar(256),
  UNIQUE (name)
);

CREATE TABLE IF NOT EXISTS roles (
  role_id BIGSERIAL PRIMARY KEY,
  role_name varchar(256) NOT NULL,
  role_source varchar(256) NOT NULL,
  description varchar(128),
  resource_id BIGINT NOT NULL,
  resource_type varchar(48) NOT NULL,
  resource_name varchar(48) NOT NULL,
  resource_verbs varchar(256) NOT NULL,
  flag INT NOT NULL
);

CREATE TABLE IF NOT EXISTS tenants (
  tenant_id BIGSERIAL PRIMARY KEY,
  tenant varchar(255) NOT NULL,
  admin_roles varchar(255),
  allowed_clusters varchar(255),
  environment_name varchar(255),
  UNIQUE(tenant)
);

CREATE TABLE IF NOT EXISTS namespaces (
  namespace_id BIGSERIAL PRIMARY KEY,
  tenant varchar(255) NOT NULL,
  namespace varchar(255) NOT NULL,
  UNIQUE(tenant, namespace)
);

CREATE TABLE IF NOT EXISTS role_binding(
  role_binding_id BIGSERIAL PRIMARY KEY,
  name varchar(256) NOT NULL,
  description varchar(256),
  role_id BIGINT NOT NULL,
  user_id BIGINT NOT NULL
);

Pulsar 使用pulsar-admin管理租户 命名空间

查看有哪些租户

bin/pulsar-admin tenants list

创建租户

 bin/pulsar-admin tenants create my-test

删除租户

bin/pulsar-admin tenants delete my-test

查看指定租户下边的命名空间

bin/pulsar-admin namespaces list my-test

创建指定租户命名空间

bin/pulsar-admin namespaces create my-test/default

删除指定租户命名空间

bin/pulsar-admin namespaces delete my-test/default

Pulsar Standalone JWT鉴权配置 采用对称密钥

通过访问portainer找到pulsar对应的container，通过console进入到pulsar。

创建base64编码密钥

bin/pulsar tokens create-secret-key --output ./my-secret.key --base64

生成token

bin/pulsar tokens create --secret-key ./my-secret.key \            --subject my-test \            --expiry-time 1y

授权

 bin/pulsar-admin namespaces grant-permission my-tenant/my-namespace \            --role my-test \            --actions produce,consume

配置standalone.conf使pulsar服务支持jwt鉴权

找到如下键值，进行修改

authenticateOriginalAuthData=trueauthenticationEnabled=trueauthenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTokenauthorizationEnabled=truesuperUserRoles=root,admin,my-testbrokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTokenbrokerClientAuthenticationParameters=token:eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJteS10ZXN0In0.hE_uDehkv92VS2b1PPkDvkSK48NUZhv2jTG7MiKqdpYtokenSecretKey=data:;base64,67r9KG04WWEpzIJ3pxFO3gO76/fzaOnh8ywADBDCcIs=

pulsar-manager创建用户密码

CSRF_TOKEN=$(curl http://localhost:7750/pulsar-manager/csrf-token)   curl \   -H 'X-XSRF-TOKEN: $CSRF_TOKEN' \   -H 'Cookie: XSRF-TOKEN=$CSRF_TOKEN;' \   -H "Content-Type: application/json" \   -X PUT http://localhost:7750/pulsar-manager/users/superuser \   -d '{"name": "pulsar", "password": "pulsar", "description": "test", "email": "[email protected]"}'