TL;DR
我们的主要目标
研究与评估
实施
JWT 策略与 Clerk 策略
Clerk 和 Novu 之间同步
- 同步用户和组织
- Clerk 与 Novu 中存储的内容
注入企业版提供商
- AuthService 和 AuthModule - 动态注入
- 存储库 - 用户、组织、成员
- 控制器
- 此方法的问题
端点修改
需要考虑和避免的要点
团队焦点
总结
事后奖励积分

这篇文章是Novu给您带来的

新总部 / 新

开源通知平台。可嵌入的通知中心、电子邮件、推送和 Slack 集成。

Moving User Management from In-House to a Product: Why We Did It and What We Learned

面向开发人员的开源通知基础设施

使用单个 API 管理多渠道通知的终极服务

探索文档 »

报告错误 · 请求功能 · 加入我们的不和谐 · 路线图 · X · 通知目录

⭐️ 为什么选择 Novu？

Novu 提供了统一的 API，可以轻松地通过多种渠道发送通知，包括应用内、推送、电子邮件、短信和聊天。借助 Novu，您可以创建自定义工作流程并为每个渠道定义条件，确保以最有效的方式传递您的通知。

✨ 特点

？适用于所有消息传递提供商的单一 API（应用内、电子邮件、短信、推送、聊天）
？完全托管的 GitOps Flow，从您的 CI 部署
？使用 Zod 或 JSON Schema 定义工作流程和步骤验证
？ React Email/Maizzle/MJML 集成
？配备CMS，用于高级布局和设计管理
？在单个仪表板中调试和分析多通道消息
？可嵌入的通知中心...

在 GitHub 上查看

长话短说

Novu 将 Clerk 实现为用户管理解决方案（身份验证基础设施），为提供 SAML 单点登录 (SSO) 功能、Google 和 GitHub 作为 OAuth 提供商、多重身份验证、基于角色的帐户控制 (RBAC) 奠定了基础），等等。

一位名叫 Adam 的开发人员在平台工程师 Denis 的大力协助下实现了它。

Moving User Management from In-House to a Product: Why We Did It and What We Learned

像大多数项目一样，这个项目是从积压工作开始的。在此之前，已有数十名客户提出要求，并在我们的路线图中大力支持了该请求。

作为通知基础设施解决方案，我们的应用程序和架构的一部分涉及通过注册、登录和会话日志来管理用户，以使用户能够邀请团队成员加入组织并管理每个角色的访问权限。

这都是关于优先事项的。 Novu 的核心重点是解决与通知和消息管理相关的所有问题，这样我们的用户就不必这样做。因此，我们将整个周期花在为构建和管理通知工作流程提供最佳体验、帮助开发人员保护他们的时间以及平滑产品和营销团队之间的协作上。
有效的用户管理不属于我们致力于的“核心价值”。

就像我们希望您将工程通知的负担交给我们的专业知识一样，我们也将工程有效用户管理的负担交给了文员的专业知识。

不用说，我们的团队从第一天起就基于定制和精心设计的架构在内部构建了出色的身份验证和授权基础设施。

随着我们的升级，我们更加专注于完善通知开发体验。

我们希望开发人员和工程师避免重新发明轮子并让 Novu 处理通知，就像我们选择在产品的其他方面利用经过验证、测试和领先的解决方案一样：用于数据库的 MongoDB、用于支付的 Stripe，以及现在文员进行用户管理。我们言行一致。

我们的首要目标

为我们的用户创造安全且易于使用的体验。

在概述该项目的初稿时，它可能显得简短而直接，甚至可能给人一种可以在周末完成的印象。

初步草案清单：

OAuth 提供商（GitHub、Google）
SAML SSO
安全会话管理
RBAC
魔法链接验证

请注意，如果初稿没有更改，则该项目尚未收到足够的反馈和输入。自然，名单就更长了。

实际清单：

使用用户凭据注册
与 OAuth 提供商（Github、Google）注册
使用用户凭据登录
使用 OAuth 提供商（Github、Google）登录
使用 SSO (SAML) 登录
从 Novu CLI 登录
从 Vercel Marketplace 登录/注册
创建组织
组织管理
用户管理（更新用户信息、凭据等......）
MFA/2FA（通过短信/电子邮件进行的 OTP、TOTP、密钥、生物识别等）
邀请函
RBAC：两个角色管理员和编辑者
- admin = 管理员可以访问网络平台上的任何页面并与之交互（因此，包括团队成员和设置）
- 编辑=编辑角色仍然是“主要内容经理”（又名产品经理或营销经理）

研究与评估

确定项目范围后，下一步是进行研究并评估实现预期结果所需的资源。

这个过程包括：

对产品的当前状态和每一层都有非常清晰的了解：
- 依赖关系
- 端点
- 建筑
- 客户端层组件和表示（前端）
- 测试
还有更多。
概述迁移规范（保留在内部且应被阻止的内容）
向后兼容性
尝试从以前的同事那里找到类似项目的参考，并从他们的流程和建议中学习
尝试寻找开源解决方案
查找是否有任何供应商（第三方解决方案）并进行比较。

还有更多。

在另一篇博客文章中，我们将探讨如何评估和比较作为服务/基础设施即服务公司的第三方解决方案（或产品）。

研究不足或评估不准确通常会导致技术债务和未来的资源损失，例如添加附加功能和维护时的工程时间，这需要重构整个事物。因此，寻找每个选项的隐藏成本。

经验丰富的团队领导知道如何评估每个选项的投资回报率 (ROI)，这有助于他们做出最佳的业务决策。

这正是我们最终得到 Clerk 的原因。他们的解决方案涵盖了我们的大多数用例，从业务角度来看，实施它们来管理用户和组织层的投资回报率是有意义的。

执行

Novu 服务包含许多微服务和方面，例如：

通知渠道（短信、电子邮件、应用内、推送、聊天等..），
通知编排（跨设备同步、摘要引擎、延迟、时区感知等..）
通知可观察性（调试、见解等）
通知内容管理（编辑器、品牌、布局、翻译、变量管理等..）
最终用户管理（用户首选项、订阅者、主题、细分、订阅管理等..）
帐户管理（SSO、基于角色的访问控制、多租户、计费等...）

下图展示了 Novu 的 API 结构的简化版本，仅关注在实现 Clerk 之前对 Novu 用户和组织的身份验证和授权。

Moving User Management from In-House to a Product: Why We Did It and What We Learned

我们使用 MongoDB 来存储 Novu 所需的所有数据，每个用户、组织、租户、订阅者、主题……简而言之，一切。

因为Clerk有自己的数据库来管理用户，所以我们需要非常仔细和精确地处理数据库之间的迁移和同步。

JWT 策略与 Clerk 策略

我们需要确保的主要事情之一是 UserSessionData 对象不会更改，以便在使用 Novu 时不会中断用户的会话。它应该保持兼容。

在这里您可以看到 jwt.stratgy.ts 文件示例：

//jwt.stratgy.ts
import type http from 'http';
import { ExtractJwt, Strategy } from 'passport-jwt';
import { PassportStrategy } from '@nestjs/passport';
import { Injectable, UnauthorizedException } from '@nestjs/common';
import { ApiAuthSchemeEnum, HttpRequestHeaderKeysEnum, UserSessionData } from '@novu/shared';
import { AuthService, Instrument } from '@novu/application-generic';
import { EnvironmentRepository } from '@novu/dal';

@Injectable()
export class JwtStrategy extends PassportStrategy(Strategy) {
  constructor(private readonly authService: AuthService, private environmentRepository: EnvironmentRepository) {
    super({
      jwtFromRequest: ExtractJwt.fromAuthHeaderAsBearerToken(),
      secretOrKey: process.env.JWT_SECRET,
      passReqToCallback: true,
    });
  }
  @Instrument()
  async validate(req: http.IncomingMessage, session: UserSessionData) {
    // Set the scheme to Bearer, meaning the user is authenticated via a JWT coming from Dashboard
    session.scheme = ApiAuthSchemeEnum.BEARER;

    const user = await this.authService.validateUser(session);
    if (!user) {
      throw new UnauthorizedException();
    }

    await this.resolveEnvironmentId(req, session);

    return session;
  }

  @Instrument()
  async resolveEnvironmentId(req: http.IncomingMessage, session: UserSessionData) {
    // Fetch the environmentId from the request header
    const environmentIdFromHeader =
      (req.headers[HttpRequestHeaderKeysEnum.NOVU_ENVIRONMENT_ID.toLowerCase()] as string) || '';

    /*
     * Ensure backwards compatibility with existing JWTs that contain environmentId
     * or cached SPA versions of Dashboard as there is no guarantee all current users
     * will have environmentId in localStorage instantly after the deployment.
     */
    const environmentIdFromLegacyAuthToken = session.environmentId;

    let currentEnvironmentId = '';

    if (environmentIdFromLegacyAuthToken) {
      currentEnvironmentId = environmentIdFromLegacyAuthToken;
    } else {
      const environments = await this.environmentRepository.findOrganizationEnvironments(session.organizationId);
      const environmentIds = environments.map((env) => env._id);
      const developmentEnvironmentId = environments.find((env) => env.name === 'Development')?._id || '';

      currentEnvironmentId = developmentEnvironmentId;

      if (environmentIds.includes(environmentIdFromHeader)) {
        currentEnvironmentId = environmentIdFromHeader;
      }
    }

    session.environmentId = currentEnvironmentId;
  }
}

登录后复制

Moving User Management from In-House to a Product: Why We Did It and What We Learned

为了保持与应用程序其余部分的兼容性，我们需要将 JWT 有效负载从 Clerk 转换为之前存在的 JWT 格式。

我们就是这样做的：

async validate(payload: ClerkJwtPayload): Promise<IJwtClaims> {
  const jwtClaims: IJwtClaims = {
    // first time its clerk_id, after sync its novu internal id
    _id: payload.externalId || payload._id,
    firstName: payload.firstName,
    lastName: payload.lastName,
    email: payload.email,
    profilePicture: payload.profilePicture,
    // first time its clerk id, after sync its novu internal id
    organizationId: payload.externalOrgId || payload.org_id,
    environmentId: payload.environmentId,
    roles: payload.org_role ? [payload.org_role.replace('org:', '')] : [],
    exp: payload.exp,
  };

  return jwtClaims;
}

登录后复制

在这里您可以看到 clerk.strategy.ts 文件示例：

import type http from 'http';
import { Injectable } from '@nestjs/common';
import { PassportStrategy } from '@nestjs/passport';
import { ExtractJwt, Strategy } from 'passport-jwt';
import { passportJwtSecret } from 'jwks-rsa';
import {
  ApiAuthSchemeEnum,
  ClerkJwtPayload,
  HttpRequestHeaderKeysEnum,
  PassportStrategyEnum,
  UserSessionData,
} from '@novu/shared';
import { EnvironmentRepository, EnvironmentEntity } from '@novu/dal';
import { LinkEntitiesService } from '../services/link-entities.service';

@Injectable()
export class ClerkStrategy extends PassportStrategy(Strategy, PassportStrategyEnum.JWT_CLERK) {
  constructor(private environmentRepository: EnvironmentRepository, private linkEntitiesService: LinkEntitiesService) {
    super({
      // ...configuration details
    });
  }

  async validate(req: http.IncomingMessage, payload: ClerkJwtPayload) {
    const { internalUserId, internalOrgId } = await this.linkEntitiesService.linkInternalExternalEntities(req, payload);

    const session: UserSessionData = {
      _id: internalUserId,
      firstName: payload.firstName,
      lastName: payload.lastName,
      email: payload.email,
      profilePicture: payload.profilePicture,
      organizationId: internalOrgId,
      roles: payload.org_role ? [payload.org_role.replace('org:', '')] : [],
      exp: payload.exp,
      iss: payload.iss,
      scheme: ApiAuthSchemeEnum.BEARER,
      environmentId: undefined,
    };

    await this.resolveEnvironmentId(req, session);

    return session;
  }

  // Other functions...
}

登录后复制

Moving User Management from In-House to a Product: Why We Did It and What We Learned

Clerk 和 Novu 之间同步

虽然目标是理想情况下仅使用 Clerk 来创建和检索用户、组织等，但不幸的是，由于需要以高性能的方式存储和查询有关用户和组织的一些元数据，因此这并不完全可能。

这是 Novu 组织存储库中的方法示例：

  async findPartnerConfigurationDetails(organizationId: string, userId: string, configurationId: string) {
    const organizationIds = await this.getUsersMembersOrganizationIds(userId);

    return await this.find(
      {
        _id: { $in: organizationIds },
        'partnerConfigurations.configurationId': configurationId,
      },
      { 'partnerConfigurations.$': 1 }
    );
  }

登录后复制

此方法使用各种 MongoDB 特定结构来过滤文档 - 使用 Clerk 无法以高性能方式重现，因为这不是用于此类查询的数据库。

我们能做的就是将有关组织的这些元数据存储在 MongoDB 组织集合中，并使用 externalId 将集合与 Clerk 数据库链接/同步。

Moving User Management from In-House to a Product: Why We Did It and What We Learned

现在我们可以根据需要结合 Clerk 和 MongoDB 来查询元数据。

async findPartnerConfigurationDetails(
  organizationId: string,
  userId: string,
  configurationId: string
): Promise<OrganizationEntity[]> {
  const clerkOrganizations = await this.getUsersMembersOrganizations(userId);

  return await this.communityOrganizationRepository.find(
    {
      _id: { $in: clerkOrganizations.map((org) => org.id) },
      'partnerConfigurations.configurationId': configurationId,
    },
    { 'partnerConfigurations.$': 1 }
  );
}

private async getUsersMembersOrganizations(userId: string): Promise<Organization[]> {
  const userOrgMemberships = await this.clerkClient.users.getOrganizationMembershipList({
    userId,
  });

  return userOrgMemberships.data.map((membership) => membership.organization);
}

登录后复制

通过调用 getUsersMembersOrganizations，findPartnerConfigurationDetails 获取必要的组织数据，以在communityOrganizationRepository 上执行过滤搜索，确保仅返回相关配置。

我们只需要在 Clerk 和 Novu 之间同步用户和组织，组织成员不需要同步。

同步用户和组织

数据库 ID 同步有两种方法：

middleware - any endpoint in API will sync the IDs if it detects that JWT doesn’t yet contain an internal ID.
webhook - as soon as the user/org is registered in Clerk, Clerk calls Novu’s API webhook, and we sync it.

Moving User Management from In-House to a Product: Why We Did It and What We Learned

Here is the flow we had in mind:

A user creates a new account via frontend using the Clerk component
Gets a new JWT containing Clerk user-id
Any request that hits the API triggers the syncing process (given it hasn’t yet happened)
A new user is created in Novu’s MongoDB containing the Clerk’s externalId
Clerk user object gets updated with Novu internal object id (saved as externalId in Clerk)
The new token returned from Clerk now contains an externalId that is equal to Novu's internal user ID.
In the Clerk strategy in validate() function on API - we set _id to equal to externalId so it is compatible with the rest of the app.

Note
In the application, we always expect Novu’s internal id on input and we always return internal id on output - its important for the application to work as is without major changes to the rest of the code.
API expects internal _id everywhere and it needs to be MongoDB ObjectID type, because it parses this user id back to ObjectID e.g. when creating new environment or any other entity which needs reference to user.

The same logic applies to organizations; just the endpoint is different.

What is stored in Clerk vs Novu

Users

For the users, we store everything in Clerk. All the properties are mostly just simple key/value pairs and we don’t need any advanced filtering on them, therefore they can be retrieved and updated directly in Clerk.

In internal MongoDB, we store just the user internal and external ids.

The original Novu user properties are stored in Clerk’s publicMetadata :

export type UserPublicMetadata = {
  profilePicture?: string | null;
  showOnBoardingTour?: number;
};

登录后复制

There are also many other attributes coming from Clerk which can be set on the user.

Organizations

For the organizations, we store everything in Clerk except for apiServiceLevel, partnerConfigurations, and branding since they are “native” to Clerk and we update those attributes directly there via frontend components and so we don’t need to sync with our internal DB after we change organization name or logo via Clerk component.

Moving User Management from In-House to a Product: Why We Did It and What We Learned

Injection of Enterprise Edition providers

The goal here was to replace the community (open source) implementation with Clerk while being minimally invasive to the application and to keep the Clerk implementation in a separate package.

This means we need to keep the changed providers (OrganizationRepository, AuthService…) on the same place with the same name so we don’t break the imports all over the place, but we need to change their body to be different based on feature flag.

The other option would be to change all of these providers in the 100+ of files and then import the EE(enterprise edition) package everywhere, which is probably not a good idea.

This turned out to be quite challenging due to the fact that users, organization and members are relatively deeply integrated to the application itself, referenced in a lot of places and they’re also tied to MongoDB specifics such as ObjectID or queries (create, update, findOne …).

The idea is to provide different implementation using NestJS dynamic custom providers where we are able to inject different class/service on compile time based on the enterprise feature flag.

This is the most promising solution we found while keeping the rest of the app mostly untouched, there are some drawbacks explained later.

AuthService & AuthModule - dynamic injection

Moving User Management from In-House to a Product: Why We Did It and What We Learned

We have two implementations of AuthService - community and enterprise one (in private package), we inject one of those as AUTH_SERVICE provider.

We need to however have a common interface for both IAuthService

Since we also need to change the AuthModule, we initialize two different modules based on the feature flag like this:

function getModuleConfig(): ModuleMetadata {
  if (process.env.NOVU_ENTERPRISE === 'true') {
    return getEEModuleConfig();
  } else {
    return getCommunityAuthModuleConfig();
  }
}

@Global()
@Module(getModuleConfig())
export class AuthModule {
  public configure(consumer: MiddlewareConsumer) {
    if (process.env.NOVU_ENTERPRISE !== 'true') {
      configure(consumer);
    }
  }
}

登录后复制

The reason why the EEModule can be a standalone module in the @novu/ee-auth package which we would just import instead of the original AuthModule and instead we are initializing one module conditionally inside API, is that we are reusing some original providers in the EE one - e.g. ApiKeyStrategy , RolesGuard, EnvironmentGuard etc which resides directly in API.

We would need to import them in the @novu/ee-auth package which would require to export these things somewhere (probably in some shared package) and it introduces other issues like circular deps etc - it can be however refactored later.

Repositories - users, organizations, members

Same logic applies for the repositories. No module is being initialized here, they’re just directly injected to the original repository classes.

Moving User Management from In-House to a Product: Why We Did It and What We Learned

Controllers

The controllers are being conditionally imported from inside @novu/api . The reason for that is the same as in the auth module, there are too many imports that the controllers uses, that we would either need to move to @novu/ee-auth or move them to a separate shared package - which would then trigger a much bigger change to the other unrelated parts of the app, which would increase the scope of this change.

function getControllers() {
  if (process.env.NOVU_ENTERPRISE === 'true') {
    return [EEOrganizationController];
  }

  return [OrganizationController];
}

@Module({
  controllers: [...getControllers()],
})
export class OrganizationModule implements NestModule {
    ...
}

登录后复制

Issues with this approach

The main issue here is the need for common interface for both of the classes - community and enterprise. You want to remain compatible in both community and enterprise versions, so when there is a this.organizationService.getOrganizations() method being called in 50 places in the app - you need an enterprise equivalent with the same name otherwise you need to change 50 places to call something else.

This results in not-so-strict typing and methods without implementation

Moving User Management from In-House to a Product: Why We Did It and What We Learned

We need to have a common interface for both, however the community one relies on MongoDB methods and needs different method arguments as the enterprise one which causes a use of any to forcefully fit both classes etc.
In some cases we don’t need the method at all, so we need to throw Not Implemented .

Endpoints modification

We modified the endpoints as follows:

AuthController: Removed and replaced by frontend calls to Clerk.
UserController: Removed, added a sync endpoint for Clerk users with MongoDB.
OrganizationController: Removed several endpoints, which can be migrated later.
InvitesController: Completely removed.
StorageModule: Completely removed.

Key points to consider and avoid

Avoid Storing Frequently Changing Properties in JWT
- Example: environmentID
- It can be cumbersome to update these properties.
Simplify Stored Data Structures
- Avoid storing complex structures in user, organization, or member records.
- Clerk performs optimally with simple key:value pairs, not arrays of objects.
Implement a User/Organization Replication Mechanism
- This helps bridge the gap during the migration period before Clerk is fully enabled.
- Use MongoDB triggers to replicate newly created users and organizations to both Clerk and your internal database.
Store Original Emails
- Do not sanitize emails as Clerk uses the original email as a unique identifier.

Team Spotlight

Lead Engineer: Adam Chmara

Platform Team Lead: Denis Kralj

Summary

Our implementation approach comes to the fact that we offloaded the Users, Organizations and Members management to Clerk.

The data property injection to Novu’s Controllers (endpoints) layer, Business layer and data layer happens based on “Enterprise” feature flag validation.

We are leveraging pre-built Clerk components on the frontend and reducing the need to build and maintain our own custom implementation on the backend.

You can also observe below the diagram of the current state after implementing Clerk.

Moving User Management from In-House to a Product: Why We Did It and What We Learned

事后诸葛亮奖励积分

当我们决定实施 Clerk 进行用户管理时，我们还选择了扩展 Clerk 未来将支持和提供的功能和特性的长期利益。

以下是我们在不久的将来可能考虑支持的一些示例：

细粒度访问控制（FGAC）
- 基于属性：FGAC 通常使用基于属性的访问控制 (ABAC) 来实现，其中访问决策基于用户、资源和环境的各种属性。属性可以包括用户角色、部门、资源类型、一天中的时间等。
- 灵活性：FGAC 通过允许详细的、基于条件的访问控制来提供更大的灵活性和粒度。这意味着权限可以根据非常具体的场景进行微调。
- 动态：FGAC 可以动态适应环境的变化，例如时间敏感的访问或基于位置的限制。
- 详细权限：FGAC 中的权限更加具体，可以根据个人操作、用户或情况进行定制。

为 Novu 提供这种级别的详细灵活性，可能超出了范围，甚至由于实施的潜在复杂性而被刮掉。

用户冒充

我们的客户成功或支持团队可以使用它来解决问题、提供支持或从模拟用户的角度测试功能，而无需知道他们的密码或其他身份验证详细信息。
- 减少诊断和解决用户问题所涉及的时间和复杂性。
- 确保所采取的支持或管理操作准确无误，因为支持人员可以像用户一样查看系统并与之交互。
- 通过提供更快、更有效的支持来提高用户满意度。

简而言之，鉴于现在身份验证基础设施已被占用，我们将能够轻松改善 Novu 用户的体验。

如果您想建议涉及 AuthN（或任何其他）的其他功能，请访问我们的路线图来审核和投票请求，或提交您的想法。