Skip to content

Commit 18ce8ea

Browse files
felix-oqfelix-oq
authored
Implement message-based CDC-Test between Notification and Pipeline (#362)
* Implement first basic contract test using pact * Fix lint errors * Integrate CDCT into GitHub Actions Workflow This approach uses the local file system for contract exchange * Fix error in CDCT-related docker files The contract files are now properly shared between providers and consumers. Previously, the provider did not use the newly generated contract files. * Delete all contract files This commit serves as a test to see if GitHub Actions commits the auto-generated contract files * Update auto-generated contracts * Create separate GitHub Actions Workflow for CDCT * Update auto-generated contracts * Improve mocking in pipeline provider test Now environment variables are properly mocked Also the main method is invoked to start the provider This removes the duplicate code in the provider test * Change pactfile write mode to overwrite Now existing pact files are entirely overwritten when consumer test execution brings changes to the files * Add CDCT for all interactions of UI and Pipeline * Replace static call to TransformationRest It is necessary to call the constructor and pass the pipeline service url * Add global test setup for UI Now when running CDCT, the previously existing contract files are deleted. Without this change, deprecated interactions would have to be deleted manually. * Fix error in Dockerfile.consumer of UI Previously the Dockerfile did not copy the pact folder. * Update auto-generated contracts * Move all interactions into a single test file With the previous approach, the order of the interactions in the contract file could be random. Now the order of the interactions in the contract file is determined by the order of the test cases. * Update auto-generated contracts * Update auto-generated contracts * Delete deprecated pact test This file was unintentionally added during merge resolution * Update auto-generated contracts * Remove obsolete common fixtures file * Implement first message-based CDC-Test The test covers an interaction between the notification service and the pipeline service * Update auto-generated contracts * Fix errors in GitHub Actions Workflow for CDCT Now the Consumer-Side fails if any Consumer-Test fails. The Provider-Side should run properly now. * Update auto-generated contracts * Make CDCT run in a single Workflow Job Previously newly generated contract files were not available to the Provider-Side Job. * Update auto-generated contracts * Add jest config file for pipeline provider test * Add new environment variable to test setup * Fix lint errors Co-authored-by: felix-oq <[email protected]>
1 parent b1b34fc commit 18ce8ea

19 files changed

+5200
-1674
lines changed

‎.github/workflows/cdct.yml

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,16 +17,32 @@ jobs:
1717
steps:
1818
- uses: actions/checkout@v2
1919

20-
- name: Build Consumer-Driven Contract Test
20+
- name: Build Consumer-Side Test for UI
2121
run: |
22-
docker-compose -f docker-compose.pact.yml build
22+
docker-compose -f ui/pact/docker-compose.consumer.yml build
2323
24-
- name: Run Consumer-Driven Contract Test
24+
- name: Build Consumer-Side Test for Notification
2525
run: |
26-
docker-compose -f docker-compose.pact.yml up
26+
docker-compose -f notification/pact/docker-compose.consumer.yml build
27+
28+
- name: Run Consumer-Side Test for UI
29+
run: |
30+
docker-compose -f ui/pact/docker-compose.consumer.yml up
31+
32+
- name: Run Consumer-Side Test for Notification
33+
run: |
34+
docker-compose -f notification/pact/docker-compose.consumer.yml up
2735
2836
- name: Commit latest contracts
2937
uses: stefanzweifel/git-auto-commit-action@v4
3038
with:
3139
commit_message: Update auto-generated contracts
3240
file_pattern: pacts/*.json
41+
42+
- name: Build Provider-Side Test
43+
run: |
44+
docker-compose -f docker-compose.provider.yml build
45+
46+
- name: Run Provider-Side Test
47+
run: |
48+
docker-compose -f docker-compose.provider.yml up --abort-on-container-exit --exit-code-from pipeline-provider

‎docker-compose.pact.yml

Lines changed: 0 additions & 24 deletions
This file was deleted.

‎docker-compose.provider.yml

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
version: "3.7"
2+
3+
services:
4+
# ----------------- PIPELINE SERVICE Provider --------------------
5+
pipeline-provider:
6+
build:
7+
context: .
8+
dockerfile: ./pipeline/pact/Dockerfile.provider
9+
depends_on:
10+
- rabbitmq-provider
11+
- pipeline-db-provider
12+
- pipeline-outboxer-provider
13+
environment:
14+
CONNECTION_RETRIES: '30'
15+
CONNECTION_BACKOFF_IN_MS: '2000'
16+
17+
POSTGRES_HOST: "pipeline-db-provider"
18+
POSTGRES_PORT: 5432
19+
POSTGRES_USER: "pipeline-service"
20+
POSTGRES_PW: "pw"
21+
POSTGRES_DB: "ods-pipelines"
22+
POSTGRES_SCHEMA: "public"
23+
POSTGRES_SSL: "false"
24+
25+
AMQP_URL: "amqp://rabbit_adm:R4bb!7_4DM_p4SS@rabbitmq-provider:5672"
26+
27+
AMQP_PIPELINE_EXECUTION_SUCCESS_TOPIC: 'pipeline.execution.success'
28+
AMQP_PIPELINE_EXECUTION_ERROR_TOPIC: 'pipeline.execution.error'
29+
AMQP_PIPELINE_CONFIG_CREATED_TOPIC: 'pipeline.config.created'
30+
AMQP_PIPELINE_CONFIG_UPDATED_TOPIC: 'pipeline.config.updated'
31+
AMQP_PIPELINE_CONFIG_DELETED_TOPIC: 'pipeline.config.deleted'
32+
33+
AMQP_DATASOURCE_EXECUTION_EXCHANGE: 'ods_global'
34+
AMQP_DATASOURCE_EXECUTION_QUEUE: 'pipeline.datasource-execution'
35+
AMQP_DATASOURCE_EXECUTION_QUEUE_TOPIC: 'datasource.execution.success'
36+
AMQP_DATASOURCE_EXECUTION_SUCCESS_TOPIC: 'datasource.execution.success'
37+
38+
AMQP_PIPELINE_EXECUTION_EXCHANGE: 'ods_global'
39+
AMQP_PIPELINE_EXECUTION_QUEUE: 'notification.pipeline-execution'
40+
volumes:
41+
- ./pacts:/app/pacts
42+
43+
pipeline-db-provider:
44+
image: postgres:13-alpine
45+
environment:
46+
POSTGRES_USER: 'pipeline-service'
47+
POSTGRES_PASSWORD: 'pw'
48+
POSTGRES_DB: 'ods-pipelines'
49+
command:
50+
# This enables and configures Postgres logical decoding feature that is needed for outboxer/debezium to work
51+
- '-cwal_level=logical'
52+
- '-cmax_wal_senders=1'
53+
- '-cmax_replication_slots=1'
54+
55+
pipeline-outboxer-provider:
56+
image: ghcr.io/jvalue/outboxer-postgres2rabbitmq
57+
environment:
58+
- OUTBOXER_DATABASE_HOSTNAME=pipeline-db-provider
59+
- OUTBOXER_DATABASE_PORT=5432
60+
- OUTBOXER_DATABASE_USER=pipeline-service
61+
- OUTBOXER_DATABASE_PASSWORD=pw
62+
- OUTBOXER_DATABASE_DBNAME=ods-pipelines
63+
- OUTBOXER_DATABASE_SERVER_NAME=pipeline-outboxer
64+
- OUTBOXER_PUBLISHER_AMQP_URL=amqp://rabbit_adm:R4bb!7_4DM_p4SS@rabbitmq-provider:5672
65+
- OUTBOXER_PUBLISHER_AMQP_EXCHANGE=ods_global
66+
- OUTBOXER_PUBLISHER_AMQP_RETRIES=30
67+
- OUTBOXER_PUBLISHER_AMQP_RETRY_DELAY_MS=2000
68+
depends_on:
69+
- pipeline-db-provider
70+
- rabbitmq-provider
71+
72+
# -------------------- RABBIT-MQ -------------------------------------------
73+
rabbitmq-provider:
74+
image: rabbitmq:3.8-management-alpine
75+
environment:
76+
RABBITMQ_ERLANG_COOKIE: 'S0m3_R4bBi7_C0ok13'
77+
RABBITMQ_DEFAULT_USER: 'rabbit_adm'
78+
RABBITMQ_DEFAULT_PASS: 'R4bb!7_4DM_p4SS'
79+
ports:
80+
- "15672:15672"
81+
- "5672:5672"

‎notification/jest.config.js

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
1+
const { defaults } = require('jest-config')
2+
13
module.exports = {
24
preset: 'ts-jest',
35
testEnvironment: 'node',
46
roots: ['./src'],
7+
testPathIgnorePatterns: [
8+
...defaults.testPathIgnorePatterns,
9+
'.*\\.pact\\.test\\.ts$'
10+
]
511
};

0 commit comments

Comments
 (0)