test: records e2e tests for the apis using keploy cli

[re-Tick]

Issue Description

Add e2e tests and mocks to test the API endpoints.

ref: #972

Keploy-CLI Integration

Install and setup my forked repo in your machine by following this guide for dev setup. and switch to the test-keploy-cli branch.

On Linux:

Download the keploy cli binary according to your local architecture. Instructions

Record Cmd

Record the testcases and mocks for the listmonk API using following cmd after initial configuration of listmonk.

sudo -E env PATH=$PATH keploy record -c "make run"

And made the API calls from here. example:

curl -u 'listmonk:listmonk' 'http://localhost:9000/api/subscribers/1'                     

This generates the keploy testcase test-1.yml in the keploy/tests directory along with postgres mocks.

- keploy
   |- tests
      |- test-1.yaml
   |- mocks.yaml 

Test Cmd

Run the recorded testcases using the test cmd of keploy CLI.

sudo -E env PATH=$PATH keploy test -c "make run"

On MacOS:

MacOS users need to install Colima and set an alias for Keploy CLI.

# create docker network for keploy CLI
docker network create keploy-network

# make alias for the keploy CLI
alias keploy='sudo docker run --pull always --name keploy-v2 -p 16789:16789 --network keploy-network --privileged --pid=host -it -v "$(pwd)":/files -v /sys/fs/cgroup:/sys/fs/cgroup -v /sys/kernel/debug:/sys/kernel/debug -v /sys/fs/bpf:/sys/fs/bpf -v /var/run/docker.sock:/var/run/docker.sock --rm ghcr.io/keploy/keploy'

Record cmd:

Start the postgres server:

docker run -p 9432:5432 -e POSTGRES_DB=listmonk -e POSTGRES_USER=listmonk -e POSTGRES_PASSWORD=listmonk --name demo-db --network keploy-network postgres:13

And edit some of the config.toml.sample fields:

address = "0.0.0.0:9000"
host = "demo-db"

And build the docker image for the listmonk: docker build -t listmonk .
Now, run the API server and do the curl requests:

keploy record -c "docker run -p 9000:9000 -e TZ=Etc/UTC --network keploy-network --name demo-app --rm listmonk"

Test Cmd:

Stop the postgres server. And run the test cmd:

keploy test -c "docker run -p 9000:9000 -e TZ=Etc/UTC --network keploy-network --name demo-app --rm listmonk"

[knadh]

Ah, nice. I see that there's no changes to the source code unlike last time. Has keploy's architecture changed since?

[re-Tick]

Yes. Now, keploy can be used as CLI and instruments the linux system calls. You can try in MacOS with co-lima and docker cmds.

[nehagup]

@knadh Thanks! we now use eBPF to instrument the applications and perform packet capture, processing and matching at the network level. Its Linux native and macOS/Windows users would either have to use VMs (lima/wsl) or via Docker (on MacOS we will have to use Colima)

[knadh]

eBPF, that's cool! I'll test this out with listmonk in a couple of weeks. Thanks again for the PR.

[knadh]

@re-Tick I downloaded the mock files from the PR (please see the tree below)

$ tree /tmp/listmonk-test-keploy-cli
/tmp/listmonk-test-keploy-cli
└── keploy
    ├── testReports
    │   ├── report-1.yaml
    │   ├── report-2.yaml
    │   ├── report-3.yaml
    │   └── report-4.yaml
    └── test-set-0
        ├── mocks.yaml
        └── tests
            ├── test-10.yaml
            ├── test-11.yaml
            ├── test-1.yaml
            ├── test-2.yaml
            ├── test-3.yaml
            ├── test-4.yaml
            ├── test-5.yaml
            ├── test-6.yaml
            ├── test-7.yaml
            ├── test-8.yaml
            └── test-9.yaml

Then ran the test command.

Testing make run

$ sudo -E keploy test -p /tmp/listmonk-test-keploy-cli -c "make run"

       ▓██▓▄
    ▓▓▓▓██▓█▓▄
     ████████▓▒
          ▀▓▓███▄      ▄▄   ▄               ▌
         ▄▌▌▓▓████▄    ██ ▓█▀  ▄▌▀▄  ▓▓▌▄   ▓█  ▄▌▓▓▌▄ ▌▌   ▓
       ▓█████████▌▓▓   ██▓█▄  ▓█▄▓▓ ▐█▌  ██ ▓█  █▌  ██  █▌ █▓
      ▓▓▓▓▀▀▀▀▓▓▓▓▓▓▌  ██  █▓  ▓▌▄▄ ▐█▓▄▓█▀ █▓█ ▀█▄▄█▀   █▓█
       ▓▌                           ▐█▌                   █▌
        ▓
  
version: 2.0.0-alpha44

Warning: delay is set to 5 seconds, incase your app takes more time to start use --delay to set custom delay
Example usage:
 sudo -E env PATH=$PATH keploy test -c "/path/to/user/app" --delay 6 

🐰 Keploy: 2023-11-11T20:46:46+05:30    INFO    cmd/test.go:192         {"keploy test and mock path": "/tmp/listmonk-test-keploy-cli/keploy", "keploy testReport path": "/tmp/listmonk-test-keploy-cli/keploy/testReports"}
🐰 Keploy: 2023-11-11T20:46:46+05:30    INFO    cmd/test.go:43  keploy configuration file not found
🐰 Keploy: 2023-11-11T20:46:50+05:30    INFO    hooks/loader.go:748     keploy initialized and probes added to the kernel.
🐰 Keploy: 2023-11-11T20:46:50+05:30    INFO    proxy/proxy.go:284      Java detected and successfully imported CA      {"path": "/usr/lib/jvm/java-17-openjdk-17.0.8.0.7-1.fc38.x86_64/lib/security/cacerts", "output": "Warning: use -cacerts option to access cacerts keystore\nCertificate was added to keystore\n"}
🐰 Keploy: 2023-11-11T20:46:50+05:30    INFO    proxy/proxy.go:285      Successfully imported CA        {"": "V2FybmluZzogdXNlIC1jYWNlcnRzIG9wdGlvbiB0byBhY2Nlc3MgY2FjZXJ0cyBrZXlzdG9yZQpDZXJ0aWZpY2F0ZSB3YXMgYWRkZWQgdG8ga2V5c3RvcmUK"}
🐰 Keploy: 2023-11-11T20:46:51+05:30    INFO    proxy/proxy.go:271      Java detected and CA already exists     {"path": "/usr/lib/jvm/java-17-openjdk-17.0.8.0.7-1.fc38.x86_64/lib/security/cacerts"}
🐰 Keploy: 2023-11-11T20:46:51+05:30    INFO    proxy/proxy.go:271      Java detected and CA already exists     {"path": "/usr/lib/jvm/java-17-openjdk-17.0.8.0.7-1.fc38.x86_64/lib/security/cacerts"}
🐰 Keploy: 2023-11-11T20:46:52+05:30    INFO    proxy/proxy.go:397      Keploy has hijacked the DNS resolution mechanism, your application may misbehave in keploy test mode if you have provided wrong domain name in your application code.
🐰 Keploy: 2023-11-11T20:46:52+05:30    INFO    proxy/proxy.go:411      Proxy started at port:16789
🐰 Keploy: 2023-11-11T20:46:52+05:30    INFO    proxy/proxy.go:568      starting DNS server at addr :16789
🐰 Keploy: 2023-11-11T20:46:52+05:30    INFO    test/test.go:241        running user application for    {"test-set": "test-set-0"}
🐰 Keploy: 2023-11-11T20:46:52+05:30    INFO    test/test.go:299                {"no of test cases": 11, "test-set": "test-set-0"}
CGO_ENABLED=0 go run -ldflags="-s -w -X 'main.buildString=v2.5.1 (#ef08495 2023-11-11T15:16:52+0000)' -X 'main.versionString=v2.5.1' -X 'main.frontendDir=frontend/dist'" cmd/*.go
/bin/sh: line 1: go: command not found
make: *** [Makefile:46: run] Error 127
🐰 Keploy: 2023-11-11T20:46:52+05:30    ERROR   hooks/launch.go:526     userApplication failed to run with the following error. Please check application logs   {"error": "exit status 2"}
🐰 Keploy: 2023-11-11T20:46:57+05:30    INFO    test/test.go:435        test report for test-set-0:     {"name: ": "report-4", "path: ": "/tmp/listmonk-test-keploy-cli/keploy/report-4"}

 <=========================================> 
  TESTRUN SUMMARY. For testrun with id: "test-set-0"
        Total tests: 0
        Total test passed: 0
        Total test failed: 0
 <=========================================> 

🐰 Keploy: 2023-11-11T20:46:57+05:30    INFO    test/test.go:181        test run completed      {"passed overall": false}
🐰 Keploy: 2023-11-11T20:46:57+05:30    INFO    hooks/loader.go:384     Exiting keploy program gracefully.
🐰 Keploy: 2023-11-11T20:46:57+05:30    INFO    hooks/loader.go:434     eBPF resources released successfully...
🐰 Keploy: 2023-11-11T20:46:57+05:30    INFO    proxy/proxy.go:996      Dns server stopped
🐰 Keploy: 2023-11-11T20:46:57+05:30    INFO    proxy/proxy.go:998      proxy stopped...

Getting this.

/bin/sh: line 1: go: command not found
make: *** [Makefile:46: run] Error 127

Testing the compiled bin

It runs, but a bunch of tests fail with the following Postgres errors.

+-------------------------------------------------------------------------------------------------------------+
|                                                DIFFS TEST-2                                                 |
+-------------------------------------------------------------------------------------------------------------+
|                     EXPECT STATUS                    |                   ACTUAL STATUS                      |
| -----------------------------------------------------+----------------------------------------------------- |
|                          200                         |                        500                           |
|                                                      |                                                      |
|                                                                                                             |
|                     EXPECT HEADER                    |                   ACTUAL HEADER                      |
| -----------------------------------------------------+----------------------------------------------------- |
|                 Content-Length: [706]                |                Content-Length: [75]                  |
|                                                      |                                                      |
|                                                                                                             |
|                      EXPECT BODY                     |                    ACTUAL BODY                       |
| -----------------------------------------------------+----------------------------------------------------- |
|    {                                                 |  {                                                   |
|   -  "data": {                                       | +  "message": "Error fetching Subscriber: pq: unex   |
|   -    "attribs": {                                  | pected Parse response '2'"                           |
|   -      "city": "Bengaluru",                        |  }   

• pq: unexpected Parse response '2'
• pq: unknown response for startup: '2'
• pq: fetching Campaign: pq: unknown response for startup: '1'

[re-Tick]

Hi @knadh, the CLI requires sudo, which changes the environment variables. To ensure it locates go/bin, we need sudo -E env PATH=$PATH. Also, add a 10-second delay before running the test command, so the app is up and ready for the test APIs. The test cmd would be:
sudo -E env PATH=$PATH keploy test -p /tmp/listmonk-test-keploy-cli -c "make run" --delay 10

[nehagup]

Hi @knadh, we are not able to replicate your Postgres issue. We have tried on both arm64 and amd64 machines with Ubuntu, Debian and Amazon Linux and have followed the following steps:

Steps
    1.    Install go v1.20.10
    2.    Install node v18.18.2.
    3.    Install yarn v1.22.21
    4.    git clone https://github.com/re-Tick/listmonk.git
    5.    cd listmonk
    6.     git pull origin test-keploy-cli
    7.    git checkout test-keploy-cli
    8.    cp config.toml.sample config.toml
    9.    make dist
    10.    Install keploy via curl -O https://raw.githubusercontent.com/keploy/keploy/main/keploy.sh && source keploy.sh and type linux in the prompt.
    11.    keploy test -c "./listmonk"

Here all tests pass:

 <=========================================> 
  TESTRUN SUMMARY. For testrun with id: "test-set-0"
        Total tests: 11
        Total test passed: 11
        Total test failed: 0
 <=========================================> 

Can you please share more information (maybe the config file) to replicate the issue?

[knadh]

Sorry, I haven't been able to find the time or bandwidth to look into this, unfortunately ;( I also realise I wouldn't have the bandwidth to maintain the mock files for the tests either.

I'll look forward to bumping into a Keploy dev at one of the local meetups, maybe at IndiaFOSS, who could run me through this.

[github-actions]

This PR has been marked 'stale' after 90 days of inactivity. If there is no further activity, it will be closed in 7 days.

[nehagup]

Hi @knadh! I’m still happy to walk you through the PR or explore collaboration— pinged you on linkedin, let me know when we can connect. :)

[knadh]

Hi @nehagup, firstly, thanks for your patience! Never got around to figuring this out. Could you please drop me an e-mail at kailash [at] nadh.in? We can schedule a call sometime next week and address this for good.