Composing images with DockerCompose

Lets create a Dockerfile to dockerize our e2e-tests module.

FROM node:7.8.0

ENV NPM_CONFIG_LOGLEVEL warn

COPY . .

RUN yarn

CMD yarn test

If create the image and run it, it will fail, because the webapp won't be running neither the backend or mongo database.

So in this project we need a complex build depending on the other modules.

But first we need to version those other modules in GitHub.

frontend

git init
# GO AHEAD AND CREATE A REPO IN GITHUB, THEN..
git remote add origin [email protected]:javierfernandes/docker-e2e-frontend.git
git pull origin master
git add .
git ci -m "Initial working version of the frontend with Dockerfile"
git push --set-upstream origin master

And the same for the backend.

Now we can grab those repositories git URLs to use them in docker compose. Create a file called docker-compose.yml in the e2e-test project.

mongo:
  image: mongo
  ports:
    - "27018:27017"
  command: "--smallfiles --logpath=/dev/null"
backend:
  build: 'https://github.com/javierfernandes/docker-e2e-backend.git#master'
  ports:
    - "3001:3001"
  links:
    - mongo
frontend:
  build: 'https://github.com/javierfernandes/docker-e2e-frontend.git#master'
  ports:
    - "3000:3000"
  links:
    - backend
tests:
  build: .
  links:
    - frontend

We can now build the image and start an instance with a single command

docker-compose up

You will see a lot of logs then, notice that each log has a different preffix & color to identify which images is logging, since we will have multiple instances starting at the same time.

For example...

mongo_1     | about to fork child process, waiting until server is ready for connections.
backend_1   | yarn start v0.22.0
frontend_1  | yarn start v0.22.0

Ok, you will notice that we are still having issues. The backend module cannot reach mongo, because it is trying to use the default connection string:

backend_1   | MongoError: failed to connect to server [localhost:27017] on first connect [MongoError: connect ECONNREFUSED 127.0.0.1:27017]

So we basically need to start the backend instructing it to connect to our mongo instance (which will have an internal IP generated by docker). We already know that the backend will try to use and env variable called MONGO_URL because we coded it like that :) So we just need to make docker-compose run the backend instance setting that environment variable. Change in docker-compose adding the environment part to backend

backend:
  build: 'https://github.com/javierfernandes/docker-e2e-backend.git#master'
  ports:
    - "3001:3001"
  links:
    - mongo
  environment: 
    MONGO_URL: 'mongodb://mongo/docker-e2e'

Notice that the host name in the url **must match the name of the mongo module of this same docker-compose file. We could have named it "e2-mongo" or "e2e-tutorial-db" whatever, but they should match !.

Now compose up again and you will see that the database error is no longer present :)

Still you will see another error...

Starting e2etests_mongo_1
Starting e2etests_backend_1
Starting e2etests_frontend_1
Starting e2etests_tests_1
Attaching to e2etests_mongo_1, e2etests_backend_1, e2etests_frontend_1, e2etests_tests_1
backend_1   | yarn start v0.22.0
frontend_1  | yarn start v0.22.0
frontend_1  | $ webpack-dev-server --history-api-fallback --progress --host 0.0.0.0 --port 3000 --inline --hot --config webpack.config.js --quiet 
backend_1   | $ babel-node ./bin/www 
tests_1     | yarn test v0.22.0
tests_1     | $ DEBUG=nightmare yarn codeceptjs run --steps --verbose 
tests_1     | yarn codeceptjs v0.22.0
tests_1     | verbose 0.442 current time: 2017-05-07T14:36:24.215Z
tests_1     | $ "/node_modules/.bin/codeceptjs" run
tests_1     | 
tests_1     | Could not load helper Nightmare from module './helper/Nightmare':
tests_1     | Required modules are not installed.
tests_1     | 
tests_1     | RUN: npm install --save-dev nightmare nightmare-upload
tests_1     | {}

It is creepy but the problem, even if it doesn't tell it expliciletly :( is that our e2e tests with codecept also cannot reach the frontend because we had a config file called codecept.json which has a hardcoded URL

    "Nightmare": {
      "url": "http://localhost:3000",
      "show": true
    }

We need to do something similar to what we did for the backend => mongo relation. We will use env vars.

But first we need to move from a codecept json config file to a js file, so that we can use env vars there.

vi codecept.conf.js

With the following content

exports.config = {
  "tests": "test/**/*_test.js",
  "timeout": 10000,
  "output": "./output",
  "helpers": {
    "Nightmare": {
      "url": process.env.CODECEPT_URL || "http://localhost:3000",
      "show": true
    }
  },
  "include": {},
  "bootstrap": false,
  "mocha": {},
  "name": "e2e-tests"
}

Then remove the old file

rm codecept.json

Now update the docker-compose.yml file so that we set those variables for the test module

tests:
  build: .
  links:
    - frontend
  environment:
    CODECEPT_URL: 'http://frontend:3000'

(Again here the host name must match the name we gave it to the frontend in this same file)

We still need to do the same for the frontend => backend relation so, in the frontend edit the webpack.config.js file and change this part to use an env var

    proxy: {
      '/api': {
        target: process.env.BACKEND_URL | 'http://localhost:3001/',
        pathRewrite: { '^/api': '' }
      }
    }

And now update the docker-compose.yml file

frontend:
  build: 'https://github.com/javierfernandes/docker-e2e-frontend.git#master'
  ports:
    - "3000:3000"
  links:
    - backend
  environment: 
    BACKEND_URL: 'http://backend:3001/'

Still if you build the image and run it

docker-compose build
docker-compose up

You will see this error

tests_1     |   1) Users List: Loads fine:
tests_1     | 
tests_1     |       expected web application to include "E2E testing with Docker"
tests_1     |       + expected - actual
tests_1     | 
tests_1     |       -Invalid Host header
tests_1     |       +E2E testing with Docker
tests_1     |

This is because our frontend (webpack-dev-server) is configured to be accessed as http://localhost:3000 but our testing project performs request to http://frontend:3000 So we need to start the frontend in a different way

Go ahead and add a new start script in package.json in the frontend

"start-e2e": "webpack-dev-server --history-api-fallback --progress --host 0.0.0.0 --port 3000 --public frontend  --inline --hot --config webpack.config.js --quiet"

(Notice that we added --public frontend. Again here this name needs to match the name of the module in the docker-compose file, since it is the alias for the server)

And now update the Dockerfile to run this command instead of start

CMD yarn start-e2e

Push changes and rebuild the image

Now it will still fail but with a new error :P

tests_1     | -- FAILURES:
tests_1     | 
tests_1     |   1) Users List: Loads fine:
tests_1     |      .wait() timed out after 1000msec

This is simply because or database is empty so there are no users to be shown.

So, we need to comeup with a script to seed mongo database before launching the tests.

Lets do this in "e2e" module

yarn add mongodb

mkdir src
touch src/seed_db.js

With content

var MongoClient = require('mongodb').MongoClient;
var ObjectId = require('mongodb').ObjectID;

var url = process.env.MONGO_URL;

MongoClient.connect(url, function(err, db) {
  const col = db.collection('users')
  col.insertOne({ name: 'Socrates', created_at: new Date() })
    .then(() => {
      col.insertOne({ name: 'Platon', created_at: new Date() })
    })
    .then(() => {
      col.insertOne({ name: 'Aristoteles', created_at: new Date() })
    })
    .then(() => {
      db.close()
    })
});

(Note: this is pure old javascript and it is not even using mongoose, because our models are defined in the backend. So probably this can be done in another better way. You need to choose between having the e2e seeds in its own project and then face this issue, or another option is to have the seeding behavior in the backend project and run it at start of this docker container, instead of the e2e. I guess that a mix would be to extract your mongoose models and common db logic into a reusable library, then have it as a dependecy in the backend, but also in the e2e module, which will have its own seeding code but reusing models. That's the most involved solution but probably the best)

And add a new script to package.json

    "populate-db": "node src/seed_db.js"

Now to test it run

MONGO_URL='mongodb://localhost/docker-e2e' yarn populate-db

(you need to have a locally installed and running mongo)

No we will change the script that starts the tests to first run this new script before running the tests. In the same package.json

    "test-docker": "yarn populate-db; DEBUG=nightmare xvfb-run --server-args='-screen 0 1024x768x24' codeceptjs run --steps --verbose",

And we need to pass the MONGO_URL env var to the test module in docker-compose.yml

tests:
  build: .
  links:
    - frontend
    - mongo
  environment:
    - CODECEPT_URL=http://frontend:3000
    - MONGO_URL=mongodb://mongo/docker-e2e
  volumes:
    - /tmp/e2e-output:/app/output

(Notice that we changed the syntax of the whole environment section and also introduced a new link to mongo)

Now rebuild the compose image and start it.

docker-compose build
docker-compose run tests

(notice that here we use docker-compose RUN instead of UP, since we want docker to end after the tests have ended, otherwise using UP will make it never complete. We also want to have the exit code from the tests as the exit code of the command)

Within the long log you should see something like this

tests_1     |    Emitted | test.passed ([object Object])
tests_1     |  ✓ OK in 6645ms
tests_1     | 
tests_1     |    Emitted | test.after
tests_1     |    [2] Queued | hook Nightmare._after()
tests_1     | Sun, 07 May 2017 20:41:28 GMT nightmare running
tests_1     |    [3] Starting recording promises
tests_1     |    Emitted | suite.after
tests_1     |    [3] Queued | hook Nightmare._afterSuite()
tests_1     | 
tests_1     |   OK  | 1 passed   // 7s
tests_1     |    Emitted | global.result ([object Object])
tests_1     | Sun, 07 May 2017 20:41:28 GMT nightmare electron child process exited with code 0: success!

Which means that the test passed :)

Huurra ! It is now working.

#

Pending

  • Optimize docker images so that they don't take forever to build (currently npm rebuild takes a lot of time :S and it was the only way to fix an issue with natives libraries)

Debugging tips

To better understand what was happening between the different issues that we faced I did some tricks.

For example to manually accessing the app from a browser in my host I edited 

sudo vi /etc/hosts

And add

127.0.0.1       frontend
127.0.0.1       backend

Then I was able to check the app in http://frontend:3000/

Also in order to check the codecept errors when test failed I configured docker-compose so that the e2e-test image shared a folder with my host.

tests:
  build: .
  links:
    - frontend
  environment:
    CODECEPT_URL: 'http://frontend:3000'
  volumes:
    - /tmp/e2e-output:/app/output

Here the last 2 lines. As codecept will write screenshots in /app/output they will be accesible from outside in /tmp/e2e-output/

< Back | Next >

results matching ""

    No results matching ""