From 700146203af6d5dcfe1a46757e70289009f527b1 Mon Sep 17 00:00:00 2001 From: Ezio Date: Thu, 3 Dec 2015 02:00:25 +0800 Subject: [PATCH 1/6] Create 20151203 Getting started with Docker by Dockerizing this Blog.md --- ...ed with Docker by Dockerizing this Blog.md | 375 ++++++++++++++++++ 1 file changed, 375 insertions(+) create mode 100644 sources/tech/20151203 Getting started with Docker by Dockerizing this Blog.md diff --git a/sources/tech/20151203 Getting started with Docker by Dockerizing this Blog.md b/sources/tech/20151203 Getting started with Docker by Dockerizing this Blog.md new file mode 100644 index 0000000000..1f69a4adba --- /dev/null +++ b/sources/tech/20151203 Getting started with Docker by Dockerizing this Blog.md @@ -0,0 +1,375 @@ +Getting started with Docker by Dockerizing this Blog +====================== +>This article covers the basic concepts of Docker and how to Dockerize an application by creating a custom Dockerfile +>Written by Benjamin Cane on 2015-12-01 10:00:00 + +Docker is an interesting technology that over the past 2 years has gone from an idea, to being used by organizations all over the world to deploy applications. In today's article I am going to cover how to get started with Docker by "Dockerizing" an existing application. The application in question is actually this very blog! + +What is Docker +============ +============ + +Before we dive into learning the basics of Docker let's first understand what Docker is and why it is so popular. Docker, is an operating system container management tool that allows you to easily manage and deploy applications by making it easy to package them within operating system containers. + +### Containers vs. Virtual Machines + +Containers may not be as familiar as virtual machines but they are another method to provide Operating System Virtualization. However, they differ quite a bit from standard virtual machines. + +Standard virtual machines generally include a full Operating System, OS Packages and eventually an Application or two. This is made possible by a Hypervisor which provides hardware virtualization to the virtual machine. This allows for a single server to run many standalone operating systems as virtual guests. + +Containers are similar to virtual machines in that they allow a single server to run multiple operating environments, these environments however, are not full operating systems. Containers generally only include the necessary OS Packages and Applications. They do not generally contain a full operating system or hardware virtualization. This also means that containers have a smaller overhead than traditional virtual machines. + +Containers and Virtual Machines are often seen as conflicting technology, however, this is often a misunderstanding. Virtual Machines are a way to take a physical server and provide a fully functional operating environment that shares those physical resources with other virtual machines. A Container is generally used to isolate a running process within a single host to ensure that the isolated processes cannot interact with other processes within that same system. In fact containers are closer to BSD Jails and chroot'ed processes than full virtual machines. + +### What Docker provides on top of containers + +Docker itself is not a container runtime environment; in fact Docker is actually container technology agnostic with efforts planned for Docker to support Solaris Zones and BSD Jails. What Docker provides is a method of managing, packaging, and deploying containers. While these types of functions may exist to some degree for virtual machines they traditionally have not existed for most container solutions and the ones that existed, were not as easy to use or fully featured as Docker. + +Now that we know what Docker is, let's start learning how Docker works by first installing Docker and deploying a public pre-built container. + +## Starting with Installation +As Docker is not installed by default step 1 will be to install the Docker package; since our example system is running Ubuntu 14.0.4 we will do this using the Apt package manager. + +# apt-get install docker.io +Reading package lists... Done +Building dependency tree +Reading state information... Done +The following extra packages will be installed: + aufs-tools cgroup-lite git git-man liberror-perl +Suggested packages: + btrfs-tools debootstrap lxc rinse git-daemon-run git-daemon-sysvinit git-doc + git-el git-email git-gui gitk gitweb git-arch git-bzr git-cvs git-mediawiki + git-svn +The following NEW packages will be installed: + aufs-tools cgroup-lite docker.io git git-man liberror-perl +0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded. +Need to get 7,553 kB of archives. +After this operation, 46.6 MB of additional disk space will be used. +Do you want to continue? [Y/n] y +To check if any containers are running we can execute the docker command using the ps option. + +# docker ps +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +The ps function of the docker command works similar to the Linux ps command. It will show available Docker containers and their current status. Since we have not started any Docker containers yet, the command shows no running containers. + +## Deploying a pre-built nginx Docker container +One of my favorite features of Docker is the ability to deploy a pre-built container in the same way you would deploy a package with yum or apt-get. To explain this better let's deploy a pre-built container running the nginx web server. We can do this by executing the docker command again, however, this time with the run option. + +# docker run -d nginx +Unable to find image 'nginx' locally +Pulling repository nginx +5c82215b03d1: Download complete +e2a4fb18da48: Download complete +58016a5acc80: Download complete +657abfa43d82: Download complete +dcb2fe003d16: Download complete +c79a417d7c6f: Download complete +abb90243122c: Download complete +d6137c9e2964: Download complete +85e566ddc7ef: Download complete +69f100eb42b5: Download complete +cd720b803060: Download complete +7cc81e9a118a: Download complete +The run function of the docker command tells Docker to find a specified Docker image and start a container running that image. By default, Docker containers run in the foreground, meaning when you execute docker run your shell will be bound to the container's console and the process running within the container. In order to launch this Docker container in the background I included the -d (detach) flag. + +By executing docker ps again we can see the nginx container running. + +# docker ps +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +f6d31ab01fc9 nginx:latest nginx -g 'daemon off 4 seconds ago Up 3 seconds 443/tcp, 80/tcp desperate_lalande +In the above output we can see the running container desperate_lalande and that this container has been built from the nginx:latest image. + +## Docker Images +Images are one of Docker's key features and is similar to a virtual machine image. Like virtual machine images, a Docker image is a container that has been saved and packaged. Docker however, doesn't just stop with the ability to create images. Docker also includes the ability to distribute those images via Docker repositories which are a similar concept to package repositories. This is what gives Docker the ability to deploy an image like you would deploy a package with yum. To get a better understanding of how this works let's look back at the output of the docker run execution. + +# docker run -d nginx +Unable to find image 'nginx' locally +The first message we see is that docker could not find an image named nginx locally. The reason we see this message is that when we executed docker run we told Docker to startup a container, a container based on an image named nginx. Since Docker is starting a container based on a specified image it needs to first find that image. Before checking any remote repository Docker first checks locally to see if there is a local image with the specified name. + +Since this system is brand new there is no Docker image with the name nginx, which means Docker will need to download it from a Docker repository. + +Pulling repository nginx +5c82215b03d1: Download complete +e2a4fb18da48: Download complete +58016a5acc80: Download complete +657abfa43d82: Download complete +dcb2fe003d16: Download complete +c79a417d7c6f: Download complete +abb90243122c: Download complete +d6137c9e2964: Download complete +85e566ddc7ef: Download complete +69f100eb42b5: Download complete +cd720b803060: Download complete +7cc81e9a118a: Download complete +This is exactly what the second part of the output is showing us. By default, Docker uses the Docker Hub repository, which is a repository service that Docker (the company) runs. + +Like GitHub, Docker Hub is free for public repositories but requires a subscription for private repositories. It is possible however, to deploy your own Docker repository, in fact it is as easy as docker run registry. For this article we will not be deploying a custom registry service. + +## Stopping and Removing the Container +Before moving on to building a custom Docker container let's first clean up our Docker environment. We will do this by stopping the container from earlier and removing it. + +To start a container we executed docker with the run option, in order to stop this same container we simply need to execute the docker with the kill option specifying the container name. + +# docker kill desperate_lalande +desperate_lalande +If we execute docker ps again we will see that the container is no longer running. + +# docker ps +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +However, at this point we have only stopped the container; while it may no longer be running it still exists. By default, docker ps will only show running containers, if we add the -a (all) flag it will show all containers running or not. + +# docker ps -a +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +f6d31ab01fc9 5c82215b03d1 nginx -g 'daemon off 4 weeks ago Exited (-1) About a minute ago desperate_lalande +In order to fully remove the container we can use the docker command with the rm option. + +# docker rm desperate_lalande +desperate_lalande +While this container has been removed; we still have a nginx image available. If we were to re-run docker run -d nginx again the container would be started without having to fetch the nginx image again. This is because Docker already has a saved copy on our local system. + +To see a full list of local images we can simply run the docker command with the images option. + +# docker images +REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE +nginx latest 9fab4090484a 5 days ago 132.8 MB +## Building our own custom image +At this point we have used a few basic Docker commands to start, stop and remove a common pre-built image. In order to "Dockerize" this blog however, we are going to have to build our own Docker image and that means creating a Dockerfile. + +With most virtual machine environments if you wish to create an image of a machine you need to first create a new virtual machine, install the OS, install the application and then finally convert it to a template or image. With Docker however, these steps are automated via a Dockerfile. A Dockerfile is a way of providing build instructions to Docker for the creation of a custom image. In this section we are going to build a custom Dockerfile that can be used to deploy this blog. + +### Understanding the Application +Before we can jump into creating a Dockerfile we first need to understand what is required to deploy this blog. + +The blog itself is actually static HTML pages generated by a custom static site generator that I wrote named; hamerkop. The generator is very simple and more about getting the job done for this blog specifically. All the code and source files for this blog are available via a public GitHub repository. In order to deploy this blog we simply need to grab the contents of the GitHub repository, install Python along with some Python modules and execute the hamerkop application. To serve the generated content we will use nginx; which means we will also need nginx to be installed. + +So far this should be a pretty simple Dockerfile, but it will show us quite a bit of the Dockerfile Syntax. To get started we can clone the GitHub repository and creating a Dockerfile with our favorite editor; vi in my case. + +# git clone https://github.com/madflojo/blog.git +Cloning into 'blog'... +remote: Counting objects: 622, done. +remote: Total 622 (delta 0), reused 0 (delta 0), pack-reused 622 +Receiving objects: 100% (622/622), 14.80 MiB | 1.06 MiB/s, done. +Resolving deltas: 100% (242/242), done. +Checking connectivity... done. +# cd blog/ +# vi Dockerfile +### FROM - Inheriting a Docker image +The first instruction of a Dockerfile is the FROM instruction. This is used to specify an existing Docker image to use as our base image. This basically provides us with a way to inherit another Docker image. In this case we will be starting with the same nginx image we were using before, if we wanted to start with a blank slate we could use the Ubuntu Docker image by specifying ubuntu:latest. + +## Dockerfile that generates an instance of http://bencane.com + +FROM nginx:latest +MAINTAINER Benjamin Cane +In addition to the FROM instruction, I also included a MAINTAINER instruction which is used to show the Author of the Dockerfile. + +As Docker supports using # as a comment marker, I will be using this syntax quite a bit to explain the sections of this Dockerfile. + +### Running a test build +Since we inherited the nginx Docker image our current Dockerfile also inherited all the instructions within the Dockerfile used to build that nginx image. What this means is even at this point we are able to build a Docker image from this Dockerfile and run a container from that image. The resulting image will essentially be the same as the nginx image but we will run through a build of this Dockerfile now and a few more times as we go to help explain the Docker build process. + +In order to start the build from a Dockerfile we can simply execute the docker command with the build option. + +# docker build -t blog /root/blog +Sending build context to Docker daemon 23.6 MB +Sending build context to Docker daemon +Step 0 : FROM nginx:latest + ---> 9fab4090484a +Step 1 : MAINTAINER Benjamin Cane + ---> Running in c97f36450343 + ---> 60a44f78d194 +Removing intermediate container c97f36450343 +Successfully built 60a44f78d194 +In the above example I used the -t (tag) flag to "tag" the image as "blog". This essentially allows us to name the image, without specifying a tag the image would only be callable via an Image ID that Docker assigns. In this case the Image ID is 60a44f78d194 which we can see from the docker command's build success message. + +In addition to the -t flag, I also specified the directory /root/blog. This directory is the "build directory", which is the directory that contains the Dockerfile and any other files necessary to build this container. + +Now that we have run through a successful build, let's start customizing this image. + +### Using RUN to execute apt-get +The static site generator used to generate the HTML pages is written in Python and because of this the first custom task we should perform within this Dockerfile is to install Python. To install the Python package we will use the Apt package manager. This means we will need to specify within the Dockerfile that apt-get update and apt-get install python-dev are executed; we can do this with the RUN instruction. + +## Dockerfile that generates an instance of http://bencane.com + +FROM nginx:latest +MAINTAINER Benjamin Cane + +## Install python and pip +RUN apt-get update +RUN apt-get install -y python-dev python-pip +In the above we are simply using the RUN instruction to tell Docker that when it builds this image it will need to execute the specified apt-get commands. The interesting part of this is that these commands are only executed within the context of this container. What this means is even though python-dev and python-pip are being installed within the container, they are not being installed for the host itself. Or to put it simplier, within the container the pip command will execute, outside the container, the pip command does not exist. + +It is also important to note that the Docker build process does not accept user input during the build. This means that any commands being executed by the RUN instruction must complete without user input. This adds a bit of complexity to the build process as many applications require user input during installation. For our example, none of the commands executed by RUN require user input. + +### Installing Python modules +With Python installed we now need to install some Python modules. To do this outside of Docker, we would generally use the pip command and reference a file within the blog's Git repository named requirements.txt. In an earlier step we used the git command to "clone" the blog's GitHub repository to the /root/blog directory; this directory also happens to be the directory that we have created the Dockerfile. This is important as it means the contents of the Git repository are accessible to Docker during the build process. + +When executing a build, Docker will set the context of the build to the specified "build directory". This means that any files within that directory and below can be used during the build process, files outside of that directory (outside of the build context), are inaccessible. + +In order to install the required Python modules we will need to copy the requirements.txt file from the build directory into the container. We can do this using the COPY instruction within the Dockerfile. + +## Dockerfile that generates an instance of http://bencane.com + +FROM nginx:latest +MAINTAINER Benjamin Cane + +## Install python and pip +RUN apt-get update +RUN apt-get install -y python-dev python-pip + +## Create a directory for required files +RUN mkdir -p /build/ + +## Add requirements file and run pip +COPY requirements.txt /build/ +RUN pip install -r /build/requirements.txt +Within the Dockerfile we added 3 instructions. The first instruction uses RUN to create a /build/ directory within the container. This directory will be used to copy any application files needed to generate the static HTML pages. The second instruction is the COPY instruction which copies the requirements.txt file from the "build directory" (/root/blog) into the /build directory within the container. The third is using the RUN instruction to execute the pip command; installing all the modules specified within the requirements.txt file. + +COPY is an important instruction to understand when building custom images. Without specifically copying the file within the Dockerfile this Docker image would not contain the requirements.txt file. With Docker containers everything is isolated, unless specifically executed within a Dockerfile a container is not likely to include required dependencies. + +### Re-running a build +Now that we have a few customization tasks for Docker to perform let's try another build of the blog image again. + +# docker build -t blog /root/blog +Sending build context to Docker daemon 19.52 MB +Sending build context to Docker daemon +Step 0 : FROM nginx:latest + ---> 9fab4090484a +Step 1 : MAINTAINER Benjamin Cane + ---> Using cache + ---> 8e0f1899d1eb +Step 2 : RUN apt-get update + ---> Using cache + ---> 78b36ef1a1a2 +Step 3 : RUN apt-get install -y python-dev python-pip + ---> Using cache + ---> ef4f9382658a +Step 4 : RUN mkdir -p /build/ + ---> Running in bde05cf1e8fe + ---> f4b66e09fa61 +Removing intermediate container bde05cf1e8fe +Step 5 : COPY requirements.txt /build/ + ---> cef11c3fb97c +Removing intermediate container 9aa8ff43f4b0 +Step 6 : RUN pip install -r /build/requirements.txt + ---> Running in c50b15ddd8b1 +Downloading/unpacking jinja2 (from -r /build/requirements.txt (line 1)) +Downloading/unpacking PyYaml (from -r /build/requirements.txt (line 2)) + +Successfully installed jinja2 PyYaml mistune markdown MarkupSafe +Cleaning up... + ---> abab55c20962 +Removing intermediate container c50b15ddd8b1 +Successfully built abab55c20962 +From the above build output we can see the build was successful, but we can also see another interesting message; ---> Using cache. What this message is telling us is that Docker was able to use its build cache during the build of this image. + +#### Docker build cache + +When Docker is building an image, it doesn't just build a single image; it actually builds multiple images throughout the build processes. In fact we can see from the above output that after each "Step" Docker is creating a new image. + + Step 5 : COPY requirements.txt /build/ + ---> cef11c3fb97c +The last line from the above snippet is actually Docker informing us of the creating of a new image, it does this by printing the Image ID; cef11c3fb97c. The useful thing about this approach is that Docker is able to use these images as cache during subsequent builds of the blog image. This is useful because it allows Docker to speed up the build process for new builds of the same container. If we look at the example above we can actually see that rather than installing the python-dev and python-pip packages again, Docker was able to use a cached image. However, since Docker was unable to find a build that executed the mkdir command, each subsequent step was executed. + +The Docker build cache is a bit of a gift and a curse; the reason for this is that the decision to use cache or to rerun the instruction is made within a very narrow scope. For example, if there was a change to the requirements.txt file Docker would detect this change during the build and start fresh from that point forward. It does this because it can view the contents of the requirements.txt file. The execution of the apt-get commands however, are another story. If the Apt repository that provides the Python packages were to contain a newer version of the python-pip package; Docker would not be able to detect the change and would simply use the build cache. This means that an older package may be installed. While this may not be a major issue for the python-pip package it could be a problem if the installation was caching a package with a known vulnerability. + +For this reason it is useful to periodically rebuild the image without using Docker's cache. To do this you can simply specify --no-cache=True when executing a Docker build. + +### Deploying the rest of the blog +With the Python packages and modules installed this leaves us at the point of copying the required application files and running the hamerkop application. To do this we will simply use more COPY and RUN instructions. + +## Dockerfile that generates an instance of http://bencane.com + +FROM nginx:latest +MAINTAINER Benjamin Cane + +## Install python and pip +RUN apt-get update +RUN apt-get install -y python-dev python-pip + +## Create a directory for required files +RUN mkdir -p /build/ + +## Add requirements file and run pip +COPY requirements.txt /build/ +RUN pip install -r /build/requirements.txt + +## Add blog code nd required files +COPY static /build/static +COPY templates /build/templates +COPY hamerkop /build/ +COPY config.yml /build/ +COPY articles /build/articles + +## Run Generator +RUN /build/hamerkop -c /build/config.yml +Now that we have the rest of the build instructions, let's run through another build and verify that the image builds successfully. + +# docker build -t blog /root/blog/ +Sending build context to Docker daemon 19.52 MB +Sending build context to Docker daemon +Step 0 : FROM nginx:latest + ---> 9fab4090484a +Step 1 : MAINTAINER Benjamin Cane + ---> Using cache + ---> 8e0f1899d1eb +Step 2 : RUN apt-get update + ---> Using cache + ---> 78b36ef1a1a2 +Step 3 : RUN apt-get install -y python-dev python-pip + ---> Using cache + ---> ef4f9382658a +Step 4 : RUN mkdir -p /build/ + ---> Using cache + ---> f4b66e09fa61 +Step 5 : COPY requirements.txt /build/ + ---> Using cache + ---> cef11c3fb97c +Step 6 : RUN pip install -r /build/requirements.txt + ---> Using cache + ---> abab55c20962 +Step 7 : COPY static /build/static + ---> 15cb91531038 +Removing intermediate container d478b42b7906 +Step 8 : COPY templates /build/templates + ---> ecded5d1a52e +Removing intermediate container ac2390607e9f +Step 9 : COPY hamerkop /build/ + ---> 59efd1ca1771 +Removing intermediate container b5fbf7e817b7 +Step 10 : COPY config.yml /build/ + ---> bfa3db6c05b7 +Removing intermediate container 1aebef300933 +Step 11 : COPY articles /build/articles + ---> 6b61cc9dde27 +Removing intermediate container be78d0eb1213 +Step 12 : RUN /build/hamerkop -c /build/config.yml + ---> Running in fbc0b5e574c5 +Successfully created file /usr/share/nginx/html//2011/06/25/checking-the-number-of-lwp-threads-in-linux +Successfully created file /usr/share/nginx/html//2011/06/checking-the-number-of-lwp-threads-in-linux + +Successfully created file /usr/share/nginx/html//archive.html +Successfully created file /usr/share/nginx/html//sitemap.xml + ---> 3b25263113e1 +Removing intermediate container fbc0b5e574c5 +Successfully built 3b25263113e1 +### Running a custom container +With a successful build we can now start our custom container by running the docker command with the run option, similar to how we started the nginx container earlier. + +# docker run -d -p 80:80 --name=blog blog +5f6c7a2217dcdc0da8af05225c4d1294e3e6bb28a41ea898a1c63fb821989ba1 +Once again the -d (detach) flag was used to tell Docker to run the container in the background. However, there are also two new flags. The first new flag is --name, which is used to give the container a user specified name. In the earlier example we did not specify a name and because of that Docker randomly generated one. The second new flag is -p, this flag allows users to map a port from the host machine to a port within the container. + +The base nginx image we used exposes port 80 for the HTTP service. By default, ports bound within a Docker container are not bound on the host system as a whole. In order for external systems to access ports exposed within a container the ports must be mapped from a host port to a container port using the -p flag. The command above maps port 80 from the host, to port 80 within the container. If we wished to map port 8080 from the host, to port 80 within the container we could do so by specifying the ports in the following syntax -p 8080:80. + +From the above command it appears that our container was started successfully, we can verify this by executing docker ps. + +# docker ps +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +d264c7ef92bd blog:latest nginx -g 'daemon off 3 seconds ago Up 3 seconds 443/tcp, 0.0.0.0:80->80/tcp blog +## Wrapping up + +At this point we now have a running custom Docker container. While we touched on a few Dockerfile instructions within this article we have yet to discuss all the instructions. For a full list of Dockerfile instructions you can checkout Docker's reference page, which explains the instructions very well. + +Another good resource is their Dockerfile Best Practices page which contains quite a few best practices for building custom Dockerfiles. Some of these tips are very useful such as strategically ordering the commands within the Dockerfile. In the above examples our Dockerfile has the COPY instruction for the articles directory as the last COPY instruction. The reason for this is that the articles directory will change quite often. It's best to put instructions that will change oftenat the lowest point possible within the Dockerfile to optimize steps that can be cached. + +In this article we covered how to start a pre-built container and how to build, then deploy a custom container. While there is quite a bit to learn about Docker this article should give you a good idea on how to get started. Of course, as always if you think there is anything that should be added drop it in the comments below. From 5102620ae4d2163a2dd09ed163b4fb69d5802eb5 Mon Sep 17 00:00:00 2001 From: ezio Date: Tue, 8 Dec 2015 10:24:21 +0800 Subject: [PATCH 2/6] update --- .../20151122 Doubly linked list in the Linux Kernel.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/sources/tech/20151122 Doubly linked list in the Linux Kernel.md b/sources/tech/20151122 Doubly linked list in the Linux Kernel.md index 4d6ff02ab8..96a515fe93 100644 --- a/sources/tech/20151122 Doubly linked list in the Linux Kernel.md +++ b/sources/tech/20151122 Doubly linked list in the Linux Kernel.md @@ -3,12 +3,14 @@ translating by Ezio Data Structures in the Linux Kernel ================================================================================ -Doubly linked list +双向链表 -------------------------------------------------------------------------------- Linux kernel provides its own implementation of doubly linked list, which you can find in the [include/linux/list.h](https://github.com/torvalds/linux/blob/master/include/linux/list.h). We will start `Data Structures in the Linux kernel` from the doubly linked list data structure. Why? Because it is very popular in the kernel, just try to [search](http://lxr.free-electrons.com/ident?i=list_head) +Linux 内核自己实现了双向链表，可以在[include/linux/list.h](https://github.com/torvalds/linux/blob/master/include/linux/list.h)找到定义。我们将会从双向链表数据结构开始`内核的数据结构`。为什么？因为它在内核里使用的很广泛，你只需要在[free-electrons.com](http://lxr.free-electrons.com/ident?i=list_head) 检索一下就知道了。 First of all, let's look on the main structure in the [include/linux/types.h](https://github.com/torvalds/linux/blob/master/include/linux/types.h): +首先让我们看一下在[include/linux/types.h](https://github.com/torvalds/linux/blob/master/include/linux/types.h) 里的主结构体： ```C struct list_head { @@ -17,6 +19,7 @@ struct list_head { ``` You can note that it is different from many implementations of doubly linked list which you have seen. For example, this doubly linked list structure from the [glib](http://www.gnu.org/software/libc/) library looks like : +你可能注意到这和你以前见过的双向链表的实现方法是不同的。举个例子来说，在[glib](http://www.gnu.org/software/libc/) 库里是这样实现的： ```C struct GList { @@ -27,8 +30,10 @@ struct GList { ``` Usually a linked list structure contains a pointer to the item. The implementation of linked list in Linux kernel does not. So the main question is - `where does the list store the data?`. The actual implementation of linked list in the kernel is - `Intrusive list`. An intrusive linked list does not contain data in its nodes - A node just contains pointers to the next and previous node and list nodes part of the data that are added to the list. This makes the data structure generic, so it does not care about entry data type anymore. +通常来说一个链表会包含一个指向某个项目的指针。但是内核的实现并没有这样做。所以问题来了：`链表在哪里保存数据呢？`。实际上内核里实现的链表实际上是`侵入式链表`。侵入式链表并不在节点内保存数据-节点仅仅包含指向前后节点的指针，然后把数据是附加到链表的。这就使得这个数据结构是通用的，使用起来就不需要考虑节点数据的类型了。 For example: +比如： ```C struct nmi_desc { From 03c4e1a402e9dd593ba5848b8b34f5bbaa3f7fa8 Mon Sep 17 00:00:00 2001 From: ezio Date: Wed, 9 Dec 2015 09:55:49 +0800 Subject: [PATCH 3/6] translate finished --- ... Doubly linked list in the Linux Kernel.md | 51 ++++++++++++++----- 1 file changed, 38 insertions(+), 13 deletions(-) diff --git a/sources/tech/20151122 Doubly linked list in the Linux Kernel.md b/sources/tech/20151122 Doubly linked list in the Linux Kernel.md index 96a515fe93..00da2d9d00 100644 --- a/sources/tech/20151122 Doubly linked list in the Linux Kernel.md +++ b/sources/tech/20151122 Doubly linked list in the Linux Kernel.md @@ -1,6 +1,4 @@ -translating by Ezio - -Data Structures in the Linux Kernel +Data Structures in the Linux Kernel——Doubly linked list ================================================================================ 双向链表 @@ -43,12 +41,14 @@ struct nmi_desc { ``` Let's look at some examples to understand how `list_head` is used in the kernel. As I already wrote about, there are many, really many different places where lists are used in the kernel. Let's look for an example in miscellaneous character drivers. Misc character drivers API from the [drivers/char/misc.c](https://github.com/torvalds/linux/blob/master/drivers/char/misc.c) is used for writing small drivers for handling simple hardware or virtual devices. Those drivers share same major number: +让我们看几个例子来理解一下在内核里是如何使用`list_head` 的。如上所述，在内核里有实在很多不同的地方用到了链表。我们来看一个在杂项字符驱动里面的使用的例子。在 [drivers/char/misc.c](https://github.com/torvalds/linux/blob/master/drivers/char/misc.c) 的杂项字符驱动API 被用来编写处理小型硬件和虚拟设备的小驱动。这些驱动共享相同的主设备号： ```C #define MISC_MAJOR 10 ``` but have their own minor number. For example you can see it with: +但是都有各自不同的次设备号。比如： ``` ls -l /dev | grep 10 @@ -75,6 +75,7 @@ crw------- 1 root root 10, 137 Mar 21 12:01 vhci ``` Now let's have a close look at how lists are used in the misc device drivers. First of all, let's look on `miscdevice` structure: +现在让我们看看它是如何使用链表的。首先看一下结构体`miscdevice`： ```C struct miscdevice @@ -91,12 +92,14 @@ struct miscdevice ``` We can see the fourth field in the `miscdevice` structure - `list` which is a list of registered devices. In the beginning of the source code file we can see the definition of misc_list: +可以看到结构体的第四个变量`list` 是所有注册过的设备的链表。在源代码文件的开始可以看到这个链表的定义： ```C static LIST_HEAD(misc_list); ``` which expands to the definition of variables with `list_head` type: +它实际上是对用`list_head` 类型定义的变量的扩展。 ```C #define LIST_HEAD(name) \ @@ -104,18 +107,21 @@ which expands to the definition of variables with `list_head` type: ``` and initializes it with the `LIST_HEAD_INIT` macro, which sets previous and next entries with the address of variable - name: +然后使用宏`LIST_HEAD_INIT` 进行初始化，这会使用变量`name` 的地址来填充`prev`和`next` 结构体的两个变量。 ```C #define LIST_HEAD_INIT(name) { &(name), &(name) } ``` Now let's look on the `misc_register` function which registers a miscellaneous device. At the start it initializes `miscdevice->list` with the `INIT_LIST_HEAD` function: +现在来看看注册杂项设备的函数`misc_register`。它在开始就用 `INIT_LIST_HEAD` 初始化了`miscdevice->list`。 ```C INIT_LIST_HEAD(&misc->list); ``` which does the same as the `LIST_HEAD_INIT` macro: +作用和宏`LIST_HEAD_INIT`一样。 ```C static inline void INIT_LIST_HEAD(struct list_head *list) @@ -126,12 +132,15 @@ static inline void INIT_LIST_HEAD(struct list_head *list) ``` In the next step after a device is created by the `device_create` function, we add it to the miscellaneous devices list with: +在函数`device_create` 创建了设备后我们就用下面的语句将设备添加到设备链表： ``` list_add(&misc->list, &misc_list); ``` Kernel `list.h` provides this API for the addition of a new entry to the list. Let's look at its implementation: +内核文件`list.h` 提供了项链表添加新项的API 接口。我们来看看它的实现： + ```C static inline void list_add(struct list_head *new, struct list_head *head) @@ -141,12 +150,14 @@ static inline void list_add(struct list_head *new, struct list_head *head) ``` It just calls internal function `__list_add` with the 3 given parameters: +实际上就是使用3个指定的参数来调用了内部函数`__list_add`： -* new - new entry. -* head - list head after which the new item will be inserted. -* head->next - next item after list head. +* new - 新项。 +* head - 新项将会被添加到`head`之前. +* head->next - `head` 之后的项。 Implementation of the `__list_add` is pretty simple: +`__list_add`的实现非常简单： ```C static inline void __list_add(struct list_head *new, @@ -161,8 +172,10 @@ static inline void __list_add(struct list_head *new, ``` Here we add a new item between `prev` and `next`. So `misc` list which we defined at the start with the `LIST_HEAD_INIT` macro will contain previous and next pointers to the `miscdevice->list`. +我们会在`prev`和`next` 之间添加一个新项。所以我们用宏`LIST_HEAD_INIT`定义的`misc` 链表会包含指向`miscdevice->list` 的向前指针和向后指针。 There is still one question: how to get list's entry. There is a special macro: +这里有一个问题：如何得到列表的内容呢？这里有一个特殊的宏： ```C #define list_entry(ptr, type, member) \ @@ -170,25 +183,29 @@ There is still one question: how to get list's entry. There is a special macro: ``` which gets three parameters: +使用了三个参数： -* ptr - the structure list_head pointer; -* type - structure type; -* member - the name of the list_head within the structure; +* ptr - 指向链表头的指针； +* type - 结构体类型; +* member - 在结构体内类型为`list_head` 的变量的名字； For example: +比如说： ```C const struct miscdevice *p = list_entry(v, struct miscdevice, list) ``` After this we can access to any `miscdevice` field with `p->minor` or `p->name` and etc... Let's look on the `list_entry` implementation: - +然后我们就可以使用`p->minor` 或者 `p->name`来访问`miscdevice`。让我们来看看`list_entry` 的实现： + ```C #define list_entry(ptr, type, member) \ container_of(ptr, type, member) ``` As we can see it just calls `container_of` macro with the same arguments. At first sight, the `container_of` looks strange: +如我们所见，它仅仅使用相同的参数调用了宏`container_of`。初看这个宏挺奇怪的： ```C #define container_of(ptr, type, member) ({ \ @@ -197,8 +214,10 @@ As we can see it just calls `container_of` macro with the same arguments. At fir ``` First of all you can note that it consists of two expressions in curly brackets. The compiler will evaluate the whole block in the curly braces and use the value of the last expression. +首先你可以注意到花括号内包含两个表达式。编译器会执行花括号内的全部语句，然后返回最后的表达式的值。 For example: +举个例子来说： ``` #include @@ -211,8 +230,10 @@ int main() { ``` will print `2`. +最终会打印`2` The next point is `typeof`, it's simple. As you can understand from its name, it just returns the type of the given variable. When I first saw the implementation of the `container_of` macro, the strangest thing I found was the zero in the `((type *)0)` expression. Actually this pointer magic calculates the offset of the given field from the address of the structure, but as we have `0` here, it will be just a zero offset along with the field width. Let's look at a simple example: +下一点就是`typeof`,它也很简单。就如你从名字所理解的，它仅仅返回了给定变量的类型。当我第一次看到宏`container_of`的实现时，让我觉得最奇怪的就是`container_of`中的0.实际上这个指针巧妙的计算了从结构体特定变量的偏移，这里的`0`刚好就是位宽里的零偏移。让我们看一个简单的例子： ```C #include @@ -220,7 +241,7 @@ The next point is `typeof`, it's simple. As you can understand from its name, it struct s { int field1; char field2; - char field3; + char field3; }; int main() { @@ -230,16 +251,20 @@ int main() { ``` will print `0x5`. +结果显示`0x5`。 The next `offsetof` macro calculates offset from the beginning of the structure to the given structure's field. Its implementation is very similar to the previous code: +下一个宏`offsetof` 会计算从结构体的某个变量的相对于结构体起始地址的偏移。它的实现和上面类似： ```C #define offsetof(TYPE, MEMBER) ((size_t) &((TYPE *)0)->MEMBER) ``` Let's summarize all about `container_of` macro. The `container_of` macro returns the address of the structure by the given address of the structure's field with `list_head` type, the name of the structure field with `list_head` type and type of the container structure. At the first line this macro declares the `__mptr` pointer which points to the field of the structure that `ptr` points to and assigns `ptr` to it. Now `ptr` and `__mptr` point to the same address. Technically we don't need this line but it's useful for type checking. The first line ensures that the given structure (`type` parameter) has a member called `member`. In the second line it calculates offset of the field from the structure with the `offsetof` macro and subtracts it from the structure address. That's all. +现在我们来总结一下宏`container_of`。只需要知道结构体里面类型为`list_head` 的变量的名字和结构体容器的类型，它可以通过结构体的变量`list_head`获得结构体的起始地址。在宏定义的第一行，声明了一个指向结构体成员变量`ptr`的指针`__mptr`，并且把`ptr` 的地址赋给它。现在`ptr` 和`__mptr` 指向了同一个地址。从技术上讲我们并不需要这一行，但是它可以方便的进行类型检查。第一行保证了特定的结构体（参数`type`）包含成员变量`member`。第二行代码会用宏`offsetof`计算成员变量相对于结构体起始地址的偏移，然后从结构体的地址减去这个偏移，最后就得到了结构体。 Of course `list_add` and `list_entry` is not the only functions which `` provides. Implementation of the doubly linked list provides the following API: +当然了`list_add` 和 `list_entry`不是``提供的唯一功能。双向链表的实现还提供了如下API： * list_add * list_add_tail @@ -254,11 +279,11 @@ Of course `list_add` and `list_entry` is not the only functions which ` Date: Wed, 9 Dec 2015 09:58:27 +0800 Subject: [PATCH 4/6] clean --- ... Doubly linked list in the Linux Kernel.md | 37 ++----------------- 1 file changed, 3 insertions(+), 34 deletions(-) diff --git a/sources/tech/20151122 Doubly linked list in the Linux Kernel.md b/sources/tech/20151122 Doubly linked list in the Linux Kernel.md index 00da2d9d00..631d918813 100644 --- a/sources/tech/20151122 Doubly linked list in the Linux Kernel.md +++ b/sources/tech/20151122 Doubly linked list in the Linux Kernel.md @@ -1,13 +1,12 @@ -Data Structures in the Linux Kernel——Doubly linked list +Linux 内核里的数据结构——双向链表 ================================================================================ 双向链表 -------------------------------------------------------------------------------- -Linux kernel provides its own implementation of doubly linked list, which you can find in the [include/linux/list.h](https://github.com/torvalds/linux/blob/master/include/linux/list.h). We will start `Data Structures in the Linux kernel` from the doubly linked list data structure. Why? Because it is very popular in the kernel, just try to [search](http://lxr.free-electrons.com/ident?i=list_head) + Linux 内核自己实现了双向链表，可以在[include/linux/list.h](https://github.com/torvalds/linux/blob/master/include/linux/list.h)找到定义。我们将会从双向链表数据结构开始`内核的数据结构`。为什么？因为它在内核里使用的很广泛，你只需要在[free-electrons.com](http://lxr.free-electrons.com/ident?i=list_head) 检索一下就知道了。 -First of all, let's look on the main structure in the [include/linux/types.h](https://github.com/torvalds/linux/blob/master/include/linux/types.h): 首先让我们看一下在[include/linux/types.h](https://github.com/torvalds/linux/blob/master/include/linux/types.h) 里的主结构体： ```C @@ -16,7 +15,6 @@ struct list_head { }; ``` -You can note that it is different from many implementations of doubly linked list which you have seen. For example, this doubly linked list structure from the [glib](http://www.gnu.org/software/libc/) library looks like : 你可能注意到这和你以前见过的双向链表的实现方法是不同的。举个例子来说，在[glib](http://www.gnu.org/software/libc/) 库里是这样实现的： ```C @@ -27,10 +25,8 @@ struct GList { }; ``` -Usually a linked list structure contains a pointer to the item. The implementation of linked list in Linux kernel does not. So the main question is - `where does the list store the data?`. The actual implementation of linked list in the kernel is - `Intrusive list`. An intrusive linked list does not contain data in its nodes - A node just contains pointers to the next and previous node and list nodes part of the data that are added to the list. This makes the data structure generic, so it does not care about entry data type anymore. 通常来说一个链表会包含一个指向某个项目的指针。但是内核的实现并没有这样做。所以问题来了：`链表在哪里保存数据呢？`。实际上内核里实现的链表实际上是`侵入式链表`。侵入式链表并不在节点内保存数据-节点仅仅包含指向前后节点的指针，然后把数据是附加到链表的。这就使得这个数据结构是通用的，使用起来就不需要考虑节点数据的类型了。 -For example: 比如： ```C @@ -40,14 +36,12 @@ struct nmi_desc { }; ``` -Let's look at some examples to understand how `list_head` is used in the kernel. As I already wrote about, there are many, really many different places where lists are used in the kernel. Let's look for an example in miscellaneous character drivers. Misc character drivers API from the [drivers/char/misc.c](https://github.com/torvalds/linux/blob/master/drivers/char/misc.c) is used for writing small drivers for handling simple hardware or virtual devices. Those drivers share same major number: 让我们看几个例子来理解一下在内核里是如何使用`list_head` 的。如上所述，在内核里有实在很多不同的地方用到了链表。我们来看一个在杂项字符驱动里面的使用的例子。在 [drivers/char/misc.c](https://github.com/torvalds/linux/blob/master/drivers/char/misc.c) 的杂项字符驱动API 被用来编写处理小型硬件和虚拟设备的小驱动。这些驱动共享相同的主设备号： ```C #define MISC_MAJOR 10 ``` -but have their own minor number. For example you can see it with: 但是都有各自不同的次设备号。比如： ``` @@ -74,7 +68,6 @@ crw------- 1 root root 10, 63 Mar 21 12:01 vga_arbiter crw------- 1 root root 10, 137 Mar 21 12:01 vhci ``` -Now let's have a close look at how lists are used in the misc device drivers. First of all, let's look on `miscdevice` structure: 现在让我们看看它是如何使用链表的。首先看一下结构体`miscdevice`： ```C @@ -91,14 +84,12 @@ struct miscdevice }; ``` -We can see the fourth field in the `miscdevice` structure - `list` which is a list of registered devices. In the beginning of the source code file we can see the definition of misc_list: 可以看到结构体的第四个变量`list` 是所有注册过的设备的链表。在源代码文件的开始可以看到这个链表的定义： ```C static LIST_HEAD(misc_list); ``` -which expands to the definition of variables with `list_head` type: 它实际上是对用`list_head` 类型定义的变量的扩展。 ```C @@ -106,21 +97,18 @@ which expands to the definition of variables with `list_head` type: struct list_head name = LIST_HEAD_INIT(name) ``` -and initializes it with the `LIST_HEAD_INIT` macro, which sets previous and next entries with the address of variable - name: 然后使用宏`LIST_HEAD_INIT` 进行初始化，这会使用变量`name` 的地址来填充`prev`和`next` 结构体的两个变量。 ```C #define LIST_HEAD_INIT(name) { &(name), &(name) } ``` -Now let's look on the `misc_register` function which registers a miscellaneous device. At the start it initializes `miscdevice->list` with the `INIT_LIST_HEAD` function: 现在来看看注册杂项设备的函数`misc_register`。它在开始就用 `INIT_LIST_HEAD` 初始化了`miscdevice->list`。 ```C INIT_LIST_HEAD(&misc->list); ``` -which does the same as the `LIST_HEAD_INIT` macro: 作用和宏`LIST_HEAD_INIT`一样。 ```C @@ -131,14 +119,12 @@ static inline void INIT_LIST_HEAD(struct list_head *list) } ``` -In the next step after a device is created by the `device_create` function, we add it to the miscellaneous devices list with: 在函数`device_create` 创建了设备后我们就用下面的语句将设备添加到设备链表： ``` list_add(&misc->list, &misc_list); ``` -Kernel `list.h` provides this API for the addition of a new entry to the list. Let's look at its implementation: 内核文件`list.h` 提供了项链表添加新项的API 接口。我们来看看它的实现： @@ -149,14 +135,12 @@ static inline void list_add(struct list_head *new, struct list_head *head) } ``` -It just calls internal function `__list_add` with the 3 given parameters: 实际上就是使用3个指定的参数来调用了内部函数`__list_add`： * new - 新项。 * head - 新项将会被添加到`head`之前. * head->next - `head` 之后的项。 -Implementation of the `__list_add` is pretty simple: `__list_add`的实现非常简单： ```C @@ -171,10 +155,8 @@ static inline void __list_add(struct list_head *new, } ``` -Here we add a new item between `prev` and `next`. So `misc` list which we defined at the start with the `LIST_HEAD_INIT` macro will contain previous and next pointers to the `miscdevice->list`. 我们会在`prev`和`next` 之间添加一个新项。所以我们用宏`LIST_HEAD_INIT`定义的`misc` 链表会包含指向`miscdevice->list` 的向前指针和向后指针。 -There is still one question: how to get list's entry. There is a special macro: 这里有一个问题：如何得到列表的内容呢？这里有一个特殊的宏： ```C @@ -182,21 +164,18 @@ There is still one question: how to get list's entry. There is a special macro: container_of(ptr, type, member) ``` -which gets three parameters: 使用了三个参数： * ptr - 指向链表头的指针； * type - 结构体类型; * member - 在结构体内类型为`list_head` 的变量的名字； -For example: 比如说： ```C const struct miscdevice *p = list_entry(v, struct miscdevice, list) ``` -After this we can access to any `miscdevice` field with `p->minor` or `p->name` and etc... Let's look on the `list_entry` implementation: 然后我们就可以使用`p->minor` 或者 `p->name`来访问`miscdevice`。让我们来看看`list_entry` 的实现： ```C @@ -204,7 +183,6 @@ After this we can access to any `miscdevice` field with `p->minor` or `p->name` container_of(ptr, type, member) ``` -As we can see it just calls `container_of` macro with the same arguments. At first sight, the `container_of` looks strange: 如我们所见，它仅仅使用相同的参数调用了宏`container_of`。初看这个宏挺奇怪的： ```C @@ -213,10 +191,8 @@ As we can see it just calls `container_of` macro with the same arguments. At fir (type *)( (char *)__mptr - offsetof(type,member) );}) ``` -First of all you can note that it consists of two expressions in curly brackets. The compiler will evaluate the whole block in the curly braces and use the value of the last expression. 首先你可以注意到花括号内包含两个表达式。编译器会执行花括号内的全部语句，然后返回最后的表达式的值。 -For example: 举个例子来说： ``` @@ -229,10 +205,8 @@ int main() { } ``` -will print `2`. 最终会打印`2` -The next point is `typeof`, it's simple. As you can understand from its name, it just returns the type of the given variable. When I first saw the implementation of the `container_of` macro, the strangest thing I found was the zero in the `((type *)0)` expression. Actually this pointer magic calculates the offset of the given field from the address of the structure, but as we have `0` here, it will be just a zero offset along with the field width. Let's look at a simple example: 下一点就是`typeof`,它也很简单。就如你从名字所理解的，它仅仅返回了给定变量的类型。当我第一次看到宏`container_of`的实现时，让我觉得最奇怪的就是`container_of`中的0.实际上这个指针巧妙的计算了从结构体特定变量的偏移，这里的`0`刚好就是位宽里的零偏移。让我们看一个简单的例子： ```C @@ -250,20 +224,16 @@ int main() { } ``` -will print `0x5`. 结果显示`0x5`。 -The next `offsetof` macro calculates offset from the beginning of the structure to the given structure's field. Its implementation is very similar to the previous code: 下一个宏`offsetof` 会计算从结构体的某个变量的相对于结构体起始地址的偏移。它的实现和上面类似： ```C #define offsetof(TYPE, MEMBER) ((size_t) &((TYPE *)0)->MEMBER) ``` -Let's summarize all about `container_of` macro. The `container_of` macro returns the address of the structure by the given address of the structure's field with `list_head` type, the name of the structure field with `list_head` type and type of the container structure. At the first line this macro declares the `__mptr` pointer which points to the field of the structure that `ptr` points to and assigns `ptr` to it. Now `ptr` and `__mptr` point to the same address. Technically we don't need this line but it's useful for type checking. The first line ensures that the given structure (`type` parameter) has a member called `member`. In the second line it calculates offset of the field from the structure with the `offsetof` macro and subtracts it from the structure address. That's all. 现在我们来总结一下宏`container_of`。只需要知道结构体里面类型为`list_head` 的变量的名字和结构体容器的类型，它可以通过结构体的变量`list_head`获得结构体的起始地址。在宏定义的第一行，声明了一个指向结构体成员变量`ptr`的指针`__mptr`，并且把`ptr` 的地址赋给它。现在`ptr` 和`__mptr` 指向了同一个地址。从技术上讲我们并不需要这一行，但是它可以方便的进行类型检查。第一行保证了特定的结构体（参数`type`）包含成员变量`member`。第二行代码会用宏`offsetof`计算成员变量相对于结构体起始地址的偏移，然后从结构体的地址减去这个偏移，最后就得到了结构体。 -Of course `list_add` and `list_entry` is not the only functions which `` provides. Implementation of the doubly linked list provides the following API: 当然了`list_add` 和 `list_entry`不是``提供的唯一功能。双向链表的实现还提供了如下API： * list_add @@ -278,8 +248,7 @@ Of course `list_add` and `list_entry` is not the only functions which ` Date: Wed, 9 Dec 2015 09:59:42 +0800 Subject: [PATCH 5/6] =?UTF-8?q?=E7=BF=BB=E8=AF=91=E5=AE=8C=E6=88=90?= =?UTF-8?q?=EF=BC=8C=E7=A7=BB=E5=8A=A8=E5=88=B0translated?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../tech/20151122 Doubly linked list in the Linux Kernel.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename {sources => translated}/tech/20151122 Doubly linked list in the Linux Kernel.md (100%) diff --git a/sources/tech/20151122 Doubly linked list in the Linux Kernel.md b/translated/tech/20151122 Doubly linked list in the Linux Kernel.md similarity index 100% rename from sources/tech/20151122 Doubly linked list in the Linux Kernel.md rename to translated/tech/20151122 Doubly linked list in the Linux Kernel.md From 6fe786d73564184151d07193f67fb85e4f345cf2 Mon Sep 17 00:00:00 2001 From: Ezio Date: Wed, 9 Dec 2015 10:10:55 +0800 Subject: [PATCH 6/6] Delete 20151203 Getting started with Docker by Dockerizing this Blog.md --- ...ed with Docker by Dockerizing this Blog.md | 375 ------------------ 1 file changed, 375 deletions(-) delete mode 100644 sources/tech/20151203 Getting started with Docker by Dockerizing this Blog.md diff --git a/sources/tech/20151203 Getting started with Docker by Dockerizing this Blog.md b/sources/tech/20151203 Getting started with Docker by Dockerizing this Blog.md deleted file mode 100644 index 1f69a4adba..0000000000 --- a/sources/tech/20151203 Getting started with Docker by Dockerizing this Blog.md +++ /dev/null @@ -1,375 +0,0 @@ -Getting started with Docker by Dockerizing this Blog -====================== ->This article covers the basic concepts of Docker and how to Dockerize an application by creating a custom Dockerfile ->Written by Benjamin Cane on 2015-12-01 10:00:00 - -Docker is an interesting technology that over the past 2 years has gone from an idea, to being used by organizations all over the world to deploy applications. In today's article I am going to cover how to get started with Docker by "Dockerizing" an existing application. The application in question is actually this very blog! - -What is Docker -============ -============ - -Before we dive into learning the basics of Docker let's first understand what Docker is and why it is so popular. Docker, is an operating system container management tool that allows you to easily manage and deploy applications by making it easy to package them within operating system containers. - -### Containers vs. Virtual Machines - -Containers may not be as familiar as virtual machines but they are another method to provide Operating System Virtualization. However, they differ quite a bit from standard virtual machines. - -Standard virtual machines generally include a full Operating System, OS Packages and eventually an Application or two. This is made possible by a Hypervisor which provides hardware virtualization to the virtual machine. This allows for a single server to run many standalone operating systems as virtual guests. - -Containers are similar to virtual machines in that they allow a single server to run multiple operating environments, these environments however, are not full operating systems. Containers generally only include the necessary OS Packages and Applications. They do not generally contain a full operating system or hardware virtualization. This also means that containers have a smaller overhead than traditional virtual machines. - -Containers and Virtual Machines are often seen as conflicting technology, however, this is often a misunderstanding. Virtual Machines are a way to take a physical server and provide a fully functional operating environment that shares those physical resources with other virtual machines. A Container is generally used to isolate a running process within a single host to ensure that the isolated processes cannot interact with other processes within that same system. In fact containers are closer to BSD Jails and chroot'ed processes than full virtual machines. - -### What Docker provides on top of containers - -Docker itself is not a container runtime environment; in fact Docker is actually container technology agnostic with efforts planned for Docker to support Solaris Zones and BSD Jails. What Docker provides is a method of managing, packaging, and deploying containers. While these types of functions may exist to some degree for virtual machines they traditionally have not existed for most container solutions and the ones that existed, were not as easy to use or fully featured as Docker. - -Now that we know what Docker is, let's start learning how Docker works by first installing Docker and deploying a public pre-built container. - -## Starting with Installation -As Docker is not installed by default step 1 will be to install the Docker package; since our example system is running Ubuntu 14.0.4 we will do this using the Apt package manager. - -# apt-get install docker.io -Reading package lists... Done -Building dependency tree -Reading state information... Done -The following extra packages will be installed: - aufs-tools cgroup-lite git git-man liberror-perl -Suggested packages: - btrfs-tools debootstrap lxc rinse git-daemon-run git-daemon-sysvinit git-doc - git-el git-email git-gui gitk gitweb git-arch git-bzr git-cvs git-mediawiki - git-svn -The following NEW packages will be installed: - aufs-tools cgroup-lite docker.io git git-man liberror-perl -0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded. -Need to get 7,553 kB of archives. -After this operation, 46.6 MB of additional disk space will be used. -Do you want to continue? [Y/n] y -To check if any containers are running we can execute the docker command using the ps option. - -# docker ps -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -The ps function of the docker command works similar to the Linux ps command. It will show available Docker containers and their current status. Since we have not started any Docker containers yet, the command shows no running containers. - -## Deploying a pre-built nginx Docker container -One of my favorite features of Docker is the ability to deploy a pre-built container in the same way you would deploy a package with yum or apt-get. To explain this better let's deploy a pre-built container running the nginx web server. We can do this by executing the docker command again, however, this time with the run option. - -# docker run -d nginx -Unable to find image 'nginx' locally -Pulling repository nginx -5c82215b03d1: Download complete -e2a4fb18da48: Download complete -58016a5acc80: Download complete -657abfa43d82: Download complete -dcb2fe003d16: Download complete -c79a417d7c6f: Download complete -abb90243122c: Download complete -d6137c9e2964: Download complete -85e566ddc7ef: Download complete -69f100eb42b5: Download complete -cd720b803060: Download complete -7cc81e9a118a: Download complete -The run function of the docker command tells Docker to find a specified Docker image and start a container running that image. By default, Docker containers run in the foreground, meaning when you execute docker run your shell will be bound to the container's console and the process running within the container. In order to launch this Docker container in the background I included the -d (detach) flag. - -By executing docker ps again we can see the nginx container running. - -# docker ps -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -f6d31ab01fc9 nginx:latest nginx -g 'daemon off 4 seconds ago Up 3 seconds 443/tcp, 80/tcp desperate_lalande -In the above output we can see the running container desperate_lalande and that this container has been built from the nginx:latest image. - -## Docker Images -Images are one of Docker's key features and is similar to a virtual machine image. Like virtual machine images, a Docker image is a container that has been saved and packaged. Docker however, doesn't just stop with the ability to create images. Docker also includes the ability to distribute those images via Docker repositories which are a similar concept to package repositories. This is what gives Docker the ability to deploy an image like you would deploy a package with yum. To get a better understanding of how this works let's look back at the output of the docker run execution. - -# docker run -d nginx -Unable to find image 'nginx' locally -The first message we see is that docker could not find an image named nginx locally. The reason we see this message is that when we executed docker run we told Docker to startup a container, a container based on an image named nginx. Since Docker is starting a container based on a specified image it needs to first find that image. Before checking any remote repository Docker first checks locally to see if there is a local image with the specified name. - -Since this system is brand new there is no Docker image with the name nginx, which means Docker will need to download it from a Docker repository. - -Pulling repository nginx -5c82215b03d1: Download complete -e2a4fb18da48: Download complete -58016a5acc80: Download complete -657abfa43d82: Download complete -dcb2fe003d16: Download complete -c79a417d7c6f: Download complete -abb90243122c: Download complete -d6137c9e2964: Download complete -85e566ddc7ef: Download complete -69f100eb42b5: Download complete -cd720b803060: Download complete -7cc81e9a118a: Download complete -This is exactly what the second part of the output is showing us. By default, Docker uses the Docker Hub repository, which is a repository service that Docker (the company) runs. - -Like GitHub, Docker Hub is free for public repositories but requires a subscription for private repositories. It is possible however, to deploy your own Docker repository, in fact it is as easy as docker run registry. For this article we will not be deploying a custom registry service. - -## Stopping and Removing the Container -Before moving on to building a custom Docker container let's first clean up our Docker environment. We will do this by stopping the container from earlier and removing it. - -To start a container we executed docker with the run option, in order to stop this same container we simply need to execute the docker with the kill option specifying the container name. - -# docker kill desperate_lalande -desperate_lalande -If we execute docker ps again we will see that the container is no longer running. - -# docker ps -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -However, at this point we have only stopped the container; while it may no longer be running it still exists. By default, docker ps will only show running containers, if we add the -a (all) flag it will show all containers running or not. - -# docker ps -a -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -f6d31ab01fc9 5c82215b03d1 nginx -g 'daemon off 4 weeks ago Exited (-1) About a minute ago desperate_lalande -In order to fully remove the container we can use the docker command with the rm option. - -# docker rm desperate_lalande -desperate_lalande -While this container has been removed; we still have a nginx image available. If we were to re-run docker run -d nginx again the container would be started without having to fetch the nginx image again. This is because Docker already has a saved copy on our local system. - -To see a full list of local images we can simply run the docker command with the images option. - -# docker images -REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE -nginx latest 9fab4090484a 5 days ago 132.8 MB -## Building our own custom image -At this point we have used a few basic Docker commands to start, stop and remove a common pre-built image. In order to "Dockerize" this blog however, we are going to have to build our own Docker image and that means creating a Dockerfile. - -With most virtual machine environments if you wish to create an image of a machine you need to first create a new virtual machine, install the OS, install the application and then finally convert it to a template or image. With Docker however, these steps are automated via a Dockerfile. A Dockerfile is a way of providing build instructions to Docker for the creation of a custom image. In this section we are going to build a custom Dockerfile that can be used to deploy this blog. - -### Understanding the Application -Before we can jump into creating a Dockerfile we first need to understand what is required to deploy this blog. - -The blog itself is actually static HTML pages generated by a custom static site generator that I wrote named; hamerkop. The generator is very simple and more about getting the job done for this blog specifically. All the code and source files for this blog are available via a public GitHub repository. In order to deploy this blog we simply need to grab the contents of the GitHub repository, install Python along with some Python modules and execute the hamerkop application. To serve the generated content we will use nginx; which means we will also need nginx to be installed. - -So far this should be a pretty simple Dockerfile, but it will show us quite a bit of the Dockerfile Syntax. To get started we can clone the GitHub repository and creating a Dockerfile with our favorite editor; vi in my case. - -# git clone https://github.com/madflojo/blog.git -Cloning into 'blog'... -remote: Counting objects: 622, done. -remote: Total 622 (delta 0), reused 0 (delta 0), pack-reused 622 -Receiving objects: 100% (622/622), 14.80 MiB | 1.06 MiB/s, done. -Resolving deltas: 100% (242/242), done. -Checking connectivity... done. -# cd blog/ -# vi Dockerfile -### FROM - Inheriting a Docker image -The first instruction of a Dockerfile is the FROM instruction. This is used to specify an existing Docker image to use as our base image. This basically provides us with a way to inherit another Docker image. In this case we will be starting with the same nginx image we were using before, if we wanted to start with a blank slate we could use the Ubuntu Docker image by specifying ubuntu:latest. - -## Dockerfile that generates an instance of http://bencane.com - -FROM nginx:latest -MAINTAINER Benjamin Cane -In addition to the FROM instruction, I also included a MAINTAINER instruction which is used to show the Author of the Dockerfile. - -As Docker supports using # as a comment marker, I will be using this syntax quite a bit to explain the sections of this Dockerfile. - -### Running a test build -Since we inherited the nginx Docker image our current Dockerfile also inherited all the instructions within the Dockerfile used to build that nginx image. What this means is even at this point we are able to build a Docker image from this Dockerfile and run a container from that image. The resulting image will essentially be the same as the nginx image but we will run through a build of this Dockerfile now and a few more times as we go to help explain the Docker build process. - -In order to start the build from a Dockerfile we can simply execute the docker command with the build option. - -# docker build -t blog /root/blog -Sending build context to Docker daemon 23.6 MB -Sending build context to Docker daemon -Step 0 : FROM nginx:latest - ---> 9fab4090484a -Step 1 : MAINTAINER Benjamin Cane - ---> Running in c97f36450343 - ---> 60a44f78d194 -Removing intermediate container c97f36450343 -Successfully built 60a44f78d194 -In the above example I used the -t (tag) flag to "tag" the image as "blog". This essentially allows us to name the image, without specifying a tag the image would only be callable via an Image ID that Docker assigns. In this case the Image ID is 60a44f78d194 which we can see from the docker command's build success message. - -In addition to the -t flag, I also specified the directory /root/blog. This directory is the "build directory", which is the directory that contains the Dockerfile and any other files necessary to build this container. - -Now that we have run through a successful build, let's start customizing this image. - -### Using RUN to execute apt-get -The static site generator used to generate the HTML pages is written in Python and because of this the first custom task we should perform within this Dockerfile is to install Python. To install the Python package we will use the Apt package manager. This means we will need to specify within the Dockerfile that apt-get update and apt-get install python-dev are executed; we can do this with the RUN instruction. - -## Dockerfile that generates an instance of http://bencane.com - -FROM nginx:latest -MAINTAINER Benjamin Cane - -## Install python and pip -RUN apt-get update -RUN apt-get install -y python-dev python-pip -In the above we are simply using the RUN instruction to tell Docker that when it builds this image it will need to execute the specified apt-get commands. The interesting part of this is that these commands are only executed within the context of this container. What this means is even though python-dev and python-pip are being installed within the container, they are not being installed for the host itself. Or to put it simplier, within the container the pip command will execute, outside the container, the pip command does not exist. - -It is also important to note that the Docker build process does not accept user input during the build. This means that any commands being executed by the RUN instruction must complete without user input. This adds a bit of complexity to the build process as many applications require user input during installation. For our example, none of the commands executed by RUN require user input. - -### Installing Python modules -With Python installed we now need to install some Python modules. To do this outside of Docker, we would generally use the pip command and reference a file within the blog's Git repository named requirements.txt. In an earlier step we used the git command to "clone" the blog's GitHub repository to the /root/blog directory; this directory also happens to be the directory that we have created the Dockerfile. This is important as it means the contents of the Git repository are accessible to Docker during the build process. - -When executing a build, Docker will set the context of the build to the specified "build directory". This means that any files within that directory and below can be used during the build process, files outside of that directory (outside of the build context), are inaccessible. - -In order to install the required Python modules we will need to copy the requirements.txt file from the build directory into the container. We can do this using the COPY instruction within the Dockerfile. - -## Dockerfile that generates an instance of http://bencane.com - -FROM nginx:latest -MAINTAINER Benjamin Cane - -## Install python and pip -RUN apt-get update -RUN apt-get install -y python-dev python-pip - -## Create a directory for required files -RUN mkdir -p /build/ - -## Add requirements file and run pip -COPY requirements.txt /build/ -RUN pip install -r /build/requirements.txt -Within the Dockerfile we added 3 instructions. The first instruction uses RUN to create a /build/ directory within the container. This directory will be used to copy any application files needed to generate the static HTML pages. The second instruction is the COPY instruction which copies the requirements.txt file from the "build directory" (/root/blog) into the /build directory within the container. The third is using the RUN instruction to execute the pip command; installing all the modules specified within the requirements.txt file. - -COPY is an important instruction to understand when building custom images. Without specifically copying the file within the Dockerfile this Docker image would not contain the requirements.txt file. With Docker containers everything is isolated, unless specifically executed within a Dockerfile a container is not likely to include required dependencies. - -### Re-running a build -Now that we have a few customization tasks for Docker to perform let's try another build of the blog image again. - -# docker build -t blog /root/blog -Sending build context to Docker daemon 19.52 MB -Sending build context to Docker daemon -Step 0 : FROM nginx:latest - ---> 9fab4090484a -Step 1 : MAINTAINER Benjamin Cane - ---> Using cache - ---> 8e0f1899d1eb -Step 2 : RUN apt-get update - ---> Using cache - ---> 78b36ef1a1a2 -Step 3 : RUN apt-get install -y python-dev python-pip - ---> Using cache - ---> ef4f9382658a -Step 4 : RUN mkdir -p /build/ - ---> Running in bde05cf1e8fe - ---> f4b66e09fa61 -Removing intermediate container bde05cf1e8fe -Step 5 : COPY requirements.txt /build/ - ---> cef11c3fb97c -Removing intermediate container 9aa8ff43f4b0 -Step 6 : RUN pip install -r /build/requirements.txt - ---> Running in c50b15ddd8b1 -Downloading/unpacking jinja2 (from -r /build/requirements.txt (line 1)) -Downloading/unpacking PyYaml (from -r /build/requirements.txt (line 2)) - -Successfully installed jinja2 PyYaml mistune markdown MarkupSafe -Cleaning up... - ---> abab55c20962 -Removing intermediate container c50b15ddd8b1 -Successfully built abab55c20962 -From the above build output we can see the build was successful, but we can also see another interesting message; ---> Using cache. What this message is telling us is that Docker was able to use its build cache during the build of this image. - -#### Docker build cache - -When Docker is building an image, it doesn't just build a single image; it actually builds multiple images throughout the build processes. In fact we can see from the above output that after each "Step" Docker is creating a new image. - - Step 5 : COPY requirements.txt /build/ - ---> cef11c3fb97c -The last line from the above snippet is actually Docker informing us of the creating of a new image, it does this by printing the Image ID; cef11c3fb97c. The useful thing about this approach is that Docker is able to use these images as cache during subsequent builds of the blog image. This is useful because it allows Docker to speed up the build process for new builds of the same container. If we look at the example above we can actually see that rather than installing the python-dev and python-pip packages again, Docker was able to use a cached image. However, since Docker was unable to find a build that executed the mkdir command, each subsequent step was executed. - -The Docker build cache is a bit of a gift and a curse; the reason for this is that the decision to use cache or to rerun the instruction is made within a very narrow scope. For example, if there was a change to the requirements.txt file Docker would detect this change during the build and start fresh from that point forward. It does this because it can view the contents of the requirements.txt file. The execution of the apt-get commands however, are another story. If the Apt repository that provides the Python packages were to contain a newer version of the python-pip package; Docker would not be able to detect the change and would simply use the build cache. This means that an older package may be installed. While this may not be a major issue for the python-pip package it could be a problem if the installation was caching a package with a known vulnerability. - -For this reason it is useful to periodically rebuild the image without using Docker's cache. To do this you can simply specify --no-cache=True when executing a Docker build. - -### Deploying the rest of the blog -With the Python packages and modules installed this leaves us at the point of copying the required application files and running the hamerkop application. To do this we will simply use more COPY and RUN instructions. - -## Dockerfile that generates an instance of http://bencane.com - -FROM nginx:latest -MAINTAINER Benjamin Cane - -## Install python and pip -RUN apt-get update -RUN apt-get install -y python-dev python-pip - -## Create a directory for required files -RUN mkdir -p /build/ - -## Add requirements file and run pip -COPY requirements.txt /build/ -RUN pip install -r /build/requirements.txt - -## Add blog code nd required files -COPY static /build/static -COPY templates /build/templates -COPY hamerkop /build/ -COPY config.yml /build/ -COPY articles /build/articles - -## Run Generator -RUN /build/hamerkop -c /build/config.yml -Now that we have the rest of the build instructions, let's run through another build and verify that the image builds successfully. - -# docker build -t blog /root/blog/ -Sending build context to Docker daemon 19.52 MB -Sending build context to Docker daemon -Step 0 : FROM nginx:latest - ---> 9fab4090484a -Step 1 : MAINTAINER Benjamin Cane - ---> Using cache - ---> 8e0f1899d1eb -Step 2 : RUN apt-get update - ---> Using cache - ---> 78b36ef1a1a2 -Step 3 : RUN apt-get install -y python-dev python-pip - ---> Using cache - ---> ef4f9382658a -Step 4 : RUN mkdir -p /build/ - ---> Using cache - ---> f4b66e09fa61 -Step 5 : COPY requirements.txt /build/ - ---> Using cache - ---> cef11c3fb97c -Step 6 : RUN pip install -r /build/requirements.txt - ---> Using cache - ---> abab55c20962 -Step 7 : COPY static /build/static - ---> 15cb91531038 -Removing intermediate container d478b42b7906 -Step 8 : COPY templates /build/templates - ---> ecded5d1a52e -Removing intermediate container ac2390607e9f -Step 9 : COPY hamerkop /build/ - ---> 59efd1ca1771 -Removing intermediate container b5fbf7e817b7 -Step 10 : COPY config.yml /build/ - ---> bfa3db6c05b7 -Removing intermediate container 1aebef300933 -Step 11 : COPY articles /build/articles - ---> 6b61cc9dde27 -Removing intermediate container be78d0eb1213 -Step 12 : RUN /build/hamerkop -c /build/config.yml - ---> Running in fbc0b5e574c5 -Successfully created file /usr/share/nginx/html//2011/06/25/checking-the-number-of-lwp-threads-in-linux -Successfully created file /usr/share/nginx/html//2011/06/checking-the-number-of-lwp-threads-in-linux - -Successfully created file /usr/share/nginx/html//archive.html -Successfully created file /usr/share/nginx/html//sitemap.xml - ---> 3b25263113e1 -Removing intermediate container fbc0b5e574c5 -Successfully built 3b25263113e1 -### Running a custom container -With a successful build we can now start our custom container by running the docker command with the run option, similar to how we started the nginx container earlier. - -# docker run -d -p 80:80 --name=blog blog -5f6c7a2217dcdc0da8af05225c4d1294e3e6bb28a41ea898a1c63fb821989ba1 -Once again the -d (detach) flag was used to tell Docker to run the container in the background. However, there are also two new flags. The first new flag is --name, which is used to give the container a user specified name. In the earlier example we did not specify a name and because of that Docker randomly generated one. The second new flag is -p, this flag allows users to map a port from the host machine to a port within the container. - -The base nginx image we used exposes port 80 for the HTTP service. By default, ports bound within a Docker container are not bound on the host system as a whole. In order for external systems to access ports exposed within a container the ports must be mapped from a host port to a container port using the -p flag. The command above maps port 80 from the host, to port 80 within the container. If we wished to map port 8080 from the host, to port 80 within the container we could do so by specifying the ports in the following syntax -p 8080:80. - -From the above command it appears that our container was started successfully, we can verify this by executing docker ps. - -# docker ps -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -d264c7ef92bd blog:latest nginx -g 'daemon off 3 seconds ago Up 3 seconds 443/tcp, 0.0.0.0:80->80/tcp blog -## Wrapping up - -At this point we now have a running custom Docker container. While we touched on a few Dockerfile instructions within this article we have yet to discuss all the instructions. For a full list of Dockerfile instructions you can checkout Docker's reference page, which explains the instructions very well. - -Another good resource is their Dockerfile Best Practices page which contains quite a few best practices for building custom Dockerfiles. Some of these tips are very useful such as strategically ordering the commands within the Dockerfile. In the above examples our Dockerfile has the COPY instruction for the articles directory as the last COPY instruction. The reason for this is that the articles directory will change quite often. It's best to put instructions that will change oftenat the lowest point possible within the Dockerfile to optimize steps that can be cached. - -In this article we covered how to start a pre-built container and how to build, then deploy a custom container. While there is quite a bit to learn about Docker this article should give you a good idea on how to get started. Of course, as always if you think there is anything that should be added drop it in the comments below.