mirror of
https://github.com/LCTT/TranslateProject.git
synced 2026-08-23 04:03:29 +08:00
Merge branch 'master' of https://github.com/LCTT/TranslateProject
Meger from LCTT.
This commit is contained in:
runningwater
75
Dict.md
75
Dict.md
@@ -14,55 +14,55 @@
|
||||
### 2.
|
||||
#### B ####
|
||||
### 1. Backbone：骨干
|
||||
>是一个网络的一部分，其作为所有网络运输的一个基本通道，其需要非常高的带宽。一个骨干网络的服务提供者连接许多企业子网和较小服务提供者的网络。一个企业骨干网络连接许多局域网和数据中心。
|
||||
> 是一个网络的一部分，其作为所有网络运输的一个基本通道，其需要非常高的带宽。一个骨干网络的服务提供者连接许多企业子网和较小服务提供者的网络。一个企业骨干网络连接许多局域网和数据中心。
|
||||
|
||||
### 2. B channel(Bearer channel)：承载信道
|
||||
>承载信道（Bearer Channel），也叫做B channel，是一个全双工DS0时间槽（64-kbps），其携带模拟语音或数字资料通过综合服务数字网（ISDN）。
|
||||
> 承载信道（Bearer Channel），也叫做B channel，是一个全双工DS0时间槽（64-kbps），其携带模拟语音或数字资料通过综合服务数字网（ISDN）。
|
||||
|
||||
### 3. Backchannel：反向通道
|
||||
>是指当其他实时在线会话在进行中时，习惯使用网络化的计算机来维持一个实时的在线会话。
|
||||
> 是指当其他实时在线会话在进行中时，习惯使用网络化的计算机来维持一个实时的在线会话。
|
||||
|
||||
### 4. Back End：后台
|
||||
>在一个计算机系统中，是指为一个前台作业提供服务的一个节点或软件程序。前台直接影响用户，后台可能与其他系统相连接，如数据库和其它系统。
|
||||
> 在一个计算机系统中，是指为一个前台作业提供服务的一个节点或软件程序。前台直接影响用户，后台可能与其他系统相连接，如数据库和其它系统。
|
||||
|
||||
### 5. Back-haul：回程线路
|
||||
>是一个通信信道，它使携带信息流到远于最终目的地的地方，然后将它送回。这样做是因为传输到更远的远程区域的代价要远比直接发送的代价低地多。
|
||||
> 是一个通信信道，它使携带信息流到远于最终目的地的地方，然后将它送回。这样做是因为传输到更远的远程区域的代价要远比直接发送的代价低地多。
|
||||
|
||||
### 6. Backoff：退避
|
||||
>是指当一个主机已经在有MAC 协议的网络中经历了一个冲突之后试图去重发之前的等待时期。这个退避时间通常是任意的来最小化相同节点再次冲突的可能性。在每次冲突后增加退避时期也能帮助预防重复碰撞，特别当这个网络负担很重时。
|
||||
> 是指当一个主机已经在有MAC 协议的网络中经历了一个冲突之后试图去重发之前的等待时期。这个退避时间通常是任意的来最小化相同节点再次冲突的可能性。在每次冲突后增加退避时期也能帮助预防重复碰撞，特别当这个网络负担很重时。
|
||||
|
||||
### 7. Backplane：附加卡
|
||||
>在许多网络中是一个物理接口模块，例如，连接在一个界面处理器或卡和在一个总线机箱内数据总线和功率分配总线之间的一个路由器或转换器。
|
||||
> 在许多网络中是一个物理接口模块，例如，连接在一个界面处理器或卡和在一个总线机箱内数据总线和功率分配总线之间的一个路由器或转换器。
|
||||
|
||||
### 8. Back Pressure：背压
|
||||
>在计算机系统中，是指网络拥塞信息逆流通过一个Internet网络。
|
||||
> 在计算机系统中，是指网络拥塞信息逆流通过一个Internet网络。
|
||||
|
||||
### 9. Balun(balanced-unbalanced)：不平衡变压器
|
||||
>意味着平衡-非平衡。不平衡变压器是一个设计用来转换平衡和不平衡之间的电信号的设备。
|
||||
> 意味着平衡-非平衡。不平衡变压器是一个设计用来转换平衡和不平衡之间的电信号的设备。
|
||||
|
||||
### 10. Baseband：基带
|
||||
>是一种类型的网络技术,在那里仅仅一种载波频率被使用。在一个基带网中，信息在传送介质中以数字的形式被携带在一个单一的多元信号通道中。
|
||||
> 是一种类型的网络技术,在那里仅仅一种载波频率被使用。在一个基带网中，信息在传送介质中以数字的形式被携带在一个单一的多元信号通道中。
|
||||
|
||||
### 11. Bastion Host：防御主机
|
||||
>是在内部网络和外部网络之间的一个网关，它被设计来防御针对内部网络的攻击。这个系统在非武装区(DMZ)的公共一边，不被防火墙或过滤路由器保护，它对攻击是完全暴露的。
|
||||
> 是在内部网络和外部网络之间的一个网关，它被设计来防御针对内部网络的攻击。这个系统在非武装区(DMZ)的公共一边，不被防火墙或过滤路由器保护，它对攻击是完全暴露的。
|
||||
|
||||
### 12: Bc(Committed Burst)：约定资讯讯务
|
||||
>是一个用在帧中继系统的术语，是一个帧中继交互网约定接受和传输和通过一个帧中继网络数据链路控制（DLC）和一个特殊的时帧的最大数据量（用比特表示）。
|
||||
> 是一个用在帧中继系统的术语，是一个帧中继交互网约定接受和传输和通过一个帧中继网络数据链路控制（DLC）和一个特殊的时帧的最大数据量（用比特表示）。
|
||||
|
||||
### 13. BCP(Best Current Practices)：最优现行方法
|
||||
>是副系列的IETF RFCs，其被用于描述在Internet上的最优配置技术。
|
||||
> 是副系列的IETF RFCs，其被用于描述在Internet上的最优配置技术。
|
||||
|
||||
### 14. BCU(Balanced Configuration Unit)：平衡配置单元
|
||||
>是一个综合的IBM解决方法，它由软件和硬件组成。BCUs是综合的和测试作为数据仓库系统的预配置功能块。
|
||||
> 是一个综合的IBM解决方法，它由软件和硬件组成。BCUs是综合的和测试作为数据仓库系统的预配置功能块。
|
||||
|
||||
### 15. BECN(Backward Explicit Congestion Notification)：显式拥塞通知
|
||||
>是在帧中继报头的一个1比特域，其发信号到任何接收帧的事物（转换器和数据终端设备），拥塞就发生在帧的反面（后面）。帧中继转换器和数据终端设备可能遵照显式拥塞通知位来减慢那个方向的数据传输率。
|
||||
> 是在帧中继报头的一个1比特域，其发信号到任何接收帧的事物（转换器和数据终端设备），拥塞就发生在帧的反面（后面）。帧中继转换器和数据终端设备可能遵照显式拥塞通知位来减慢那个方向的数据传输率。
|
||||
|
||||
### 16. BER(Bit Error Rate)：误码率
|
||||
>是接收到的位包含错误的比率。BER通常被表示成十足的负面力量。
|
||||
> 是接收到的位包含错误的比率。BER通常被表示成十足的负面力量。
|
||||
|
||||
### 17. BIP(Bit Interleaved Parity)：位交叉奇偶校验
|
||||
>一个用在ATM中的术语，是一个通常用来检测链接错误的一种方法。一个检测位或字被嵌入到以前发生阻塞或帧的链接中。位错误在有效载荷中能够作为维护信息被删除和报告。
|
||||
> 一个用在ATM中的术语，是一个通常用来检测链接错误的一种方法。一个检测位或字被嵌入到以前发生阻塞或帧的链接中。位错误在有效载荷中能够作为维护信息被删除和报告。
|
||||
|
||||
#### C ####
|
||||
|
||||
@@ -90,69 +90,74 @@
|
||||
> 指 Linux 内核的 live patch 支持。
|
||||
|
||||
### 2. LTS(Long Term Support)：长期支持
|
||||
>该缩写词多见于操作系统发行版或者软件发行版名称中，表明该版本属于长期支持版。
|
||||
> 该缩写词多见于操作系统发行版或者软件发行版名称中，表明该版本属于长期支持版。
|
||||
|
||||
#### M ####
|
||||
|
||||
#### N ####
|
||||
|
||||
#### O ####
|
||||
### 1. Orchestration：编排
|
||||
> 描述复杂计算机系统、中间件（middleware）和业务的自动化的安排、协调和管理（来自维基百科）。
|
||||
|
||||
#### P ####
|
||||
### 1.P-code(Pseudo-code)：伪代码语言
|
||||
>一种解释型语言，执行方式介于编译型语言和解释型语言之间。和解释型语言一样，伪代码编程语言无需编译，在执行时自动转换成二进制形式。然而，和编译型语言不同的是，这种可执行的二进制文件是以伪代码的形式而不是机器语言的形式存储的。伪代码语言的例子有 Java、Python 和 REXX/Object REXX。
|
||||
### 1. P-code(Pseudo-code)：伪代码语言
|
||||
> 一种解释型语言，执行方式介于编译型语言和解释型语言之间。和解释型语言一样，伪代码编程语言无需编译，在执行时自动转换成二进制形式。然而，和编译型语言不同的是，这种可执行的二进制文件是以伪代码的形式而不是机器语言的形式存储的。伪代码语言的例子有 Java、Python 和 REXX/Object REXX。
|
||||
|
||||
### 2. PAM(Pluggable Authentication Modules)：可插拔认证模块
|
||||
>用于系统安全性的可替换的用户认证模块，它允许在不知道将使用何种认证方案的情况下进行编程。这允许将来用其它模块来替换某个模块，却无需重写软件。
|
||||
> 用于系统安全性的可替换的用户认证模块，它允许在不知道将使用何种认证方案的情况下进行编程。这允许将来用其它模块来替换某个模块，却无需重写软件。
|
||||
|
||||
### 3. Port/Ported/Porting：移植
|
||||
>一个过程，即获取为某个操作系统平台编写的程序，并对其进行修改使之能在另一 OS 上运行，并且具有类似的功能。
|
||||
> 一个过程，即获取为某个操作系统平台编写的程序，并对其进行修改使之能在另一 OS 上运行，并且具有类似的功能。
|
||||
|
||||
### 4. POSIX(Portable Operating System Interface for uniX)：UNIX 可移植操作系统接口
|
||||
>一组编程接口标准，它们规定如何编写应用程序源代码以便应用程序可在操作系统之间移植。POSIX 基于 UNIX，它是 The Open Group 的 X/Open 规范的基础。
|
||||
> 一组编程接口标准，它们规定如何编写应用程序源代码以便应用程序可在操作系统之间移植。POSIX 基于 UNIX，它是 The Open Group 的 X/Open 规范的基础。
|
||||
|
||||
#### Q ####
|
||||
|
||||
#### R ####
|
||||
### 1. RCS(Revision Control System)：修订控制系统
|
||||
>一组程序，它们控制组环境下文件的共享访问并跟踪文本文件的变化。常用于维护源代码模块的编码工作。
|
||||
> 一组程序，它们控制组环境下文件的共享访问并跟踪文本文件的变化。常用于维护源代码模块的编码工作。
|
||||
|
||||
### 2. RFS(Remote File Sharing)：远程文件共享
|
||||
>一个程序，它让用户访问其它计算机上的文件，就好象文件在用户的系统上一样。
|
||||
> 一个程序，它让用户访问其它计算机上的文件，就好象文件在用户的系统上一样。
|
||||
|
||||
#### S ####
|
||||
### 1. shebang [ʃɪ'bæŋ]：释伴
|
||||
>Shebang（也称为Hashbang）是一个由井号和叹号构成的字符序列（#!），出现在文本文件的第一行的前两个字符，后跟解释器路径，如：#!/bin/sh，这通常是Linux中shell脚本的标准起始行。
|
||||
>长期以来，shebang都没有正式的中文名称。Linux中国翻译组将其翻译为：释伴，即解释伴随行的简称，同时又是shebang的音译。
|
||||
> Shebang（也称为Hashbang）是一个由井号和叹号构成的字符序列（#!），出现在文本文件的第一行的前两个字符，后跟解释器路径，如：#!/bin/sh，这通常是Linux中shell脚本的标准起始行。
|
||||
> 长期以来，shebang都没有正式的中文名称。Linux中国翻译组将其翻译为：释伴，即解释伴随行的简称，同时又是shebang的音译。
|
||||
|
||||
### 2. Spool(Simultaneous Peripheral Operation On-Line)：假脱机
|
||||
>将数据发送给一个程序，该程序将该数据信息放入队列以备将来使用（例如，打印假脱机程序）
|
||||
> 将数据发送给一个程序，该程序将该数据信息放入队列以备将来使用（例如，打印假脱机程序）
|
||||
|
||||
### 2. Steganography：隐写术
|
||||
>将一段信息隐藏在另一段信息中的做法。一个示例是在数字化照片中放置不可见的数字水印。
|
||||
> 将一段信息隐藏在另一段信息中的做法。一个示例是在数字化照片中放置不可见的数字水印。
|
||||
|
||||
### 3. Swap：交换
|
||||
>暂时将数据（程序和／或数据文件）从随机存取存储器移到磁盘存储器（换出），或反方向移动（换入），以允许处理比物理内存所能容纳的更多的程序和数据。
|
||||
> 暂时将数据（程序和／或数据文件）从随机存取存储器移到磁盘存储器（换出），或反方向移动（换入），以允许处理比物理内存所能容纳的更多的程序和数据。
|
||||
|
||||
### 4. Scheduling：调度
|
||||
> 将任务分配至资源的过程，在计算机或生产处理中尤为重要（来自维基百科）。
|
||||
|
||||
#### T ####
|
||||
### 1. Time-sharing：分时
|
||||
>一种允许多个用户分享处理器的方法，它以时间为基础给每个用户分配一部分处理器资源，按照这些时间段轮流运行每个用户的进程。
|
||||
> 一种允许多个用户分享处理器的方法，它以时间为基础给每个用户分配一部分处理器资源，按照这些时间段轮流运行每个用户的进程。
|
||||
|
||||
### 2. TL;DR:长篇摘要
|
||||
>Too Long；Didn't Read的缩写词，即太长，未阅的意思。该词多见于互联网社区论坛中，用于指出该文太长，没有阅读，或者标示出一篇长文章的摘要。在论坛回复中，该缩写词也多作为灌水用。因此，Linux中国翻译组将其翻译为：长篇摘要。
|
||||
> Too Long；Didn't Read的缩写词，即太长，未阅的意思。该词多见于互联网社区论坛中，用于指出该文太长，没有阅读，或者标示出一篇长文章的摘要。在论坛回复中，该缩写词也多作为灌水用。因此，Linux中国翻译组将其翻译为：长篇摘要。
|
||||
|
||||
#### U ####
|
||||
|
||||
#### V ####
|
||||
### 1. VRML(Virtual Reality Modeling Language)：虚拟现实建模语言
|
||||
>一种主要基于 Web 的语言，用于 3D 效果（如构建遍历）。
|
||||
> 一种主要基于 Web 的语言，用于 3D 效果（如构建遍历）。
|
||||
|
||||
#### W ####
|
||||
### 1. Wrapper：封装器
|
||||
>用于启动另一个程序的程序。
|
||||
> 用于启动另一个程序的程序。
|
||||
|
||||
#### X ####
|
||||
|
||||
#### Y ####
|
||||
|
||||
#### Z ####
|
||||
#### Z ####
|
||||
|
||||
192
published/20161005 Docker Engine swarm mode - Intro tutorial.md
Normal file
192
published/20161005 Docker Engine swarm mode - Intro tutorial.md
Normal file
@@ -0,0 +1,192 @@
|
||||
Docker 引擎的 Swarm 模式：入门教程
|
||||
============================
|
||||
|
||||
Swarm，听起来像是一个朋克摇滚乐队。但它确实是个新的编排机制，抑或者是，一个 [Docker][1] 现有编排体制的改进。简单来讲，如果你在用一个旧版本的 Docker，你必须手动配置 Swarm 来创建 Docker 集群。从 [1.12 版][2]开始，Docker 引擎集成了一个原生的实现（LCTT 译注：见下文）来支持无缝的集群设置。也就是为什么会有这篇文章。
|
||||
|
||||
在这篇教程中，我将带你体验一下编排后的 Docker 将能做的事情。这篇文章并不是包含所有细节（如 BnB 一般）或是让你对其全知全能，但它能带你踏上你的集群之路。在我的带领下开始吧。
|
||||
|
||||
|
||||
|
||||
### 技术概要
|
||||
|
||||
如果把 Docker 详细而又好用的文档照搬到这里那将太丢人了，所以我将简要概括下这个技术的概要。我们已经有了 Docker，对吧。现在，你想要更多的服务器作为 Docker 主机，但同时你希望它们属于同一个逻辑上的实体。也就是说，你想建立一个集群。
|
||||
|
||||
我们先从一个主机组成的集群开始。当你在一个主机上初始化一个 Swarm 集群，这台主机将成为这个集群的管理者（manager）。从技术角度来讲，它成为了共识组（consensus group）中的一个<ruby>节点<rt>node</rt></ruby>。其背后的数学逻辑建立在 [Raft][3] 算法之上。管理者（manager）负责调度任务。而具体的任务则会委任给各个加入了 Swarm 集群的工作者（worker）节点。这些操作将由 Node API 所管理。虽说我讨厌 API 这个词汇，但我必须在这里用到它。
|
||||
|
||||
Service API 是这个实现中的第二个组件。它允许管理者（manager）节点在所有的 Swarm 集群节点上创建一个分布式的服务。这个服务可以被复制（replicated），也就是说它们（LCTT 译注：指这些服务）会由平衡机制被分配到集群中（LCTT 译注：指 replicated 模式，多个容器实例将会自动调度任务到集群中的一些满足条件的节点），或者可以分配给全局（LCTT 译注：指 global 模式），也就是说每个节点都会运行一个容器实例。
|
||||
|
||||
此外还有更多的功课需要做，但这些信息已经足够你上路了。现在，我们开始整些实际的。我们的目标平台是 [CentOS 7.2][4]，有趣的是在我写这篇教程的时候，它的软件仓库中只有 1.10 版的 Docker，也就是说我必须手动更新以使用 Swarm。我们将在另一篇教程中讨论这个问题。接下来我们还有一个跟进的指南，其中涵盖了如何将新的节点加入我们现有的集群（LCTT 译注：指刚刚建立的单节点集群），并且我们将使用 [Fedora][5] 进行一个非对称的配置。至此，请确保正确的配置已经就位，并有一个工作的集群启动并正在运行（LCTT 译注：指第一个节点的 Docker 已经安装并已进入 Swarm 模式，但到这里笔者并没有介绍如何初始化 Swarm 集群，不过别担心下章会讲）。
|
||||
|
||||
### 配置镜像和服务
|
||||
|
||||
我将尝试配置一个负载均衡的 [Apache][6] 服务，并使用多个容器实例通过唯一的 IP 地址提供页面内容。挺标准的吧（LCTT 译注：指这个负载均衡的网页服务器）。这个例子同时也突出了你想要使用集群的大多数原因：可用性、冗余、横向扩展以及性能。当然，你同时需要考虑[网络][7]和[储存][8]这两块，但它们超出了这篇指南所涉及的范围了。
|
||||
|
||||
这个 Dockerfile 模板其实可以在官方镜像仓库里的 httpd 下找到。你只需一个最简单的设置来起步。至于如何下载或创建自己的镜像，请参考我的入门指南，链接可以在这篇教程的顶部可以找到。
|
||||
|
||||
```
|
||||
docker build -t my-apache2 .
|
||||
Sending build context to Docker daemon 2.048 kB
|
||||
Step 1 : FROM httpd:2.4
|
||||
Trying to pull repository docker.io/library/httpd ...
|
||||
2.4: Pulling from docker.io/library/httpd
|
||||
|
||||
8ad8b3f87b37: Pull complete
|
||||
c95e1f92326d: Pull complete
|
||||
96e8046a7a4e: Pull complete
|
||||
00a0d292c371: Pull complete
|
||||
3f7586acab34: Pull complete
|
||||
Digest: sha256:3ad4d7c4f1815bd1c16788a57f81b413...a915e50a0d3a4
|
||||
Status: Downloaded newer image for docker.io/httpd:2.4
|
||||
---> fe3336dd034d
|
||||
Step 2 : COPY ../public-html/ /usr/local/apache2/htdocs/
|
||||
...
|
||||
```
|
||||
|
||||
|
||||
|
||||
在你继续下面的步骤之前，你应该确保你能无错误的启动一个容器实例并能链接到这个网页服务器上（LCTT 译注：使用下面的命令）。一旦你确保你能连上，我们就可以开始着手创建一个分布式的服务。
|
||||
|
||||
```
|
||||
docker run -dit --name my-running-app my-apache2
|
||||
```
|
||||
|
||||
将这个 IP 地址输入浏览器，看看会出现什么。
|
||||
|
||||
### Swarm 初始化和配置
|
||||
|
||||
下一步就是启动 Swarm 集群了。你将需要这些最基础的命令来开始，它们与 Docker 博客中的例子非常相似：
|
||||
|
||||
```
|
||||
docker service create --name frontend --replicas 5 -p 80:80/tcp my-apache2:latest
|
||||
```
|
||||
|
||||
这里我们做了什么？我们创建了一个叫做 `frontent` 的服务，它有五个容器实例。同时我们还将主机的 80 端口和这些容器的 80 端口相绑定。我们将使用刚刚新创建的 Apache 镜像来做这个测试。然而，当你在自己的电脑上直接键入上面的指令时，你将看到下面的错误：
|
||||
|
||||
```
|
||||
docker service create --name frontend --replicas 5 -p 80:80/tcp my-apache2:latest
|
||||
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
|
||||
```
|
||||
|
||||
这意味着你没有将你的主机（节点）配置成一个 Swarm 管理者（manager）。你可以在这台主机上初始化 Swarm 集群或是让它加入一个现有的集群。由于我们目前还没有一个现成的集群，我们将初始化它（LCTT 译注：指初始化 Swarm 集群并使当前节点成为 manager）：
|
||||
|
||||
```
|
||||
docker swarm init
|
||||
Swarm initialized: current node (dm58mmsczqemiikazbfyfwqpd) is now a manager.
|
||||
```
|
||||
|
||||
为了向这个 Swarm 集群添加一个工作者（worker），请执行下面的指令：
|
||||
|
||||
```
|
||||
docker swarm join \
|
||||
--token SWMTKN-1-4ofd46a2nfyvrqwu8w5oeetukrbylyznxla
|
||||
9srf9vxkxysj4p8-eu5d68pu5f1ci66s7w4wjps1u \
|
||||
10.0.2.15:2377
|
||||
```
|
||||
|
||||
为了向这个 Swarm 集群添加一个管理者（manager），请执行 `docker swarm join-token manager` 并按照指示操作。
|
||||
|
||||
操作后的输出不用解释已经很清楚明了。我们成功的创建了一个 Swarm 集群。新的节点们将需要正确的令牌（token）来加入这个 Swarm 集群。如果你需要配置防火墙，你还需找到它的 IP 地址和端口（LCTT 译注：指 Docker 的 Swarm 模式通讯所需的端口，默认 2377）。此外，你还可以向 Swarm 集群中添加管理者节点。现在，重新执行刚刚的服务创建指令：
|
||||
|
||||
```
|
||||
docker service create --name frontend --replicas 5 -p 80:80/tcp my-apache2:latest
|
||||
6lrx1vhxsar2i50is8arh4ud1
|
||||
```
|
||||
|
||||
### 测试连通性
|
||||
|
||||
现在，我们来验证下我们的服务是否真的工作了。从某些方面讲，这很像我们在 [Vagrant][9] 和 [coreOS][10] 中做的事情那样。毕竟它们的原理几乎相同。相同指导思想的不同实现罢了（LCTT 译注：笔者观点，无法苟同）。首先需要确保 `docker ps` 能够给出正确的输出。你应该能看到所创建服务的多个容器副本。
|
||||
|
||||
```
|
||||
docker ps
|
||||
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
|
||||
NAMES
|
||||
cda532f67d55 my-apache2:latest "httpd-foreground"
|
||||
2 minutes ago Up 2 minutes 80/tcp frontend.1.2sobjfchdyucschtu2xw6ms9a
|
||||
75fe6e0aa77b my-apache2:latest "httpd-foreground"
|
||||
2 minutes ago Up 2 minutes 80/tcp frontend.4.ag77qtdeby9fyvif5v6c4zcpc
|
||||
3ce824d3151f my-apache2:latest "httpd-foreground"
|
||||
2 minutes ago Up 2 minutes 80/tcp frontend.2.b6fqg6sf4hkeqs86ps4zjyq65
|
||||
eda01569181d my-apache2:latest "httpd-foreground"
|
||||
2 minutes ago Up 2 minutes 80/tcp frontend.5.0rmei3zeeh8usagg7fn3olsp4
|
||||
497ef904e381 my-apache2:latest "httpd-foreground"
|
||||
2 minutes ago Up 2 minutes 80/tcp frontend.3.7m83qsilli5dk8rncw3u10g5a
|
||||
```
|
||||
|
||||
我也测试了不同的、非常规的端口，它们都能正常工作。对于你如何连接服务器和收取请求你将会有很多可配置的余地。你可以使用 localhost 或者 Docker 网络接口（笔者注：应该是指 Docker 的默认网桥 docker0，其网关为 172.17.0.1） IP 地址的正确端口去访问。下面的例子使用了端口 1080：
|
||||
|
||||
|
||||
|
||||
至此，这是一个非常粗略、简单的开始。真正的挑战是创建一个优化过的、可扩展的服务，但是它们需要一个准确的技术用例。此外，你还会用到 `docker info` 和 `docker service`（还有 `inspect` 和 `ps`）命令来详细了解你的集群是如何工作的。
|
||||
|
||||
### 可能会遇到的问题
|
||||
|
||||
你可能会在把玩 Docker 和 Swarm 时遇到一些小的问题（也许没那么小）。比如 SELinux 也许会抱怨你正在执行一些非法的操作（LCTT 译注：指在强制访问控制策略中没有权限的操作）。然而，这些错误和警告应该不会对你造成太多阻碍。
|
||||
|
||||
|
||||
|
||||
- `docker service` 不是一条命令（`docker service is not a docker command`）
|
||||
|
||||
当你尝试执行必须的命令去创建一个复制模式（replicated）的服务时，你可能会遇到一条错误说 `docker: 'service' is not a docker command`（LCTT 译注：见下面的例子）。这表示你的 Docker 版本不对（使用 `-v` 选项来检查）。我们将在将来的教程讨论如何修复这个问题。
|
||||
|
||||
```
|
||||
docker service create --name frontend --replicas 5 -p 80:80/tcp my-apache2:latest
|
||||
docker: 'service' is not a docker command.
|
||||
```
|
||||
|
||||
- `docker tag` 无法识别（`docker tag not recognized`）
|
||||
|
||||
你也许会看到下面的错误：
|
||||
|
||||
```
|
||||
docker service create -name frontend -replicas 5 -p 80:80/tcp my-apache2:latest
|
||||
Error response from daemon: rpc error: code = 3 desc = ContainerSpec: "-name" is not a valid repository/tag
|
||||
```
|
||||
|
||||
关于这个错误已经有多个相关的[讨论][11]和[帖子][12]了。其实这个错误也许相当无辜。你也许是从浏览器粘贴的命令，在浏览器中的横线也许没被正确解析（笔者注：应该用 `--name` 而不是 `-name`）。就是这么简单的原因所导致的。
|
||||
|
||||
### 扩展阅读
|
||||
|
||||
关于这个话题还有很多可谈的，包含 1.12 版之前的 Swarm 集群实现（笔者注：旧的 Swarm 集群实现，下文亦作`独立版本`，需要 Consul 等应用提供服务发现），以及当前的 Docker 版本提供的（笔者注：新的 Swarm 集群实现，亦被称为 Docker 引擎的 Swarm 模式）。也就是说，请别偷懒花些时间阅读以下内容：
|
||||
|
||||
- Docker Swarm [概述][13]（独立版本的 Swarm 集群安装）
|
||||
- [构建][14]一个生产环境的 Swarm 集群（独立版本安装）
|
||||
- [安装并创建][15]一个 Docker Swarm 集群（独立版本安装）
|
||||
- Docker 引擎 Swarm [概述][16]（对于 1.12 版）
|
||||
- [Swarm][17] 模式入门（对于 1.12 版）
|
||||
|
||||
### 总结
|
||||
|
||||
你总算看到这里了。到这里仍然无法保证你学到了什么，但我相信你还是会觉得这篇文章有些用的。它涵盖了一些基础的概念，以及一个 Swarm 集群模式是如何工作的以及它能做什么的概述，与此同时我们也成功的下载了并创建了我们的网页服务器的镜像，并且在之后基于它运行了多个集群式的容器实例。虽然我们目前只在单一节点做了以上实验，但是我们会在将来解释清楚（LCTT 译注：以便解释清楚多节点的 Swarm 集群操作）。并且我们解决了一些常见的问题。
|
||||
|
||||
我希望你能认为这篇指南足够有趣。结合着我过去所写的关于 Docker 的文章，这些文章应该能给你一个像样的解释，包括：怎么样操作镜像、网络栈、储存、以及现在的集群。就当热身吧。的确，请享受并期待在新的 Docker 教程中与你见面。我控几不住我记几啊。
|
||||
|
||||
祝你愉快。
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: http://www.dedoimedo.com/computers/docker-swarm-intro.html
|
||||
|
||||
作者：[Dedoimedo][a]
|
||||
译者：[Viz](https://github.com/vizv)
|
||||
校对：[wxy](https://github.com/wxy)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:http://www.dedoimedo.com/computers/docker-swarm-intro.html
|
||||
[1]:http://www.dedoimedo.com/computers/docker-guide.html
|
||||
[2]:https://blog.docker.com/2016/06/docker-1-12-built-in-orchestration/
|
||||
[3]:https://en.wikipedia.org/wiki/Raft_%28computer_science%29
|
||||
[4]:http://www.dedoimedo.com/computers/lenovo-g50-centos-xfce.html
|
||||
[5]:http://www.dedoimedo.com/computers/fedora-24-gnome.html
|
||||
[6]:https://hub.docker.com/_/httpd/
|
||||
[7]:http://www.dedoimedo.com/computers/docker-networking.html
|
||||
[8]:http://www.dedoimedo.com/computers/docker-data-volumes.html
|
||||
[9]:http://www.dedoimedo.com/computers/vagrant-intro.html
|
||||
[10]:http://www.dedoimedo.com/computers/vagrant-coreos.html
|
||||
[11]:https://github.com/docker/docker/issues/24192
|
||||
[12]:http://stackoverflow.com/questions/38618609/docker-swarm-1-12-name-option-not-recognized
|
||||
[13]:https://docs.docker.com/swarm/
|
||||
[14]:https://docs.docker.com/swarm/install-manual/
|
||||
[15]:https://docs.docker.com/swarm/install-w-machine/
|
||||
[16]:https://docs.docker.com/engine/swarm/
|
||||
[17]:https://docs.docker.com/engine/swarm/swarm-tutorial/
|
||||
183
published/20161031 An introduction to Linux filesystems.md
Normal file
183
published/20161031 An introduction to Linux filesystems.md
Normal file
@@ -0,0 +1,183 @@
|
||||
Linux 文件系统概览
|
||||
====
|
||||
|
||||
|
||||
|
||||
|
||||
本文旨在高屋建瓴地来讨论 Linux 文件系统概念，而不是对某种特定的文件系统，比如 EXT4 是如何工作的进行具体的描述。另外，本文也不是一个文件系统命令的教程。
|
||||
|
||||
每台通用计算机都需要将各种数据存储在硬盘驱动器（HDD）或其他类似设备上，比如 USB 存储器。这样做有两个原因。首先，当计算机关闭以后，内存（RAM）会失去存于它里面的内容。尽管存在非易失类型的 RAM，在计算机断电以后还能把数据存储下来（比如采用 USB 闪存和固态硬盘的闪存），但是，闪存和标准的、易失性的 RAM，比如 DDR3 以及其他相似类型的 RAM 相比，要贵很多。
|
||||
|
||||
数据需要存储在硬盘驱动上的另一个原因是，即使是标准的 RAM 也要比普通硬盘贵得多。尽管 RAM 和硬盘的价格都在迅速下降，但是 RAM 的价格依旧在以字节为单位来计算。让我们进行一个以字节为单位的快速计算：基于 16 GB 大的 RAM 的价格和 2 TB 大的硬盘驱动的价格。计算显示 RAM 的价格大约比硬盘驱动贵 71 倍。今天，一个典型的 RAM 的价格大约是 0.000000004373750 美元/每字节。
|
||||
|
||||
直观的展示一下在很久以前 RAM 的价格，在计算机发展的非常早的时期，其中一种类型的 RAM 是基于在 CRT 屏幕上的点。这种 RAM 非常昂贵，大约 1 美元/每字节。
|
||||
|
||||
### 定义
|
||||
|
||||
你可能听过其他人以各种不同和令人迷惑的方式谈论过文件系统。文件系统这个单词本身有多重含义，你需要从一个讨论或文件的上下文中理解它的正确含义。
|
||||
|
||||
我将根据我所观察到的在不同情况下使用“文件系统”这个词来定义它的不同含义。注意，尽管我试图遵循标准的“官方”含义，但是我打算基于它的不同用法来定义这个术语（如下）。这就是说我将在本文的后续章节中进行更详细的探讨。
|
||||
|
||||
1. 始于顶层 root（/）目录的整个 Linux 目录结构。
|
||||
2. 特定类型的数据存储格式，比如 EXT3、EXT4、BTRFS 以及 XFS 等等。Linux 支持近百种类型的文件系统，包括一些非常老的以及一些最新的。每一种文件系统类型都使用它自己独特的元数据结构来定义数据是如何存储和访问的。
|
||||
3. 用特定类型的文件系统格式化后的分区或逻辑卷，可以挂载到 Linux 文件系统的指定挂载点上。
|
||||
|
||||
### 文件系统的基本功能
|
||||
|
||||
磁盘存储是文件系统必须的功能，它与之伴生的有一些有趣而且不可或缺的细节。很明显，文件系统是用来为非易失数据的存储提供空间，这是它的基本功能。然而，它还有许多从需求出发的重要功能。
|
||||
|
||||
所有文件系统都需要提供一个名字空间，这是一种命名和组织方法。它定义了文件应该如何命名、文件名的最大长度，以及所有可用字符集中可用于文件名中字符集子集。它也定义了一个磁盘上数据的逻辑结构，比如使用目录来组织文件而不是把所有文件聚集成一个单一的、巨大的文件混合体。
|
||||
|
||||
定义名字空间以后，元数据结构是为该名字空间提供逻辑基础所必须的。这包括所需数据结构要能够支持分层目录结构，同时能够通过结构来确定硬盘空间中的块是已用的或可用的，支持修改文件或目录的名字，提供关于文件大小、创建时间、最后访问或修改时间等信息，以及位置或数据所属的文件在磁盘空间中的位置。其他的元数据用来存储关于磁盘细分的高级信息，比如逻辑卷和分区。这种更高层次的元数据以及它所代表的结构包含描述文件系统存储在驱动器或分区中的信息，但与文件系统元数据无关，与之独立。
|
||||
|
||||
文件系统也需要一个应用程序接口（API），从而提供了对文件系统对象，比如文件和目录进行操作的系统功能调用的访问。API 也提供了诸如创建、移动和删除文件的功能。它也提供了算法来确定某些信息，比如文件存于文件系统中的位置。这样的算法可以用来解释诸如磁盘速度和最小化磁盘碎片等术语。
|
||||
|
||||
现代文件系统还提供一个安全模型，这是一个定义文件和目录的访问权限的方案。Linux 文件系统安全模型确保用户只能访问自己的文件，而不能访问其他用户的文件或操作系统本身。
|
||||
|
||||
最后一块组成部分是实现这些所有功能所需要的软件。Linux 使用两层软件实现的方式来提高系统和程序员的效率。
|
||||
|
||||
|
||||
|
||||
*图片 1：Linux 两层文件系统软件实现。*
|
||||
|
||||
这两层中的第一层是 Linux 虚拟文件系统。虚拟文件系统提供了内核和开发者访问所有类型文件系统的的单一命令集。虚拟文件系统软件通过调用特殊设备驱动来和不同类型的文件系统进行交互。特定文件系统的设备驱动是第二层实现。设备驱动程序将文件系统命令的标准集解释为在分区或逻辑卷上的特定类型文件系统命令。
|
||||
|
||||
### 目录结构
|
||||
|
||||
作为一个通常来说非常有条理的处女座，我喜欢将东西存储在更小的、有组织的小容器中，而不是存于同一个大容器中。目录的使用使我能够存储文件并在我想要查看这些文件的时候也能够找到它们。目录也被称为文件夹，之所以被称为文件夹，是因为其中的文件被类比存放于物理桌面上。
|
||||
|
||||
在 Linux 和其他许多操作系统中，目录可以被组织成树状的分层结构。在 [Linux 文件系统层次标准][10]中定义了 Linux 的目录结构（LCTT 译注：可参阅[这篇][23]）。当通过目录引用来访问目录时，更深层目录名字是通过正斜杠（/）来连接，从而形成一个序列，比如 `/var/log` 和 `/var/spool/mail` 。这些被称为路径。
|
||||
|
||||
下表提供了标准的、众所周知的、预定义的顶层 Linux 目录及其用途的简要清单。
|
||||
|
||||
| 目录 | 描述 |
|
||||
| ------------- | ---------------------------------------- |
|
||||
| **/ (root 文件系统)** | root 文件系统是文件系统的顶级目录。它必须包含在挂载其它文件系统前需要用来启动 Linux 系统的全部文件。它必须包含需要用来启动剩余文件系统的全部可执行文件和库。文件系统启动以后，所有其他文件系统作为 root 文件系统的子目录挂载到标准的、预定义好的挂载点上。 |
|
||||
| **/bin** | `/bin` 目录包含用户的可执行文件。 |
|
||||
| /boot | 包含启动 Linux 系统所需要的静态引导程序和内核可执行文件以及配置文件。 |
|
||||
| **/dev** | 该目录包含每一个连接到系统的硬件设备的设备文件。这些文件不是设备驱动，而是代表计算机上的每一个计算机能够访问的设备。 |
|
||||
| **/etc** | 包含主机计算机的本地系统配置文件。 |
|
||||
| /home | 主目录存储用户文件，每一个用户都有一个位于 `/home` 目录中的子目录（作为其主目录）。 |
|
||||
| **/lib** | 包含启动系统所需要的共享库文件。 |
|
||||
| /media | 一个挂载外部可移动设备的地方，比如主机可能连接了一个 USB 驱动器。 |
|
||||
| /mnt | 一个普通文件系统的临时挂载点（如不可移动的介质），当管理员对一个文件系统进行修复或在其上工作时可以使用。 |
|
||||
| /opt | 可选文件，比如供应商提供的应用程序应该安装在这儿。 |
|
||||
| **/root** | 这不是 root（`/`）文件系统。它是 root 用户的主目录。 |
|
||||
| **/sbin** | 系统二进制文件。这些是用于系统管理的可执行文件。 |
|
||||
| /tmp | 临时目录。被操作系统和许多程序用来存储临时文件。用户也可能临时在这儿存储文件。注意，存储在这儿的文件可能在任何时候在没有通知的情况下被删除。 |
|
||||
| /usr | 该目录里面包含可共享的、只读的文件，包括可执行二进制文件和库、man 文件以及其他类型的文档。 |
|
||||
| /var | 可变数据文件存储在这儿。这些文件包括日志文件、MySQL 和其他数据库的文件、Web 服务器的数据文件、邮件以及更多。 |
|
||||
|
||||
*表 1：Linux 文件系统层次结构的顶层*
|
||||
|
||||
这些目录以及它们的子目录如表 1 所示，在所有子目录中，粗体的目录组成了 root 文件系统的必需部分。也就是说，它们不能创建为一个分离的文件系统并且在开机时进行挂载。这是因为它们（特别是它们包含的内容）必须在系统启动的时候出现，从而系统才能正确启动。
|
||||
|
||||
`/media` 目录和 `/mnt` 目录是 root 文件系统的一部分，但是它们从来不包含任何数据，因为它们只是一个临时挂载点。
|
||||
|
||||
表 1 中剩下的非粗体的目录不需要在系统启动过程中出现，但会在之后挂载到 root 文件系统上，在开机阶段，它们为主机进行准备，从而执行有用的工作。
|
||||
|
||||
请参考官方 [Linux 文件系统层次标准][11]（FHS）网页来了解这些每一个目录以及它们的子目录的更多细节。维基百科上也有关于 [FHS][12] 的一个很好的介绍。应该尽可能的遵循这些标准，从而确保操作和功能的一致性。无论在主机上使用什么类型的文件系统，该层次目录结构都是相同的。
|
||||
|
||||
### Linux 统一目录结构
|
||||
|
||||
在一些非 Linux 操作系统的个人电脑上，如果有多个物理硬盘驱动器或多个分区，每一个硬盘或分区都会分配一个驱动器号。知道文件或程序位于哪一个硬盘驱动器上是很有必要的，比如 `C:` 或 `D:` 。然后，你可以在命令中使用驱动器号，以 `D:` 为例，为了进入 `D:` 驱动器，你可以使用 `cd` 命令来更改工作目录为正确的目录，从而定位需要的文件。每一个硬盘驱动器都有自己单独的、完整的目录树。
|
||||
|
||||
Linux 文件系统将所有物理硬盘驱动器和分区统一为一个目录结构。它们均从顶层 root 目录（`/`）开始。所有其它目录以及它们的子目录均位于单一的 Linux 根目录下。这意味着只有一棵目录树来搜索文件和程序。
|
||||
|
||||
因为只有一个文件系统，所以 `/home`、`/tmp`、`/var`、`/opt` 或 `/usr` 能够创建在和 root（`/`）文件系统不同的物理硬盘驱动器、分区或逻辑分区上，然后挂载到一个挂载点（目录）上，从而作为 root 文件系统树的一部分。甚至可移动驱动器，比如 USB 驱动器或一个外接的 USB 或 ESATA 硬盘驱动器均可以挂载到 root 文件系统上，成为目录树不可或缺的部分。
|
||||
|
||||
当从 Linux 发行版的一个版本升级到另一个版本或从一个发行版更改到另一个发行版的时候，就会很清楚地看到这样创建到不同分区的好处。通常情况下，除了任何像 Fedora 中的 `dnf-upgrade` 之类的升级工具，会明智地在升级过程中偶尔重新格式化包含操作系统的硬盘驱动来删除那些长期积累的垃圾。如果 `/home` 目录是 root 文件系统的一部分（位于同一个硬盘驱动器），那么它也会被格式化，然后需要通过之前的备份恢复。如果 /home 目录作为一个分离的文件系统，那么安装程序将会识别到，并跳过它的格式化。对于存储数据库、邮箱、网页和其它可变的用户以及系统数据的 `/var` 目录也是这样的。
|
||||
|
||||
将 Linux 系统目录树的某些部分作为一个分离的文件系统还有一些其他原因。比如，在很久以前，我还不知道将所有需要的 Linux 目录均作为 root（`/`）文件系统的一部分可能存在的问题，于是，一些非常大的文件填满了 `/home` 目录。因为 `/home` 目录和 `/tmp` 目录均不是分离的文件系统，而是 root 文件系统的简单子目录，整个 root 文件系统就被填满了。于是就不再有剩余空间可以让操作系统用来存储临时文件或扩展已存在数据文件。首先，应用程序开始抱怨没有空间来保存文件，然后，操作系统也开始异常行动。启动到单用户模式，并清除了 `/home` 目录中的多余文件之后，终于又能够重新工作了。然后，我使用非常标准的多重文件系统设置来重新安装 Linux 系统，从而避免了系统崩溃的再次发生。
|
||||
|
||||
我曾经遇到一个情况，Linux 主机还在运行，但是却不允许用户通过 GUI 桌面登录。我可以通过使用[虚拟控制台][13]之一，通过命令行界面（CLI）本地登录，然后远程使用 SSH 。问题的原因是因为 `/tmp` 文件系统满了，因此 GUI 桌面登录时所需要的一些临时文件不能被创建。因为命令行界面登录不需要在 `/tmp` 目录中创建文件，所以无可用空间并不会阻止我使用命令行界面来登录。在这种情况下，`/tmp` 目录是一个分离的文件系统，在 `/tmp` 所位于的逻辑卷上还有大量的可用空间。我简单地[扩展了 /tmp 逻辑卷][14]的容量到能够容纳主机所需要的临时文件，于是问题便解决了。注意，这个解决方法不需要重启，当 `/tmp` 文件系统扩大以后，用户就可以登录到桌面了。
|
||||
|
||||
当我在一家很大的科技公司当实验室管理员的时候，遇到过另外一个故障。开发者将一个应用程序安装到了一个错误的位置（`/var`）。结果该应用程序崩溃了，因为 `/var` 文件系统满了，由于缺乏空间，存储于 `/var/log` 中的日志文件无法附加新的日志消息。然而，系统仍然在运行，因为 root 文件系统和 `/tmp` 文件系统还没有被填满。删除了该应用程序并重新安装在 `/opt` 文件系统后，问题便解决了。
|
||||
|
||||
### 文件系统类型
|
||||
|
||||
Linux 系统支持大约 100 种分区类型的读取，但是只能对很少的一些进行创建和写操作。但是，可以挂载不同类型的文件系统在同一个 root 文件系统上，并且是很常见的。在这样的背景下，我们所说的文件系统一词是指在硬盘驱动器或逻辑卷上的一个分区中存储和管理用户数据所需要的结构和元数据。能够被 Linux 系统的 `fdisk` 命令识别的文件系统类型的完整列表[在此][24]，你可以感受一下 Linux 系统对许多类型的系统的高度兼容性。
|
||||
|
||||
Linux 支持读取这么多类型的分区系统的主要目的是为了提高兼容性，从而至少能够与一些其他计算机系统的文件系统进行交互。下面列出了在 Fedora 中创建一个新的文件系统时的所有可选类型：
|
||||
|
||||
* btrfs
|
||||
* **cramfs**
|
||||
* **ext2**
|
||||
* **ext3**
|
||||
* **ext4**
|
||||
* fat
|
||||
* gfs2
|
||||
* hfsplus
|
||||
* minix
|
||||
* **msdos**
|
||||
* ntfs
|
||||
* reiserfs
|
||||
* **vfat**
|
||||
* xfs
|
||||
|
||||
其他发行版支持创建的文件系统类型不同。比如，CentOS 6 只支持创建上表中标为黑体的文件系统类型。
|
||||
|
||||
### 挂载
|
||||
|
||||
在 Linux 系统上“<ruby>挂载<rt>mount</rt></ruby>”文件系统的术语是指在计算机发展的早期，磁带或可移动的磁盘组需要需要物理地挂载到一个合适的驱动器设备上。当通过物理的方式放置到驱动器上以后，操作系统会逻辑地挂载位于磁盘上的文件系统，从而操作系统、应用程序和用户才能够访问文件系统中的内容。
|
||||
|
||||
一个挂载点简单的来说就是一个目录，就像任何其它目录一样，是作为 root 文件系统的一部分创建的。所以，比如，home 文件系统是挂载在目录 `/home` 下。文件系统可以被挂载到其他非 root 文件系统的挂载点上，但是这并不常见。
|
||||
|
||||
在 Linux 系统启动阶段的最初阶段，root 文件系统就会被挂载到 root 目录下（`/`）。其它文件系统在之后通过 SystemV 下的 `rc` 或更新一些的 Linux 发行版中的 `systemd` 等 Linux 启动程序挂载。在启动进程中文件系统的挂载是由 `/etc/fstab` 配置文件管理的。一个简单的记忆方法是，fstab 代表“<ruby>文件系统表<rt>file system table</rt></ruby>”，它包含了需要挂载的文件系统的列表，这些文件系统均指定了挂载点，以及针对特定文件系统可能需要的选项。
|
||||
|
||||
使用 `mount` 命令可以把文件系统挂载到一个已有的目录/挂载点上。通常情况下，任何作为挂载点的目录都应该是空的且不包含任何其他文件。Linux 系统不会阻止用户挂载一个已被挂载了文件系统的目录或将文件系统挂载到一个包含文件的目录上。如果你将文件系统挂载到一个已有的目录或文件系统上，那么其原始内容将会被隐藏，只有新挂载的文件系统的内容是可见的。
|
||||
|
||||
### 结论
|
||||
|
||||
我希望通过这篇文章，阐明了围绕文件系统这个术语的一些可能的模糊之处。我花费了很长的时间，以及在一个良师的帮助下才真正理解和欣赏到 Linux 文件系统的复杂性、优雅性和功能以及它的全部含义。
|
||||
|
||||
如果你有任何问题，请写到下面的评论中，我会尽力来回答它们。
|
||||
|
||||
### 下个月
|
||||
|
||||
Linux 的另一个重要概念是：[万物皆为文件][15]。这个概念对用户和系统管理员来说有一些有趣和重要的实际应用。当我说完这个理由之后，你可能会想阅读我的文章：[万物皆为文件][15]，这篇文章会在我下个月计划写的关于 `/dev` 目录的文章之前写完。（LCTT 译注，也可参阅[这篇][25]）
|
||||
|
||||
（题图 : 原始图片来自 Rikki Endsley. [CC BY-SA 4.0][9]）
|
||||
|
||||
-----------------
|
||||
|
||||
作者简介：
|
||||
|
||||
David Both 居住在美国北卡罗纳州的首府罗利，是一个 Linux 开源贡献者。他已经从事 IT 行业 40 余年，在 IBM 教授 OS/2 20 余年。1981 年，他在 IBM 开发了第一个关于最初的 IBM 个人电脑的培训课程。他也曾在 Red Hat 教授 RHCE 课程，也曾供职于 MCI worldcom，Cico 以及北卡罗纳州等。他已经为 Linux 开源社区工作近 20 年。
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://opensource.com/life/16/10/introduction-linux-filesystems
|
||||
|
||||
作者：[David Both][a]
|
||||
译者：[ucasFL](https://github.com/ucasFL)
|
||||
校对：[wxy](https://github.com/wxy)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://opensource.com/users/dboth
|
||||
[1]:https://opensource.com/resources/what-is-linux?intcmp=70160000000h1jYAAQ&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;utm_source=intcallout&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;utm_campaign=linuxcontent
|
||||
[2]:https://opensource.com/resources/what-are-linux-containers?intcmp=70160000000h1jYAAQ&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;utm_source=intcallout&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;utm_campaign=linuxcontent
|
||||
[3]:https://developers.redhat.com/promotions/linux-cheatsheet/?intcmp=70160000000h1jYAAQ&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;utm_source=intcallout&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;utm_campaign=linuxcontent
|
||||
[4]:https://developers.redhat.com/cheat-sheet/advanced-linux-commands-cheatsheet?intcmp=70160000000h1jYAAQ&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;utm_source=intcallout&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;utm_campaign=linuxcontent
|
||||
[5]:https://opensource.com/tags/linux?intcmp=70160000000h1jYAAQ&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;utm_source=intcallout&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;utm_campaign=linuxcontent
|
||||
[6]:https://opensource.com/life/16/10/introduction-linux-filesystems?rate=Qyf2jgkdgrj5_zfDwadBT8KsHZ2Gp5Be2_tF7R-s02Y
|
||||
[7]:https://opensource.com/users/dboth
|
||||
[8]:https://opensource.com/user/14106/feed
|
||||
[9]:https://creativecommons.org/licenses/by-sa/4.0/
|
||||
[10]:http://www.pathname.com/fhs/
|
||||
[11]:http://www.pathname.com/fhs/
|
||||
[12]:https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard
|
||||
[13]:https://en.wikipedia.org/wiki/Virtual_console
|
||||
[14]:https://opensource.com/business/16/9/linux-users-guide-lvm
|
||||
[15]:https://opensource.com/life/15/9/everything-is-a-file
|
||||
[16]:https://opensource.com/users/dboth
|
||||
[17]:https://opensource.com/users/dboth
|
||||
[18]:https://opensource.com/users/dboth
|
||||
[19]:https://opensource.com/life/16/10/introduction-linux-filesystems#comments
|
||||
[20]:https://opensource.com/tags/linux
|
||||
[21]:https://opensource.com/tags/sysadmin
|
||||
[22]:https://opensource.com/participate
|
||||
[23]:https://linux.cn/article-6132-1.html
|
||||
[24]:https://www.win.tue.nl/~aeb/partitions/partition_types-1.html
|
||||
[25]:https://linux.cn/article-7669-1.html
|
||||
@@ -0,0 +1,73 @@
|
||||
为什么我们比以往更需要开放的领导人
|
||||
============================================================
|
||||
|
||||
> 不断变化的社会和文化条件正促使着开放的领导。
|
||||
|
||||
|
||||
|
||||
|
||||
领导力就是力量。更具体地说，领导力是影响他人行动的力量。 关于领导力的神话不仅可以让人联想到人类浪漫的一面而且还有人类境况险恶的一面。 我们最终决定如何领导才能决定其真正的本质。
|
||||
|
||||
现代许多对领导力的理解都是在战争中诞生的，在那里，领导力意味着熟练地执行命令和控制思想。 在现代商业的大部分时间里，我们都是以一个到达权力顶峰的伟大的男人或女人作为领导，并通过地位来发挥力量的。 这种传统的通过等级和报告关系的领导方式严重依赖于正式的权威。 这些结构中的权威通过垂直层次结构向下流动，并沿命令链的形式存在。
|
||||
|
||||
然而，在 20 世纪后期，一些东西开始改变。 新技术打开了全球化的大门，从而使团队更加分散。 我们投入人力资本的方式开始转变，永远地改变了人们之间的沟通方式。组织内部的人开始感觉得到了责任感，他们要求对自己的成功(和失败)拥有归属感。 领导者不再是权力的唯一拥有者。 21世纪的领导者带领 21 世纪的组织开始了解授权、协作、责任和清晰的沟通是一种新型权力的本质。 这些新领导人开始分享权力——他们无保留地信任他们的追随者。
|
||||
|
||||
随着组织继续变得更加开放，即使是没有“领导力”头衔的人也会感到有责任推动变革。 这些组织消除了等级制度的枷锁，让工人们以他们认为合适的方式去工作。 历史暴露了 20 世纪领导人倾向通过单边决策和单向信息流来扼杀敏捷性。 但是，新世纪的领导者却是确定一个组织，让由它授权的若干个体来完成一些事情。 重点是权力赋予若干个体——坦率地说，一个领导者不能在任何时候出现在所有的地方，做出所有的决定。
|
||||
|
||||
因此，领导人也开始变得开放。
|
||||
|
||||
### 控制
|
||||
|
||||
当旧式领导人专注于指挥和控制的地位权力时，一个开放的领导者通过新形式的组织管理方式、新技术和其他减少摩擦的方式，将组织控制权放在了其它人身上，这样可以更有效的方式实现集体行动的方式。 这些领导者了解信任的力量，相信追随者总是会表现出主动性、参与性和独立性。 而这种新的领导方式需要在战术上有所转变——从告诉人们如何去做，到向他们展示如何去做，并在路上指导他们。开放的领导人很快就发现，领导力不是影响我们发挥进步的力量，而是我们在组织成员中分配的力量和信心。 21 世纪的领导者专注于社区和对他人的教化。最后，开放的领导者并不是专注于自我，而是无私的。
|
||||
|
||||
### 交流
|
||||
|
||||
20 世纪的领导者人组织并控制整个组织的信息的流动。 然而，开放的领导者试图通过与团队成员共享信息和背景（以及权力）来组织一个组织。 这些领导人摧毁了领地，谦逊前行，分享着前所未有的力量。 集体赋权和参与的协作创造了灵活性，分担责任，所有权，尤其是幸福。 当一个组织的成员被授权做他们的工作时，他们比等级层次的同事更快乐（因而更有生产力）。
|
||||
|
||||
### 信任
|
||||
|
||||
开放的领导者接受不确定性，相信他们的追随者在正确的时间做正确的事情。 他们拥有比传统对手，有更高的吸引人力资本效率的能力。 再说一次：他们不会像命令和控制的微观管理者那样运作。 提高透明度，而不是暗箱操作，他们尽可能的把决策和行动放在公开场合，解释决策的基础，并假设员工对组织内的情况有高度的把握。开放领导者的操作的前提是，如果没有他们的持续干预，该组织的人力资本就更有能力取得成功。
|
||||
|
||||
### 自治权
|
||||
|
||||
在 20 世纪具有强大指挥和控制力的领导者专注于某些权力的时候，一个开放的领导者更多地关注组织内个人的实际活动。 当领导者专注于个人时，他们就能够更好地训练和指导团队成员。 从这个角度来看，一个开放的领导者关注的是与组织的愿景和使命一致的行为和行动。最后，一个开放的领导者被看作是团队中的一员，而不是团队的领导者。 这并不意味着领导人放弃了权力的地位，而是低估了这一点，以分享权力，并通过自主创造成果赋予个人权力。
|
||||
|
||||
### 赋权
|
||||
|
||||
开放的领导人把重点放在授予组织成员的权力上。 在这个过程中承认领导者在组织人力资本中的技能、能力和信任，从而为整个团队带来了积极的动力和意愿。 最终，赋权就是帮助追随者相信他们自己的能力。 那些相信自己拥有个人权力的追随者更有可能采取主动行动、制定和实现更高的目标，并在困难的环境下坚持下去。 最终，开放组织的概念是关于包容性，每个人都是属于自己的，个性和不同的观点对于成功是至关重要的。 一个开放的组织及其开放的领导者提供了一种社区的感觉，而成员则受到组织的使命或目的的驱动。 这会产生一种比个人更大的归属感。 个性创造了成员之间的幸福和工作满意度。 反过来，又实现了更高的效率和成功。
|
||||
|
||||
我们都应该为 21 世纪领导人所要求的开放性而努力。 这需要自我反省，好奇心，尤其是它正在进行的改变。 通过新的态度和习惯，我们逐渐发现了一个真正的开放领导者，并且希望我们在适应 21 世纪的领导风格的同时，也开始采纳这些理念。
|
||||
|
||||
是的，领导力就是力量。我们如何利用这种权力决定了我们组织的成败。 那些滥用权力的人不会持久，但那些分享权力和庆祝他人的人会更持久。 通过阅读 [这本书][7]，你可以在开放组织及其领导的持续对话中开始发挥重要作用。 在[本卷][8]的结论中，您将找到与开放组织社区联系的额外资源和机会，以便您也可以与我们聊天、思考和成长。 欢迎来到谈话——欢光临！
|
||||
|
||||
_这篇文章最初是作为《开放组织领导手册》的引言出现的，它现在可以[从 Opensource.com 中可获得][5]。_
|
||||
|
||||
（ Image by : opensource.com）
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
作者简介：
|
||||
|
||||
Philip A Foster - Dr. Philip A. Foster 是一名领导/商业教练兼顾问兼兼职教授。 他是企业运营、组织发展、展望和战略领导层的著名思想领袖。 Dr. Foster 通过设计和实施战略、战略预见和规划来促进变革。
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://opensource.com/open-organization/17/2/need-open-leaders-more-ever
|
||||
|
||||
作者：[Philip A Foster][a]
|
||||
译者：[TimeBear](https://github.com/TimeBear)
|
||||
校对：[wxy](https://github.com/wxy)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://opensource.com/users/maximumchange
|
||||
[1]:https://opensource.com/open-organization/resources/leaders-manual?src=too_resource_menu
|
||||
[2]:https://opensource.com/open-organization/resources/field-guide?src=too_resource_menu
|
||||
[3]:https://opensource.com/open-organization/resources/open-org-definition?src=too_resource_menu
|
||||
[4]:https://opensource.com/open-organization/resources/open-decision-framework?src=too_resource_menu
|
||||
[5]:https://opensource.com/open-organization/resources/leaders-manual
|
||||
[6]:https://opensource.com/open-organization/17/2/need-open-leaders-more-ever?rate=c_9hT0EKbdXcTGRl-YW0QgW60NsRwO2a4RaplUKfvXs
|
||||
[7]:https://opensource.com/open-organization/resources/leaders-manual
|
||||
[8]:https://opensource.com/open-organization/resources/leaders-manual
|
||||
[9]:https://opensource.com/user/15497/feed
|
||||
[10]:https://opensource.com/users/maximumchange
|
||||
@@ -0,0 +1,151 @@
|
||||
Docker 引擎的 Swarm 模式：添加工作者节点教程
|
||||
================
|
||||
|
||||
让我们继续几周前在 CentOS 7.2 中开始的工作。 在本[指南][1]中，我们学习了如何初始化以及启动 Docker 1.12 中内置的原生的集群以及编排功能。但是我们只有管理者（manager）节点还没有其它工作者（worker）节点。今天我们会展开讲述这个。
|
||||
|
||||
我将向你展示如何将不对称节点添加到 Sawrm 中，比如一个与 CentOS 相邻的 [Fedora 24][2]，它们都将加入到集群中，还有相关很棒的负载均衡等等。当然这并不是轻而易举的，我们会遇到一些障碍，所以它应该是非常有趣的。
|
||||
|
||||
|
||||
|
||||
### 先决条件
|
||||
|
||||
在将其它节点成功加入 Swarm 之前，我们需要做几件事情。理想情况下，所有节点都应该运行相同版本的 Docker，为了支持原生的编排功能，它的版本至少应该为 1.12。像 CentOS 一样，Fedora 内置的仓库没有最新的构建版本，所以你需要手动构建，或者使用 Docker 仓库手动[添加和安装][3]正确的版本，并修复一些依赖冲突。我已经向你展示了如何在 CentOS 中操作，经过是相同的。
|
||||
|
||||
此外，所有节点都需要能够相互通信。这就需要有正确的路由和防火墙规则，这样管理者（manager）和工作者（worker）节点才能互相通信。否则，你无法将节点加入 Swarm 中。最简单的解决方法是临时清除防火墙规则 （`iptables -F`），但这可能会损害你的安全。请确保你完全了解你正在做什么，并为你的节点和端口创建正确的规则。
|
||||
|
||||
> Error response from daemon: Timeout was reached before node was joined. The attempt to join the swarm will continue in the background. Use the "docker info" command to see the current swarm status of your node.
|
||||
|
||||
> 守护进程的错误响应：节点加入之前已超时。尝试加入 Swarm 的请求将在后台继续进行。使用 “docker info” 命令查看节点的当前 Swarm 状态。
|
||||
|
||||
你需要在主机上提供相同的 Docker 镜像。在上一个教程中我们创建了一个 Apache 映像，你需要在你的工作者（worker）节点上执行相同操作，或者分发已创建的镜像。如果你不这样做，你会遇到错误。如果你在设置 Docker 上需要帮助，请阅读我的[介绍指南][4]和[网络教程][5]。
|
||||
|
||||
```
|
||||
7vwdxioopmmfp3amlm0ulimcu \_ websky.11 my-apache2:latest
|
||||
localhost.localdomain Shutdown Rejected 7 minutes ago
|
||||
"No such image: my-apache2:lat&"
|
||||
```
|
||||
|
||||
### 现在开始
|
||||
|
||||
现在我们有一台启动了 CentOS 机器，并成功地创建了容器。你可以使用主机端口连接到该服务，这一切都看起来很好。目前，你的 Swarm 只有管理者（manager）。
|
||||
|
||||
|
||||
|
||||
### 加入工作者（worker）
|
||||
|
||||
要添加新的节点，你需要使用 `join` 命令。但是你首先必须提供令牌、IP 地址和端口，以便工作者（woker）节点能正确地对 Swarm 管理器进行身份验证。接着（在 Fedora 上）执行：
|
||||
|
||||
```
|
||||
[root@localhost ~]# docker swarm join-token worker
|
||||
To add a worker to this swarm, run the following command:
|
||||
|
||||
docker swarm join \
|
||||
--token SWMTKN-1-0xvojvlza90nrbihu6gfu3qm34ari7lwnza ... \
|
||||
192.168.2.100:2377
|
||||
```
|
||||
|
||||
如果你不修复防火墙和路由规则，你会得到超时错误。如果你已经加入了 Swarm，重复 `join` 命令会收到错误：
|
||||
|
||||
```
|
||||
Error response from daemon: This node is already part of a swarm. Use "docker swarm leave" to leave this swarm and join another one.
|
||||
```
|
||||
|
||||
如果有疑问，你可以离开 Swarm，然后重试：
|
||||
|
||||
```
|
||||
[root@localhost ~]# docker swarm leave
|
||||
Node left the swarm.
|
||||
|
||||
docker swarm join --token
|
||||
SWMTKN-1-0xvojvlza90nrbihu6gfu3qnza4 ... 192.168.2.100:2377
|
||||
This node joined a swarm as a worker.
|
||||
```
|
||||
|
||||
在工作者（worker）节点中，你可以使用 `docker info` 来检查状态：
|
||||
|
||||
```
|
||||
Swarm: active
|
||||
NodeID: 2i27v3ce9qs2aq33nofaon20k
|
||||
Is Manager: false
|
||||
Node Address: 192.168.2.103
|
||||
|
||||
Likewise, on the manager:
|
||||
|
||||
Swarm: active
|
||||
NodeID: cneayene32jsb0t2inwfg5t5q
|
||||
Is Manager: true
|
||||
ClusterID: 8degfhtsi7xxucvi6dxvlx1n4
|
||||
Managers: 1
|
||||
Nodes: 3
|
||||
Orchestration:
|
||||
Task History Retention Limit: 5
|
||||
Raft:
|
||||
Snapshot Interval: 10000
|
||||
Heartbeat Tick: 1
|
||||
Election Tick: 3
|
||||
Dispatcher:
|
||||
Heartbeat Period: 5 seconds
|
||||
CA Configuration:
|
||||
Expiry Duration: 3 months
|
||||
Node Address: 192.168.2.100
|
||||
```
|
||||
|
||||
### 创建或缩放服务
|
||||
|
||||
现在，我们需要看下 Docker 是否以及如何在节点间分发容器。我的测试展示了一个在非常轻的负载下相当简单的平衡算法。试了一两次之后，即使在我尝试缩放并更新之后，Docker 也没有将运行的服务重新分配给新的 worker。同样，有一次，它在工作者（worker）节点上创建了一个新的服务。也许这是最好的选择。
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
*在新的工作者（worker）节点上完整创建新的服务。*
|
||||
|
||||
过了一段时间，两个容器之间的现有服务有一些重新分配，但这需要一些时间。新服务工作正常。这只是一个前期观察，所以我现在不能说更多。现在是开始探索和调整的新起点。
|
||||
|
||||
|
||||
|
||||
*负载均衡过了一会工作了。*
|
||||
|
||||
### 总结
|
||||
|
||||
Docker 是一只灵巧的小野兽，它仍在继续长大，变得更复杂、更强大，当然也更优雅。它被一个大企业吃掉只是一个时间问题。当它带来了原生的编排功能时，Swarm 模式运行得很好，但是它不只是几个容器而已，而是充分利用了其算法和可扩展性。
|
||||
|
||||
我的教程展示了如何将 Fedora 节点添加到由 CentOS 运行的群集中，并且两者能并行工作。关于负载平衡还有一些问题，但这是我将在以后的文章中探讨的。总而言之，我希望这是一个值得记住的一课。我们已经解决了在尝试设置 Swarm 时可能遇到的一些先决条件和常见问题，同时我们启动了一堆容器，我们甚至简要介绍了如何缩放和分发服务。要记住，这只是一个开始。
|
||||
|
||||
干杯。
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
作者简介：
|
||||
|
||||
我是 Igor Ljubuncic。现在大约 38 岁，已婚但还没有孩子。我现在在一个大胆创新的云科技公司做首席工程师。直到大约 2015 年初时，我还在一个全世界最大的 IT 公司之一中做系统架构工程师，和一个工程计算团队开发新的基于 Linux 的解决方案，优化内核以及攻克 Linux 的问题。在那之前，我是一个为高性能计算环境设计创新解决方案的团队的技术领导。还有一些其他花哨的头衔，包括系统专家、系统程序员等等。所有这些都曾是我的爱好，但从 2008 年开始成为了我的付费工作。还有什么比这更令人满意的呢？
|
||||
|
||||
从 2004 年到 2008 年间，我曾通过作为医学影像行业的物理学家来糊口。我的工作专长集中在解决问题和算法开发。为此，我广泛地使用了 Matlab，主要用于信号和图像处理。另外，我得到了几个主要的工程方法学的认证，包括 MEDIC 六西格玛绿带、试验设计以及统计工程学。
|
||||
|
||||
我也开始写书，包括奇幻类和 Linux 上的技术性工作。彼此交融。
|
||||
|
||||
要查看我开源项目、出版物和专利的完整列表，请滚动到下面。
|
||||
|
||||
有关我的奖项，提名和 IT 相关认证的完整列表，请稍等一下。
|
||||
|
||||
-------------
|
||||
|
||||
|
||||
via: http://www.dedoimedo.com/computers/docker-swarm-adding-worker-nodes.html
|
||||
|
||||
作者：[Igor Ljubuncic][a]
|
||||
译者：[geekpi](https://github.com/geekpi)
|
||||
校对：[wxy](https://github.com/wxy)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:http://www.dedoimedo.com/faq.html
|
||||
[1]:https://linux.cn/article-8888-1.html
|
||||
[2]:http://www.dedoimedo.com/computers/fedora-24-gnome.html
|
||||
[3]:http://www.dedoimedo.com/computers/docker-centos-upgrade-latest.html
|
||||
[4]:http://www.dedoimedo.com/computers/docker-guide.html
|
||||
[5]:http://www.dedoimedo.com/computers/docker-networking.html
|
||||
71
published/20170310 Developer-defined application delivery.md
Normal file
71
published/20170310 Developer-defined application delivery.md
Normal file
@@ -0,0 +1,71 @@
|
||||
开发者定义的应用交付
|
||||
============================================================
|
||||
|
||||
> 负载均衡器如何帮助你解决分布式系统的复杂性。
|
||||
|
||||
|
||||
|
||||
|
||||
原生云应用旨在利用分布式系统的性能、可扩展性和可靠性优势。不幸的是，分布式系统往往以额外的复杂性为代价。由于你程序的各个组件跨网络分布，并且这些网络有通信障碍或者性能降级，因此你的分布式程序组件需要能够继续独立运行。
|
||||
|
||||
为了避免程序状态的不一致，分布式系统设计应该有一个共识，即组件会失效。没有什么比在网络中更突出了。因此，在其核心，分布式系统在很大程度上依赖于负载平衡——请求分布于两个或多个系统，以便在面临网络中断时具有弹性，并在系统负载波动时水平缩放时。
|
||||
|
||||
随着分布式系统在原生云程序的设计和交付中越来越普及，负载平衡器在现代应用程序体系结构的各个层次都影响了基础设施设计。在大多数常见配置中，负载平衡器部署在应用程序前端，处理来自外部世界的请求。然而，微服务的出现意味着负载平衡器可以在幕后发挥关键作用：即管理_服务_之间的流。
|
||||
|
||||
因此，当你使用原生云程序和分布式系统时，负载均衡器将承担其他角色：
|
||||
|
||||
* 作为提供缓存和增加安全性的**反向代理**，因为它成为外部客户端的中间人。
|
||||
* 作为通过提供协议转换（例如 REST 到 AMQP）的 **API 网关**。
|
||||
* 它可以处理**安全性**（即运行 Web 应用程序防火墙）。
|
||||
* 它可能承担应用程序管理任务，如速率限制和 HTTP/2 支持。
|
||||
|
||||
鉴于它们的扩展能力远大于平衡流量，<ruby>负载平衡器<rt>load balancer</rt></ruby>可以更广泛地称为<ruby>应用交付控制器<rt>Application Delivery Controller</rt></ruby>（ADC）。
|
||||
|
||||
### 开发人员定义基础设施
|
||||
|
||||
从历史上看，ADC 是由 IT 专业人员购买、部署和管理的，最常见运行企业级架构的应用程序。对于物理负载平衡器设备（如 F5、Citrix、Brocade等），这种情况在很大程度上仍然存在。具有分布式系统设计和临时基础设施的云原生应用要求负载平衡器与它们运行时的基础设施 (如容器) 一样具有动态特性。这些通常是软件负载均衡器（例如来自公共云提供商的 NGINX 和负载平衡器）。云原生应用通常是开发人员主导的计划，这意味着开发人员正在创建应用程序（例如微服务器）和基础设施（Kubernetes 和 NGINX）。开发人员越来越多地对负载平衡 (和其他) 基础设施的决策做出或产生重大影响。
|
||||
|
||||
作为决策者，云原生应用的开发人员通常不会意识到企业基础设施需求或现有部署的影响，同时要考虑到这些部署通常是新的，并且经常在公共或私有云环境中进行部署。云技术将基础设施抽象为可编程 API，开发人员正在定义应用程序在该基础设施的每一层的构建方式。在有负载平衡器的情况下，开发人员会选择要使用的类型、部署方式以及启用哪些功能。它们以编程的方式对负载平衡器的行为进行编码 —— 随着程序在部署的生存期内增长、收缩和功能上进化时，它如何动态响应应用程序的需要。开发人员将基础设施定义为代码 —— 包括基础设施配置及其运维。
|
||||
|
||||
### 开发者为什么定义基础设施？
|
||||
|
||||
编写如何构建和部署应用程序的代码实践已经发生了根本性的转变，它体现在很多方面。简而言之，这种根本性的转变是由两个因素推动的：将新的应用功能推向市场所需的时间（_上市时间_）以及应用用户从产品中获得价值所需的时间（_获益时间_）。因此，新的程序写出来就被持续地交付（作为服务），无需下载和安装。
|
||||
|
||||
上市时间和获益时间的压力并不是新的，但由于其他因素的加剧，这些因素正在加强开发者的决策权力：
|
||||
|
||||
* 云：通过 API 将基础设施定义为代码的能力。
|
||||
* 伸缩：需要在大型环境中高效运维。
|
||||
* 速度：马上需要交付应用功能，为企业争取竞争力。
|
||||
* 微服务：抽象框架和工具选择，进一步赋予开发人员基础设施决策权力。
|
||||
|
||||
除了上述因素外，值得注意的是开源的影响。随着开源软件的普及和发展，开发人员手中掌握了许多应用程序基础设施 - 语言、运行时环境、框架、数据库、负载均衡器、托管服务等。微服务的兴起使应用程序基础设施的选择民主化，允许开发人员选择最佳的工具。在选择负载平衡器的情况下，那些与云原生应用的动态特质紧密集成并响应的那些人将超人一等。
|
||||
|
||||
### 总结
|
||||
|
||||
当你在仔细考虑你的云原生应用设计时，请与我一起讨论“[在云中使用 NGINX 和 Kubernetes 进行负载平衡][8]”。我们将检测不同公共云和容器平台的负载平衡功能，并通过一个宏应用的案例研究。我们将看看它是如何被变成较小的、独立的服务，以及 NGINX 和 Kubernetes 的能力是如何拯救它的。
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
作者简介：
|
||||
|
||||
Lee Calcote 是一位创新思想领袖，对开发者平台和云、容器、基础设施和应用的管理软件充满热情。先进的和新兴的技术一直是 Calcote 在 SolarWinds、Seagate、Cisco 和 Pelco 时的关注重点。他是技术会议和聚会的组织者、写作者、作家、演讲者，经常活跃在技术社区。
|
||||
|
||||
----------------------------
|
||||
|
||||
via: https://www.oreilly.com/learning/developer-defined-application-delivery
|
||||
|
||||
作者：[Lee Calcote][a]
|
||||
译者：[geekpi](https://github.com/geekpi)
|
||||
校对：[wxy](https://github.com/wxy)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://www.oreilly.com/people/7f693-lee-calcote
|
||||
[1]:https://pixabay.com/en/ship-containers-products-shipping-84139/
|
||||
[2]:https://conferences.oreilly.com/velocity/vl-ca?intcmp=il-webops-confreg-na-vlca17_new_site_velocity_sj_17_cta
|
||||
[3]:https://www.oreilly.com/people/7f693-lee-calcote
|
||||
[4]:http://www.oreilly.com/pub/e/3864?intcmp=il-webops-webcast-reg-webcast_new_site_developer_defined_application_delivery_text_cta
|
||||
[5]:https://www.oreilly.com/learning/developer-defined-application-delivery?imm_mid=0ee8c5&cmp=em-webops-na-na-newsltr_20170310
|
||||
[6]:https://conferences.oreilly.com/velocity/vl-ca?intcmp=il-webops-confreg-na-vlca17_new_site_velocity_sj_17_cta
|
||||
[7]:https://conferences.oreilly.com/velocity/vl-ca?intcmp=il-webops-confreg-na-vlca17_new_site_velocity_sj_17_cta
|
||||
[8]:http://www.oreilly.com/pub/e/3864?intcmp=il-webops-webcast-reg-webcast_new_site_developer_defined_application_delivery_body_text_cta
|
||||
178
published/20170422 FEWER MALLOCS IN CURL.md
Normal file
178
published/20170422 FEWER MALLOCS IN CURL.md
Normal file
@@ -0,0 +1,178 @@
|
||||
减少 curl 中内存分配操作（malloc）
|
||||
===========================================================
|
||||
|
||||
|
||||
|
||||
今天我在 libcurl 内部又做了[一个小改动][4]，使其做更少的 malloc。这一次，泛型链表函数被转换成更少的 malloc (这才是链表函数应有的方式，真的)。
|
||||
|
||||
### 研究 malloc
|
||||
|
||||
几周前我开始研究内存分配。这很容易，因为多年前我们 curl 中就已经有内存调试和日志记录系统了。使用 curl 的调试版本，并在我的构建目录中运行此脚本：
|
||||
|
||||
```
|
||||
#!/bin/sh
|
||||
export CURL_MEMDEBUG=$HOME/tmp/curlmem.log
|
||||
./src/curl http://localhost
|
||||
./tests/memanalyze.pl -v $HOME/tmp/curlmem.log
|
||||
```
|
||||
|
||||
对于 curl 7.53.1，这大约有 115 次内存分配。这算多还是少？
|
||||
|
||||
内存日志非常基础。为了让你有所了解，这是一个示例片段：
|
||||
|
||||
```
|
||||
MEM getinfo.c:70 free((nil))
|
||||
MEM getinfo.c:73 free((nil))
|
||||
MEM url.c:294 free((nil))
|
||||
MEM url.c:297 strdup(0x559e7150d616) (24) = 0x559e73760f98
|
||||
MEM url.c:294 free((nil))
|
||||
MEM url.c:297 strdup(0x559e7150d62e) (22) = 0x559e73760fc8
|
||||
MEM multi.c:302 calloc(1,480) = 0x559e73760ff8
|
||||
MEM hash.c:75 malloc(224) = 0x559e737611f8
|
||||
MEM hash.c:75 malloc(29152) = 0x559e737a2bc8
|
||||
MEM hash.c:75 malloc(3104) = 0x559e737a9dc8
|
||||
```
|
||||
|
||||
### 检查日志
|
||||
|
||||
然后，我对日志进行了更深入的研究，我意识到在相同的代码行做了许多小内存分配。我们显然有一些相当愚蠢的代码模式，我们分配一个结构体，然后将该结构添加到链表或哈希，然后该代码随后再添加另一个小结构体，如此这般，而且经常在循环中执行。（我在这里说的是_我们_，不是为了责怪某个人，当然大部分的责任是我自己……）
|
||||
|
||||
这两种分配操作将总是成对地出现，并被同时释放。我决定解决这些问题。做非常小的（小于 32 字节）的分配也是浪费的，因为非常多的数据将被用于（在 malloc 系统内）跟踪那个微小的内存区域。更不用说堆碎片了。
|
||||
|
||||
因此，将该哈希和链表代码修复为不使用 malloc 是快速且简单的方法，对于最简单的 “curl http://localhost” 传输，它可以消除 20％ 以上的 malloc。
|
||||
|
||||
此时，我根据大小对所有的内存分配操作进行排序，并检查所有最小的分配操作。一个突出的部分是在 `curl_multi_wait()` 中，它是一个典型的在 curl 传输主循环中被反复调用的函数。对于大多数典型情况，我将其转换为[使用堆栈][5]。在大量重复的调用函数中避免 malloc 是一件好事。
|
||||
|
||||
### 重新计数
|
||||
|
||||
现在，如上面的脚本所示，同样的 `curl localhost` 命令从 curl 7.53.1 的 115 次分配操作下降到 80 个分配操作，而没有牺牲任何东西。轻松地有 26％ 的改善。一点也不差！
|
||||
|
||||
由于我修改了 `curl_multi_wait()`，我也想看看它实际上是如何改进一些稍微更高级一些的传输。我使用了 [multi-double.c][6] 示例代码，添加了初始化内存记录的调用，让它使用 `curl_multi_wait()`，并且并行下载了这两个 URL：
|
||||
|
||||
```
|
||||
http://www.example.com/
|
||||
http://localhost/512M
|
||||
```
|
||||
|
||||
第二个文件是 512 兆字节的零，第一个文件是一个 600 字节的公共 html 页面。这是 [count-malloc.c 代码][7]。
|
||||
|
||||
首先，我使用 7.53.1 来测试上面的例子，并使用 `memanalyze` 脚本检查：
|
||||
|
||||
```
|
||||
Mallocs: 33901
|
||||
Reallocs: 5
|
||||
Callocs: 24
|
||||
Strdups: 31
|
||||
Wcsdups: 0
|
||||
Frees: 33956
|
||||
Allocations: 33961
|
||||
Maximum allocated: 160385
|
||||
```
|
||||
|
||||
好了，所以它总共使用了 160KB 的内存，分配操作次数超过 33900 次。而它下载超过 512 兆字节的数据，所以它每 15KB 数据有一次 malloc。是好是坏？
|
||||
|
||||
回到 git master，现在是 7.54.1-DEV 的版本 - 因为我们不太确定当我们发布下一个版本时会变成哪个版本号。它可能是 7.54.1 或 7.55.0，它还尚未确定。我离题了，我再次运行相同修改的 multi-double.c 示例，再次对内存日志运行 memanalyze，报告来了：
|
||||
|
||||
```
|
||||
Mallocs: 69
|
||||
Reallocs: 5
|
||||
Callocs: 24
|
||||
Strdups: 31
|
||||
Wcsdups: 0
|
||||
Frees: 124
|
||||
Allocations: 129
|
||||
Maximum allocated: 153247
|
||||
```
|
||||
|
||||
我不敢置信地反复看了两遍。发生什么了吗？为了仔细检查，我最好再运行一次。无论我运行多少次，结果还是一样的。
|
||||
|
||||
### 33961 vs 129
|
||||
|
||||
在典型的传输中 `curl_multi_wait()` 被调用了很多次，并且在传输过程中至少要正常进行一次内存分配操作，因此删除那个单一的微小分配操作对计数器有非常大的影响。正常的传输也会做一些将数据移入或移出链表和散列操作，但是它们现在也大都是无 malloc 的。简单地说：剩余的分配操作不会在传输循环中执行，所以它们的重要性不大。
|
||||
|
||||
以前的 curl 是当前示例分配操作数量的 263 倍。换句话说：新的是旧的分配操作数量的 0.37％ 。
|
||||
|
||||
另外还有一点好处，新的内存分配量更少，总共减少了 7KB（4.3％）。
|
||||
|
||||
### malloc 重要吗？
|
||||
|
||||
在几个 G 内存的时代里，在传输中有几个 malloc 真的对于普通人有显著的区别吗？对 512MB 数据进行的 33832 个额外的 malloc 有什么影响？
|
||||
|
||||
为了衡量这些变化的影响，我决定比较 localhost 的 HTTP 传输，看看是否可以看到任何速度差异。localhost 对于这个测试是很好的，因为没有网络速度限制，更快的 curl 下载也越快。服务器端也会相同的快/慢，因为我将使用相同的测试集进行这两个测试。
|
||||
|
||||
我相同方式构建了 curl 7.53.1 和 curl 7.54.1-DEV，并运行这个命令：
|
||||
|
||||
```
|
||||
curl http://localhost/80GB -o /dev/null
|
||||
```
|
||||
|
||||
下载的 80GB 的数据会尽可能快地写到空设备中。
|
||||
|
||||
我获得的确切数字可能不是很有用，因为它将取决于机器中的 CPU、使用的 HTTP 服务器、构建 curl 时的优化级别等，但是相对数字仍然应该是高度相关的。新代码对决旧代码！
|
||||
|
||||
7.54.1-DEV 反复地表现出更快 30％！我的早期版本是 2200MB/秒增加到当前版本的超过 2900 MB/秒。
|
||||
|
||||
这里的要点当然不是说它很容易在我的机器上使用单一内核以超过 20GB/秒的速度来进行 HTTP 传输，因为实际上很少有用户可以通过 curl 做到这样快速的传输。关键在于 curl 现在每个字节的传输使用更少的 CPU，这将使更多的 CPU 转移到系统的其余部分来执行任何需要做的事情。或者如果设备是便携式设备，那么可以省电。
|
||||
|
||||
关于 malloc 的成本：512MB 测试中，我使用旧代码发生了 33832 次或更多的分配。旧代码以大约 2200MB/秒的速率进行 HTTP 传输。这等于每秒 145827 次 malloc - 现在它们被消除了！600 MB/秒的改进意味着每秒钟 curl 中每个减少的 malloc 操作能额外换来多传输 4300 字节。
|
||||
|
||||
### 去掉这些 malloc 难吗？
|
||||
|
||||
一点也不难，非常简单。然而，有趣的是，在这个旧项目中，仍然有这样的改进空间。我有这个想法已经好几年了，我很高兴我终于花点时间来实现。感谢我们的测试套件，我可以有相当大的信心做这个“激烈的”内部变化，而不会引入太可怕的回归问题。由于我们的 API 很好地隐藏了内部，所以这种变化可以完全不改变任何旧的或新的应用程序……
|
||||
|
||||
（是的，我还没在版本中发布该变更，所以这还有风险，我有点后悔我的“这很容易”的声明……）
|
||||
|
||||
### 注意数字
|
||||
|
||||
curl 的 git 仓库从 7.53.1 到今天已经有 213 个提交。即使我没有别的想法，可能还会有一次或多次的提交，而不仅仅是内存分配对性能的影响。
|
||||
|
||||
### 还有吗？
|
||||
|
||||
还有其他类似的情况么？
|
||||
|
||||
也许。我们不会做很多性能测量或比较，所以谁知道呢，我们也许会做更多的愚蠢事情，我们可以收手并做得更好。有一个事情是我一直想做，但是从来没有做，就是添加所使用的内存/malloc 和 curl 执行速度的每日“监视” ，以便更好地跟踪我们在这些方面不知不觉的回归问题。
|
||||
|
||||
### 补遗，4/23
|
||||
|
||||
（关于我在 hacker news、Reddit 和其它地方读到的关于这篇文章的评论）
|
||||
|
||||
有些人让我再次运行那个 80GB 的下载，给出时间。我运行了三次新代码和旧代码，其运行“中值”如下：
|
||||
|
||||
旧代码：
|
||||
|
||||
```
|
||||
real 0m36.705s
|
||||
user 0m20.176s
|
||||
sys 0m16.072s
|
||||
```
|
||||
|
||||
新代码：
|
||||
|
||||
```
|
||||
real 0m29.032s
|
||||
user 0m12.196s
|
||||
sys 0m12.820s
|
||||
```
|
||||
|
||||
承载这个 80GB 文件的服务器是标准的 Apache 2.4.25，文件存储在 SSD 上，我的机器的 CPU 是 i7 3770K 3.50GHz 。
|
||||
|
||||
有些人也提到 `alloca()` 作为该补丁之一也是个解决方案，但是 `alloca()` 移植性不够，只能作为一个孤立的解决方案，这意味着如果我们要使用它的话，需要写一堆丑陋的 `#ifdef`。
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://daniel.haxx.se/blog/2017/04/22/fewer-mallocs-in-curl/
|
||||
|
||||
作者：[DANIEL STENBERG][a]
|
||||
译者：[geekpi](https://github.com/geekpi)
|
||||
校对：[wxy](https://github.com/wxy)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://daniel.haxx.se/blog/author/daniel/
|
||||
[1]:https://daniel.haxx.se/blog/author/daniel/
|
||||
[2]:https://daniel.haxx.se/blog/2017/04/22/fewer-mallocs-in-curl/
|
||||
[3]:https://daniel.haxx.se/blog/2017/04/22/fewer-mallocs-in-curl/#comments
|
||||
[4]:https://github.com/curl/curl/commit/cbae73e1dd95946597ea74ccb580c30f78e3fa73
|
||||
[5]:https://github.com/curl/curl/commit/5f1163517e1597339d
|
||||
[6]:https://github.com/curl/curl/commit/5f1163517e1597339d
|
||||
[7]:https://gist.github.com/bagder/dc4a42cb561e791e470362da7ef731d3
|
||||
106
published/20170517 Security Debt is an Engineers Problem.md
Normal file
106
published/20170517 Security Debt is an Engineers Problem.md
Normal file
@@ -0,0 +1,106 @@
|
||||
安全债务是工程师的问题
|
||||
============================================================
|
||||
|
||||
|
||||
|
||||
|
||||
>Keziah Plattner of AirBnBSecurity.
|
||||
|
||||
在上个月旧金山 Twitter 总部举办的 [WomenWhoCode Connect][5] 活动中参会者了解到，就像组织会形成技术债务一样，如果他们不相应地计划，也会形成一个名为“安全债务”的东西。
|
||||
|
||||
甲骨文首席安全官 [Mary Ann Davidson][6] 与 [WomenWhoCode][8] 的 [Zassmin Montes de Oca][7] 在一个面对开发人员的安全性的主题谈话中强调，安全性已经成为软件开发过程中每步的重要组成部分，
|
||||
|
||||
在过去，除银行外，安全性几乎被所有人忽视。但安全性比以往任何时候都更重要，因为现在有这么多接入点。我们已经进入[物联网][9]的时代，窃贼可以通过劫持你的冰箱而了解到你不在家的情况。
|
||||
|
||||
Davidson 负责 Oracle 的保障，“我们确保为建立的一切构建安全性，无论是内部部署产品、云服务，甚至是设备，我们在客户的网站建立有支持小组并报告数据给我们，帮助我们做诊断 - 每件事情都必须对其进行安全保护。”
|
||||
|
||||
|
||||
|
||||
*Plattner 与 #WWCConnect 人群交谈*
|
||||
|
||||
AirBnB 的 [Keziah Plattner][10] 在分组会议中回应了这个看法。她说：“大多数开发者并不认为安全是他们的工作，但这必须改变。”
|
||||
|
||||
她分享了工程师的四项基本安全原则。首先，安全债务是昂贵的。现在有很多人在谈论[技术债务][11]，她认为这些谈话应该也包括安全债务。
|
||||
|
||||
Plattner 说：“历史上这个看法是‘我们会稍后考虑安全’”。当公司抓住软件效率和增长的唾手可得的成果时，他们忽视了安全性，但最初的不安全设计可能在未来几年会引发问题。
|
||||
|
||||
她说，很难为现有的脆弱系统增加安全性。即使你知道安全漏洞在哪里，并且有进行更改的时间和资源的预算，重新设计一个安全系统也是耗时和困难的。
|
||||
|
||||
她说，所以这就是关键，从一开始就建立安全性。将安全性视为技术债务的一部分以避免这个问题，并涵盖所有可能性。
|
||||
|
||||
根据 Plattner 说的，最重要的是难以让人们改变行为。没有人会自愿改变，她说，即使你指出新的行为更安全。他们也只不过是点点头而已。
|
||||
|
||||
Davidson 说，工程师们需要开始考虑他们的代码如何被攻击，并从这个角度进行设计。她说她只有两个规则。第一个从不信任任何未验证的数据；规则二参见规则一。
|
||||
|
||||
她笑着说：“人们一直这样做。他们说：‘我的客户端给我发送数据，所以没有问题’。千万不要……”。
|
||||
|
||||
Plattner说，安全的第二个关键是“永远不信任用户”。
|
||||
|
||||
Davidson 以另外一种说法表示：“我的工作是做专业的偏执狂。”她一直担心有人或许无意中会破坏她的系统。这不是学术性的考虑，最近已经有通过 IoT 设备的拒绝服务攻击。
|
||||
|
||||
### Little Bobby Tables
|
||||
|
||||
Plattner 说：“如果你安全计划的一部分是信任用户做正确的事情，那么无论你有什么其他安全措施，你系统本质上是不安全的。”
|
||||
|
||||
她解释说，重要的是要净化所有的用户输入，如 [XKCD 漫画][12]中的那样，一位妈妈干掉整个学校的数据库——因为她的儿子的中间名是 “DropTable Students”（LCTT 译注：看不懂的[点这里][17]）。
|
||||
|
||||
|
||||
|
||||
所以净化所有的用户输入。你一定检查一下。
|
||||
|
||||
她展示了一个 JavaScript 开发者在开源软件中使用 eval 的例子。她警告说：“一个好的基本规则是‘从不使用 eval()’”。 [eval()] [13] 函数会执行 JavaScript 代码。“如果你这样做，你正在向任意用户开放你的系统。”
|
||||
|
||||
Davidson 警告说，她甚至偏执到将文档中的示例代码的安全测试也包括在内。她笑着说：“我们都知道没有人会去复制示例代码”。她强调指出，任何代码都应进行安全检查。
|
||||
|
||||
|
||||
|
||||
*让它容易*
|
||||
|
||||
Plattner 的第三个建议：要使安全容易实施。她建议采取阻力最小的道路。
|
||||
|
||||
对外，使用户<ruby>默认采用<rt>option out</rt></ruby>安全措施而不是<ruby>可选采用<rt>option in</rt></ruby>，或者更好使其成为强制性的措施。她说，改变人们的行为是科技中最难的问题。一旦用户习惯以非安全的方式使用你的产品，让他们改进会变得非常困难。
|
||||
|
||||
在公司内部，她建议制定安全标准，因此这不是个别开发人员需要考虑的内容。例如，将数据加密作为服务，这样工程师可以只需要调用服务就可以加密或解密数据。
|
||||
|
||||
她说，确保公司注重安全环境。在让整个公司切换到好的安全习惯。
|
||||
|
||||
你的最薄弱的环节决定了你的安全水准，所以重要的是每个人都有良好的个人安全习惯，并具有良好的企业安全环境。
|
||||
|
||||
在 Oracle，他们已经全面覆盖安全的各个环节。Davidson 表示，她厌倦了向没有安全培训的大学毕业的工程师解释安全性，所以她写了 Oracle 的第一个编码标准，现在已经有数百个页面之多以及很多贡献者，还有一些课程是强制性的。它们具有符合安全要求的度量标准。这些课程不仅适用于工程师，也适用于文档作者。她说：“这是一种文化。”
|
||||
|
||||
没有提及密码的关于安全性的讨论怎么能是安全的？Plattner 说：“每个人都应该使用一个好的密码管理器，在工作中应该是强制性的，还有双重身份验证。”
|
||||
|
||||
她说，基本的密码原则应该是每个工程师日常生活的一部分。密码中最重要的是它们的长度和熵（使按键的集合尽可能地随机）。强健的密码熵检查器对此非常有用。她建议使用 Dropbox 开源的熵检查器 [zxcvbn][14]。
|
||||
|
||||
Plattner 说，另一个诀窍是在验证用户输入时使用一些故意减慢速度的算法，如 [bcrypt][15]。慢速并不困扰大多数合法用户，但会让那些试图强行进行密码尝试的黑客难受。
|
||||
|
||||
Davidson 说：“所有这些都为那些想要进入技术安全领域的人提供了工作安全保障，我们在各种地方放了各种代码，这就产生了系统性风险。只要我们继续在技术领域做有趣的事情，我不认为任何人不想要在安全中工作。”
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://thenewstack.io/security-engineers-problem/
|
||||
|
||||
作者：[TC Currie][a]
|
||||
译者：[geekpi](https://github.com/geekpi)
|
||||
校对：[wxy](https://github.com/wxy)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://thenewstack.io/author/tc/
|
||||
[1]:http://twitter.com/share?url=https://thenewstack.io/security-engineers-problem/&text=Security+Debt+is+an+Engineer%E2%80%99s+Problem+
|
||||
[2]:http://www.facebook.com/sharer.php?u=https://thenewstack.io/security-engineers-problem/
|
||||
[3]:http://www.linkedin.com/shareArticle?mini=true&url=https://thenewstack.io/security-engineers-problem/
|
||||
[4]:https://thenewstack.io/security-engineers-problem/#disqus_thread
|
||||
[5]:http://connect2017.womenwhocode.com/
|
||||
[6]:https://www.linkedin.com/in/mary-ann-davidson-235ba/
|
||||
[7]:https://www.linkedin.com/in/zassmin/
|
||||
[8]:https://www.womenwhocode.com/
|
||||
[9]:https://www.thenewstack.io/tag/Internet-of-Things
|
||||
[10]:https://twitter.com/ittskeziah
|
||||
[11]:https://martinfowler.com/bliki/TechnicalDebt.html
|
||||
[12]:https://xkcd.com/327/
|
||||
[13]:https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/eval
|
||||
[14]:https://blogs.dropbox.com/tech/2012/04/zxcvbn-realistic-password-strength-estimation/
|
||||
[15]:https://en.wikipedia.org/wiki/Bcrypt
|
||||
[16]:https://thenewstack.io/author/tc/
|
||||
[17]:https://www.explainxkcd.com/wiki/index.php/Little_Bobby_Tables
|
||||
@@ -1,210 +1,185 @@
|
||||
8款适合树莓派使用的 IDE
|
||||
8 款适合树莓派使用的 IDE
|
||||
============================================================
|
||||
|
||||
|
||||
|
||||
|
||||
树莓派是一种微型的单板电脑（SBC），已经在学校的计算机科学教学中掀起了一场革命，但同样，它也给软件开发者带来了福音。目前，树莓派获得的知名度远远超出了它原本的目标市场，而且正在应用于机器人项目中。
|
||||
|
||||
_树莓派是一种微型的单板电脑，已经在计算机科学掀起了一场革命，它被设计用于教学。但同样也给软件开发者带来了福音。目前，树莓派获得的知名度远远超出了它原本的目标市场，而且正在应用于机器人项目中。_
|
||||
|
||||
树莓派是一个可以运行 Linux 操作系统的微型开发板，由英国树莓派基金会开发，用来在英国和发展中国家促进学校的基础计算机科学教育。树莓派拥有 USB 接口，能够支持多种即插即用外围设备，比如键盘，鼠标，打印机等。它包含了一个 HDMI（高清多媒体界面）端口，为用户提供视频输出。信用卡大小的尺寸使得树莓派非常便携且价格便宜。仅需一个 5V 的 micro-USB 电源供电，类似于给手机用的充电器一样。
|
||||
树莓派是一个可以运行 Linux 操作系统的微型开发板计算机，由英国树莓派基金会开发，用来在英国和发展中国家促进学校的基础计算机科学教育。树莓派拥有 USB 接口，能够支持多种即插即用外围设备，比如键盘、鼠标、打印机等。它包含了一个 HDMI（高清多媒体界面）端口，可以为用户提供视频输出。信用卡大小的尺寸使得树莓派非常便携且价格便宜。仅需一个 5V 的 micro-USB 电源供电，类似于给手机用的充电器一样。
|
||||
|
||||
多年来，树莓派基金会已经推出了几个不同版本的树莓派产品。 第一个版本是树莓派 1 B 型，随后是一个相对简单便宜的 A 型。在 2014 年，基金会推出了一个增强版本 —— 树莓派 1 B+。在 2015 年，基金会推出了全新设计的版本，售价为 5 美元，命名为树莓派 Zero。
|
||||
多年来，树莓派基金会已经推出了几个不同版本的树莓派产品。 第一个版本是树莓派 1B 型，随后是一个相对简单便宜的 A 型。在 2014 年，基金会推出了一个增强版本 —— 树莓派 1B+。在 2015 年，基金会推出了全新设计的版本，售价为 5 美元，命名为树莓派 Zero。
|
||||
|
||||
在 2016 年 2 月，树莓派 3 B 型发布，这也是现在可用的主要型号。在 2017 年，基金会发布了树莓派 Zero 的新型号树莓派 Zero W （W = wireless 无线）。
|
||||
在 2016 年 2 月，树莓派 3B 型发布，这也是现在可用的主要型号。在 2017 年，基金会发布了树莓派 Zero 的新型号树莓派 Zero W （W = wireless 无线）。
|
||||
|
||||
在不久的将来，一个提高技术规格的型号将会到来，为嵌入式系统发烧友、研究员、爱好者和工程师们通过多功能的方式来开发实时应用提供一个稳健的平台。
|
||||
在不久的将来，一个提高了技术规格的型号将会到来，为嵌入式系统发烧友、研究员、爱好者和工程师们用其开发多种功能的实时应用提供一个稳健的平台。
|
||||
|
||||
[][3]
|
||||
![][3]
|
||||
|
||||
图 1 ：树莓派
|
||||
*图 1 ：树莓派*
|
||||
|
||||
**树莓派作为一个高效的编程设备**
|
||||
### 树莓派是一个高效的编程设备
|
||||
|
||||
在给树莓派供电后，启动运行 LXDE 窗口管理器，用户会获得一个完整的基于 Debian 的 Linux 操作系统，即 Raspbian。Raspbian 操作系统为用户提供了众多免费的开源程序，涵盖了程序设计、游戏、应用以及教育方面。
|
||||
在给树莓派供电后，启动运行 LXDE 窗口管理器，用户会获得一个完整的基于 Debian 的 Linux 操作系统，即 Raspbian。Raspbian 操作系统为用户提供了众多自由开源的程序，涵盖了程序设计、游戏、应用以及教育方面。
|
||||
|
||||
树莓派的官方编程语言是 Python ，并已预装在了 Paspbian 操作系统上。结合树莓派和 Python 的集成开发环境 IDLE3 ，让程序员能够开发各种基于 Python 的程序。
|
||||
树莓派的官方编程语言是 Python ，并已预装在了 Paspbian 操作系统上。结合树莓派和 Python 的集成开发环境 IDLE3 ，可以让程序员能够开发各种基于 Python 的程序。
|
||||
|
||||
除了 Python ，树莓派还支持多种其他语言。并且可以使用一些免费的开源 IDE （集成开发环境）。允许程序员、开发者和应用工程师在树莓派上开发程序和应用。
|
||||
除了 Python ，树莓派还支持多种其它语言。并且可以使用一些自由开源的 IDE （集成开发环境）。允许程序员、开发者和应用工程师在树莓派上开发程序和应用。
|
||||
|
||||
[][4]
|
||||
### 树莓派上的最佳 IDE
|
||||
|
||||
图 2 ：BlueJ 的 GUI 界面
|
||||
作为一名程序员和开发者，你需要的首先就是有一个 IDE ，这是一个集成了开发者和程序员编写、编译和测试软件所需的的基本工具的综合软件套件。IDE 包含了代码编辑器、编译或解释程序和调试器，并允许开发者通过一个图形用户界面（GUI）来访问。IDE 的主要目的之一是提供一个整合单元来统一功能设置，减少组合多个开发工具的必要配置。
|
||||
|
||||
[][5]
|
||||
|
||||
图 3 ： Geany IDE 的 GUI 界面
|
||||
|
||||
**树莓派上的最佳 IDE**
|
||||
|
||||
作为一名程序员和开发者，你需要的第一件事就是有一个 IDE ，这是一个集成了开发者和程序员编写、编译和测试软件所需的的基本工具的综合软件套件。IDE 包含了一个代码编辑器，一个编译或解释程序和一个调试器，并允开发者通过一个图形用户界面（GUI）来访问。IDE 的主要目的之一是提供一个整合单元来统一功能设置，减少组合多个开发工具的必要配置。
|
||||
|
||||
IDE 的用户界面与文字处理程序相似，在工具栏提供颜色编码、源代码格式化、错误诊断、报告以及智能代码补全工具。IDE 被设计用来整合第三方版本控制库如 GitHub 或 Apache Subversion 。一些 IDE 专注于特定的编程语言，允许一个匹配编程语言的功能集，当然也有一些是支持多种语言的。
|
||||
IDE 的用户界面与文字处理程序相似，在工具栏提供颜色编码、源代码格式化、错误诊断、报告以及智能代码补全工具。IDE 被设计用来整合第三方版本控制库如 GitHub 或 Apache Subversion 。一些 IDE 专注于特定的编程语言，支持一个匹配该编程语言的功能集，当然也有一些是支持多种语言的。
|
||||
|
||||
树莓派上拥有丰富的 IDE ，为程序员提供友好界面来开发源代码、应用程序以及系统程序。
|
||||
|
||||
就让我们来探索最适合树莓派的 IDE 吧。
|
||||
|
||||
[][6]
|
||||
#### BlueJ
|
||||
|
||||
图 4 ：Adafruit WebIDE 的 GUI 界面
|
||||
![][4]
|
||||
|
||||
**BlueJ**
|
||||
*图 2 ：BlueJ 的 GUI 界面*
|
||||
|
||||
BlueJ 是一款致力于 Java 编程语言的 IDE ，主要是为教育目的而开发的。它也支持小型的软件开发项目。BlueJ 由澳大利亚的莫纳什大学的 Michael Kolling 和 John Rosenburg 在 2000 年作为 Blue 系统的继任者而开发的，后来在 2009 年 3 月成为自由开源软件。
|
||||
|
||||
BlueJ 提供一种高效的方式来学习面向对象的编程概念，图形用户界面为应用程序提供像 UML 图一样的类结构。每一个像类、对象和函数调用这样基于 OOPS 的概念，都可以通过基于交互的设计来表示。
|
||||
BlueJ 提供一种学习面向对象的编程概念的高效的方式，图形用户界面为应用程序提供像 UML 图一样的类结构。每一个像类、对象和函数调用这样基于 OOPS 的概念，都可以通过基于交互的设计来表示。
|
||||
|
||||
_**特性**_
|
||||
**特性：**
|
||||
|
||||
* _简单的交互界面:_ 与 NetBeans 或 Eclipse 这样的专业界面相比，BlueJ 的用户界面更加简易学。使开发者可以专注于编程而不是环境。
|
||||
|
||||
* _便携:_ BlueJ 支持多种平台如 Windows、Linux 以及 Mac OS X ， 可以免安装直接运行。
|
||||
|
||||
* _新的创新:_ BlueJ IDE 在对象工作台、代码块和范围着色方面有着大量的创新，使新手体验到开发的乐趣。
|
||||
|
||||
* _强大的技术支持:_ BlueJ 拥有一个核心功能团队来解答疑问，并且在 24 小时内为开发者的各种问题提供解决方案。
|
||||
* _简单的交互界面：_ 与 NetBeans 或 Eclipse 这样的专业界面相比，BlueJ 的用户界面更加简易学。使开发者可以专注于编程而不是环境。
|
||||
* _便携：_ BlueJ 支持多种平台如 Windows、Linux 以及 Mac OS X ， 可以免安装直接运行。
|
||||
* _新的创新：_ BlueJ IDE 在对象工作台、代码块和范围着色方面有着大量的创新，使新手体验到开发的乐趣。
|
||||
* _强大的技术支持：_ BlueJ 拥有一个核心功能团队来解答疑问，并且在 24 小时内为开发者的各种问题提供解决方案。
|
||||
|
||||
**最新版本：** 4.0.1
|
||||
|
||||
**Geany IDE**
|
||||
#### Geany IDE
|
||||
|
||||
Geany IDE 使用了 Scintilla 和 GTK+ 的集成开发环境支持，被认为是一个非常轻量级的基于 GUI 的文本编辑器。 Geany 的独特之处在于它被设计为独立于特殊的桌面环境并且仅需要较少数量的依赖包。只需要 GTK2 运行库就可以运行。Geany IDE 支持多种编程语言如 C、C++、C#、Java、HTML、PHP、Python、Perl、Ruby、Erlang 和 LaTeX 。
|
||||
![][5]
|
||||
|
||||
_**特性**_
|
||||
*图 3 ： Geany IDE 的 GUI 界面*
|
||||
|
||||
Geany IDE 使用了 Scintilla 和 GTK+ 的集成开发环境支持，被认为是一个非常轻量级的基于 GUI 的文本编辑器。 Geany 的独特之处在于它被设计为独立于特定的桌面环境，并且仅需要较少数量的依赖包。只需要 GTK2 运行库就可以运行。Geany IDE 支持多种编程语言如 C、C++、C#、Java、HTML、PHP、Python、Perl、Ruby、Erlang 和 LaTeX 。
|
||||
|
||||
**特性：**
|
||||
|
||||
* 代码自动补全和简单的代码导航。
|
||||
|
||||
* 高效的语法高亮和代码折叠.
|
||||
|
||||
* 高效的语法高亮和代码折叠。
|
||||
* 支持嵌入式终端仿真器，拥有高度可扩展性，可以免费下载大量功能丰富的插件。
|
||||
|
||||
* 简单的项目管理并支持多种文件类型，包括 C、Java、PHP、HTML、Python、Perl 等。
|
||||
|
||||
* 高度定制的界面，可以添加或删除设置、栏及窗口。
|
||||
|
||||
**最新版本：** 1.30.1
|
||||
|
||||
[][7]
|
||||
#### Adafruit WebIDE
|
||||
|
||||
图 5 ：AlgoIDE 的 GUI 界面
|
||||
![][6]
|
||||
|
||||
[][8]
|
||||
|
||||
图 6 ：Ninja IDE 的 GUI 界面
|
||||
|
||||
**Adafruit WebIDE**
|
||||
*图 4 ：Adafruit WebIDE 的 GUI 界面*
|
||||
|
||||
Adafruit WebIDE 为树莓派用户提供一个基于 Web 的界面来执行编程功能，并且允许开发者编译多种语言的源代码如 Python、Ruby、JavaScript 等。
|
||||
|
||||
Adafruit IDE 允许开发者把代码放在 GIT 仓库，这样就可以通过 GitHub 在任何地方进行访问。
|
||||
|
||||
_**特性**_
|
||||
**特性：**
|
||||
|
||||
* 可以通过 Web 浏览器的 8080 端口或 80 端口进行访问。
|
||||
|
||||
* 支持源代码的简单编译和运行。
|
||||
|
||||
* 配备一个调试器和可视器来进行正确追踪，代码导航以及测试源代码。
|
||||
|
||||
**AlgoIDE**
|
||||
#### AlgoIDE
|
||||
|
||||
![][7]
|
||||
|
||||
*图 5 ：AlgoIDE 的 GUI 界面*
|
||||
|
||||
AlgoIDE 结合了一个脚本语言和一个 IDE 环境，它被设计用来将编程与下一步的示例一起来运行。AlgoIDE 包含了一个强大的调试器、 实时范围管理器并且一步一步的执行代码。针对全年龄人群而设计，用来设计程序以及对算法进行大量的研究。
|
||||
|
||||
AlgoIDE 支持多种类型的语言如 C、C++、Python、Java、Smalltalk、Objective C、ActionScript 等。
|
||||
|
||||
_**特性**_
|
||||
**特性：**
|
||||
|
||||
* 代码自动缩进和补全。
|
||||
|
||||
* 高效的语法高亮和错误管理。
|
||||
|
||||
* 包含了一个调试器、范围管理器和动态帮助系统。
|
||||
|
||||
* 支持 GUI 和传统的 Logo 程序语言 Turtle 来进行源代码开发。
|
||||
|
||||
**最新版本：** 2016-12-08 （上次更新时间）
|
||||
|
||||
**Ninja IDE**
|
||||
#### Ninja IDE
|
||||
|
||||
![][8]
|
||||
|
||||
图 6 ：Ninja IDE 的 GUI 界面
|
||||
|
||||
Ninja IDE （“Ninja-IDE Is Not Just Another IDE”的缩写），由 Diego Sarmentero 、Horacio Duranm Gabriel Acosta 、Pedro Mourelle 和 Jose Rostango 设计，使用纯 Python 编写并且支持多种平台运行如 Linux 、Mac OS X 和 Windows 。Ninja IDE 被认为是一个跨平台的 IDE 软件，尤其是用来设计基于 Python 的应用程序。
|
||||
|
||||
Ninja IDE 是非常轻量级的并能执行多种功能如文件处理、代码定位、跳转行、标签、代码自动缩进和编辑器缩放。除了 Python ，这款 IDE 也支持几种其他语言。
|
||||
Ninja IDE 是非常轻量级的，并能执行多种功能如文件处理、代码定位、跳转行、标签、代码自动缩进和编辑器缩放。除了 Python ，这款 IDE 也支持几种其他语言。
|
||||
|
||||
_**特性**_
|
||||
|
||||
* _高效的代码编辑器:_ Ninja-IDE 被认为是最有效的代码编辑器，因为它能执行多种功能如代码补全和缩进，以及助手功能。
|
||||
|
||||
* _错误和 PEP8 查找器:_ 高亮显示文件中的静态和 PEP8 错误。
|
||||
|
||||
* _代码定位器:_ 使用此功能，快速直接访问能够访问的文件。用户可以使用快捷键 “CTRL+K” 进行输入，IDE 会找到特定的文本。
|
||||
**特性：**
|
||||
|
||||
* _高效的代码编辑器：_ Ninja-IDE 被认为是最有效的代码编辑器，因为它能执行多种功能如代码补全和缩进，以及助手功能。
|
||||
* _错误和 PEP8 查找器：_ 高亮显示文件中的静态和 PEP8 错误。
|
||||
* _代码定位器：_ 使用此功能，快速直接访问能够访问的文件。用户可以使用快捷键 “CTRL+K” 进行输入，IDE 会找到特定的文本。
|
||||
* 独特的项目管理功能以及大量的插件使得具有 Ninja-IDE 高度可扩展性。
|
||||
|
||||
**最新版本：** 2.3
|
||||
|
||||
[][9]
|
||||
#### Lazarus IDE
|
||||
|
||||
![][9]
|
||||
|
||||
图 7 ：Lazarus IDE 的 GUI 界面
|
||||
|
||||
**Lazarus IDE**
|
||||
|
||||
Lazarus IDE 是由 Cliff Baeseman、Shane Miller 和 Michael A. Hess 于 1999 年 2 月 开发。它被视为是一款用于应用程序快速开发的基于 GUI 的跨平台 IDE ，使用的是 Free Pascal 编译器。Lazarus IDE 继承了 Free Pascal 的三个主要特性 —— 编译速度、执行速度和交叉编译。可以在多种操作系统上对应用程序进行交叉编译，如 Windows 、Linux 、Mac OS X 等。
|
||||
|
||||
这款 IDE 由 Lazarus 组件库组成。这些组件库以一个单一和带有不同的特定平台实现的统一接口的形式为开发者提供了多种配套设施。它支持“一次编写，随处编译”的原则。
|
||||
|
||||
_**特性**_
|
||||
**特性：**
|
||||
|
||||
* 强大而快速的处理各种类型的源代码，同时支持性能测试。
|
||||
|
||||
* 易用的 GUI ，支持组件拖拽功能。可以通过 Lazarus 包文件为 IDE 添加附加组件。
|
||||
|
||||
* 使用新功能加强的 Free Pascal ，可以用来开发 Android 应用。
|
||||
|
||||
* 高可扩展性、开放源代码并支持多种框架来编译其他语言。
|
||||
|
||||
**最新版本：** 1.6.4
|
||||
|
||||
**Codeblock IDE**
|
||||
#### Codeblock IDE
|
||||
|
||||
Codeblock IDE 是用 C++ 编写的，使用了 wxWidgets 作为 GUI 库，发布于 2005 年。它是一款免费、开源、跨平台的 IDE ，支持多种类型的编译器如 GCC 、Clang 和 Visual C++ 。
|
||||
![][10]
|
||||
|
||||
*图 8 ： Codeblock IDE 界面*
|
||||
|
||||
Codeblock IDE 是用 C++ 编写的，使用了 wxWidgets 作为 GUI 库，发布于 2005 年。它是一款自由开源、跨平台的 IDE ，支持多种类型的编译器如 GCC 、Clang 和 Visual C++ 。
|
||||
|
||||
Codeblock IDE 高度智能并且可以支持多种功能，如语法高亮、代码折叠、代码补全和缩进，同时也拥有一些扩展插件来进行定制。它可以在 Windows 、Mac OS X 和 Linux 操作系统上运行。
|
||||
|
||||
_**特性**_
|
||||
**特性：**
|
||||
|
||||
* 支持多种类型的编译器如 GCC 、Visual C++ 、Borland C++ 、Watcom 、Intel C++ 等。主要针对 C++ 而设计，不过现在也支持其他的一些语言。
|
||||
|
||||
* 智能的调试器，允许用户通过访问本地函数符号和参数显示，用户自定义监视、调用堆栈、自定义内存转储、线程切换以及 GNU 调试接口调试程序。
|
||||
|
||||
* 支持多种功能用来从 Dev-C++ 、Visual C++ 等平台迁移代码。
|
||||
|
||||
* 使用自定义系统和 XML 扩展文件来存储信息。
|
||||
|
||||
**最新版本：** 16.01
|
||||
|
||||
[][10]
|
||||
#### Greenfoot IDE
|
||||
|
||||
图 8 ： Codeblock IDE 界面
|
||||
![][11]
|
||||
|
||||
[][11]
|
||||
|
||||
图 9 ： Greenfoot IDE 界面
|
||||
|
||||
**Greenfoot IDE**
|
||||
*图 9 ： Greenfoot IDE 界面*
|
||||
|
||||
Greenfoot IDE 是由肯特大学的 Michael Kolling 设计。它是一款基于 Java 的跨平台 IDE ，针对中学和大学教育目的而设计。Greenfoot IDE 的功能有项目管理、代码自动补全、语法高亮并提供一个简易的 GUI 界面。
|
||||
|
||||
Greenfoot IDE 编程包括两个主类的子类 —— World 和 Actor 。 World 表示主要执行发生的类，Actors 是已经存在且活动于 World 中的对象。
|
||||
|
||||
_**特性**_
|
||||
**特性：**
|
||||
|
||||
* 简单易用的 GUI ，比 BlueJ 和其他的 IDE 交互性更强。
|
||||
|
||||
* 易于新手和初学者上手。
|
||||
|
||||
* 在执行 Java 代码方面非常强大。
|
||||
|
||||
* 支持 GNOME/KDE/X11 图形环境。
|
||||
|
||||
* 其他功能包括项目管理、自动补全、语法高亮以及错误自动校正。
|
||||
|
||||
**最新版本：** 3.1.0
|
||||
@@ -221,9 +196,9 @@ Anand Nayyar
|
||||
|
||||
via: http://opensourceforu.com/2017/06/top-ides-raspberry-pi/
|
||||
|
||||
作者：[Anand Nayyar ][a]
|
||||
作者：[Anand Nayyar][a]
|
||||
译者：[softpaopao](https://github.com/softpaopao)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
校对：[wxy](https://github.com/wxy)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
@@ -1,45 +1,32 @@
|
||||
开发一个 Linux 调试器（七）：源码层断点
|
||||
开发一个 Linux 调试器（七）：源码级断点
|
||||
============================================================
|
||||
|
||||
在内存地址上设置断点是可以的，但它没有提供最方便用户的工具。我们希望能够在源代码行和函数入口地址上设置断点，以便我们可以在与代码相同的抽象级中别进行调试。
|
||||
在内存地址上设置断点虽然不错，但它并没有提供最方便用户的工具。我们希望能够在源代码行和函数入口地址上设置断点，以便我们可以在与代码相同的抽象级别中进行调试。
|
||||
|
||||
这篇文章将会添加源码层断点到我们的调试器中。通过所有我们已经支持的，这比起最初听起来容易得多。我们还将添加一个命令来获取符号的类型和地址，这对于定位代码或数据以及理解链接概念非常有用。
|
||||
|
||||
* * *
|
||||
这篇文章将会添加源码级断点到我们的调试器中。通过所有我们已经支持的功能，这要比起最初听起来容易得多。我们还将添加一个命令来获取符号的类型和地址，这对于定位代码或数据以及理解链接概念非常有用。
|
||||
|
||||
### 系列索引
|
||||
|
||||
随着后面文章的发布，这些链接会逐渐生效。
|
||||
|
||||
1. [准备环境][1]
|
||||
|
||||
2. [断点][2]
|
||||
|
||||
3. [寄存器和内存][3]
|
||||
|
||||
4. [Elves 和 dwarves][4]
|
||||
|
||||
5. [源码和信号][5]
|
||||
|
||||
6. [源码层逐步执行][6]
|
||||
|
||||
7. [源码层断点][7]
|
||||
|
||||
6. [源码级逐步执行][6]
|
||||
7. [源码级断点][7]
|
||||
8. [调用栈][8]
|
||||
|
||||
9. 读取变量
|
||||
|
||||
10. 之后步骤
|
||||
|
||||
* * *
|
||||
|
||||
### 断点
|
||||
|
||||
### DWARF
|
||||
#### DWARF
|
||||
|
||||
[Elves 和 dwarves][9] 这篇文章，描述了 DWARF 调试信息是如何工作的，以及如何用它来将机器码映射到高层源码中。回想一下，DWARF 包含函数的地址范围和一个允许你在抽象层之间转换代码位置的行表。我们将使用这些功能来实现我们的断点。
|
||||
[Elves 和 dwarves][4] 这篇文章，描述了 DWARF 调试信息是如何工作的，以及如何用它来将机器码映射到高层源码中。回想一下，DWARF 包含了函数的地址范围和一个允许你在抽象层之间转换代码位置的行表。我们将使用这些功能来实现我们的断点。
|
||||
|
||||
### 函数入口
|
||||
#### 函数入口
|
||||
|
||||
如果你考虑重载、成员函数等等，那么在函数名上设置断点可能有点复杂，但是我们将遍历所有的编译单元，并搜索与我们正在寻找的名称匹配的函数。DWARF 信息如下所示：
|
||||
|
||||
@@ -85,13 +72,13 @@ void debugger::set_breakpoint_at_function(const std::string& name) {
|
||||
}
|
||||
```
|
||||
|
||||
这代码看起来有点奇怪的唯一一点是 `++entry`。 问题是函数的 `DW_AT_low_pc` 不指向该函数的用户代码的起始地址，它指向 prologue 的开始。编译器通常会输出一个函数的 prologue 和 epilogue，它们用于执行保存和恢复堆栈、操作堆栈指针等。这对我们来说不是很有用，所以我们将入口行加一来获取用户代码的第一行而不是 prologue。DWARF 行表实际上具有一些功能，用于将入口标记为函数 prologue 之后的第一行，但并不是所有编译器都输出该函数，因此我采用了原始的方法。
|
||||
这代码看起来有点奇怪的唯一一点是 `++entry`。 问题是函数的 `DW_AT_low_pc` 不指向该函数的用户代码的起始地址，它指向 prologue 的开始。编译器通常会输出一个函数的 prologue 和 epilogue，它们用于执行保存和恢复堆栈、操作堆栈指针等。这对我们来说不是很有用，所以我们将入口行加一来获取用户代码的第一行而不是 prologue。DWARF 行表实际上具有一些功能，用于将入口标记为函数 prologue 之后的第一行，但并不是所有编译器都输出它，因此我采用了原始的方法。
|
||||
|
||||
### 源码行
|
||||
#### 源码行
|
||||
|
||||
要在高层源码行上设置一个断点，我们要将这个行号转换成 DWARF 中的一个地址。我们将遍历编译单元，寻找一个名称与给定文件匹配的编译单元，然后查找与给定行对应的入口。
|
||||
|
||||
DWARF 看山去有点像这样：
|
||||
DWARF 看上去有点像这样：
|
||||
|
||||
```
|
||||
.debug_line: line number info for a single cu
|
||||
@@ -119,7 +106,7 @@ IS=val ISA number, DI=val discriminator value
|
||||
|
||||
```
|
||||
|
||||
所以如果我们想要在 `ab.cpp` 的第五行设置一个断点，我们查找与行 (`0x004004e3`) 相关的入口并设置一个断点。
|
||||
所以如果我们想要在 `ab.cpp` 的第五行设置一个断点，我们将查找与行 (`0x004004e3`) 相关的入口并设置一个断点。
|
||||
|
||||
```
|
||||
void debugger::set_breakpoint_at_source_line(const std::string& file, unsigned line) {
|
||||
@@ -138,13 +125,11 @@ void debugger::set_breakpoint_at_source_line(const std::string& file, unsigned l
|
||||
}
|
||||
```
|
||||
|
||||
我这里的 `is_suffix` hack，这样你可以为 `a/b/c.cpp` 输入 `c.cpp`。当然你应该使用大小写敏感路径处理库或者其他东西。我很懒。`entry.is_stmt` 是检查行表入口是否被标记为一个语句的开头，这是由编译器根据它认为是断点的最佳目标的地址设置的。
|
||||
|
||||
* * *
|
||||
我这里做了 `is_suffix` hack，这样你可以输入 `c.cpp` 代表 `a/b/c.cpp` 。当然你实际上应该使用大小写敏感路径处理库或者其它东西，但是我比较懒。`entry.is_stmt` 是检查行表入口是否被标记为一个语句的开头，这是由编译器根据它认为是断点的最佳目标的地址设置的。
|
||||
|
||||
### 符号查找
|
||||
|
||||
当我们在对象文件层时，符号是王者。函数用符号命名，全局变量用符号命名，得到一个符号，我们得到一个符号，每个人都得到一个符号。 在给定的对象文件中，一些符号可能引用其他对象文件或共享库，链接器将从符号引用创建一个可执行程序。
|
||||
当我们在对象文件层时，符号是王者。函数用符号命名，全局变量用符号命名，你得到一个符号，我们得到一个符号，每个人都得到一个符号。 在给定的对象文件中，一些符号可能引用其他对象文件或共享库，链接器将从符号引用创建一个可执行程序。
|
||||
|
||||
可以在正确命名的符号表中查找符号，它存储在二进制文件的 ELF 部分中。幸运的是，`libelfin` 有一个不错的接口来做这件事，所以我们不需要自己处理所有的 ELF 的事情。为了让你知道我们在处理什么，下面是一个二进制文件的 `.symtab` 部分的转储，它由 `readelf` 生成：
|
||||
|
||||
@@ -222,7 +207,7 @@ Num: Value Size Type Bind Vis Ndx Name
|
||||
|
||||
你可以在对象文件中看到用于设置环境的很多符号，最后还可以看到 `main` 符号。
|
||||
|
||||
我们对符号的类型、名称和值（地址）感兴趣。我们有一个 `symbol_type` 类型的枚举，并使用一个 `std::string` 作为名称，`std::uintptr_t` 作为地址：
|
||||
我们对符号的类型、名称和值（地址）感兴趣。我们有一个该类型的 `symbol_type` 枚举，并使用一个 `std::string` 作为名称，`std::uintptr_t` 作为地址：
|
||||
|
||||
```
|
||||
enum class symbol_type {
|
||||
@@ -265,7 +250,7 @@ symbol_type to_symbol_type(elf::stt sym) {
|
||||
};
|
||||
```
|
||||
|
||||
最后我们要查找符号。为了说明的目的，我循环查找符号表的 ELF 部分，然后收集我在其中找到的任意符号到 `std::vector` 中。更智能的实现将建立从名称到符号的映射，这样你只需要查看一次数据就行了。
|
||||
最后我们要查找符号。为了说明的目的，我循环查找符号表的 ELF 部分，然后收集我在其中找到的任意符号到 `std::vector` 中。更智能的实现可以建立从名称到符号的映射，这样你只需要查看一次数据就行了。
|
||||
|
||||
```
|
||||
std::vector<symbol> debugger::lookup_symbol(const std::string& name) {
|
||||
@@ -287,16 +272,12 @@ std::vector<symbol> debugger::lookup_symbol(const std::string& name) {
|
||||
}
|
||||
```
|
||||
|
||||
* * *
|
||||
|
||||
### 添加命令
|
||||
|
||||
一如往常，我们需要添加一些更多的命令来向用户暴露功能。对于断点，我使用 GDB 风格的接口，其中断点类型是通过你传递的参数推断的，而不用要求显式切换：
|
||||
|
||||
* `0x<hexadecimal>` -> 断点地址
|
||||
|
||||
* `<line>:<filename>` -> 断点行号
|
||||
|
||||
* `<anything else>` -> 断点函数名
|
||||
|
||||
```
|
||||
@@ -326,11 +307,9 @@ else if(is_prefix(command, "symbol")) {
|
||||
}
|
||||
```
|
||||
|
||||
* * *
|
||||
|
||||
### 测试一下
|
||||
|
||||
在一个简单的二进制文件上启动调试器，并设置源代码级别的断点。在一些 `foo` 上设置一个断点，看到我的调试器停在它上面是我这个项目最有价值的时刻之一。
|
||||
在一个简单的二进制文件上启动调试器，并设置源代码级别的断点。在一些 `foo` 函数上设置一个断点，看到我的调试器停在它上面是我这个项目最有价值的时刻之一。
|
||||
|
||||
符号查找可以通过在程序中添加一些函数或全局变量并查找它们的名称来进行测试。请注意，如果你正在编译 C++ 代码，你还需要考虑[名称重整][10]。
|
||||
|
||||
@@ -342,19 +321,19 @@ else if(is_prefix(command, "symbol")) {
|
||||
|
||||
via: https://blog.tartanllama.xyz/c++/2017/06/19/writing-a-linux-debugger-source-break/
|
||||
|
||||
作者：[Simon Brand ][a]
|
||||
作者：[Simon Brand][a]
|
||||
译者：[geekpi](https://github.com/geekpi)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
校对：[wxy](https://github.com/wxy)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://twitter.com/TartanLlama
|
||||
[1]:https://blog.tartanllama.xyz/c++/2017/03/21/writing-a-linux-debugger-setup/
|
||||
[2]:https://blog.tartanllama.xyz/c++/2017/03/24/writing-a-linux-debugger-breakpoints/
|
||||
[3]:https://blog.tartanllama.xyz/c++/2017/03/31/writing-a-linux-debugger-registers/
|
||||
[4]:https://blog.tartanllama.xyz/c++/2017/04/05/writing-a-linux-debugger-elf-dwarf/
|
||||
[5]:https://blog.tartanllama.xyz/c++/2017/04/24/writing-a-linux-debugger-source-signal/
|
||||
[6]:https://blog.tartanllama.xyz/c++/2017/05/06/writing-a-linux-debugger-dwarf-step/
|
||||
[1]:https://linux.cn/article-8626-1.html
|
||||
[2]:https://linux.cn/article-8645-1.html
|
||||
[3]:https://linux.cn/article-8663-1.html
|
||||
[4]:https://linux.cn/article-8719-1.html
|
||||
[5]:https://linux.cn/article-8812-1.html
|
||||
[6]:https://linux.cn/article-8813-1.html
|
||||
[7]:https://blog.tartanllama.xyz/c++/2017/06/19/writing-a-linux-debugger-source-break/
|
||||
[8]:https://blog.tartanllama.xyz/c++/2017/06/24/writing-a-linux-debugger-unwinding/
|
||||
[9]:https://blog.tartanllama.xyz/c++/2017/04/05/writing-a-linux-debugger-elf-dwarf/
|
||||
@@ -0,0 +1,150 @@
|
||||
18 个开源的项目本地化翻译工具
|
||||
============================================================
|
||||
|
||||
> <ruby>本地化<rt>Localization</rt></ruby>（L10N）在适应项目方面为世界各地的用户发挥着关键作用。
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
本地化在定制开源项目以适应世界各地用户的需求方面发挥着核心作用。 除了代码之外，语言翻译也是世界各地人们贡献和参与开源项目的主要方式之一。
|
||||
|
||||
有专门针对语言服务行业特有的工具（听到这件事是不是很惊讶？），这使得高品质的本地化过程可以很顺畅。 本地化工具的类别包括:
|
||||
|
||||
* 计算机辅助翻译工具（CAT）
|
||||
* 机器翻译引擎（MT）
|
||||
* 翻译管理系统（TMS）
|
||||
* 术语管理工具
|
||||
* 本地化自动化工具
|
||||
|
||||
这些工具的专有版本可能相当昂贵。一个 SDL Trados Studio （领先的 CAT 工具）的许可证可能要花费数千欧元，即使这样，它只能一个人使用，并且定制功能也是有限的（注意，它们的费用也很高）。开源项目希望本地化到多种语言，简化本地化过程，所以希望找到开源工具来节省资金，并可以通过定制获得所需的灵活性。我对许多开源本地化工具项目进行了深入的调查，以帮助您决定使用什么。
|
||||
|
||||
### 计算机辅助翻译工具（CAT）
|
||||
|
||||
|
||||
|
||||
*OmegaT CAT 工具。在这里您可以发现翻译记忆（模糊匹配）和术语回顾（术语表）特性。OmegaT 在 GPL v3 许可证之下发布。*
|
||||
|
||||
CAT 工具是语言服务行业的主要工具。 顾名思义，CAT 工具可以帮助翻译人员尽快完成翻译、双语审查和单语审查的任务，并通过重用翻译内容（也称为翻译记忆），达到尽可能高的一致性。 <ruby>翻译记忆<rt>translation memory</rt></ruby>和<ruby>术语回忆<rt>terminology recall</rt></ruby>是 CAT 工具的两个主要特性。它们能够使译者在新项目中重用以前项目中翻译的内容。这使得他们可以在较短的时间内翻译大量的文字，同时通过术语和风格的一致性保持较高水平的质量。这对于本地化特别方便，因为许多软件和 web UI 中的文本在平台和应用程序中通常是相同的。 尽管 CAT 工具是独立的软件，但需要翻译人员在本地使用它们并合并到中央存储库。
|
||||
|
||||
**可用工具：**
|
||||
|
||||
* [OmegaT][7]
|
||||
* [OmegaT+][8]
|
||||
* [OpenTM2][9]
|
||||
* [Anaphraseus][10]
|
||||
* [字幕翻译器][11]
|
||||
|
||||
### 机器翻译引擎（MT）
|
||||
|
||||
|
||||
|
||||
机器翻译引擎自动将文本从一种语言翻译到另一种语言。机器翻译引擎被分成三种主要的方法：基于规则、统计式和神经网络式（这是新技术）。最广泛的机器翻译引擎方法是统计式，简而言之，通过使用 [_n_-gram 模型][29] 对带注释的双语语料库数据进行统计分析，得出关于两种语言之间的相互关联性。当将新的源语言短语引入到引擎进行翻译时，它会在其分析的语料库数据中查找与目标语言产生统计相关的对等物。机器翻译引擎可以作为翻译人员的生产力辅助工具，将他们的主要任务从将源文本转换为目标文本，改变为对机器翻译引擎的目标语言输出结果的后期编辑。我不建议在本地化工作中使用原始的机器翻译引擎输出结果，但是如果您的社区接受了后期编辑的培训，那么机器翻译引擎可以成为一个有用的工具，帮助他们做出大量的贡献。
|
||||
|
||||
**可用工具：**
|
||||
|
||||
* [Apertium][12]
|
||||
* [Moses][13]
|
||||
|
||||
### 翻译管理系统（TMS）
|
||||
|
||||
|
||||
|
||||
*如上是 Mozilla 的 Pontoon 翻译管理系统用户界面。使用所见即所得编辑方式，您可以在上下文根据语境翻译内容，在翻译的同时保证质量。 Pontoon 在 BSD 3 句版许可证（新款或修订版）之下发布。*
|
||||
|
||||
翻译管理系统工具是基于 web 的平台，允许您管理本地化项目，并使翻译人员和审阅人员能够做他们最擅长的事情。 大多数翻译管理系统工具旨在通过包括版本控制系统（VCS）集成、云服务集成、项目报告以及标准的翻译记忆和术语回忆功能，实现本地化过程中的许多手工部分的自动化。这些工具最适合于社区本地化或翻译项目，因为它们允许大量的翻译人员和审阅人员为一个项目做出贡献。一些人还使用所见即所得编辑器为他们的翻译者提供翻译语境。这种增加的语境可以提高翻译的准确性，减少译者在用户界面里翻译和审查翻译之间需要等待的时间。
|
||||
|
||||
**可用工具：**
|
||||
|
||||
* [Pontoon][14]
|
||||
* [Pootle][15]
|
||||
* [Weblate][16]
|
||||
* [Translate5][17]
|
||||
* [GlobalSight][18]
|
||||
* [Zanata][19]
|
||||
* [Jabylon][20]
|
||||
|
||||
### 术语管理工具
|
||||
|
||||
|
||||
|
||||
*杨百翰大学 (Brigham Young University) 的 BaseTerm 工具显示了新术语条目的对话窗口。 BaseTerm 在 Eclipse 公共许可证之下发布。*
|
||||
|
||||
术语管理工具为您提供 GUI 来创建术语资源（称为术语库）以添加语境并确保翻译的一致性。这些资源在帮助翻译人员的翻译过程中用于 CAT 工具和 TMS 平台。 对于一个术语基于语境可以是名词或动词的语言，术语管理工具允许您添加标记其词性、方言、单语定义以及上下文线索的术语元数据。 术语管理通常是本地化过程中使用不多的部分，但也是同样重要的部分。 在开源软件和专有软件的生态系统中，只有少量的可选产品。
|
||||
|
||||
**查看工具**
|
||||
|
||||
* [BaseTerm][21]
|
||||
* [Terminator][22]
|
||||
|
||||
### 自动本地化工具
|
||||
|
||||
|
||||
|
||||
*Okapi 框架的 Ratel 和 Rainbow 组件。 图片由 Okapi 框架提供。Okapi 框架在 Apache 许可证 2.0 之下发布。*
|
||||
|
||||
自动本地化工具便于您处理本地化数据。这可以包括文本提取、文件格式转换、标记化、VCS 同步、术语提取、预翻译和对通用的本地化标准文件格式的各种质量检查。在一些工具套件中，如 Okapi 框架，您可以创建用于执行各种本地化任务的自动化流程。这对于各种情况都非常有用，但是它们的主要功能是通过自动化许多任务来节省时间。它们还可以让你更接近一个根据连续的本地化流程。
|
||||
|
||||
**查看工具**
|
||||
|
||||
* [Okapi Framework][23]
|
||||
* [Mojito][24]
|
||||
|
||||
### 为什么开源是关键
|
||||
|
||||
本地化在开源时是最强力有效的。 这些工具应该让您和您的社区能够将您的项目本地化为尽可能多的语言。
|
||||
|
||||
想了解更多吗? 看看这些附加资源：
|
||||
|
||||
* [自由/开源的机器翻译软件][25] 列表
|
||||
* _[开放翻译工具][5]_ 电子书
|
||||
|
||||
（题图： opensource.com）
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
作者简介：
|
||||
|
||||
Jeff Beatty - Jeff Beatty 是 Mozilla 公司本地化的负责人， Mozilla 是流行的开源 web 浏览器 Firefox 的制造商。 他拥有利默里克大学（University of Limerick）多语言计算和本地化专业硕士学位。 Jeff 还在全球知名刊物中担任本地化专家，如《经济学人》 (The Economist)、《世界报》 (El Universal)、多语种杂志等。 Jeff 旨在展示 Mozilla 的本地化程序，创建颠覆性的开源翻译技术，并充当传播桥梁。
|
||||
|
||||
-----------
|
||||
|
||||
via: https://opensource.com/article/17/6/open-source-localization-tools
|
||||
|
||||
作者：[Jeff Beatty][a]
|
||||
译者：[TimeBear](https://github.com/TimeBear)
|
||||
校对：[wxy](https://github.com/wxy)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://opensource.com/users/guerojeff
|
||||
[1]:https://opensource.com/file/357341
|
||||
[2]:https://opensource.com/file/357331
|
||||
[3]:https://opensource.com/file/357326
|
||||
[4]:https://opensource.com/file/357336
|
||||
[5]:https://booki.flossmanuals.net/open-translation-tools/index
|
||||
[6]:https://opensource.com/article/17/6/open-source-localization-tools?rate=QVxhNMYU2Rzrul0hmvzCTqOUinduQ8Au5L8sT8bVbFk
|
||||
[7]:http://www.omegat.org/
|
||||
[8]:http://omegatplus.sourceforge.net/
|
||||
[9]:http://opentm2.org/
|
||||
[10]:http://anaphraseus.sourceforge.net/
|
||||
[11]:http://www.mironto.sk/
|
||||
[12]:http://www.apertium.org/
|
||||
[13]:http://www.statmt.org/moses/
|
||||
[14]:http://pontoon.mozilla.org/
|
||||
[15]:http://pootle.translatehouse.org/
|
||||
[16]:https://weblate.org/
|
||||
[17]:http://translate5.net/
|
||||
[18]:http://www.globalsight.com/
|
||||
[19]:http://zanata.org/
|
||||
[20]:http://jabylon.org/
|
||||
[21]:http://certsoftadmin.byu.edu/baseterm/termbase/search_all
|
||||
[22]:https://github.com/translate/terminator
|
||||
[23]:http://okapiframework.org/
|
||||
[24]:http://www.mojito.global/
|
||||
[25]:http://fosmt.org/
|
||||
[26]:https://www.openwest.org/custom/description.php?id=156
|
||||
[27]:https://www.openwest.org/
|
||||
[28]:https://opensource.com/user/143806/feed
|
||||
[29]:https://en.wikipedia.org/wiki/N-gram#n-gram_models
|
||||
[30]:https://opensource.com/users/guerojeff
|
||||
71
published/20170629 Kubernetes Why does it matter.md
Normal file
71
published/20170629 Kubernetes Why does it matter.md
Normal file
@@ -0,0 +1,71 @@
|
||||
Kubernetes 为什么这么重要？
|
||||
============================================================
|
||||
|
||||
> 在开发和部署云原生应用程序时，运行容器化负载的 Kubernetes 平台起到了重大作用。
|
||||
|
||||
|
||||
|
||||
自然而然的，开发和部署云原生应用程序已经变得非常受欢迎。对于一个允许快速部署和连续交付的 bug 修复和新功能的流程来说，它有明显的优势，但是没有人会谈到鸡和鸡蛋问题：怎样才能达成这样的目的呢？从头开始构建基础设施和开发流程来开发和维护云原生应用程序是个不简单的、耗时的任务。
|
||||
|
||||
[Kubernetes][3] 是一个相对较新的运行容器化负载的平台，它解决了这些问题。它原本是 Google 内部的一个项目，Kubernetes 在 2015 年被捐赠给了[云原生计算基金会][4]，并吸引了来自世界各地开源社区的开发人员。 Kubernetes 的设计基于 Google 15 年的在生产和开发环境运维的经验。由于它是开源的，任何人都可以下载并使用它，并实现其带来的优势。
|
||||
|
||||
那么为什么 Kubernetes 会有这么大的惊喜呢？我认为它在像 OpenStack 这样的基础架构即服务（IaaS）和完整的平台即服务 （PaaS）的资源之间达到了最佳平衡，它的底层运行时实现完全由供应商控制。Kubernetes 提供了两个优势：对管理基础设施的抽象，以及深入裸机进行故障排除的工具和功能。
|
||||
|
||||
### IaaS 与 PaaS
|
||||
|
||||
OpenStack 被大多数人归类为 IaaS 解决方案，其中物理资源池（如处理器、网络和存储）在不同用户之间分配和共享。它使用传统的基于硬件的虚拟化实现用户之间的隔离。
|
||||
|
||||
OpenStack 的 REST API 允许使用代码自动创建基础架构，但是这就是问题所在。IaaS 产品输出的也是基础设施。其创建后，支持和管理那些更多的基础设施的服务方式并不多。在一定程度上，OpenStack 生产的底层基础架构（如服务器和 IP 地址）成为管理工作的重中之重。一个众所周知的结果是虚拟机（VM）的无序蔓延，而同样的情况也出现于网络、加密密钥和存储卷方面。这样，开发人员建立和维护应用程序的时间就更少了。
|
||||
|
||||
像其它基于集群的解决方案一样，Kubernetes 以单个服务器级别的方式运行，以实现水平缩放。它可以轻松添加新的服务器，并立即在新硬件上安排负载。类似地，当服务器没有被有效利用或需要维护时，可以从集群中删除服务器。其它 Kubernetes 可以自动处理的其他任务是编排活动，如工作调度、健康监测和维护高可用性。
|
||||
|
||||
网络是另一个可能难以在 IaaS 环境中可靠编排的领域。微服务之间通过 IP 地址通信可能是很棘手的。Kubernetes 实现了 IP 地址管理、负载均衡、服务发现和 DNS 名称注册，以在集群内提供无痛、透明的网络环境。
|
||||
|
||||
### 专为部署而设计
|
||||
|
||||
一旦创建了运行应用程序的环境，部署就是一件小事了。可靠地部署一个应用程序是说起来容易做起来难的任务 —— 它并不是最简单的。Kubernetes 相对其他环境的巨大优势是，部署是一等公民。
|
||||
|
||||
使用一个单独的 Kubernetes 命令行界面（CLI）的命令，可以描述应用程序并将其安装在群集上。Kubernetes 从初始部署、推出新版本以及（当一个关键功能出现问题时）进行回滚，实现了应用程序的整个生命周期。运行中的部署也可以暂停和恢复。拥有现成的、内置的工具和支持应用程序部署，而不用自己构建部署系统，这是不容小觑的优点。Kubernetes 用户既不必重新发明应用程序部署的轮子，也不会发现这是一项艰巨的任务。
|
||||
|
||||
Kubernetes 还可以监控运行中的部署的状态。虽然你可以在 IaaS 环境中像编写部署过程一样编写这个功能，但这是一个非常困难的任务，而这样的情况还比比皆是。
|
||||
|
||||
### 专为 DevOps 而设计
|
||||
|
||||
随着你在开发和部署 Kubernetes 应用程序方面获得更多经验，你将沿着与 Google 和其他前行者相同的路径前行。你将发现有几种 Kubernetes 功能对于多服务应用程序的有效开发和故障排除是非常重要的。
|
||||
|
||||
首先，Kubernetes 能够通过日志或 SSH（安全 shell）轻松检查正在运行的服务的能力非常重要。通过一条命令行调用，管理员可以检查在 Kubernetes 下运行的服务的日志。这可能听起来像一个简单的任务，但在 IaaS 环境中，除非你已经做了一些工作，否则这并不容易。大型应用程序通常具有专门用于日志收集和分析的硬件和人员。在Kubernetes 中的日志可能不能替代完整功能的日志和指标解决方案，但它足以提供基本的故障排除。
|
||||
|
||||
第二，Kubernetes 提供内置的密钥管理。从头开发过自己的部署系统的团队知道的另一个问题是，将敏感数据（如密码和 API 令牌）安全地部署到虚拟机上很困难。通过将密钥管理变成一等公民，Kubernetes 可以避免你的团队发明自己的不安全的、错误的密钥分发系统或在部署脚本中硬编码凭据。
|
||||
|
||||
最后，Kubernetes 有一些用于自动进行缩放、负载均衡和重新启动应用程序的功能。同样，这些功能是开发人员在使用 IaaS 或裸机时要自己编写的。你的 Kubernetes 应用程序的缩放和运行状况检查在服务定义中进行声明，而 Kubernetes 会确保正确数量的实例健康运行。
|
||||
|
||||
### 总结
|
||||
|
||||
IaaS 和 PaaS 系统之间的差异是巨大的，包括 PaaS 可以节省大量的开发和调试时间。作为一种 PaaS，Kubernetes 实现了强大而有效的功能，可帮助你开发、部署和调试云原生应用程序。它的架构和设计代表了数十年的难得的经验，而你的团队能够免费获得该优势。
|
||||
|
||||
|
||||
（题图：opensource.com）
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
作者简介：
|
||||
|
||||
Tim Potter - Tim 是 Hewlett Packard Enterprise 的高级软件工程师。近二十年来，他一直致力于自由和开源软件的开发工作，其中包括 Samba、Wireshark、OpenPegasus 和 Docker 等多个项目。Tim 博客在 https://elegantinfrastructure.com/ ，关于 Docker、Kubernetes 和其他基础设施相关主题。
|
||||
|
||||
-----
|
||||
|
||||
|
||||
via: https://opensource.com/article/17/6/introducing-kubernetes
|
||||
|
||||
作者：[Tim Potter][a]
|
||||
译者：[geekpi](https://github.com/geekpi)
|
||||
校对：[wxy](https://github.com/wxy)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://opensource.com/users/tpot
|
||||
[1]:https://opensource.com/article/17/6/introducing-kubernetes?rate=RPoUoHXYQXbTb7DHQCDsHgR1ZcfLSoquZ8xVZzfMtxM
|
||||
[2]:https://opensource.com/user/63281/feed
|
||||
[3]:https://kubernetes.io/
|
||||
[4]:https://www.cncf.io/
|
||||
[5]:https://opensource.com/users/tpot
|
||||
@@ -0,0 +1,58 @@
|
||||
IoT 边缘计算框架的新进展
|
||||
---
|
||||
|
||||
|
||||
|
||||
> 开源项目 EdgeX Foundry 旨在开发一个标准化的互操作物联网边缘计算框架。
|
||||
|
||||
4 月份时， Linux 基金组织[启动](http://linuxgizmos.com/open-source-group-focuses-on-industrial-iot-gateway-middleware/)了一个开源项目 [EdgeX Foundry](https://www.edgexfoundry.org/) ，用于为物联网边缘计算开发一个标准化互操作框架。 就在最近， EdgeX Foundry 又[宣布](https://www.edgexfoundry.org/announcement/2017/07/17/edgex-foundry-builds-momentum-for-a-iot-interoperability-and-a-unified-marketplace-with-eight-new-members/)新增了 8 个成员，其总成员达到 58 位。
|
||||
|
||||
这些新成员是 Absolute、IoT Impact LABS、inwinStack、Parallel Machines、Queen's University Belfast、RIOT、Toshiba Digital Solutions Corporation 和 Tulip Interfaces。 其原有成员包括 AMD、Analog Devices、Canonical/Ubuntu、Cloud Foundry、Dell、Linaro、Mocana、NetFoundry、 Opto 22、RFMicron 和 VMWare 等其他公司或组织。
|
||||
|
||||
EdgeX Foundry 项目构建于戴尔早期的基于 Apache2.0 协议的 [FUSE](https://medium.com/@gigastacey/dell-plans-an-open-source-iot-stack-3dde43f24feb) 物联网中间件框架之上，其中包括十几个微服务和超过 12.5 万行代码。在 FUSE 合并了类同项目 AllJoyn-compliant IoTX 之后，Linux 基金会协同 Dell 创立了 EdgeX Foundry ，后者是由 EdgeX Foundry 现有成员 Two Bulls 和 Beechwood 发起的项目。
|
||||
|
||||
EdgeX Foundry 将创造一个互操作性的、即插即用组件的物联网边缘计算的生态系统。开源的 EdgeX 栈将协调各种传感器网络协议与多种云平台及分析平台。该框架旨在充分挖掘横跨边缘计算、安全、系统管理和服务等模块间的互操作性代码。
|
||||
|
||||
对于项目成员及其客户来说，其关键的好处是在于能将各种预先认证的软件集成到许多 IoT 网关和智能边缘设备上。 在 Linux.com 的一次采访中，[IoT Impact LABS](https://iotimpactlabs.com/) 的首席工程师 Dan Mahoney 说：“现实中，EdgeX Foundry 降低了我们在部署多供应商解决方案时所面对的挑战。”
|
||||
|
||||
在 Linux 基金会仍然将其 AllSeen Alliance 项目下的 AllJoyn 规范合并到 [IoTivity](https://www.linux.com/news/how-iotivity-and-alljoyn-could-combine) 标准的情况下，为什么会发起了另外一个物联网标准化项目（EdgeX Foundry） 呢？ 原因之一，EdgeX Foundry 不同于 IoTivity，IoTivity 主要解决工业物联网问题，而 EdgeX Foundry 旨在解决消费级和工业级物联网全部的问题。 更具体来说， EdgeX Foundry 旨在成为网关和智能终端的通用中间件。 EdgeX Foundry 与 IoTivity 的另一个不同在于，前者希望借助预认证的终端塑造一种新产品，后者更多解决现存产品之间的互操作性。
|
||||
|
||||
Linux 基金会 IoT 高级总监 Philip DesAutels 说：“IoTivity 提供实现设备之间无缝连接的协议， 而 EdgeX Foundry 提供了一个边缘计算框架。EdgeX Foundry 能够兼容如 IoTivity、 BacNet、 EtherCat 等任何协议设备，从而实现集成多协议通信系统的通用边缘计算框架，该项目的目标是为构建互操作组件的生态系统的过程中，降低不确定性，缩短市场化时间，更好地产生规模效应。”
|
||||
|
||||
上个月, 由 [Open Connectivity Foundation](https://openconnectivity.org/developer/specifications/international-standards) （OCF）和 Linux 基金组织共同发起的 IoTivity 项目发布了 [IoTivity 1.3](https://wiki.iotivity.org/release_note_1.3.0)，该版本增加了与其曾经的对手 AllJoyn spec 的纽带，也增加了对于 OCF 的 UPnP 设备发现标准的接口。 预计在 [IoTivity 2.0](https://www.linux.com/news/iotivity-20-whats-store) 中， IoTivity 和 AllJoyn 将会更进一步深入集成。
|
||||
|
||||
DesAutels 告诉 linux.com，IoTivity 和 EdgeX 是“高度互补的”，其“原因是 EdgeX Foundry 项目的几个成员也是 IoTivity 或 OCF 的成员，如此更强化了 IoTivity 和 EdgeX 的合作关系。”
|
||||
|
||||
尽管 IoTivity 和 EdgeX 都宣称是跨平台的，包括在 CPU 架构和 OS 方面，但是二者还是存在一定区别。 IoTivity 最初是基于 Linux 平台设计，兼容 Ubuntu、Tizen 和 Android 等 Linux 系列 OS，后来逐步扩展到 Windows 和 iOS 操作系统。与之对应的 EdgeX 设计之初就是基于跨平台的理念，其完美兼容于各种 CPU 架构，支持 Linux, Windows 和 Mac OS 等操作系统， 未来还将兼容于实时操作系统（RTOS）。”
|
||||
|
||||
EdgeX 的新成员 [RIOT](https://riot-os.org/) 提供了一个开源的面向物联网的项目 RIOT RTOS。RIOT 的主要维护者 Thomas Eichinger 在一次表彰讲话中说：“由于 RIOT 初衷就是致力于解决 linux 不太适应的问题， 故对于 RIOT 社区来说，参加和支持类似于 EdgeX Foundry 等边缘计算的开源组织的积极性是自然而然的。”
|
||||
|
||||
### 传感器集成的简化
|
||||
|
||||
IoT Impact LABS （即 Impact LABS 或直接称为 LABS）是另一个 EdgeX 新成员。 该公司推出了一个独特的业务模式，旨在帮助中小企业度过物联网解决方案的试用阶段。该公司的大部分客户，其中包括几个 EdgeX Foundry 的项目成员，是致力于建设智慧城市、基础设施再利用、提高食品安全，以及解决社会面临的自然资源缺乏的挑战。
|
||||
|
||||
Dan Mahoney 说：“在 LABS 我们花费了很多时间来调和试点客户的解决方案之间的差异性。 EdgeX Foundry 可以最小化部署边缘软件系统的工作量，从而使我们能够更快更好地部署高质量的解决方案。”
|
||||
|
||||
该框架在涉及多个供应商、多种类型传感器的场景尤其凸显优势。“Edgex Foundry 将为我们提供快速构建可以控制所有部署的传感器的网关的能力。” Mahoney 补充说到。传感器制造商将借助 EdgeX SDK 烧写应用层协议驱动到边缘设备，该协议能够兼容多供应商和解决方案。
|
||||
|
||||
### 边缘分析能力的构建
|
||||
|
||||
当我们问到， Mahoney 的公司希望见到 EdgeX Foundry 怎样的发展时，他说：“我们喜见乐闻的一个目标是有更多有效的工业协议成为设备服务，这是一个更清晰的边缘计算实现之路。”
|
||||
|
||||
在工业物联网和消费级物联网中边缘计算都呈现增长趋势。 在后者，我们已经看到如 Alexa 的智能声控以及录像分析等几个智能家居系统[集成了边缘计算分析](https://www.linux.com/news/smart-linux-home-hubs-mix-iot-ai)技术。 这减轻了云服务平台的计算负荷，但同时也带来了安全、隐私，以及由于供应商中断或延迟问题引起的服务中断问题。
|
||||
|
||||
对于工业物联网网关，延迟问题成为首要的问题。因此，在物联网网关方面出现了一些类似于云服务功能的扩展。 其中一个解决方案是，为了安全将一些云服务上的安全保障应用借助容器如 [RIOS 与 Ubuntu 内核快照机制](https://www.linux.com/news/future-iot-containers-aim-solve-security-crisis)等方式集成到嵌入式设备。 另一种方案是，开发 IoT 生态系统，迁移云功能到边缘计算上。上个月，Amazon 为基于 linux 的网关发布了实现 [AWS Greengrass](http://linuxgizmos.com/amazon-releases-aws-greengrass-for-local-iot-processing-on-linux-devices/) 物联网协议栈的 AWS lambda。 该软件能够使 AWS 计算、消息路由、数据缓存和同步能力在诸如物联网网关等联网设备上完成。
|
||||
|
||||
分析能力是 EdgeX Foundry 发展路线上的一个关键功能要点。 发起成员之一 Cloud Foundry 其旨在集成其主要的工业应用平台到边缘设备。 另一个新成员 [Parallel Machines](https://www.parallelmachines.com/) 则计划利用 EdgeX 将 AI 带到边缘设备。
|
||||
|
||||
EdgeX Foundry 仍然在项目早期， 软件仍然在 α 阶段，其成员在上个月（六月份）才刚刚进行了第一次全体成员大会。同时该项目已经为新开发者准备了一些初始训练课程，另外从[这里](https://wiki.edgexfoundry.org/)也能获取更多的信息。
|
||||
|
||||
----
|
||||
|
||||
via: [https://www.linux.com/blog/2017/7/iot-framework-edge-computing-gains-ground](https://www.linux.com/blog/2017/7/iot-framework-edge-computing-gains-ground)
|
||||
|
||||
作者： [ERIC BROWN](https://www.linux.com/users/ericstephenbrown)
|
||||
译者：[penghuster](https://github.com/penghuster)
|
||||
校对：[wxy](https://github.com/wxy)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
@@ -0,0 +1,86 @@
|
||||
创建更好的灾难恢复计划
|
||||
============================================================
|
||||
|
||||
|
||||
|
||||
> Tanya Reilly 的五个问题：相互依赖的服务如何使恢复更加困难，为什么有意并预先管理依赖是个好主意。
|
||||
|
||||
我最近请 Google 的网站可靠性工程师 Tanya Reilly 分享了她关于如何制定更好的灾难恢复计划的想法。Tanya 将在 10 月 1 日到 4 日在纽约举行的 O'Reilly Velocity Conference 上发表了一个题为《[你有没有试着把它关闭之后再打开？] [9]》的演讲。
|
||||
|
||||
### 1、 在计划备份系统策略时，人们最常犯的错误是什么？
|
||||
|
||||
经典的一条是“**你不需要备份策略，你需要一个恢复策略**”。如果你有备份，但你尚未测试恢复它们，那么你没有真正的备份。测试不仅仅意味着知道你可以获得数据，还意味着知道如何把它放回数据库，如何处理增量更改，甚至如果你需要的话，如何重新安装整个系统。这意味着确保你的恢复路径不依赖于与数据同时丢失的某些系统。
|
||||
|
||||
但测试恢复是枯燥的。这是人们在忙碌时会偷工减料的那类事情。这值得花时间使其尽可能简单、无痛、自动化，永远不要靠任何人的意志力！同时，你必须确保有关人员知道该怎么做，所以定期进行大规模的灾难测试是很好的。恢复演练是个好方法，可以找出该过程的文档是否缺失或过期，或者你是否没有足够的资源（磁盘、网络等）来传输和重新插入数据。
|
||||
|
||||
### 2、 创建<ruby>灾难恢复<rt>disaster recovery</rt></ruby> （DR） 计划最常见的挑战是什么？
|
||||
|
||||
我认为很多 DR 是一种事后的想法：“我们有这个很棒的系统，我们的业务依赖它……我猜我们应该为它做 DR？”而且到那时，系统会非常复杂，充满相互依赖关系，很难复制。
|
||||
|
||||
第一次安装的东西，它通常是由人手动调整才正常工作的，有时那是个具体特定的版本。当你构建_第二_个时，很难确定它是完全一样的。即使在具有严格的配置管理的站点中，你也可能丢了某些东西，或者过期了。
|
||||
|
||||
例如，如果你已经失去对解密密钥的访问权限，那么加密备份没有太多用处。而且任何只在灾难中使用的部分都可能从你上次检查它们过后就破环了。确保你已经涵盖了所有东西的唯一方法做认真地故障切换。当你准备好了的，就计划一下你的灾难（演练）吧！
|
||||
|
||||
如果你可以设计系统，以使灾难恢复模式成为正常运行的一部分，那么情况会更好。如果你的服务从一开始就被设计为可复制的，添加更多的副本就是一个常规的操作并可能是自动化的。没有新的方法，这只是一个容量问题。但是，系统中仍然存在一些只能在一个或两个地方运行的组件。偶然计划中的假灾难能够很好地将它们暴露出来。
|
||||
|
||||
顺便说一句，那些被遗忘的组件可能包括仅在一个人的大脑中的信息，所以如果你自己发现说：“我们不能在 X 休假回来前进行 DR 故障切换测试”，那么那个人是一个危险的单点失败。
|
||||
|
||||
仅在灾难中使用的部分系统需要最多的测试，否则在需要时会失败。这个部分越少越安全，且辛苦的测试工作也越少。
|
||||
|
||||
### 3、 为什么服务相互依赖使得灾难恢复更加困难？
|
||||
|
||||
如果你只有一个二进制文件，那么恢复它是比较容易的：你做个二进制备份就行。但是我们越来越多地将通用功能分解成单独的服务。微服务意味着我们有更多的灵活性和更少地重新发明轮子：如果我们需要一个后端做一些事情，并且有一个已经存在，那么很好，我们就可以使用它。但是一些需要保留很大的依赖关系，因为它很快会变得纠缠。
|
||||
|
||||
你可能知道你直接使用的后端，但是你可能不会注意到有新的后端添加到你使用的库中。你可能依赖于某个东西，它也间接依赖于你。在依赖中断之后，你可能会遇到一个死锁：两个系统都不能启动，直到另一个运行并提供一些功能。这是一个困难的恢复情况！
|
||||
|
||||
你甚至可以最终遇到间接依赖于自身的东西，例如你需要配置启动网络的设备，但在网络关闭时无法访问该设备。人们通常会提前考虑这些循环依赖，并且有某种后备计划，但是这些本质上是不太行得通的方式：它们只适用于极端情况，并且以不同的方式使用你的系统、进程或代码。这意味着，它们很可能有一个不会被发现的问题，直到你真的，真的需要它们的工作的时候才发现。
|
||||
|
||||
### 4、 你建议人们在感觉需要之前就开始有意管理其依赖关系，以防止潜在的灾难性系统故障。为什么这很重要，你有什么建议有效地做到这一点？
|
||||
|
||||
管理你的依赖关系对于确保你可以从灾难中恢复至关重要。它使操作系统更容易。如果你的依赖不可靠，那么你就不可靠，所以你需要知道它们是什么。
|
||||
|
||||
虽然在它们变得混乱后也可以开始管理依赖关系，但是如果你早点开始，它会变得更容易一些。你可以设置使用各种服务策略——例如，你必须在堆栈中的这一层依赖于这组系统。你可以通过使其成为设计文件审查的常规部分，引入考虑依赖关系的习惯。但请记住，依赖关系列表将很快变得陈旧。如果你有程序化的发现依赖关系的方式，甚至强制实施依赖，这是最好的。 [我的 Velocity 谈话][10]涵盖了我们如何做到这一点。
|
||||
|
||||
早期开始的另一个优点是，你可以将服务拆分为垂直“层”，每个层中的功能必须能够在下一个层启动之前完全在线。所以，例如，你可以说网络必须能够完全启动而不借助任何其他服务。然后说，你的存储系统应该仅仅依赖于网络，程序后端应该仅仅依赖于网络和存储，等等。不同的层次对于不同的架构是有意义的。
|
||||
|
||||
如果你提前计划，新服务更容易选择依赖关系。每个服务应该只依赖堆栈中较低的服务。你仍然可以结束循环，在相同的层次服务上批次依赖 —— 但是它们可以更加紧密地包含，并且在逐个基础上处理更容易。
|
||||
|
||||
### 5、 你对 Velocity NY 的其他部分感兴趣么？
|
||||
|
||||
我整个星期二和星期三的时间表都完成了！正如你可能收集的那样，我非常关心大型相互依赖的系统的可管理性，所以我期待听到 [Carin Meier 关于管理系统复杂性的想法][11]、[Sarah Wells 的微服务][12]和 [Baron 的可观察性][13] 的谈话。我非常着迷听到 [Jon Moore 关于 Comcast 如何从年度发布到每天发布的故事][14]。作为一个前系统管理员，我很期待听到 [Bryan Liles 对这个职位走向的看法][15]。
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
作者简介：
|
||||
|
||||
Nikki McDonald 是 O'Reilly Media，Inc. 的内容总监。她住在密歇根州的安娜堡市。
|
||||
|
||||
Tanya Reilly 自 2005 年以来一直是 Google 的系统管理员和站点可靠性工程师，致力于分布式锁、负载均衡和引导等底层基础架构。在加入 Google 之前，她是爱尔兰最大的 ISP eircom.net 的系统管理员，在这之前她担当了一个小型软件公司的整个 IT 部门。
|
||||
|
||||
----------------------------
|
||||
|
||||
via: https://www.oreilly.com/ideas/creating-better-disaster-recovery-plans
|
||||
|
||||
作者：[Nikki McDonald][a], [Tanya Reilly][b]
|
||||
译者：[geekpi](https://github.com/geekpi)
|
||||
校对：[wxy](https://github.com/wxy)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://www.oreilly.com/people/nikki-mcdonald
|
||||
[b]:https://www.oreilly.com/people/5c97a-tanya-reilly
|
||||
[1]:https://pixabay.com/en/crane-baukran-load-crane-crane-arm-2436704/
|
||||
[2]:https://conferences.oreilly.com/velocity/vl-ny?intcmp=il-webops-confreg-reg-vlny17_new_site_right_rail_cta
|
||||
[3]:https://www.oreilly.com/people/nikki-mcdonald
|
||||
[4]:https://www.oreilly.com/people/5c97a-tanya-reilly
|
||||
[5]:https://conferences.oreilly.com/velocity/vl-ny?intcmp=il-webops-confreg-reg-vlny17_new_site_creating_better_disaster_recovery_plans_text_cta
|
||||
[6]:https://www.oreilly.com/ideas/creating-better-disaster-recovery-plans
|
||||
[7]:https://conferences.oreilly.com/velocity/vl-ny?intcmp=il-webops-confreg-reg-vlny17_new_site_right_rail_cta
|
||||
[8]:https://conferences.oreilly.com/velocity/vl-ny?intcmp=il-webops-confreg-reg-vlny17_new_site_right_rail_cta
|
||||
[9]:https://conferences.oreilly.com/velocity/vl-ny/public/schedule/detail/61400?intcmp=il-webops-confreg-reg-vlny17_new_site_creating_better_disaster_recovery_plans_body_text_cta
|
||||
[10]:https://conferences.oreilly.com/velocity/vl-ny/public/schedule/detail/61400?intcmp=il-webops-confreg-reg-vlny17_new_site_creating_better_disaster_recovery_plans_body_text_cta
|
||||
[11]:https://conferences.oreilly.com/velocity/vl-ny/public/schedule/detail/62779?intcmp=il-webops-confreg-reg-vlny17_new_site_creating_better_disaster_recovery_plans_body_text_cta
|
||||
[12]:https://conferences.oreilly.com/velocity/vl-ny/public/schedule/detail/61597?intcmp=il-webops-confreg-reg-vlny17_new_site_creating_better_disaster_recovery_plans_body_text_cta
|
||||
[13]:https://conferences.oreilly.com/velocity/vl-ny/public/schedule/detail/61630?intcmp=il-webops-confreg-reg-vlny17_new_site_creating_better_disaster_recovery_plans_body_text_cta
|
||||
[14]:https://conferences.oreilly.com/velocity/vl-ny/public/schedule/detail/62733?intcmp=il-webops-confreg-reg-vlny17_new_site_creating_better_disaster_recovery_plans_body_text_cta
|
||||
[15]:https://conferences.oreilly.com/velocity/vl-ny/public/schedule/detail/62893?intcmp=il-webops-confreg-reg-vlny17_new_site_creating_better_disaster_recovery_plans_body_text_cta
|
||||
@@ -3,55 +3,50 @@
|
||||
|
||||
|
||||
|
||||
作为一名开发者，我会尝试留意那些我可能不会每天使用的技术进步。了解这些技术至关重要，因为它们可能会间接影响到我的工作。比如[由 Docker 推动][8]的，近期正在兴起的容器化技术，可用于托管 Web 应用，并方便地调整其规模。从技术层面来讲，我并不是一个 DevOps，但当我每天构建 Web 应用时，多去留意这些技术如何去发展，会对我有所裨益。
|
||||
作为一名开发者，我会尝试留意那些我可能不会每天使用的技术的进步。了解这些技术至关重要，因为它们可能会间接影响到我的工作。比如[由 Docker 推动][8]的、近期正在兴起的容器化技术，可用于上规模地托管 Web 应用。从技术层面来讲，我并不是一个 DevOps，但当我每天构建 Web 应用时，多去留意这些技术如何去发展，会对我有所裨益。
|
||||
|
||||
这种进步的一个绝佳的例子，是近一段时间高速发展的容器编排平台。它允许你轻松地部署、管理容器化应用，并对它们的规模进行调整。目前看来，容器编排的流行工具有 [Kubernetes （来自 Google）][9]，[Docker Swarm][10] 和 [Apache Mesos][11]。如果你想较好的了解上面那些技术以及它们的区别，我推荐你看一下[这篇文章][12]。
|
||||
|
||||
在这片文章中，我们将会从一些简单的操作开始，看一看 Kubernetes 平台，将一个 WordPress 网站部署在本地机器上的一个单节点集群中。
|
||||
在这篇文章中，我们将会从一些简单的操作开始，了解一下 Kubernetes 平台，看看如何将一个 WordPress 网站部署在本地机器上的一个单节点集群中。
|
||||
|
||||
### 安装 Kubernetes
|
||||
|
||||
在 [Kubernetes 文档][13]中有一个很好的互动教程，涵盖了很多东西。但出于本文的目的，我只会介绍在 MacOS 中，Kuberentes 的安装和使用。
|
||||
在 [Kubernetes 文档][13]中有一个很好的互动教程，涵盖了很多东西。但出于本文的目的，我只会介绍在 MacOS 中 Kuberentes 的安装和使用。
|
||||
|
||||
我们要做的第一件事是在你的本地主机中安装 Kubernetes. 我们将使用一个叫做 [MiniKube][14] 的工具，它专门用于在你的机器上方便地设置一个 Kubernetes 集群用于测试。
|
||||
我们要做的第一件事是在你的本地主机中安装 Kubernetes。我们将使用一个叫做 [MiniKube][14] 的工具，它专门用于在你的机器上方便地设置一个用于测试的 Kubernetes 集群。
|
||||
|
||||
根据 Minikube 文档，在我们开始之前，有一些先决条件。首先要保证你已经安装了一个 Hypervisor （我将会使用 Virtualbox）。接下来，我们需要[安装 Kubernetes 命令行工具][15]（也就是 `kubectl`）。如果你在用 Homebrew，这一步非常简单，只需要运行命令：
|
||||
|
||||
```
|
||||
$ brew install kubectl
|
||||
|
||||
```
|
||||
|
||||
现在我们可以真正 [安装 Minikube][16] 了：
|
||||
|
||||
```
|
||||
$ curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.21.0/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
|
||||
|
||||
```
|
||||
|
||||
最后，我们想要[启动 Minicube][17]，创建一个虚拟机，来作为我们的单节点 Kubernetes 集群。现在我要说一点：尽管我们在本文中只在本地运行它，但是在[真正的服务器][18]上运行 Kubernetes 集群时，后面提到的大多数概念都会适用。在多节点集群上，“主节点”将负责管理其它工作节点（虚拟机或物理服务器），并且 Kubernetes 将会在集群中自动进行容器的分发和调度。
|
||||
最后，我们要[启动 Minicube][17] 创建一个虚拟机，来作为我们的单节点 Kubernetes 集群。现在我要说一点：尽管我们在本文中只在本地运行它，但是在[真正的服务器][18]上运行 Kubernetes 集群时，后面提到的大多数概念都会适用。在多节点集群上，“主节点”将负责管理其它工作节点（虚拟机或物理服务器），并且 Kubernetes 将会在集群中自动进行容器的分发和调度。
|
||||
|
||||
```
|
||||
$ minikube start --vm-driver=virtualbox
|
||||
|
||||
```
|
||||
|
||||
### 安装 Helm
|
||||
|
||||
现在，本机中应该有一个正在运行的（单节点）Kubernetes 集群了。我们现在可以用任何方式来与 Kubernetes 交互。如果你想现在可以玩它，我觉得 [kubernetesbyexample.com][19] 会很好地想你介绍 Kubernetes 的概念和术语。
|
||||
现在，本机中应该有一个正在运行的（单节点）Kubernetes 集群了。我们现在可以用任何方式来与 Kubernetes 交互。如果你想现在可以体验一下，我觉得 [kubernetesbyexample.com][19] 可以很好地向你介绍 Kubernetes 的概念和术语。
|
||||
|
||||
虽然我们可以手动配置这些东西，但实际上我们将会使用另外的工具，来将我们的 WordPress 应用部署到 Kubernetes 集群中。[Helm][20] 被称为“Kubernetes 的包管理工具”，它可以让你轻松地在你的集群中部署预构建的软件包，也就是“图表”。你可以把图表看做一组专为特定应用（如 WordPress）而设计的容器定义和配置。首先我们在本地主机上安装 Helm：
|
||||
虽然我们可以手动配置这些东西，但实际上我们将会使用另外的工具，来将我们的 WordPress 应用部署到 Kubernetes 集群中。[Helm][20] 被称为“Kubernetes 的包管理工具”，它可以让你轻松地在你的集群中部署预构建的软件包，也就是“<ruby>图表<rt>chart</rt></ruby>”。你可以把图表看做一组专为特定应用（如 WordPress）而设计的容器定义和配置。首先我们在本地主机上安装 Helm：
|
||||
|
||||
```
|
||||
$ brew install kubernetes-helm
|
||||
|
||||
```
|
||||
|
||||
然后我们需要在集群中安装 Helm. 幸运的是，只需要运行下面的命令就好：
|
||||
然后我们需要在集群中安装 Helm。 幸运的是，只需要运行下面的命令就好：
|
||||
|
||||
```
|
||||
$ helm init
|
||||
|
||||
```
|
||||
|
||||
### 安装 WordPress
|
||||
@@ -60,20 +55,18 @@ $ helm init
|
||||
|
||||
```
|
||||
$ helm install --namespace wordpress --name wordpress --set serviceType=NodePort stable/wordpress
|
||||
|
||||
```
|
||||
|
||||
这条命令将会在容器中运行 WordPress，并在容器中运行 MariaDB 作为数据库。它在 Kubernetes 中被称为“Pod”。一个 [Pod][22] 基本上可视为一个或多个应用程序容器和一些这些容器的共享资源（例如存储卷，网络等）的组合的抽象。
|
||||
这条命令将会在容器中安装并运行 WordPress，并在容器中运行 MariaDB 作为数据库。它在 Kubernetes 中被称为“Pod”。一个 [Pod][22] 基本上可视为一个或多个应用程序容器和这些容器的一些共享资源（例如存储卷，网络等）的组合的抽象。
|
||||
|
||||
我们需要给这个部署一个名字和一个命名空间，以保证他们有序，而且方便查找。我们同样会给 `NodePort` 设置 `serviceType`。这一步非常重要，因为在默认设置中，服务类型会被设置为 `LoadBalancer`。由于我们的集群现在没有负载均衡器，所以我们将无法在集群外访问我们的 WordPress 站点。
|
||||
我们需要给这个部署一个名字和一个命名空间，以将它们组织起来并便于查找。我们同样会将 `serviceType` 设置为 `NodePort` 。这一步非常重要，因为在默认设置中，服务类型会被设置为 `LoadBalancer`。由于我们的集群现在没有负载均衡器，所以我们将无法在集群外访问我们的 WordPress 站点。
|
||||
|
||||
在输出数据的最后一部分，你会注意到一些关于访问你的 WordPress 站点的实用信息。运行这些命令，你可以获取到我们的 WordPress 站点的外部 IP 地址和端口：
|
||||
在输出数据的最后一部分，你会注意到一些关于访问你的 WordPress 站点的有用的命令。运行那些命令，你可以获取到我们的 WordPress 站点的外部 IP 地址和端口：
|
||||
|
||||
```
|
||||
$ export NODE_PORT=$(kubectl get --namespace wordpress -o jsonpath="{.spec.ports[0].nodePort}" services wordpress-wordpress)
|
||||
$ export NODE_IP=$(kubectl get nodes --namespace wordpress -o jsonpath="{.items[0].status.addresses[0].address}")
|
||||
$ echo http://$NODE_IP:$NODE_PORT/admin
|
||||
|
||||
```
|
||||
|
||||
你现在访问刚刚生成的 URL（忽略 `/admin` 部分），就可以看到 WordPress 已经在你的 Kubernetes 集群中运行了！
|
||||
@@ -84,7 +77,6 @@ Kubernetes 等服务编排平台的一个伟大之处，在于它将应用的扩
|
||||
|
||||
```
|
||||
$ kubectl get deployments --namespace=wordpress
|
||||
|
||||
```
|
||||
|
||||
[][23]
|
||||
@@ -93,24 +85,22 @@ $ kubectl get deployments --namespace=wordpress
|
||||
|
||||
```
|
||||
$ kubectl scale --replicas 2 deployments wordpress-wordpress --namespace=wordpress
|
||||
|
||||
```
|
||||
|
||||
再次运行 `kubectl get deployments`，我们现在应该会看到下面的场景：
|
||||
|
||||
[][24]
|
||||
|
||||
你刚刚扩大了你的 WordPress 站点规模！超级简单，对不对？现在我们有多个 WordPress 容器，可以在他们之中对流量进行负载均衡。想了解 Kubernetes 扩展的更多信息，参见[这篇指南][25]。
|
||||
你刚刚扩大了你的 WordPress 站点规模！超级简单，对不对？现在我们有了多个 WordPress 容器，可以在它们之中对流量进行负载均衡。想了解 Kubernetes 扩展的更多信息，参见[这篇指南][25]。
|
||||
|
||||
### 高可用
|
||||
|
||||
Kubernetes 等平台的的另一大特色在于，它不单单能进行方便的扩展，还可以通过自愈组建来提供高可用性。假设我们的 WordPress 部署因为某些原因失效了，那 Kubernetes 会立刻自动替换掉这个部署。我们可以通过删除我们 WordPress 部署的一个 pod 来模拟这个过程。
|
||||
Kubernetes 等平台的的另一大特色在于，它不单单能进行方便的扩展，还可以通过自愈组件来提供高可用性。假设我们的一个 WordPress 部署因为某些原因失效了，那 Kubernetes 会立刻自动替换掉这个部署。我们可以通过删除我们 WordPress 部署的一个 pod 来模拟这个过程。
|
||||
|
||||
首先运行命令，获取 pod 列表：
|
||||
|
||||
```
|
||||
$ kubectl get pods --namespace=wordpress
|
||||
|
||||
```
|
||||
|
||||
[][26]
|
||||
@@ -118,11 +108,10 @@ $ kubectl get pods --namespace=wordpress
|
||||
然后删除其中一个 pod：
|
||||
|
||||
```
|
||||
$ kubectl delete pod {POD-ID} --namespace=wordpress
|
||||
|
||||
$ kubectl delete pod wordpress-wordpress-876183909-jqc8s --namespace=wordpress
|
||||
```
|
||||
|
||||
如果你再次运行 `kubectl get pods` 命令，应该会看到 Kubernetes 立刻换上了新的 pod。
|
||||
如果你再次运行 `kubectl get pods` 命令，应该会看到 Kubernetes 立刻换上了新的 pod （`3l167`）。
|
||||
|
||||
[][27]
|
||||
|
||||
@@ -131,14 +120,11 @@ $ kubectl delete pod {POD-ID} --namespace=wordpress
|
||||
我们只是简单了解了 Kubernetes 能完成工作的表面。如果你想深入研究，我建议你查看以下功能：
|
||||
|
||||
* [平行扩展][2]
|
||||
|
||||
* [自动恢复][3]
|
||||
|
||||
* [自愈][3]
|
||||
* [自动更新及回滚][4]
|
||||
|
||||
* [密钥管理][5]
|
||||
|
||||
你在容器平台上运行过 WordPress 吗？有没有使用过 Kubernetes（或其它容器编排平台），有没有什么好的提示？你通常会怎么扩展你的 WordPress 站点？请在评论中告诉我们。
|
||||
你在容器平台上运行过 WordPress 吗？有没有使用过 Kubernetes（或其它容器编排平台），有没有什么好的技巧？你通常会怎么扩展你的 WordPress 站点？请在评论中告诉我们。
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
@@ -151,9 +137,9 @@ Gilbert 喜欢构建软件。从 jQuery 脚本到 WordPress 插件，再到完
|
||||
|
||||
via: https://deliciousbrains.com/running-wordpress-kubernetes-cluster/
|
||||
|
||||
作者：[ Gilbert Pellegrom][a]
|
||||
作者：[Gilbert Pellegrom][a]
|
||||
译者：[StdioA](https://github.com/StdioA)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
校对：[wxy](https://github.com/wxy)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
333
published/20170823 How Machines Learn A Practical Guide.md
Normal file
333
published/20170823 How Machines Learn A Practical Guide.md
Normal file
@@ -0,0 +1,333 @@
|
||||
机器学习实践指南
|
||||
============================================================
|
||||
|
||||
|
||||
|
||||
你可能在各种应用中听说过<ruby>机器学习<rt>machine learning</rt></ruby>（ML），比如垃圾邮件过滤、光学字符识别（OCR）和计算机视觉。
|
||||
|
||||
开启机器学习之旅是一个涉及多方面的漫长旅途。对于新手，有很多的书籍，有学术论文，有指导练习，有独立项目。在这些众多的选择里面，很容易迷失你最初想学习的目标。
|
||||
|
||||
所以在今天的文章中，我会列出 7 个步骤（和 50 多个资源）帮助你开启这个令人兴奋的计算机科学领域的大门，并逐渐成为一个机器学习高手。
|
||||
|
||||
请注意，这个资源列表并不详尽，只是为了让你入门。 除此之外，还有更多的资源。
|
||||
|
||||
### 1、 学习必要的背景知识
|
||||
|
||||
你可能还记得 DataCamp 网站上的[学习数据科学][77]这篇文章里面的信息图：数学和统计学是开始机器学习（ML）的关键。 基础可能看起来很容易，因为它只有三个主题。 但不要忘记这些实际上是三个广泛的话题。
|
||||
|
||||
在这里需要记住两件非常重要的事情：
|
||||
|
||||
- 首先，你一定会需要一些进一步的指导，以了解开始机器学习需要覆盖哪些知识点。
|
||||
- 其次，这些是你进一步学习的基础。 不要害怕花时间，有了这些知识你才能构建一切。
|
||||
|
||||
第一点很简单：学习线性代数和统计学是个好主意。这两门知识是必须要理解的。但是在你学习的同时，也应该尝试学习诸如最优化和高等微积分等主题。当你越来越深入 ML 的时候，它们就能派上用场。
|
||||
|
||||
如果是从零开始的，这里有一些入门指南可供参考：
|
||||
|
||||
* [Khan 学院][1] 对于初学者是非常好的资源，可以考虑学习他们的线性代数和微积分课程。
|
||||
* 在 [麻省理工学院 OpenCourseWare][2] 网站上学习[线性代数][3] 课程。
|
||||
* [Coursera course][4] 网站上对描述统计学、概率论和推论统计学的介绍内容。
|
||||
|
||||
|
||||
|
||||
*统计学是学习 ML 的关键之一*
|
||||
|
||||
如果你更多喜欢阅读书籍，请参考以下内容：
|
||||
|
||||
* <ruby>[线性代数及其应用][5]<rt>Linear Algebra and Its Applications</rt></ruby>
|
||||
* <ruby>[应用线性代数][6]<rt>Applied Linear Algebra</rt></ruby>
|
||||
* <ruby>[线性代数解决的 3000 个问题][7]<rt>3,000 Solved Problems in Linear Algebra</rt></ruby>
|
||||
* [麻省理工学院在线教材][8]
|
||||
|
||||
然而，在大多数情况下，你已经对统计学和数学有了一个初步的了解。很有可能你已经浏览过上面列举的的那些资源。
|
||||
|
||||
在这种情况下，诚实地回顾和评价你的知识是一个好主意，是否有一些领域是需要复习的，或者现在掌握的比较好的？
|
||||
|
||||
如果你一切都准备好了，那么现在是时候使用 R 或者 Python 应用这些知识了。作为一个通用的指导方针，选择一门语言开始是个好主意。另外，你仍然可以将另一门语言加入到你的技能池里。
|
||||
|
||||
为什么这些编程知识是必需的？
|
||||
|
||||
嗯，你会看到上面列出的课程（或你在学校或大学学习的课程）将为你提供关于数学和统计学主题的更理论性的介绍（而不是应用性的）。 然而，ML 非常便于应用，你需要能够应用你所学到的所有主题。 所以最好再次复习一遍之前的材料，但是这次需要付诸应用。
|
||||
|
||||
如果你想掌握 R 和 Python 的基础，可以看以下课程：
|
||||
|
||||
* DataCamp 上关于 Python 或者 R 的介绍性课程： [Python 语言数据科学介绍][9] 或者 [R 语言编程介绍][10]。
|
||||
* Edx 上关于 Python 或者 R 的介绍性课程： [Python 语言数据科学介绍][11] 和 [R 语言数据科学介绍][12]。
|
||||
* 还有很多其他免费的课程。查看 [Coursera][13] 或者 [Codeacademy][14] 了解更多。
|
||||
|
||||
当你打牢基础知识后，请查看 DataCamp 上的博客 [Python 统计学：40+ 数据科学资源][78]。 这篇文章提供了统计学方面的 40 多个资源，这些资源都是你开始数据科学（以及 ML）需要学习的。
|
||||
|
||||
还要确保你查看了关于向量和数组的 [这篇 SciPy 教程][79]文章，以及使用 Python 进行科学计算的[研讨会][80]。
|
||||
|
||||
要使用 Python 和微积分进行实践，你可以了解下 [SymPy 软件包][81]。
|
||||
|
||||
### 2、 不要害怕在 ML 的“理论”上浪费时间
|
||||
|
||||
很多人并不会花很多精力去浏览理论材料，因为理论是枯燥的、无聊的。但从长远来看，在理论知识上投入时间是至关重要的、非常值得的。 你将会更好地了解机器学习的新进展，也能和背景知识结合起来。 这将有助于你保持学习积极性。
|
||||
|
||||
此外，理论并不会多无聊。 正如你在介绍中所看到的，你可以借助非常多的资料深入学习。
|
||||
|
||||
书籍是吸收理论知识的最佳途径之一。 它们可以让你停下来想一会儿。 当然，看书是一件非常平静的事情，可能不符合你的学习风格。 不过，请尝试阅读下列书籍，看看它是否适合你：
|
||||
|
||||
* <ruby>[机器学习教程][15]<rt>Machine Learning textbook</rt></ruby>， Tom Mitchell 著，书可能比较旧，但是却很经典。这本书很好的解释介绍了机器学习中最重要的课题，步骤详尽，逐层深入。
|
||||
* <ruby>机器学习: 使数据有意义的算法艺术和科学<rt>Machine Learning: The Art and Science of Algorithms that Make Sense of Data</rt></ruby>（你可以在[这里][16]看到这本书的幻灯片版本）：这本书对初学者来说非常棒。 里面讨论了许多实践中的应用程序，其中有一些是在 Tom Mitchell 的书中缺少的。
|
||||
* <ruby>[机器学习之向往][17]<rt>Machine Learning Yearning</rt></ruby> ：这本书由<ruby>吴恩达<rt>Andrew Ng</rt></ruby>编写的，仍未完本，但对于那些正在学习 ML 的学生来说，这一定是很好的参考资料。
|
||||
* <ruby>[算法与数据结构][18]<rt>Algorithms and Data Structures</rt></ruby> 由 Jurg Nievergelt 和 Klaus Hinrichs 著。
|
||||
* 也可以参阅 Matthew North 的<ruby>[面向大众的数据挖掘][19]<rt>Data Mining for the Masses</rt></ruby>。 你会发现这本书引导你完成一些最困难的主题。
|
||||
* <ruby>[机器学习介绍][20]<rt>Introduction to Machine Learning</rt></ruby> 由 Alex Smola 和 S.V.N. Vishwanathan 著。
|
||||
|
||||
|
||||
|
||||
*花些时间看书并研究其中涵盖的资料*
|
||||
|
||||
视频和慕课对于喜欢边听边看来学习的人来说非常棒。 慕课和视频非常的多，多到可能你都很难找到适合你的。 下面列出了最知名的几个：
|
||||
|
||||
* [这个著名的机器学习慕课][21]，是<ruby>吴恩达<rt>Andrew Ng</rt></ruby>讲的，介绍了机器学习及其理论。 别担心，这个慕课讲的非常好，一步一步深入，所以对初学者来说非常适用。
|
||||
* [麻省理工学院 Open Courseware 的 6034 课程的节目清单][22]，已经有点前沿了。 在你开始本系列之前，你需要做一些 ML 理论方面的准备工作，但是你不会后悔的。
|
||||
|
||||
在这一点上，重要的是要将各种独立的技术融会贯通，形成整体的结构图。 首先了解关键的概念：<ruby>监督学习<rt>supervised learning</rt></ruby>和<ruby>无监督学习<rt>unsupervised learning</rt></ruby>的区别、分类和回归等。 手动（书面）练习可以派上用场，能帮你了解算法是如何工作的以及如何应用这些算法。 在大学课程里你经常会找到一些书面练习，可以看看波特兰州立大学的 [ML 课程][82]。
|
||||
|
||||
### 3、 开始动手
|
||||
|
||||
通过看书和看视频了解理论和算法都非常好，但是需要超越这一阶段，就要开始做一些练习。你要学着去实现这些算法，应用学到的理论。
|
||||
|
||||
首先，有很多介绍 Python 和 R 方面的机器学习的基础知识。当然最好的方法就是使用交互式教程：
|
||||
|
||||
* [Python 机器学习：Scikit-Learn 教程][23]，在这篇教程里面，你可以学到使用 Scikit-Learn 构建模型的 KMeans 和支持向量机（SVM）相关的知名算法。
|
||||
* [给初学者的 R 语言机器学习教程][24] 用 R 中的类和 caret 包介绍机器学习。
|
||||
* [Keras 教程：Python 深度学习[25] 涵盖了如何一步一步的为分类和回归任务构建多层感知器（MLP）。
|
||||
|
||||
还请查看以下静态的（非互动的）教程，这些需要你在 IDE 中操作：
|
||||
|
||||
* [循序渐进：Python 机器学习][26]： 一步一步地学习 Scikit-Learn。
|
||||
* [循序渐进：使用 Keras 开发你的第一个神经网络][27]： 按这个教程一步一步地使用 Keras 开发你的第一个神经网络。
|
||||
* 你可以考虑看更多的教程，但是[机器学习精要][28]这篇教程是非常好的。
|
||||
|
||||
除了教程之外，还有一些课程。参加课程可以帮助你系统性地应用学到的概念。 经验丰富的导师很有帮助。 以下是 Python 和机器学习的一些互动课程：
|
||||
|
||||
* [用 scikit-learn 做监督学习][29]： 学习如何构建预测模型，调整参数，并预测在未知数据上执行的效果。你将使用 Scikit-Learn 操作真实世界的数据集。
|
||||
* [用 Python 做无监督学习][30]： 展示给你如何从未标记的数据集进行聚类、转换、可视化和提取关键信息。 在课程结束时，还会构建一个推荐系统。
|
||||
* [Python 深度学习][31]： 你将获得如何使用 Keras 2.0 进行深度学习的实践知识，Keras 2.0 是前沿的 Python 深度学习库 Keras 的最新版本。
|
||||
* [在 Python 中应用机器学习][32]： 将学习者引入到机器学习实践中，更多地关注技术和方法，而不是这些方法背后的统计学知识。
|
||||
|
||||
|
||||
|
||||
*理论学习之后，花点时间来应用你所学到的知识。*
|
||||
|
||||
对于那些正在学习 R 语言机器学习的人，还有这些互动课程：
|
||||
|
||||
* [机器学习介绍][33] 可以让你宏观了解机器学习学科最常见的技术和应用，还可以更多地了解不同机器学习模型的评估和训练。这门课程剩下的部分重点介绍三个最基本的机器学习任务： 分类、回归和聚类。
|
||||
* [R 语言无监督学习][34] ，用 R 语言从 ML 角度提供聚类和降维的基本介绍。 可以让你尽快获得数据的关键信息。
|
||||
* [实操机器学习][35]涵盖了构建和应用预测功能的基本组成部分，其重点是实际应用。
|
||||
|
||||
最后，还有很多书籍以偏向实践的方式介绍了 ML 主题。 如果你想借助书籍内容和 IDE 来学习，请查看这些书籍：
|
||||
|
||||
* <ruby>[Python 机器学习][36]<rt>Python Machine Learning Book</rt></ruby>，Sebastian Raschka 著。
|
||||
* <ruby>[人工神经网络与深度学习导论：Python 应用实用指南][37]<rt>Introduction to Artificial Neural Networks and Deep Learning: A Practical Guide with Applications in Python</rt></ruby>，Sebastian Raschka 著。
|
||||
* <ruby>[R 语言机器学习][38]<rt>Machine Learning with R</rt></ruby>，Brett Lantz 著。
|
||||
|
||||
### 4、 练习
|
||||
|
||||
实践比使用 Python 进行练习和修改材料更重要。 这一步对我来说可能是最难的。 在做了一些练习后看看其他人是如何实现 ML 算法的。 然后，开始你自己的项目，阐述你对 ML 算法和理论的理解。
|
||||
|
||||
最直接的方法之一就是将练习的规模做得更大些。 要做一个更大的练习，就需要你做更多的数据清理和功能工程。
|
||||
|
||||
- 从 [Kaggle][39] 开始。 如果你需要额外的帮助来征服所谓的“数据恐惧”，请查看 [Kaggle 的 Python 机器学习教程][40] 和 [Kaggle 的 R 语言机器学习教程][41]。 这些将带给您快速的提升。
|
||||
- 此后，你也可以自己开始挑战。 查看这些网站，您可以在其中找到大量的 ML 数据集：[UCI 机器学习仓库][42]，[用于机器学习的公开数据集][43] 和 [data.world][44]。
|
||||
|
||||
|
||||
|
||||
*熟能生巧。*
|
||||
|
||||
### 5、 项目
|
||||
|
||||
虽然做一些小的练习也不错，但是在最后，您需要做一个项目，可以在其中展示您对使用到的 ML 算法的理解。
|
||||
|
||||
最好的练习是实现你自己的 ML 算法。 您可以在以下页面中阅读更多关于为什么您应该做这样的练习，以及您可以从中学到什么内容：
|
||||
|
||||
- [为什么有许多先进的 API，比如 tensorflow，还需要自己手动实现机器学习的算法？][45]
|
||||
- [为什么要从头开始实现机器学习算法？][46]
|
||||
- [使用 Python 从头开始实现一个分类器，我能从中学到什么？][47]
|
||||
|
||||
接下来，您可以查看以下文章和仓库。 可以从中获得一些灵感，并且了解他们是如何实现 ML 算法的。
|
||||
|
||||
- [如何实现机器学习算法][48]
|
||||
- [从头开始学习机器学习][49]
|
||||
- [从头开始学习机器学习算法][50]
|
||||
|
||||
|
||||
|
||||
*开始时项目可能会很难，但是可以极大增加你的理解。*
|
||||
|
||||
### 6、 不要停止
|
||||
|
||||
对 ML 的学习永远不能停止，即使你在这个领域工作了十年，总是有新的东西要学习，许多人都将会证实这一点。
|
||||
|
||||
例如，ML 趋势，比如<ruby>深度学习<rt>deep learning</rt></ruby>现在就很受欢迎。你也可以专注于那些现在不怎么火，但是将来会火的话题上。如果你想了解更多，可以看看[这个有趣的问题和答案][83]。
|
||||
|
||||
当你苦恼于掌握基础知识时，你最先想到的可能不是论文。 但是它们是你紧跟最新研究的一个途径。 论文并不适合刚刚开始学习的人，但是绝对适合高级人员。
|
||||
|
||||
- [20 篇最新的机器学习和深度学习领域的顶级研究论文][51]
|
||||
- [机器学习研究杂志][52]
|
||||
- [优秀的深度学习论文][53]
|
||||
- [机器学习的一些最好的研究论文和书籍][54]
|
||||
|
||||
其他技术也是需要考虑的。 但是当你刚开始学习时，不要担心这些。 例如，您可以专注于 Python 或 R 语言 （取决于你已经知道哪一个），并把它到你的技能池里。 你可以通过这篇文章来查找一些感兴趣的资源。
|
||||
|
||||
如果您还想转向大数据，您可以考虑研究 Spark。 这里有一些有趣的资源：
|
||||
|
||||
* [在 R 语言中使用 sparklyr 来了解 Spark][55]
|
||||
* [Spark 数据科学与工程][56]
|
||||
* [介绍 Apache Spark][57]
|
||||
* [Apache Spark 分布式机器学习][58]
|
||||
* [用 Apache Spark 进行大数据分析][59]
|
||||
* [初学者指南：用 Python 操作 Apache Spark][60]
|
||||
* [PySpark RDD 速查表][61]
|
||||
* [PySpark SQL 速查表][62]
|
||||
|
||||
其他编程语言，比如 Java、JavaScript、C 和 C++ 在 ML 中越来越重要。 从长远来看，您可以考虑将其中一种语言添加到学习列表中。 你可以使用这些博客文章来指导你选择：
|
||||
|
||||
* [机器学习和数据科学最流行的编程语言][63]
|
||||
* [机器学习和数据科学最流行的语言是...][64]
|
||||
|
||||
|
||||
|
||||
*学无止境。*
|
||||
|
||||
### 7、 利用一切可以利用的资源
|
||||
|
||||
机器学习是一个充满难度的话题，有时候可能会让你失去动力。 或者也许你觉得你需要点改变。 在这种情况下，请记住，有很多资源可以让你打消掉这种想法。 查看以下资源：
|
||||
|
||||
**播客**是可以让你继续你的 ML 旅程，紧跟这个领域最新的发展的伟大资源：
|
||||
|
||||
* [谈论机器][65]
|
||||
* [数据怀疑论者][66]
|
||||
* [线性化][67]
|
||||
* [本周的机器学习及 AI][68]
|
||||
* [机器学习 101][69]
|
||||
|
||||
当然，还有更多的播客。
|
||||
|
||||
**文档和软件包源代码**是深入了解 ML 算法的实现的两种方法。 查看这些仓库：
|
||||
|
||||
* [Scikit-Learn][70]：知名的 Python ML 软件包
|
||||
* [Keras][71]： Python 深度学习软件包
|
||||
* [caret][72]： 非常受欢迎的用于分类和回归训练 R 软件包
|
||||
|
||||
**可视化**是深入 ML 理论的最新也是最流行的方式之一。 它们对初学者来说非常棒，但对于更高级的学习者来说也是非常有趣的。 你肯定会被下面这些可视化资源所吸引，它们能让你更加了解 ML 的工作原理：
|
||||
|
||||
- [机器学习的可视化介绍][73]
|
||||
- [Distill][74] 使 ML 研究清晰，动态和生动。
|
||||
- 如果你想玩下神经网络架构，可以看下 [Tensorflow - 神经网络游乐场][75]。
|
||||
- 更多的看这里：[机器学习算法最佳的可视化方法是什么？][76]
|
||||
|
||||
|
||||
|
||||
*学习中的一些变化更加能激励你。*
|
||||
|
||||
### 现在你可以开始了
|
||||
|
||||
现在一切都取决于你自己了。学习机器学习是一个持续的过程，所以开始的越早就会越好。 运用你手边的一切工具开始吧。 祝你好运，并确保让我们知道你的进步。
|
||||
|
||||
_这篇文章是我基于 Quora 问题（[小白该如何开始机器学习][84]）给出的答案。_
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
作者简介：
|
||||
|
||||
Karlijn Willems，数据科学记者
|
||||
|
||||
-----------------------
|
||||
|
||||
via: https://medium.freecodecamp.org/how-machines-learn-a-practical-guide-203aae23cafb
|
||||
|
||||
作者：[Karlijn Willems][a]
|
||||
译者：[Flowsnow](https://github.com/Flowsnow)
|
||||
校对：[wxy](https://github.com/wxy)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://medium.freecodecamp.org/@kacawi
|
||||
[1]:http://www.khanacademy.org/
|
||||
[2]:https://ocw.mit.edu/index.htm
|
||||
[3]:https://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-spring-2010/
|
||||
[4]:https://www.coursera.org/learn/basic-statistics
|
||||
[5]:https://www.amazon.com/Linear-Algebra-Its-Applications-4th/dp/0030105676
|
||||
[6]:https://www.amazon.com/Applied-Linear-Algebra-3rd-Noble/dp/0130412600
|
||||
[7]:https://www.amazon.de/Solved-Problems-Linear-Algebra-Schaums/dp/0070380236
|
||||
[8]:https://ocw.mit.edu/courses/online-textbooks/
|
||||
[9]:https://www.datacamp.com/courses/intro-to-python-for-data-science
|
||||
[10]:https://www.datacamp.com/courses/free-introduction-to-r
|
||||
[11]:https://www.edx.org/course/introduction-python-data-science-microsoft-dat208x-5
|
||||
[12]:https://www.edx.org/course/introduction-r-data-science-microsoft-dat204x-4
|
||||
[13]:http://www.coursera.org/
|
||||
[14]:https://www.codecademy.com/
|
||||
[15]:http://www.cs.cmu.edu/~tom/mlbook.html
|
||||
[16]:http://www.cs.bris.ac.uk/~flach/mlbook/materials/mlbook-beamer.pdf
|
||||
[17]:http://www.mlyearning.org/
|
||||
[18]:https://www.amazon.com/Algorithms-Data-Structures-Applications-Practitioner/dp/0134894286
|
||||
[19]:https://www.amazon.com/Data-Mining-Masses-Matthew-North/dp/0615684378
|
||||
[20]:http://alex.smola.org/drafts/thebook.pdf
|
||||
[21]:https://www.coursera.org/learn/machine-learning
|
||||
[22]:https://youtu.be/TjZBTDzGeGg?list=PLnvKubj2-I2LhIibS8TOGC42xsD3-liux
|
||||
[23]:https://www.datacamp.com/community/tutorials/machine-learning-python
|
||||
[24]:https://www.datacamp.com/community/tutorials/machine-learning-in-r
|
||||
[25]:https://www.datacamp.com/community/tutorials/deep-learning-python
|
||||
[26]:http://machinelearningmastery.com/machine-learning-in-python-step-by-step/
|
||||
[27]:http://machinelearningmastery.com/tutorial-first-neural-network-python-keras/
|
||||
[28]:http://www.machinelearningmastery.com/
|
||||
[29]:https://www.datacamp.com/courses/supervised-learning-with-scikit-learn
|
||||
[30]:https://www.datacamp.com/courses/unsupervised-learning-in-python
|
||||
[31]:https://www.datacamp.com/courses/deep-learning-in-python
|
||||
[32]:https://www.coursera.org/learn/python-machine-learning
|
||||
[33]:https://www.datacamp.com/courses/introduction-to-machine-learning-with-r
|
||||
[34]:https://www.datacamp.com/courses/unsupervised-learning-in-r
|
||||
[35]:https://www.coursera.org/learn/practical-machine-learning
|
||||
[36]:https://github.com/rasbt/python-machine-learning-book
|
||||
[37]:https://github.com/rasbt/deep-learning-book
|
||||
[38]:https://books.google.be/books/about/Machine_Learning_with_R.html?id=ZQu8AQAAQBAJ&amp;amp;amp;amp;amp;source=kp_cover&amp;amp;amp;amp;amp;redir_esc=y
|
||||
[39]:http://www.kaggle.com/
|
||||
[40]:https://www.datacamp.com/community/open-courses/kaggle-python-tutorial-on-machine-learning
|
||||
[41]:https://www.datacamp.com/community/open-courses/kaggle-tutorial-on-machine-learing-the-sinking-of-the-titanic
|
||||
[42]:http://archive.ics.uci.edu/ml/
|
||||
[43]:http://homepages.inf.ed.ac.uk/rbf/IAPR/researchers/MLPAGES/mldat.htm
|
||||
[44]:https://data.world/
|
||||
[45]:https://www.quora.com/Why-is-there-a-need-to-manually-implement-machine-learning-algorithms-when-there-are-many-advanced-APIs-like-tensorflow-available
|
||||
[46]:http://www.kdnuggets.com/2016/05/implement-machine-learning-algorithms-scratch.html
|
||||
[47]:http://www.jeannicholashould.com/what-i-learned-implementing-a-classifier-from-scratch.html
|
||||
[48]:http://machinelearningmastery.com/how-to-implement-a-machine-learning-algorithm/
|
||||
[49]:https://github.com/eriklindernoren/ML-From-Scratch
|
||||
[50]:https://github.com/madhug-nadig/Machine-Learning-Algorithms-from-Scratch
|
||||
[51]:http://www.kdnuggets.com/2017/04/top-20-papers-machine-learning.html
|
||||
[52]:http://www.jmlr.org/
|
||||
[53]:https://github.com/terryum/awesome-deep-learning-papers
|
||||
[54]:https://www.quora.com/What-are-some-of-the-best-research-papers-books-for-Machine-learning
|
||||
[55]:https://www.datacamp.com/courses/introduction-to-spark-in-r-using-sparklyr
|
||||
[56]:https://www.edx.org/xseries/data-science-engineering-apache-spark
|
||||
[57]:https://www.edx.org/course/introduction-apache-spark-uc-berkeleyx-cs105x
|
||||
[58]:https://www.edx.org/course/distributed-machine-learning-apache-uc-berkeleyx-cs120x
|
||||
[59]:https://www.edx.org/course/big-data-analysis-apache-spark-uc-berkeleyx-cs110x
|
||||
[60]:https://www.datacamp.com/community/tutorials/apache-spark-python
|
||||
[61]:https://www.datacamp.com/community/blog/pyspark-cheat-sheet-python
|
||||
[62]:https://www.datacamp.com/community/blog/pyspark-sql-cheat-sheet
|
||||
[63]:https://fossbytes.com/popular-top-programming-languages-machine-learning-data-science/
|
||||
[64]:http://www.kdnuggets.com/2017/01/most-popular-language-machine-learning-data-science.html
|
||||
[65]:http://www.thetalkingmachines.com/
|
||||
[66]:https://dataskeptic.com/
|
||||
[67]:http://lineardigressions.com/
|
||||
[68]:https://twimlai.com/
|
||||
[69]:http://www.learningmachines101.com/
|
||||
[70]:https://github.com/scikit-learn/scikit-learn
|
||||
[71]:http://www.github.com/fchollet/keras
|
||||
[72]:http://topepo/caret
|
||||
[73]:http://www.r2d3.us/visual-intro-to-machine-learning-part-1/
|
||||
[74]:http://distill.pub/
|
||||
[75]:http://playground.tensorflow.org/
|
||||
[76]:https://www.quora.com/What-are-the-best-visualizations-of-machine-learning-algorithms
|
||||
[77]:https://www.datacamp.com/community/tutorials/learn-data-science-infographic
|
||||
[78]:https://www.datacamp.com/community/tutorials/python-statistics-data-science
|
||||
[79]:https://www.datacamp.com/community/tutorials/python-scipy-tutorial
|
||||
[80]:http://www.math.pitt.edu/~siam/workshops/python10/python.pdf
|
||||
[81]:http://docs.sympy.org/latest/tutorial/calculus.html
|
||||
[82]:http://web.cecs.pdx.edu/~mm/MachineLearningSpring2017/
|
||||
[83]:https://www.quora.com/Should-I-quit-machine-learning
|
||||
[84]:https://www.quora.com/How-does-a-total-beginner-start-to-learn-machine-learning/answer/Karlijn-Willems-1
|
||||
419
published/GDB-common-commands.md
Normal file
419
published/GDB-common-commands.md
Normal file
@@ -0,0 +1,419 @@
|
||||
常用 GDB 命令中文速览
|
||||
============
|
||||
|
||||
### 目录
|
||||
|
||||
- [break](#break) -- 在指定的行或函数处设置断点，缩写为 `b`
|
||||
- [info breakpoints](#info-breakpoints) -- 打印未删除的所有断点，观察点和捕获点的列表，缩写为 `i b`
|
||||
- [disable](#disable) -- 禁用断点，缩写为 `dis`
|
||||
- [enable](#enable) -- 启用断点
|
||||
- [clear](#clear) -- 清除指定行或函数处的断点
|
||||
- [delete](#delete) -- 删除断点，缩写为 `d`
|
||||
- [tbreak](#tbreak) -- 设置临时断点，参数同 `break`，但在程序第一次停住后会被自动删除
|
||||
- [watch](#watch) -- 为表达式（或变量）设置观察点，当表达式（或变量）的值有变化时，暂停程序执行
|
||||
- [step](#step) -- 单步跟踪，如果有函数调用，会进入该函数，缩写为 `s`
|
||||
- [reverse-step](#reverse-step) -- 反向单步跟踪，如果有函数调用，会进入该函数
|
||||
- [next](#next) -- 单步跟踪，如果有函数调用，不会进入该函数，缩写为 `n`
|
||||
- [reverse-next](#reverse-next) -- 反向单步跟踪，如果有函数调用，不会进入该函数
|
||||
- [return](#return) -- 使选定的栈帧返回到其调用者
|
||||
- [finish](#finish) -- 执行直到选择的栈帧返回，缩写为 `fin`
|
||||
- [until](#until) -- 执行直到达到当前栈帧中当前行后的某一行（用于跳过循环、递归函数调用），缩写为 `u`
|
||||
- [continue](#continue) -- 恢复程序执行，缩写为 `c`
|
||||
- [print](#print) -- 打印表达式 EXP 的值，缩写为 `p`
|
||||
- [x](#x) -- 查看内存
|
||||
- [display](#display) -- 每次程序停止时打印表达式 EXP 的值（自动显示）
|
||||
- [info display](#info-display) -- 打印早先设置为自动显示的表达式列表
|
||||
- [disable display](#disable-display) -- 禁用自动显示
|
||||
- [enable display](#enable-display) -- 启用自动显示
|
||||
- [undisplay](#undisplay) -- 删除自动显示项
|
||||
- [help](#help) -- 打印命令列表（带参数时查找命令的帮助），缩写为 `h`
|
||||
- [attach](#attach) -- 挂接到已在运行的进程来调试
|
||||
- [run](#run) -- 启动被调试的程序，缩写为 `r`
|
||||
- [backtrace](#backtrace) -- 查看程序调用栈的信息，缩写为 `bt`
|
||||
- [ptype](#ptype) -- 打印类型 TYPE 的定义
|
||||
|
||||
------
|
||||
|
||||
### break
|
||||
|
||||
使用 `break` 命令（缩写 `b`）来设置断点。
|
||||
|
||||
用法：
|
||||
|
||||
- `break` 当不带参数时，在所选栈帧中执行的下一条指令处设置断点。
|
||||
- `break <function-name>` 在函数体入口处打断点，在 C++ 中可以使用 `class::function` 或 `function(type, ...)` 格式来指定函数名。
|
||||
- `break <line-number>` 在当前源码文件指定行的开始处打断点。
|
||||
- `break -N` `break +N` 在当前源码行前面或后面的 `N` 行开始处打断点，`N` 为正整数。
|
||||
- `break <filename:linenum>` 在源码文件 `filename` 的 `linenum` 行处打断点。
|
||||
- `break <filename:function>` 在源码文件 `filename` 的 `function` 函数入口处打断点。
|
||||
- `break <address>` 在程序指令的地址处打断点。
|
||||
- `break ... if <cond>` 设置条件断点，`...` 代表上述参数之一（或无参数），`cond` 为条件表达式，仅在 `cond` 值非零时暂停程序执行。
|
||||
|
||||
详见[官方文档][1]。
|
||||
|
||||
### info breakpoints
|
||||
|
||||
查看断点，观察点和捕获点的列表。
|
||||
|
||||
用法：
|
||||
|
||||
- `info breakpoints [list...]`
|
||||
- `info break [list...]`
|
||||
- `list...` 用来指定若干个断点的编号（可省略），可以是 `2`， `1-3`， `2 5` 等。
|
||||
|
||||
### disable
|
||||
|
||||
禁用一些断点。参数是用空格分隔的断点编号。要禁用所有断点，不加参数。
|
||||
|
||||
禁用的断点不会被忘记，但直到重新启用才有效。
|
||||
|
||||
用法：
|
||||
|
||||
- `disable [breakpoints] [list...]`
|
||||
- `breakpoints` 是 `disable` 的子命令（可省略），`list...` 同 `info breakpoints` 中的描述。
|
||||
|
||||
详见[官方文档][2]。
|
||||
|
||||
### enable
|
||||
|
||||
启用一些断点。给出断点编号（以空格分隔）作为参数。没有参数时，所有断点被启用。
|
||||
|
||||
用法：
|
||||
|
||||
- `enable [breakpoints] [list...]` 启用指定的断点（或所有定义的断点）。
|
||||
- `enable [breakpoints] once list...` 临时启用指定的断点。GDB 在停止您的程序后立即禁用这些断点。
|
||||
- `enable [breakpoints] delete list...` 使指定的断点启用一次，然后删除。一旦您的程序停止，GDB 就会删除这些断点。等效于用 `tbreak` 设置的断点。
|
||||
|
||||
`breakpoints` 同 `disable` 中的描述。
|
||||
|
||||
详见[官方文档][2]。
|
||||
|
||||
### clear
|
||||
|
||||
在指定行或函数处清除断点。参数可以是行号，函数名称或 `*` 跟一个地址。
|
||||
|
||||
用法：
|
||||
|
||||
- `clear` 当不带参数时，清除所选栈帧在执行的源码行中的所有断点。
|
||||
- `clear <function>`, `clear <filename:function>` 删除在命名函数的入口处设置的任何断点。
|
||||
- `clear <linenum>`, `clear <filename:linenum>` 删除在指定的文件指定的行号的代码中设置的任何断点。
|
||||
- `clear <address>` 清除指定程序指令的地址处的断点。
|
||||
|
||||
详见[官方文档][3]。
|
||||
|
||||
### delete
|
||||
|
||||
删除一些断点或自动显示表达式。参数是用空格分隔的断点编号。要删除所有断点，不加参数。
|
||||
|
||||
用法： `delete [breakpoints] [list...]`
|
||||
|
||||
详见[官方文档][3]。
|
||||
|
||||
### tbreak
|
||||
|
||||
设置临时断点。参数形式同 `break` 一样。
|
||||
|
||||
除了断点是临时的之外，其他同 `break` 一样，所以在命中时会被删除。
|
||||
|
||||
详见[官方文档][1]。
|
||||
|
||||
### watch
|
||||
|
||||
为表达式设置观察点。
|
||||
|
||||
用法： `watch [-l|-location] <expr>` 每当一个表达式的值改变时，观察点就会暂停程序执行。
|
||||
|
||||
如果给出了 `-l` 或者 `-location`，则它会对 `expr` 求值并观察它所指向的内存。例如，`watch *(int *)0x12345678` 将在指定的地址处观察一个 4 字节的区域（假设 int 占用 4 个字节）。
|
||||
|
||||
详见[官方文档][4]。
|
||||
|
||||
### step
|
||||
|
||||
单步执行程序，直到到达不同的源码行。
|
||||
|
||||
用法： `step [N]` 参数 `N` 表示执行 N 次（或由于另一个原因直到程序停止）。
|
||||
|
||||
警告：如果当控制在没有调试信息的情况下编译的函数中使用 `step` 命令，则执行将继续进行，直到控制到达具有调试信息的函数。 同样，它不会进入没有调试信息编译的函数。
|
||||
|
||||
要执行没有调试信息的函数，请使用 `stepi` 命令，详见后文。
|
||||
|
||||
详见[官方文档][5]。
|
||||
|
||||
### reverse-step
|
||||
|
||||
反向单步执行程序，直到到达另一个源码行的开头。
|
||||
|
||||
用法： `reverse-step [N]` 参数 `N` 表示执行 N 次（或由于另一个原因直到程序停止）。
|
||||
|
||||
详见[官方文档][6]。
|
||||
|
||||
### next
|
||||
|
||||
单步执行程序，执行完子程序调用。
|
||||
|
||||
用法： `next [N]`
|
||||
|
||||
与 `step` 不同，如果当前的源代码行调用子程序，则此命令不会进入子程序，而是将其视为单个源代码行，继续执行。
|
||||
|
||||
详见[官方文档][5]。
|
||||
|
||||
### reverse-next
|
||||
|
||||
反向步进程序，执行完子程序调用。
|
||||
|
||||
用法： `reverse-next [N]`
|
||||
|
||||
如果要执行的源代码行调用子程序，则此命令不会进入子程序，调用被视为一个指令。
|
||||
|
||||
参数 `N` 表示执行 N 次（或由于另一个原因直到程序停止）。
|
||||
|
||||
详见[官方文档][6]。
|
||||
|
||||
### return
|
||||
|
||||
您可以使用 `return` 命令取消函数调用的执行。如果你给出一个表达式参数，它的值被用作函数的返回值。
|
||||
|
||||
用法： `return <expression>` 将 `expression` 的值作为函数的返回值并使函数直接返回。
|
||||
|
||||
详见[官方文档][7]。
|
||||
|
||||
### finish
|
||||
|
||||
执行直到选定的栈帧返回。
|
||||
|
||||
用法： `finish` 返回后，返回的值将被打印并放入到值历史记录中。
|
||||
|
||||
详见[官方文档][5]。
|
||||
|
||||
### until
|
||||
|
||||
执行直到程序到达当前栈帧中当前行之后（与 [break](#break) 命令相同的参数）的源码行。此命令用于通过一个多次的循环，以避免单步执行。
|
||||
|
||||
用法：`until <location>` 或 `u <location>` 继续运行程序，直到达到指定的位置，或者当前栈帧返回。
|
||||
|
||||
详见[官方文档][5]。
|
||||
|
||||
### continue
|
||||
|
||||
在信号或断点之后，继续运行被调试的程序。
|
||||
|
||||
用法： `continue [N]` 如果从断点开始，可以使用数字 `N` 作为参数，这意味着将该断点的忽略计数设置为 `N - 1`(以便断点在第 N 次到达之前不会中断)。如果启用了非停止模式（使用 `show non-stop` 查看），则仅继续当前线程，否则程序中的所有线程都将继续。
|
||||
|
||||
详见[官方文档][5]。
|
||||
|
||||
### print
|
||||
|
||||
求值并打印表达式 EXP 的值。可访问的变量是所选栈帧的词法环境，以及范围为全局或整个文件的所有变量。
|
||||
|
||||
用法：
|
||||
|
||||
- `print [expr]` 或 `print /f [expr]` `expr` 是一个（在源代码语言中的）表达式。
|
||||
|
||||
默认情况下，`expr` 的值以适合其数据类型的格式打印；您可以通过指定 `/f` 来选择不同的格式，其中 `f` 是一个指定格式的字母；详见[输出格式][9]。
|
||||
|
||||
如果省略 `expr`，GDB 再次显示最后一个值。
|
||||
|
||||
要以每行一个成员带缩进的格式打印结构体变量请使用命令 `set print pretty on`，取消则使用命令 `set print pretty off`。
|
||||
|
||||
可使用命令 `show print` 查看所有打印的设置。
|
||||
|
||||
详见[官方文档][8]。
|
||||
|
||||
### x
|
||||
|
||||
检查内存。
|
||||
|
||||
用法： `x/nfu <addr>` 或 `x <addr>` `n`、`f` 和 `u` 都是可选参数，用于指定要显示的内存以及如何格式化。`addr` 是要开始显示内存的地址的表达式。
|
||||
|
||||
`n` 重复次数（默认值是 1），指定要显示多少个单位（由 `u` 指定）的内存值。
|
||||
|
||||
`f` 显示格式（初始默认值是 `x`），显示格式是 `print('x'，'d'，'u'，'o'，'t'，'a'，'c'，'f'，'s')` 使用的格式之一，再加 `i`（机器指令）。
|
||||
|
||||
`u` 单位大小，`b` 表示单字节，`h` 表示双字节，`w` 表示四字节，`g` 表示八字节。
|
||||
|
||||
例如：
|
||||
|
||||
`x/3uh 0x54320` 表示从地址 0x54320 开始以无符号十进制整数的格式，双字节为单位来显示 3 个内存值。
|
||||
|
||||
`x/16xb 0x7f95b7d18870` 表示从地址 0x7f95b7d18870 开始以十六进制整数的格式，单字节为单位显示 16 个内存值。
|
||||
|
||||
详见[官方文档][10]。
|
||||
|
||||
### display
|
||||
|
||||
每次程序暂停时，打印表达式 EXP 的值。
|
||||
|
||||
用法： `display <expr>`, `display/fmt <expr>` 或 `display/fmt <addr>` `fmt` 用于指定显示格式。像 [print](#print) 命令里的 `/f` 一样。
|
||||
|
||||
对于格式 `i` 或 `s`，或者包括单位大小或单位数量，将表达式 `addr` 添加为每次程序停止时要检查的内存地址。
|
||||
|
||||
详见[官方文档][11]。
|
||||
|
||||
### info display
|
||||
|
||||
打印自动显示的表达式列表，每个表达式都带有项目编号，但不显示其值。
|
||||
|
||||
包括被禁用的表达式和不能立即显示的表达式（当前不可用的自动变量）。
|
||||
|
||||
### undisplay
|
||||
|
||||
取消某些表达式在程序暂停时的自动显示。参数是表达式的编号（使用 `info display` 查询编号）。不带参数表示取消所有自动显示表达式。
|
||||
|
||||
`delete display` 具有与此命令相同的效果。
|
||||
|
||||
### disable display
|
||||
|
||||
禁用某些表达式在程序暂停时的自动显示。禁用的显示项目不会被自动打印，但不会被忘记。 它可能稍后再次被启用。
|
||||
|
||||
参数是表达式的编号（使用 `info display` 查询编号）。不带参数表示禁用所有自动显示表达式。
|
||||
|
||||
### enable display
|
||||
|
||||
启用某些表达式在程序暂停时的自动显示。
|
||||
|
||||
参数是重新显示的表达式的编号（使用 `info display` 查询编号）。不带参数表示启用所有自动显示表达式。
|
||||
|
||||
### help
|
||||
|
||||
打印命令列表。
|
||||
|
||||
您可以使用不带参数的 `help`（缩写为 `h`）来显示命令的类别名的简短列表。
|
||||
|
||||
使用 `help <class>` 您可以获取该类中的各个命令的列表。使用 `help <command>` 显示如何使用该命令。
|
||||
|
||||
详见[官方文档][12]。
|
||||
|
||||
### attach
|
||||
|
||||
挂接到 GDB 之外的进程或文件。该命令可以将进程 ID 或设备文件作为参数。
|
||||
|
||||
对于进程 ID，您必须具有向进程发送信号的权限，并且必须具有与调试器相同的有效的 uid。
|
||||
|
||||
用法： `attach <process-id>` GDB 在安排调试指定的进程之后做的第一件事是暂停该进程。
|
||||
|
||||
无论是通过 `attach` 命令挂接的进程还是通过 `run` 命令启动的进程，您都可以使用的 GDB 命令来检查和修改挂接的进程。
|
||||
|
||||
详见[官方文档][13]。
|
||||
|
||||
### run
|
||||
|
||||
启动被调试的程序。
|
||||
|
||||
可以直接指定参数，也可以用 [set args][15] 设置（启动所需的）参数。
|
||||
|
||||
例如： `run arg1 arg2 ...` 等效于
|
||||
|
||||
```
|
||||
set args arg1 arg2 ...
|
||||
run
|
||||
```
|
||||
|
||||
还允许使用 `>`、 `<` 或 `>>` 进行输入和输出重定向。
|
||||
|
||||
详见[官方文档][14]。
|
||||
|
||||
### backtrace
|
||||
|
||||
打印整体栈帧信息。
|
||||
|
||||
- `bt` 打印整体栈帧信息，每个栈帧一行。
|
||||
- `bt n` 类似于上，但只打印最内层的 n 个栈帧。
|
||||
- `bt -n` 类似于上，但只打印最外层的 n 个栈帧。
|
||||
- `bt full n` 类似于 `bt n`，还打印局部变量的值。
|
||||
|
||||
`where` 和 `info stack`(缩写 `info s`) 是 `backtrace` 的别名。调用栈信息类似如下：
|
||||
|
||||
```
|
||||
(gdb) where
|
||||
#0 vconn_stream_run (vconn=0x99e5e38) at lib/vconn-stream.c:232
|
||||
#1 0x080ed68a in vconn_run (vconn=0x99e5e38) at lib/vconn.c:276
|
||||
#2 0x080dc6c8 in rconn_run (rc=0x99dbbe0) at lib/rconn.c:513
|
||||
#3 0x08077b83 in ofconn_run (ofconn=0x99e8070, handle_openflow=0x805e274 <handle_openflow>) at ofproto/connmgr.c:1234
|
||||
#4 0x08075f92 in connmgr_run (mgr=0x99dc878, handle_openflow=0x805e274 <handle_openflow>) at ofproto/connmgr.c:286
|
||||
#5 0x08057d58 in ofproto_run (p=0x99d9ba0) at ofproto/ofproto.c:1159
|
||||
#6 0x0804f96b in bridge_run () at vswitchd/bridge.c:2248
|
||||
#7 0x08054168 in main (argc=4, argv=0xbf8333e4) at vswitchd/ovs-vswitchd.c:125
|
||||
```
|
||||
|
||||
详见[官方文档][16]。
|
||||
|
||||
### ptype
|
||||
|
||||
打印类型 TYPE 的定义。
|
||||
|
||||
用法： `ptype[/FLAGS] TYPE-NAME | EXPRESSION`
|
||||
|
||||
参数可以是由 `typedef` 定义的类型名， 或者 `struct STRUCT-TAG` 或者 `class CLASS-NAME` 或者 `union UNION-TAG` 或者 `enum ENUM-TAG`。
|
||||
|
||||
根据所选的栈帧的词法上下文来查找该名字。
|
||||
|
||||
类似的命令是 `whatis`，区别在于 `whatis` 不展开由 `typedef` 定义的数据类型，而 `ptype` 会展开，举例如下：
|
||||
|
||||
```
|
||||
/* 类型声明与变量定义 */
|
||||
typedef double real_t;
|
||||
struct complex {
|
||||
real_t real;
|
||||
double imag;
|
||||
};
|
||||
typedef struct complex complex_t;
|
||||
|
||||
complex_t var;
|
||||
real_t *real_pointer_var;
|
||||
```
|
||||
|
||||
这两个命令给出了如下输出：
|
||||
|
||||
```
|
||||
(gdb) whatis var
|
||||
type = complex_t
|
||||
(gdb) ptype var
|
||||
type = struct complex {
|
||||
real_t real;
|
||||
double imag;
|
||||
}
|
||||
(gdb) whatis complex_t
|
||||
type = struct complex
|
||||
(gdb) whatis struct complex
|
||||
type = struct complex
|
||||
(gdb) ptype struct complex
|
||||
type = struct complex {
|
||||
real_t real;
|
||||
double imag;
|
||||
}
|
||||
(gdb) whatis real_pointer_var
|
||||
type = real_t *
|
||||
(gdb) ptype real_pointer_var
|
||||
type = double *
|
||||
```
|
||||
|
||||
详见[官方文档][17]。
|
||||
|
||||
------
|
||||
|
||||
### 参考资料
|
||||
|
||||
- [Debugging with GDB](https://sourceware.org/gdb/current/onlinedocs/gdb/)
|
||||
|
||||
------
|
||||
|
||||
译者：[robot527](https://github.com/robot527)
|
||||
校对：[mudongliang](https://github.com/mudongliang)， [wxy](https://github.com/wxy)
|
||||
|
||||
[1]: https://sourceware.org/gdb/current/onlinedocs/gdb/Set-Breaks.html
|
||||
[2]: https://sourceware.org/gdb/current/onlinedocs/gdb/Disabling.html
|
||||
[3]: https://sourceware.org/gdb/current/onlinedocs/gdb/Delete-Breaks.html
|
||||
[4]: https://sourceware.org/gdb/current/onlinedocs/gdb/Set-Watchpoints.html
|
||||
[5]: https://sourceware.org/gdb/current/onlinedocs/gdb/Continuing-and-Stepping.html
|
||||
[6]: https://sourceware.org/gdb/current/onlinedocs/gdb/Reverse-Execution.html
|
||||
[7]: https://sourceware.org/gdb/current/onlinedocs/gdb/Returning.html
|
||||
[8]: https://sourceware.org/gdb/current/onlinedocs/gdb/Data.html
|
||||
[9]: https://sourceware.org/gdb/current/onlinedocs/gdb/Output-Formats.html
|
||||
[10]: https://sourceware.org/gdb/current/onlinedocs/gdb/Memory.html
|
||||
[11]: https://sourceware.org/gdb/current/onlinedocs/gdb/Auto-Display.html
|
||||
[12]: https://sourceware.org/gdb/current/onlinedocs/gdb/Help.html
|
||||
[13]: https://sourceware.org/gdb/current/onlinedocs/gdb/Attach.html
|
||||
[14]: https://sourceware.org/gdb/current/onlinedocs/gdb/Starting.html
|
||||
[15]: https://sourceware.org/gdb/current/onlinedocs/gdb/Arguments.html
|
||||
[16]: https://sourceware.org/gdb/current/onlinedocs/gdb/Backtrace.html
|
||||
[17]: https://sourceware.org/gdb/current/onlinedocs/gdb/Symbols.html
|
||||
@@ -1,4 +1,4 @@
|
||||
translating by XYenChi
|
||||
XYenChi is translating
|
||||
A 5-step plan to encourage your team to make changes on your project
|
||||
============================================================
|
||||
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
Translating by gitlilys
|
||||
|
||||
GOOGLE CHROME–ONE YEAR IN
|
||||
========================================
|
||||
|
||||
|
||||
@@ -1,83 +0,0 @@
|
||||
Translated by DPueng
|
||||
|
||||
Why we need open leaders more than ever
|
||||
============================================================
|
||||
|
||||
### Changing social and cultural conditions are giving rise to open leadership.
|
||||
|
||||
Posted 02 Feb 2017[Philip A Foster][10][Feed][9]13[up][6]
|
||||
|
||||
>Image by : opensource.com
|
||||
|
||||
Leadership is power. More specifically, leadership is the power to influence the actions of others. The mythology of leadership can certainly conjure images of not only the romantic but also the sinister side of the human condition. How we ultimately decide to engage in leadership determines its true nature.
|
||||
|
||||
Many modern understandings of leadership are born out of warfare, where leadership is the skillful execution of command-and-control thinking. For most of the modern era of business, then, we engaged leadership as some great man or woman arriving at the pinnacle of power and exerting this power through position. Such traditional leadership relies heavily on formal lines of authority through hierarchies and reporting relationships. Authority in these structures flows down through the vertical hierarchy and exists along formal lines in the chain of command.
|
||||
|
||||
>Open leaders quickly discover that leadership is not about the power we exert to influence progress, but the power and confidence we distribute among the members of the organization.
|
||||
|
||||
However, in the late 20th century, something began to change. New technologies opened doors to globalism and thus more dispersed teams. The way we engaged human capital began to shift, forever changing the way people communicate with each other. People inside organizations began to feel empowered, and they demanded a sense of ownership of their successes (and failures). Leaders were no longer the sole owners of power. The 21st century leader leading the 21st century organization began to understand empowerment, collaboration, accountability, and clear communication were the essence of a new kind of power. These new leaders began _sharing_ that power—and they implicitly trusted their followers.
|
||||
|
||||
As organizations continue becoming more open, even individuals without "leadership" titles feel empowered to drive change. These organizations remove the chains of hierarchy and untether workers to do their jobs in the ways they best see fit. History has exposed 20th century leaders' tendencies to strangle agility through unilateral decision-making and unidirectional information flows. But the new century's leader best defines an organization by the number of individuals it empowers to get something done. There's power in numbers—and, frankly, one leader cannot be in all places at all times, making all the decisions.
|
||||
|
||||
So leaders are becoming open, too.
|
||||
|
||||
### Control
|
||||
|
||||
Where the leaders of old are focused on command-and-control positional power, an open leader cedes organizational control to others via new forms of organizational governance, new technologies, and other means of reducing friction, thereby enabling collective action in a more efficient manner. These leaders understand the power of trust, and believe followers will always show initiative, engagement, and independence. And this new brand of leadership requires a shift in tactics—from _telling people what to do_ to _showing them what to do_ and _coaching them along the way_. Open leaders quickly discover that leadership is not about the power we exert to influence progress, but the power and confidence we _distribute_ among the members of the organization. The 21stcentury leader is focused on community and the edification of others. In the end, the open leader is not focused on self but is selfless.
|
||||
|
||||
### Communication
|
||||
|
||||
The 20th century leader hordes and controls the flow of information throughout the organization. The open leader, however, seeks to engage an organization by sharing information and context (as well as authority) with members of a team. These leaders destroy fiefdoms, walk humbly, and share power like never before. The collective empowerment and engaged collaboration they inspire create agility, shared responsibility, ownership—and, above all, happiness. When members of an organization are empowered to do their jobs, they're happier (and thus more productive) than their hierarchical counterparts.
|
||||
|
||||
### Trust
|
||||
|
||||
Open leaders embrace uncertainty and trust their followers to do the right thing at the right time. They possess an ability to engage human capital at a higher level of efficiency than their traditional counterparts. Again: They don't operate as command-and-control micromanagers. Elevating transparency, they don't operate in hiding, and they do their best to keep decisions and actions out in the open, explaining the basis on which decisions get made and assuming employees have a high level grasp of situations within the organization. Open leaders operate from the premise that the organization's human capital is more than capable of achieving success without their constant intervention.
|
||||
|
||||
### Autonomy
|
||||
|
||||
Where the powerful command-and-control 20th century leader is focused on some _position_ of power, an open leader is more interested in the actual _role_ an individual plays within the organization. When a leader is focused on an _individual_, they're better able to coach and mentor members of a team. From this perspective, an open leader is focused on modeling behaviors and actions that are congruent with the organization's vision and mission. In the end, an open leader is very much seen as a member of the team rather than the _head_ of the team. This does not mean the leader abdicates a position of authority, but rather understates it in an effort to share power and empower individuals through autonomy to create results.
|
||||
|
||||
### Empowerment
|
||||
|
||||
Open leaders are focused on granting authority to members of an organization. This process acknowledges the skills, abilities, and trust the leader has in the organization's human capital, and thereby creates positive motivation and willingness for the entire team to take risks. Empowerment, in the end, is about helping followers believe in their own abilities. Followers who believe that they have personal power are more likely to undertake initiatives, set and achieve higher goals, and persist in the face of difficult circumstances. Ultimately the concept of an open organization is about inclusivity, where everyone belongs and individuality and differing opinions are essential to success. An open organization and its open leaders offer a sense of community, and members are motivated by the organization's mission or purpose. This creates a sense of belonging to something bigger than the individual. Individuality creates happiness and job satisfaction among its members. In turn, higher degrees of efficiency and success are achieved.
|
||||
|
||||
>More Open Organization Resources
|
||||
|
||||
* [Download the Open Organization Leaders Manual][1]
|
||||
* [Download the Open Organization Field Guide][2]
|
||||
* [What is an Open Organization?][3]
|
||||
* [What is an Open Decision?][4]
|
||||
|
||||
We should all strive for the openness the 21st century leader requires. This requires self-examination, curiosity—and, above all, it's ongoing process of change. Through new attitudes and habits, we move toward the discovery of what an open leader really _is _and _does,_ and hopefully we begin to take on those ideals as we adapt our leadership styles to the 21st century.
|
||||
|
||||
Yes, leadership is power. How we use that power determines the success or failure of our organizations. Those who abuse power don't last, but those who share power and celebrate others do. By reading [this book][7], you are beginning to play an important role in the ongoing conversation of the open organization and its leadership. And at the conclusion of [this volume][8], you'll find additional resources and opportunities to connect with the open organization community, so that you too can chat, think, and grow with us. Welcome to the conversation—welcome to the journey!
|
||||
|
||||
_This article originally appeared as the introduction to _The Open Organization Leaders Manual_, now [available from Opensource.com][5]._
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
译者简介：
|
||||
|
||||
Philip A Foster - Dr. Philip A. Foster is a leadership/business coach and consultant and Adjunct Professor. He is a noted Thought Leader in Business Operations, Organizational Development, Foresight and Strategic Leadership. Dr. Foster facilitates change through the design and implementation of strategies, strategic foresight, and planning.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://opensource.com/open-organization/17/2/need-open-leaders-more-ever
|
||||
|
||||
作者：[Philip A Foster][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://opensource.com/users/maximumchange
|
||||
[1]:https://opensource.com/open-organization/resources/leaders-manual?src=too_resource_menu
|
||||
[2]:https://opensource.com/open-organization/resources/field-guide?src=too_resource_menu
|
||||
[3]:https://opensource.com/open-organization/resources/open-org-definition?src=too_resource_menu
|
||||
[4]:https://opensource.com/open-organization/resources/open-decision-framework?src=too_resource_menu
|
||||
[5]:https://opensource.com/open-organization/resources/leaders-manual
|
||||
[6]:https://opensource.com/open-organization/17/2/need-open-leaders-more-ever?rate=c_9hT0EKbdXcTGRl-YW0QgW60NsRwO2a4RaplUKfvXs
|
||||
[7]:https://opensource.com/open-organization/resources/leaders-manual
|
||||
[8]:https://opensource.com/open-organization/resources/leaders-manual
|
||||
[9]:https://opensource.com/user/15497/feed
|
||||
[10]:https://opensource.com/users/maximumchange
|
||||
@@ -1,4 +1,4 @@
|
||||
Translating by windcode
|
||||
translating---geekpi
|
||||
|
||||
# Why DevOps is the end of security as we know it
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
Education of a Programmer
|
||||
============================================================
|
||||
|
||||
_When I left Microsoft in October 2016 after almost 21 years there and almost 35 years in the industry, I took some time to reflect on what I had learned over all those years. This is a lightly edited version of that post. Pardon the length!_
|
||||
_When I left Microsoft in October 2016 after almost 21 years there and almost 35 years in the industry, I took some time to reflect on what I had learned over all those years. This is a lightly edited version of that post. Pardon the length!_
|
||||
|
||||
There are an amazing number of things you need to know to be a proficient programmer — details of languages, APIs, algorithms, data structures, systems and tools. These things change all the time — new languages and programming environments spring up and there always seems to be some hot new tool or language that “everyone” is using. It is important to stay current and proficient. A carpenter needs to know how to pick the right hammer and nail for the job and needs to be competent at driving the nail straight and true.
|
||||
|
||||
|
||||
@@ -1,7 +1,3 @@
|
||||
申请翻译 by WangYueScream
|
||||
==================================
|
||||
|
||||
|
||||
A Window Into the Linux Desktop
|
||||
============================================================
|
||||
|
||||
|
||||
@@ -1,103 +0,0 @@
|
||||
Security Debt is an Engineer’s Problem
|
||||
============================================================
|
||||
|
||||
|
||||
|
||||
|
||||
>Keziah Plattner of AirBnBSecurity.
|
||||
|
||||
Just like organizations can build up technical debt, so too can they also build up something called “security debt,” if they don’t plan accordingly, attendees learned at the [WomenWhoCode Connect ][5]event at Twitter headquarters in San Francisco last month.
|
||||
|
||||
Security has got to be integral to every step of the software development process, stressed [Mary Ann Davidson][6], Oracle’s Chief Security Officer, in a keynote talk with about security for developers with [Zassmin Montes de Oca][7] of [WomenWhoCode][8].
|
||||
|
||||
In the past, security used to be ignored by pretty much everyone, except banks. But security is more critical than it has ever been because there are so many access points. We’ve entered the era of [Internet of Things][9], where thieves can just hack your fridge to see that you’re not home.
|
||||
|
||||
Davidson is in charge of assurance at Oracle, “making sure we build security into everything we build, whether it’s an on-premise product, whether it’s a cloud service, even devices we have that support group builds at customer sites and reports data back to us, helping us do diagnostics — every single one of those things has to have security engineered into it.”
|
||||
|
||||
|
||||
|
||||
Plattner talking to a capacity crowd at #WWCConnect
|
||||
|
||||
AirBnB’s [Keziah Plattner][10] echoed that sentiment in her breakout session. “Most developers don’t see security as their job,” she said, “but this has to change.”
|
||||
|
||||
She shared four basic security principles for engineers. First, security debt is expensive. There’s a lot of talk about [technical debt ][11]and she thinks security debt should be included in those conversations.
|
||||
|
||||
“This historical attitude is ‘We’ll think about security later,’” Plattner said. As companies grab the low-hanging fruit of software efficiency and growth, they ignore security, but an initial insecure design can cause problems for years to come.
|
||||
|
||||
It’s very hard to add security to an existing vulnerable system, she said. Even when you know where the security holes are and have budgeted the time and resources to make the changes, it’s time-consuming and difficult to re-engineer a secure system.
|
||||
|
||||
So it’s key, she said, to build security into your design from the start. Think of security as part of the technical debt to avoid. And cover all possibilities.
|
||||
|
||||
Most importantly, according to Plattner, is the difficulty in getting to people to change their behavior. No one will change voluntarily, she said, even when you point out that the new behavior is more secure. We all nodded.
|
||||
|
||||
Davidson said engineers need to start thinking about how their code could be attacked, and design from that perspective. She said she only has two rules. The first is never trust any unvalidated data and rule two is see rule one.
|
||||
|
||||
“People do this all the time. They say ‘My client sent me the data so it will be fine.’ Nooooooooo,” she said, to laughs.
|
||||
|
||||
The second key to security, Plattner said, is “never trust users.”
|
||||
|
||||
Davidson put it another way: “My job is to be a professional paranoid.” She worries all the time about how someone might breach her systems even inadvertently. This is not academic, there has been recent denial of service attacks through IoT devices.
|
||||
|
||||
### Little Bobby Tables
|
||||
|
||||
If part of your security plan is trusting users to do the right thing, your system is inherently insecure regardless of whatever other security measures you have in place, said Plattner.
|
||||
|
||||
It’s important to properly sanitize all user input, she explained, showing the [XKCD cartoon][12] where a mom wiped out an entire school database because her son’s middle name was “DropTable Students.”
|
||||
|
||||
So sanitize all user input. Check.
|
||||
|
||||
She showed an example of JavaScript developers using Eval on open source. “A good ground rule is ‘Never use eval(),’” she cautioned. The [eval() ][13]function evaluates JavaScript code. “You’re opening your system to random users if you do.”
|
||||
|
||||
Davidson cautioned that her paranoia extends to including security testing your example code in documentation. “Because we all know no one ever copies sample code,” she said to laughter. She underscored the point that any code should be subject to security checks.
|
||||
|
||||
|
||||
|
||||
Make it easy
|
||||
|
||||
Plattner’s suggestion three: Make security easy. Take the path of least resistance, she suggested.
|
||||
|
||||
Externally, make users opt out of security instead of opting in, or, better yet, make it mandatory. Changing people’s behavior is the hardest problem in tech, she said. Once users get used to using your product in a non-secure way, getting them to change in the future is extremely difficult.
|
||||
|
||||
Internal to your company, she suggested make tools that standardize security so it’s not something individual developers need to think about. For example, encrypting data as a service so engineers can just call the service to encrypt or decrypt data.
|
||||
|
||||
Make sure that your company is focused on good security hygiene, she said. Switch to good security habits across the company.
|
||||
|
||||
You’re only secure as your weakest link, so it’s important that each individual also has good personal security hygiene as well as having good corporate security hygiene.
|
||||
|
||||
At Oracle, they’ve got this covered. Davidson said she got tired of explaining security to engineers who graduated college with absolutely no security training, so she wrote the first coding standards at Oracle. There are now hundreds of pages with lots of contributors, and there are classes that are mandatory. They have metrics for compliance to security requirements and measure it. The classes are not just for engineers, but for doc writers as well. “It’s a cultural thing,” she said.
|
||||
|
||||
And what discussion about security would be secure without a mention of passwords? While everyone should be using a good password manager, Plattner said, but they should be mandatory for work, along with two-factor authentication.
|
||||
|
||||
Basic password principles should be a part of every engineer’s waking life, she said. What matters most in passwords is their length and entropy — making the collection of keystrokes as random as possible. A robust password entropy checker is invaluable for this. She recommends [zxcvbn][14], the Dropbox open-source entropy checker.
|
||||
|
||||
Another trick is to use something intentionally slow like [bcrypt][15] when authenticating user input, said Plattner. The slowness doesn’t bother most legit users but irritates hackers who try to force password attempts.
|
||||
|
||||
All of this adds up to job security for anyone wanting to get into the security side of technology, said Davidson. We’re putting more code more places, she said, and that creates systemic risk. “I don’t think anybody is not going to have a job in security as long as we keep doing interesting things in technology.”
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://thenewstack.io/security-engineers-problem/
|
||||
|
||||
作者：[TC Currie][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://thenewstack.io/author/tc/
|
||||
[1]:http://twitter.com/share?url=https://thenewstack.io/security-engineers-problem/&text=Security+Debt+is+an+Engineer%E2%80%99s+Problem+
|
||||
[2]:http://www.facebook.com/sharer.php?u=https://thenewstack.io/security-engineers-problem/
|
||||
[3]:http://www.linkedin.com/shareArticle?mini=true&url=https://thenewstack.io/security-engineers-problem/
|
||||
[4]:https://thenewstack.io/security-engineers-problem/#disqus_thread
|
||||
[5]:http://connect2017.womenwhocode.com/
|
||||
[6]:https://www.linkedin.com/in/mary-ann-davidson-235ba/
|
||||
[7]:https://www.linkedin.com/in/zassmin/
|
||||
[8]:https://www.womenwhocode.com/
|
||||
[9]:https://www.thenewstack.io/tag/Internet-of-Things
|
||||
[10]:https://twitter.com/ittskeziah
|
||||
[11]:https://martinfowler.com/bliki/TechnicalDebt.html
|
||||
[12]:https://xkcd.com/327/
|
||||
[13]:https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/eval
|
||||
[14]:https://blogs.dropbox.com/tech/2012/04/zxcvbn-realistic-password-strength-estimation/
|
||||
[15]:https://en.wikipedia.org/wiki/Bcrypt
|
||||
[16]:https://thenewstack.io/author/tc/
|
||||
@@ -1,122 +0,0 @@
|
||||
translating by @explosic4
|
||||
|
||||
Why working openly is hard when you just want to get stuff done
|
||||
============================================================
|
||||
|
||||
### Learn how to create a book using the Open Decision Framework.
|
||||
|
||||
|
||||
|
||||
>Image by : opensource.com
|
||||
|
||||
Three letters guide the way I work: GSD—get stuff done. Over the years, I've managed to blend concepts like feedback loops (from lean methodologies) and iterative improvement (from Agile) into my everyday work habits so I can better GSD (if I can use that as a verb). This means being extremely efficient with my time: outlining clear, discrete goals; checking completed items off a master list; and advancing projects forward iteratively and constantly. But can someone still GSD while defaulting to open? Or is this when getting stuff done comes to a grinding halt? Most would assume the worst, but I found that's not necessarily the case.
|
||||
|
||||
Working in the open and using guidance from the [Open Decision Framework][6] can get projects off to a slower start. But during a recent project, we made the decision—right at the beginning—to work openly and collaborate with our community.
|
||||
|
||||
Open Organization resources
|
||||
|
||||
* [Download the Open Organization Guide to IT Culture Change][1]
|
||||
|
||||
* [Download the Open Organization Leaders Manual][2]
|
||||
|
||||
* [What is an Open Organization?][3]
|
||||
|
||||
* [What is an Open Decision?][4]
|
||||
|
||||
It was the best decision we could have made.
|
||||
|
||||
Let's take a look at a few unexpected consequences from this experience and see how we can incorporate a GSD mentality into the Open Decision Framework.
|
||||
|
||||
### Building community
|
||||
|
||||
In November 2014, I undertook a new project: build a community around the concepts in _The Open Organization_ , a forthcoming (at the time) book by Red Hat CEO Jim Whitehurst. I thought, "Cool, that sounds like a challenge—I'm in!" Then [impostor syndrome][7] set in. I started thinking: "What in the world are we going to do, and what would success look like?"
|
||||
|
||||
_Spoiler alert_ . At the end of the book, Jim recommends that readers visit Opensource.com to continue the conversation about openness and management in the 21st century. So, in May 2015, my team launched a new section of the site dedicated to those ideas. We planned to engage in some storytelling, just like we always do at Opensource.com—this time around the ideas and concepts in the book. Since then, we've published new articles every week, hosted an online book club complete with Twitter chats, and turned _The Open Organization_ into [a book series][8].
|
||||
|
||||
We produced the first three installments of our book series in-house, releasing one every six months. When we finished one, we'd announce it to the community. Then we'd get to work on the next one, and the cycle would continue.
|
||||
|
||||
Working this way, we saw great success. Nearly 3,000 people have registered to receive the [latest book in the series][9], _The Open Organization Leaders Manual_ . And we've maintained our six-month cadence, which means the next book would coincide with the second anniversary of the book.
|
||||
|
||||
Behind the scenes, our process for creating the books is fairly straightforward: We collect our best-of-best stories about particular aspects of working openly, organize them into a compelling narrative, recruit writers to fill some gaps, typeset everything with open tools, collaborate with designers on a cover, and release. Working like this allows us to stay on our own timeline—GSD, full steam ahead. By the [third book][10], we seemed to have perfected the process.
|
||||
|
||||
That all changed when we began planning the latest volume in _Open Organization_ series, one focused on the intersection of open organizations and IT culture. I proposed using the Open Decision Framework because I wanted this book to be proof that working openly produced better results, even though I knew it would completely change our approach to the work. In spite of a fairly aggressive timeline (about two-and-a-half months), we decided to try it.
|
||||
|
||||
### Creating a book with the Open Decision Framework
|
||||
|
||||
The Open Decision Framework lists four phases that constitute the open decision-making process. Here's what we did during each (and how it worked out).
|
||||
|
||||
### 1\. Ideation
|
||||
|
||||
First, we drafted a document outlining a tentative vision for the project. We needed something we could begin sharing with potential "customers" (in our case, potential stakeholders and authors). Then we scheduled interviews with source matter experts we thought would be interested in the project—people who would give us raw, honest feedback about it. Enthusiasm and guidance from those experts validated our idea and gave us the feedback we needed to move forward. If we hadn't gotten that validation, we'd have gone back to our proposal and made a decision on where to pivot and start over.
|
||||
|
||||
### 2\. Planning and research
|
||||
|
||||
With validation after several interviews, we prepared to [announce the project publically on Opensource.com][11]. At the same time, we [launched the project on GitHub][12], offering a description, prospective timeline, and set of constraints. The project announcement was so well-received that all remaining holes in our proposed table of contents were filled within 72 hours. In addition (and more importantly), readers proposed ideas for chapters that _weren't _ already in the table of contents—things they thought might enhance the vision we'd initially sketched.
|
||||
|
||||
We experienced Linus' Law firsthand: "With more eyes, all _typos_ are shallow."
|
||||
|
||||
Looking back, I get the sense that working openly on Phases 1 and 2 really didn't negatively impacted our ability to GSD. In fact, working this way had a huge upside: identifying and filling content gaps. We didn't just fill them; we filled them _rapidly_ and with chapter ideas, we never would have considered on our own. This didn't necessarily involve more work—just work of a different type. We found ourselves asking people in our limited network to write chapters, then managing incoming requests, setting context, and pointing people in the right direction.
|
||||
|
||||
### 3\. Design, development, and testing
|
||||
|
||||
This point in the project was all about project management, cat herding, and maintaining expectations. We were on a deadline, and we communicated that early and often. We also used a tactic of creating a list of contributors and stakeholders and keeping them updated and informed along the entire journey, particularly at milestones we'd identified on GitHub.
|
||||
|
||||
Eventually, our book needed a title. We gathered lots of feedback on what the title should be, and more importantly what it _shouldn't_ be. We [opened an issue][13] as one way of gathering feedback, then openly shared that my team would be making the final decision. When we were ready to announce the final title, my colleague Bryan Behrenshausen did a great job [sharing the context for the decision][14]. People seemed to be happy with it—even if they didn't agree with where we landed with the final title.
|
||||
|
||||
Book "testing" involved extensive [proofreading][15]. The community really stepped up to answer this "help wanted" request. We received approximately 80 comments on the GitHub issue outlining the proofing process (not to mention numerous additional interactions from others via email and other feedback channels).
|
||||
|
||||
With respect to getting things done: In this phase, we experienced [Linus' Law][16] firsthand: "With more eyes, all _typos_ are shallow." Had we used the internalized method we'd used for our three previous book projects, the entire burden of proofing would have fallen on our shoulders (as it did for those books!). Instead, community members graciously helped us carry the burden of proofing, and our work shifted from proofing itself (though we still did plenty of that) to managing all the change requests coming in. This was a much-welcomed change for our team and a chance for the community to participate. We certainly would have finished the proofing faster if we'd done it ourselves, but working on this in the open undeniably allowed us to catch a greater number of errors in advance of our deadline.
|
||||
|
||||
### 4\. Launch
|
||||
|
||||
And here we are, on the cusp of launching the final (or is it just the first?) version of the book.
|
||||
|
||||
Following the Open Decision Framework was key to the success of the Guide to IT Culture Change.
|
||||
|
||||
Our approach to launch consists of two phases. First, in keeping with our public project timeline, we quietly soft-launched the book days ago so our community of contributors could help us test the [download form][17]. The second phase begins right now, with the final, formal announcement of the book's [general availability][18]. Of course, we'll continue accepting additional feedback post-launch, as is the open source way.
|
||||
|
||||
### Achievement unlocked
|
||||
|
||||
Following the Open Decision Framework was key to the success of the _Guide to IT Culture Change_ . By working with our customers and stakeholders, sharing our constraints, and being transparent with the work, we exceeded even our own expectations for the book project.
|
||||
|
||||
I was definitely pleased with the collaboration, feedback, and activity we experienced throughout the entire project. And although the feeling of anxiety about not getting stuff done as quickly as I'd liked loomed over me for a time, I soon realized that opening up the process actually allowed us to get _more_ done than we would have otherwise. That should be evident based on some of the outcomes I outlined above.
|
||||
|
||||
So perhaps I should reconsider my GSD mentality and expand it to GMD: Get **more** done—and, in this case, with better results.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
作者简介：
|
||||
|
||||
Jason Hibbets - Jason Hibbets is a senior community evangelist in Corporate Marketing at Red Hat where he is a community manager for Opensource.com. He has been with Red Hat since 2003 and is the author of The foundation for an open source city. Prior roles include senior marketing specialist, project manager, Red Hat Knowledgebase maintainer, and support engineer. Follow him on Twitter:
|
||||
|
||||
-----------
|
||||
|
||||
via: https://opensource.com/open-organization/17/6/working-open-and-gsd
|
||||
|
||||
作者：[Jason Hibbets ][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://opensource.com/users/jhibbets
|
||||
[1]:https://opensource.com/open-organization/resources/culture-change?src=too_resource_menu
|
||||
[2]:https://opensource.com/open-organization/resources/leaders-manual?src=too_resource_menu
|
||||
[3]:https://opensource.com/open-organization/resources/open-org-definition?src=too_resource_menu
|
||||
[4]:https://opensource.com/open-organization/resources/open-decision-framework?src=too_resource_menu
|
||||
[5]:https://opensource.com/open-organization/17/6/working-open-and-gsd?rate=ZgpGc0D07SjGkTOf708lnNqbF_HvkhXTXeSzRKMhvVM
|
||||
[6]:https://opensource.com/open-organization/resources/open-decision-framework
|
||||
[7]:https://opensource.com/open-organization/17/5/team-impostor-syndrome
|
||||
[8]:https://opensource.com/open-organization/resources
|
||||
[9]:https://opensource.com/open-organization/resources/leaders-manual
|
||||
[10]:https://opensource.com/open-organization/resources/leaders-manual
|
||||
[11]:https://opensource.com/open-organization/17/3/announcing-it-culture-book
|
||||
[12]:https://github.com/open-organization-ambassadors/open-org-it-culture
|
||||
[13]:https://github.com/open-organization-ambassadors/open-org-it-culture/issues/20
|
||||
[14]:https://github.com/open-organization-ambassadors/open-org-it-culture/issues/20#issuecomment-297970303
|
||||
[15]:https://github.com/open-organization-ambassadors/open-org-it-culture/issues/29
|
||||
[16]:https://en.wikipedia.org/wiki/Linus%27s_Law
|
||||
[17]:https://opensource.com/open-organization/resources/culture-change
|
||||
[18]:https://opensource.com/open-organization/resources/culture-change
|
||||
[19]:https://opensource.com/user/10530/feed
|
||||
[20]:https://opensource.com/users/jhibbets
|
||||
@@ -1,4 +1,4 @@
|
||||
Translating by sanfusu
|
||||
translating by flankershen
|
||||
|
||||
Network automation with Ansible
|
||||
================
|
||||
@@ -967,7 +967,6 @@ Ansible’s ease of use and agentless architecture accounts for the platform’s
|
||||
|
||||
There is no device left behind when using Ansible for network automation.
|
||||
|
||||
|
||||
-----------
|
||||
|
||||
作者简介：
|
||||
@@ -976,7 +975,6 @@ There is no device left behind when using Ansible for network automation.
|
||||
|
||||
Jason Edelman, CCIE 15394 & VCDX-NV 167, is a born and bred network engineer from the great state of New Jersey. He was the typical “lover of the CLI” or “router jockey.” At some point several years ago, he made the decision to focus more on software, development practices, and how they are converging with network engineering. Jason currently runs a boutique consulting firm, Network to Code, helping vendors and end users take advantage of new tools and technologies to reduce their operational inefficiencies. Jason has a Bachelor’s...
|
||||
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://www.oreilly.com/learning/network-automation-with-ansible
|
||||
|
||||
@@ -1,131 +0,0 @@
|
||||
Best Linux Adobe Alternatives You Need to Know ###translating by ninaiwohe109
|
||||
============================================================
|
||||
|
||||
|
||||
|
||||
|
||||
Looking for Adobe alternatives as a Linux user? You’re not alone. If you’re an avid graphic designer, you’re probably well-versed at avoiding pricey Adobe products. Still, Linux users are typically the ones granted the shortest end of the stick when it comes to Adobe. Adobe alternatives are a must – but what exactly are the best options available?
|
||||
|
||||
[Ads by Google][4]
|
||||
|
||||
It ultimately comes down to the specific Adobe program and what you hope to accomplish. Fortunately, as necessity remains the mother of all invention, others have answered the call. The result is a series of highly efficient stand-ins for Adobe software.
|
||||
|
||||
### Evince (Adobe Acrobat)
|
||||
|
||||
|
||||
|
||||
Like Adobe Acrobat Reader, [Evince][6] is a “document viewer for multiple documents.” For example, users may rely on Evince as a PDF viewer. It also supports various comic book formats (cbr, cbz, cb7, and cbt). You can find a [complete list of supported formats][7] at the Evince website.
|
||||
|
||||
Linux users don’t have to look high and low for Evince, as it comes standard. You can head to the official site for updates when needed.
|
||||
|
||||
### Pixlr (Adobe Photoshop)
|
||||
|
||||
|
||||
|
||||
The wonderful thing about [Pixlr][8] is this Adobe alternative’s various tools are available online. If you have an Internet connection, then you have a powerful image editing workspace.
|
||||
|
||||
[Pixlr Editor][9] is a capable stand-in for Photoshop, as you can work with layers and related effects. It also has some nifty drawing and color-editing tools. [Pixlr Express][10] doesn’t have as many features, as it’s mainly for enhancing images. Adjust the color and clarity, and also add some Instagram-friendly effects!
|
||||
|
||||
It’s incredible what you can accomplish with Pixlr, and it’s completely free.
|
||||
|
||||
### Inkscape (Adobe Illustrator)
|
||||
|
||||
|
||||
|
||||
[Inkscape][11] is another well-recommended free Adobe alternative. It functions mainly as a “professional vector graphics editor.” In addition to Illustrator, Inkscape also gets held up against Corel Draw, Freehand, and Xara X.
|
||||
|
||||
Its vector design tools are handy for crafting logos and “high scalability” artwork. Inkscape includes drawing, shape, and text tools. Layer tools allow you to lock, group, or hide individual layers.
|
||||
|
||||
### Pinegrow Web Editor (Adobe Dreamweaver)
|
||||
|
||||
|
||||
|
||||
[Pinegrow Web Editor][12] is a great substitute for Dreamweaver on Linux. The program lets you work on HTML sites right on your desktop.
|
||||
|
||||
Instead of just creating with code (and having to preview later), Pinegrow provides a detailed visual editing experience. You can see and test your HTML projects live, learning in real time whether links work or if images are where they’re supposed to be. Pinegrow also comes with a WordPress theme builder.
|
||||
|
||||
Try it free for 30 days. If you like it, you can buy it for a one-time payment of $49.
|
||||
|
||||
### Scribus (Adobe InDesign)
|
||||
|
||||
|
||||
|
||||
[Scribus][13] is probably the closest substitute you’ll find to Adobe’s InDesign. According to its developers, you should [strongly consider][14] Scribus because “it’s reliable and free.”
|
||||
|
||||
In actuality, Scribus is more than an excellent desktop publishing tool; it’s a great _self-_ publishing tool. Why rely on expensive businesses to create high-quality magazines and books when you can do it yourself? Scribus currently allows designers to work with a 200-color palette, with [promises to double the number][15] of colors with the next stable version.
|
||||
|
||||
### digiKam (Adobe Lightroom)
|
||||
|
||||
|
||||
|
||||
[digiKam][16] is perhaps the best Lightroom alternative at the moment for Linux users. Features include the capacity to import photographs, organize image collections, enhance images, create slideshows, and more.
|
||||
|
||||
Its sleek design and advanced features are an actual labor of love; the people behind digiKam are, in fact, photographers. More than that, they wanted the ability to do in Linux what others can accomplish with Lightroom.
|
||||
|
||||
### Webflow (Adobe Muse)
|
||||
|
||||
|
||||
|
||||
[Webflow][17] is yet another website that proves how much you can accomplish without having to download software. A very handy Adobe alternative for Muse, Webflow is ideal for creating highly-responsive website designs.
|
||||
|
||||
One of the best aspects of Webflow is that you don’t need to do your own coding. Just drag and drop images and write text. Webflow does the “heavy lifting” for you. There’s the option of building a website completely from scratch, or you can use various templates. Although free, the premium options boast additional features, such as the ability to easily export HTML and CSS to use elsewhere.
|
||||
|
||||
### Tupi (Adobe Animate)
|
||||
|
||||
|
||||
|
||||
[Tupi][18] is an Adobe alternative for Animate – or anyone who isn’t [too keen on Flash][19] these days. Sure, Tupi’s creators state that it’s not meant to be competition for Flash. However, the ability to work with HTML5 doesn’t stop it from being an ideal substitute.
|
||||
|
||||
Draw and animate in 2D on your PC or tablet. Unsure how to get started? Learn how to make cutout animations and more using the site’s [YouTube tutorials][20].
|
||||
|
||||
### Black Magic Fusion (Adobe After Effects)
|
||||
|
||||
|
||||
|
||||
[Black Magic Fusion][21] was destined to upstage Adobe’s After Effects. This visual effects software is the outcome of roughly 25 years of development! Fusion is routinely used to create impressive effects in Hollywood movies and television shows – driving home its detailed and stylish capabilities.
|
||||
|
||||
Fusion works by using nodes, “small icons that represent effects, filters, and other processing .” These nodes are linked together to create a series of sophisticated visual effects. The program includes a host of features such as image retouching, object tracking, and mind-blowing 3D effects.
|
||||
|
||||
You have the option of a free version or shelling out $995 for Fusion Studio. To help you decide, [you can compare][22]free and premium Fusion features.
|
||||
|
||||
[Ads by Google][5]
|
||||
|
||||
### Conclusion
|
||||
|
||||
As you can see, others are way ahead of you regarding Adobe alternatives. Thanks to open source ingenuity, significantly improved substitutes continue to be released. We could soon see a complete suite created just for Linux users. Until then, you’ll have these alternatives to choose from at your leisure.
|
||||
|
||||
Know of any useful Adobe alternatives not mentioned here? Share software recommendations in the comment section below.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://www.maketecheasier.com/adobe-alternatives-for-linux/
|
||||
|
||||
作者：[ Toni Matthews-El][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://www.maketecheasier.com/author/ttmatthe/
|
||||
[1]:https://www.maketecheasier.com/author/ttmatthe/
|
||||
[2]:https://www.maketecheasier.com/adobe-alternatives-for-linux/#comments
|
||||
[3]:https://www.maketecheasier.com/category/linux-tips/
|
||||
[4]:https://support.google.com/adsense/troubleshooter/1631343
|
||||
[5]:https://support.google.com/adsense/troubleshooter/1631343
|
||||
[6]:https://wiki.gnome.org/Apps/Evince
|
||||
[7]:https://wiki.gnome.org/Apps/Evince/SupportedDocumentFormats
|
||||
[8]:https://pixlr.com/
|
||||
[9]:https://pixlr.com/editor/
|
||||
[10]:https://pixlr.com/express/
|
||||
[11]:https://inkscape.org/en/
|
||||
[12]:http://pinegrow.com/
|
||||
[13]:https://www.scribus.net/
|
||||
[14]:https://www.scribus.net/why-on-earth-should-i-use-scribus-2/
|
||||
[15]:https://www.scribus.net/because-color-matters/
|
||||
[16]:http://digikam.org/
|
||||
[17]:https://webflow.com/
|
||||
[18]:http://www.maefloresta.com/portal/
|
||||
[19]:https://www.maketecheasier.com/sites-moving-away-flash/
|
||||
[20]:https://www.youtube.com/user/maefloresta
|
||||
[21]:https://www.blackmagicdesign.com/
|
||||
[22]:https://www.blackmagicdesign.com/products/fusion/compare
|
||||
@@ -1,177 +0,0 @@
|
||||
yzca Translating
|
||||
Docker Engine swarm mode - Intro tutorial
|
||||
============================
|
||||
|
||||
Sounds like a punk rock band. But it is the brand new orchestration mechanism, or rather, an improvement of the orchestration available in [Docker][1]. To keep it short and sweet, if you are using an older version of Docker, you will manually need to setup Swarm to create Docker clusters. Starting with [version 1.12][2], the Docker engine comes with a native implementation allowing a seamless clustering setup. The reason why we are here.
|
||||
|
||||
In this tutorial, I will try to give you a taste of what Docker can do when it comes to orchestration. This article is by no means all inclusive (bed & breakfast) or all-knowing, but it has what it takes to embark you on your clustering journey. After me.
|
||||
|
||||
|
||||
|
||||
### Technology overview
|
||||
|
||||
It would be a shame for me to rehash the very detailed and highly useful Docker documentation article, so I will just outline a brief overview of the technology. So we have Docker, right. Now, you want to use more than a single server as a Docker host, but you want them to belong to the same logical entity. Hence, clustering.
|
||||
|
||||
Let's start by a cluster of one. When you initiate swarm on a host, it becomes a manager of the cluster. Technically speaking, it becomes a consensus group of one node. The mathematical logic behind is based on the [Raft][3] algorithm. The manager is responsible for scheduling tasks. The tasks will be delegated to worker nodes, once and if they join the swarm. This is governed by the Node API. I hate the word API, but I must use it here.
|
||||
|
||||
The Service API is the second component of this implementation. It allows manager nodes to create distributed services on all of the nodes in the swarm. The services can be replicated, meaning they are spread across the cluster using balancing mechanisms, or they can be global, meaning an instance of the service will be running on each node.
|
||||
|
||||
There's much more at work here, but this is good enough to get you primed and pumped. Now, let's do some actual hands-on stuff. Our target platform is [CentOS 7.2][4], which is quite interesting, because at the time I wrote this tutorial, it only had Docker 1.10 in the repos, and I had to manually upgrade the framework to use swarm. We will discuss this in a separate tutorial. Then, we will also have a follow-up guide, where we will join new nodes into our cluster, and we will try an asymmetric setup with [Fedora][5]. At this point, please assume the correct setup is in place, and let's get a cluster service up and running.
|
||||
|
||||
### Setup image & service
|
||||
|
||||
I will try to setup a load-balanced [Apache][6] service, with multiple instances serving content via a single IP address. Pretty standard. It also highlights the typical reasons why you would go with a cluster configuration - availability, redundancy, horizontal scaling, and performance. Of course, you also need to take into consideration the [networking][7] piece, as well as [storage][8], but that's something that goes beyond the immediate scope of this guide.
|
||||
|
||||
The actual Dockerfile template is available in the official repository under httpd. You will need a minimal setup to get underway. The details on how to download images, how to create your own and such are available in my intro guide, linked at the beginning of this tutorial.
|
||||
|
||||
docker build -t my-apache2 .
|
||||
Sending build context to Docker daemon 2.048 kB
|
||||
Step 1 : FROM httpd:2.4
|
||||
Trying to pull repository docker.io/library/httpd ...
|
||||
2.4: Pulling from docker.io/library/httpd
|
||||
|
||||
8ad8b3f87b37: Pull complete
|
||||
c95e1f92326d: Pull complete
|
||||
96e8046a7a4e: Pull complete
|
||||
00a0d292c371: Pull complete
|
||||
3f7586acab34: Pull complete
|
||||
Digest: sha256:3ad4d7c4f1815bd1c16788a57f81b413...a915e50a0d3a4
|
||||
Status: Downloaded newer image for docker.io/httpd:2.4
|
||||
---> fe3336dd034d
|
||||
Step 2 : COPY ../public-html/ /usr/local/apache2/htdocs/
|
||||
...
|
||||
|
||||
|
||||
|
||||
Before you go any further, you should start a single instance and see that your container is created without any errors and that you can connect to the Web server. Once we establish that, we will create a distributed service.
|
||||
|
||||
docker run -dit --name my-running-app my-apache2
|
||||
|
||||
Check the IP address, punch into a browser, see what gives.
|
||||
|
||||
### Swarm initiation and setup
|
||||
|
||||
The next step is to get swarm going. Here's the most basic of commands that will get you underway, and it is very similar to the example used in the Docker blog:
|
||||
|
||||
docker service create --name frontend --replicas 5 -p 80:80/tcp my-apache2:latest
|
||||
|
||||
What do we have here? We are creating a service called frontend, with five container instances. We are also binding our hostPort 80 with the containerPort 80. And we are using my freshly created Apache image for this. However, when you do this, you will get the following error:
|
||||
|
||||
docker service create --name frontend --replicas 5 -p 80:80/tcp my-apache2:latest
|
||||
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
|
||||
|
||||
This means you have not setup the current host (node) to be a swarm manager. You either need to init the swarm or join an existing one. Since we do not have one yet, we will now initialize it:
|
||||
|
||||
docker swarm init
|
||||
Swarm initialized: current node (dm58mmsczqemiikazbfyfwqpd) is now a manager.
|
||||
|
||||
To add a worker to this swarm, run the following command:
|
||||
|
||||
docker swarm join \
|
||||
--token SWMTKN-1-4ofd46a2nfyvrqwu8w5oeetukrbylyznxla
|
||||
9srf9vxkxysj4p8-eu5d68pu5f1ci66s7w4wjps1u \
|
||||
10.0.2.15:2377
|
||||
|
||||
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
|
||||
|
||||
The output is fairly self explanatory. We have created a swarm. New nodes will need to use the correct token to join the swarm. You also have the IP address and port identified, if you require firewall rules. Moreover, you can add managers to the swarm, too. Now, rerun the service create command:
|
||||
|
||||
docker service create --name frontend --replicas 5 -p 80:80/tcp my-apache2:latest
|
||||
6lrx1vhxsar2i50is8arh4ud1
|
||||
|
||||
### Test connectivity
|
||||
|
||||
Now, let's check that our service actually works. In a way, this is similar to what we did with [Vagrant][9] and [coreOS][10]. After all, the concepts are almost identical. It's just different implementation of the same idea. First, docker ps should show the right output. You should have multiple replicas for the created service.
|
||||
|
||||
docker ps
|
||||
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
|
||||
NAMES
|
||||
cda532f67d55 my-apache2:latest "httpd-foreground"
|
||||
2 minutes ago Up 2 minutes 80/tcp frontend.1.2sobjfchdyucschtu2xw6ms9a
|
||||
75fe6e0aa77b my-apache2:latest "httpd-foreground"
|
||||
2 minutes ago Up 2 minutes 80/tcp frontend.4.ag77qtdeby9fyvif5v6c4zcpc
|
||||
3ce824d3151f my-apache2:latest "httpd-foreground"
|
||||
2 minutes ago Up 2 minutes 80/tcp frontend.2.b6fqg6sf4hkeqs86ps4zjyq65
|
||||
eda01569181d my-apache2:latest "httpd-foreground"
|
||||
2 minutes ago Up 2 minutes 80/tcp frontend.5.0rmei3zeeh8usagg7fn3olsp4
|
||||
497ef904e381 my-apache2:latest "httpd-foreground"
|
||||
2 minutes ago Up 2 minutes 80/tcp frontend.3.7m83qsilli5dk8rncw3u10g5a
|
||||
|
||||
I also tested with different, non-default ports, and it works well. You have a lot of leeway in how you can connect to the server and get the response. You can use localhost or the docker interface IP address with the correct port. The example below shows port 1080:
|
||||
|
||||
|
||||
|
||||
Now, this a very rough, very simple beginning. The real challenge is in creating optimized, scalable services, but they do require a proper technical use case. Moreover, you should also use the docker info and docker service (inspect|ps) commands to learn more about how your cluster is behaving.
|
||||
|
||||
### Possible problems
|
||||
|
||||
You may encounter some small (or not so small) issues while playing with Docker and swarm. For example, SELinux may complain that you are trying to do something illegal. However, the errors and warnings should not impede you too much.
|
||||
|
||||
|
||||
|
||||
### Docker service is not a docker command
|
||||
|
||||
When you try to run the necessary command to start a replicated service, you get an error that says docker: 'service' is not a docker command. This means that you do not have the right version of Docker (check with -v). We will fix this in a follow-up tutorial.
|
||||
|
||||
docker service create --name frontend --replicas 5 -p 80:80/tcp my-apache2:latest
|
||||
docker: 'service' is not a docker command.
|
||||
|
||||
### Docker tag not recognized
|
||||
|
||||
You may also see the following error:
|
||||
|
||||
docker service create -name frontend -replicas 5 -p 80:80/tcp my-apache2:latest
|
||||
Error response from daemon: rpc error: code = 3 desc = ContainerSpec: "-name" is not a valid repository/tag
|
||||
|
||||
There are several [discussions][11] [threads][12] around this. The error may actually be quite innocent. You may have copied the command from a browser, and the hyphens may not be parsed correctly. As simple as that.
|
||||
|
||||
### More reading
|
||||
|
||||
There's a lot more to be said on this topic, including the Swarm implementation prior to Docker 1.12, as well as the current version of the Docker engine. To wit, please do not be lazy and spend some time reading:
|
||||
|
||||
Docker Swarm [overview][13] (for standalone Swarm installations)
|
||||
|
||||
[Build][14] a Swarm cluster for production (standalone setups)
|
||||
|
||||
[Install and create][15] a Docker Swarm (standalone setups)
|
||||
|
||||
Docker engine swarm [overview][16] (for version 1.12)
|
||||
|
||||
Getting started with [swarm][17] mode (for version 1.12)
|
||||
|
||||
### Conclusion
|
||||
|
||||
There you go. Nothing too grand at this point, but I believe you will find the article useful. It covers several key concepts, there's an overview of how the swarm mode works and what it does, and we successfully managed to download and create our own Web server image and then run several clustered instances of it. We did this on a single node for now, but we will expand in the future. Also, we tackled some common problems.
|
||||
|
||||
I hope you find this guide interesting. Combined with my previous work on Docker, this should give you a decent understand of how to work with images, the networking stack, storage, and now clusters. Warming up. Indeed, enjoy and see you soon with fresh new tutorials on Docker. I just can't contain [sic] myself.
|
||||
|
||||
Cheers.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: http://www.dedoimedo.com/computers/docker-swarm-intro.html
|
||||
|
||||
作者：[Dedoimedo ][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:http://www.dedoimedo.com/computers/docker-swarm-intro.html
|
||||
[1]:http://www.dedoimedo.com/computers/docker-guide.html
|
||||
[2]:https://blog.docker.com/2016/06/docker-1-12-built-in-orchestration/
|
||||
[3]:https://en.wikipedia.org/wiki/Raft_%28computer_science%29
|
||||
[4]:http://www.dedoimedo.com/computers/lenovo-g50-centos-xfce.html
|
||||
[5]:http://www.dedoimedo.com/computers/fedora-24-gnome.html
|
||||
[6]:https://hub.docker.com/_/httpd/
|
||||
[7]:http://www.dedoimedo.com/computers/docker-networking.html
|
||||
[8]:http://www.dedoimedo.com/computers/docker-data-volumes.html
|
||||
[9]:http://www.dedoimedo.com/computers/vagrant-intro.html
|
||||
[10]:http://www.dedoimedo.com/computers/vagrant-coreos.html
|
||||
[11]:https://github.com/docker/docker/issues/24192
|
||||
[12]:http://stackoverflow.com/questions/38618609/docker-swarm-1-12-name-option-not-recognized
|
||||
[13]:https://docs.docker.com/swarm/
|
||||
[14]:https://docs.docker.com/swarm/install-manual/
|
||||
[15]:https://docs.docker.com/swarm/install-w-machine/
|
||||
[16]:https://docs.docker.com/engine/swarm/
|
||||
[17]:https://docs.docker.com/engine/swarm/swarm-tutorial/
|
||||
@@ -1,159 +0,0 @@
|
||||
【toutoudnf 翻译中】
|
||||
A public cloud migration in 22 days
|
||||
============================================================
|
||||
|
||||
|
||||
>Lush's Oxford St, UK store. Credit: Lush.
|
||||
|
||||
### Lush says it’s possible.
|
||||
|
||||
|
||||
Migrating your core operations from one public cloud to another in less than one month may seem like a farfetched goal, but British cosmetics giant Lush reckons it can be done.
|
||||
|
||||
Last September Lush - who you might recognise as the company behind the candy-coloured, sweet smelling bath and body products - was nearing the end of its contract with its existing infrastructure provider, understood to be [UK-based Memset][5].
|
||||
|
||||
Memset had been hosting Lush's Drupal-based commerce environment out of Amazon Web Services for a few years, but the retailer wanted out.
|
||||
|
||||
The arrangement was 'awkward' and rigid, according to Lush chief digital officer and heir to the company throne Jack Constantine (his parents founded the business in 1995).
|
||||
|
||||
“We were in a contract that we weren’t really comfortable with, and we wanted to have a look and see what else we could go for,” he told the Google Cloud Next conference in San Francisco today.
|
||||
|
||||
“It was a very closed environment [which] made it difficult for us to get visibility of everything we wanted to be able to move over.
|
||||
|
||||
"[We] could either sign up for another year, and have that commitment and think up a long-term plan where we had more control ... but [we] would have ended up struggling."
|
||||
|
||||
After scouring the market Lush landed on Google’s Cloud Platform. The company was already familiar with Google, having migrated from Scalix to Google Apps (now known as G Suite) in [late 2013][6].
|
||||
|
||||
However, it had less than a few months to make the migration, both in time for the end of its existing contract on December 22 as well as the critical Christmas shopping period.
|
||||
|
||||
“So it wasn’t just a little bit business critical. We were talking peak trade time. It was a huge deal,” Constantine said.
|
||||
|
||||
Lush’s lack of bureaucracy meant Constantine was able to make a quick decision on vendor selection, and “then the team just powered through”, he said.
|
||||
|
||||
They also prioritised optimising the "monolithic" Drupal application specifically for the migration, pushing back bug fixes until later.
|
||||
|
||||
Lush started the physical migration on December 1 and completed it on December 22.
|
||||
|
||||
The team came up against challenges “like with any migration”, Constantine said - “you have to worry about getting your data from one place to another, you have to make sure you have consistency, and customer, product data etc. needs to be up and stable”.
|
||||
|
||||
But the CDO said one thing that got the company through the incredibly tight timeframe was the team’s lack of alternatives: there was no fallback plan.
|
||||
|
||||
“About a week before the deadline my colleague had a conversation with our Google partner on the phone, they were getting a bit nervous about whether this was going to happen, and they asked us what Plan B was. My colleague said ‘Plan B is to make Plan A happen, that’s it’,” Constantine said.
|
||||
|
||||
“When you throw a hard deadline like that it can sound a bit unachieveable, but [you need to keep] that focus on people believing that this is a goal that we can achieve in that timeframe, and not letting people put up the blockers and say ‘we’re going to have to delay this and that’.
|
||||
|
||||
“Yes everybody gets very tense but you achieve a lot. You actually get through it and nail it. All the things you need to get done, get done.”
|
||||
|
||||
The focus now is on moving the commerce application to a microservices architecture, while looking into various Google tools like the Kubernetes container management system and Spanner relational database.
|
||||
|
||||
The retailer also recently built a prototype point-of-sale system using GCP and Android, which it is currently playing around with, Constantine said.
|
||||
|
||||
Allie Coyne travelled to Google Cloud Next as a guest of Google
|
||||
|
||||
|
||||
Lush's Oxford St, UK store. Credit: Lush.
|
||||
|
||||
### Lush says it’s possible.
|
||||
|
||||
Migrating your core operations from one public cloud to another in less than one month may seem like a farfetched goal, but British cosmetics giant Lush reckons it can be done.
|
||||
|
||||
Last September Lush - who you might recognise as the company behind the candy-coloured, sweet smelling bath and body products - was nearing the end of its contract with its existing infrastructure provider, understood to be [UK-based Memset][1].
|
||||
|
||||
Memset had been hosting Lush's Drupal-based commerce environment out of Amazon Web Services for a few years, but the retailer wanted out.
|
||||
|
||||
The arrangement was 'awkward' and rigid, according to Lush chief digital officer and heir to the company throne Jack Constantine (his parents founded the business in 1995).
|
||||
|
||||
“We were in a contract that we weren’t really comfortable with, and we wanted to have a look and see what else we could go for,” he told the Google Cloud Next conference in San Francisco today.
|
||||
|
||||
“It was a very closed environment [which] made it difficult for us to get visibility of everything we wanted to be able to move over.
|
||||
|
||||
"[We] could either sign up for another year, and have that commitment and think up a long-term plan where we had more control ... but [we] would have ended up struggling."
|
||||
|
||||
After scouring the market Lush landed on Google’s Cloud Platform. The company was already familiar with Google, having migrated from Scalix to Google Apps (now known as G Suite) in [late 2013][2].
|
||||
|
||||
However, it had less than a few months to make the migration, both in time for the end of its existing contract on December 22 as well as the critical Christmas shopping period.
|
||||
|
||||
“So it wasn’t just a little bit business critical. We were talking peak trade time. It was a huge deal,” Constantine said.
|
||||
|
||||
Lush’s lack of bureaucracy meant Constantine was able to make a quick decision on vendor selection, and “then the team just powered through”, he said.
|
||||
|
||||
They also prioritised optimising the "monolithic" Drupal application specifically for the migration, pushing back bug fixes until later.
|
||||
|
||||
Lush started the physical migration on December 1 and completed it on December 22.
|
||||
|
||||
The team came up against challenges “like with any migration”, Constantine said - “you have to worry about getting your data from one place to another, you have to make sure you have consistency, and customer, product data etc. needs to be up and stable”.
|
||||
|
||||
But the CDO said one thing that got the company through the incredibly tight timeframe was the team’s lack of alternatives: there was no fallback plan.
|
||||
|
||||
“About a week before the deadline my colleague had a conversation with our Google partner on the phone, they were getting a bit nervous about whether this was going to happen, and they asked us what Plan B was. My colleague said ‘Plan B is to make Plan A happen, that’s it’,” Constantine said.
|
||||
|
||||
“When you throw a hard deadline like that it can sound a bit unachieveable, but [you need to keep] that focus on people believing that this is a goal that we can achieve in that timeframe, and not letting people put up the blockers and say ‘we’re going to have to delay this and that’.
|
||||
|
||||
“Yes everybody gets very tense but you achieve a lot. You actually get through it and nail it. All the things you need to get done, get done.”
|
||||
|
||||
The focus now is on moving the commerce application to a microservices architecture, while looking into various Google tools like the Kubernetes container management system and Spanner relational database.
|
||||
|
||||
The retailer also recently built a prototype point-of-sale system using GCP and Android, which it is currently playing around with, Constantine said.
|
||||
|
||||
|
||||
Lush's Oxford St, UK store. Credit: Lush.
|
||||
|
||||
### Lush says it’s possible.
|
||||
|
||||
Migrating your core operations from one public cloud to another in less than one month may seem like a farfetched goal, but British cosmetics giant Lush reckons it can be done.
|
||||
|
||||
Last September Lush - who you might recognise as the company behind the candy-coloured, sweet smelling bath and body products - was nearing the end of its contract with its existing infrastructure provider, understood to be [UK-based Memset][3].
|
||||
|
||||
Memset had been hosting Lush's Drupal-based commerce environment out of Amazon Web Services for a few years, but the retailer wanted out.
|
||||
|
||||
The arrangement was 'awkward' and rigid, according to Lush chief digital officer and heir to the company throne Jack Constantine (his parents founded the business in 1995).
|
||||
|
||||
“We were in a contract that we weren’t really comfortable with, and we wanted to have a look and see what else we could go for,” he told the Google Cloud Next conference in San Francisco today.
|
||||
|
||||
“It was a very closed environment [which] made it difficult for us to get visibility of everything we wanted to be able to move over.
|
||||
|
||||
"[We] could either sign up for another year, and have that commitment and think up a long-term plan where we had more control ... but [we] would have ended up struggling."
|
||||
|
||||
After scouring the market Lush landed on Google’s Cloud Platform. The company was already familiar with Google, having migrated from Scalix to Google Apps (now known as G Suite) in [late 2013][4].
|
||||
|
||||
However, it had less than a few months to make the migration, both in time for the end of its existing contract on December 22 as well as the critical Christmas shopping period.
|
||||
|
||||
“So it wasn’t just a little bit business critical. We were talking peak trade time. It was a huge deal,” Constantine said.
|
||||
|
||||
Lush’s lack of bureaucracy meant Constantine was able to make a quick decision on vendor selection, and “then the team just powered through”, he said.
|
||||
|
||||
They also prioritised optimising the "monolithic" Drupal application specifically for the migration, pushing back bug fixes until later.
|
||||
|
||||
Lush started the physical migration on December 1 and completed it on December 22.
|
||||
|
||||
The team came up against challenges “like with any migration”, Constantine said - “you have to worry about getting your data from one place to another, you have to make sure you have consistency, and customer, product data etc. needs to be up and stable”.
|
||||
|
||||
But the CDO said one thing that got the company through the incredibly tight timeframe was the team’s lack of alternatives: there was no fallback plan.
|
||||
|
||||
“About a week before the deadline my colleague had a conversation with our Google partner on the phone, they were getting a bit nervous about whether this was going to happen, and they asked us what Plan B was. My colleague said ‘Plan B is to make Plan A happen, that’s it’,” Constantine said.
|
||||
|
||||
“When you throw a hard deadline like that it can sound a bit unachieveable, but [you need to keep] that focus on people believing that this is a goal that we can achieve in that timeframe, and not letting people put up the blockers and say ‘we’re going to have to delay this and that’.
|
||||
|
||||
“Yes everybody gets very tense but you achieve a lot. You actually get through it and nail it. All the things you need to get done, get done.”
|
||||
|
||||
The focus now is on moving the commerce application to a microservices architecture, while looking into various Google tools like the Kubernetes container management system and Spanner relational database.
|
||||
|
||||
The retailer also recently built a prototype point-of-sale system using GCP and Android, which it is currently playing around with, Constantine said.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://www.itnews.com.au/news/a-public-cloud-migration-in-22-days-454186
|
||||
|
||||
作者：[Allie Coyne ][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://www.itnews.com.au/author/allie-coyne-461593
|
||||
[1]:http://www.memset.com/about-us/case-studies/lush-cosmetics/
|
||||
[2]:https://cloud.googleblog.com/2013/12/google-apps-helps-eco-cosmetics-company.html
|
||||
[3]:http://www.memset.com/about-us/case-studies/lush-cosmetics/
|
||||
[4]:https://cloud.googleblog.com/2013/12/google-apps-helps-eco-cosmetics-company.html
|
||||
[5]:http://www.memset.com/about-us/case-studies/lush-cosmetics/
|
||||
[6]:https://cloud.googleblog.com/2013/12/google-apps-helps-eco-cosmetics-company.html
|
||||
@@ -1,457 +0,0 @@
|
||||
Translating by trnhoe
|
||||
From Node to Go: A High-Level Comparison
|
||||
============================================================
|
||||
|
||||
At XO Group, we primarily work with Node and Ruby to build out our system of interconnected services. We get the implicit performance benefits of Node mixed with the access to a large, established repository of packages. We also have the ability to easily break out plugins and modules that can be published and reused across the company. This greatly increases developer efficiency and allows us to make scalable and reliable applications in a short amount of time. Furthermore, the large Node community makes it easy for our engineers to contribute open source software (see [BunnyBus][9] or [Felicity][10]).
|
||||
|
||||
Although a good portion of my college days and early career was spent using strict compiled languages, like C++ and C#, I eventually shifted to using Javascript. While I love the freedom and flexibility, I recently found myself nostalgic for static and structured languages. That’s when a coworker turned my attention to Go.
|
||||
|
||||
Coming from Javascript, there are some similarities between the two languages. Both are very fast, fairly easy to learn, have an expressive syntax, and a niche in the development community. There isn’t a perfect programming language and you should always choose a language that fits the project at hand; In this post, I will attempt to illustrate some of the key differences between the two languages at a high level and hopefully encourage anyone new to Go to give it a ̶g̶o̶ chance.
|
||||
|
||||
|
||||
* * *
|
||||
|
||||
### General Differences
|
||||
|
||||
Before we dive into specifics, we should understand some important distinctions between the two languages.
|
||||
|
||||
Go, or Golang, is a free, open-source programming language created by Google in 2007\. It was designed to be fast and simple. Go is compiled down into machine code, which is where its speed is derived from. Debugging is fairly easy with a compiled language because you are able to catch a large chunk of errors early on. It is also a strongly typed language which helps with data integrity and finding type errors at compile time.
|
||||
|
||||
Javascript, on the other hand, is a loosely-typed language. Aside from the added burden of data validation and “truthy” evaluation pitfalls, using a loosely-typed language can can have its own benefits. There is no need for interfaces or generics and currying/flexible arity make functions extremely versatile. Javascript is interpreted at runtime, which can lead to issues with error handling and debugging. Node is a Javascript runtime built on Google’s V8 virtual machine making it a lightweight and fast platform for web development.
|
||||
|
||||
* * *
|
||||
|
||||
### Syntax
|
||||
|
||||
Coming from Javascript, Go’s simple and intuitive syntax was very inviting. Since both languages’ syntaxes are said to have evolved from C, there is quite a bit of overlap. Go is commonly referred to as an ‘easy language to learn.’ This is due to the developer-friendly tools, pared-down syntax, and opinionated conventions.
|
||||
|
||||
Go has a number of built-in features that make development a bit easier. The standard Go build tool lets you compile your code down into a binary file or executable with the go build command. Running tests with the built-in test suite are as simple as calling go test. Things like natively-supported concurrency are even available at the language level.
|
||||
|
||||
According to the [Go developers at Google][11], programming today is too complicated with too much “book keeping, repetition, and clerical work.” This is why the Go’s syntax was designed to be clean and simple in order to reduce clutter, increase efficiency, and improve readability. It also encourages developers to write explicit, easy to understand code. As a result, Go only has [25 unique keywords][12] and one type of loop (for-loop) as opposed to [~84 keywords][13] (reserved words, objects, properties, and methods) in Javascript.
|
||||
|
||||
In order to illustrate some syntactical differences and similarities, let’s look at a couple of examples:
|
||||
|
||||
* Punctuation: Go strips out any superfluous punctuation in order to increase efficiency and readability. Although Javascript’s use of punctuation is somewhat minimal (see: [Lisp][1]) and often optional, I definitely enjoy the simplicity with Go.
|
||||
|
||||
```
|
||||
// Javascript with parentheses and semicolons
|
||||
for (var i = 0; i < 10; i++) {
|
||||
console.log(i);
|
||||
}
|
||||
```
|
||||
|
||||
Punctuation in Javascript
|
||||
|
||||
```
|
||||
// Go uses minimal punctuation
|
||||
for i := 0; i < 10; i++ {
|
||||
fmt.Println(i)
|
||||
}
|
||||
```
|
||||
Punctuation in Go
|
||||
|
||||
* Assignment: Since Go is strongly typed, you have access type inference on initialization with the := operator to reduce [stuttering][2], whereas Javascript declares types on runtime.
|
||||
|
||||
|
||||
```
|
||||
// Javascript assignment
|
||||
var foo = "bar";
|
||||
```
|
||||
|
||||
Assignment in Javascript
|
||||
|
||||
```
|
||||
// Go assignment
|
||||
var foo string //without type derivation
|
||||
foo = "bar"
|
||||
|
||||
foo := "bar" //with type derivation
|
||||
```
|
||||
|
||||
Assignment in Go</figcaption>
|
||||
|
||||
* Exporting: In Javascript, you must explicitly export from a module. In Go, any capitalized functions will be exported.
|
||||
|
||||
|
||||
```
|
||||
const Bar = () => {};
|
||||
|
||||
module.exports = {
|
||||
Bar
|
||||
}
|
||||
```
|
||||
|
||||
Exporting in Javascript
|
||||
|
||||
```
|
||||
// Go export
|
||||
package foo //define package name
|
||||
func Bar (s string) string {
|
||||
//Bar will be exported
|
||||
}
|
||||
```
|
||||
|
||||
Exporting in Go
|
||||
|
||||
* Importing: The _required_ library is necessary for importing dependencies/modules in Javascript, whereas Go utilizes the native import keyword with the import path to the package. Another distinction is that, unlike Node’s central NPM repository for packages, Go uses URLs for the import path on non-standard libraries in order to directly clone dependencies from their origin. Although this provides a simple and intuitive dependency management, versioning packages can be a bit more tedious in Go as opposed to updating the _package.json_ file in Node.
|
||||
```
|
||||
// Javascript import
|
||||
var foo = require('foo');
|
||||
foo.bar();
|
||||
```
|
||||
|
||||
Importing in Javascript
|
||||
|
||||
|
||||
```
|
||||
// Go import
|
||||
import (
|
||||
"fmt" // part of Go’s standard library
|
||||
"github.com/foo/foo" // imported directly from repository
|
||||
)
|
||||
foo.Bar()
|
||||
```
|
||||
|
||||
Importing in Go
|
||||
|
||||
* Returns: Go’s multiple value returns allow for elegantly passing and handling values and errors, as well as reducing the improper passing of values by reference. In Javascript, multiple values must be returned by an array or object.
|
||||
|
||||
```
|
||||
// Javascript - return multiple values
|
||||
function foo() {
|
||||
return {a: 1, b: 2};
|
||||
}
|
||||
const { a, b } = foo();
|
||||
```
|
||||
|
||||
Returns in Javascript
|
||||
|
||||
```
|
||||
// Go - return multiple values
|
||||
func foo() (int, int) {
|
||||
return 1, 2
|
||||
}
|
||||
a, b := foo()
|
||||
```
|
||||
|
||||
Returns in Go
|
||||
|
||||
|
||||
* Errors: Go encourages catching errors often and where they occur as opposed to bubbling the error up in a callback in Node.
|
||||
|
||||
```
|
||||
// Node error handling
|
||||
foo('bar', function(err, data) {
|
||||
//handle error
|
||||
}
|
||||
```
|
||||
|
||||
Errors in Javascript
|
||||
|
||||
|
||||
```
|
||||
//Go error handling
|
||||
foo, err := bar()
|
||||
if err != nil {
|
||||
// handle error with defer, panic, recover, or log.fatal, etc...
|
||||
}
|
||||
```
|
||||
|
||||
Errors in Go
|
||||
|
||||
* Variadic Functions: Both Go and Javascript support functions that accept a fluid number of arguments.
|
||||
|
||||
```
|
||||
function foo (...args) {
|
||||
console.log(args.length);
|
||||
}
|
||||
|
||||
foo(); // 0
|
||||
foo(1, 2, 3); // 3
|
||||
```
|
||||
|
||||
Variadic Function in Javascript
|
||||
|
||||
```
|
||||
func foo (args ...int) {
|
||||
fmt.Println(len(args))
|
||||
}
|
||||
|
||||
func main() {
|
||||
foo() // 0
|
||||
foo(1,2,3) // 3
|
||||
}
|
||||
```
|
||||
|
||||
Variadic Function in Go
|
||||
|
||||
|
||||
* * *
|
||||
|
||||
### Communities
|
||||
|
||||
Although Go and Node have their differences when it comes to which programming paradigms they enable to be easier, they both have unique and supportive followings. One area where Node outshines Go is in the sheer size of their package library and community. Node package manager (NPM), the largest package registry in the world, has over [410,000 packages growing at an alarming rate of 555 new packages per day][14]. That number may seem staggering (and it is), however, something to keep in mind is that many of these packages are redundant and/or non-production quality. In contrast, Go has about 130,000 packages.
|
||||
|
||||
|
||||
|
||||
Module Counts for Node and Go
|
||||
|
||||
Although Node and Go are around the same age, Javascript is more widely used — boasting a large development and open-source community. This is of course because Node was developed for the general public with a robust package manager from the start while Go was specifically built for Google. [The Spectrum ratings][15] below show the top web development languages based on current trends.
|
||||
|
||||
|
||||
|
||||
Spectrum Ratings for top 7 web development programming languages
|
||||
|
||||
While Javascript’s popularity seems to have stayed relatively static over recent years, [Go has been trending up][16].
|
||||
|
||||
|
||||
|
||||
|
||||
Programming language trends
|
||||
|
||||
* * *
|
||||
|
||||
### Performance
|
||||
|
||||
What if your primary concern is speed? In this day and age, it seems performance optimizations are more important than ever. People don’t like to wait for information. In fact, [40% of users will abandon your site if it takes longer than 3 seconds to load][17].
|
||||
|
||||
Node is often touted as a highly performant because of it’s non-blocking asynchronous I/O. Also, as I mentioned before, Node is run on Google’s V8 engine which was optimized for dynamic languages. Go on the other hand was designed with speed in mind. [The developers at Google][18] achieved this by building “an expressive but lightweight type system; concurrency and garbage collection; rigid dependency specification; and so on.”
|
||||
|
||||
To compare the performance of Node and Go, I ran a couple of tests. These focus on the rudimentary, low-level abilities of the languages. If I had been testing something like HTTP requests or time-intensive processes, I would have used Go’s language-level concurrency tools (goroutines/channels). Instead, I stuck to basic features of each language (see [Concurrency in Three Flavors][19] for a deeper look into goroutines and channels).
|
||||
|
||||
I also included Python in the benchmarks so we feel good about the Node and Go results no matter what.
|
||||
|
||||
#### Loop/Arithmetic
|
||||
|
||||
Iterating through a billion items and adding them up:
|
||||
|
||||
```
|
||||
var r = 0;
|
||||
for (var c = 0; c < 1000000000; c++) {
|
||||
r += c;
|
||||
}
|
||||
```
|
||||
|
||||
Node
|
||||
|
||||
```
|
||||
package main
|
||||
func main() {
|
||||
var r int
|
||||
for c := 0; c < 1000000000; c++ {
|
||||
r += c
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
go
|
||||
|
||||
|
||||
```
|
||||
sum(xrange(1000000000))
|
||||
```
|
||||
|
||||
Python
|
||||
|
||||
|
||||
|
||||
Results
|
||||
|
||||
The clear loser here is Python clocking in at over 7 seconds. On the other hand, both Node and Go were extremely efficient, clocking in at 900 ms and 408 ms, respectively.
|
||||
|
||||
_Edit: As some of the comments suggest, Python’s performance could be improved. The results have been updated to reflect those changes. Also, the use of PyPy greatly improves the performance. When run using Python 3.6.1 and PyPy 3.5.7, the performance improves to 1.234 seconds, but still falls short of Go and Node._
|
||||
|
||||
#### I/O
|
||||
|
||||
Iterating over 1 million numbers and writing them to a file:
|
||||
|
||||
|
||||
```
|
||||
var fs = require('fs');
|
||||
var wstream = fs.createWriteStream('node');
|
||||
|
||||
for (var c = 0; c < 1000000; ++c) {
|
||||
wstream.write(c.toString());
|
||||
}
|
||||
wstream.end();
|
||||
```
|
||||
|
||||
Node
|
||||
|
||||
```
|
||||
package main
|
||||
|
||||
import (
|
||||
"io"
|
||||
"os"
|
||||
"strconv"
|
||||
)
|
||||
|
||||
func main() {
|
||||
file, _ := os.Create("go")
|
||||
for c := 0; c < 1000000; c++ {
|
||||
num := strconv.Itoa(c)
|
||||
io.WriteString(file, num)
|
||||
}
|
||||
file.Close()
|
||||
}
|
||||
```
|
||||
|
||||
go
|
||||
|
||||
```
|
||||
with open("python", "a") as text_file:
|
||||
for i in range(1000000):
|
||||
text_file.write(str(i))
|
||||
```
|
||||
|
||||
Python
|
||||
|
||||
|
||||
|
||||
Results
|
||||
|
||||
Once again, Python is third at 7.94 seconds. The gap between Node and Go is small in this test, with Node taking about 1.164 seconds and Go taking 1.477 seconds (although this includes the time it takes for the Go code to compile via go run — the compiled binary shaves off another ~200 ms).
|
||||
|
||||
#### Bubble Sort
|
||||
|
||||
Iterating 10 million times over a 10-item array and sorting:
|
||||
|
||||
```
|
||||
const toBeSorted = [1, 3, 2, 4, 8, 6, 7, 2, 3, 0];
|
||||
|
||||
function bubbleSort(input) {
|
||||
var n = input.length;
|
||||
var swapped = true;
|
||||
while (swapped) {
|
||||
swapped = false;
|
||||
for (var i = 0; i < n; i++) {
|
||||
if (input[i - 1] > input [i]) {
|
||||
[input[i], input[i - 1]] = [input[i - 1], input[i]];
|
||||
swapped = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (var c = 0; c < 10000000; c++) {
|
||||
bubbleSort(toBeSorted);
|
||||
}
|
||||
```
|
||||
|
||||
Node
|
||||
|
||||
```
|
||||
package main
|
||||
|
||||
var toBeSorted [10]int = [10]int{1, 3, 2, 4, 8, 6, 7, 2, 3, 0}
|
||||
|
||||
func bubbleSort(input [10]int) {
|
||||
n := len(input)
|
||||
swapped := true
|
||||
for swapped {
|
||||
swapped = false
|
||||
for i := 1; i < n; i++ {
|
||||
if input[i-1] > input[i] {
|
||||
input[i], input[i-1] = input[i-1], input[i]
|
||||
swapped = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func main() {
|
||||
for c := 0; c < 10000000; c++ {
|
||||
bubbleSort(toBeSorted)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
go
|
||||
|
||||
|
||||
```
|
||||
toBeSorted = [1, 3, 2, 4, 8, 6, 7, 2, 3, 0]
|
||||
|
||||
def bubbleSort(input):
|
||||
length = len(input)
|
||||
swapped = True
|
||||
|
||||
while swapped:
|
||||
swapped = False
|
||||
for i in range(1,length):
|
||||
if input[i - 1] > input[i]:
|
||||
input[i], input[i - 1] = input[i - 1], input[i]
|
||||
swapped = True
|
||||
|
||||
for i in range(10000000):
|
||||
bubbleSort(toBeSorted)
|
||||
```
|
||||
Python
|
||||
|
||||
|
||||
|
||||
|
||||
<figcaption class="imageCaption" style="position: relative; left: 0px; width: 700px; top: 0px; margin-top: 10px; color: rgba(0, 0, 0, 0.6); outline: 0px; text-align: center; z-index: 300; --baseline-multiplier:0.157; font-family: medium-content-sans-serif-font, "Lucida Grande", "Lucida Sans Unicode", "Lucida Sans", Geneva, Arial, sans-serif; font-feature-settings: 'liga' 1, 'lnum' 1; font-size: 14px; line-height: 1.4; letter-spacing: 0px;">Results</figcaption>
|
||||
|
||||
As usual, Python’s performance was the poorest, completing the task at hand in about 13 seconds. Go was able to finish the task over two times faster than Node.
|
||||
|
||||
#### Verdict
|
||||
|
||||
Go is the clear winner in all three tests, but Node, for the most part, performs admirably. And Python was there, too. To be clear, performance isn’t everything when choosing a programming language. If your application doesn’t need to process high amounts data, then the differences in performance between Node and Go may be negligible. For some additional comparisons on performance, see the following:
|
||||
|
||||
* [Node Vs. Go][3]
|
||||
* [Multiple Language Performance Test][4]
|
||||
* [Benchmarks Game][5]
|
||||
|
||||
|
||||
* * *
|
||||
|
||||
### Conclusion
|
||||
|
||||
This post is not to prove that one language is better than another. Every programming language has its place in the software development community for one reason or another. My intentions were to highlight the differences between Go and Node, as well as promote exposure to a new web development language. When choosing a language for a given project, there are a lot of different factors to consider including developer familiarity, cost, and practicality. I encourage a thorough low-level analysis when deciding what language is right for you.
|
||||
|
||||
As we have seen, there are several benefits to Go. The raw performance, simple syntax, and relatively shallow learning curve make it ideal for scalable and secure web applications. With it’s fast growth in adoption and community involvement, there is no reason Go can’t become a prominent player in modern web development. That being said, I believe that Node is moving in the right direction to remain a powerful and useful language if implemented correctly. It has a large following and active community that makes it a simple platform for getting a web application up and running in no time.
|
||||
|
||||
* * *
|
||||
|
||||
### Resources
|
||||
|
||||
If you are interested in learning more about Go, consider the following resources:
|
||||
|
||||
* [Golang Website][6]
|
||||
* [Golang Wiki][7]
|
||||
* [Golang Subreddit][8]
|
||||
|
||||
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://medium.com/xo-tech/from-node-to-go-a-high-level-comparison-56c8b717324a#.byltlz535
|
||||
|
||||
作者：[John Stamatakos][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://medium.com/@johnstamatakos?source=post_header_lockup
|
||||
[1]:https://en.wikipedia.org/wiki/Lisp_%28programming_language%29
|
||||
[2]:https://golang.org/doc/faq#principles
|
||||
[3]:https://jaxbot.me/articles/node-vs-go-2014
|
||||
[4]:https://hashnode.com/post/comparison-nodejs-php-c-go-python-and-ruby-cio352ydg000ym253frmfnt70
|
||||
[5]:https://benchmarksgame.alioth.debian.org/u64q/compare.php?lang=go&lang2=node
|
||||
[6]:https://golang.org/doc/#learning
|
||||
[7]:https://github.com/golang/go/wiki/Learn
|
||||
[8]:https://www.reddit.com/r/golang/
|
||||
[9]:https://medium.com/xo-tech/bunnybus-building-a-data-transit-system-b9647f6283e5#.l64fdvfys
|
||||
[10]:https://medium.com/xo-tech/introducing-felicity-7b6d0b734ce#.hmloiiyx8
|
||||
[11]:https://golang.org/doc/faq
|
||||
[12]:https://golang.org/ref/spec#Keywords
|
||||
[13]:https://www.w3schools.com/js/js_reserved.asp
|
||||
[14]:http://www.modulecounts.com/
|
||||
[15]:http://spectrum.ieee.org/static/interactive-the-top-programming-languages-2016
|
||||
[16]:http://www.tiobe.com/tiobe-index/
|
||||
[17]:https://hostingfacts.com/internet-facts-stats-2016/
|
||||
[18]:https://golang.org/doc/faq
|
||||
[19]:https://medium.com/xo-tech/concurrency-in-three-flavors-51ed709876fb#.khvqrttxa
|
||||
@@ -1,175 +0,0 @@
|
||||
GHLandy Translating
|
||||
|
||||
LFCS sed Command
|
||||
=====================
|
||||
|
||||
Another useful command for the Linux Foundation Certified SysAdmin (LFCS) is the 'sed' Command. The initials stand for 'Streaming EDitor'.
|
||||
|
||||
The 'sed' Command is an editor that edits the file as a stream. To stream a file is to pass it from another command using pipes (> or |) or load it directly into 'sed'.
|
||||
|
||||
The command works like any other editor except that the file is not displayed and allowed to be edited visually. Commands are passed to 'sed' to manipulate the stream.
|
||||
|
||||
With 'sed' there are five basic things which can be done. Of course, with 'sed' being so powerful there are more advanced functions, but you only need to concentrate on the basic five. The five types of functionality are as follows:
|
||||
|
||||
1. Search
|
||||
|
||||
2. Replace
|
||||
|
||||
3. Delete
|
||||
|
||||
4. Adding
|
||||
|
||||
5. Change/Transform
|
||||
|
||||
Before getting into the command parameters we need to look over the basic syntax.
|
||||
|
||||
**Syntax**
|
||||
|
||||
The syntax for the 'sed' command is:
|
||||
|
||||
_sed [options] commands [file-to-edit]_
|
||||
|
||||
The options will be covered in this article in the appropriate sections. The commands are the search and replacement patterns which can be Regex Expressions. Read on to see how 'sed' works and learn the basic commands. As I mentioned before, 'sed' is a very powerful tool and has more options available than I will cover in the article.
|
||||
|
||||
**Example File**
|
||||
|
||||
If you open a Terminal you can create a file to be used for the 'sed' examples. Perform the following commands:
|
||||
|
||||
_cd ~
|
||||
grep --help >grephelp.txt_
|
||||
|
||||
You should now have a file named 'grephelp.txt' in your HOME folder. The contents of the file are the help instructions for the command 'grep'.
|
||||
|
||||
**Search**
|
||||
|
||||
Searching for specific strings is a common ability of editors and performing searches in 'sed' is no exception.
|
||||
|
||||
Searches can be performed to find a string in a file. Let's look at a basic search.
|
||||
|
||||
If we wanted to search through our example file for the word 'PATTERN' we would use the command and see the results in Figure 1:
|
||||
|
||||
_sed -n 's/PATTERN/PATTERN/p' grephelp.txt_
|
||||
|
||||
**NOTE:** If you cut and paste the command make sure you replace the single quotes with that of a standard single quote on your keyboard.
|
||||
|
||||
|
||||
|
||||
**FIGURE 1**
|
||||
|
||||
The parameter '-n' is used to suppress the printing of each line automatically. These lines include lines which do not include the search pattern. By using the '-n' the lines with a matching result will be displayed. Every line which is streamed into 'sed' will be printed to the standard out (stdout). If you run the above command without the '-n' option you will have a line for every line in the original file as well as a duplicate line for each match.
|
||||
|
||||
The file name to search is 'grephelp.txt' which we created in the 'Example File' section.
|
||||
|
||||
The portion left is _'s/PATTERN/PATTERN/p'_ . There are basically four sections to this section. The first section 's' specifies to perform a substitution, or a search and replace.
|
||||
|
||||
The second and third part of the remaining portion are the patterns. The first is the pattern to search for and the last is the pattern to replace the matching string within the stream. In this case we are finding the string 'PATTERN' and replacing it with 'PATTERN'. By finding and replacing the same string we are not changing the file at all, even on the screen.
|
||||
|
||||
The last command is 'p'. The 'p' command specifies to print the new line after the substitution is made. Of course, there is no change since the substitution is the same string. Since we are suppressing printing lines with ‘-n’ parameter only the changed lines will be printed with the ‘p’ command.
|
||||
|
||||
The whole command allows us to perform a search and see the matching results.
|
||||
|
||||
**Replace**
|
||||
|
||||
When searching for specific strings you may want to replace the matching string with a new one. Replacing strings with another is very common.
|
||||
|
||||
We can perform the same search with the following command and the results are shown in Figure 2:
|
||||
|
||||
_sed -n 's/PATTERN/Pattern/p' grephelp.txt_
|
||||
|
||||
|
||||
|
||||
**FIGURE 2**
|
||||
|
||||
In this instance, the string 'PATTERN' is changed to 'Pattern' and displayed. If you view the file with the command 'cat grephelp.txt' you will see that the file has not changed. The change was made only to the output on the screen. You could pipe the output to another file with the command:
|
||||
|
||||
_sed 's/PATTERN/Pattern/' grephelp.txt > grephelp1.txt_
|
||||
|
||||
A new file called 'grephelp1.txt' would now exist which held the changed file. If the 'p' was left in as the fourth option then the problem is that each line that had a string substituted would exist twice in the file. We also drop the '-n' parameter to allow all the lines to print.
|
||||
|
||||
Another way to replace the string with the same string is to use the ‘&’ symbol to represent the search string. For example, the command ‘s/PATTERN/&/p’ would be the same. We can add to the string, such as adding an ‘S’, with the command ‘s/PATTERN/&S/p’.
|
||||
|
||||
What if we wish to replace only a certain pattern in each line? It is possible to specify the specific occurrence of a match to replace. Of course, the replacement will be a specific numbered occurrence on each line. For example, there are a lot of dashes on the sample file. Some lines have at least two dashes so we can replace the second dash in each line with another character. The command to replace the second dash (-) in each line with an asterisk (*) would be:
|
||||
|
||||
sed 's/-/*/2' grephelp.txt
|
||||
|
||||
Here, we are performing a substitution with the initial ‘s’. The character ‘-’ is replaced with the ‘*’. The ‘2’ shows we are wanting to replace the second instance of the ‘-’ on each line if it exists. A sample result is shown in Figure 3\. If we left out the command ‘2’ then the first occurrence of a dash is replaced. Not every dash of a line is replaced only the first one.
|
||||
|
||||
|
||||
|
||||
**FIGURE 3**
|
||||
|
||||
If you want to search and replace all dashes on a line with an asterisk, use the ‘g’ command:
|
||||
|
||||
_sed 's/-/*/g' grephelp.txt_
|
||||
|
||||
Commands can also be combined. Let’s say you want to change all dashes starting at the second occurrence to the end, the command would be:
|
||||
|
||||
_sed 's/-/*/2g' grephelp.txt_
|
||||
|
||||
Now every dash from the second to the last on every line is replaced with an asterisk.
|
||||
|
||||
**Delete**
|
||||
|
||||
Many times during a search you may want to remove the search string completely.
|
||||
|
||||
For instance, if you wanted to remove all dashes from the file you could use the command:
|
||||
|
||||
_sed ‘s/-//g’ grephelp.txt_
|
||||
|
||||
The replacement string is left as a blank so the matching string is removed or deleted.
|
||||
|
||||
**Adding**
|
||||
|
||||
When a match is found you can add a line of specific text to make the line stand out for viewing or printing.
|
||||
|
||||
If you want the new line to be inserted after the match use the ‘a’ command followed by the string for the new line. Also include the string to match. For example, we can find a ‘--’ and add a line after the matched line. The string on the added line will be ‘double dash before this line’.
|
||||
|
||||
_sed '/--/ a "double dash before this line"' grephelp.txt_
|
||||
|
||||
If you want to place the line before the line containing the matched string use the ‘i’ command as follows:
|
||||
|
||||
_sed '/--/ i "double dash after this line"' grephelp.txt_
|
||||
|
||||
**Change/Transform**
|
||||
|
||||
If a line needs to be changed or transformed you can use the command ‘c’.
|
||||
|
||||
Let’s say we have a document which has some private information and we need to change any line which contains a specific string. The ‘c’ command will change the whole line and not just the search string.
|
||||
|
||||
Let’s say in our example file we want to block out every line which contains the word ‘PATTERN’. The changed line will read ‘This line is Top Secret’. The command is:
|
||||
|
||||
_sed ‘/PATTERN/ c This line is Top Secret’ grephelp.txt_
|
||||
|
||||
A Transformation can be performed to change case of specific letters. For instance, we can change all lower-case ‘a’ to uppercase ‘A’ with the command ‘y’ as follows:
|
||||
|
||||
_sed ‘y/a/A/’ grephelp.txt_
|
||||
|
||||
Multiple letters can be specified such as ‘abdg’ as in the following command:
|
||||
|
||||
_sed ‘y/abdg/ABDG/’ grephelp.txt_
|
||||
|
||||
Make sure the second set of letters are in the same order as the first or they can be replaced and transformed. For example, the string ‘y/a/D/’ would replace all lower-case ‘a’ with an upper-case ‘D’.
|
||||
|
||||
**In-place change**
|
||||
|
||||
If you actually want to make changes to the file you are using, use the ‘-i’ option.
|
||||
|
||||
For example, to change the word ‘PATTERN’ to ‘Pattern’ and have the changes made to the file, the command would be:
|
||||
|
||||
sed -i 's/PATTERN/Pattern/' grephelp.txt
|
||||
|
||||
The file ‘grephelp.txt’ will now be changed. The ‘-i’ option can be used with any of the above commands to change the original file content.
|
||||
|
||||
Practice with these commands and make sure you understand them. The ‘sed’ command is very powerful.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://www.linux.org/threads/lfcs-sed-command.4561/
|
||||
|
||||
作者：[Jarret B ][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://www.linux.org/threads/lfcs-sed-command.4561/
|
||||
@@ -1,4 +1,3 @@
|
||||
translated by zhousiyu325
|
||||
What all you need to know about HTML5
|
||||
============================================================
|
||||
|
||||
|
||||
@@ -1,111 +0,0 @@
|
||||
翻译中 by zky001
|
||||
# [Open source social robot kit runs on Raspberry Pi and Arduino][22]
|
||||
|
||||
|
||||
|
||||
|
||||
Thecorpora’s Scratch-ready “Q.bo One” robot is based on the RPi 3 and Arduino, and offers stereo cams, mics, a speaker, and visual and language recognition.
|
||||
|
||||
In 2010, robotics developer Francisco Paz and his Barcelona-based Thecorpora startup introduced the first [Qbo][6] “Cue-be-oh” robot as an open source proof-of-concept and research project for exploring AI capabilities in multi-sensory, interactive robots. Now, after a preview in February at Mobile World Congress, Thecorpora has gone to Indiegogo to launch the first mass produced version of the social robot in partnership with Arrow.
|
||||
|
||||
|
||||
[][7] [][8]
|
||||
**Q.bo One from angle (left) and top**
|
||||
|
||||
|
||||
Like the original, the new Q.bo One has a spherical head with eyes (dual stereoscopic cameras), ears (3x mics), and mouth (speakers), and is controlled by WiFi and Bluetooth. The Q.bo One also similarly features open source Linux software and open spec hardware. Instead of using an Intel Atom-based Mini-ITX board, however, it runs Raspbian on a Raspberry Pi 3 linked to an Arduino compatible mainboard.
|
||||
|
||||
|
||||
[][9]
|
||||
**Q.bo One side views**
|
||||
|
||||
|
||||
The Q.bo One is available on Indiegogo through mid-July starting at $369 (early bird) or $399 in kit form including its built-in Raspberry Pi 3 and Arduino-based “Qboard” controller board. It also sells for $499 fully assembled. The Indiegogo campaign is currently about 15 percent toward its flexible $100,000 goal, and shipments are due in December.
|
||||
|
||||
More proficient roboticists and embedded developers may instead want the $99 package with just the RPi and Qboard PCBs and software, or the $249 version, which gives you the robot kit without the boards. With this package, you could replace the Qboard with your own Arduino controller, and swap out the RPi 3 for another Linux SBC. Thecorpora lists the Banana Pi, BeagleBone, Tinker Board, and [soon to be retired Intel Edison][10], as examples of compatible alternatives.
|
||||
|
||||
<center>
|
||||
[][11]
|
||||
**Q.bo One kit**
|
||||
(click image to enlarge)
|
||||
</center>
|
||||
|
||||
Unlike the 2010 Qbo, the Q.bo One is not mobile aside from its spherical head, which swivels in its base with the help of dual servos in order to track voices and motion. The Robotis Dynamixel servos, which are also found in the open source, Raspberry Pi based [TurtleBot 3][23] robot kit, can move up and down in addition to left and right.
|
||||
|
||||
<center>
|
||||
[][12] [][13]
|
||||
**Q.bo One detail view (left) and Qboard detail**
|
||||
(click images to enlarge)
|
||||
</center>
|
||||
|
||||
The Q.bo One can also be compared with the similarly stationary, Linux-based [Jibo][24] “social robot,” which launched on Indiegogo in 2014 to the tune of $3.6 million. The Jibo has yet to ship, however, with the [latest delays][25] pushing it toward a release sometime this year.
|
||||
|
||||
|
|
||||
|
||||
|
||||
**Q.bo One** |
|
||||
|
||||
We’ll go out on a limb and predict the Q.bo One will ship closer to its Dec. 2017 target. The core technology and AI software has been proven, and so are the Raspberry Pi and Arduino technologies. The Qboard mainboard has already been built and certified for manufacturing by Arrow.
|
||||
|
||||
The open source design suggests that even a mobile version wouldn’t be out of the question. That would make it more like the rolling, humanoid [Pepper][14], a similarly AI-infused conversational robot from Softbank and Aldeberan.
|
||||
|
||||
The Q.bo One has added a few tricks since the original, such as a “mouth” formed by 20 LEDs that light up in different, programmable patterns to mimic lips moving during speech. There are also three touch sensors around head if you want to tap the bot to get its attention. But all you really need to do is speak, and the Q.bo One will swivel and gaze adoringly at you like a cocker spaniel.
|
||||
|
||||
Interfaces include everything you have on the Raspberry Pi 3, which just demolished the competition in our [2017 hacker board survey][15]. An antenna mount is provided for the RPi 3’s WiFi and Bluetooth radios.
|
||||
|
||||
<center>
|
||||
[][16] [][17]
|
||||
**Q.bo One software architecture (left) and Q.bo One with Scratch screen**
|
||||
(click images to enlarge)
|
||||
</center>
|
||||
|
||||
The Qboard (also referred to as the Q.board) runs Arduino code on an Atmel ATSAMD21 MCU, and houses the three microphones, speaker, touch sensors, Dynamixel controller, and the LED matrix for the mouth. Other features include GPIO, an I2C interface, and a micro-USB port that can connect to a desktop computer.
|
||||
|
||||
The Q.bo One can recognize faces and track movements, and the bot can even recognize itself in a mirror. With the help of a cloud connection, the robot can recognize and converse with other Q.bo One bots. The robot can respond to questions with the help of natural language processing, and read aloud with text-to-speech
|
||||
|
||||
Scratch programming is available, enabling the robot’s main function, which is to teach kids about robots and programming. The robot is also designed for educators and makers, and can act as a companion to the elderly.
|
||||
|
||||
The Raspbian based software uses OpenCV for vision processing, and can be programmed with a wide variety of languages including C++. The software also offers hooks to IBM Bluemix, NodeRED, and ROS. Presumably, you could also integrate an [Alexa][18] or [Google Assistant][19] voice agent, although Thecorpora makes no mention of this.
|
||||
|
||||
|
||||
|
||||
**Further information**
|
||||
|
||||
The Q.bo One is available on Indiegogo through mid-July starting at $369 for the full kit and $499 fully assembled. Shipments are expected in Dec. 2017\. More information may be found on the [Q.bo One Indiegogo page][20] and [Thecorpora website][21].
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: http://linuxgizmos.com/open-source-social-robot-kit-runs-on-raspberry-pi-and-arduino/
|
||||
|
||||
作者：[ Eric Brown][a]
|
||||
译者：[zky001](https://github.com/zky001)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:http://linuxgizmos.com/open-source-social-robot-kit-runs-on-raspberry-pi-and-arduino/
|
||||
[1]:http://twitter.com/share?url=http://linuxgizmos.com/open-source-social-robot-kit-runs-on-raspberry-pi-and-arduino/&text=Open+source+social+robot+kit+runs+on+Raspberry+Pi+and+Arduino+
|
||||
[2]:https://plus.google.com/share?url=http://linuxgizmos.com/open-source-social-robot-kit-runs-on-raspberry-pi-and-arduino/
|
||||
[3]:http://www.facebook.com/sharer.php?u=http://linuxgizmos.com/open-source-social-robot-kit-runs-on-raspberry-pi-and-arduino/
|
||||
[4]:http://www.linkedin.com/shareArticle?mini=true&url=http://linuxgizmos.com/open-source-social-robot-kit-runs-on-raspberry-pi-and-arduino/
|
||||
[5]:http://reddit.com/submit?url=http://linuxgizmos.com/open-source-social-robot-kit-runs-on-raspberry-pi-and-arduino/&title=Open%20source%20social%20robot%20kit%20runs%20on%20Raspberry%20Pi%20and%20Arduino
|
||||
[6]:http://linuxdevices.linuxgizmos.com/open-source-robot-is-all-eyes/
|
||||
[7]:http://linuxgizmos.com/files/thecorpora_qboone.jpg
|
||||
[8]:http://linuxgizmos.com/files/thecorpora_qboone2.jpg
|
||||
[9]:http://linuxgizmos.com/files/thecorpora_qboone_side.jpg
|
||||
[10]:http://linuxgizmos.com/intel-pulls-the-plug-on-its-joule-edison-and-galileo-boards/
|
||||
[11]:http://linuxgizmos.com/files/thecorpora_qboone_kit.jpg
|
||||
[12]:http://linuxgizmos.com/files/thecorpora_qboone_detail.jpg
|
||||
[13]:http://linuxgizmos.com/files/thecorpora_qboone_qboard.jpg
|
||||
[14]:http://linuxgizmos.com/worlds-first-emotional-robot-runs-linux/
|
||||
[15]:http://linuxgizmos.com/2017-hacker-board-survey-raspberry-pi-still-rules-but-x86-sbcs-make-gains/
|
||||
[16]:http://linuxgizmos.com/files/thecorpora_qboone_arch.jpg
|
||||
[17]:http://linuxgizmos.com/files/thecorpora_qboone_scratch.jpg
|
||||
[18]:http://linuxgizmos.com/how-to-add-alexa-to-your-raspberry-pi-3-gizmo/
|
||||
[19]:http://linuxgizmos.com/free-raspberry-pi-voice-kit-taps-google-assistant-sdk/
|
||||
[20]:https://www.indiegogo.com/projects/q-bo-one-an-open-source-robot-for-everyone#/
|
||||
[21]:http://thecorpora.com/
|
||||
[22]:http://linuxgizmos.com/open-source-social-robot-kit-runs-on-raspberry-pi-and-arduino/
|
||||
[23]:http://linuxgizmos.com/ubuntu-driven-turtlebot-gets-a-major-rev-with-a-pi-or-joule-in-the-drivers-seat/
|
||||
[24]:http://linuxgizmos.com/cheery-social-robot-owes-it-all-to-its-inner-linux/
|
||||
[25]:https://www.slashgear.com/jibo-delayed-to-2017-as-social-robot-hits-more-hurdles-20464725/
|
||||
@@ -1,178 +0,0 @@
|
||||
translating by cycoe
|
||||
|
||||
18 open source translation tools to localize your project
|
||||
============================================================
|
||||
|
||||
### Localization plays a key role in adapting projects for users around the world.
|
||||
|
||||
|
||||
|
||||
>Image by : opensource.com
|
||||
|
||||
Localization plays a central role in the ability to customize an open source project to suit the needs of users around the world. Besides coding, language translation is one of the main ways people around the world contribute to and engage with open source projects.
|
||||
|
||||
There are tools specific to the language services industry (surprised to hear that's a thing?) that enable a smooth localization process with a high level of quality. Categories that localization tools fall into include:
|
||||
|
||||
* Computer-assisted translation (CAT) tools
|
||||
|
||||
* Machine translation (MT) engines
|
||||
|
||||
* Translation management systems (TMS)
|
||||
|
||||
* Terminology management tools
|
||||
|
||||
* Localization automation tools
|
||||
|
||||
The proprietary versions of these tools can be quite expensive. A single license for SDL Trados Studio (the leading CAT tool) can cost thousands of euros, and even then it is only useful for one individual and the customizations are limited (and psst, they cost more, too). Open source projects looking to localize into many languages and streamline their localization processes will want to look at open source tools to save money and get the flexibility they need with customization. I've compiled this high-level survey of many of the open source localization tool projects out there to help you decide what to use.
|
||||
|
||||
### Computer-assisted translation (CAT) tools
|
||||
|
||||
### [omegat_cat.png][1]
|
||||
|
||||
|
||||
|
||||
OmegaT CAT tool. Here you see the translation memory (Fuzzy Matches) and terminology recall (Glossary) features at work. OmegaT is licensed under the GNU Public License version 3+.
|
||||
|
||||
CAT tools are a staple of the language services industry. As the name implies, CAT tools help translators perform the tasks of translation, bilingual review, and monolingual review as quickly as possible and with the highest possible consistency through reuse of translated content (also known as translation memory). Translation memory and terminology recall are two central features of CAT tools. They enable a translator to reuse previously translated content from old projects in new projects. This allows them to translate a high volume of words in a shorter amount of time while maintaining a high level of quality through terminology and style consistency. This is especially handy for localization, as text in a lot of software and web UIs is often the same across platforms and applications. CAT tools are standalone pieces of software though, requiring translators that use them to work locally and merge to a central repository.
|
||||
|
||||
**Tools to check out:**
|
||||
|
||||
* [OmegaT][7]
|
||||
|
||||
* [OmegaT+][8]
|
||||
|
||||
* [OpenTM2][9]
|
||||
|
||||
* [Anaphraseus][10]
|
||||
|
||||
* [Subtitles Translator][11]
|
||||
|
||||
### Machine translation (MT) engines
|
||||
|
||||
|
||||
|
||||
MT engines automate the transfer of text from one language to another. MT is broken up into three primary methodologies: rules-based, statistical, and neural (which is the new player). The most widespread MT methodology is statistical, which (in very brief terms) draws conclusions about the interconnectedness of a pair of languages by running statistical analyses over annotated bilingual corpus data using [_n_ -gram models][29]. When a new source language phrase is introduced to the engine for translation, it looks within its analyzed corpus data to find statistically relevant equivalents, which it produces in the target language. MT can be useful as a productivity aid to translators, changing their primary task from translating a source text to a target text to post-editing the MT engine's target language output. I don't recommend using raw MT output in localizations, but if your community is trained in the art of post-editing, MT can be a useful tool to help them make large volumes of contributions.
|
||||
|
||||
**Tools to check out:**
|
||||
|
||||
* [Apertium][12]
|
||||
|
||||
* [Moses][13]
|
||||
|
||||
### Translation management systems (TMS)
|
||||
|
||||
### [mozilla_pontoon.png][2]
|
||||
|
||||
|
||||
|
||||
Mozilla's Pontoon translation management system user interface. With WYSIWYG editing, you can translate content in context and simultaneously perform translation and quality assurance. Pontoon is licensed under the BSD 3-clause New or Revised License.
|
||||
|
||||
TMS tools are web-based platforms that allow you to manage a localization project and enable translators and reviewers to do what they do best. Most TMS tools aim to automate many manual parts of the localization process by including version control system (VCS) integrations, cloud services integrations, project reporting, as well as the standard translation memory and terminology recall features. These tools are most amenable to community localization or translation projects, as they allow large groups of translators and reviewers to contribute to a project. Some also use a WYSIWYG editor to give translators context for their translations. This added context improves translation accuracy and cuts down on the amount of time a translator has to wait between doing the translation and reviewing the translation within the user interface.
|
||||
|
||||
**Tools to check out**
|
||||
|
||||
* [Pontoon][14]
|
||||
|
||||
* [Pootle][15]
|
||||
|
||||
* [Weblate][16]
|
||||
|
||||
* [Translate5][17]
|
||||
|
||||
* [GlobalSight][18]
|
||||
|
||||
* [Zanata][19]
|
||||
|
||||
* [Jabylon][20]
|
||||
|
||||
### Terminology management tools
|
||||
|
||||
### [baseterm_term_entry_example.png][3]
|
||||
|
||||
|
||||
|
||||
Brigham Young University's BaseTerm tool displays the new-term entry dialogue window. BaseTerm is licensed under the Eclipse Public License.
|
||||
|
||||
Terminology management tools give you a GUI to create terminology resources (known as termbases) to add context and ensure translation consistency. These resources are consumed by CAT tools and TMS platforms to aid translators in the process of translation. For languages in which a term could be either a noun or a verb based on the context, terminology management tools allows you to add metadata for a term that labels its gender, part of speech, monolingual definition, as well as context clues. Terminology management is often an underserved, but no less important, part of the localization process. In both the open source and proprietary ecosystems, there are only a small handful of options available.
|
||||
|
||||
**Tools to check out**
|
||||
|
||||
* [BaseTerm][21]
|
||||
|
||||
* [Terminator][22]
|
||||
|
||||
### Localization automation tools
|
||||
|
||||
### [okapi_framework.jpg][4]
|
||||
|
||||
|
||||
|
||||
The Ratel and Rainbow components of the Okapi Framework. Photo courtesy of the Okapi Framework. The Okapi Framework is licensed under the Apache License version 2.0.
|
||||
|
||||
Localization automation tools facilitate the way you process localization data. This can include text extraction, file format conversion, tokenization, VCS synchronization, term extraction, pre-translation, and various quality checks over common localization standard file formats. In some tool suites, like the Okapi Framework, you can create automation pipelines for performing various localization tasks. This can be very useful for a variety of situations, but their main utility is in the time they save by automating many tasks. They can also move you closer to a more continuous localization process.
|
||||
|
||||
**Tools to check out**
|
||||
|
||||
* [Okapi Framework][23]
|
||||
|
||||
* [Mojito][24]
|
||||
|
||||
### Why open source is key
|
||||
|
||||
Localization is most powerful and effective when done in the open. These tools should give you and your communities the power to localize your projects into as many languages as humanly possible.
|
||||
|
||||
Want to learn more? Check out these additional resources:
|
||||
|
||||
* [Free/open-source machine translation software][25] list
|
||||
|
||||
* _[Open Translation Tools][5]_ e-book
|
||||
|
||||
_Jeff Beatty will be talking about [open source localization tools][26] at [OpenWest][27], which will be held July 12-15 in Salt Lake City._
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
作者简介：
|
||||
|
||||
Jeff Beatty - Jeff Beatty is the Head of Localization at Mozilla, the makers of the popular open source web browser, Firefox. He holds a MSc in Multilingual Computing and Localisation from the University of Limerick. Jeff has also been featured as a localization expert in prominent global publications, such as The Economist, El Universal, Multilingual Magazine and others. Jeff aims to showcase Mozilla's localization program, create disruptive, open source translation technology, and serve as intermediary
|
||||
|
||||
-----------
|
||||
|
||||
via: https://opensource.com/article/17/6/open-source-localization-tools
|
||||
|
||||
作者：[Jeff Beatty][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://opensource.com/users/guerojeff
|
||||
[1]:https://opensource.com/file/357341
|
||||
[2]:https://opensource.com/file/357331
|
||||
[3]:https://opensource.com/file/357326
|
||||
[4]:https://opensource.com/file/357336
|
||||
[5]:https://booki.flossmanuals.net/open-translation-tools/index
|
||||
[6]:https://opensource.com/article/17/6/open-source-localization-tools?rate=QVxhNMYU2Rzrul0hmvzCTqOUinduQ8Au5L8sT8bVbFk
|
||||
[7]:http://www.omegat.org/
|
||||
[8]:http://omegatplus.sourceforge.net/
|
||||
[9]:http://opentm2.org/
|
||||
[10]:http://anaphraseus.sourceforge.net/
|
||||
[11]:http://www.mironto.sk/
|
||||
[12]:http://www.apertium.org/
|
||||
[13]:http://www.statmt.org/moses/
|
||||
[14]:http://pontoon.mozilla.org/
|
||||
[15]:http://pootle.translatehouse.org/
|
||||
[16]:https://weblate.org/
|
||||
[17]:http://translate5.net/
|
||||
[18]:http://www.globalsight.com/
|
||||
[19]:http://zanata.org/
|
||||
[20]:http://jabylon.org/
|
||||
[21]:http://certsoftadmin.byu.edu/baseterm/termbase/search_all
|
||||
[22]:https://github.com/translate/terminator
|
||||
[23]:http://okapiframework.org/
|
||||
[24]:http://www.mojito.global/
|
||||
[25]:http://fosmt.org/
|
||||
[26]:https://www.openwest.org/custom/description.php?id=156
|
||||
[27]:https://www.openwest.org/
|
||||
[28]:https://opensource.com/user/143806/feed
|
||||
[29]:https://en.wikipedia.org/wiki/N-gram#n-gram_models
|
||||
[30]:https://opensource.com/users/guerojeff
|
||||
@@ -1,4 +1,4 @@
|
||||
translating by chenxinlong
|
||||
Translating by trnhoe.
|
||||
An introduction to functional programming in JavaScript
|
||||
============================================================
|
||||
|
||||
|
||||
@@ -1,71 +0,0 @@
|
||||
Translating by Zhipeng-li
|
||||
Kubernetes: Why does it matter?
|
||||
============================================================
|
||||
|
||||
### The Kubernetes platform for running containerized workloads takes on some of the heavy lifting when developing and deploying cloud-native applications.
|
||||
|
||||
|
||||
|
||||
>Image by : opensource.com
|
||||
|
||||
Developing and deploying cloud-native applications has become very popular—for very good reasons. There are clear advantages to a process that allows rapid deployment and continuous delivery of bug fixes and new features, but there's a chicken-and-egg problem no one talks about: How do you get there from here? Building the infrastructure and developing processes to develop and maintain cloud-native applications—all from scratch—are non-trivial, time-intensive tasks.
|
||||
|
||||
[Kubernetes][3], a relatively new platform for running containerized workloads, addresses these problems. Originally an internal project within Google, Kubernetes was donated to the [Cloud Native Computing Foundation][4] in 2015 and has attracted developers from the open source community around the world. Kubernetes' design is based on 15 years of experience in running both production and development workloads. Since it is open source, anyone can download and use it and realize its benefits.
|
||||
|
||||
So why is such a big fuss being made over Kubernetes? I believe that it hits a sweet spot between an Infrastructure as a Service (IaaS) solution, like OpenStack, and a full Platform as a Service (PaaS) resource where the lower-level runtime implementation is completely controlled by a vendor. Kubernetes provides the benefits of both worlds: abstractions to manage infrastructure, as well as tools and features to drill down to bare metal for troubleshooting.
|
||||
|
||||
### IaaS vs. PaaS
|
||||
|
||||
OpenStack is classified by most people as an IaaS solution, where pools of physical resources, such as processors, networking, and storage, are allocated and shared among different users. Isolation between users is implemented using traditional, hardware-based virtualization.
|
||||
|
||||
OpenStack's REST API allows infrastructure to be created automatically using code, but therein lies the problem. The output of the IaaS product is yet more infrastructure. There's not much in the way of services to support and manage the extra infrastructure once it has been created. After a certain point, it becomes a lot of work to manage the low-level infrastructure, such as servers and IP addresses, produced by OpenStack. One well-known outcome is virtual machine (VM) sprawl, but the same concept applies to networks, cryptographic keys, and storage volumes. This leaves less time for developers to work on building and maintaining an application.
|
||||
|
||||
Like other cluster-based solutions, Kubernetes operates at the individual server level to implement horizontal scaling. New servers can be added easily and workloads scheduled on the hardware immediately. Similarly, servers can be removed from the cluster when they're not being utilized effectively or when maintenance is needed. Orchestration activities, such as job scheduling, health monitoring, and maintaining high availability, are other tasks automatically handled by Kubernetes.
|
||||
|
||||
Networking is another area that can be difficult to reliably orchestrate in an IaaS environment. Communication of IP addresses between services to link microservices can be particularly tricky. Kubernetes implements IP address management, load balancing, service discovery, and DNS name registration to provide a headache-free, transparent networking environment within the cluster.
|
||||
|
||||
### Designed for deployment
|
||||
|
||||
Once you have created the environment to run your application, there is the small matter of deploying it. Reliably deploying an application is one of those tasks that's easily said, but not easily done—not in the slightest. The huge advantage that Kubernetes has over other environments is that deployment is a first-class citizen.
|
||||
|
||||
There is a single command, using the Kubernetes command-line interface (CLI), that takes a description of the application and installs it on the cluster. Kubernetes implements the entire application lifecycle from initial deployment, rolling out new releases as well as rolling them back—a critical feature when things go wrong. In-progress deployments can also be paused and resumed. The advantage of having existing, built-in tools and support for application deployment, rather than building a deployment system yourself, cannot be overstated. Kubernetes users do not have to reinvent the application deployment wheel nor discover what a difficult task it is.
|
||||
|
||||
Kubernetes also has the facility to monitor the status of an in-progress deployment. While you can write this in an IaaS environment, like the deployment process itself, it's a surprisingly difficult task where corner cases abound.
|
||||
|
||||
### Designed for DevOps
|
||||
|
||||
As you gain more experience in developing and deploying applications for Kubernetes, you will be traveling the same path that Google and others have before you. You'll discover there are several Kubernetes features that are essential to effectively developing and troubleshooting a multi-service application.
|
||||
|
||||
First, Kubernetes' ability to easily examine the logs or SSH (secure shell) into a running service is vitally important. With a single command line invocation, an administrator can examine the logs of a service running under Kubernetes. This may sound like a simple task, but in an IaaS environment it's not easy unless you have already put some work into it. Large applications often have hardware and personnel dedicated just for log collection and analysis. Logging in Kubernetes may not replace a full-featured logging and metrics solution, but it provides enough to enable basic troubleshooting.
|
||||
|
||||
Second, Kubernetes offers built-in secret management. Another hitch known by teams who have developed their own deployment systems from scratch is that deploying sensitive data, such as passwords and API tokens, securely to VMs is hard. By making secrets first-class citizens, Kubernetes stops your team from inventing its own insecure, buggy secret-distribution system or just hardcoding credentials in deployment scripts.
|
||||
|
||||
Finally, there is a slew of features in Kubernetes for automatically scaling, load-balancing, and restarting your application. Again, these features are tempting targets for developers to write when using IaaS or bare metal. Scaling and health checks for your Kubernetes application are declared in the service definition, and Kubernetes ensures that the correct number of instances is running and healthy.
|
||||
|
||||
### Conclusion
|
||||
|
||||
The differences between IaaS and PaaS systems are enormous, including that PaaS can save a vast amount of development and debugging time. As a PaaS, Kubernetes implements a potent and effective set of features to help you develop, deploy, and debug cloud-native applications. Its architecture and design represent decades of hard-won experience that can your team can take advantage of—for free.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
作者简介：
|
||||
|
||||
Tim Potter - Tim is a senior software engineer working at Hewlett Packard Enterprise. He has been a contributor to free and open source software for nearly two decades working on a variety of projects including Samba, Wireshark, OpenPegasus, and Docker. Tim blogs at https://elegantinfrastructure.com/ about Docker, Kubernetes and other infrastructure-related topics.
|
||||
|
||||
-----
|
||||
|
||||
|
||||
via: https://opensource.com/article/17/6/introducing-kubernetes
|
||||
|
||||
作者：[ Tim Potter][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://opensource.com/users/tpot
|
||||
[1]:https://opensource.com/article/17/6/introducing-kubernetes?rate=RPoUoHXYQXbTb7DHQCDsHgR1ZcfLSoquZ8xVZzfMtxM
|
||||
[2]:https://opensource.com/user/63281/feed
|
||||
[3]:https://kubernetes.io/
|
||||
[4]:https://www.cncf.io/
|
||||
[5]:https://opensource.com/users/tpot
|
||||
@@ -1,312 +0,0 @@
|
||||
MonkeyDEcho translated
|
||||
|
||||
[MySQL infrastructure testing automation at GitHub][31]
|
||||
============================================================
|
||||
|
||||
Our MySQL infrastructure is a critical component to GitHub. MySQL serves GitHub.com, GitHub’s API, authentication and more. Every `git` request touches MySQL in some way. We are tasked with keeping the data available, and maintaining its integrity. Even while our MySQL clusters serve traffic, we need to be able to perform tasks such as heavy duty cleanups, ad-hoc updates, online schema migrations, cluster topology refactoring, pooling and load balancing and more. We have the infrastructure to automate away such operations; in this post we share a few examples of how we build trust in our infrastructure through continuous testing. It is essentially how we sleep well at night.
|
||||
|
||||
### Backups[][36]
|
||||
|
||||
It is incredibly important to take backups of your data. If you are not taking backups of your database, it is likely a matter of time before this will become an issue. Percona [Xtrabackup][37]is the tool we have been using for issuing full backups for our MySQL databases. If there is data that we need to be certain is saved, we have a server that is backing up the data.
|
||||
|
||||
In addition to the full binary backups, we run logical backups several times a day. These backups allow our engineers to get a copy of recent data. There are times that they would like a complete set of data from a table so they can test an index change on a production sized table or see data from a certain point of time. Hubot allows us to restore a backed up table and will ping us when the table is ready to use.
|
||||
|
||||
|
||||
**tomkrouper**.mysql backup-list locations
|
||||
|
||||
**Hubot**
|
||||
```
|
||||
+-----------+------------+---------------+---------------------+---------------------+----------------------------------------------+
|
||||
| Backup ID | Table Name | Donor Host | Backup Start | Backup End | File Name |
|
||||
+-----------+------------+---------------+---------------------+---------------------+----------------------------------------------+
|
||||
| 1699494 | locations | db-mysql-0903 | 2017-07-01 22:09:17 | 2017-07-01 22:09:17 | backup-mycluster-locations-1498593122.sql.gz |
|
||||
| 1699133 | locations | db-mysql-0903 | 2017-07-01 16:11:37 | 2017-07-01 16:11:39 | backup-mycluster-locations-1498571521.sql.gz |
|
||||
| 1698772 | locations | db-mysql-0903 | 2017-07-01 10:09:21 | 2017-07-01 10:09:22 | backup-mycluster-locations-1498549921.sql.gz |
|
||||
| 1698411 | locations | db-mysql-0903 | 2017-07-01 04:12:32 | 2017-07-01 04:12:32 | backup-mycluster-locations-1498528321.sql.gz |
|
||||
| 1698050 | locations | db-mysql-0903 | 2017-06-30 22:18:23 | 2017-06-30 22:18:23 | backup-mycluster-locations-1498506721.sql.gz |
|
||||
| ...
|
||||
| 1262253 | locations | db-mysql-0088 | 2016-08-01 01:58:51 | 2016-08-01 01:58:54 | backup-mycluster-locations-1470034801.sql.gz |
|
||||
| 1064984 | locations | db-mysql-0088 | 2016-04-04 13:07:40 | 2016-04-04 13:07:43 | backup-mycluster-locations-1459494001.sql.gz |
|
||||
+-----------+------------+---------------+---------------------+---------------------+----------------------------------------------+
|
||||
|
||||
```
|
||||
|
||||
|
||||
**tomkrouper**.mysql restore 1699133
|
||||
|
||||
**Hubot**A restore job has been created for the backup job 1699133\. You will be notified in #database-ops when the restore is complete.
|
||||
|
||||
**Hubot**[@tomkrouper][1]: the locations table has been restored as locations_2017_07_01_16_11 in the restores database on db-mysql-0482
|
||||
|
||||
The data is loaded onto a non-production database which is accessible to the engineer requesting the restore.
|
||||
|
||||
The last way we keep a “backup” of data around is we use [delayed replicas][38]. This is less of a backup and more of a safeguard. For each production cluster we have a host that has replication delayed by 4 hours. If a query is run that shouldn’t have, we can run `mysql panic` in chatops. This will cause all of our delayed replicas to stop replication immediately. This will also page the on-call DBA. From there we can use delayed replica to verify there is an issue, and then fast forward the binary logs to the point right before the error. We can then restore this data to the master, thus recovering data to that point.
|
||||
|
||||
Backups are great, however they are worthless if some unknown or uncaught error occurs corrupting the backup. A benefit of having a script to restore backups is it allows us to automate the verification of backups via cron. We have set up a dedicated host for each cluster that runs a restore of the latest backup. This ensures that the backup ran correctly and that we are able to retrieve the data from the backup.
|
||||
|
||||
Depending on dataset size, we run several restores per day. Restored servers are expected to join the replication stream and to be able to catch up with replication. This tests not only that we took a restorable backup, but also that we correctly identified the point in time at which it was taken and can further apply changes from that point in time. We are alerted if anything goes wrong in the restore process.
|
||||
|
||||
We furthermore track the time the restore takes, so we have a good idea of how long it will take to build a new replica or restore in cases of emergency.
|
||||
|
||||
The following is an output from an automated restore process, written by Hubot in our robots chat room.
|
||||
|
||||
|
||||
**Hubot**gh-mysql-backup-restore: db-mysql-0752: restore_log.id = 4447
|
||||
gh-mysql-backup-restore: db-mysql-0752: Determining backup to restore for cluster 'prodcluster'.
|
||||
gh-mysql-backup-restore: db-mysql-0752: Enabling maintenance mode
|
||||
gh-mysql-backup-restore: db-mysql-0752: Setting orchestrator downtime
|
||||
gh-mysql-backup-restore: db-mysql-0752: Disabling Puppet
|
||||
gh-mysql-backup-restore: db-mysql-0752: Stopping MySQL
|
||||
gh-mysql-backup-restore: db-mysql-0752: Removing MySQL files
|
||||
gh-mysql-backup-restore: db-mysql-0752: Running gh-xtrabackup-restore
|
||||
gh-mysql-backup-restore: db-mysql-0752: Restore file: xtrabackup-notify-2017-07-02_0000.xbstream
|
||||
gh-mysql-backup-restore: db-mysql-0752: Running gh-xtrabackup-prepare
|
||||
gh-mysql-backup-restore: db-mysql-0752: Starting MySQL
|
||||
gh-mysql-backup-restore: db-mysql-0752: Update file ownership
|
||||
gh-mysql-backup-restore: db-mysql-0752: Upgrade MySQL
|
||||
gh-mysql-backup-restore: db-mysql-0752: Stopping MySQL
|
||||
gh-mysql-backup-restore: db-mysql-0752: Starting MySQL
|
||||
gh-mysql-backup-restore: db-mysql-0752: Backup Host: db-mysql-0034
|
||||
gh-mysql-backup-restore: db-mysql-0752: Setting up replication
|
||||
gh-mysql-backup-restore: db-mysql-0752: Starting replication
|
||||
gh-mysql-backup-restore: db-mysql-0752: Replication catch-up
|
||||
gh-mysql-backup-restore: db-mysql-0752: Restore complete (replication running)
|
||||
gh-mysql-backup-restore: db-mysql-0752: Enabling Puppet
|
||||
gh-mysql-backup-restore: db-mysql-0752: Disabling maintenance mode
|
||||
gh-mysql-backup-restore: db-mysql-0752: Setting orchestrator downtime
|
||||
gh-mysql-backup-restore: db-mysql-0752: Restore process complete.
|
||||
|
||||
One thing we use backups for is adding a new replica to an existing set of MySQL servers. We will initiate the build of a new server, and once we are notified it is ready, we can start a restore of the latest backup for that particular cluster. We have a script in place that runs all of the restore commands that we would otherwise have to do by hand. Our automated restore system essentially uses the same script. This simplifies the system build process and allows us to have a host up and running with a handful of chat commands opposed to dozens of manual processes. Shown below is a restore kicked manually in chat:
|
||||
|
||||
|
||||
**jessbreckenridge**.mysql backup-restore -H db-mysql-0007 -o -r magic_word=daily_rotating_word
|
||||
|
||||
**Hubot**[@jessbreckenridge][2] gh-mysql-backup-restore: db-mysql-0007: Determining backup to restore for cluster 'mycluster'.
|
||||
[@jessbreckenridge][3] gh-mysql-backup-restore: db-mysql-0007: restore_log.id = 4449
|
||||
[@jessbreckenridge][4] gh-mysql-backup-restore: db-mysql-0007: Enabling maintenance mode
|
||||
[@jessbreckenridge][5] gh-mysql-backup-restore: db-mysql-0007: Setting orchestrator downtime
|
||||
[@jessbreckenridge][6] gh-mysql-backup-restore: db-mysql-0007: Disabling Puppet
|
||||
[@jessbreckenridge][7] gh-mysql-backup-restore: db-mysql-0007: Stopping MySQL
|
||||
[@jessbreckenridge][8] gh-mysql-backup-restore: db-mysql-0007: Removing MySQL files
|
||||
[@jessbreckenridge][9] gh-mysql-backup-restore: db-mysql-0007: Running gh-xtrabackup-restore
|
||||
[@jessbreckenridge][10] gh-mysql-backup-restore: db-mysql-0007: Restore file: xtrabackup-mycluster-2017-07-02_0015.xbstream
|
||||
[@jessbreckenridge][11] gh-mysql-backup-restore: db-mysql-0007: Running gh-xtrabackup-prepare
|
||||
[@jessbreckenridge][12] gh-mysql-backup-restore: db-mysql-0007: Update file ownership
|
||||
[@jessbreckenridge][13] gh-mysql-backup-restore: db-mysql-0007: Starting MySQL
|
||||
[@jessbreckenridge][14] gh-mysql-backup-restore: db-mysql-0007: Upgrade MySQL
|
||||
[@jessbreckenridge][15] gh-mysql-backup-restore: db-mysql-0007: Stopping MySQL
|
||||
[@jessbreckenridge][16] gh-mysql-backup-restore: db-mysql-0007: Starting MySQL
|
||||
[@jessbreckenridge][17] gh-mysql-backup-restore: db-mysql-0007: Setting up replication
|
||||
[@jessbreckenridge][18] gh-mysql-backup-restore: db-mysql-0007: Starting replication
|
||||
[@jessbreckenridge][19] gh-mysql-backup-restore: db-mysql-0007: Backup Host: db-mysql-0201
|
||||
[@jessbreckenridge][20] gh-mysql-backup-restore: db-mysql-0007: Replication catch-up
|
||||
[@jessbreckenridge][21] gh-mysql-backup-restore: db-mysql-0007: Replication behind by 4589 seconds, waiting 1800 seconds before next check.
|
||||
[@jessbreckenridge][22] gh-mysql-backup-restore: db-mysql-0007: Restore complete (replication running)
|
||||
[@jessbreckenridge][23] gh-mysql-backup-restore: db-mysql-0007: Enabling puppet
|
||||
[@jessbreckenridge][24] gh-mysql-backup-restore: db-mysql-0007: Disabling maintenance mode
|
||||
|
||||
### Failovers[][39]
|
||||
|
||||
[We use orchestrator][40] to perform automated failovers for masters and intermediate masters. We expect `orchestrator` to correctly detect master failure, designate a replica for promotion, heal the topology under said designated replica, make the promotion. We expect VIPs to change, pools to change, clients to reconnect, `puppet` to run essential components on promoted master, and more. A failover is a complex task that touches many aspects of our infrastructure.
|
||||
|
||||
To build trust in our failovers we set up a _production-like_ , test cluster, and we continuously crash it to observe failovers.
|
||||
|
||||
The _production-like_ cluster is a replication setup that is identical in all aspects to our production clusters: types of hardware, operating systems, MySQL versions, network environments, VIP, `puppet` configurations, [haproxy setup][41], etc. The only thing different to this cluster is that it doesn’t send/receive production traffic.
|
||||
|
||||
We emulate a write load on the test cluster, while avoiding replication lag. The write load is not too heavy, but has queries that are intentionally contending to write on same datasets. This isn’t too interesting in normal times, but proves to be useful upon failovers, as we will shortly describe.
|
||||
|
||||
Our test cluster has representative servers from three data centers. We would _like_ the failover to promote a replacement replica from within the same data center. We would _like_ to be able to salvage as many replicas as possible under such constraint. We _require_ that both apply whenever possible. `orchestrator` has no prior assumption on the topology; it must react on whatever the state was at time of the crash.
|
||||
|
||||
We, however, are interested in creating complex and varying scenarios for failovers. Our failover testing script prepares the grounds for the failover:
|
||||
|
||||
* It identifies existing master
|
||||
|
||||
* It refactors the topology to have representatives of all three data centers under the master. Different DCs have different network latencies and are expected to react in different timing to master’s crash.
|
||||
|
||||
* It chooses a crash method. We choose from shooting the master (`kill -9`) or network partitioning it: `iptables -j REJECT` (nice-ish) or `iptables -j DROP`(unresponsive).
|
||||
|
||||
The script proceeds to crash the master by chosen method, and waits for `orchestrator` to reliably detect the crash and to perform failover. While we expect detection and promotion to both complete within `30` seconds, the script relaxes this expectation a bit, and sleeps for a designated time before looking into failover results. It will then:
|
||||
|
||||
* Check that a new (different) master is in place
|
||||
|
||||
* There is a good number of replicas in the cluster
|
||||
|
||||
* The master is writable
|
||||
|
||||
* Writes to the master are visible on the replicas
|
||||
|
||||
* Internal service discovery entries are updated (identity of new master is as expected; old master removed)
|
||||
|
||||
* Other internal checks
|
||||
|
||||
These tests confirm that the failover was successful, not only MySQL-wise but also on our larger infrastructure scope. A VIP has been assumed; specific services have been started; information got to where it was supposed to go.
|
||||
|
||||
The script further proceeds to restore the failed server:
|
||||
|
||||
* Restoring it from backup, thereby implicitly testing our backup/restore procedure
|
||||
|
||||
* Verifying server configuration is as expected (the server no longer believes it’s the master)
|
||||
|
||||
* Returning it to the replication cluster, expecting to find data written on the master
|
||||
|
||||
Consider the following visualization of a scheduled failover test: from having a well-running cluster, to seeing problems on some replicas, to diagnosing the master (`7136`) is dead, to choosing a server to promote (`a79d`), refactoring the topology below that server, to promoting it (failover successful), to restoring the dead master and placing it back into the cluster.
|
||||
|
||||
|
||||
|
||||
#### What would a test failure look like?
|
||||
|
||||
Our testing script uses a stop-the-world approach. A single failure in any of the failover components fails the entire test, disabling any future automated tests until a human resolves the matter. We get alerted and proceed to check the status and logs.
|
||||
|
||||
The script would fail on an unacceptable detection or failover time; on backup/restore issues; on losing too many servers; on unexpected configuration following the failover; etc.
|
||||
|
||||
We need to be certain `orchestrator` connects the servers correctly. This is where the contending write load comes useful: if set up incorrectly, replication is easily susceptible to break. We would get `DUPLICATE KEY` or other errors to suggest something went wrong.
|
||||
|
||||
This is particularly important as we make improvements and introduce new behavior to `orchestrator`, and allows us to test such changes in a safe environment.
|
||||
|
||||
#### Coming up: chaos testing
|
||||
|
||||
The testing procedure illustrated above will catch (and has caught) problems on many parts of our infrastructure. Is it enough?
|
||||
|
||||
In a production environment there’s always something else. Something about the particular test method that won’t apply to our production clusters. They don’t share the same traffic and traffic manipulation, nor the exact same set of servers. The types of failure can vary.
|
||||
|
||||
We are designing chaos testing for our production clusters. Chaos testing would literally destroy pieces in our production, but on expected schedule and under sufficiently controlled manner. Chaos testing introduces a higher level of trust in the recovery mechanism and affects (thus tests) larger parts of our infrastructure and application.
|
||||
|
||||
This is delicate work: while we acknowledge the need for chaos testing, we also wish to avoid unnecessary impact to our service. Different tests will differ in risk level and impact, and we will work to ensure availability of our service.
|
||||
|
||||
### Schema migrations[][42]
|
||||
|
||||
[We use gh-ost][43] to run live schema migrations. `gh-ost` is stable, but also under active developments, with major new features being added or planned.
|
||||
|
||||
`gh-ost` migrates tables by copying data onto a _ghost_ table, applying ongoing changes intercepted by the binary logs onto the _ghost_ table, even as the original table is being written to. It then swaps the _ghost_ table in place of the original table. At migration completion GitHub proceeds to work with a table generated and populated by `gh-ost`.
|
||||
|
||||
At this time almost all of GitHub’s MySQL data has been recreated by `gh-ost`, and most of it multiple times. We must have high trust in `gh-ost` to let it tamper with our data over and over again, even in face of active development. Here’s how we gain this trust.
|
||||
|
||||
`gh-ost` provides a testing-in-production capability. It supports running a migration on a replica, in much the same way as it would run on the master: `gh-ost` would connect to the replica and treat it as if it were the master. It would parse its binary logs the same way it would for a real master migration. However it would copy rows and apply binlog events to the replica, and avoid making writes onto the master.
|
||||
|
||||
We run `gh-ost`-dedicated replicas in production. These replicas do not serve production traffic. Each such replica retrieves the current list of production tables and iterates them in random order. One by one it picks a table and performs a replica-migration on that table. The migration doesn’t actually modify table structure, but instead runs a trivial `ENGINE=InnoDB`. The test runs the migration even as the table is being used in production, thus copying real production data and applying true production traffic off the binary logs.
|
||||
|
||||
These migrations can be audited. Here’s how we can inspect status of running tests from chat:
|
||||
|
||||
|
||||
**ggunson**.migration test-status
|
||||
|
||||
**Hubot**
|
||||
```
|
||||
# Migrating `prod`.`pull_requests`; Ghost table is `prod`.`_pull_requests_gho`
|
||||
# Migrating ghost-db-mysql-0007:3306; inspecting ghost-db-mysql-0007:3306; executing on ghost-db-mysql-0007
|
||||
# Migration started at Mon Jan 30 02:13:39 -0800 2017
|
||||
# chunk-size: 2500; max-lag-millis: 1500ms; max-load: Threads_running=30; critical-load: Threads_running=1000; nice-ratio: 0.000000
|
||||
# throttle-additional-flag-file: /tmp/gh-ost.throttle
|
||||
# panic-flag-file: /tmp/ghost-test-panic.flag
|
||||
# Serving on unix socket: /tmp/gh-ost.test.sock
|
||||
Copy: 57992500/86684838 66.9%; Applied: 57708; Backlog: 1/100; Time: 3h28m38s(total), 3h28m36s(copy); streamer: mysql-bin.000576:142993938; State: migrating; ETA: 1h43m12s
|
||||
```
|
||||
|
||||
When a test migration completes copying of table data it stops replication and performs the cut-over, replacing the original table with the _ghost_ table, and then swaps back. We’re not interested in actually replacing the data. Instead we are left with both the original table and the _ghost_ table, which should both be identical. We verify that by checksumming the entire table data for both tables.
|
||||
|
||||
A test can complete with:
|
||||
|
||||
* _success_ : All went well and checksum is identical. We expect to see this.
|
||||
|
||||
* _failure_ : Execution problem. This can occasionally happen due to the migration process being killed, a replication issue etc., and is typically unrelated to `gh-ost` itself.
|
||||
|
||||
* _checksum failure_ : table data inconsistency. For a tested branch, this call for fixes. For an ongoing `master` branch test, this would imply immediate blocking of production migrations. We don’t get the latter.
|
||||
|
||||
Test results are audited, sent to robot chatrooms, sent as events to our metrics systems. Each vertical line in the following graph represents a successful migration test:
|
||||
|
||||
|
||||
|
||||
These tests run continuously. We are notified by alerts in case of failures. And of course we can always visit the robots chatroom to know what’s going on.
|
||||
|
||||
#### Testing new versions
|
||||
|
||||
We continuously improve `gh-ost`. Our development flow is based on `git` branches, which we then offer to merge via [pull requests][44].
|
||||
|
||||
A submitted `gh-ost` pull request goes through Continuous Integration (CI) which runs basic compilation and unit tests. Once past this, the PR is technically eligible for merging, but even more interestingly it is [eligible for deployment via Heaven][45]. Being the sensitive component in our infrastructure that it is, we take care to deploy `gh-ost` branches for intensive testing before merging into `master`.
|
||||
|
||||
|
||||
**shlomi-noach**.deploy gh-ost/fix-reappearing-throttled-reasons to prod/ghost-db-mysql-0007
|
||||
|
||||
**Hubot**[@shlomi-noach][25] is deploying gh-ost/fix-reappearing-throttled-reasons (baee4f6) to production (ghost-db-mysql-0007).
|
||||
[@shlomi-noach][26]'s production deployment of gh-ost/fix-reappearing-throttled-reasons (baee4f6) is done! (2s)
|
||||
[@shlomi-noach][27], make sure you watch for exceptions in haystack
|
||||
|
||||
**jonahberquist**.deploy gh-ost/interactive-command-question to prod/ghost-db-mysql-0012
|
||||
|
||||
**Hubot**[@jonahberquist][28] is deploying gh-ost/interactive-command-question (be1ab17) to production (ghost-db-mysql-0012).
|
||||
[@jonahberquist][29]'s production deployment of gh-ost/interactive-command-question (be1ab17) is done! (2s)
|
||||
[@jonahberquist][30], make sure you watch for exceptions in haystack
|
||||
|
||||
**shlomi-noach**.wcid gh-ost
|
||||
|
||||
**Hubot**shlomi-noach testing fix-reappearing-throttled-reasons 41 seconds ago: ghost-db-mysql-0007
|
||||
jonahberquist testing interactive-command-question 7 seconds ago: ghost-db-mysql-0012
|
||||
|
||||
Nobody is in the queue.
|
||||
|
||||
Some PRs are small and do not affect the data itself. Changes to status messages, interactive commands etc. are of lesser impact to the `gh-ost` app. Others pose significant changes to the migration logic and operation. We would tests these rigorously, running through our production tables fleet until satisfied these changes do not pose data corruption threat.
|
||||
|
||||
### Summary[][46]
|
||||
|
||||
Throughout testing we build trust in our systems. By automating these tests, in production, we get repetitive confirmation that everything is working as expected. As we continue to develop our infrastructure we also follow up by adapting tests to cover the newest changes.
|
||||
|
||||
Production always surprises with scenarios not covered by tests. The more we test on production environment, the more input we get on our app’s expectations and our infrastructure’s capabilities.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://githubengineering.com/mysql-testing-automation-at-github/
|
||||
|
||||
作者：[tomkrouper ][a], [Shlomi Noach][b]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://github.com/tomkrouper
|
||||
[b]:https://github.com/shlomi-noach
|
||||
[1]:https://github.com/tomkrouper
|
||||
[2]:https://github.com/jessbreckenridge
|
||||
[3]:https://github.com/jessbreckenridge
|
||||
[4]:https://github.com/jessbreckenridge
|
||||
[5]:https://github.com/jessbreckenridge
|
||||
[6]:https://github.com/jessbreckenridge
|
||||
[7]:https://github.com/jessbreckenridge
|
||||
[8]:https://github.com/jessbreckenridge
|
||||
[9]:https://github.com/jessbreckenridge
|
||||
[10]:https://github.com/jessbreckenridge
|
||||
[11]:https://github.com/jessbreckenridge
|
||||
[12]:https://github.com/jessbreckenridge
|
||||
[13]:https://github.com/jessbreckenridge
|
||||
[14]:https://github.com/jessbreckenridge
|
||||
[15]:https://github.com/jessbreckenridge
|
||||
[16]:https://github.com/jessbreckenridge
|
||||
[17]:https://github.com/jessbreckenridge
|
||||
[18]:https://github.com/jessbreckenridge
|
||||
[19]:https://github.com/jessbreckenridge
|
||||
[20]:https://github.com/jessbreckenridge
|
||||
[21]:https://github.com/jessbreckenridge
|
||||
[22]:https://github.com/jessbreckenridge
|
||||
[23]:https://github.com/jessbreckenridge
|
||||
[24]:https://github.com/jessbreckenridge
|
||||
[25]:https://github.com/shlomi-noach
|
||||
[26]:https://github.com/shlomi-noach
|
||||
[27]:https://github.com/shlomi-noach
|
||||
[28]:https://github.com/jonahberquist
|
||||
[29]:https://github.com/jonahberquist
|
||||
[30]:https://github.com/jonahberquist
|
||||
[31]:https://githubengineering.com/mysql-testing-automation-at-github/
|
||||
[32]:https://github.com/tomkrouper
|
||||
[33]:https://github.com/tomkrouper
|
||||
[34]:https://github.com/shlomi-noach
|
||||
[35]:https://github.com/shlomi-noach
|
||||
[36]:https://githubengineering.com/mysql-testing-automation-at-github/#backups
|
||||
[37]:https://www.percona.com/software/mysql-database/percona-xtrabackup
|
||||
[38]:https://dev.mysql.com/doc/refman/5.6/en/replication-delayed.html
|
||||
[39]:https://githubengineering.com/mysql-testing-automation-at-github/#failovers
|
||||
[40]:http://githubengineering.com/orchestrator-github/
|
||||
[41]:https://githubengineering.com/context-aware-mysql-pools-via-haproxy/
|
||||
[42]:https://githubengineering.com/mysql-testing-automation-at-github/#schema-migrations
|
||||
[43]:http://githubengineering.com/gh-ost-github-s-online-migration-tool-for-mysql/
|
||||
[44]:https://github.com/github/gh-ost/pulls
|
||||
[45]:https://githubengineering.com/deploying-branches-to-github-com/
|
||||
[46]:https://githubengineering.com/mysql-testing-automation-at-github/#summary
|
||||
@@ -1,337 +0,0 @@
|
||||
translating----geekpi
|
||||
|
||||
Writing a Linux Debugger Part 9: Handling variables
|
||||
============================================================
|
||||
|
||||
Variables are sneaky. At one moment they’ll be happily sitting in registers, but as soon as you turn your head they’re spilled to the stack. Maybe the compiler completely throws them out of the window for the sake of optimization. Regardless of how often variables move around in memory, we need some way to track and manipulate them in our debugger. This post will teach you more about handling variables in your debugger and demonstrate a simple implementation using `libelfin`.
|
||||
|
||||
* * *
|
||||
|
||||
### Series index
|
||||
|
||||
1. [Setup][1]
|
||||
|
||||
2. [Breakpoints][2]
|
||||
|
||||
3. [Registers and memory][3]
|
||||
|
||||
4. [Elves and dwarves][4]
|
||||
|
||||
5. [Source and signals][5]
|
||||
|
||||
6. [Source-level stepping][6]
|
||||
|
||||
7. [Source-level breakpoints][7]
|
||||
|
||||
8. [Stack unwinding][8]
|
||||
|
||||
9. [Handling variables][9]
|
||||
|
||||
10. [Advanced topics][10]
|
||||
|
||||
* * *
|
||||
|
||||
Before you get started, make sure that the version of `libelfin` you are using is the [`fbreg` branch of my fork][11]. This contains some hacks to support getting the base of the current stack frame and evaluating location lists, neither of which are supported by vanilla `libelfin`. You might need to pass `-gdwarf-2` to GCC to get it to generate compatible DWARF information. But before we get into the implementation, I’ll give a more detailed description of how locations are encoded in DWARF 5, which is the most recent specification. If you want more information than what I write here, then you can grab the standard from [here][12].
|
||||
|
||||
### DWARF locations
|
||||
|
||||
The location of a variable in memory at a given moment is encoded in the DWARF information using the `DW_AT_location`attribute. Location descriptions can be either single location descriptions, composite location descriptions, or location lists.
|
||||
|
||||
* Simple location descriptions describe the location of one contiguous piece (usually all) of an object. A simple location description may describe a location in addressable memory, or in a register, or the lack of a location (with or without a known value).
|
||||
* Example:
|
||||
* `DW_OP_fbreg -32`
|
||||
|
||||
* A variable which is entirely stored -32 bytes from the stack frame base
|
||||
|
||||
* Composite location descriptions describe an object in terms of pieces, each of which may be contained in part of a register or stored in a memory location unrelated to other pieces.
|
||||
* Example:
|
||||
* `DW_OP_reg3 DW_OP_piece 4 DW_OP_reg10 DW_OP_piece 2`
|
||||
|
||||
* A variable whose first four bytes reside in register 3 and whose next two bytes reside in register 10.
|
||||
|
||||
* Location lists describe objects which have a limited lifetime or change location during their lifetime.
|
||||
* Example:
|
||||
* `<loclist with 3 entries follows>`
|
||||
* `[ 0]<lowpc=0x2e00><highpc=0x2e19>DW_OP_reg0`
|
||||
|
||||
* `[ 1]<lowpc=0x2e19><highpc=0x2e3f>DW_OP_reg3`
|
||||
|
||||
* `[ 2]<lowpc=0x2ec4><highpc=0x2ec7>DW_OP_reg2`
|
||||
|
||||
* A variable whose location moves between registers depending on the current value of the program counter
|
||||
|
||||
The `DW_AT_location` is encoded in one of three different ways, depending on the kind of location description. `exprloc`s encode simple and composite location descriptions. They consist of a byte length followed by a DWARF expression or location description. `loclist`s and `loclistptr`s encode location lists. They give indexes or offsets into the `.debug_loclists` section, which describes the actual location lists.
|
||||
|
||||
### DWARF Expressions
|
||||
|
||||
The actual location of the variables is computed using DWARF expressions. These consist of a series of operations which operate on a stack of values. There are an impressive number of DWARF operations available, so I won’t explain them all in detail. Instead I’ll give a few examples from each class of expression to give you a taste of what is available. Also, don’t get scared off by these; `libelfin` will take care off all of this complexity for us.
|
||||
|
||||
* Literal encodings
|
||||
* `DW_OP_lit0`, `DW_OP_lit1`, …, `DW_OP_lit31`
|
||||
* Push the literal value on to the stack
|
||||
|
||||
* `DW_OP_addr <addr>`
|
||||
* Pushes the address operand on to the stack
|
||||
|
||||
* `DW_OP_constu <unsigned>`
|
||||
* Pushes the unsigned value on to the stack
|
||||
|
||||
* Register values
|
||||
* `DW_OP_fbreg <offset>`
|
||||
* Pushes the value found at the base of the stack frame, offset by the given value
|
||||
|
||||
* `DW_OP_breg0`, `DW_OP_breg1`, …, `DW_OP_breg31 <offset>`
|
||||
* Pushes the contents of the given register plus the given offset to the stack
|
||||
|
||||
* Stack operations
|
||||
* `DW_OP_dup`
|
||||
* Duplicate the value at the top of the stack
|
||||
|
||||
* `DW_OP_deref`
|
||||
* Treats the top of the stack as a memory address, and replaces it with the contents of that address
|
||||
|
||||
* Arithmetic and logical operations
|
||||
* `DW_OP_and`
|
||||
* Pops the top two values from the stack and pushes back the logical `AND` of them
|
||||
|
||||
* `DW_OP_plus`
|
||||
* Same as `DW_OP_and`, but adds the values
|
||||
|
||||
* Control flow operations
|
||||
* `DW_OP_le`, `DW_OP_eq`, `DW_OP_gt`, etc.
|
||||
* Pops the top two values, compares them, and pushes `1` if the condition is true and `0`otherwise
|
||||
|
||||
* `DW_OP_bra <offset>`
|
||||
* Conditional branch: if the top of the stack is not `0`, skips back or forward in the expression by `offset`
|
||||
|
||||
* Type conversions
|
||||
* `DW_OP_convert <DIE offset>`
|
||||
* Converts value on the top of the stack to a different type, which is described by the DWARF information entry at the given offset
|
||||
|
||||
* Special operations
|
||||
* `DW_OP_nop`
|
||||
* Do nothing!
|
||||
|
||||
### DWARF types
|
||||
|
||||
DWARF’s representation of types needs to be strong enough to give debugger users useful variable representations. Users most often want to be able to debug at the level of their application rather than at the level of their machine, and they need a good idea of what their variables are doing to achieve that.
|
||||
|
||||
DWARF types are encoded in DIEs along with the majority of the other debug information. They can have attributes to indicate their name, encoding, size, endianness, etc. A myriad of type tags are available to express pointers, arrays, structures, typedefs, anything else you could see in a C or C++ program.
|
||||
|
||||
Take this simple structure as an example:
|
||||
|
||||
```
|
||||
struct test{
|
||||
int i;
|
||||
float j;
|
||||
int k[42];
|
||||
test* next;
|
||||
};
|
||||
```
|
||||
|
||||
The parent DIE for this struct is this:
|
||||
|
||||
```
|
||||
< 1><0x0000002a> DW_TAG_structure_type
|
||||
DW_AT_name "test"
|
||||
DW_AT_byte_size 0x000000b8
|
||||
DW_AT_decl_file 0x00000001 test.cpp
|
||||
DW_AT_decl_line 0x00000001
|
||||
|
||||
```
|
||||
|
||||
The above says that we have a structure called `test` of size `0xb8`, declared at line `1` of `test.cpp`. All there are then many children DIEs which describe the members.
|
||||
|
||||
```
|
||||
< 2><0x00000032> DW_TAG_member
|
||||
DW_AT_name "i"
|
||||
DW_AT_type <0x00000063>
|
||||
DW_AT_decl_file 0x00000001 test.cpp
|
||||
DW_AT_decl_line 0x00000002
|
||||
DW_AT_data_member_location 0
|
||||
< 2><0x0000003e> DW_TAG_member
|
||||
DW_AT_name "j"
|
||||
DW_AT_type <0x0000006a>
|
||||
DW_AT_decl_file 0x00000001 test.cpp
|
||||
DW_AT_decl_line 0x00000003
|
||||
DW_AT_data_member_location 4
|
||||
< 2><0x0000004a> DW_TAG_member
|
||||
DW_AT_name "k"
|
||||
DW_AT_type <0x00000071>
|
||||
DW_AT_decl_file 0x00000001 test.cpp
|
||||
DW_AT_decl_line 0x00000004
|
||||
DW_AT_data_member_location 8
|
||||
< 2><0x00000056> DW_TAG_member
|
||||
DW_AT_name "next"
|
||||
DW_AT_type <0x00000084>
|
||||
DW_AT_decl_file 0x00000001 test.cpp
|
||||
DW_AT_decl_line 0x00000005
|
||||
DW_AT_data_member_location 176(as signed = -80)
|
||||
|
||||
```
|
||||
|
||||
Each member has a name, a type (which is a DIE offset), a declaration file and line, and a byte offset into the structure where the member is located. The types which are pointed to come next.
|
||||
|
||||
```
|
||||
< 1><0x00000063> DW_TAG_base_type
|
||||
DW_AT_name "int"
|
||||
DW_AT_encoding DW_ATE_signed
|
||||
DW_AT_byte_size 0x00000004
|
||||
< 1><0x0000006a> DW_TAG_base_type
|
||||
DW_AT_name "float"
|
||||
DW_AT_encoding DW_ATE_float
|
||||
DW_AT_byte_size 0x00000004
|
||||
< 1><0x00000071> DW_TAG_array_type
|
||||
DW_AT_type <0x00000063>
|
||||
< 2><0x00000076> DW_TAG_subrange_type
|
||||
DW_AT_type <0x0000007d>
|
||||
DW_AT_count 0x0000002a
|
||||
< 1><0x0000007d> DW_TAG_base_type
|
||||
DW_AT_name "sizetype"
|
||||
DW_AT_byte_size 0x00000008
|
||||
DW_AT_encoding DW_ATE_unsigned
|
||||
< 1><0x00000084> DW_TAG_pointer_type
|
||||
DW_AT_type <0x0000002a>
|
||||
|
||||
```
|
||||
|
||||
As you can see, `int` on my laptop is a 4-byte signed integer type, and `float` is a 4-byte float. The integer array type is defined by pointing to the `int` type as its element type, a `sizetype` (think `size_t`) as the index type, with `2a` elements. The `test*` type is a `DW_TAG_pointer_type` which references the `test` DIE.
|
||||
|
||||
* * *
|
||||
|
||||
### Implementing a simple variable reader
|
||||
|
||||
As mentioned, `libelfin` will deal with most of the complexity for us. However, it doesn’t implement all of the different methods for representing variable locations, and handling a lot of them in our code would get pretty complex. As such, I’ve chosen to only support `exprloc`s for now. Feel free to add support for more types of expression. If you’re really feeling brave, submit some patches to `libelfin` to help complete the necessary support!
|
||||
|
||||
Handling variables is mostly down to locating the different parts in memory or registers, then reading or writing is the same as you’ve seen before. I’ll only show you how to implement reading for the sake of simplicity.
|
||||
|
||||
First we need to tell `libelfin` how to read registers from our process. We do this by creating a class which inherits from `expr_context` and uses `ptrace` to handle everything:
|
||||
|
||||
```
|
||||
class ptrace_expr_context : public dwarf::expr_context {
|
||||
public:
|
||||
ptrace_expr_context (pid_t pid) : m_pid{pid} {}
|
||||
|
||||
dwarf::taddr reg (unsigned regnum) override {
|
||||
return get_register_value_from_dwarf_register(m_pid, regnum);
|
||||
}
|
||||
|
||||
dwarf::taddr pc() override {
|
||||
struct user_regs_struct regs;
|
||||
ptrace(PTRACE_GETREGS, m_pid, nullptr, ®s);
|
||||
return regs.rip;
|
||||
}
|
||||
|
||||
dwarf::taddr deref_size (dwarf::taddr address, unsigned size) override {
|
||||
//TODO take into account size
|
||||
return ptrace(PTRACE_PEEKDATA, m_pid, address, nullptr);
|
||||
}
|
||||
|
||||
private:
|
||||
pid_t m_pid;
|
||||
};
|
||||
```
|
||||
|
||||
The reading will be handled by a `read_variables` function in our `debugger` class:
|
||||
|
||||
```
|
||||
void debugger::read_variables() {
|
||||
using namespace dwarf;
|
||||
|
||||
auto func = get_function_from_pc(get_pc());
|
||||
|
||||
//...
|
||||
}
|
||||
```
|
||||
|
||||
The first thing we do above is find the function which we’re currently in. Then we need to loop through the entries in that function, looking for variables:
|
||||
|
||||
```
|
||||
for (const auto& die : func) {
|
||||
if (die.tag == DW_TAG::variable) {
|
||||
//...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
We get the location information by looking up the `DW_AT_location` entry in the DIE:
|
||||
|
||||
```
|
||||
auto loc_val = die[DW_AT::location];
|
||||
```
|
||||
|
||||
Then we ensure that it’s an `exprloc` and ask `libelfin` to evaluate the expression for us:
|
||||
|
||||
```
|
||||
if (loc_val.get_type() == value::type::exprloc) {
|
||||
ptrace_expr_context context {m_pid};
|
||||
auto result = loc_val.as_exprloc().evaluate(&context);
|
||||
```
|
||||
|
||||
Now that we’ve evaluated the expression, we need to read the contents of the variable. It could be in memory or a register, so we’ll handle both cases:
|
||||
|
||||
```
|
||||
switch (result.location_type) {
|
||||
case expr_result::type::address:
|
||||
{
|
||||
auto value = read_memory(result.value);
|
||||
std::cout << at_name(die) << " (0x" << std::hex << result.value << ") = "
|
||||
<< value << std::endl;
|
||||
break;
|
||||
}
|
||||
|
||||
case expr_result::type::reg:
|
||||
{
|
||||
auto value = get_register_value_from_dwarf_register(m_pid, result.value);
|
||||
std::cout << at_name(die) << " (reg " << result.value << ") = "
|
||||
<< value << std::endl;
|
||||
break;
|
||||
}
|
||||
|
||||
default:
|
||||
throw std::runtime_error{"Unhandled variable location"};
|
||||
}
|
||||
```
|
||||
|
||||
As you can see I’ve simply printed out the value without interpreting it based on the type of the variable. Hopefully from this code you can see how you could support writing variables, or searching for variables with a given name.
|
||||
|
||||
Finally we can add this to our command parser:
|
||||
|
||||
```
|
||||
else if(is_prefix(command, "variables")) {
|
||||
read_variables();
|
||||
}
|
||||
```
|
||||
|
||||
### Testing it out
|
||||
|
||||
Write a few small functions which have some variables, compile it without optimizations and with debug info, then see if you can read the values of your variables. Try writing to the memory address where a variable is stored and see the behaviour of the program change.
|
||||
|
||||
* * *
|
||||
|
||||
Nine posts down, one to go! Next time I’ll be talking about some more advanced concepts which might interest you. For now you can find the code for this post [here][13]
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://blog.tartanllama.xyz/writing-a-linux-debugger-variables/
|
||||
|
||||
作者：[ Simon Brand][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://www.twitter.com/TartanLlama
|
||||
[1]:https://blog.tartanllama.xyz/writing-a-linux-debugger-setup/
|
||||
[2]:https://blog.tartanllama.xyz/writing-a-linux-debugger-breakpoints/
|
||||
[3]:https://blog.tartanllama.xyz/writing-a-linux-debugger-registers/
|
||||
[4]:https://blog.tartanllama.xyz/writing-a-linux-debugger-elf-dwarf/
|
||||
[5]:https://blog.tartanllama.xyz/writing-a-linux-debugger-source-signal/
|
||||
[6]:https://blog.tartanllama.xyz/writing-a-linux-debugger-dwarf-step/
|
||||
[7]:https://blog.tartanllama.xyz/writing-a-linux-debugger-source-break/
|
||||
[8]:https://blog.tartanllama.xyz/writing-a-linux-debugger-unwinding/
|
||||
[9]:https://blog.tartanllama.xyz/writing-a-linux-debugger-variables/
|
||||
[10]:https://blog.tartanllama.xyz/writing-a-linux-debugger-advanced-topics/
|
||||
[11]:https://github.com/TartanLlama/libelfin/tree/fbreg
|
||||
[12]:http://dwarfstd.org/
|
||||
[13]:https://github.com/TartanLlama/minidbg/tree/tut_variable
|
||||
@@ -1,149 +0,0 @@
|
||||
Writing a Linux Debugger Part 10: Advanced topics
|
||||
============================================================
|
||||
|
||||
We’re finally here at the last post of the series! This time I’ll be giving a high-level overview of some more advanced concepts in debugging: remote debugging, shared library support, expression evaluation, and multi-threaded support. These ideas are more complex to implement, so I won’t walk through how to do so in detail, but I’m happy to answer questions about these concepts if you have any.
|
||||
|
||||
* * *
|
||||
|
||||
### Series index
|
||||
|
||||
1. [Setup][1]
|
||||
|
||||
2. [Breakpoints][2]
|
||||
|
||||
3. [Registers and memory][3]
|
||||
|
||||
4. [Elves and dwarves][4]
|
||||
|
||||
5. [Source and signals][5]
|
||||
|
||||
6. [Source-level stepping][6]
|
||||
|
||||
7. [Source-level breakpoints][7]
|
||||
|
||||
8. [Stack unwinding][8]
|
||||
|
||||
9. [Handling variables][9]
|
||||
|
||||
10. [Advanced topics][10]
|
||||
|
||||
* * *
|
||||
|
||||
### Remote debugging
|
||||
|
||||
Remote debugging is very useful for embedded systems or debugging the effects of environment differences. It also sets a nice divide between the high-level debugger operations and the interaction with the operating system and hardware. In fact, debuggers like GDB and LLDB operate as remote debuggers even when debugging local programs. The general architecture is this:
|
||||
|
||||
|
||||
|
||||
The debugger is the component which we interact with through the command line. Maybe if you’re using an IDE there’ll be another layer on top which communicates with the debugger through the _machine interface_ . On the target machine (which may be the same as the host) there will be a _debug stub_ , which in theory is a very small wrapper around the OS debug library which carries out all of your low-level debugging tasks like setting breakpoints on addresses. I say “in theory” because stubs are getting larger and larger these days. The LLDB debug stub on my machine is 7.6MB, for example. The debug stub communicates with the debugee process using some OS-specific features (in our case, `ptrace`), and with the debugger though some remote protocol.
|
||||
|
||||
The most common remote protocol for debugging is the GDB remote protocol. This is a text-based packet format for communicating commands and information between the debugger and debug stub. I won’t go into detail about it, but you can read all you could want to know about it [here][11]. If you launch LLDB and execute the command `log enable gdb-remote packets` then you’ll get a trace of all packets sent through the remote protocol. On GDB you can write `set remotelogfile <file>` to do the same.
|
||||
|
||||
As a simple example, here’s the packet to set a breakpoint:
|
||||
|
||||
```
|
||||
$Z0,400570,1#43
|
||||
|
||||
```
|
||||
|
||||
`$` marks the start of the packet. `Z0` is the command to insert a memory breakpoint. `400570` and `1` are the argumets, where the former is the address to set a breakpoint on and the latter is a target-specific breakpoint kind specifier. Finally, the `#43` is a checksum to ensure that there was no data corruption.
|
||||
|
||||
The GDB remote protocol is very easy to extend for custom packets, which is very useful for implementing platform- or language-specific functionality.
|
||||
|
||||
* * *
|
||||
|
||||
### Shared library and dynamic loading support
|
||||
|
||||
The debugger needs to know what shared libraries have been loaded by the debuggee so that it can set breakpoints, get source-level information and symbols, etc. As well as finding libraries which have been dynamically linked against, the debugger must track libraries which are loaded at runtime through `dlopen`. To facilitate this, the dynamic linker maintains a _rendezvous structure_ . This structure maintains a linked list of shared library descriptors, along with a pointer to a function which is called whenever the linked list is updated. This structure is stored where the `.dynamic` section of the ELF file is loaded, and is initialized before program execution.
|
||||
|
||||
A simple tracing algorithm is this:
|
||||
|
||||
* The tracer looks up the entry point of the program in the ELF header (or it could use the auxillary vector stored in `/proc/<pid>/aux`)
|
||||
|
||||
* The tracer places a breakpoint on the entry point of the program and begins execution.
|
||||
|
||||
* When the breakpoint is hit, the address of the rendezvous structure is found by looking up the load address of `.dynamic` in the ELF file.
|
||||
|
||||
* The rendezvous structure is examined to get the list of currently loaded libraries.
|
||||
|
||||
* A breakpoint is set on the linker update function.
|
||||
|
||||
* Whenever the breakpoint is hit, the list is updated.
|
||||
|
||||
* The tracer infinitely loops, continuing the program and waiting for a signal until the tracee signals that it has exited.
|
||||
|
||||
I’ve written a small demonstration of these concepts, which you can find [here][12]. I can do a more detailed write up of this in the future if anyone is interested.
|
||||
|
||||
* * *
|
||||
|
||||
### Expression evaluation
|
||||
|
||||
Expression evaluation is a feature which lets users evaluate expressions in the original source language while debugging their application. For example, in LLDB or GDB you could execute `print foo()` to call the `foo` function and print the result.
|
||||
|
||||
Depending on how complex the expression is, there are a few different ways of evaluating it. If the expression is a simple identifier, then the debugger can look at the debug information, locate the variable and print out the value, just like we did in the last part of this series. If the expression is a bit more complex, then it may be possible to compile the code to an intermediate representation (IR) and interpret that to get the result. For example, for some expressions LLDB will use Clang to compile the expression to LLVM IR and interpret that. If the expression is even more complex, or requires calling some function, then the code might need to be JITted to the target and executed in the address space of the debuggee. This involves calling `mmap` to allocate some executable memory, then the compiled code is copied to this block and is executed. LLDB does this by using LLVM’s JIT functionality.
|
||||
|
||||
If you want to know more about JIT compilation, I’d highly recommend [Eli Bendersky’s posts on the subject][13].
|
||||
|
||||
* * *
|
||||
|
||||
### Multi-threaded debugging support
|
||||
|
||||
The debugger shown in this series only supports single threaded applications, but to debug most real-world applications, multi-threaded support is highly desirable. The simplest way to support this is to trace thread creation and parse the procfs to get the information you want.
|
||||
|
||||
The Linux threading library is called `pthreads`. When `pthread_create` is called, the library creates a new thread using the `clone` syscall, and we can trace this syscall with `ptrace` (assuming your kernel is older than 2.5.46). To do this, you’ll need to set some `ptrace` options after attaching to the debuggee:
|
||||
|
||||
```
|
||||
ptrace(PTRACE_SETOPTIONS, m_pid, nullptr, PTRACE_O_TRACECLONE);
|
||||
```
|
||||
|
||||
Now when `clone` is called, the process will be signaled with our old friend `SIGTRAP`. For the debugger in this series, you can add a case to `handle_sigtrap` which can handle the creation of the new thread:
|
||||
|
||||
```
|
||||
case (SIGTRAP | (PTRACE_EVENT_CLONE << 8)):
|
||||
//get the new thread ID
|
||||
unsigned long event_message = 0;
|
||||
ptrace(PTRACE_GETEVENTMSG, pid, nullptr, message);
|
||||
|
||||
//handle creation
|
||||
//...
|
||||
```
|
||||
|
||||
Once you’ve got that, you can look in `/proc/<pid>/task/` and read the memory maps and suchlike to get all the information you need.
|
||||
|
||||
GDB uses `libthread_db`, which provides a bunch of helper functions so that you don’t need to do all the parsing and processing yourself. Setting up this library is pretty weird and I won’t show how it works here, but you can go and read [this tutorial][14] if you’d like to use it.
|
||||
|
||||
The most complex part of multithreaded support is modelling the thread state in the debugger, particularly if you want to support [non-stop mode][15] or some kind of heterogeneous debugging where you have more than just a CPU involved in your computation.
|
||||
|
||||
* * *
|
||||
|
||||
### The end!
|
||||
|
||||
Whew! This series took a long time to write, but I learned a lot in the process and I hope it was helpful. Get in touch on Twitter [@TartanLlama][16] or in the comments section if you want to chat about debugging or have any questions about the series. If there are any other debugging topics you’d like to see covered then let me know and I might do a bonus post.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://blog.tartanllama.xyz/writing-a-linux-debugger-advanced-topics/
|
||||
|
||||
作者：[Simon Brand ][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://www.twitter.com/TartanLlama
|
||||
[1]:https://blog.tartanllama.xyz/writing-a-linux-debugger-setup/
|
||||
[2]:https://blog.tartanllama.xyz/writing-a-linux-debugger-breakpoints/
|
||||
[3]:https://blog.tartanllama.xyz/writing-a-linux-debugger-registers/
|
||||
[4]:https://blog.tartanllama.xyz/writing-a-linux-debugger-elf-dwarf/
|
||||
[5]:https://blog.tartanllama.xyz/writing-a-linux-debugger-source-signal/
|
||||
[6]:https://blog.tartanllama.xyz/writing-a-linux-debugger-dwarf-step/
|
||||
[7]:https://blog.tartanllama.xyz/writing-a-linux-debugger-source-break/
|
||||
[8]:https://blog.tartanllama.xyz/writing-a-linux-debugger-unwinding/
|
||||
[9]:https://blog.tartanllama.xyz/writing-a-linux-debugger-variables/
|
||||
[10]:https://blog.tartanllama.xyz/writing-a-linux-debugger-advanced-topics/
|
||||
[11]:https://sourceware.org/gdb/onlinedocs/gdb/Remote-Protocol.html
|
||||
[12]:https://github.com/TartanLlama/dltrace
|
||||
[13]:http://eli.thegreenplace.net/tag/code-generation
|
||||
[14]:http://timetobleed.com/notes-about-an-odd-esoteric-yet-incredibly-useful-library-libthread_db/
|
||||
[15]:https://sourceware.org/gdb/onlinedocs/gdb/Non_002dStop-Mode.html
|
||||
[16]:https://twitter.com/TartanLlama
|
||||
@@ -1,3 +1,5 @@
|
||||
translating by Flowsnow
|
||||
|
||||
The user’s home dashboard in our app, AlignHow we built our first full-stack JavaScript web app in three weeks
|
||||
============================================================
|
||||
|
||||
@@ -176,7 +178,6 @@ For now, however, we’re happy with the final product, and with the immeasurabl
|
||||
|
||||
Team Align: Sara Kladky (left), Melanie Mohn (center), and myself.
|
||||
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://medium.com/ladies-storm-hackathons/how-we-built-our-first-full-stack-javascript-web-app-in-three-weeks-8a4668dbd67c?imm_mid=0f581a&cmp=em-web-na-na-newsltr_20170816
|
||||
|
||||
@@ -1,389 +0,0 @@
|
||||
translating by flowsow
|
||||
|
||||
How Machines Learn: A Practical Guide
|
||||
============================================================
|
||||
|
||||
|
||||
|
||||
You may have heard about machine learning from interesting applications like spam filtering, optical character recognition, and computer vision.
|
||||
|
||||
Getting started with machine learning is long process that involves going through several resources. There are books for newbies, academic papers, guided exercises, and standalone projects. It’s easy to lose track of what you need to learn among all these options.
|
||||
|
||||
So in today’s post, I’ll list seven steps (and 50+ resources) that can help you get started in this exciting field of Computer Science, and ramp up toward becoming a machine learning hero.
|
||||
|
||||
Note that this list of resources is not exhaustive and is meant to get you started. There are many more resources around.
|
||||
|
||||
### 1\. Get the necessary background knowledge
|
||||
|
||||
You might remember from DataCamp’s [Learn Data Science][77] infographic that mathematics and statistics are key to starting machine learning (ML). The foundations might seem quite easy because it’s just three topics. But don’t forget that these are in fact three broad topics.
|
||||
|
||||
There are two things that are very important to keep in mind here:
|
||||
|
||||
* First, you’ll definitely want some further guidance on what exactly you need to cover to get started.
|
||||
|
||||
* Second, these are the foundations of your further learning. Don’t be scared to take your time. Get the knowledge on which you’ll build everything.
|
||||
|
||||
The first point is simple: it’s a good idea to cover linear algebra and statistics. These two are the bare minimum that one should understand. But while you’re at it, you should also try to cover topics such as optimization and advanced calculus. They will come in handy when you’re getting deeper into ML.
|
||||
|
||||
Here are some pointers on where to get started if you are starting from zero:
|
||||
|
||||
* [Khan Academy][1] is a good resource for beginners. Consider taking the Linear Algebra and Calculus courses.
|
||||
|
||||
* Go to [MIT OpenCourseWare][2] and take the[ Linear Algebra][3] course.
|
||||
|
||||
* Take [this Coursera course][4] for an introduction to descriptive statistics, probability theory, and inferential statistics.
|
||||
|
||||
|
||||
|
||||
Statistics is one of the keys to learning ML
|
||||
|
||||
If you’re more into books, consider the following:
|
||||
|
||||
* [_Linear Algebra and Its Applications_][5] _,_
|
||||
|
||||
* [_Applied Linear Algebra_][6] ,
|
||||
|
||||
* [_3,000 Solved Problems in Linear Algebra_][7] _,_
|
||||
|
||||
* [MIT Online Texbooks][8]
|
||||
|
||||
However, in most cases, you’ll start off already knowing some things about statistics and mathematics. Or maybe you have already gone through all the theory resources listed above.
|
||||
|
||||
In these cases, it’s a good idea to recap and assess your knowledge honestly. Are there any areas that you need to revise or are you good for now?
|
||||
|
||||
If you’re all set, it’s time to go ahead and apply all that knowledge with R or Python. As a general guideline, it’s a good idea to pick one and get started with that language. Later, you can still add the other programming language to your skill set.
|
||||
|
||||
Why is all this programming knowledge necessary?
|
||||
|
||||
Well, you’ll see that the courses listed above (or those you have taken in school or university) will provide you with a more theoretical (and not applied) introduction to mathematics and statistics topics. However, ML is very applied and you’ll need to be able to apply all the topics you have learned. So it’s a good idea to go over the materials again, but this time in an applied way.
|
||||
|
||||
If you want to master the basics of R and Python, consider the following courses:
|
||||
|
||||
* DataCamp’s introductory Python or R courses: [Intro to Python for Data Science][9] or [Introduction to R Programming][10].
|
||||
|
||||
* Introductory Python and R courses from Edx: [Introduction to Python for Data Science][11] and [Introduction to R for Data Science][12].
|
||||
|
||||
* There are many other free courses out there. Check out [Coursera][13] or [Codeacademy][14] for more.
|
||||
|
||||
When you have nailed down the basics, check out DataCamp’s blog on the [40+ Python Statistics For Data Science Resources][78]. This post offers 40+ resources on the statistics topics you need to know to get started with data science (and by extension also ML).
|
||||
|
||||
Also make sure you check out [this SciPy tutorial][79] on vectors and arrays and [this workshop][80] on Scientific Computing with Python.
|
||||
|
||||
To get hands-on with Python and calculus, you can check out the [SymPy package][81].
|
||||
|
||||
### 2\. Don’t be scared to invest in the “theory” of ML
|
||||
|
||||
A lot of people don’t make the effort to go through some more theoretical material because it’s “dry” or “boring.” But going through the theory and really investing your time in it is essential and invaluable in the long run. You’ll better understand new advancements in machine learning, and you’ll be able to link back to your background knowledge. This will help you stay motivated.
|
||||
|
||||
Additionally, the theory doesn’t need to be boring. As you read in the introduction, there are so many materials that will make it easier for you to get into it.
|
||||
|
||||
Books are one of the best ways to absorb the theoretical knowledge. They force you to stop and think once in a while. Of course, reading books is a very static thing to do and it might not agree with your learning style. Nonetheless, try out the following books and see if it might be something for you:
|
||||
|
||||
* [_Machine Learning textbook_][15] , by Tom Mitchell might be old but it’s gold. This book goes over the most important topics in machine learning in a well-explained and step-by-step way.
|
||||
|
||||
* _Machine Learning: The Art and Science of Algorithms that Make Sense of Data _ (you can see the slides of the book [here][16]): this book is great for beginners. There are many real-life applications discussed, which you might find lacking in Tom Mitchell’s book.
|
||||
|
||||
* [_Machine Learning Yearning_][17] : this book by Andrew Ng is not yet complete, but it’s bound to be an excellent reference for those who are learning ML.
|
||||
|
||||
* [_Algorithms and Data Structures_][18] by Jurg Nievergelt and Klaus Hinrichs
|
||||
|
||||
* Also check out the [_Data Mining for the Masses_][19] by Matthew North. You’ll find that this book guides you through some of the most difficult topics.
|
||||
|
||||
* [_Introduction to Machine Learning_][20] by Alex Smola and S.V.N. Vishwanathan.
|
||||
|
||||
|
||||
|
||||
Take your time to read books and to study the material covered in them
|
||||
|
||||
Videos / MOOCs are awesome for those who learn by watching and listening. There are a lot of MOOCs and videos out there, but it can also be hard to find your way through all those materials. Below is a list of the most notable ones:
|
||||
|
||||
* [This well-known Machine Learning MOOC][21], taught by Andrew Ng, introduces you to Machine Learning and the theory. Don’t worry — it’s well-explained and takes things step-by-step, so it’s excellent for beginners.
|
||||
|
||||
* The [playlist of the MIT Open Courseware 6034 course][22]: already a bit more advanced. You’ll definitely need some previous work on ML theory before you start this series, but you won’t regret it.
|
||||
|
||||
At this point, it’s important for you to go over the separate techniques and grasp the whole picture. This starts with understanding key concepts: the distinction between supervised and unsupervised learning, classification and regression, and so on. Manual (written) exercises can come in handy. They can help you understand how algorithms work and how you should go about them. You’ll most often find these written exercises in courses from universities. Check out [this ML course][82] by Portland State University.
|
||||
|
||||
### 3\. Get hands-on
|
||||
|
||||
Knowing the theory and understanding the algorithms by reading and watching is all good. But you also need to surpass this stage and get started with some exercises. You’ll learn to implement these algorithms and apply the theory that you’ve learned.
|
||||
|
||||
First, you have tutorials which introduce you to the basics of machine learning in Python and R. The best way is, of course, to go for interactive tutorials:
|
||||
|
||||
* In [Python Machine Learning: Scikit-Learn Tutorial][23], you will learn more about well-known algorithms KMeans and Support Vector Machines (SVM) to construct models with Scikit-Learn.
|
||||
|
||||
* [Machine Learning in R for beginners][24] introduces you to ML in R with the class and caret packages.
|
||||
|
||||
* [Keras Tutorial: Deep Learning in Python covers ][25]how to build Multi-Layer Perceptrons (MLPs) for classification and regression tasks, step-by-step.
|
||||
|
||||
Also check out the following tutorials, which are static and will require you to work in an IDE:
|
||||
|
||||
* [Machine Learning in Python, Step By Step][26]: step-by-step tutorial with Scikit-Learn.
|
||||
|
||||
* [Develop Your First Neural Network in Python With Keras Step-By-Step][27]: learn how to develop your first neural network with Keras thanks to this tutorial.
|
||||
|
||||
* There are many more that you can consider, but the tutorials of [Machine Learning Mastery][28] are very good.
|
||||
|
||||
Besides the tutorials, there are also courses. Taking courses will help you apply the concepts that you’ve learned in a focused way. Experienced instructors will help you. Here are some interactive courses for Python and ML:
|
||||
|
||||
* [Supervised Learning with scikit-learn][29]: you’ll learn how to build predictive models, tune their parameters, and predict how well they will perform on unseen data. All while using real world datasets. You’ll do so with Scikit-Learn.
|
||||
|
||||
* [Unsupervised Learning in Python][30]: shows you how to cluster, transform, visualize, and extract insights from unlabeled datasets. At the end of the course, you’ll build a recommender system.
|
||||
|
||||
* [Deep Learning in Python][31]: you’ll gain hands-on, practical knowledge of how to use deep learning with Keras 2.0, the latest version of a cutting-edge library for deep learning in Python.
|
||||
|
||||
* [Applied Machine Learning in Python][32]: introduces the learner to applied ML and focuses more on the techniques and methods than on the statistics behind these methods.
|
||||
|
||||
|
||||
|
||||
|
||||
After the theory, take your time to apply the knowledge you have gained.
|
||||
|
||||
For those who are learning ML with R, there are also these interactive courses:
|
||||
|
||||
* [Introduction to Machine Learning][33] gives you a broad overview of the discipline’s most common techniques and applications. You’ll gain more insight into the assessment and training of different ML models. The rest of the course focuses on an introduction to three of the most basic ML tasks: classification, regression, and clustering.
|
||||
|
||||
* [R: Unsupervised Learning][34] provides a basic introduction to clustering and dimensionality reduction in R from a ML perspective. This allows you to get from data to insights as quickly as possible.
|
||||
|
||||
* [Practical Machine Learning][35] covers the basic components of building and applying prediction functions with an emphasis on practical applications.
|
||||
|
||||
Lastly, there are also books that go over ML topics in a very applied way. If you’re looking to learn with the help of text and an IDE, check out these books:
|
||||
|
||||
* The [_Python Machine Learning Book_][36] by Sebastian Raschka
|
||||
|
||||
* The [Introduction to Artificial Neural Networks and Deep Learning: A Practical Guide with Applications in Python][37] by Sebastian Raschka
|
||||
|
||||
* [_Machine Learning with R_][38] by Brett Lantz
|
||||
|
||||
### 4\. Practice
|
||||
|
||||
Practice is even more important than getting hands-on and revising the material with Python. This step was probably the hardest one for me. Check out how other people have implemented ML algorithms when you have done some exercises. Then, get started on your own projects that illustrate your understanding of ML algorithms and theories.
|
||||
|
||||
One of the most straightforward ways is to see the exercises a tiny bit bigger. You want to do a bigger exercise which requires you to do more data cleaning and feature engineering.
|
||||
|
||||
* Start with[ Kaggle][39]. If you need additional help to conquer the so-called “data fear,” check out the [Kaggle Python Tutorial on Machine Learning][40]and[ Kaggle R Tutorial on Machine Learning][41]. These will bring you up to speed in no time.
|
||||
|
||||
* Afterwards, you can also start doing challenges by yourself. Check out these sites, where you can find lots of ML datasets: [UCI Machine Learning Repository][42], [Public datasets for machine learning][43], and [data.world][44].
|
||||
|
||||
|
||||
|
||||
Practice makes perfect.
|
||||
|
||||
### 5\. Projects
|
||||
|
||||
Doing small exercises is good. But in the end, you’ll want to make a project in which you can demonstrate your understanding of the ML algorithms with which you’ve been working.
|
||||
|
||||
The best exercise is to implement your own ML algorithm. You can read more about why you should do this exercise and what you can learn from it in the following pages:
|
||||
|
||||
* [Why is there a need to manually implement machine learning algorithms when there are many advanced APIs like tensorflow available?][45]
|
||||
|
||||
* [Why Implement Machine Learning Algorithms From Scratch?][46]
|
||||
|
||||
* [What I Learned Implementing a Classifier from Scratch in Python][47]
|
||||
|
||||
Next, you can check out the following posts and repositories. They’ll give you some inspiration from others and will show how they have implemented ML algorithms.
|
||||
|
||||
* [How to Implement a Machine Learning Algorithm][48]
|
||||
|
||||
* [ML From Scratch][49]
|
||||
|
||||
* [Machine Learning Algorithms From Scratch][50]
|
||||
|
||||
|
||||
|
||||
|
||||
Projects can be hard at start, but they’ll increase your understanding even more.
|
||||
|
||||
### 6\. Don’t stop
|
||||
|
||||
Learning ML is something that should never stop. As many will confirm, there are always new things to learn — even when you’ve been working in this area for a decade.
|
||||
|
||||
There are, for example, ML trends such as deep learning which are very popular right now. You might also focus on other topics that aren’t central at this point but which might be in the future. Check out this [interesting question and the answers][83] if you want to know more.
|
||||
|
||||
Papers may not be the first thing that spring to mind when you’re worried about mastering the basics. But they are your way to get up to date with the latest research. Papers are not for those who are just starting out. They are definitely a good fit for those who are more advanced.
|
||||
|
||||
* [Top 20 Recent Research Papers on Machine Learning and Deep Learning][51]
|
||||
|
||||
* [Journal of Machine Learning Research][52]
|
||||
|
||||
* [Awesome Deep Learning Papers][53]
|
||||
|
||||
* [What are some of the best research papers/books for Machine learning?][54]
|
||||
|
||||
Other technologies are also something to consider. But don’t worry about them when you’re just starting out. You can, for example, focus on adding Python or R (depending on which one you already know) to your skill set. You can look through this post to find interesting resources.
|
||||
|
||||
If you also want to move towards big data, you could consider looking into Spark. Here are some interesting resources:
|
||||
|
||||
* [Introduction to Spark in R with sparklyr][55]
|
||||
|
||||
* [Data Science And Engineering With Spark][56]
|
||||
|
||||
* [Introduction to Apache Spark][57]
|
||||
|
||||
* [Distributed Machine Learning with Apache Spark][58]
|
||||
|
||||
* [Big Data Analysis with Apache Spark][59]
|
||||
|
||||
* [Apache Spark in Python: Beginner’s Guide][60]
|
||||
|
||||
* [PySpark RDD Cheat Sheet][61]
|
||||
|
||||
* [PySpark SQL Cheat Sheet][62].
|
||||
|
||||
Other programming languages, such as Java, JavaScript, C, and C++ are gaining importance in ML. In the long run, you can consider also adding one of these languages to your to-do list. You can use these blog posts to guide your choice:
|
||||
|
||||
* [Most Popular Programming Languages for Machine Learning and Data Science][63]
|
||||
|
||||
* [The Most Popular Language For Machine Learning And Data Science Is…][64]
|
||||
|
||||
|
||||
|
||||
You’re never done learning.
|
||||
|
||||
### 7\. Make use of all the material that is out there
|
||||
|
||||
Machine learning is a difficult topic which can make you lose your motivation at some point. Or maybe you feel you need a change. In such cases, remember that there’s a lot of material on which you can fall back. Check out the following resources:
|
||||
|
||||
Podcasts. Great resource for continuing your journey into ML and staying up-to-date with the latest developments in the field:
|
||||
|
||||
* [Talking Machines][65]
|
||||
|
||||
* [Data Skeptic][66]
|
||||
|
||||
* [Linear Digressions][67]
|
||||
|
||||
* [This Week in Machine Learning & AI][68]
|
||||
|
||||
* [Learning Machines 101][69]
|
||||
|
||||
There are, of course, many more podcasts.
|
||||
|
||||
Documentation and package source code are two ways to get deeper into the implementation of the ML algorithms. Check out some of these repositories:
|
||||
|
||||
* [Scikit- Learn][70]: Well-known Python ML package
|
||||
|
||||
* [Keras][71]: Deep learning package for Python
|
||||
|
||||
* [caret][72]: very popular R package for Classification and Regression Training
|
||||
|
||||
Visualizations are one of the newest and trendiest ways to get into the theory of ML. They’re fantastic for beginners, but also very interesting for more advanced learners. The following visualizations will intrigue you and will help you gain more understanding into the workings of ML:
|
||||
|
||||
* [A visual introduction to machine learning][73]
|
||||
|
||||
* [Distill][74] makes ML Research clear, dynamic and vivid.
|
||||
|
||||
* [Tensorflow — Neural Network Playground][75] if you’re looking to play around with neural network architectures.
|
||||
|
||||
* More here:[ What are the best visualizations of machine learning algorithms?][76]
|
||||
|
||||
|
||||
|
||||
Some variety in your learning can and will motivate you even more.
|
||||
|
||||
### You Can Get Started Now
|
||||
|
||||
Now it’s up to you. Learning ML is something that’s a continuous process, so the sooner you get started, the better. You have all of the tools in your hands now to get started. Good luck and make sure to let us know how you’re progressing.
|
||||
|
||||
_This post is based on an answer I gave to the Quora question _ [_How Does A Total Beginner Start To Learn Machine Learning_][84] _._
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
作者简介：
|
||||
|
||||
Karlijn Willems
|
||||
|
||||
Data Science Journalist
|
||||
|
||||
-----------------------
|
||||
|
||||
via: https://medium.freecodecamp.org/how-machines-learn-a-practical-guide-203aae23cafb
|
||||
|
||||
作者：[ Karlijn Willems][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://medium.freecodecamp.org/@kacawi
|
||||
[1]:http://www.khanacademy.org/
|
||||
[2]:https://ocw.mit.edu/index.htm
|
||||
[3]:https://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-spring-2010/
|
||||
[4]:https://www.coursera.org/learn/basic-statistics
|
||||
[5]:https://www.amazon.com/Linear-Algebra-Its-Applications-4th/dp/0030105676
|
||||
[6]:https://www.amazon.com/Applied-Linear-Algebra-3rd-Noble/dp/0130412600
|
||||
[7]:https://www.amazon.de/Solved-Problems-Linear-Algebra-Schaums/dp/0070380236
|
||||
[8]:https://ocw.mit.edu/courses/online-textbooks/
|
||||
[9]:https://www.datacamp.com/courses/intro-to-python-for-data-science
|
||||
[10]:https://www.datacamp.com/courses/free-introduction-to-r
|
||||
[11]:https://www.edx.org/course/introduction-python-data-science-microsoft-dat208x-5
|
||||
[12]:https://www.edx.org/course/introduction-r-data-science-microsoft-dat204x-4
|
||||
[13]:http://www.coursera.org/
|
||||
[14]:https://www.codecademy.com/
|
||||
[15]:http://www.cs.cmu.edu/~tom/mlbook.html
|
||||
[16]:http://www.cs.bris.ac.uk/~flach/mlbook/materials/mlbook-beamer.pdf
|
||||
[17]:http://www.mlyearning.org/
|
||||
[18]:https://www.amazon.com/Algorithms-Data-Structures-Applications-Practitioner/dp/0134894286
|
||||
[19]:https://www.amazon.com/Data-Mining-Masses-Matthew-North/dp/0615684378
|
||||
[20]:http://alex.smola.org/drafts/thebook.pdf
|
||||
[21]:https://www.coursera.org/learn/machine-learning
|
||||
[22]:https://youtu.be/TjZBTDzGeGg?list=PLnvKubj2-I2LhIibS8TOGC42xsD3-liux
|
||||
[23]:https://www.datacamp.com/community/tutorials/machine-learning-python
|
||||
[24]:https://www.datacamp.com/community/tutorials/machine-learning-in-r
|
||||
[25]:https://www.datacamp.com/community/tutorials/deep-learning-python
|
||||
[26]:http://machinelearningmastery.com/machine-learning-in-python-step-by-step/
|
||||
[27]:http://machinelearningmastery.com/tutorial-first-neural-network-python-keras/
|
||||
[28]:http://www.machinelearningmastery.com/
|
||||
[29]:https://www.datacamp.com/courses/supervised-learning-with-scikit-learn
|
||||
[30]:https://www.datacamp.com/courses/unsupervised-learning-in-python
|
||||
[31]:https://www.datacamp.com/courses/deep-learning-in-python
|
||||
[32]:https://www.coursera.org/learn/python-machine-learning
|
||||
[33]:https://www.datacamp.com/courses/introduction-to-machine-learning-with-r
|
||||
[34]:https://www.datacamp.com/courses/unsupervised-learning-in-r
|
||||
[35]:https://www.coursera.org/learn/practical-machine-learning
|
||||
[36]:https://github.com/rasbt/python-machine-learning-book
|
||||
[37]:https://github.com/rasbt/deep-learning-book
|
||||
[38]:https://books.google.be/books/about/Machine_Learning_with_R.html?id=ZQu8AQAAQBAJ&source=kp_cover&redir_esc=y
|
||||
[39]:http://www.kaggle.com/
|
||||
[40]:https://www.datacamp.com/community/open-courses/kaggle-python-tutorial-on-machine-learning
|
||||
[41]:https://www.datacamp.com/community/open-courses/kaggle-tutorial-on-machine-learing-the-sinking-of-the-titanic
|
||||
[42]:http://archive.ics.uci.edu/ml/
|
||||
[43]:http://homepages.inf.ed.ac.uk/rbf/IAPR/researchers/MLPAGES/mldat.htm
|
||||
[44]:https://data.world/
|
||||
[45]:https://www.quora.com/Why-is-there-a-need-to-manually-implement-machine-learning-algorithms-when-there-are-many-advanced-APIs-like-tensorflow-available
|
||||
[46]:http://www.kdnuggets.com/2016/05/implement-machine-learning-algorithms-scratch.html
|
||||
[47]:http://www.jeannicholashould.com/what-i-learned-implementing-a-classifier-from-scratch.html
|
||||
[48]:http://machinelearningmastery.com/how-to-implement-a-machine-learning-algorithm/
|
||||
[49]:https://github.com/eriklindernoren/ML-From-Scratch
|
||||
[50]:https://github.com/madhug-nadig/Machine-Learning-Algorithms-from-Scratch
|
||||
[51]:http://www.kdnuggets.com/2017/04/top-20-papers-machine-learning.html
|
||||
[52]:http://www.jmlr.org/
|
||||
[53]:https://github.com/terryum/awesome-deep-learning-papers
|
||||
[54]:https://www.quora.com/What-are-some-of-the-best-research-papers-books-for-Machine-learning
|
||||
[55]:https://www.datacamp.com/courses/introduction-to-spark-in-r-using-sparklyr
|
||||
[56]:https://www.edx.org/xseries/data-science-engineering-apache-spark
|
||||
[57]:https://www.edx.org/course/introduction-apache-spark-uc-berkeleyx-cs105x
|
||||
[58]:https://www.edx.org/course/distributed-machine-learning-apache-uc-berkeleyx-cs120x
|
||||
[59]:https://www.edx.org/course/big-data-analysis-apache-spark-uc-berkeleyx-cs110x
|
||||
[60]:https://www.datacamp.com/community/tutorials/apache-spark-python
|
||||
[61]:https://www.datacamp.com/community/blog/pyspark-cheat-sheet-python
|
||||
[62]:https://www.datacamp.com/community/blog/pyspark-sql-cheat-sheet
|
||||
[63]:https://fossbytes.com/popular-top-programming-languages-machine-learning-data-science/
|
||||
[64]:http://www.kdnuggets.com/2017/01/most-popular-language-machine-learning-data-science.html
|
||||
[65]:http://www.thetalkingmachines.com/
|
||||
[66]:https://dataskeptic.com/
|
||||
[67]:http://lineardigressions.com/
|
||||
[68]:https://twimlai.com/
|
||||
[69]:http://www.learningmachines101.com/
|
||||
[70]:https://github.com/scikit-learn/scikit-learn
|
||||
[71]:http://www.github.com/fchollet/keras
|
||||
[72]:http://topepo/caret
|
||||
[73]:http://www.r2d3.us/visual-intro-to-machine-learning-part-1/
|
||||
[74]:http://distill.pub/
|
||||
[75]:http://playground.tensorflow.org/
|
||||
[76]:https://www.quora.com/What-are-the-best-visualizations-of-machine-learning-algorithms
|
||||
[77]:https://www.datacamp.com/community/tutorials/learn-data-science-infographic
|
||||
[78]:https://www.datacamp.com/community/tutorials/python-statistics-data-science
|
||||
[79]:https://www.datacamp.com/community/tutorials/python-scipy-tutorial
|
||||
[80]:http://www.math.pitt.edu/~siam/workshops/python10/python.pdf
|
||||
[81]:http://docs.sympy.org/latest/tutorial/calculus.html
|
||||
[82]:http://web.cecs.pdx.edu/~mm/MachineLearningSpring2017/
|
||||
[83]:https://www.quora.com/Should-I-quit-machine-learning
|
||||
[84]:https://www.quora.com/How-does-a-total-beginner-start-to-learn-machine-learning/answer/Karlijn-Willems-1
|
||||
@@ -0,0 +1,88 @@
|
||||
Translating by Penney94
|
||||
[GIVE AWAY YOUR CODE, BUT NEVER YOUR TIME][23]
|
||||
============================================================
|
||||
|
||||
As software developers, I think we can agree that open-source code has [transformed the world][9]. Its public nature tears down the walls that prevent some pieces of software from becoming the best they can be. The problem is that too many valuable projects stagnate, with burned-out leaders:
|
||||
|
||||
> “I do not have the time or energy to invest in open source any more. I am not being paid at all to do any open source work, and so the work that I do there is time that I could be spending doing ‘life stuff’, or writing…It’s for this reason that I've decided to end all my engagements with open source effective today.”
|
||||
>
|
||||
> —[Ryan Bigg, former maintainer of several Ruby and Elixir projects][1]
|
||||
>
|
||||
> “It’s also been a massive opportunity cost because of all the things I haven’t learned or done in the meantime because FubuMVC takes up so much of my time and that’s the main reason that it has to stop now.”
|
||||
>
|
||||
> —[Jeremy Miller, former project lead of FubuMVC][2]
|
||||
>
|
||||
> “When we decide to start having kids, I will probably quit open source for good…I anticipate that ultimately this will be the solution to my problem: the nuclear option.”
|
||||
>
|
||||
> —[Nolan Lawson, one of the maintainers of PouchDB][3]
|
||||
|
||||
What we need is a new industry norm, that project leaders will _always_ be compensated for their time. We also need to bury the idea that any developer who submits an issue or pull request is automatically entitled to the attention of a maintainer.
|
||||
|
||||
Let’s first review how an open-source code base works in the market. It is a building block. It is [utility software][10], a cost that must be incurred by a business to make profit elsewhere. The community around the software grows if users can both understand the purpose of the code and see that it is a better value than the alternatives (closed-source off-the-shelf, custom in-house solution, etc.). It can be better, cheaper, or both.
|
||||
|
||||
If an organization needs to improve the code, they are free to hire any developer they want. It’s usually [in their interest][11] to contribute the improvement back to the community because, due to the complexity of merging, that’s the only way they can easily receive future improvements from other users. This “gravity” tends to hold communities together.
|
||||
|
||||
But it also burdens project maintainers since they must respond to these incoming improvements. And what do they get in return? At best, a community contribution may be something they can use in the future but not right now. At worst, it is nothing more than a selfish request wearing the mask of altruism.
|
||||
|
||||
One class of open-source projects has avoided this trap. What do Linux, MySQL, Android, Chromium, and .NET Core have in common, besides being famous? They are all _strategically important_ to one or more big-business interests because they complement those interests. [Smart companies commoditize their complements][12] and there’s no commodity cheaper than open-source software. Red Hat needs companies using Linux in order to sell Enterprise Linux, Oracle uses MySQL as a gateway drug that leads to MySQL Enterprise, Google wants everyone in the world to have a phone and web browser, and Microsoft is trying to hook developers on a platform and then pull them into the Azure cloud. These projects are all directly funded by the respective companies.
|
||||
|
||||
But what about the rest of the projects out there, that aren’t at the center of a big player’s strategy?
|
||||
|
||||
If you’re the leader of one of these projects, charge an annual fee for community membership. _Open source, closed community._ The message to users should be “do whatever you want with the code, but _pay us for our time_ if you want to influence the project’s future.” Lock non-paying users out of the forum and issue tracker, and ignore their emails. People who don’t pay should feel like they are missing out on the party.
|
||||
|
||||
Also charge contributors for the time it takes to merge nontrivial pull requests. If a particular submission will not immediately benefit you, charge full price for your time. Be disciplined and [remember YAGNI][13].
|
||||
|
||||
Will this lead to a drastically smaller community, and more forks? Absolutely. But if you persevere in building out your vision, and it delivers value to anyone else, they will pay as soon as they have a contribution to make. _Your willingness to merge contributions is [the scarce resource][4]._ Without it, users must repeatedly reconcile their changes with every new version you release.
|
||||
|
||||
Restricting the community is especially important if you want to maintain a high level of [conceptual integrity][14] in the code base. Headless projects with [liberal contribution policies][15] have less of a need to charge.
|
||||
|
||||
To implement larger pieces of your vision that do not justify their cost for your business alone, but may benefit others, [crowdfund][16]. There are many success stories:
|
||||
|
||||
> [Font Awesome 5][5]
|
||||
>
|
||||
> [Ruby enVironment Management (RVM)][6]
|
||||
>
|
||||
> [Django REST framework 3][7]
|
||||
|
||||
[Crowdfunding has limitations][17]. It [doesn’t work][18] for [huge projects][19]. But again, open-source code is utility software, which doesn’t need ambitious, risky game-changers. It has already [permeated every industry][20] with only incremental updates.
|
||||
|
||||
These ideas represent a sustainable path forward, and they could also fix the [diversity problem in open source][21], which may be rooted in its historically-unpaid nature. But above all, let’s remember that we only have [so many keystrokes left in our lives][22], and that we will someday regret the ones we waste.
|
||||
|
||||
_When I say “open source”, I mean code [licensed][8] in a way that it can be used to build proprietary things. This usually means a permissive license (MIT or Apache or BSD), but not always. Linux is the core of today’s tech industry, yet it is licensed under the GPL._
|
||||
|
||||
Thanks to Jason Haley, Don McNamara, Bryan Hogan, and Nadia Eghbal for reading drafts of this.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: http://wgross.net/essays/give-away-your-code-but-never-your-time
|
||||
|
||||
作者：[William Gross][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:http://wgross.net/#about-section
|
||||
[1]:http://ryanbigg.com/2015/11/open-source-work
|
||||
[2]:https://jeremydmiller.com/2014/04/03/im-throwing-in-the-towel-in-fubumvc/
|
||||
[3]:https://nolanlawson.com/2017/03/05/what-it-feels-like-to-be-an-open-source-maintainer/
|
||||
[4]:https://hbr.org/2010/11/column-to-win-create-whats-scarce
|
||||
[5]:https://www.kickstarter.com/projects/232193852/font-awesome-5
|
||||
[6]:https://www.bountysource.com/teams/rvm/fundraiser
|
||||
[7]:https://www.kickstarter.com/projects/tomchristie/django-rest-framework-3
|
||||
[8]:https://choosealicense.com/
|
||||
[9]:https://www.wired.com/insights/2013/07/in-a-world-without-open-source/
|
||||
[10]:https://martinfowler.com/bliki/UtilityVsStrategicDichotomy.html
|
||||
[11]:https://tessel.io/blog/67472869771/monetizing-open-source
|
||||
[12]:https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/
|
||||
[13]:https://martinfowler.com/bliki/Yagni.html
|
||||
[14]:http://wiki.c2.com/?ConceptualIntegrity
|
||||
[15]:https://opensource.com/life/16/5/growing-contributor-base-modern-open-source
|
||||
[16]:https://poststatus.com/kickstarter-open-source-project/
|
||||
[17]:http://blog.felixbreuer.net/2013/04/24/crowdfunding-for-open-source.html
|
||||
[18]:https://www.indiegogo.com/projects/geary-a-beautiful-modern-open-source-email-client#/
|
||||
[19]:http://www.itworld.com/article/2708360/open-source-tools/canonical-misses-smartphone-crowdfunding-goal-by--19-million.html
|
||||
[20]:http://www.infoworld.com/article/2914643/open-source-software/rise-and-rise-of-open-source.html
|
||||
[21]:http://readwrite.com/2013/12/11/open-source-diversity/
|
||||
[22]:http://keysleft.com/
|
||||
[23]:http://wgross.net/essays/give-away-your-code-but-never-your-time
|
||||
467
sources/tech/20170908 Betting on the Web.md
Normal file
467
sources/tech/20170908 Betting on the Web.md
Normal file
@@ -0,0 +1,467 @@
|
||||
[Betting on the Web][27]
|
||||
============================================================
|
||||
|
||||
|
||||
|
||||
_Note: I just spoke at [Coldfront 2017][12] about why I’m such a big proponent of the Web. What follows is essentially that talk as a blog post (I’ll add a link to the video once it is published)._
|
||||
|
||||
_Also: the Starbucks PWA mentioned in the talk has shipped! 🎉_
|
||||
|
||||
I’m _not_ going to tell you what to do. Instead, I’m going to explain why I’ve chosen to bet my whole career on this crazy Web thing. "Betting" sounds a bit haphazard, it’s more calculated than that. It would probably be better described as "investing."
|
||||
|
||||
Investing what? Our time and attention.
|
||||
|
||||
Many of us only have maybe 6 or so _really_ productive hours per day when we’re capable of being super focused and doing our absolute best work. So how we chose to invest that very limited time is kind of a big deal. Even though I really enjoy programming I rarely do it aimlessly just for the pure joy of it. Ultimately, I’m investing that productive time expecting to get _some kind of return_ even if it’s just mastering something or solving a difficult problem.
|
||||
|
||||
[### "So what, what’s your point?"][28]
|
||||
|
||||
> > More than most of us realize we are _constantly_ investing
|
||||
|
||||
Sure, someone may be paying for our time directly but there’s more to it than just trading hours for money. In the long run, what we chose to invest our professional efforts into has other effects:
|
||||
|
||||
**1\. Building Expertise:** We learn as we work and gain valuable experience in the technologies and platform we’re investing in. That expertise impacts our future earning potential and what types of products we’re capable of building.
|
||||
|
||||
**2\. Building Equity:** Hopefully we’re generating equity and adding value to whatever product we’re building.
|
||||
|
||||
**3\. Shaping tomorrow’s job market:** We’re building tomorrow’s legacy code today™. Today’s new hotness is tomorrow’s maintenance burden. In many cases the people that initially build a product or service are not the ones that ultimately maintain it. This means the technology choices we make when building a new product or service, determine whether or not there will be jobs later that require expertise in that particular platform/technology. So, those tech choices _literally shape tomorrow’s job market!_
|
||||
|
||||
**4\. Body of knowledge:** As developers, we’re pretty good at sharing what we learn. We blog, we "Stack-Overflow", etc. These things all contribute to the corpus of knowledge available about that given platform which adds significant value by making it easier/faster for others to build things using these tools.
|
||||
|
||||
**5\. Open Source:** We solve problems and share our work. When lots of developers do this it adds _tremendous value_ to the technologies and platforms these tools are for. The sheer volume of work that we _don’t have to do_ because we can use someone else’s library that already does it is mind-boggling. Millions and millions of hours of development work are available to us for free with a simple `npm install`.
|
||||
|
||||
**6\. Building apps for users on that platform:** Last but not least, without apps there is no platform. By making more software available to end users, we’re contributing significant value to the platforms that run our apps.
|
||||
|
||||
Looking at that list, the last four items are not about _us_ at all. They represent other significant long-term impacts.
|
||||
|
||||
> > We often have a broader impact than we realize
|
||||
|
||||
We’re not just investing time into a job, we're also shaping the platform, community, and technologies we use.
|
||||
|
||||
We’re going to come back to this, but hopefully, recognizing that greater impact can help us make better investments.
|
||||
|
||||
[### With all investing comes _risk_][29]
|
||||
|
||||
We can’t talk about investing without talking about risk. So what are some of the potential risks?
|
||||
|
||||
[### Are we building for the right platform?][30]
|
||||
|
||||
Platform stability is indeed A Thing™. Just ask a Flash developer, Windows Phone developer, or Blackberry developer. Platforms _can_ go away.
|
||||
|
||||
If we look at those three platforms, what do they have in common? They’re _closed_ platforms. What I mean is there’s a single controlling interest. When you build for them, you’re building for a specific operating system and coding against a particular implementation as opposed to coding against a set of _open standards_ . You could argue, that at least to some degree, Flash died because of its "closed-ness". Regardless, one thing is clear from a risk mitigation perspective: open is better than closed.
|
||||
|
||||
the Web is _incredibly_ open. It would be quite difficult for any one entity to kill it off.
|
||||
|
||||
Now, for Windows Phone/Blackberry it failed due to a lack of interested users... or was it lack of interested developers??
|
||||
|
||||
|
||||
|
||||
Maybe if Ballmer ☝️ has just yelled "developers" _one more time_ we’d all have Windows Phones in our pockets right now 😜.
|
||||
|
||||
From a risk mitigation perspective, two things are clear with regard to platform stability:
|
||||
|
||||
1. Having _many users_ is better than having few users
|
||||
|
||||
2. Having _more developers_ building for the platform is better than having few developers
|
||||
|
||||
> > There is no bigger more popular open platform than the Web
|
||||
|
||||
[### Are we building the right software?][31]
|
||||
|
||||
Many of us are building apps. Well, we used to build "applications" but that wasn’t nearly cool enough. So now we build "apps" instead 😎.
|
||||
|
||||
What does "app" mean to a user? This is important because I think it’s changed a bit over the years. To a user, I would suggest it basically means: "a thing I put on my phone."
|
||||
|
||||
But for our purposes I want to get a bit more specific. I’d propose that an app is really:
|
||||
|
||||
1. An "ad hoc" user interface
|
||||
|
||||
2. That is local(ish) to the device
|
||||
|
||||
The term "ad hoc" is Latin and translates to **"for this"**. This actually matches pretty closely with what Apple’s marketing campaigns have been teaching the masses:
|
||||
|
||||
> There’s an app **for that**
|
||||
>
|
||||
> – Apple
|
||||
|
||||
The point is it helps you _do_ something. The emphasis is on action. I happen to think this is largely the difference between a "site" and an "app". A news site for example has articles that are resources in and of themselves. Where a news app is software that runs on the device that helps you consume news articles.
|
||||
|
||||
Another way to put it would be that site is more like a book, while an app is a tool.
|
||||
|
||||
[### Should we be building apps at all?!][32]
|
||||
|
||||
Remember when chatbots were supposed to take over the world? Or perhaps we’ll all be walking around with augmented reality glasses and that’s how we’ll interact with the world?
|
||||
|
||||
I’ve heard it said that "the future app is _no_ app" and virtual assistants will take over everything.
|
||||
|
||||
|
||||
|
||||
I’ve had one of these sitting in my living room for a couple of years, but I find it all but useless. It’s just a nice bluetooth speaker that I can yell at to play me music.
|
||||
|
||||
But I find it very interesting that:
|
||||
|
||||
> > Even Alexa has an app!
|
||||
|
||||
Why? Because there’s no screen! As it turns out these "ad hoc visual interfaces" are extremely efficient.
|
||||
|
||||
Sure, I can yell out "Alexa, what’s the weather going to be like today" and I’ll hear a reply with high and low and whether it’s cloudy, rainy, or sunny. But in that same amount of time, I can pull my phone out tap the weather app and before Alexa can finish telling me those 3 pieces of data, I can visually scan the entire week’s worth of data, air quality, sunrise/sunset times, etc. It’s just _so much more_ efficient as a mechanism for consuming this type of data.
|
||||
|
||||
As a result of that natural efficiency, I believe that having a visual interface is going to continue to be useful for all sorts of things for a long time to come.
|
||||
|
||||
That’s _not_ to say virtual assistants aren’t useful! Google Assistant on my Pixel is quite useful in part because it can show me answers and can tolerate vagueness in a way that an app with a fixed set of buttons never could.
|
||||
|
||||
But, as is so often the case with new useful tech, rarely does it complete replace everything that came before it, instead, it augments what we already have.
|
||||
|
||||
[### If apps are so great why are we so "apped out"?][33]
|
||||
|
||||
How do we explain that supposed efficiency when there’s data like this?
|
||||
|
||||
* [65% of smartphone users download zero apps per month][13]
|
||||
|
||||
* [More than 75% of app downloads open an app once and never come back][14]
|
||||
|
||||
I think to answer that we have to really look at what isn’t working well.
|
||||
|
||||
[### What sucks about apps?][34]
|
||||
|
||||
1. **Downloading them certainly sucks.** No one wants to open an app store, search for the app they’re trying to find, then wait to download the huge file. These days a 50mb app is pretty small. Facebook for iOS 346MB, Twitter iOS 212MB.
|
||||
|
||||
2. **Updating them sucks.** Every night I plug in my phone I download a whole slew of app updates that I, as a user, **could not possibly care less about**. In addition, many of these apps are things I installed _once_ and will **never open again, ever!**. I’d love to know the global stats on how much bandwidth has been wasted on app updates for apps that were never opened again.
|
||||
|
||||
3. **Managing them sucks.** Sure, when I first got an iPhone ages ago and could first download apps my home screen was impeccable. Then when we got folders!! Wow... what an amazing development! Now I could finally put all those pesky uninstallable Apple apps in a folder called "💩" and pretend they didn’t exist. But now, my home screen is a bit of a disaster. Sitting there dragging apps around is not my idea of a good time. So eventually things get all cluttered up again.
|
||||
|
||||
The thing I’ve come to realize, is this:
|
||||
|
||||
> > We don’t care how they got there. We only care that they’re _there_ when we need them.
|
||||
|
||||
For example, I love to go mountain biking and I enjoy tracking my rides with an app called Strava. I get all geared up for my ride, get on my bike and then go, "Oh right, gotta start Strava." So I pull out my phone _with my gloves on_ and go: "Ok Google, open Strava".
|
||||
|
||||
I _could not care less_ about where that app was or where it came from when I said that.
|
||||
|
||||
I don’t care if it was already installed, I don’t care if it never existed on my home screen, or if it was generated out of thin air on the spot.
|
||||
|
||||
> > Context is _everything_ !
|
||||
|
||||
If I’m at a parking meter, I want the app _for that_ . If I’m visiting Portland, I want their public transit app.
|
||||
|
||||
But I certainly _do not_ want it as soon as I’ve left.
|
||||
|
||||
If I’m at a conference, I might want a conference app to see the schedule, post questions to speakers, or whatnot. But wow, talk about something that quickly becomes worthless as soon as that conference is over!
|
||||
|
||||
As it turns out the more "ad hoc" these things are, the better! The more _disposable_ and _re-inflatable_ the better!
|
||||
|
||||
Which also reminds me of something that I feel like we often forget. We always assume people want our shiny apps and we measure things like "engagement" and "time spent in the app" when really, and there certainly are exceptions to this such as apps that are essentially entertainment, but often...
|
||||
|
||||
> > People don’t want to use your app. They want _to be done_ using your app.
|
||||
|
||||
[### Enter PWAs][35]
|
||||
|
||||
I’ve been contracting with Starbucks for the past 18 months. They’ve taken on the ambitious project of essentially re-building a lot of their web stuff in Node.js and React. One of the things I’ve helped them with (and pushed hard for) was to build a PWA (Progressive Web App) that could provide similar functionality as their native apps. Coincidentally it was launched today: [https://preview.starbucks.com][18]!
|
||||
|
||||
<twitterwidget class="twitter-tweet twitter-tweet-rendered" id="twitter-widget-0" data-tweet-id="905931990444244995" style="box-sizing: inherit; max-width: 100%; position: static; visibility: visible; display: block; transform: rotate(0deg); width: 500px; min-width: 220px; margin-top: 10px; margin-bottom: 10px;">[View image on Twitter][10] [][5]
|
||||
|
||||
> [ Follow][1] [ David Brunelle @davidbrunelle][6]
|
||||
>
|
||||
> My team at [@Starbucks][7] has been building a PWA, and it's now in beta! Check it out at [https://preview.starbucks.com ][8] if you're an existing customer!
|
||||
>
|
||||
> [<time class="dt-updated" datetime="2017-09-07T23:13:12+0000" pubdate="" title="Time posted: September 07, 2017 23:13:12 (UTC)">7:13 AM - Sep 8, 2017</time>][9]
|
||||
>
|
||||
> * [ 4949 Replies][2]
|
||||
>
|
||||
> * [ 140140 Retweets][3]
|
||||
>
|
||||
> * [ 454454 likes][4]
|
||||
|
||||
[Twitter Ads info and privacy][11]</twitterwidget>
|
||||
|
||||
This gives is a nice real world example:
|
||||
|
||||
* Starbucks iOS: 146MB
|
||||
|
||||
* Starbucks PWA: ~600KB
|
||||
|
||||
The point is there’s a _tremendous_ size difference.
|
||||
|
||||
It’s 0.4% of the size. To put it differently, I could download the PWA **243 times**in the same amount of time it would take to download the iOS app. Then, of course on iOS it then also still has to install and boot up!
|
||||
|
||||
Personally, I’d have loved it if the app ended up even smaller and there are plans to shrink it further. But even still, they’re _not even on the same planet_ in terms of file-size!
|
||||
|
||||
Market forces are _strongly_ aligned with PWAs here:
|
||||
|
||||
* Few app downloads
|
||||
|
||||
* User acquisition is _hard_
|
||||
|
||||
* User acquisition is _expensive_
|
||||
|
||||
If the goal is to get people to sign up for the rewards program, that type of size difference could very well make the difference of getting someone signed up and using the app experience (via PWA) by the time they reach the front of the line at Starbucks or not.
|
||||
|
||||
User acquisition is hard enough already, the more time and barriers that can be removed from that process, the better.
|
||||
|
||||
[### Quick PWA primer][36]
|
||||
|
||||
As mentioned, PWA stands for "Progressive Web Apps" or, as I like to call them: "Web Apps" 😄
|
||||
|
||||
Personally I’ve been trying to build what a user would define as an "app" with web technology for _years_ . But until PWAs came along, as hard as we tried, you couldn’t quite build a _real app_ with just web tech. Honestly, I kinda hate myself for saying that, but in terms of something that a user would understand as an "app" I’m afraid that statement has probably true until very recently.
|
||||
|
||||
So what’s a PWA? As one of its primary contributors put it:
|
||||
|
||||
> It’s just a website that took all the right vitamins.
|
||||
>
|
||||
> – Alex Russell
|
||||
|
||||
It involves a few specific technologies, namely:
|
||||
|
||||
* Service Worker. Which enable true reliability on the web. What I mean by that is I can build an app that as long as you loaded it while you were online, from then on it will _always_ open, even if you’re not. This puts it on equal footing with other apps.
|
||||
|
||||
* HTTPS. Requires encrypted connections
|
||||
|
||||
* Web App Manifest. A simple JSON file that describes your application. What icons to use is someone adds it to their home screen, what its name is, etc.
|
||||
|
||||
There are plenty of other resources about PWAs on the web. The point for my purposes is:
|
||||
|
||||
> > It is now possible to build PWAs that are _indistinguishable_ from their native counter parts
|
||||
|
||||
They can be up and running in a fraction of the time whether or not they were already "installed" and unlike "apps" can be saved as an app on the device _at the user’s discretion!_
|
||||
|
||||
Essentially they’re really great for creating "ad hoc" experiences that can be "cold started" on a whim nearly as fast as if it were already installed.
|
||||
|
||||
I’ve said it before and I’ll say it again:
|
||||
|
||||
> PWAs are the biggest thing to happen to the mobile web since the iPhone.
|
||||
>
|
||||
> – Um... that was me
|
||||
|
||||
[### Let’s talk Internet of things][37]
|
||||
|
||||
I happen to think that PWAs + IoT = ✨ MAGIC ✨. As several smart folks have pointed out.
|
||||
|
||||
The one-app-per-device approach to smart devices probably isn’t particularly smart.
|
||||
|
||||
It doesn’t scale well and it completely fails in terms of "ad hoc"-ness. Sure, if I have a Nest thermostat and Phillips Hue lightbulbs, it’s reasonable to have two apps installed. But even that sucks as soon as I want someone else to be able to use control them. If _I just let you into my house_ , trust me... I’m perfectly happy to let you flip a light switch, you’re in my house, after all. But for the vast majority of these things there’s no concept of "nearby apps" and, it’s silly for my guest (or a house-sitter) to download an app they don’t actually want, just so I can let them control my lights.
|
||||
|
||||
The whole "nearby apps" thing has so many uses:
|
||||
|
||||
* thermostat
|
||||
|
||||
* lights
|
||||
|
||||
* locks
|
||||
|
||||
* garage doors
|
||||
|
||||
* parking meter
|
||||
|
||||
* setting refrigerator temp
|
||||
|
||||
* conference apps
|
||||
|
||||
Today there are lots of new capabilities being added to the web to enable web apps to interact with physical devices in the real world. Things like WebUSB, WebBluetooth, WebNFC, and efforts like [Physical Web][19]. Even for things like Augmented (and Virtual) reality, the idea of the items we want to interact with having URLs makes so much sense and I can’t imagine a better, more flexible use of those URLs than for them to point to a PWA that lets you interact with that device!
|
||||
|
||||
[### Forward looking statements...][38]
|
||||
|
||||
I’ve been talking about all this in terms of investing. If you’ve ever read any company statement that discusses the future you always see this line explaining that things that are about to be discussed contains "forward looking statements" that may or may not ultimately happen.
|
||||
|
||||
So, here are _my_ forward looking statements.
|
||||
|
||||
[### 1\. PWA-only startups][39]
|
||||
|
||||
Given the cost (and challenge) of user-acquisition and the quality of app you can build with PWAs these days, I feel like this is inevitable. If you’re trying to get something off the ground, it just isn’t very efficient to spin up _three whole teams_ to build for iOS, Android, and the Web.
|
||||
|
||||
[### 2\. PWAs listed in App Stores][40]
|
||||
|
||||
So, there’s a problem with "web only" which is that for the good part of a decade we’ve been training users to look for apps in the app store for their given platform. So if you’re already a recognized brand, especially if you already have a native app that you’re trying to replace, it simply isn’t smart for you _not to exist_ in the app stores.
|
||||
|
||||
So, some of this isn’t all that "forward looking" as it turns out [Microsoft has already committed to listing PWAs in the Windows Store][20], more than once!
|
||||
|
||||
**They haven’t even finished implementing Service Worker in Edge yet!** But they’re already committing hard to PWAs. In addition to post linked above, one of their lead Developer Relations folks, Aaron Gustafson just [wrote an article for A List Apart][21] telling everyone to build PWAs.
|
||||
|
||||
But if you think about it from their perspective, of course they should do that! As I said earlier they’ve struggled to attract developer to build for their mobile phones. In fact, they’ve at times _paid_ companies to write apps for them simply to make sure apps exist so that users will be able to have apps they want when using a Windows Phone. Remember how I said developer time is a scarce resource and without apps, the platform is worthless? So _of course_ they should add first class support for PWAs. If you build a PWA like a lot of folks are doing then TADA!!! 🎉 You just made a Windows/Windows Phone app!
|
||||
|
||||
I’m of the opinion that the writing is on the wall for Google to do the same thing. It’s pure speculation, but it certainly seems like they are taking steps that suggest they may be planning on listing PWAs too. Namely that the Chrome folks recently shipped a feature referred to as "WebAPKs" for Chrome stable on Android (yep, everyone). In the past I’ve [explained in more detail][22] why I think this is a big deal. But a shorted version would be that before this change, sure you could save a PWA to your home screen... _But_ , in reality it was actually a glorified bookmark. That’s what changes with WebAPKs. Instead, when you add a PWA to your home screen it generates and "side loads" an actual `.apk`file on the fly. This allows that PWA to enjoy some privileges that were simply impossible until the operating system recognized it as "an app." For example:
|
||||
|
||||
* You can now mute push notifications for a specific PWA without muting it for all of Chrome.
|
||||
|
||||
* The PWA is listed in the "app tray" that shows all installed apps (previously it was just the home screen).
|
||||
|
||||
* You can see power usage, and permissions granted to the PWA just like any other app.
|
||||
|
||||
* The app developer can now update the icon for the app by publishing an update to the app manifest. Before, there was no way to updated the icon once it had been added.
|
||||
|
||||
* And a slew of other similar benefits...
|
||||
|
||||
If you’ve ever installed an Android app from a source other than the Play Store (or carriers/OEMs store) you know that you have to flip a switch in settings to allow installs from "untrusted sources". So, how then, you might ask, can they generate and install an actual `.apk` file for a PWA without requiring that you change that setting? As it turns out the answer is quite simple: Use a trusted source!
|
||||
|
||||
> > As it turns out WebAPKs are managed through Google Play Services!
|
||||
|
||||
I’m no rocket scientist, but based on their natural business alignment with the web, their promotion of PWAs, the lengths they’ve gone to to grant PWAs equal status on the operating system as native apps, it only seems natural that they’d eventually _list them in the store_ .
|
||||
|
||||
Additionally, if Google did start listing PWAs in the Play Store both them and Microsoft would be doing it _leaving Apple sticking out like a sore thumb and looking like the laggard_ . Essentially, app developers would be able to target a _massive_ number of users on a range of platforms with a single well-built PWA. But, just like developers grew to despise IE for not keeping up with the times and forcing them to jump through extra hoops to support it, the same thing would happen here. Apple does _not_ want to be the next IE and I’ve already seen many prominent developers suggesting they already are.
|
||||
|
||||
Which bring us to another forward-looking statement:
|
||||
|
||||
[### 3\. PWAs on iOS][41]
|
||||
|
||||
Just a few weeks ago the Safari folks announced that Service Worker is now [officially under development][23].
|
||||
|
||||
[### 4\. PWAs everywhere][42]
|
||||
|
||||
I really think we’ll start seeing them everywhere:
|
||||
|
||||
* Inside VR/AR/MR experiences
|
||||
|
||||
* Inside chat bots (again, pulling up an ad-hoc interface is so much more efficient).
|
||||
|
||||
* Inside Xbox?!
|
||||
|
||||
As it turns out, if you look at Microsoft’s status page for Edge about Service Worker you see this:
|
||||
|
||||
|
||||
|
||||
I hinted at this already, but I also think PWAs pair very nicely with virtual assistants being able to pull up an PWA on a whim without requiring it to already be installed would add tremendous power to the virtual assistant. Incidentally, this also becomes easier if there’s a known "registered" name of a PWA listed in an app store.
|
||||
|
||||
Some other fun use cases:
|
||||
|
||||
* Apparently the new digital menu displays in McDonald’s Restaurants (at least in the U.S.) are actually a web app built with Polymer ([source][15]). I don’t know if there’s a Service Worker or not, but it would make sense for there to be.
|
||||
|
||||
* Sports score boards!? I’m a [independent consultant][16], and someone approached me about potentially using a set of TVs and web apps to build a score keeping system at an arena. Point is, there are so many cool examples!
|
||||
|
||||
The web really is the universal platform!
|
||||
|
||||
[### For those who think PWAs are just a Google thing][43]
|
||||
|
||||
First off, I’m pretty sure Microsoft, Opera, Firefox, and Samsung folks would want to punch you for that. It [simply isn’t true][24] and increasingly we’re seeing a lot more compatibility efforts between browser vendors.
|
||||
|
||||
For example: check out the [Web Platform Tests][25] which is essentially Continuous Integration for web features that are run against new releases of major browsers. Some folks will recall that when Apple first claimed they implemented IndexedDb in Safari, the version they shipped was essentially unusable because it had major shortcomings and bugs.
|
||||
|
||||
Now, with the WPTs, you can drill into these features (to quite some detail) and see whether a given browser passes or fails. No more claiming "we shipped!" but not actually shipping.
|
||||
|
||||
[### What about feature "x" on platform "y" that we need?][44]
|
||||
|
||||
It could well be that you have a need that isn’t yet covered by the web platform. In reality, that list is getting shorter and shorter, also... HAVE YOU ASKED?! Despite what it may feel like, browser vendors eagerly want to know what you’re trying to do that you can’t. If there are missing features, be loud, be nice, but from my experience it’s worth making your desires known.
|
||||
|
||||
Also, it doesn’t take much to wrap a web view and add hooks into the native OS that your JavaScript can call to do things that aren’t _quite_ possible yet.
|
||||
|
||||
But that also brings me to another point, in terms of investing, as the world’s greatest hockey player said:
|
||||
|
||||
> Skate to where the puck is going, not where it has been.
|
||||
>
|
||||
> – Wayne Gretzky
|
||||
|
||||
Based on what I’ve outlined thus far, it could be more risky to building an entire application for a whole other platform that you ultimately may not need than to at least exhaust your options seeing what you can do with the Web first.
|
||||
|
||||
So to line ’em up in terms of PWA support:
|
||||
|
||||
* Chrome: yup
|
||||
|
||||
* Firefox: yup
|
||||
|
||||
* Opera: yup
|
||||
|
||||
* Samsung Internet ([the 3rd largest browser surprise!][17]): yup
|
||||
|
||||
* Microsoft: huge public commitment
|
||||
|
||||
* Safari: at least implementing Service Worker
|
||||
|
||||
[### Ask them add your feature!][45]
|
||||
|
||||
Sure, it may not happen, it may take a long time but _at least_ try. Remember, developers have a lot more influence over platforms than we typically realize. Make. your. voice. heard.
|
||||
|
||||
[### Side note about React-Native/Expo][46]
|
||||
|
||||
These projects are run by awesome people, the tech is incredibly impressive. If you’re Facebook and you’re trying to consolidate your development efforts, for the same basic reasons as why it makes sense for them to create their on [VM for running PHP][26]. They have realities to deal with at a scale that most of us will never have to deal with. Personally, I’m not Facebook.
|
||||
|
||||
As a side note, I find it interesting that building native apps and having as many people do that as possible, plays nicely into their advertising competition with Google.
|
||||
|
||||
It just so happens that Google is well positioned to capitalize off of people using the Web. Inversely, I’m fairly certain Facebook wouldn’t mind that ad revenue _not_ going Google. Facebook, seemingly would much rather _be_ your web, that be part of the Web.
|
||||
|
||||
Anyway, all that aside, for me it’s also about investing well.
|
||||
|
||||
By building a native app you’re volunteering for a 30% app-store tax. Plus, like we covered earlier odds are that no one wants to go download your app. Also, though it seems incredibly unlikely, I feel compelled to point out that in terms of "openness" Apple’s App Store is very clearly _anything_ but that. Apple could decide one day that they really don’t like how it’s possible to essentially circumvent their normal update/review process when you use Expo. One day they could just decide to reject all React Native apps. I really don’t think they would because of the uproar it would cause. I’m simply pointing out that it’s _their_ platform and they would have _every_ right to do so.
|
||||
|
||||
[### So is it all about investing for your own gain?][47]
|
||||
|
||||
So far, I’ve presented all this from kind of a cold, heartless investor perspective: getting the most for your time.
|
||||
|
||||
But, that’s not the whole story is it?
|
||||
|
||||
Life isn’t all about me. Life isn’t all about us.
|
||||
|
||||
I want to invest in platforms that increase opportunities **for others**. Personally, I really hope the next friggin’ Mark Zuckerburg isn’t an ivy-league dude. Wouldn’t it be amazing if instead the next huge success was, I don’t know, perhaps a young woman in Nairobi or something? The thing is, if owning an iPhone is a prerequisite for building apps, it _dramatically_ decreases the odds of something like that happening. I feel like the Web really is the closest thing we have to a level playing field.
|
||||
|
||||
**I want to invest in and improve _that_ platform!**
|
||||
|
||||
This quote really struck me and has stayed with me when thinking about these things:
|
||||
|
||||
> If you’re the kind of person who tends to succeed in what you start,
|
||||
>
|
||||
> changing what you start could be _the most extraordinary thing_ you could do.
|
||||
>
|
||||
> – Anand Giridharadas
|
||||
|
||||
Thanks for your valuable attention ❤️. I’ve presented the facts as I see them and I’ve done my best not to "should on you."
|
||||
|
||||
Ultimately though, no matter how prepared we are or how much research we’ve done; investing is always a bit of a gamble.
|
||||
|
||||
So I guess the only thing left to say is:
|
||||
|
||||
> > I’m all in.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://joreteg.com/blog/betting-on-the-web
|
||||
|
||||
作者：[Joreteg][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://joreteg.com/
|
||||
[1]:https://twitter.com/davidbrunelle
|
||||
[2]:https://twitter.com/intent/tweet?in_reply_to=905931990444244995
|
||||
[3]:https://twitter.com/intent/retweet?tweet_id=905931990444244995
|
||||
[4]:https://twitter.com/intent/like?tweet_id=905931990444244995
|
||||
[5]:https://twitter.com/davidbrunelle/status/905931990444244995/photo/1
|
||||
[6]:https://twitter.com/davidbrunelle
|
||||
[7]:https://twitter.com/Starbucks
|
||||
[8]:https://t.co/tEUXM8BLgP
|
||||
[9]:https://twitter.com/davidbrunelle/status/905931990444244995
|
||||
[10]:https://twitter.com/davidbrunelle/status/905931990444244995/photo/1
|
||||
[11]:https://support.twitter.com/articles/20175256
|
||||
[12]:https://2017.coldfront.co/
|
||||
[13]:https://qz.com/253618/most-smartphone-users-download-zero-apps-per-month/
|
||||
[14]:http://fortune.com/2016/05/19/app-economy/
|
||||
[15]:https://twitter.com/AJStacy06/status/857628546507968512
|
||||
[16]:http://consulting.joreteg.com/
|
||||
[17]:https://medium.com/samsung-internet-dev/think-you-know-the-top-web-browsers-458a0a070175
|
||||
[18]:https://preview.starbucks.com/
|
||||
[19]:https://google.github.io/physical-web/
|
||||
[20]:https://blogs.windows.com/msedgedev/2016/07/08/the-progress-of-web-apps/
|
||||
[21]:https://alistapart.com/article/yes-that-web-project-should-be-a-pwa
|
||||
[22]:https://joreteg.com/blog/installing-web-apps-for-real
|
||||
[23]:https://webkit.org/status/#specification-service-workers
|
||||
[24]:https://jakearchibald.github.io/isserviceworkerready/
|
||||
[25]:http://wpt.fyi/
|
||||
[26]:http://hhvm.com/
|
||||
[27]:https://joreteg.com/blog/betting-on-the-web
|
||||
[28]:https://joreteg.com/blog/betting-on-the-web#quotso-what-whats-your-pointquot
|
||||
[29]:https://joreteg.com/blog/betting-on-the-web#with-all-investing-comes
|
||||
[30]:https://joreteg.com/blog/betting-on-the-web#are-we-building-for-the-right-platform
|
||||
[31]:https://joreteg.com/blog/betting-on-the-web#are-we-building-the-right-software
|
||||
[32]:https://joreteg.com/blog/betting-on-the-web#should-we-be-building-apps-at-all
|
||||
[33]:https://joreteg.com/blog/betting-on-the-web#if-apps-are-so-great-why-are-we-so-quotapped-outquot
|
||||
[34]:https://joreteg.com/blog/betting-on-the-web#what-sucks-about-apps
|
||||
[35]:https://joreteg.com/blog/betting-on-the-web#enter-pwas
|
||||
[36]:https://joreteg.com/blog/betting-on-the-web#quick-pwa-primer
|
||||
[37]:https://joreteg.com/blog/betting-on-the-web#lets-talk-internet-of-things
|
||||
[38]:https://joreteg.com/blog/betting-on-the-web#forward-looking-statements
|
||||
[39]:https://joreteg.com/blog/betting-on-the-web#1-pwa-only-startups
|
||||
[40]:https://joreteg.com/blog/betting-on-the-web#2-pwas-listed-in-app-stores
|
||||
[41]:https://joreteg.com/blog/betting-on-the-web#3-pwas-on-ios
|
||||
[42]:https://joreteg.com/blog/betting-on-the-web#4-pwas-everywhere
|
||||
[43]:https://joreteg.com/blog/betting-on-the-web#for-those-who-think-pwas-are-just-a-google-thing
|
||||
[44]:https://joreteg.com/blog/betting-on-the-web#what-about-feature-quotxquot-on-platform-quotyquot-that-we-need
|
||||
[45]:https://joreteg.com/blog/betting-on-the-web#ask-them-add-your-feature
|
||||
[46]:https://joreteg.com/blog/betting-on-the-web#side-note-about-react-nativeexpo
|
||||
[47]:https://joreteg.com/blog/betting-on-the-web#so-is-it-all-about-investing-for-your-own-gain
|
||||
@@ -0,0 +1,513 @@
|
||||
What every software engineer should know about search
|
||||
============================================================
|
||||
|
||||
|
||||
|
||||
|
||||
### Want to build or improve a search experience? Start here.
|
||||
|
||||
Ask a software engineer: “[How would you add search functionality to your product?][78]” or “[How do I build a search engine?][79]” You’ll probably immediately hear back something like: “Oh, we’d just launch an ElasticSearch cluster. Search is easy these days.”
|
||||
|
||||
But is it? Numerous current products [still][80] [have][81] [suboptimal][82] [search][83] [experiences][84]. Any true search expert will tell you that few engineers have a very deep understanding of how search engines work, knowledge that’s often needed to improve search quality.
|
||||
|
||||
Even though many open source software packages exist, and the research is vast, the knowledge around building solid search experiences is limited to a select few. Ironically, [searching online][85] for search-related expertise doesn’t yield any recent, thoughtful overviews.
|
||||
|
||||
#### Emoji Legend
|
||||
|
||||
```
|
||||
❗ “Serious” gotcha: consequences of ignorance can be deadly
|
||||
🔷 Especially notable idea or piece of technology
|
||||
☁️ ️Cloud/SaaS
|
||||
🍺 Open source / free software
|
||||
🦏 JavaScript
|
||||
🐍 Python
|
||||
☕ Java
|
||||
🇨 C/C++
|
||||
```
|
||||
|
||||
### Why read this?
|
||||
|
||||
Think of this post as a collection of insights and resources that could help you to build search experiences. It can’t be a complete reference, of course, but hopefully we can improve it based on feedback (please comment or reach out!).
|
||||
|
||||
I’ll point at some of the most popular approaches, algorithms, techniques, and tools, based on my work on general purpose and niche search experiences of varying sizes at Google, Airbnb and several startups.
|
||||
|
||||
❗️Not appreciating or understanding the scope and complexity of search problems can lead to bad user experiences, wasted engineering effort, and product failure.
|
||||
|
||||
If you’re impatient or already know a lot of this, you might find it useful to jump ahead to the tools and services sections.
|
||||
|
||||
### Some philosophy
|
||||
|
||||
This is a long read. But most of what we cover has four underlying principles:
|
||||
|
||||
#### 🔷 Search is an inherently messy problem:
|
||||
|
||||
* Queries are highly variable. The search problems are highly variablebased on product needs.
|
||||
|
||||
* Think about how different Facebook search (searching a graph of people).
|
||||
|
||||
* YouTube search (searching individual videos).
|
||||
|
||||
* Or how different both of those are are from Kayak ([air travel planning is a really hairy problem][2]).
|
||||
|
||||
* Google Maps (making sense of geo-spacial data).
|
||||
|
||||
* Pinterest (pictures of a brunch you might cook one day).
|
||||
|
||||
#### Quality, metrics, and processes matter a lot:
|
||||
|
||||
* There is no magic bullet (like PageRank) nor a magic ranking formula that makes for a good approach. Processes are always evolving collection of techniques and processes that solve aspects of the problem and improve overall experience, usually gradually and continuously.
|
||||
|
||||
* ❗️In other words, search is not just just about building software that does ranking or retrieval (which we will discuss below) for a specific domain. Search systems are usually an evolving pipeline of components that are tuned and evolve over time and that build up to a cohesive experience.
|
||||
|
||||
* In particular, the key to success in search is building processes for evaluation and tuning into the product and development cycles. A search system architect should think about processes and metrics, not just technologies.
|
||||
|
||||
#### Use existing technologies first:
|
||||
|
||||
* As in most engineering problems, don’t reinvent the wheel yourself. When possible, use existing services or open source tools. If an existing SaaS (such as [Algolia][3] or managed Elasticsearch) fits your constraints and you can afford to pay for it, use it. This solution will likely will be the best choice for your product at first, even if down the road you need to customize, enhance, or replace it.
|
||||
|
||||
#### ❗️Even if you buy, know the details:
|
||||
|
||||
* Even if you are using an existing open source or commercial solution, you should have some sense of the complexity of the search problem and where there are likely to be pitfalls.
|
||||
|
||||
### Theory: the search problem
|
||||
|
||||
Search is different for every product, and choices depend on many technical details of the requirements. It helps to identify the key parameters of your search problem:
|
||||
|
||||
1. Size: How big is the corpus (a complete set of documents that need to be searched)? Is it thousands or billions of documents?
|
||||
|
||||
2. Media: Are you searching through text, images, graphical relationships, or geospatial data?
|
||||
|
||||
3. 🔷 Corpus control and quality: Are the sources for the documents under your control, or coming from a (potentially adversarial) third party? Are all the documents ready to be indexed or need to be cleaned up and selected?
|
||||
|
||||
4. Indexing speed: Do you need real-time indexing, or is building indices in batch is fine?
|
||||
|
||||
5. Query language: Are the queries structured, or you need to support unstructured ones?
|
||||
|
||||
6. Query structure: Are your queries textual, images, sounds? Street addresses, record ids, people’s faces?
|
||||
|
||||
7. Context-dependence: Do the results depend on who the user is, what is their history with the product, their geographical location, time of the day etc?
|
||||
|
||||
8. Suggest support: Do you need to support incomplete queries?
|
||||
|
||||
9. Latency: What are the serving latency requirements? 100 milliseconds or 100 seconds?
|
||||
|
||||
10. Access control: Is it entirely public or should users only see a restricted subset of the documents?
|
||||
|
||||
11. Compliance: Are there compliance or organizational limitations?
|
||||
|
||||
12. Internationalization: Do you need to support documents with multilingual character sets or Unicode? (Hint: Always use UTF-8 unless you really know what you’re doing.) Do you need to support a multilingual corpus? Multilingual queries?
|
||||
|
||||
Thinking through these points up front can help you make significant choices designing and building individual search system components.
|
||||
|
||||
** 此处有Canvas,请手动处理 **
|
||||
|
||||
|
||||
A production indexing pipeline.
|
||||
|
||||
### Theory: the search pipeline
|
||||
|
||||
Now let’s go through a list of search sub-problems. These are usually solved by separate subsystems that form a pipeline. What that means is that a given subsystem consumes the output of previous subsystems, and produces input for the following subsystems.
|
||||
|
||||
This leads to an important property of the ecosystem: once you change how an upstream subsystem works, you need to evaluate the effect of the change and possibly change the behavior downstream.
|
||||
|
||||
Here are the most important problems you need to solve:
|
||||
|
||||
#### Index selection:
|
||||
|
||||
given a set of documents (e.g. the entirety of the Internet, all the Twitter posts, all the pictures on Instagram), select a potentially smaller subset of documents that may be worthy for consideration as search results and only include those in the index, discarding the rest. This is done to keep your indexes compact, and is almost orthogonal to selecting the documents to show to the user. Examples of particular classes of documents that don’t make the cut may include:
|
||||
|
||||
#### Spam:
|
||||
|
||||
oh, all the different shapes and sizes of search spam! A giant topic in itself, worthy of a separate guide. [A good web spam taxonomy overview][86].
|
||||
|
||||
#### Undesirable documents:
|
||||
|
||||
domain constraints might require filtering: [porn][87], illegal content, etc. The techniques are similar to spam filtering, probably with extra heuristics.
|
||||
|
||||
#### Duplicates:
|
||||
|
||||
Or near-duplicates and redundant documents. Can be done with [Locality-sensitive hashing][88], [similarity measures][89], clustering techniques or even [clickthrough data][90]. A [good overview][91] of techniques.
|
||||
|
||||
#### Low-utility documents:
|
||||
|
||||
The definition of utility depends highly on the problem domain, so it’s hard to recommend the approaches here. Some ideas are: it might be possible to build a utility function for your documents; heuristics might work, or example an image that contains only black pixels is not a useful document; utility might be learned from user behavior.
|
||||
|
||||
#### Index construction:
|
||||
|
||||
For most search systems, document retrieval is performed using an [inverted index][92] — often just called the index.
|
||||
|
||||
* The index is a mapping of search terms to documents. A search term could be a word, an image feature or any other document derivative useful for query-to-document matching. The list of the documents for a given term is called a [posting list][1]. It can be sorted by some metric, like document quality.
|
||||
|
||||
* Figure out whether you need to index the data in real time.❗️Many companies with large corpora of documents use a batch-oriented indexing approach, but then find this is unsuited to a product where users expect results to be current.
|
||||
|
||||
* With text documents, term extraction usually involves using NLP techniques, such as stop lists, [stemming][4] and [entity extraction][5]; for images or videos computer vision methods are used etc.
|
||||
|
||||
* In addition, documents are mined for statistical and meta information, such as references to other documents (used in the famous [PageRank][6]ranking signal), [topics][7], counts of term occurrences, document size, entities A mentioned etc. That information can be later used in ranking signal construction or document clustering. Some larger systems might contain several indexes, e.g. for documents of different types.
|
||||
|
||||
* Index formats. The actual structure and layout of the index is a complex topic, since it can be optimized in many ways. For instance there are [posting lists compression methods][8], one could target [mmap()able data representation][9] or use[ LSM-tree][10] for continuously updated index.
|
||||
|
||||
#### Query analysis and document retrieval:
|
||||
|
||||
Most popular search systems allow non-structured queries. That means the system has to extract structure out of the query itself. In the case of an inverted index, you need to extract search terms using [NLP][93] techniques.
|
||||
|
||||
The extracted terms can be used to retrieve relevant documents. Unfortunately, most queries are not very well formulated, so it pays to do additional query expansion and rewriting, like:
|
||||
|
||||
* [Term re-weighting][11].
|
||||
|
||||
* [Spell checking][12]. Historical query logs are very useful as a dictionary.
|
||||
|
||||
* [Synonym matching][13]. [Another survey][14].
|
||||
|
||||
* [Named entity recognition][15]. A good approach is to use [HMM-based language modeling][16].
|
||||
|
||||
* Query classification. Detect queries of particular type. For example, Google Search detects queries that contain a geographical entity, a porny query, or a query about something in the news. The retrieval algorithm can then make a decision about which corpora or indexes to look at.
|
||||
|
||||
* Expansion through [personalization][17] or [local context][18]. Useful for queries like “gas stations around me”.
|
||||
|
||||
#### Ranking:
|
||||
|
||||
Given a list of documents (retrieved in the previous step), their signals, and a processed query, create an optimal ordering (ranking) for those documents.
|
||||
|
||||
Originally, most ranking models in use were hand-tuned weighted combinations of all the document signals. Signal sets might include PageRank, clickthrough data, topicality information and [others][94].
|
||||
|
||||
To further complicate things, many of those signals, such as PageRank, or ones generated by [statistical language models][95] contain parameters that greatly affect the performance of a signal. Those have to be hand-tuned too.
|
||||
|
||||
Lately, 🔷 [learning to rank][96], signal-based discriminative supervised approaches are becoming more and more popular. Some popular examples of LtR are [McRank][97] and [LambdaRank][98] from Microsoft, and [MatrixNet][99] from Yandex.
|
||||
|
||||
A new, [vector space based approach][100] for semantic retrieval and ranking is gaining popularity lately. The idea is to learn individual low-dimensional vector document representations, then build a model which maps queries into the same vector space.
|
||||
|
||||
Then, retrieval is just finding several documents that are closest by some metric (e.g. Eucledian distance) to the query vector. Ranking is the distance itself. If the mapping of both the documents and queries is built well, the documents are chosen not by a fact of presence of some simple pattern (like a word), but how close the documents are to the query by _meaning_ .
|
||||
|
||||
### Indexing pipeline operation
|
||||
|
||||
Usually, each of the above pieces of the pipeline must be operated on a regular basis to keep the search index and search experience current.
|
||||
|
||||
❗️Operating a search pipeline can be complex and involve a lot of moving pieces. Not only is the data moving through the pipeline, but the code for each module and the formats and assumptions embedded in the data will change over time.
|
||||
|
||||
A pipeline can be run in “batch” or based on a regular or occasional basis (if indexing speed does not need to be real time) or in a streamed way (if real-time indexing is needed) or based on certain triggers.
|
||||
|
||||
Some complex search engines (like Google) have several layers of pipelines operating on different time scales — for example, a page that changes often (like [cnn.com][101]) is indexed with a higher frequency than a static page that hasn’t changed in years.
|
||||
|
||||
### Serving systems
|
||||
|
||||
Ultimately, the goal of a search system is to accept queries, and use the index to return appropriately ranked results. While this subject can be incredibly complex and technical, we mention a few of the key aspects to this part of the system.
|
||||
|
||||
* Performance: users notice when the system they interact with is laggy. ❗️Google has done [extensive research][19], and they have noticed that number of searches falls 0.6%, when serving is slowed by 300ms. They recommend to serve results under 200 ms for most of your queries. A good article [on the topic][20]. This is the hard part: the system needs to collect documents from, possibly, many computers, than merge them into possible a very long list and then sort that list in the ranking order. To complicate things further, ranking might be query-dependent, so, while sorting, the system is not just comparing 2 numbers, but performing computation.
|
||||
|
||||
* 🔷 Caching results: is often necessary to achieve decent performance. ❗️ But caches are just one large gotcha. The might show stale results when indices are updated or some results are blacklisted. Purging caches is a can of warm of itself: a search system might not have the capacity to serve the entire query stream with an empty (cold) cache, so the [cache needs to be pre-warmed][21] before the queries start arriving. Overall, caches complicate a system’s performance profile. Choosing a cache size and a replacement algorithm is also a [challenge][22].
|
||||
|
||||
* Availability: is often defined by an uptime/(uptime + downtime) metric. When index is distributed, in order to serve any search results, the system often needs to query all the shards for their share of results. ❗️That means, that if one shard is unavailable, the entire search system is compromised. The more machines are involved in serving the index — the higher the probability of one of them becoming defunct and bringing the whole system down.
|
||||
|
||||
* Managing multiple indices: Indices for large systems may separated into shards (pieces) or divided by media type or indexing cadence (fresh versus long-term indices). Results can then be merged.
|
||||
|
||||
* Merging results of different kinds: e.g. Google showing results from Maps, News etc.
|
||||
|
||||
** 此处有Canvas,请手动处理 **
|
||||
|
||||
|
||||
A human rater. Yeah, you should still have those.
|
||||
|
||||
### Quality, evaluation, and improvement
|
||||
|
||||
So you’ve launched your indexing pipeline and search servers, and it’s all running nicely. Unfortunately the road to a solid search experience only begins with running infrastructure.
|
||||
|
||||
Next, you’ll need to build a set of processes around continuous search quality evaluation and improvement. In fact, this is actually most of the work and the hardest problem you’ll have to solve.
|
||||
|
||||
🔷 What is quality? First, you’ll need to determine (and get your boss or the product lead to agree), what quality means in your case:
|
||||
|
||||
* Self-reported user satisfaction (includes UX)
|
||||
|
||||
* Perceived relevance of the returned results (not including UX)
|
||||
|
||||
* Satisfaction relative to competitors
|
||||
|
||||
* Satisfaction relative performance of the previous version of the search engine (e.g. last week)
|
||||
|
||||
* [User engagement][23]
|
||||
|
||||
Metrics: Some of these concepts can be quite hard to quantify. On the other hand, it’s incredibly useful to be able to express how well a search engine is performing in a single number, a quality metric.
|
||||
|
||||
Continuously computing such a metric for your (and your competitors’) system you can both track your progress and explain how well you are doing to your boss. Here are some classical ways to quantify quality, that can help you construct your magic quality metric formula:
|
||||
|
||||
* [Precision][24] and [recall][25] measure how well the retrieved set of documents corresponds to the set you expected to see.
|
||||
|
||||
* [F score][26] (specifically F1 score) is a single number, that represents both precision and recall well.
|
||||
|
||||
* [Mean Average Precision][27] (MAP) allows to quantify the relevance of the top returned results.
|
||||
|
||||
* 🔷 [Normalized Discounted Cumulative Gain][28] (nDCG) is like MAP, but weights the relevance of the result by its position.
|
||||
|
||||
* [Long and short clicks][29] — Allow to quantify how useful the results are to the real users.
|
||||
|
||||
* [A good detailed overview][30].
|
||||
|
||||
🔷 Human evaluations: Quality metrics might seem like statistical calculations, but they can’t all be done by automated calculations. Ultimately, metrics need to represent subjective human evaluation, and this is where a “human in the loop” comes into play.
|
||||
|
||||
❗️Skipping human evaluation is probably the most spread reason of sub-par search experiences.
|
||||
|
||||
Usually, at early stages the developers themselves evaluate the results manually. At later point [human raters][102] (or assessors) may get involved. Raters typically use custom tools to look at returned search results and provide feedback on the quality of the results.
|
||||
|
||||
Subsequently, you can use the feedback signals to guide development, help make launch decisions or even feed them back into the index selection, retrieval or ranking systems.
|
||||
|
||||
Here is the list of some other types of human-driven evaluation, that can be done on a search system:
|
||||
|
||||
* Basic user evaluation: The user ranks their satisfaction with the whole experience
|
||||
|
||||
* Comparative evaluation: Compare with other search results (compare with search results from earlier versions of the system or competitors)
|
||||
|
||||
* Retrieval evaluation: The query analysis and retrieval quality is often evaluated using manually constructed query-document sets. A user is shown a query and the list of the retrieved documents. She can then mark all the documents that are relevant to the query, and the ones that are not. The resulting pairs of (query, [relevant docs]) are called a “golden set”. Golden sets are remarkably useful. For one, an engineer can set up automatic retrieval regression tests using those sets. The selection signal from golden sets can also be fed back as ground truth to term re-weighting and other query re-writing models.
|
||||
|
||||
* Ranking evaluation: Raters are presented with a query and two documents side-by-side. The rater must choose the document that fits the query better. This creates a partial ordering on the documents for a given query. That ordering can be later be compared to the output of the ranking system. The usual ranking quality measures used are MAP and nDCG.
|
||||
|
||||
#### Evaluation datasets:
|
||||
|
||||
One should start thinking about the datasets used for evaluation (like “golden sets” mentioned above) early in the search experience design process. How you collect and update them? How you push them to the production eval pipeline? Is there a built-in bias?
|
||||
|
||||
Live experiments:
|
||||
|
||||
After your search engine catches on and gains enough users, you might want to start conducting [live search experiments][103] on a portion of your traffic. The basic idea is to turn some optimization on for a group of people, and then compare the outcome with that of a “control” group — a similar sample of your users that did not have the experiment feature on for them. How you would measure the outcome is, once again, very product specific: it could be clicks on results, clicks on ads etc.
|
||||
|
||||
Evaluation cycle time: How fast you improve your search quality is directly related to how fast you can complete the above cycle of measurement and improvement. It is essential from the beginning to ask yourself, “how fast can we measure and improve our performance?”
|
||||
|
||||
Will it take days, hours, minutes or seconds to make changes and see if they improve quality? ❗️Running evaluation should also be as easy as possible for the engineers and should not take too much hands-on time.
|
||||
|
||||
### 🔷 So… How do I PRACTICALLY build it?
|
||||
|
||||
This blogpost is not meant as a tutorial, but here is a brief outline of how I’d approach building a search experience right now:
|
||||
|
||||
1. As was said above, if you can afford it — just buy the existing SaaS (some good ones are listed below). An existing service fits if:
|
||||
|
||||
* Your experience is a “connected” one (your service or app has internet connection).
|
||||
|
||||
* Does it support all the functionality you need out of box? This post gives a pretty good idea of what functions would you want. To name a few, I’d at least consider: support for the media you are searching; real-time indexing support; query flexibility, including context-dependent queries.
|
||||
|
||||
* Given the size of the corpus and the expected [QpS][31], can you afford to pay for it for the next 12 months?
|
||||
|
||||
* Can the service support your expected traffic within the required latency limits? In case when you are querying the service from an app, make sure that the given service is accessible quickly enough from where your users are.
|
||||
|
||||
2\. If a hosted solution does not fit your needs or resources, you probably want to use one of the open source libraries or tools. In case of connected apps or websites, I’d choose ElasticSearch right now. For embedded experiences, there are multiple tools below.
|
||||
|
||||
3\. You most likely want to do index selection and clean up your documents (say extract relevant text from HTML pages) before uploading them to the search index. This will decrease the index size and make getting to good results easier. If your corpus fits on a single machine, just write a script (or several) to do that. If not, I’d use [Spark][104].
|
||||
|
||||
** 此处有Canvas,请手动处理 **
|
||||
|
||||
|
||||
You can never have too many tools.
|
||||
|
||||
### ☁️ SaaS
|
||||
|
||||
☁️ 🔷[Algolia][105] — a proprietary SaaS that indexes a client’s website and provides an API to search the website’s pages. They also have an API to submit your own documents, support context dependent searches and serve results really fast. If I were building a web search experience right now and could afford it, I’d probably use Algolia first — and buy myself time to build a comparable search experience.
|
||||
|
||||
* Various ElasticSearch providers: AWS (☁️ [ElasticSearch Cloud)][32], ☁️[elastic.co][33] and from ☁️ [Qbox][34].
|
||||
|
||||
* ☁️[ Azure Search][35] — a SaaS solution from Microsoft. Accessible through a REST API, it can scale to billions of documents. Has a Lucene query interface to simplify migrations from Lucene-based solutions.
|
||||
|
||||
* ☁️[ Swiftype][36] — an enterprise SaaS that indexes your company’s internal services, like Salesforce, G Suite, Dropbox and the intranet site.
|
||||
|
||||
### Tools and libraries
|
||||
|
||||
🍺☕🔷[ Lucene][106] is the most popular IR library. Implements query analysis, index retrieval and ranking. Either of the components can be replaced by an alternative implementation. There is also a C port — 🍺[Lucy][107].
|
||||
|
||||
* 🍺☕🔷[ Solr][37] is a complete search server, based on Lucene. It’s a part of the [Hadoop][38] ecosystem of tools.
|
||||
|
||||
* 🍺☕🔷[ Hadoop][39] is the most widely used open source MapReduce system, originally designed as a indexing pipeline framework for Solr. It has been gradually loosing ground to 🍺[Spark][40] as the batch data processing framework used for indexing. ☁️[EMR][41] is a proprietary implementation of MapReduce on AWS.
|
||||
|
||||
* 🍺☕🔷 [ElasticSearch][42] is also based on Lucene ([feature comparison with Solr][43]). It has been getting more attention lately, so much that a lot of people think of ES when they hear “search”, and for good reasons: it’s well supported, has [extensive API][44], [integrates with Hadoop][45] and [scales well][46]. There are open source and [Enterprise][47] versions. ES is also available as a SaaS on Can scale to billions of documents, but scaling to that point can be very challenging, so typical scenario would involve orders of magnitude smaller corpus.
|
||||
|
||||
* 🍺🇨 [Xapian][48] — a C++-based IR library. Relatively compact, so good for embedding into desktop or mobile applications.
|
||||
|
||||
* 🍺🇨 [Sphinx][49] — an full-text search server. Has a SQL-like query language. Can also act as a [storage engine for MySQL][50] or used as a library.
|
||||
|
||||
* 🍺☕ [Nutch][51] — a web crawler. Can be used in conjunction with Solr. It’s also the tool behind [🍺Common Crawl][52].
|
||||
|
||||
* 🍺🦏 [Lunr][53] — a compact embedded search library for web apps on the client-side.
|
||||
|
||||
* 🍺🦏 [searchkit][54] — a library of web UI components to use with ElasticSearch.
|
||||
|
||||
* 🍺🦏 [Norch][55] — a [LevelDB][56]-based search engine library for Node.js.
|
||||
|
||||
* 🍺🐍 [Whoosh][57] — a fast, full-featured search library implemented in pure Python.
|
||||
|
||||
* OpenStreetMaps has it’s own 🍺[deck of search software][58].
|
||||
|
||||
### Datasets
|
||||
|
||||
A few fun or useful data sets to try building a search engine or evaluating search engine quality:
|
||||
|
||||
* 🍺🔷 [Commoncrawl][59] — a regularly-updated open web crawl data. There is a [mirror on AWS][60], accessible for free within the service.
|
||||
|
||||
* 🍺🔷 [Openstreetmap data dump][61] is a very rich source of data for someone building a geospacial search engine.
|
||||
|
||||
* 🍺 [Google Books N-grams][62] can be very useful for building language models.
|
||||
|
||||
* 🍺 [Wikipedia dumps][63] are a classic source to build, among other things, an entity graph out of. There is a [wide range of helper tools][64] available.
|
||||
|
||||
* [IMDb dumps][65] are a fun dataset to build a small toy search engine for.
|
||||
|
||||
### References
|
||||
|
||||
* [Modern Information Retrieval][66] by R. Baeza-Yates and B. Ribeiro-Neto is a good, deep academic treatment of the subject. This is a good overview for someone completely new to the topic.
|
||||
|
||||
* [Information Retrieval][67] by S. Büttcher, C. Clarke and G. Cormack is another academic textbook with a wide coverage and is more up-to-date. Covers learn-to-rank and does a pretty good job at discussing theory of search systems evaluation. Also is a good overview.
|
||||
|
||||
* [Learning to Rank][68] by T-Y Liu is a best theoretical treatment of LtR. Pretty thin on practical aspects though. Someone considering building an LtR system should probably check this out.
|
||||
|
||||
* [Managing Gigabytes][69] — published in 1999, is still a definitive reference for anyone embarking on building an efficient index of a significant size.
|
||||
|
||||
* [Text Retrieval and Search Engines][70] — a MOOC from Coursera. A decent overview of basics.
|
||||
|
||||
* [Indexing the World Wide Web: The Journey So Far][71] ([PDF][72]), an overview of web search from 2012, by Ankit Jain and Abhishek Das of Google.
|
||||
|
||||
* [Why Writing Your Own Search Engine is Hard][73] a classic article from 2004 from Anna Patterson.
|
||||
|
||||
* [https://github.com/harpribot/awesome-information-retrieval][74] — a curated list of search-related resources.
|
||||
|
||||
* A [great blog][75] on everything search by [Daniel Tunkelang][76].
|
||||
|
||||
* Some good slides on [search engine evaluation][77].
|
||||
|
||||
This concludes my humble attempt to make a somewhat-useful “map” for an aspiring search engine engineer. Did I miss something important? I’m pretty sure I did — you know, [the margin is too narrow][108] to contain this enormous topic. Let me know if you think that something should be here and is not — you can reach [me][109] at[ forwidur@gmail.com][110] or at [@forwidur][111].
|
||||
|
||||
> P.S. — This post is part of a open, collaborative effort to build an online reference, the Open Guide to Practical AI, which we’ll release in draft form soon. See [this popular guide][112] for an example of what’s coming. If you’d like to get updates on or help with with this effort, sign up [here][113].
|
||||
|
||||
> Special thanks to [Joshua Levy][114], [Leo Polovets][115] and [Abhishek Das][116] for reading drafts of this and their invaluable feedback!
|
||||
|
||||
> Header image courtesy of [Mickaël Forrett][117]. The beautiful toolbox is called [The Studley Tool Chest][118].
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
作者简介：
|
||||
|
||||
Max Grigorev
|
||||
distributed systems, data, AI
|
||||
|
||||
-------------
|
||||
|
||||
|
||||
via: https://medium.com/startup-grind/what-every-software-engineer-should-know-about-search-27d1df99f80d
|
||||
|
||||
作者：[Max Grigorev][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://medium.com/@forwidur?source=post_header_lockup
|
||||
[1]:https://en.wikipedia.org/wiki/Inverted_index
|
||||
[2]:http://www.demarcken.org/carl/papers/ITA-software-travel-complexity/ITA-software-travel-complexity.pdf
|
||||
[3]:https://www.algolia.com/
|
||||
[4]:https://en.wikipedia.org/wiki/Stemming
|
||||
[5]:https://en.wikipedia.org/wiki/Named-entity_recognition
|
||||
[6]:http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf
|
||||
[7]:https://gofishdigital.com/semantic-topic-modeling/
|
||||
[8]:https://nlp.stanford.edu/IR-book/html/htmledition/postings-file-compression-1.html
|
||||
[9]:https://deplinenoise.wordpress.com/2013/03/31/fast-mmapable-data-structures/
|
||||
[10]:https://en.wikipedia.org/wiki/Log-structured_merge-tree
|
||||
[11]:http://orion.lcg.ufrj.br/Dr.Dobbs/books/book5/chap11.htm
|
||||
[12]:http://norvig.com/spell-correct.html
|
||||
[13]:http://nlp.stanford.edu/IR-book/html/htmledition/query-expansion-1.html
|
||||
[14]:https://www.iro.umontreal.ca/~nie/IFT6255/carpineto-Survey-QE.pdf
|
||||
[15]:https://en.wikipedia.org/wiki/Named-entity_recognition
|
||||
[16]:http://www.aclweb.org/anthology/P02-1060
|
||||
[17]:https://en.wikipedia.org/wiki/Personalized_search
|
||||
[18]:http://searchengineland.com/future-search-engines-context-217550
|
||||
[19]:http://services.google.com/fh/files/blogs/google_delayexp.pdf
|
||||
[20]:http://highscalability.com/latency-everywhere-and-it-costs-you-sales-how-crush-it
|
||||
[21]:https://stackoverflow.com/questions/22756092/what-does-it-mean-by-cold-cache-and-warm-cache-concept
|
||||
[22]:https://en.wikipedia.org/wiki/Cache_performance_measurement_and_metric
|
||||
[23]:http://blog.popcornmetrics.com/5-user-engagement-metrics-for-growth/
|
||||
[24]:https://en.wikipedia.org/wiki/Information_retrieval#Precision
|
||||
[25]:https://en.wikipedia.org/wiki/Information_retrieval#Recall
|
||||
[26]:https://en.wikipedia.org/wiki/F1_score
|
||||
[27]:http://fastml.com/what-you-wanted-to-know-about-mean-average-precision/
|
||||
[28]:https://en.wikipedia.org/wiki/Discounted_cumulative_gain
|
||||
[29]:http://www.blindfiveyearold.com/short-clicks-versus-long-clicks
|
||||
[30]:https://arxiv.org/pdf/1302.2318.pdf
|
||||
[31]:https://en.wikipedia.org/wiki/Queries_per_second
|
||||
[32]:https://aws.amazon.com/elasticsearch-service/
|
||||
[33]:https://www.elastic.co/
|
||||
[34]:https://qbox.io/
|
||||
[35]:https://azure.microsoft.com/en-us/services/search/
|
||||
[36]:https://swiftype.com/
|
||||
[37]:http://lucene.apache.org/solr/
|
||||
[38]:http://hadoop.apache.org/
|
||||
[39]:http://hadoop.apache.org/
|
||||
[40]:http://spark.apache.org/
|
||||
[41]:https://aws.amazon.com/emr/
|
||||
[42]:https://www.elastic.co/products/elasticsearch
|
||||
[43]:http://solr-vs-elasticsearch.com/
|
||||
[44]:https://www.elastic.co/guide/en/elasticsearch/reference/current/docs.html
|
||||
[45]:https://github.com/elastic/elasticsearch-hadoop
|
||||
[46]:https://www.elastic.co/guide/en/elasticsearch/guide/current/distributed-cluster.html
|
||||
[47]:https://www.elastic.co/cloud/enterprise
|
||||
[48]:https://xapian.org/
|
||||
[49]:http://sphinxsearch.com/
|
||||
[50]:https://mariadb.com/kb/en/mariadb/sphinx-storage-engine/
|
||||
[51]:https://nutch.apache.org/
|
||||
[52]:http://commoncrawl.org/
|
||||
[53]:https://lunrjs.com/
|
||||
[54]:https://github.com/searchkit/searchkit
|
||||
[55]:https://github.com/fergiemcdowall/norch
|
||||
[56]:https://github.com/google/leveldb
|
||||
[57]:https://bitbucket.org/mchaput/whoosh/wiki/Home
|
||||
[58]:http://wiki.openstreetmap.org/wiki/Search_engines
|
||||
[59]:http://commoncrawl.org/
|
||||
[60]:https://aws.amazon.com/public-datasets/common-crawl/
|
||||
[61]:http://wiki.openstreetmap.org/wiki/Downloading_data
|
||||
[62]:http://commondatastorage.googleapis.com/books/syntactic-ngrams/index.html
|
||||
[63]:https://dumps.wikimedia.org/
|
||||
[64]:https://www.mediawiki.org/wiki/Alternative_parsers
|
||||
[65]:http://www.imdb.com/interfaces
|
||||
[66]:https://www.amazon.com/dp/0321416910
|
||||
[67]:https://www.amazon.com/dp/0262528878/
|
||||
[68]:https://www.amazon.com/dp/3642142664/
|
||||
[69]:https://www.amazon.com/dp/1558605703
|
||||
[70]:https://www.coursera.org/learn/text-retrieval
|
||||
[71]:https://research.google.com/pubs/pub37043.html
|
||||
[72]:https://pdfs.semanticscholar.org/28d8/288bff1b1fc693e6d80c238de9fe8b5e8160.pdf
|
||||
[73]:http://queue.acm.org/detail.cfm?id=988407
|
||||
[74]:https://github.com/harpribot/awesome-information-retrieval
|
||||
[75]:https://medium.com/@dtunkelang
|
||||
[76]:https://www.cs.cmu.edu/~quixote/
|
||||
[77]:https://web.stanford.edu/class/cs276/handouts/lecture8-evaluation_2014-one-per-page.pdf
|
||||
[78]:https://stackoverflow.com/questions/34314/how-do-i-implement-search-functionality-in-a-website
|
||||
[79]:https://www.quora.com/How-to-build-a-search-engine-from-scratch
|
||||
[80]:https://github.com/isaacs/github/issues/908
|
||||
[81]:https://www.reddit.com/r/Windows10/comments/4jbxgo/can_we_talk_about_how_bad_windows_10_search_sucks/d365mce/
|
||||
[82]:https://www.reddit.com/r/spotify/comments/2apwpd/the_search_function_sucks_let_me_explain/
|
||||
[83]:https://medium.com/@RohitPaulK/github-issues-suck-723a5b80a1a3#.yp8ui3g9i
|
||||
[84]:https://thenextweb.com/opinion/2016/01/11/netflix-search-sucks-flixed-fixes-it/
|
||||
[85]:https://www.google.com/search?q=building+a+search+engine
|
||||
[86]:http://airweb.cse.lehigh.edu/2005/gyongyi.pdf
|
||||
[87]:https://www.researchgate.net/profile/Gabriel_Sanchez-Perez/publication/262371199_Explicit_image_detection_using_YCbCr_space_color_model_as_skin_detection/links/549839cf0cf2519f5a1dd966.pdf
|
||||
[88]:https://en.wikipedia.org/wiki/Locality-sensitive_hashing
|
||||
[89]:https://en.wikipedia.org/wiki/Similarity_measure
|
||||
[90]:https://www.microsoft.com/en-us/research/wp-content/uploads/2011/02/RadlinskiBennettYilmaz_WSDM2011.pdf
|
||||
[91]:http://infolab.stanford.edu/~ullman/mmds/ch3.pdf
|
||||
[92]:https://en.wikipedia.org/wiki/Inverted_index
|
||||
[93]:https://en.wikipedia.org/wiki/Natural_language_processing
|
||||
[94]:http://backlinko.com/google-ranking-factors
|
||||
[95]:http://times.cs.uiuc.edu/czhai/pub/slmir-now.pdf
|
||||
[96]:https://en.wikipedia.org/wiki/Learning_to_rank
|
||||
[97]:https://papers.nips.cc/paper/3270-mcrank-learning-to-rank-using-multiple-classification-and-gradient-boosting.pdf
|
||||
[98]:https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/lambdarank.pdf
|
||||
[99]:https://yandex.com/company/technologies/matrixnet/
|
||||
[100]:https://arxiv.org/abs/1708.02702
|
||||
[101]:http://cnn.com/
|
||||
[102]:http://static.googleusercontent.com/media/www.google.com/en//insidesearch/howsearchworks/assets/searchqualityevaluatorguidelines.pdf
|
||||
[103]:https://googleblog.blogspot.co.uk/2008/08/search-experiments-large-and-small.html
|
||||
[104]:https://spark.apache.org/
|
||||
[105]:https://www.algolia.com/
|
||||
[106]:https://lucene.apache.org/
|
||||
[107]:https://lucy.apache.org/
|
||||
[108]:https://www.brainyquote.com/quotes/quotes/p/pierredefe204944.html
|
||||
[109]:https://www.linkedin.com/in/grigorev/
|
||||
[110]:mailto:forwidur@gmail.com
|
||||
[111]:https://twitter.com/forwidur
|
||||
[112]:https://github.com/open-guides/og-aws
|
||||
[113]:https://upscri.be/d29cfe/
|
||||
[114]:https://twitter.com/ojoshe
|
||||
[115]:https://twitter.com/lpolovets
|
||||
[116]:https://www.linkedin.com/in/abhishek-das-3280053/
|
||||
[117]:https://www.behance.net/gallery/3530289/-HORIZON-
|
||||
[118]:https://en.wikipedia.org/wiki/Henry_O._Studley
|
||||
@@ -0,0 +1,118 @@
|
||||
当你只想将事情搞定时，为什么开放式工作这么难？
|
||||
============================================================
|
||||
|
||||
### 学习使用开放式决策框架来写一本书
|
||||
|
||||
|
||||
>图片来源 : opensource.com
|
||||
|
||||
GSD（get stuff done 的缩写，即搞定）指导着我的工作方式。数年来，我将各种方法论融入我日常工作的习惯中，包括精益方法的反馈循环，和敏捷开发的迭代优化，以此来更好地 GSD（如果把 GSD 当作动词的话）。这意味着我必须非常有效地利用我的时间：列出清晰，各自独立的目标；标记已完成的项目；用迭代的方式地持续推进项目进度。但是当我们默认使用开放的时仍然能够 GSD 吗？又或者 GSD 的方法完全行不通呢？大多数人都认为这会导致糟糕的状况，但我发现事实并不一定这样。
|
||||
|
||||
在开放的环境中工作，遵循[开放式决策框架][6]中的指导，会让项目起步变慢。但是在最近的一个项目中，我们作出了一个决定，一个从开始就正确的决定：以开放的方式工作，并与我们的社群一起合作。
|
||||
|
||||
关于开放式组织的资料
|
||||
|
||||
* [下载《开放式组织 IT 文化变革指南》][1]
|
||||
* [下载《开放式组织领袖手册》][2]
|
||||
* [什么是开放式组织][3]
|
||||
* [什么是开放决策][4]
|
||||
|
||||
这是我们能做的最好的决定。
|
||||
|
||||
我们来看看这次经历带来的意想不到的结果，再看看我们如何将 GSD 思想融入开放式组织框架。
|
||||
|
||||
### 建立社区
|
||||
|
||||
2014 年 10 月，我接手了一个新的项目：当时红帽的 CEO Jim Whitehurst 即将推出一本新书《开放式组织》，我要根据书中提出的概念，建立一个社区。“太棒了，这听起来是一个挑战，我加入了！”我这样想。但不久，[冒牌者综合征][7]便出现了，我又开始想：“我们究竟要做什么呢？怎样才算成功呢？”
|
||||
|
||||
让我剧透一下，在这本书的结尾处，Jim 鼓励读者访问 Opensource.com，继续探讨 21 世纪的开放和管理。所以，在 2015 年 5 月，我们的团队在网站上建立了一个新的板块来讨论这些想法。我们计划讲一些故事，就像我们在 Opensource.com 上常做的那样，只不过这次围绕着书中的观点与概念。之后，我们每周都发布新的文章，在 Twitter 上举办了一个在线的读书俱乐部，还将《开放式组织》打造成了系列书籍。
|
||||
|
||||
我们内部独自完成了该系列书籍的前三期，每隔六个月发布一期。每完成一期，我们就向社区发布。然后我们继续完成下一期的工作，如此循环下去。
|
||||
|
||||
这种工作方式，让我们看到了很大的成功。近 3000 人订阅了[该系列的新书][9]，《开放式组织领袖手册》。我们用 6 个月的周期来完成这个项目，这样新书的发行日正好是前书的两周年纪念日。
|
||||
|
||||
在这样的背景下，我们完成这本书的方式是简单直接的：针对开放工作这个主题，我们收集了最好的故事，并将它们组织起来形成文章，招募作者填补一些内容上的空白，使用开源工具调整字体样式，与设计师一起完成封面，最终发布这本书。这样的工作方式使得我们能按照自己的时间线（GSD）全速前进。到[第三本书][10]时，我们的工作流已经基本完善了。
|
||||
|
||||
然而这一切在我们计划开始《开放式组织》的最后一本书时改变了，这本书将重点放在开放式组织和 IT 文化的交融上。我提议使用开放式决策框架来完成这本书，因为我想通过这本书证明开放式的工作方法能得到更好的结果，尽管我知道这可能会完全改变我们的工作方式。时间非常紧张（只有两个半月），但我们还是决定试一试。
|
||||
|
||||
### 用开放式决策框架来完成一本书
|
||||
|
||||
开放式决策框架列出了组成开放决策制定过程的 4 个阶段。下面是我们在每个阶段中的工作情况（以及开放是如何帮助完成工作的）。
|
||||
|
||||
### 1\. 构思
|
||||
|
||||
我们首先写了一份草稿，罗列了对项目设想的愿景。我们需要拿出东西来和潜在的“顾客”分享（在这个例子中，“顾客”指潜在的利益相关者和作者）。然后我们约了一些领域专家面谈，这些专家能够给我们直接的诚实的意见。这些专家表现出的热情与他们提供的指导验证了我们的想法，同时提出了反馈意见使我们能继续向前。如果我们没有得到这些验证，我们会退回到我们最初的想法，再决定从哪里重新开始。
|
||||
|
||||
### 2\. 计划与研究
|
||||
|
||||
经过几次面谈，我们准备在 [Opensource.com 上公布这个项目][11]。同时，我们在 [Github 上也公布了这个项目][12], 提供了项目描述，预计的时间线，并阐明了我们所受的约束。这次公布得到了很好的效果，我们最初计划的目录中欠缺了一些内容，在项目公布之后的 72 小时内就被补充完整了。另外（也是更重要的），读者针对一些章节，提出了本不在我们计划中的想法，但是读者觉得这些想法能够补充我们最初设想的版本。
|
||||
|
||||
我们体会到了 [Linus 法则][16]： "With more eyes, all _typos_ are shallow."
|
||||
|
||||
回顾过去，我觉得在项目的第一和第二个阶段，开放项目并不会影响我们搞定项目的能力。事实上，这样工作有一个很大的好处：发现并填补内容的空缺。我们不只是填补了空缺，我们是迅速地就填补了空缺，并且还是用我们自己从未考虑过的点子。这并不一定要求我们做更多的工作，只是改变了我们的工作方式。我们动用有限的人脉，邀请别人来写作，再组织收到的内容，设置上下文，将人们导向正确的方向。
|
||||
|
||||
### 3\. 设计，开发和测试
|
||||
|
||||
项目的这个阶段完全围绕项目管理，管理一些像猫一样特立独行的人，并处理项目的预期。我们有明确的截止时间，我们提前沟通，频繁沟通。我们还使用了一个战略：列出了贡献者和利益相关者，在项目的整个过程中向他们告知项目的进度，尤其是我们在 Github 上标出的里程碑。
|
||||
|
||||
最后，我们的书需要一个名字。我们收集了许多反馈，指出书名应该是什么，更重要的反馈指出了书名不应该是什么。我们通过 [Github 上的 issue][13] 收集反馈意见，并公开表示我们的团队将作最后的决定。当我们准备宣布最后的书名时，我的同事 Bryan Behrenshausen 做了很好的工作，[分享了我们作出决定的过程][14]。人们似乎对此感到高兴——即使他们不同意我们最后的书名。
|
||||
|
||||
书的“测试”阶段需要大量的[校对][15]。社区成员真的参与到回答这个“求助”贴中来。我们在 GitHub issue 上收到了大约 80 条意见，汇报校对工作的进度（更不用说通过电子邮件和其他反馈渠道获得的许多额外的反馈）。
|
||||
|
||||
关于搞定任务：在这个阶段，我们亲身体会了 [Linus 法则][16]："With more eyes, all _typos_ are shallow." 如果我们像前三本书一样自己独立完成，那么整个校对的负担就会落在我们的肩上（就像这些书一样）！相反，社区成员慷慨地帮我们承担了校对的重担，我们的工作从自己校对（尽管我们仍然做了很多工作）转向管理所有的 change requests。对我们团队来说，这是一个受大家欢迎的改变；对社区来说，这是一个参与的机会。如果我们自己做的话，我们肯定能更快地完成校对，但是在开放的情况下，我们在截止日期之前发现了更多的错误，这一点毋庸置疑。
|
||||
|
||||
### 4\. Launch
|
||||
|
||||
### 4\. 发布
|
||||
|
||||
好了，我们现在推出了这本书的最终版本。（或者只是第一版？）
|
||||
|
||||
遵循开放决策框架是《IT 文化变革指南》成功的关键。
|
||||
|
||||
我们把发布分为两个阶段。首先，根据我们的公开的项目时间表，在最终日期之前的几天，我们安静地推出了这本书，以便让我们的社区贡献者帮助我们测试[下载表格][17]。第二阶段也就是现在，这本书的[通用版][18]的正式公布。当然，我们在发布后的仍然接受反馈，开源方式也正是如此。
|
||||
|
||||
### 成就解锁
|
||||
|
||||
遵循开放式决策框架是《IT 文化变革指南》成功的关键。通过与客户和利益相关者的合作，分享我们的制约因素，工作透明化，我们甚至超出了自己对图书项目的期望。
|
||||
|
||||
我对整个项目中的合作，反馈和活动感到非常满意。虽然有一段时间内没有像我想要的那样快速完成任务，这让我有一种焦虑感，但我很快就意识到，开放这个过程实际上让我们能完成更多的事情。基于上面我的概述这一点显而易见。
|
||||
|
||||
所以也许我应该重新考虑我的 GSD 心态，并将其扩展到 GMD：Get **more** done，搞定**更多**工作，并且就这个例子说，取得更好的结果。
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
作者简介：
|
||||
|
||||
Jason Hibbets - Jason Hibbets 是 Red Hat 企业营销中的高级社区传播者，也是 Opensource.com 的社区经理。 他自2003年以来一直在 Red Hat，并且是开源城市基金会的创立者。之前的职位包括高级营销专员，项目经理，Red Hat 知识库维护人员和支持工程师。
|
||||
|
||||
-----------
|
||||
|
||||
via: https://opensource.com/open-organization/17/6/working-open-and-gsd
|
||||
|
||||
作者：[Jason Hibbets ][a]
|
||||
译者：[explosic4](https://github.com/explosic4)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://opensource.com/users/jhibbets
|
||||
[1]:https://opensource.com/open-organization/resources/culture-change?src=too_resource_menu
|
||||
[2]:https://opensource.com/open-organization/resources/leaders-manual?src=too_resource_menu
|
||||
[3]:https://opensource.com/open-organization/resources/open-org-definition?src=too_resource_menu
|
||||
[4]:https://opensource.com/open-organization/resources/open-decision-framework?src=too_resource_menu
|
||||
[5]:https://opensource.com/open-organization/17/6/working-open-and-gsd?rate=ZgpGc0D07SjGkTOf708lnNqbF_HvkhXTXeSzRKMhvVM
|
||||
[6]:https://opensource.com/open-organization/resources/open-decision-framework
|
||||
[7]:https://opensource.com/open-organization/17/5/team-impostor-syndrome
|
||||
[8]:https://opensource.com/open-organization/resources
|
||||
[9]:https://opensource.com/open-organization/resources/leaders-manual
|
||||
[10]:https://opensource.com/open-organization/resources/leaders-manual
|
||||
[11]:https://opensource.com/open-organization/17/3/announcing-it-culture-book
|
||||
[12]:https://github.com/open-organization-ambassadors/open-org-it-culture
|
||||
[13]:https://github.com/open-organization-ambassadors/open-org-it-culture/issues/20
|
||||
[14]:https://github.com/open-organization-ambassadors/open-org-it-culture/issues/20#issuecomment-297970303
|
||||
[15]:https://github.com/open-organization-ambassadors/open-org-it-culture/issues/29
|
||||
[16]:https://en.wikipedia.org/wiki/Linus%27s_Law
|
||||
[17]:https://opensource.com/open-organization/resources/culture-change
|
||||
[18]:https://opensource.com/open-organization/resources/culture-change
|
||||
[19]:https://opensource.com/user/10530/feed
|
||||
[20]:https://opensource.com/users/jhibbets
|
||||
101
translated/tech/20141107 Dont Waste Time Writing Perfect Code.md
Normal file
101
translated/tech/20141107 Dont Waste Time Writing Perfect Code.md
Normal file
@@ -0,0 +1,101 @@
|
||||
不要浪费时间写出完美的代码
|
||||
============================================================
|
||||
|
||||
|
||||
系统可以持续 5 年或 10 年甚至 20 年或者更多年。但是，特定代码行的生命，即使是设计，通常要短得多：当你通过不同的方法来解决问题，它会有几个月或几天甚至几分钟的生命。
|
||||
|
||||
### 一些代码比其他代码重要
|
||||
|
||||
通过研究[代码如何随时间变化][4]，Michael Feathers 确定了[一个代码库曲线][5]。每个系统都有代码，通常有很多是一次性写的，永远都不会改变。但是有少量的代码，包括最重要和最有用的代码，会一次又一次地改变、几次重构或者从头重写。
|
||||
|
||||
当你在一个系统中有更多体验，或者有一个问题领域或体系结构方法时，应该更容易了解并预测什么代码将永远改变，哪些代码将永远不会改变：什么代码重要，什么代码不重要。
|
||||
|
||||
### 我们应该尝试编写完美的代码么？
|
||||
|
||||
我们知道我们应该写[干净的代码][6]，代码是一致的，很明显也要尽可能简单的。
|
||||
|
||||
有些人把这变成极端，他们迫使自己写出[美丽][7]、优雅，接近[完美][8]的代码，[痴迷重构][9]并且纠结每个细节。
|
||||
|
||||
但是，如果代码只写一次而不改变，或者如果在另一个极端下，它一直在改变的话，就如尝试写完美的需求护着尝试完美的前期设计那样，写完美的代码难道不是既浪费又没有必要（也不可能实现）的么？
|
||||
|
||||
> “你不能写完美的软件。这受伤么？作为生活的公理接受它、拥抱它、庆祝它。因为完美的软件不存在。计算机的短暂历史中没有人写过一个完美的软件。你不可能成为第一个。除非你接受这个事实，否则你最终会浪费时间和精力追逐不可能的梦想。”
|
||||
> Andrew Hunt，[务实的程序员: 从熟练工到大师][10]
|
||||
|
||||
一次性写的代码不需要美观优雅。但它必须是正确的、可以理解的 - 因为不会改变的代码在系统的整个生命周期内可能仍然被阅读很多次。它不需要干净和紧凑 - 只要干净够了。代码中[复制和粘贴][11]和其他小的裁剪是允许的，至少要达到这点。这是永远不需要打磨的代码。即使周围的其他代码正在更改，这也是不需要重构的代码（除非你需要更改它）。这是不值得花费额外时间的代码。
|
||||
|
||||
你一直在改变的代码怎么样？纠结风格以及提出最优雅的解决方案是浪费时间，因为这段代码可能会再次更改，甚至可能会在几天或几周内重写。因此，每当你进行更改时，都会[痴迷重构][12]代码，或者没有重构没有改变的代码，因为它可能会更好。代码总是可以更好。但这并不重要。
|
||||
|
||||
重要的是：代码是否做了应该做的 - 是正确的、可用的和高效的？它可以[处理错误和不良数据][13]而不会崩溃 - 至少[安全地失败][14]？调试容易吗？改变是否容易安全？这些不是美的主观方面。这些是成功与失败实际措施之间的差异。
|
||||
|
||||
### 务实编码和重构
|
||||
|
||||
精益发展的核心思想是：不要浪费时间在不重要的事情上。这应该提醒我们该如何编写代码，以及我们如何重构它、审查它、测试它。
|
||||
|
||||
为了让工作完成，只[重构你需要的][15] - [Martin Fowler][16] 称之为机会主义重构（理解、清理、[童子军规则][17] ）和准备重构。足够使变化更加容易和安全，而不是更多。如果你不改变代码，那么它并不会如看起来的那么重要。
|
||||
|
||||
在代码审查中，只聚焦在[重要的事上][18]。代码是否正确？有防御吗？是否安全？你能理解么？改变是否安全？
|
||||
|
||||
忘记风格（除非风格变成无法理解）。让你的 IDE 处理格式化。不要争议代码是否应该是“更多的 OO”。只要它有意义，它是否适当地遵循这种或那种模式并不重要。无论你喜欢还是不喜欢都没关系。无论你有更好的方式做到这一点并不重要 - 除非你在教新接触这个平台或者语言的人，而且需要在做代码审查时做一部分指导。
|
||||
|
||||
写测试很重要。测试涵盖主要流程和重要的意外情况。测试让你用最少的工作获得最多的信息和最大的信心。[大面积覆盖测试，或小型测试][19] - 都没关系，只要他们做这个工作，在编写代码之前或之后编写测试并不重要。
|
||||
|
||||
### 不是（只是）关于代码
|
||||
|
||||
建筑和工程隐喻从未对软件有效。我们不是设计和建造几年或几代将保持基本不变的桥梁或摩天大楼。我们构建的更加弹性和抽象，更加短暂的东西。代码写来是被修改的 - 这就是为什么它被称为“软件”。
|
||||
|
||||
> “经过五年的使用和修改，成功的软件程序的源码通常和它的原始形式完全无法识别，而五年后的成功建筑几乎没有变化。”
|
||||
> Kevin Tate，[可持续软件开发][20]
|
||||
|
||||
我们需要将代码看作是我们工作的一个暂时的人工品：
|
||||
|
||||
> 有时候面对更重要的事情时，我们被引导迷信代码。我们经常有一个错觉，让发出的产品有价值的是代码，然而实际上可能是对问题领域的了解、设计难题的进展甚至是客户反馈。
|
||||
> Dan Grover，[Code and Creative Destruction][21]
|
||||
|
||||
迭代开发教会我们来体验和研究我们工作的结果 - 我们是否解决了这个问题，如果没有，我们学到了什么，我们如何改进？软件构建从不会完成。即使设计和代码是正确的，它们也可能只是一段时间，直到环境要求再次更改或替换为更好的东西。
|
||||
|
||||
我们需要编写好的代码：代码可以理解、正确、安全和可靠。我们需要重构和审查它，并写出好的有用的测试，同时知道这其中一些或者所有的代码，可能会很快被抛弃，或者它可能永远不会被再被查看，或者它可能根本不用。我们需要认识到，我们的一些工作必然会被浪费，并为此而进行优化。做需要做的，没有别的了。不要浪费时间尝试编写完美的代码。
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
作者简介：
|
||||
|
||||
Jim Bird
|
||||
|
||||
|
||||
我是一名经验丰富的软件开发经理，项目经理和 CTO，专注于软件开发和维护、软件质量和安全性方面的困难问题。在过去 15 年中，我一直在管理建立全球证券交易所和投资银行电子交易平台的团队。我特别感兴趣的是，小团队在构建真正的软件中如何有效率：在可靠性，性能和适应性极限限制下的高质量，安全系统。
|
||||
|
||||
------
|
||||
|
||||
via: https://dzone.com/articles/dont-waste-time-writing
|
||||
|
||||
作者：[Jim Bird][a]
|
||||
译者：[geekpi](https://github.com/geekpi)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://dzone.com/users/722527/jim.bird.html
|
||||
[1]:https://dzone.com/users/722527/jim.bird.html
|
||||
[2]:https://dzone.com/users/722527/jim.bird.html
|
||||
[3]:https://dzone.com/articles/dont-waste-time-writing?utm_source=wanqu.co&utm_campaign=Wanqu%20Daily&utm_medium=website#
|
||||
[4]:http://www.youtube.com/watch?v=0eAhzJ_KM-Q
|
||||
[5]:http://swreflections.blogspot.ca/2012/10/bad-things-happen-to-good-code.html
|
||||
[6]:http://www.amazon.com/Clean-Code-Handbook-Software-Craftsmanship/dp/0132350882
|
||||
[7]:http://www.makinggoodsoftware.com/2011/03/27/the-obsession-with-beautiful-code-the-refactor-syndrome/
|
||||
[8]:http://stackoverflow.com/questions/1196405/how-to-keep-yourself-from-perfectionism-when-coding
|
||||
[9]:http://programmers.stackexchange.com/questions/43506/is-it-bad-to-have-an-obsessive-refactoring-disorder
|
||||
[10]:https://pragprog.com/the-pragmatic-programmer
|
||||
[11]:http://swreflections.blogspot.com/2012/03/is-copy-and-paste-programming-really.html
|
||||
[12]:http://programmers.stackexchange.com/questions/43506/is-it-bad-to-have-an-obsessive-refactoring-disorder
|
||||
[13]:http://swreflections.blogspot.com/2012/03/defensive-programming-being-just-enough.html
|
||||
[14]:https://buildsecurityin.us-cert.gov/articles/knowledge/principles/failing-securely
|
||||
[15]:http://swreflections.blogspot.com/2012/04/what-refactoring-is-and-what-it-isnt.html
|
||||
[16]:http://martinfowler.com/articles/workflowsOfRefactoring/
|
||||
[17]:http://programmer.97things.oreilly.com/wiki/index.php/The_Boy_Scout_Rule
|
||||
[18]:http://randomthoughtsonjavaprogramming.blogspot.com/2014/08/building-real-software-dont-waste-time.html
|
||||
[19]:http://swreflections.blogspot.com/2012/08/whats-better-big-fat-tests-or-little.html
|
||||
[20]:http://www.amazon.com/Sustainable-Software-Development-Agile-Perspective/dp/0321286081
|
||||
[21]:http://dangrover.com/2013/07/16/code-and-creative-destruction/
|
||||
[22]:https://dzone.com/devops-tutorials-tools-news
|
||||
[23]:https://dzone.com/articles/dont-waste-time-writing?utm_source=wanqu.co&utm_campaign=Wanqu%20Daily&utm_medium=website#
|
||||
[24]:https://dzone.com/go?i=228233&u=https%3A%2F%2Foffers.automic.com%2Fblueprint-to-continuous-delivery-with-automic-release-automation%3Futm_campaign%3DAMER%252520Online%252520Syndication%252520DZone%252520Platinum%252520Sponsorship%252520Ads%252520JULY-2017%26utm_source%3DDzone%252520Ads%26utm_medium%3DBlueprint%252520to%252520CD
|
||||
@@ -0,0 +1,127 @@
|
||||
你需要知道的最佳 Linux Adobe 替代品
|
||||
============================================================
|
||||
|
||||
|
||||
|
||||
|
||||
作为一名 Linux 用户正在寻找 Adobe 的替代品？你不是一个人。如果你是一个狂热的平面设计师，那么你可能很擅长避开昂贵的 Adobe 产品。不过，对于 Adobe 来说，Linux 用户通常是不利的一面。因此，Adobe 的替代品是必须的 - 但是最好的选择是什么？
|
||||
|
||||
它最终归结到具体的 Adobe 程序和你希望完成的事情。幸运的是，由于需求是所有发明之母，有人响应了号召。其结果是出现了一系列高效的 Adobe 插件。
|
||||
|
||||
### Evince (Adobe Acrobat)
|
||||
|
||||
|
||||
|
||||
就像 Adobe Acrobat Reader 一样，[Evince][6] 是一个“支持多种文档格式的文档查看器”。例如，用户可以使用 Evince 作为 PDF 查看器。它还支持各种漫画书格式（cbr、cbz、cb7 和 cbt）。你可以在 Evince 网站上找到[支持格式的完整列表][7]。
|
||||
|
||||
Evince 没有高看或低看 Linux 用户，因为它是标准的。你可以在需要时前往官方网站进行更新。
|
||||
|
||||
### Pixlr (Adobe Photoshop)
|
||||
|
||||
|
||||
|
||||
关于 [Pixlr][8] 的很棒的一点在于 Adobe 的各种替代工具可以在线获得。如果你有一个互联网连接，那么你就有一个强大的图像编辑工作区。
|
||||
|
||||
[Pixlr Editor][9] 是 Photoshop 的一个功能强大的插件，因为你可以使用图层和相关效果。它还有一些漂亮的绘图和颜色编辑工具。[Pixlr Express][10] 没有这么多功能，因为它主要用于图像增强。调整颜色和清晰度，并增加一些 Instagram 友好的效果！
|
||||
|
||||
你可以通过 Pixlr 完成任务，这是完全免费的。
|
||||
|
||||
### Inkscape (Adobe Illustrator)
|
||||
|
||||
|
||||
|
||||
[Inkscape][11]是另一个值得推荐的免费 Adobe 替代品。它主要作为一个“专业的矢量图形编辑器”。除了 Illustrator，Inkscape 也受到了 Corel Draw、Freehand 和 Xara X 的限制。
|
||||
|
||||
它的矢量设计工具可用于制作 logo 和“高可伸缩性”艺术品。Inkscape 包含绘图、形状和文本工具。图层工具允许你锁定、分组或隐藏单个图层。
|
||||
|
||||
### Pinegrow Web Editor (Adobe Dreamweaver)
|
||||
|
||||
|
||||
|
||||
[Pinegrow Web Editor][12] 是 Linux 上 Dreamweaver 的绝佳替代品。该程序可让你在桌面上直接制作 HTML 网站。
|
||||
|
||||
不仅是使用代码创建（而且需要稍后预览），Pinegrow 可以提供详细的视觉编辑体验。你可以直接查看和测试你的 HTML 项目，实时了解链接是否正常工作，或者图片是否在它该在的地方。Pinegrow 还附带了 WordPress 主题构建器。
|
||||
|
||||
免费试用 30 天。如果你喜欢，你可以一次性支付 $49 购买。
|
||||
|
||||
### Scribus (Adobe InDesign)
|
||||
|
||||
|
||||
|
||||
[Scribus][13] 可能是最接近 Adobe InDesign 的替代品。根据开发者的说法，你应该[认真考虑使用][14] Scribus，因为它是可靠和免费的。
|
||||
|
||||
实际上，Scribus 不仅仅是一个出色的桌面出版工具，也是一个很好的_自我_出版工具。当你可以自己做高质量的杂志和书籍时，为什么要依靠昂贵的商业软件来创建？Scribus 目前允许设计师使用一个 200 色的调色板，下一个稳定版中[承诺将会加倍颜色][15]。
|
||||
|
||||
### digiKam (Adobe Lightroom)
|
||||
|
||||
|
||||
|
||||
[digiKam][16] 也许是目前 Linux 用户最好的 Lightroom 选择。功能包括导入照片、整理图片集、图像增强、创建幻灯片等功能。
|
||||
|
||||
它的时尚设计和先进的功能是真正的爱的劳动。实际上，digiKam 背后的人是摄影师。不仅如此，他们希望在 Linux 中完成在 Lightroom 能做的任何工作。
|
||||
|
||||
### Webflow (Adobe Muse)
|
||||
|
||||
|
||||
|
||||
[Webflow][17] 是另一个可以证明你无需下载软件而可以完成很多事的网站。一个非常方便的 Adobe Muse 替代品，Webflow 是创建高响应式网站设计的理想选择。
|
||||
|
||||
Webflow 的最好的一方面是你不需要自己编写代码。你只需拖放图像并写入文本。Webflow 为你做了所有杂事。你可以从头开始构建网站，也可以使用各种模板。虽然是免费的，但是高级选项还有额外的功能，如能够轻松地导出 HTML 和 CSS 以在其他地方使用。
|
||||
|
||||
### Tupi (Adobe Animate)
|
||||
|
||||
|
||||
|
||||
[Tupi][18] 是 Adobe Animate 的替代品，或者那些[不太热衷于 Flash 的人][19]。当然，Tupi 的作者说这并不是与 Flash 竞争。然而，能够使用 HTML5 并不能阻止它成为理想的替代品。
|
||||
|
||||
在 PC 或平板电脑上绘制 2D 动画。不确定如何开始？使用网站的[ YouTube 教程][20]了解如何制作剪贴画动画以及更多。
|
||||
|
||||
### Black Magic Fusion (Adobe After Effects)
|
||||
|
||||
|
||||
|
||||
[Black Magic Fusion][21]注定是 Adobe After Effects 的替代者。这个视觉效果软件有大约 25 年的开发！Fusion 通常用于在好莱坞电影和电视节目中制造令人印象深刻的效果 - 将其详细和时尚的功能带到家庭。
|
||||
|
||||
Fusion 通过使用节点，那些“代表效果、过滤器和其他处理的小图标”工作。将这些节点连接在一起，创建一系列复杂的视觉效果。该程序包括许多功能，如图片修饰、对象跟踪和令人兴奋的 3D 效果。
|
||||
|
||||
你可以选择免费版或者 $995 的 Fusion Studio。为了帮助你决定，[你可以比较][22]免费和高级版的 Fusion 功能。
|
||||
|
||||
### 总结
|
||||
|
||||
如你所见，其他的远远超过你的 Adobe 替代品。由于开源独创性，显著提升的替代品的持续发布。我们很快就会看到一个仅为 Linux 用户创建的完整套件。在此之前，你可以随意选择这些替代品。
|
||||
|
||||
知道这里没有提到的任何有用的 Adobe 替代品？在下面的评论区分享软件建议。
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://www.maketecheasier.com/adobe-alternatives-for-linux/
|
||||
|
||||
作者：[ Toni Matthews-El][a]
|
||||
译者：[geekpi](https://github.com/geekpi)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://www.maketecheasier.com/author/ttmatthe/
|
||||
[1]:https://www.maketecheasier.com/author/ttmatthe/
|
||||
[2]:https://www.maketecheasier.com/adobe-alternatives-for-linux/#comments
|
||||
[3]:https://www.maketecheasier.com/category/linux-tips/
|
||||
[4]:https://support.google.com/adsense/troubleshooter/1631343
|
||||
[5]:https://support.google.com/adsense/troubleshooter/1631343
|
||||
[6]:https://wiki.gnome.org/Apps/Evince
|
||||
[7]:https://wiki.gnome.org/Apps/Evince/SupportedDocumentFormats
|
||||
[8]:https://pixlr.com/
|
||||
[9]:https://pixlr.com/editor/
|
||||
[10]:https://pixlr.com/express/
|
||||
[11]:https://inkscape.org/en/
|
||||
[12]:http://pinegrow.com/
|
||||
[13]:https://www.scribus.net/
|
||||
[14]:https://www.scribus.net/why-on-earth-should-i-use-scribus-2/
|
||||
[15]:https://www.scribus.net/because-color-matters/
|
||||
[16]:http://digikam.org/
|
||||
[17]:https://webflow.com/
|
||||
[18]:http://www.maefloresta.com/portal/
|
||||
[19]:https://www.maketecheasier.com/sites-moving-away-flash/
|
||||
[20]:https://www.youtube.com/user/maefloresta
|
||||
[21]:https://www.blackmagicdesign.com/
|
||||
[22]:https://www.blackmagicdesign.com/products/fusion/compare
|
||||
@@ -1,197 +0,0 @@
|
||||
Linux 文件系统简介
|
||||
====
|
||||
|
||||
|
||||
图片来源 : 原始图片来自 Rikki Endsley. [CC BY-SA 4.0][9]
|
||||
|
||||
本文旨在对 Linux 文件系统的概念进行高层次的讨论，而不是对某种特定的文件系统，比如 EXT4 是如何工作的进行底层的描述。另外，本文也不是一个文件系统命令的教程。
|
||||
|
||||
每台通用计算机都需要将各种数据存储在硬盘驱动器（HDD）或其他类似设备上，比如 USB 存储器。这样做有两个原因。首先，当计算机关闭以后，内存（RAM）会失去存于它里面的内容。尽管存在非易失类型的 RAM，在计算机断电以后还能把数据存储下来（比如采用 USB 闪存和固态硬盘的 flash RAM），但是，flash RAM 和标准的、易失性的 RAM，比如 DDR3 以及其他相似类型的 RAM 相比，要贵很多。
|
||||
|
||||
数据需要存储在硬盘驱动上的另一个原因是，即使是标准 RAM 也要比普通硬盘贵得多。尽管 RAM 和硬盘的价格都在迅速下降，但是 RAM 的价格依旧在以字节为单位来计算。让我们进行一个以字节为单位的快速计算：基于 16 GB 大的 RAM 的价格和 2 TB 大的硬盘驱动的价格。计算显示 RAM 的价格大约比硬盘驱动贵 71 倍。今天，一个典型的 RAM 的价格大约是 0.000000004373750 美元/每字节。
|
||||
|
||||
直观的展示一下在很久以前 RAM 的价格，在计算机发展的非常早的时期，其中一种类型的 RAM 是基于在 CRT 屏幕上的点。这种 RAM 非常昂贵，大约 1 美元/每字节。
|
||||
|
||||
### 定义
|
||||
|
||||
你可能听过其他人以各种不同和迷惑的方式谈论文件系统。这个单词本身有多重含义，你需要从一个讨论或文件的上下文中理解它的正确含义。
|
||||
|
||||
我将根据我所观察到的在不同情况下使用“文件系统”这个词来定义它的不同含义。注意，尽管我试图遵循标准的“官方”含义，但是我打算基于它的不同用法来定义这个术语。这意味着我将在本文的后续章节中进行更详细的探讨。
|
||||
|
||||
1. 整个 Linux 目录结构从顶层 root （/）目录开始。
|
||||
2. 一种特定类型的数据存储格式，比如 EXT3、EXT4、BTRFS 以及 XFS 等等。Linux 几乎支持 100 种类型的文件系统，包括一些非常老的以及一些最新的。每一种文件系统类型都使用它自己独特的元数据结构来定义数据是如何存储和访问的。
|
||||
3. 用特定类型的文件系统格式化后的分区或逻辑卷，可以挂载到Linux文件系统的指定挂载点上。
|
||||
|
||||
|
||||
### 文件系统的基本功能
|
||||
|
||||
磁盘存储是必须的，它伴有一些有趣而且不可避免的细节。很明显，文件系统是用来为非易失数据的存储提供空间，这是它的基本功能。然而，它还有许多从需求出发的重要功能。
|
||||
|
||||
所有文件系统都需要提供一个名字空间，这是一种命名和组织方法。它定义了文件应该如何命令，文件名的最大长度，以及所有可用字符集中可用于文件名中字符集。它也定义了一个磁盘上数据的逻辑结构，比如使用目录来组织文件而不是把所有文件聚集成一个单一的大文件。
|
||||
|
||||
定义名字空间以后，元数据结构是为该名字空间提供逻辑基础所必须的。这包括所需数据结构要能够支持分层目录结构，同时能够通过结构来确定硬盘空间中的块已用或可用，支持维护文件或目录的名字，提供关于文件大小、创建时间、最后访问或修改时间等信息，以及位置或数据所属的文件在磁盘空间中的位置。其他的元数据用来存储关于磁盘细分的高级信息，比如逻辑容量和分区。这种更高层次的元数据以及它所代表的结构包含描述文件系统存储在驱动或分区中的信息，但与文件系统元数据无关，与之独立。
|
||||
|
||||
文件系统也需要一个应用程序接口（API），从而提供了对维护文件系统对象，比如文件和目录的系统功能调用的访问。API 也提供了诸如创建、移动和删除文件的功能。它也提供了算法来确定文件存于文件系统中的位置。这样的算法可以用来解释诸如磁盘速度和最小化磁盘碎片等术语。
|
||||
|
||||
现代文件系统还提供一个安全模型，这是一个定义文件和目录的访问权限的方案。Linux 文件系统安全模型确保用户只能访问自己的文件，而不能访问其他用户的文件或操作系统本身。
|
||||
|
||||
最后的构建块是实现这些所有功能所需要的软件。Linux 使用两层的软件实现的方式来提高系统和程序员的效率。
|
||||
|
||||
|
||||
|
||||
|
||||
图片 1：Linux 两层文件系统软件实现。
|
||||
|
||||
第一层实现是 Linux 虚拟文件系统。虚拟文件系统提供了内核和开发者访问所有类型文件系统的的单一命令集。虚拟文件系统软件通过调用特殊设备驱动来和不同类型的文件系统进行交互。文件系统特定设备驱动是第二层实现。设备驱动程序将文件系统命令的标准集解释为在分区或逻辑卷上的特定类型文件系统命令。
|
||||
|
||||
### 目录结构
|
||||
|
||||
作为一个通常来说非常有条理的处女座，我喜欢将东西存储在更小的、有组织的小容器中，而不是存于同一个大容器中。目录的使用使我能够存储文件，同时当我想要查看这些文件的时候也能够定位。目录也被称为文件夹，之所以被称为文件夹，是因为其中的文件被类比存于物理桌面上。
|
||||
|
||||
在 Linux 和其他许多操作系统中，目录可以被组织成树状的分层结构。在 [Linux 文件系统层次标准][10]中定义了 Linux 文件系统层次。当通过目录引用来访问目录时，更深层目录名字是通过正斜杠（/）来连接，从而形成一个序列的，比如 `/var/log` 和 `/var/spool/mail` 。这些被称为路径。
|
||||
|
||||
下表提供了标准的、众所周知的、定义好的顶层 Linux 目录及其用途的简要清单。
|
||||
|
||||
| 目录 | 描述 |
|
||||
| ------------- | ---------------------------------------- |
|
||||
| / (root 文件系统) | root 文件系统是文件系统的顶级目录。它必须包含在挂载其他文件系统前需要用来启动 Linux 系统的全部文件。它必须包含需要用来启动剩余文件系统的全部可执行文件和库。文件系统启动以后，所有其他文件系统作为 root 文件系统的子目录挂载到标准的、预定义好的挂载点上。 |
|
||||
| /bin | `/bin` 目录包含用户的可执行文件。 |
|
||||
| /boot | 包含启动 Linux 系统所需要的静态引导程序和内核可执行文件以及配置文件。 |
|
||||
| /dev | 该目录包含每一个连接到系统的硬件设备的设备文件。这些文件不是设备驱动，而是代表计算机上的每一个计算机能够访问的设备。 |
|
||||
| /etc | 包含主机计算机的本地系统配置文件。 |
|
||||
| /home | `Home` 目录存储用户文件，每一个用户都有一个子目录位于 `/home` 目录中。 |
|
||||
| /lib | 包含启动系统所需要的共享库文件。 |
|
||||
| /media | 一个挂载外部可移动设备的地方，比如 USB 驱动可能连接到主机。 |
|
||||
| /mnt | 一个普通文件系统的临时挂载点（如不可移动媒介），当管理员对一个文件系统进行修复或在其上工作时可以使用。 |
|
||||
| /opt | 可选文件比如供应商提供的应用程序应该安装在这儿。 |
|
||||
| /root | 这不是 `root`（/）文件系统。它是 root 用户的 `home` 目录。 |
|
||||
| /sbin | 系统二进制文件。这些是用于系统管理的可执行文件。 |
|
||||
| /tmp | 临时目录。被操作系统和许多程序用来存储临时文件。用户也可能临时在这儿存储文件。注意，存储在这儿的文件可能在任何时候在没有通知的情况下被删除。 |
|
||||
| /var | 可变数据文件存储在这儿。这些文件包括日志文件，MySQL 和其他数据库文件，网络服务器数据文件，邮件以及更多。 |
|
||||
| /usr | 该目录里面包含共享只读文件，包括可执行二进制文件和库，man 文件以及其他类型的文档。 |
|
||||
|
||||
*表 1：Linux 文件系统顶层*
|
||||
|
||||
这些目录以及它们的子目录如表 1 所示，在所有子目录中，有蓝绿色背景的目录组成了 root 文件系统的一部分。也就是说，它们不能作为一个分离的文件系统并且在开机时进行挂载。这是因为它们（特别是它们包含的内容）必须在系统启动的时候出现，从而系统才能正确启动。
|
||||
|
||||
`/media` 目录和 `/mnt` 目录是 root 文件系统的一部分，但是它们从来不包含任何数据，因为它们只是一个临时挂载点。
|
||||
|
||||
表 1 中剩下的没有背景颜色的目录不需要在系统启动过程中出现，但会在之后挂载到 root 文件系统上，在开机阶段，它们为主机进行准备，从而执行有用的工作。
|
||||
|
||||
请参考官方 [Linux 文件系统层次标准][11]（FHS）网页来来了解这些每一个目录以及它们的子目录的更多细节。维基百科上也有关于 [FHS][12] 的一个很好的介绍。应该尽可能的遵循这些标准，从而确保操作和功能的一致性。无论在主机上使用什么类型的文件系统，该层次目录结构都是相同的。
|
||||
|
||||
### Linux 统一目录结构
|
||||
|
||||
在一些非 Linux 操作系统的个人电脑上，如果有多个物理硬盘驱动或多重分区，每一个硬盘或分区都会分配一个驱动器号。知道文件或程序位于哪一个硬盘驱动器上是很有必要的，比如 C: 或 D: 。然后，你可以在命令中使用驱动器号，以 `D:` 为例，为了进入 `D:` 驱动器，你可以使用 `cd` 命令来更改工作目录为正确的目录，从而定位需要的文件。每一个硬盘驱动器都有自己单独的、完整的目录树。
|
||||
|
||||
Linux 文件系统将所有物理硬盘驱动器和分区统一为一个目录结构。它们均从顶层 root 目录（/）开始。所有其他目录以及它们的子目录均位于单一的 Linux 根目录下。这意味着只有一棵目录树来搜索文件和程序。
|
||||
|
||||
因为只有一个文件系统，所以 /home、/tmp、/var、/opt 或 /usr 能够创建在和 root（/）文件系统不同的物理硬盘驱动器、分区或逻辑分区上，然后挂载到一个挂载点（目录）上，从而作为 root 文件系统树的一部分。甚至可移动驱动，比如 USB 驱动或一个外部 USB 驱动或 ESATA 硬盘驱动均可以挂载到 root 文件系统上，称为目录树不可或缺的部分。
|
||||
|
||||
当 Linux 发行版从一个版本升级到另一个版本或从一个版本更改到另一个版本的时候，就会很清楚地看到单文件系统的好处。通常情况下，除了任何像 Fedora 中的 dnf-upgrade 之类的升级工具，在升级过程中偶尔重新格式化包含操作系统的硬盘驱动来删除那些长期积累的垃圾是很明智的。如果 /home 目录是 root 文件系统的一部分（位于同一个硬盘驱动），那么它也会被格式化，然后通过之前的备份恢复。如果 /home 目录作为一个分离的文件系统，那么安装程序将会识别到，并跳过它的格式化。对于存储数据库、邮箱、网页和其他可变用户以及系统数据的 /var 目录也是同样的。
|
||||
|
||||
将 Linux 系统目录树的某些部分作为一个分离的文件系统还有一些其他原因。比如，在很久以前，我还不知道将所有需要的 Linux 目录均作为 root（/）文件系统的一部分可能存在的问题，于是，一些非常大的文件填满了 /home 目录。因为 /home 目录和 /tmp 目录均不是分离的文件系统，而是 root 文件系统的简单子目录，整个 root 文件系统就被填满了。于是就不再有剩余空间可以让操作系统用来存储临时文件或扩展已存在数据文件。首先，应用程序开始抱怨没有空间来保存文件，然后，操作系统也开始异常行动。启动到单用户模式，并清除了 /home 目录中的多余文件之后，终于又能够重新工作了。然后，我使用非常标准的多重文件系统设置来重新安装 Linux 系统，从而避免了系统崩溃的再次发生。
|
||||
|
||||
我曾经遇到一个情况，Linux 主机还在运行，但是却不运行用户通过 GUI 桌面登录。我可以通过局部使用[虚拟控制台][13]，通过命令行界面（CLI）登录，然后远程使用 SSH 。问题的原因是因为 /tmp 文件系统满了，因此 GUI 桌面登录时所需要的一些临时文件不能被创建。因为命令行界面登录不需要在 /tmp 目录中创建文件，所以无可用空间并不会阻止我使用命令行界面来登录。在这种情况下，/tmp 目录是一个分离的文件系统，在 /tmp 所位于的逻辑卷上还有大量的可用空间。我简单地[扩展了 /tmp 逻辑卷][14]的容量到能够容纳主机所需要的临时文件，于是问题便解决了。注意，这个解决方法不需要重启，当 /tmp 文件系统扩大以后，用户就可以登录到桌面了。
|
||||
|
||||
当我在一家很大的科技公司当实验室管理员的时候，遇到过另外一个故障。开发者将一个应用程序安装到了一个错误的位置（/var）。该应用程序崩溃了，因为 /var 文件系统满了，由于缺乏空间，当产生新消息的时候，存储于 /var/log 中的日志文件无法扩展。然而，系统仍然在运行，因为 root 文件系统和 /tmp 文件系统还没有被填满。删除了该应用程序并重新安装在 /opt 文件系统后，问题便解决了。
|
||||
|
||||
### 文件系统类型
|
||||
|
||||
Linux 系统支持大约 100 中分区类型的读取，但是只能对很少的一些进行创建和写操作。但是，挂载不同类型的文件系统在相同的 root 文件系统上是可能的，并且是很常见的。在这样的背景下，我们是根据在硬盘驱动器或逻辑卷上的一个分区中存储和管理用户数据所需要的结构和元数据来讨论文件系统的。能够被 Linux 系统的 `fdisk` 命令识别的文件系统类型的完整列表如下表所示，因此，你应该有这样的感觉，Linux 系统对许多类型的系统均有很高的兼容性。
|
||||
|
||||
```
|
||||
0 Empty 24 NEC DOS 81 Minix / old Lin bf Solaris 1 FAT12 27 Hidden NTFS Win 82 Linux swap / So c1 DRDOS/sec (FAT- 2 XENIX root 39 Plan 9 83 Linux c4 DRDOS/sec (FAT- 3 XENIX usr 3c PartitionMagic 84 OS/2 hidden or c6 DRDOS/sec (FAT- 4 FAT16 <32M 40 Venix 80286 85 Linux extended c7 Syrinx 5 Extended 41 PPC PReP Boot 86 NTFS volume set da Non-FS data 6 FAT16 42 SFS 87 NTFS volume set db CP/M / CTOS / . 7 HPFS/NTFS/exFAT 4d QNX4.x 88 Linux plaintext de Dell Utility 8 AIX 4e QNX4.x 2nd part 8e Linux LVM df BootIt 9 AIX bootable 4f QNX4.x 3rd part 93 Amoeba e1 DOS access a OS/2 Boot Manag 50 OnTrack DM 94 Amoeba BBT e3 DOS R/O b W95 FAT32 51 OnTrack DM6 Aux 9f BSD/OS e4 SpeedStor c W95 FAT32 (LBA) 52 CP/M a0 IBM Thinkpad hi ea Rufus alignment e W95 FAT16 (LBA) 53 OnTrack DM6 Aux a5 FreeBSD eb BeOS fs f W95 Ext'd (LBA) 54 OnTrackDM6 a6 OpenBSD ee GPT 10 OPUS 55 EZ-Drive a7 NeXTSTEP ef EFI (FAT-12/16/ 11 Hidden FAT12 56 Golden Bow a8 Darwin UFS f0 Linux/PA-RISC b 12 Compaq diagnost 5c Priam Edisk a9 NetBSD f1 SpeedStor 14 Hidden FAT16 <3 61 SpeedStor ab Darwin boot f4 SpeedStor 16 Hidden FAT16 63 GNU HURD or Sys af HFS / HFS+ f2 DOS secondary 17 Hidden HPFS/NTF 64 Novell Netware b7 BSDI fs fb VMware VMFS 18 AST SmartSleep 65 Novell Netware b8 BSDI swap fc VMware VMKCORE 1b Hidden W95 FAT3 70 DiskSecure Mult bb Boot Wizard hid fd Linux raid auto 1c Hidden W95 FAT3 75 PC/IX bc Acronis FAT32 L fe LANstep 1e Hidden W95 FAT1 80 Old Minix be Solaris boot ff BBT
|
||||
```
|
||||
|
||||
Linux 支持读这么多类型的分区系统的主要目的是为了提高兼容性，从而至少能够与一些其他计算机系统文件类型进行交互。下面列出了在 Fedora 中创建一个新的文件系统时的所有可选类型：
|
||||
|
||||
* btrfs
|
||||
|
||||
* **cramfs**
|
||||
|
||||
* **ext2**
|
||||
|
||||
* **ext3**
|
||||
|
||||
* **ext4**
|
||||
|
||||
* fat
|
||||
|
||||
* gfs2
|
||||
|
||||
* hfsplus
|
||||
|
||||
* minix
|
||||
|
||||
* **msdos**
|
||||
|
||||
* ntfs
|
||||
|
||||
* reiserfs
|
||||
|
||||
* **vfat**
|
||||
|
||||
* xfs
|
||||
|
||||
其他发行版支持创建不同的文件系统类型。比如，CentOS 6 只支持创建上表中标为黑体的文件系统类型。
|
||||
|
||||
### 挂载
|
||||
|
||||
在 Linux 系统上“挂载”文件系统的术语是指在计算机发展的早期，磁带或可移动的磁盘组需要需要物理地挂载到一个合适的驱动设备上。当通过物理的方式放置驱动以后，操作系统会逻辑地挂载位于磁盘上的文件系统，从而操作系统、应用程序和用户才能够访问文件系统中的内容。
|
||||
|
||||
一个挂载点就是一个简单的目录，就像任何其它目录一样，是作为 root 文件系统的一部分创建的。所以，比如，home 文件系统是挂载在目录 /home 下。文件系统可以被挂载到其他非 root 文件系统的挂载点上，但是这并不常见。
|
||||
|
||||
在 Linux 系统启动阶段的早期，root 文件系统就会被挂载到 root 目录下（/）。其他文件系统在之后通过 SystemV 下的 `rc` 或更新的 Linux 发行版中的 `systemd` 等 Linux 启动程序挂载。在启动进程中挂载文件系统是由 `/etc/fstab` 配置文件管理的。一个简单的记忆方法是，fstab 代表“文件系统表”，它包含了需要挂载的文件系统的列表，这些文件系统均指定了挂载点，以及针对特定文件系统可能需要的选项。
|
||||
|
||||
使用 `mount` 命令可以把文件系统挂载到一个存在的目录/挂载点上。通常情况下，任何作为挂载点的目录都应该是空的且不包含任何其他文件。Linux 系统会阻止用户挂载一个已被挂载的文件系统或将文件系统挂载到一个包含文件的目录上。如果你将文件系统挂载到一个已经存在的目录或文件系统上，那么原始内容将会被隐藏，只有新挂载的文件系统的内容是可见的。
|
||||
|
||||
### 结论
|
||||
|
||||
我希望通过这篇文章，围绕文件系统这个术语的一些可能的迷糊都被清除了。我花费了很长的时间，以及在一个良师的帮助下才真正理解和欣赏到 Linux 文件系统的复杂性、优雅性和功能以及它的全部含义。
|
||||
|
||||
如果你有任何问题，请写到下面的评论中，我会尽力来回答它们。
|
||||
|
||||
### 下个月
|
||||
|
||||
Linux 的另一个重要概念是：万物皆为文件。这个概念对用户和系统管理员来说有一些有趣和重要的实际应用。当我说完这个理由之后，你可能会想阅读我的文章：[万物皆为文件][15]，这篇文章是在我下个月计划写的关于 /dev 目录的文章之前。
|
||||
|
||||
-----------------
|
||||
|
||||
作者简介：
|
||||
|
||||
David Both 居住在美国北卡罗纳州的首府罗利，是一个 Linux 开源贡献者。他已经从事 IT 行业 40 余年，在 IBM 教授 OS/2 20 余年。1981 年，他在 IBM 开发了第一个关于最初的 IBM 个人电脑的培训课程。他也曾在 Red Hat 教授 RHCE 课程，也曾供职于 MCI worldcom，Cico 以及北卡罗纳州等。他已经为 Linux 开源社区工作近 20 年。
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://opensource.com/life/16/10/introduction-linux-filesystems
|
||||
|
||||
作者：[David Both][a]
|
||||
译者：[ucasFL](https://github.com/ucasFL)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://opensource.com/users/dboth
|
||||
[1]:https://opensource.com/resources/what-is-linux?intcmp=70160000000h1jYAAQ&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;utm_source=intcallout&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;utm_campaign=linuxcontent
|
||||
[2]:https://opensource.com/resources/what-are-linux-containers?intcmp=70160000000h1jYAAQ&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;utm_source=intcallout&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;utm_campaign=linuxcontent
|
||||
[3]:https://developers.redhat.com/promotions/linux-cheatsheet/?intcmp=70160000000h1jYAAQ&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;utm_source=intcallout&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;utm_campaign=linuxcontent
|
||||
[4]:https://developers.redhat.com/cheat-sheet/advanced-linux-commands-cheatsheet?intcmp=70160000000h1jYAAQ&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;utm_source=intcallout&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;utm_campaign=linuxcontent
|
||||
[5]:https://opensource.com/tags/linux?intcmp=70160000000h1jYAAQ&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;utm_source=intcallout&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;utm_campaign=linuxcontent
|
||||
[6]:https://opensource.com/life/16/10/introduction-linux-filesystems?rate=Qyf2jgkdgrj5_zfDwadBT8KsHZ2Gp5Be2_tF7R-s02Y
|
||||
[7]:https://opensource.com/users/dboth
|
||||
[8]:https://opensource.com/user/14106/feed
|
||||
[9]:https://creativecommons.org/licenses/by-sa/4.0/
|
||||
[10]:http://www.pathname.com/fhs/
|
||||
[11]:http://www.pathname.com/fhs/
|
||||
[12]:https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard
|
||||
[13]:https://en.wikipedia.org/wiki/Virtual_console
|
||||
[14]:https://opensource.com/business/16/9/linux-users-guide-lvm
|
||||
[15]:https://opensource.com/life/15/9/everything-is-a-file
|
||||
[16]:https://opensource.com/users/dboth
|
||||
[17]:https://opensource.com/users/dboth
|
||||
[18]:https://opensource.com/users/dboth
|
||||
[19]:https://opensource.com/life/16/10/introduction-linux-filesystems#comments
|
||||
[20]:https://opensource.com/tags/linux
|
||||
[21]:https://opensource.com/tags/sysadmin
|
||||
[22]:https://opensource.com/participate
|
||||
@@ -1,149 +0,0 @@
|
||||
# Docker Sawrm 模式 - 添加 worker 节点教程
|
||||
|
||||
让我们继续几周前在 CentOS 7.2 中开始的工作。 在本[指南][1]中，我们学习了如何初始化以及启动 Docker 1.12 中内置的本地集群以及编排功能。但是我们只有管理节点还没有其他 worker 节点。今天我们会展开这个。
|
||||
|
||||
我将向你展示如何将不对称节点添加到 Sawrm 中，也就是 [Fedora 24][2] 将与 CentOS 相邻，它们都将加入到集群中，还有相关很棒的负载均衡等等。当然这并不是微不足道的，我们会遇到一些障碍，所以它应该是非常有趣的。
|
||||
|
||||
|
||||
|
||||
### 先决条件
|
||||
|
||||
在将其他节点成功加入 Swarm 之前，我们需要做几件事情。理想情况下，所有节点都应该运行相同版本的 Docker，为了支持本地编排，它的版本至少应该为 1.12。像 CentOS 一样，Fedora 内置的仓库没有最新的构建，所以你需要手动或者使用 Docker 仓库手动[添加并安装][3]正确的版本，并修复一些依赖冲突。我已经向你展示了如何在 CentOS 中操作，练习是相同的。
|
||||
|
||||
此外，所有节点都需要能够相互通信。这就需要有正确的路由和防火墙规则，这样管理和 worker 节点才能互相通信。否则，你无法将节点加入 Swarm 中。最简单的解决方法是临时刷新防火墙规则 （iptables -F），但这可能会损害你的安全。请确保你完全了解你正在做什么，并为你的节点和端口创建正确的规则。
|
||||
|
||||
守护进程的错误响应：节点加入之前已超时。尝试加入 Swarm 的请求将在后台继续进行。使用 “docker info” 命令查看节点的当前 Swarm 状态。
|
||||
|
||||
你需要在主机上提供相同的 Docker 镜像。在上一个教程中我们创建了一个 Apache 映像，你需要在你的 worker 节点上执行相同操作，或者分发创建的镜像。如果你不这样做，你会遇到错误。如果你在设置 Docker 上需要帮助，请阅读我的[介绍指南][4]和[网络教程][5]。
|
||||
|
||||
```
|
||||
7vwdxioopmmfp3amlm0ulimcu \_ websky.11 my-apache2:latest
|
||||
localhost.localdomain Shutdown Rejected 7 minutes ago
|
||||
"No such image: my-apache2:lat&"
|
||||
```
|
||||
|
||||
### 现在开始
|
||||
|
||||
现在我们有一台 CentOS 机器并启动了，并成功创建了容器。你可以使用主机端口连接到服务，这一切都看起来很好。目前，你的 Swarm 只有管理节点。
|
||||
|
||||
|
||||
|
||||
### 加入 workers
|
||||
|
||||
要添加新的节点，你需要使用 join 命令。但是你首先必须提供令牌、IP 地址和端口，以便 woker 节点能正确地对 Swarm 管理器进行身份验证。接着执行（在 Fedora 上）：
|
||||
|
||||
```
|
||||
[root@localhost ~]# docker swarm join-token worker
|
||||
要将 worker 添加大这个 Swarm 中，运行下面的命令：
|
||||
|
||||
docker swarm join \
|
||||
--token SWMTKN-1-0xvojvlza90nrbihu6gfu3qm34ari7lwnza ... \
|
||||
192.168.2.100:2377
|
||||
```
|
||||
|
||||
如果你不修复防火墙和路由规则，你会得到超时错误。如果你已经加入了 Swarm，重复 join 命令会收到错误：
|
||||
|
||||
```
|
||||
Error response from daemon: This node is already part of a swarm. Use "docker swarm leave" to leave this swarm and join another one.
|
||||
```
|
||||
|
||||
如果有疑问，你可以离开 Swarm，然后重试：
|
||||
|
||||
```
|
||||
[root@localhost ~]# docker swarm leave
|
||||
Node left the swarm.
|
||||
|
||||
docker swarm join --token
|
||||
SWMTKN-1-0xvojvlza90nrbihu6gfu3qnza4 ... 192.168.2.100:2377
|
||||
This node joined a swarm as a worker.
|
||||
```
|
||||
|
||||
在 worker 节点中，你可以使用 “docker info” 来检查状态：
|
||||
|
||||
```
|
||||
Swarm: active
|
||||
NodeID: 2i27v3ce9qs2aq33nofaon20k
|
||||
Is Manager: false
|
||||
Node Address: 192.168.2.103
|
||||
|
||||
Likewise, on the manager:
|
||||
|
||||
Swarm: active
|
||||
NodeID: cneayene32jsb0t2inwfg5t5q
|
||||
Is Manager: true
|
||||
ClusterID: 8degfhtsi7xxucvi6dxvlx1n4
|
||||
Managers: 1
|
||||
Nodes: 3
|
||||
Orchestration:
|
||||
Task History Retention Limit: 5
|
||||
Raft:
|
||||
Snapshot Interval: 10000
|
||||
Heartbeat Tick: 1
|
||||
Election Tick: 3
|
||||
Dispatcher:
|
||||
Heartbeat Period: 5 seconds
|
||||
CA Configuration:
|
||||
Expiry Duration: 3 months
|
||||
Node Address: 192.168.2.100
|
||||
```
|
||||
|
||||
### 创建或缩放服务
|
||||
|
||||
现在，我们需要看下 Docker 是否以及如何在节点间分发容器。我的测试展示了一个在非常轻的负载下相当简单的平衡算法。试了一两次之后，即使在我尝试缩放并更新之后，Docker 也没有将运行的服务重新分配给新的 worker。同样，有一次，它在 worker 节点上创建了一个新的服务。也许这是最好的选择。
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
在新的 worker 节点上创建完整新的服务。
|
||||
|
||||
过了一段时间，两个容器之间的现有服务有一些重新分配，但这需要一些时间。新服务工作正常。这只是一个前期观察，所以我现在不能说更多。现在是开始探索和调整的新起点。
|
||||
|
||||
|
||||
|
||||
负载均衡过了一会工作了。
|
||||
|
||||
### 总结
|
||||
|
||||
Docker 是一只灵巧的小野兽，它只会继续扩大，更复杂，更强大，当然也更优雅。它被一个大企业吃掉只是一个时间问题。当它涉及本地编排时，Swarm 模式运行得很好，但是它不仅仅需要几个容器来充分利用其算法和可扩展性。
|
||||
|
||||
我的教程展示了如何将 Fedora 节点添加到由 CentOS 运行的群集中，并且两者能并行工作。关于负载平衡还有一些问题，但这是我将在以后的文章中探讨的。总而言之，我希望这是一个值得记住的教训。我们已经解决了在尝试设置 Swarm 时可能遇到的一些先决条件和常见问题，同时我们启动了一堆容器，我们甚至简要介绍了如何缩放和分发服务。要记住，这只是一个开始。
|
||||
|
||||
干杯。
|
||||
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
作者简介：
|
||||
|
||||
我是 Igor Ljubuncic。现在大约 38 岁，已婚但还没有孩子。我现在在一个大胆创新的云科技公司做首席工程师。直到大约 2015 年初，我还在一个全世界最大的 IT 公司之一中做系统架构工程师，和一个工程计算团队开发新的基于 Linux 的解决方案，优化内核以及攻克 Linux 的问题。在那之前，我是一个为高性能计算环境设计创新解决方案的团队的技术领导。还有一些其他花哨的头衔，包括系统专家、系统程序员等等。所有这些都曾是我的爱好，但从 2008 年开始成为了我的付费工作。还有什么比这更令人满意的呢？
|
||||
|
||||
从 2004 年到 2008 年间，我曾通过作为医学影像行业的物理学家来糊口。我的工作专长集中在解决问题和算法开发。为此，我广泛地使用了 Matlab，主要用于信号和图像处理。另外，我得到了几个主要的工程方法学的认证，包括 MEDIC 六西格玛绿带、试验设计以及统计工程学。
|
||||
|
||||
我也开始写书，包括奇幻类和 Linux 上的技术性工作。彼此交融。
|
||||
|
||||
要查看我开源项目、出版物和专利的完整列表，请滚动到下面。
|
||||
|
||||
有关我的奖项，提名和 IT 相关认证的完整列表，请稍等一下。
|
||||
|
||||
-------------
|
||||
|
||||
|
||||
via: http://www.dedoimedo.com/computers/docker-swarm-adding-worker-nodes.html
|
||||
|
||||
作者：[Igor Ljubuncic][a]
|
||||
译者：[geekpi](https://github.com/geekpi)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:http://www.dedoimedo.com/faq.html
|
||||
[1]:http://www.dedoimedo.com/computers/docker-swarm-intro.html
|
||||
[2]:http://www.dedoimedo.com/computers/fedora-24-gnome.html
|
||||
[3]:http://www.dedoimedo.com/computers/docker-centos-upgrade-latest.html
|
||||
[4]:http://www.dedoimedo.com/computers/docker-guide.html
|
||||
[5]:http://www.dedoimedo.com/computers/docker-networking.html
|
||||
@@ -0,0 +1,68 @@
|
||||
22 天内迁移到公共云
|
||||
============================================================
|
||||
|
||||
|
||||
>Lush's Oxford St, UK store. Credit: Lush.
|
||||
|
||||
### Lush 说这是可能的。
|
||||
|
||||
|
||||
在不到一个月内将你的核心业务从一个公共云迁移到另一个公共云看起来可能像一个遥远的目标，但是英国化妆品巨头 Lush 认为可以做到这一点。
|
||||
|
||||
去年九月 Lush - 你认为可能是糖果色、好闻的沐浴和护肤产品背后的公司- 与已有的基础设施供应商[位于英国的 Memset][5]的合同快要到期。
|
||||
|
||||
Memset 已经在亚马逊的网络服务中托管了 Lush 的基于 Drupal 商业环境好几年了，但零售商想要退出。
|
||||
|
||||
根据 Lush 的首席数字官和公司的继承人 Jack Constantine（他的父母在 1995 年成立该公司）的说法，安排是“尴尬”和僵硬的。
|
||||
|
||||
他今天在旧金山举行的 Google Cloud Next 会议上说：“我们在一份我们不太满意的合同中，我们想看看我们还能做些什么。”
|
||||
|
||||
“这是一个非常封闭的环境，这使我们很难看清下一步做什么。”
|
||||
|
||||
“(我们) 可以再注册一年，并有这样的承诺，想出一个有更多的控制权的长期计划，但是（我们）最终会挣扎。”
|
||||
|
||||
在淘遍市场后，Lush 目标放在 Google 的云平台上。该公司已经熟悉 Google，已于[ 2013 年底][6]从 Scalix 迁移到 Google Apps（现称为 G Suite）上。
|
||||
|
||||
然而，不得不在几个月不到的时间内进行迁移，一是在 12 月 22 日现有合同截止，二是在圣诞节购物的关键时期。
|
||||
|
||||
Constantine 说：“所以这不仅仅是一个关键的业务。我们在说高峰交易时间。这是一个巨大的交易。”
|
||||
|
||||
Lush 没有官僚主义转移意味着 Constantine 能够在选择供应商上快速决定。他说：“接着团队只要全力进行”
|
||||
|
||||
他们还优先优化专门用于迁移的“一体化” Drupal 程序，bug 推到以后再修复。
|
||||
|
||||
Lush 12 月 1 日开始物理迁移，12 月 22 日完成。
|
||||
|
||||
团队“像其他迁移一样”遇到了挑战，Constantine 说：“你不得不会担心将数据从一个地方传输到另一个地方，你必须确保一致性，客户，产品数据等需要稳定。”
|
||||
|
||||
但是，CDO 表示，让公司通过这个难以置信的紧张时间表是因为团队缺乏备选方案: 没有后备计划。
|
||||
|
||||
Constantine 说：“在截止日期前的一个星期，我的同事和我们的 Google 合作伙伴打了电话，他们对这是否会发生有点紧张，他们问我们 Plan B 是什么，我的同事说：Plan B 就是让 Plan A 发生，就是这样”。
|
||||
|
||||
“当你抛出这样一个听起来有点难以置信的艰难的截止日期时，但是（你们需要保持）关注那些认为这是我们可在这个时间范围内可以实现的目标的人，而不是那些放置阻碍说‘我们要延期’的人”。
|
||||
|
||||
“是的，每个人都很紧张，但你实现了很多。你实际上完成并盯牢了它。你所要做的就是完成、完成。”
|
||||
|
||||
现在的重点是将商业应用转移到微服务架构，同时研究各种 Google 工具，如 Kubernetes 容器管理系统和 Spanner 关系数据库。
|
||||
|
||||
Constantine 说，零售商最近也建立了使用 GCP 和 Android 的原型销售点系统。
|
||||
|
||||
Allie Coyne 作为 Google 的客人前往 Google Cloud Next
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://www.itnews.com.au/news/a-public-cloud-migration-in-22-days-454186
|
||||
|
||||
作者：[Allie Coyne ][a]
|
||||
译者：[geekpi](https://github.com/geekpi)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://www.itnews.com.au/author/allie-coyne-461593
|
||||
[1]:http://www.memset.com/about-us/case-studies/lush-cosmetics/
|
||||
[2]:https://cloud.googleblog.com/2013/12/google-apps-helps-eco-cosmetics-company.html
|
||||
[3]:http://www.memset.com/about-us/case-studies/lush-cosmetics/
|
||||
[4]:https://cloud.googleblog.com/2013/12/google-apps-helps-eco-cosmetics-company.html
|
||||
[5]:http://www.memset.com/about-us/case-studies/lush-cosmetics/
|
||||
[6]:https://cloud.googleblog.com/2013/12/google-apps-helps-eco-cosmetics-company.html
|
||||
@@ -1,75 +0,0 @@
|
||||
开发者定义的应用交付
|
||||
============================================================
|
||||
|
||||
负载均衡器如何帮助你管理分布式系统的复杂性。
|
||||
|
||||
|
||||
|
||||
|
||||
原生云应用旨在利用分布式系统的性能、可扩展性和可靠性优势。不幸的是，分布式系统往往以额外的复杂性为代价。由于你程序的各个组件分布在网络中，并且这些网络有通信障碍或者性能降级，因此你的分布式程序组件需要继续独立运行。
|
||||
|
||||
为了避免程序状态的不一致，分布式系统设计应该有一个共识，即组件会失效。没有什么比网络更突出。因此，在其核心，分布式系统在很大程度上依赖于负载平衡-跨两个或多个系统的请求分布，以便在面临网络中断和在系统负载波动时水平缩放时具有弹性。
|
||||
|
||||
Get O'Reilly's weekly Systems Engineering and Operations newsletter[
|
||||
|
||||
][5]
|
||||
|
||||
随着分布式系统在原生云程序的设计和交付中越来越普及，负载平衡器在现代应用程序体系结构的各个层次都浸透了基础结构设计。在常见配置中，负载平衡器部署在应用程序前面，处理来自外部世界的请求。然而，微服务的出现意味着负载平衡器在幕后发挥关键作用：即管理_服务_之间的流。
|
||||
|
||||
因此，当你使用原生云程序和分布式系统时，负载均衡器将承担其他角色：
|
||||
|
||||
* 作为提供缓存和增加安全性的反向代理，因为它成为外部客户端的中间件。
|
||||
* 作为通过提供协议转换（例如 REST 到 AMQP）的 API 网关。
|
||||
* 它可以处理安全性（即运行 Web 应用程序防火墙）。
|
||||
* 它可能承担应用程序管理任务，如速率限制和 HTTP/2 支持。
|
||||
|
||||
鉴于它们的扩展能力远大于平衡流量，负载平衡器可以更广泛地称为应用交付控制器（ADC）。
|
||||
|
||||
### 开发人员定义基础设施
|
||||
|
||||
从历史上看，ADC 是由 IT 专业人员购买、部署和管理的，最常见的是运行企业架构的应用程序。对于物理负载平衡器设备（如 F5、Citrix、Brocade等），这种情况在很大程度上仍然存在。具有分布式系统设计和临时基础结构的云原生应用要求负载平衡器与它们运行时的基础结构 (如容器) 一样具有动态特性。这些通常是软件负载均衡器（例如来自公共云提供商的 NGINX 和负载平衡器）。云原生应用通常是开发人员主导的计划，这意味着开发人员正在创建应用程序（例如微服务器）和基础设施（Kubernetes 和 NGINX）。开发人员越来越多地对负载平衡 (和其他) 基础结构的决策做出或产生大量影响。
|
||||
|
||||
作为决策者，云原生应用的开发人员通常不会意识到企业基础架构要求或现有部署的影响，同时考虑到这些部署通常是新的，并且经常在公共或私有云环境中进行部署。云技术将基础设施抽象为可编程 API，开发人员正在定义应用程序在该基础架构的每一层构建的方式。在有负载平衡器的情况下，开发人员会选择要使用的类型，部署方式以及启用哪些功能。它们以编程方式对负载平衡器的行为进行编码 - 随着程序在部署的生存期内增长、收缩和功能上进化时，它如何动态响应应用程序的需要。开发人员将基础结构定义为代码-包括基础结构配置和代码操作。
|
||||
|
||||
### 开发者为什么定义基础架构？
|
||||
|
||||
编写这个代码-_如何构建和部署应用程序_-的实践已经发生了根本性的转变，它体现在很多方面。令人遗憾的是，这种根本性的转变是由两个因素驱动的：将新的应用功能推向市场（_上市时间_）所需的时间以及应用用户从产品（_时间到价值_）中获得价值所需的时间。因此，新的程序写出来被持续地交付（作为服务），没有下载和安装。
|
||||
|
||||
上市时间和时间价值的压力并不是新的，但由于其他因素的加剧，这些因素正在加强开发者的决策权力：
|
||||
|
||||
* 云：通过 API 定义基础架构作为代码的能力。
|
||||
* 伸缩：需要在大型环境中高效运行操作。
|
||||
* 速度：马上需要交付应用功能，为企业争取竞争力。
|
||||
* 微服务：抽象框架和工具选择，进一步赋予开发人员基础架构决策权力。
|
||||
|
||||
除了上述因素外，值得注意的是开源的影响。随着开源软件的普及和发展，开发人员掌握了许多应用程序基础设施 - 语言、运行时、框架、数据库、负载均衡器、托管服务等。微服务的兴起使应用程序基础设施的选择民主化，允许开发人员选择最佳的工具。在选择负载平衡器的情况下，与云原生应用的动态性质紧密集成并响应的那些应用程序将上升到最高。
|
||||
|
||||
### 总结
|
||||
|
||||
当你在仔细考虑你的云原生应用设计时，请与我一起讨论_[在云中使用 NGINX 和 Kubernetes 进行负载平衡][8]_。我们将检测不同公共云和容器平台的负载平衡功能，并通过一个宏应用的案例研究。我们将看看它是如何被变成成较小的，独立的服务，以及 NGINX 和 Kubernetes 的能力如何拯救它的。
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
作者简介：
|
||||
|
||||
Lee Calcote 是一位创新的思想领袖，对开发者平台和云、容器、基础设施和应用的管理软件充满热情。先进的和新兴的技术一直是 Calcote 在 SolarWinds、Seagate、Cisco 和 Pelco 时的关注重点。技术会议和聚会的组织者、写作者、作家、演讲者，他活跃在技术社区。
|
||||
|
||||
----------------------------
|
||||
|
||||
via: https://www.oreilly.com/learning/developer-defined-application-delivery
|
||||
|
||||
作者：[Lee Calcote][a]
|
||||
译者：[geekpi](https://github.com/geekpi)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://www.oreilly.com/people/7f693-lee-calcote
|
||||
[1]:https://pixabay.com/en/ship-containers-products-shipping-84139/
|
||||
[2]:https://conferences.oreilly.com/velocity/vl-ca?intcmp=il-webops-confreg-na-vlca17_new_site_velocity_sj_17_cta
|
||||
[3]:https://www.oreilly.com/people/7f693-lee-calcote
|
||||
[4]:http://www.oreilly.com/pub/e/3864?intcmp=il-webops-webcast-reg-webcast_new_site_developer_defined_application_delivery_text_cta
|
||||
[5]:https://www.oreilly.com/learning/developer-defined-application-delivery?imm_mid=0ee8c5&cmp=em-webops-na-na-newsltr_20170310
|
||||
[6]:https://conferences.oreilly.com/velocity/vl-ca?intcmp=il-webops-confreg-na-vlca17_new_site_velocity_sj_17_cta
|
||||
[7]:https://conferences.oreilly.com/velocity/vl-ca?intcmp=il-webops-confreg-na-vlca17_new_site_velocity_sj_17_cta
|
||||
[8]:http://www.oreilly.com/pub/e/3864?intcmp=il-webops-webcast-reg-webcast_new_site_developer_defined_application_delivery_body_text_cta
|
||||
@@ -0,0 +1,453 @@
|
||||
从 Node 到 Go：深层比较
|
||||
============================================================
|
||||
|
||||
在 XO 公司，我们最初使用 Node 和 Ruby 构建互联服务系统。我们享受 Node 带来的明显性能优势，以及可以访问已有的大型软件包仓库。我们也可以轻松地在公司内部发布并复用已有的插件和模块。极大地提高了开发效率，使得我们可以快速编写可拓展和可靠的应用。庞大的 Node 社区使我们的工程师向开源软件贡献更加容易。(比如 [BunnyBus][9] 和 [Felicity][10])
|
||||
|
||||
虽然我在大学时期和刚刚工作的一些时间在使用更严格的编译语言，比如 C++ 和 C#，后来我开始使用 JavaScript。我很喜欢它的自由和灵活，但是我最近开始怀念静态和结构化的语言。因为当时有一个同事让我对 Go 语言产生了兴趣。
|
||||
|
||||
我从写 JavaScript 到写 Go，我发现两种语言有很多相似之处。两者速度都很快并且易于上手，都具有充满表现力的语法，并且在开发者社区中都有很多工作机会。没有完美的编程语言，所以你应该总是选择一个适合手头项目的语言。我将要在这篇文章中说明这两种语言深层次上的关键区别，希望能鼓励没有用过 Go 语言的用户使用 Go 。
|
||||
|
||||
* * *
|
||||
|
||||
### 大体上的差异
|
||||
|
||||
在深入细节之前，我们应该先了解一下两种语言之间的重要区别。
|
||||
|
||||
Go，或称 Golang，是 Google 在 2007 年创建的免费开源编程语言。它被设计成快速和简单的。Go 被直接编译成机器码，这就是它速度的来源。使用编译语言调试是相当容易的，因为你可以在早期捕获大量错误。Go 也是一种强类型的语言，它有助于数据完整，并可以在编译时查找类型错误。
|
||||
|
||||
另一方面，JavaScript 是一种弱类型语言。忽略验证数据的类型和真值判断陷阱带来的额外负担外，使用弱类型语言也有自己的好处。比起使用接口和范型，柯里化和可变的形参个数让函数变得更加灵活。JavaScript 在运行时被解释，这可能导致错误处理和调试的问题。Node 是一款基于 Google V8 虚拟机的 JavaScript 运行库，这使它成为一个轻量和快速的 Web 开发平台。
|
||||
|
||||
* * *
|
||||
|
||||
### 语法
|
||||
|
||||
作为原来的 JavaScript 开发者，Go 简单和直观的语法很吸引我。由于两种语言的语法都是从 C 语言演变的，所以它们的语法有很多相同之处。Go 被普遍认为是一种容易学习的语言。那时因为它的对开发者友好的工具，精简的语法和自以为是的约定（存疑）。
|
||||
|
||||
Go 包含大量的自带特性有助于简化开发。你可以用标准 Go 构建工具把你的程序用 go build 命令编译成二进制可执行文件。使用内置的测试套件运行测试只需要运行 go test。 诸如原生支持的并发等特性甚至在语言层面上提供。
|
||||
|
||||
[Google 的 Go 开发者][11]认为，现在的编程太复杂了，“太多的记账一样，重复劳动和文书工作”。这就是为什么 Go 的语法被设计得简单和干净，以减少混乱，提高效率和增强可读性。它还鼓励开发人员编写明确，易于理解的代码。Go 只有 [25 个唯一关键字][12]和唯一一种循环（for 循环），而不像 JavaScript 有 [ ~84 个关键字][13]（包括保留字，对象，属性和方法）。
|
||||
|
||||
为了说明语法的一些差异和相似之处，我们来看几个例子：
|
||||
|
||||
* 标点符号： Go 去除了所有多余的符号以提高效率和可读性。尽管 JavaScript 中需要符号的地方也不多（见: [Lisp][1]），而且经常是可选的，但我更加喜欢 Go 的简单。
|
||||
|
||||
```
|
||||
// JavaScript 的逗号和分号
|
||||
for (var i = 0; i < 10; i++) {
|
||||
console.log(i);
|
||||
}
|
||||
```
|
||||
|
||||
JavaScript 中的标点
|
||||
|
||||
```
|
||||
// Go uses minimal punctuation 使用最少数量标点的 Go
|
||||
for i := 0; i < 10; i++ {
|
||||
fmt.Println(i)
|
||||
}
|
||||
```
|
||||
Go 中的标点
|
||||
|
||||
* 赋值：由于 Go 是强类型语言，所以你在初始化变量时可以使用 := 操作符来进行类型推断，以避免[重复声明][2]，而 JavaScript 则在运行时声明类型。
|
||||
|
||||
|
||||
```
|
||||
// Javascript assignment JavaScript 赋值
|
||||
var foo = "bar";
|
||||
```
|
||||
|
||||
JavaScript 中的赋值
|
||||
|
||||
```
|
||||
// Go 的赋值
|
||||
var foo string //不使用类型推导
|
||||
foo = "bar"
|
||||
|
||||
foo := "bar" //使用类型推导
|
||||
```
|
||||
|
||||
Go 的赋值
|
||||
|
||||
* 导出：在 JavaScript 中，你必须从某个模块中显式地导出。 在 Go 中，任何大写的函数将被默认导出。
|
||||
|
||||
|
||||
```
|
||||
const Bar = () => {};
|
||||
|
||||
module.exports = {
|
||||
Bar
|
||||
}
|
||||
```
|
||||
|
||||
JavaScript 中的导出
|
||||
|
||||
```
|
||||
// Go 中的导出
|
||||
package foo //define package name 定义包名
|
||||
func Bar (s string) string {
|
||||
//Bar will be exported Bar 将被导出为 Bar
|
||||
}
|
||||
```
|
||||
|
||||
Go 中的导出
|
||||
|
||||
* 导入：在 JavaScript 中被导入的依赖项和模块所依赖的库是必需的，而 Go 则利用原生的 import 关键字通过包的路径导入模块。另一个区别是，与Node的中央NPM存储库不同，Go使用URL作为路径来导入非标准库的包，这是为了从包的源码仓库直接克隆依赖。
|
||||
```
|
||||
// Javascript 的导入
|
||||
var foo = require('foo');
|
||||
foo.bar();
|
||||
```
|
||||
|
||||
JavaScript 的导入
|
||||
|
||||
|
||||
```
|
||||
// Go 的导入
|
||||
import (
|
||||
"fmt" // part of Go’s standard library Go 的标准库部分
|
||||
"github.com/foo/foo" // imported directly from repository 直接从仓库导入
|
||||
)
|
||||
foo.Bar()
|
||||
```
|
||||
|
||||
Go 的导入
|
||||
|
||||
* 返回多值：通过 Go 的多值返回特性可以优雅地传递和处理返回值和错误，并且通过传递引用代替不正确的值传递。在 JavaScript 中需要通过一个对象或者数组来返回多个值。
|
||||
|
||||
```
|
||||
// Javascript - 返回多值
|
||||
function foo() {
|
||||
return {a: 1, b: 2};
|
||||
}
|
||||
const { a, b } = foo();
|
||||
```
|
||||
|
||||
JavaScript 的返回
|
||||
|
||||
```
|
||||
// Go - 返回多值
|
||||
func foo() (int, int) {
|
||||
return 1, 2
|
||||
}
|
||||
a, b := foo()
|
||||
```
|
||||
|
||||
Go 的返回
|
||||
|
||||
|
||||
* 错误处理：Go 推荐在错误出现的地方捕获它们，而不是像 Node 一样在回调中让错误冒泡。
|
||||
|
||||
```
|
||||
// Node 的错误处理
|
||||
foo('bar', function(err, data) {
|
||||
//handle error
|
||||
}
|
||||
```
|
||||
|
||||
JavaScript 的错误处理
|
||||
|
||||
|
||||
```
|
||||
//Go 的错误处理
|
||||
foo, err := bar()
|
||||
if err != nil {
|
||||
// handle error with defer, panic, recover, or log.fatal, etc...
|
||||
}
|
||||
```
|
||||
|
||||
Go 的错误处理
|
||||
|
||||
* 可变参数函数：Go 和 JavaScript 的函数都支持传入不定数量的参数。
|
||||
|
||||
```
|
||||
function foo (...args) {
|
||||
console.log(args.length);
|
||||
}
|
||||
|
||||
foo(); // 0
|
||||
foo(1, 2, 3); // 3
|
||||
```
|
||||
|
||||
JavaScript 中的可变参数函数
|
||||
|
||||
```
|
||||
func foo (args ...int) {
|
||||
fmt.Println(len(args))
|
||||
}
|
||||
|
||||
func main() {
|
||||
foo() // 0
|
||||
foo(1,2,3) // 3
|
||||
}
|
||||
```
|
||||
|
||||
Go 中的可变参数函数
|
||||
|
||||
* * *
|
||||
|
||||
### 社区
|
||||
|
||||
当比较 Go 和 Node 提供的编程范式哪种更方便时，两边都有不同的拥护者。Node 在软件包数量和社区的大小上完全胜过了 Go。Node 包管理器（NPM），世界上最大的软件仓库，拥有[超过 410,000 个软件包，每天以 555 个新软件包的惊人速度增长][14]。这个数字可能看起来令人吃惊（确实是），但是需要注意的是，这些包许多是重复的和/或质量不足以在生产环境。 相比之下，Go 大约有 13 万个包。
|
||||
|
||||
|
||||
|
||||
Node 和 Go 包的数量
|
||||
|
||||
尽管 Node 和 Go 岁数相仿，JavaScript 使用更加广泛，并拥有巨大的开发者和开源社区。因为 Node 是为所有人开发的并在开始的时候就带有一个强壮的包管理器而 Go 是特地为 Google 开发的。下面的[排行榜][15]显示了当前流行的的顶尖 Web 开发语言。
|
||||
|
||||
|
||||
|
||||
Web 开发语言排行榜前 7 名
|
||||
|
||||
JavaScript 的受欢迎程度近年来似乎保持相对稳定，而 [Go 一直在保持上升趋势][16]。
|
||||
|
||||
|
||||
|
||||
|
||||
Programming language trends 编程语言趋势
|
||||
|
||||
* * *
|
||||
|
||||
### 性能
|
||||
|
||||
如果你的主要关注点是速度呢？当今似乎人们比以前更重视性能的优化。用户不喜欢等待信息。 事实上，如果网页的加载时间超过 3 秒，[40％ 的用户会放弃访问您的网站][17]。
|
||||
|
||||
Node 经常被认为是高性能的语言，因为它的非阻塞异步 I/O。另外，正如我之前提到的，Node 运行在针对动态语言进行了优化的 Google V8 引擎上。而 Go 的设计也考虑到速度。[Google 的开发者][18]也达成了这一目标，通过建立了一个“充满表现力而轻量级的类型系统，并发和垃圾回收机制，强制地指定依赖版本等等”。
|
||||
|
||||
我运行了一些测试来比较 Node 和 Go 之间的性能。这些测试注重于语言提供的初级能力。如果我准备测试例如 HTTP 请求 或者 CPU 密集型运算，我会使用 Go 语言级别的并发工具(goroutines/channels)。但是我更注重于各个语言提供的基本特性(看 [三种并发方法][19] 了解关于 goroutines 和 channels 的更多知识)。
|
||||
|
||||
我在基准测试中也加入了 Python，所以无论如何我们对 Node 和 Go 的结果都很满意。
|
||||
|
||||
#### 循环/算术
|
||||
|
||||
迭代十亿项并把它们相加：
|
||||
|
||||
```
|
||||
var r = 0;
|
||||
for (var c = 0; c < 1000000000; c++) {
|
||||
r += c;
|
||||
}
|
||||
```
|
||||
|
||||
Node
|
||||
|
||||
```
|
||||
package main
|
||||
func main() {
|
||||
var r int
|
||||
for c := 0; c < 1000000000; c++ {
|
||||
r += c
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
go
|
||||
|
||||
|
||||
```
|
||||
sum(xrange(1000000000))
|
||||
```
|
||||
|
||||
Python
|
||||
|
||||
|
||||
|
||||
结果
|
||||
|
||||
这里的输家无疑是 Python，花了超过 7 秒的 cpu 时间。而 Node 和 Go 都相当高效，分别用了 900 ms 和 408 ms。
|
||||
|
||||
修正：由于一些评论表明 Python 的性能还可以提高。我更新了结果来反映这些变化。同时，使用 PyPy 大大地提高了性能。当使用 Python 3.6.1 和 PyPy 3.5.7 运行时，性能提升到 1.234 秒，但仍然不及 Go 和 Node 。_
|
||||
|
||||
#### I/O
|
||||
|
||||
遍历一百万个数字并将其写入一个文件。
|
||||
|
||||
|
||||
```
|
||||
var fs = require('fs');
|
||||
var wstream = fs.createWriteStream('node');
|
||||
|
||||
for (var c = 0; c < 1000000; ++c) {
|
||||
wstream.write(c.toString());
|
||||
}
|
||||
wstream.end();
|
||||
```
|
||||
|
||||
Node
|
||||
|
||||
```
|
||||
package main
|
||||
|
||||
import (
|
||||
"io"
|
||||
"os"
|
||||
"strconv"
|
||||
)
|
||||
|
||||
func main() {
|
||||
file, _ := os.Create("go")
|
||||
for c := 0; c < 1000000; c++ {
|
||||
num := strconv.Itoa(c)
|
||||
io.WriteString(file, num)
|
||||
}
|
||||
file.Close()
|
||||
}
|
||||
```
|
||||
|
||||
go
|
||||
|
||||
```
|
||||
with open("python", "a") as text_file:
|
||||
for i in range(1000000):
|
||||
text_file.write(str(i))
|
||||
```
|
||||
|
||||
Python
|
||||
|
||||
|
||||
|
||||
结果
|
||||
|
||||
Python 以 7.94 秒再次排名第三。 这次测试中，Node 和 Go 之间的差距很小，Node 花费大约 1.164 秒，Go 花费了1.477秒（尽管这包括了编译 Go 的代码所花费的时间—使用编译好的文件将缩短 ~200 ms 的整体时间）。
|
||||
|
||||
#### 冒泡排序
|
||||
|
||||
将含有十个元素的数组排序一千万次
|
||||
|
||||
```
|
||||
const toBeSorted = [1, 3, 2, 4, 8, 6, 7, 2, 3, 0];
|
||||
|
||||
function bubbleSort(input) {
|
||||
var n = input.length;
|
||||
var swapped = true;
|
||||
while (swapped) {
|
||||
swapped = false;
|
||||
for (var i = 0; i < n; i++) {
|
||||
if (input[i - 1] > input [i]) {
|
||||
[input[i], input[i - 1]] = [input[i - 1], input[i]];
|
||||
swapped = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (var c = 0; c < 10000000; c++) {
|
||||
bubbleSort(toBeSorted);
|
||||
}
|
||||
```
|
||||
|
||||
Node
|
||||
|
||||
```
|
||||
package main
|
||||
|
||||
var toBeSorted [10]int = [10]int{1, 3, 2, 4, 8, 6, 7, 2, 3, 0}
|
||||
|
||||
func bubbleSort(input [10]int) {
|
||||
n := len(input)
|
||||
swapped := true
|
||||
for swapped {
|
||||
swapped = false
|
||||
for i := 1; i < n; i++ {
|
||||
if input[i-1] > input[i] {
|
||||
input[i], input[i-1] = input[i-1], input[i]
|
||||
swapped = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func main() {
|
||||
for c := 0; c < 10000000; c++ {
|
||||
bubbleSort(toBeSorted)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
go
|
||||
|
||||
|
||||
```
|
||||
toBeSorted = [1, 3, 2, 4, 8, 6, 7, 2, 3, 0]
|
||||
|
||||
def bubbleSort(input):
|
||||
length = len(input)
|
||||
swapped = True
|
||||
|
||||
while swapped:
|
||||
swapped = False
|
||||
for i in range(1,length):
|
||||
if input[i - 1] > input[i]:
|
||||
input[i], input[i - 1] = input[i - 1], input[i]
|
||||
swapped = True
|
||||
|
||||
for i in range(10000000):
|
||||
bubbleSort(toBeSorted)
|
||||
```
|
||||
Python
|
||||
|
||||
|
||||
|
||||
|
||||
<figcaption class="imageCaption" style="position: relative; left: 0px; width: 700px; top: 0px; margin-top: 10px; color: rgba(0, 0, 0, 0.6); outline: 0px; text-align: center; z-index: 300; --baseline-multiplier:0.157; font-family: medium-content-sans-serif-font, "Lucida Grande", "Lucida Sans Unicode", "Lucida Sans", Geneva, Arial, sans-serif; font-feature-settings: 'liga' 1, 'lnum' 1; font-size: 14px; line-height: 1.4; letter-spacing: 0px;">Results</figcaption>
|
||||
|
||||
像刚才一样，Python 的表现是最差的，大约花费 13 秒完成了任务。 Go 完成任务的速度是 Node 的两倍多。
|
||||
|
||||
#### 判决
|
||||
|
||||
Go 无疑是这三个测试中的赢家，而 Node 大部分表现都很出色。Python 也表现不错。要清楚，性能不是选择编程语言需要考虑的全部内容。如果您的应用不需要处理大量数据，那么 Node 和 Go 之间的性能差异可能是微不足道的。 有关性能的一些其他比较，请参阅以下内容：
|
||||
|
||||
* [Node Vs. Go][3]
|
||||
* [Multiple Language Performance Test][4]
|
||||
* [Benchmarks Game][5]
|
||||
|
||||
|
||||
* * *
|
||||
|
||||
### 结论
|
||||
|
||||
这个帖子不是为了证明一种语言比另一种语言更好。由于各种原因，每种编程语言都在软件开发社区中占有一席之地。 我的意图是强调 Go 和 Node 之间的差异，并且促进展示一种新的 Web 开发语言。 在为一个项目选择语言时，有各种因素需要考虑，比如开发人员的熟悉程度，花费和实用性。 我鼓励在决定哪种语言适合您时进行一次彻底的底层分析。
|
||||
|
||||
正如我们所看到的，Go 有如下的优点：接近底层语言的性能，简单的语法和相对简单的学习曲线使它成为构建可拓展和安全的 Web 应用的理想选择。随着 Go 的使用率和社区活动的快速增长，它将会成为现代网络开发中的重要角色。话虽如此，我相信如果 Node 被正确地实现，它正在向正确的方向努力，仍然是一种强大而有用的语言。它具有大量的追随者和活跃的社区，使其成为一个简单的平台，可以让 Web 应用在任何时候启动和运行。
|
||||
|
||||
* * *
|
||||
|
||||
### 资料
|
||||
|
||||
如果你对学习 Go 语言感兴趣，可以参阅下面的资源：
|
||||
|
||||
* [Golang 网站][6]
|
||||
* [Golang Wiki][7]
|
||||
* [Golang Subreddit][8]
|
||||
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://medium.com/xo-tech/from-node-to-go-a-high-level-comparison-56c8b717324a#.byltlz535
|
||||
|
||||
作者：[John Stamatakos][a]
|
||||
译者：[trnhoe](https://github.com/trnhoe)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://medium.com/@johnstamatakos?source=post_header_lockup
|
||||
[1]:https://en.wikipedia.org/wiki/Lisp_%28programming_language%29
|
||||
[2]:https://golang.org/doc/faq#principles
|
||||
[3]:https://jaxbot.me/articles/node-vs-go-2014
|
||||
[4]:https://hashnode.com/post/comparison-nodejs-php-c-go-python-and-ruby-cio352ydg000ym253frmfnt70
|
||||
[5]:https://benchmarksgame.alioth.debian.org/u64q/compare.php?lang=go&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;lang2=node
|
||||
[6]:https://golang.org/doc/#learning
|
||||
[7]:https://github.com/golang/go/wiki/Learn
|
||||
[8]:https://www.reddit.com/r/golang/
|
||||
[9]:https://medium.com/xo-tech/bunnybus-building-a-data-transit-system-b9647f6283e5#.l64fdvfys
|
||||
[10]:https://medium.com/xo-tech/introducing-felicity-7b6d0b734ce#.hmloiiyx8
|
||||
[11]:https://golang.org/doc/faq
|
||||
[12]:https://golang.org/ref/spec#Keywords
|
||||
[13]:https://www.w3schools.com/js/js_reserved.asp
|
||||
[14]:http://www.modulecounts.com/
|
||||
[15]:http://spectrum.ieee.org/static/interactive-the-top-programming-languages-2016
|
||||
[16]:http://www.tiobe.com/tiobe-index/
|
||||
[17]:https://hostingfacts.com/internet-facts-stats-2016/
|
||||
[18]:https://golang.org/doc/faq
|
||||
[19]:https://medium.com/xo-tech/concurrency-in-three-flavors-51ed709876fb#.khvqrttxa
|
||||
@@ -1,153 +0,0 @@
|
||||
Curl 中的更少的 Malloc
|
||||
===========================================================
|
||||
|
||||
|
||||
|
||||
今天我在 libcurl 内部又做了[一个小变化][4]，使其做更少的 malloc。这一次，泛型链表函数被转换成更少的 malloc (链表函数应该表现的方式，真的)。
|
||||
|
||||
### 研究 malloc
|
||||
|
||||
几周前我开始研究内存分配。这很容易，因为我们 curl 中已经有多年的内存调试和日志记录系统。使用 curl 的调试版本，并在我的构建目录中运行此脚本：
|
||||
|
||||
```
|
||||
#!/bin/sh
|
||||
export CURL_MEMDEBUG=$HOME/tmp/curlmem.log
|
||||
./src/curl http://localhost
|
||||
./tests/memanalyze.pl -v $HOME/tmp/curlmem.log
|
||||
```
|
||||
|
||||
对于 curl 7.53.1，这大约有 115 次内存分配。这算多还是少？
|
||||
|
||||
内存日志非常基本。为了让你了解，这是一个示例代码片段：
|
||||
|
||||
```
|
||||
MEM getinfo.c:70 free((nil))
|
||||
MEM getinfo.c:73 free((nil))
|
||||
MEM url.c:294 free((nil))
|
||||
MEM url.c:297 strdup(0x559e7150d616) (24) = 0x559e73760f98
|
||||
MEM url.c:294 free((nil))
|
||||
MEM url.c:297 strdup(0x559e7150d62e) (22) = 0x559e73760fc8
|
||||
MEM multi.c:302 calloc(1,480) = 0x559e73760ff8
|
||||
MEM hash.c:75 malloc(224) = 0x559e737611f8
|
||||
MEM hash.c:75 malloc(29152) = 0x559e737a2bc8
|
||||
MEM hash.c:75 malloc(3104) = 0x559e737a9dc8
|
||||
```
|
||||
|
||||
### 检查日志
|
||||
|
||||
然后，我对日志进行了更深入的研究，我意识到在相同的代码行做了许多小内存分配。我们显然有一些相当愚蠢的代码模式，我们将分配一个结构体，然后将该结构添加到链表或哈希，然后该代码随后再添加另一个小结构体和类似的代码，然后经常在循环中执行。（我在这里说的是_我们_，以避免责怪任何人，但当然我自己也有大部分的责任。。。）
|
||||
|
||||
这两个分配将总是成对发生，同时它们将被释放。我决定解决这些问题。做非常小的（小于 32 字节）的分配也是浪费的，因为非常多的数据将被用于跟踪那个微小的内存区域（在 malloc 系统内）。更不用说堆碎片了。
|
||||
|
||||
因此，将哈希和链表代码修复为不使用 malloc 是快速且简单的方法，它可以清除 20％ 以上的 malloc，以便进行简单的 “curl http://localhost” 传输。
|
||||
|
||||
此时，我根据大小对所有分配进行排序，并检查所有最小的分配。一个突出的部分是在 _curl_multi_wait()_ 中，它是一个典型的在 curl
|
||||
传输主循环中被反复调用的函数。对于大多数典型情况，我将其转换为[使用堆栈][5]。在大量重复的调用函数中避免 malloc 是一件好事。
|
||||
|
||||
### 重新计数
|
||||
|
||||
现在，如上面的脚本所示，相同的 “curl localhost” 命令从 curl 7.53.1 的 115 次分配下降到 80 个分配而没有牺牲任何东西。轻松地有 26％ 的改善。一点也不差！
|
||||
|
||||
由于我修改了 curl_multi_wait()，我也想看看它实际上是如何改进一些稍微更高级的传输。我使用了 [multi-double.c][6] 示例代码，添加了初始内存记录的调用，让它使用 curl_multi_wait()，并并行下载了这两个 URL：
|
||||
|
||||
```
|
||||
http://www.example.com/
|
||||
http://localhost/512M
|
||||
```
|
||||
|
||||
第二个是 512 兆字节的零，第一个是一个 600 字节的公共 html 页面。这是[ count-malloc.c 代码][7]。
|
||||
|
||||
首先，我使用 7.53.1 来测试上面的例子，并使用 memaalyze 脚本检查：
|
||||
|
||||
```
|
||||
Mallocs: 33901
|
||||
Reallocs: 5
|
||||
Callocs: 24
|
||||
Strdups: 31
|
||||
Wcsdups: 0
|
||||
Frees: 33956
|
||||
Allocations: 33961
|
||||
Maximum allocated: 160385
|
||||
```
|
||||
|
||||
好的，所以它总共使用了 160KB 的内存，分配次数超过 33,900 次。但是，它下载超过 512 兆字节的数据，所以它每 15KB 数据有一次 malloc。是好是坏？
|
||||
|
||||
回到 git master，现在是 7.54.1-DEV 的版本 - 因为我们不太确定当我们发布下一个版本时会变成哪个版本号。它可能是 7.54.1 或 7.55.0，它还尚未确定。我离题了，我再次运行相同修改的 multi-double.c 示例，再次在内存日志中运行 memaalyze，报告来了：
|
||||
|
||||
```
|
||||
Mallocs: 69
|
||||
Reallocs: 5
|
||||
Callocs: 24
|
||||
Strdups: 31
|
||||
Wcsdups: 0
|
||||
Frees: 124
|
||||
Allocations: 129
|
||||
Maximum allocated: 153247
|
||||
```
|
||||
|
||||
我不得不看了两次。我做错什么了吗？为了仔细检查，我最好再运行一次。无论我运行多少次，结果还是一样的。
|
||||
|
||||
### 33,961 vs 129
|
||||
|
||||
在典型的传输中被调用了很多次，并且在传输过程中至少有一次进行内存分配，因此删除那个单一的微小分配对计数器有非常大的影响。正常的传输也会将数据添加或移移出链表并进行散列，但是它们现在也大都是无 malloc 的。简单地说：剩余的分配不会在传输循环中完成，所以它们的重要性不大。
|
||||
|
||||
以前的 curl 是当前对此示例分配数量的 263 倍。换句话说：新的是旧的分配数量的 0.37％
|
||||
|
||||
另外还有一点，新的内存总共减少了 7KB（4.3％）。
|
||||
|
||||
### malloc 重要吗？
|
||||
|
||||
在几千兆字节的 RAM 时代里，在传输中有几个 mallocs 真的对于普通人有显著的区别吗？对 512MB 数据完成的 33,832 个额外的 malloc 有什么影响？
|
||||
|
||||
为了衡量这些变化的影响，我决定比较本地主机的 HTTP 传输，看看是否可以看到任何速度差异。localhost 对于这个测试是很好的，因为没有网络速度限制，但更快的 curl 下载也越快。服务器端也会相同的快/慢，因为我将使用相同的集合进行这两个测试。
|
||||
|
||||
我相同方式构建了 curl 7.53.1 和 curl 7.54.1-DEV，并运行这个命令：
|
||||
|
||||
```
|
||||
curl http://localhost/80GB -o /dev/null
|
||||
```
|
||||
|
||||
它尽快地会将 80GB 的数据写到空设备中。
|
||||
|
||||
我获得的确切数字可能不是很有趣，因为它将取决于机器中的 CPU、使用的 HTTP 服务器、构建 curl 时的优化级别等，但是相关数字仍然应该是高度相关的。新代码 vs 旧代码。
|
||||
|
||||
7.54.1-DEV 反复表现更快 30％！我的早期版本是 2200MB/秒增加到当前版本的超过 2900 MB /秒。
|
||||
|
||||
这里要点当然不是说它很容易在我的机器上使用单一内核以超过 20GB/秒的速度传输 HTTP，因为实际上很少有用户可以通过 curl 快速传输。关键在于 curl 现在传输每字节使用较少的 CPU，这将使更多的 CPU 转移到系统的其余部分来执行任何需要做的事情。或者如果设备是便携式设备，那么可以省电。
|
||||
|
||||
关于 malloc 的成本：512MB 测试中我使用旧代码发生了 33832 次或更多的分配。旧代码以大约 2200MB/秒的速率传输 HTTP。这等于每秒 145,827 次 mallocs - 现在它们被删除了！一个 600 MB/秒的改进意味着 curl 每个 malloc 能比过去多传输 4300 字节。
|
||||
|
||||
### 去掉这些 malloc 难吗？
|
||||
|
||||
一点也不，非常直接。然而，有趣的是，在这个旧项目中，仍然有这样的变化空间。我有这个想法已经好几年了，我很高兴我终于花点时间来实现。感谢我们的测试套件，我可以有相当大的信心做这个“激烈的”内部变化，而不会引入太可怕的回归问题。由于我们的 API 很好地隐藏了内部，所以这种变化可以完全不改变任何旧的或新的应用程序。。
|
||||
|
||||
（是的，我还没在版本中发布变更，所以这还有风险，我不得不后悔我的“这很容易”的声明。。。）
|
||||
|
||||
### 注意数字
|
||||
|
||||
curl 的 git repo 从 7.53.1 到今天已经有 213 个提交。即使我不能想到任何更改，可能还会有一次或多次的提交，而不仅仅是内存分配对性能的影响。
|
||||
|
||||
### 更多信息？
|
||||
|
||||
还有其他类似的情况么？
|
||||
|
||||
也许。我们不会做很多性能测量或比较，所以谁知道，我们可能还会做更多的愚蠢的事情，我们可以停止做，并做得更好。我一直想做的一件事，但是从来没有这样做的是添加每天“监视”内存/mallocs，以及 curl 执行速度，以便更好地跟踪我们在这些方面不知不觉的倒退。
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://daniel.haxx.se/blog/2017/04/22/fewer-mallocs-in-curl/
|
||||
|
||||
作者：[DANIEL STENBERG ][a]
|
||||
译者：[geekpi](https://github.com/geekpi)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://daniel.haxx.se/blog/author/daniel/
|
||||
[1]:https://daniel.haxx.se/blog/author/daniel/
|
||||
[2]:https://daniel.haxx.se/blog/2017/04/22/fewer-mallocs-in-curl/
|
||||
[3]:https://daniel.haxx.se/blog/2017/04/22/fewer-mallocs-in-curl/#comments
|
||||
[4]:https://github.com/curl/curl/commit/cbae73e1dd95946597ea74ccb580c30f78e3fa73
|
||||
[5]:https://github.com/curl/curl/commit/5f1163517e1597339d
|
||||
[6]:https://github.com/curl/curl/commit/5f1163517e1597339d
|
||||
[7]:https://gist.github.com/bagder/dc4a42cb561e791e470362da7ef731d3
|
||||
173
translated/tech/20170529 LFCS sed Command.md
Normal file
173
translated/tech/20170529 LFCS sed Command.md
Normal file
@@ -0,0 +1,173 @@
|
||||
LFCS sed 命令
|
||||
=====================
|
||||
|
||||
Linux 基金会认证系统管理员（LFCS）的另一个有用的命令是 “sed”，最初表示“流式编辑器” （Streaming EDitor）。
|
||||
|
||||
“sed” 命令是一个可以将文件作为流进行编辑的编辑器。流式传输文件的方法是从另一个命令使用管道（> 或 |）传递，或将其直接加载到 “sed” 中。
|
||||
|
||||
该命令的工作方式与其他编辑器相同，只是文件不显示，也不允许可视化编辑。命令被传递给 “sed” 来操纵流。
|
||||
|
||||
用 “sed” 可以做五件基本的事。当然，“sed” 如此强大，还有其他高级的功能，但你只需要集中精力在五件基本的事上。五种功能类型如下：
|
||||
|
||||
1. 搜索
|
||||
|
||||
2. 替换
|
||||
|
||||
3. 删除
|
||||
|
||||
4. 添加
|
||||
|
||||
5. 改变/变换
|
||||
|
||||
在深入命令参数之前，我们需要看看基本的语法。
|
||||
|
||||
**语法**
|
||||
|
||||
“sed” 命令的语法是：
|
||||
|
||||
_sed [选项] 命令 [要编辑的文件]_
|
||||
|
||||
本文将在适当的部分中介绍这些选项。命令是可以是正则表达式的搜索和替换模式。继续阅读了解 “sed” 如何工作并学习基本命令。正如我之前提到的，“sed” 是一个非常强大的工具，有更多的选项可用，我将在本文中介绍。
|
||||
|
||||
**示例文件**
|
||||
|
||||
如果你打开一个终端，那你可以创建一个用于 “sed” 示例的文件。执行以下命令：
|
||||
|
||||
_cd ~
|
||||
grep --help >grephelp.txt_
|
||||
|
||||
你现在应该在 HOME 文件夹中有一个名为 “grephelp.txt” 的文件。该文件的内容是命令 “grep” 的帮助说明。
|
||||
|
||||
**搜索**
|
||||
|
||||
搜索特定字符串是编辑器的常见功能，在 “sed” 中执行搜索也不例外。
|
||||
|
||||
执行搜索以在文件中查找字符串。我们来看一下基本的搜索。
|
||||
|
||||
如果我们想在示例文件搜索 “PATTERN” 这个词，我们将使用这个命令，在图 1 中查看结果：
|
||||
|
||||
_sed -n 's/PATTERN/PATTERN/p' grephelp.txt_
|
||||
|
||||
**注意：** 如果剪切粘贴命令，请确保将单引号替换为键盘上的标准单引号。
|
||||
|
||||
|
||||
|
||||
**图 1**
|
||||
|
||||
参数 “-n” 用于自动抑制每行的打印。这些行包括不包含搜索模式的行。通过使用 “-n”，将显示有匹配结果的行。流入 “sed” 的每一行将被打印到标准输出（stdout）。如果你不使用 “-n” 选项运行上述命令，你将看到原始文件的每一行，以及每个匹配的重复行。
|
||||
|
||||
要搜索的文件名是我们在“示例文件”部分中创建的 “grephelp.txt”。
|
||||
|
||||
剩下的部分是 _'s/PATTERN/PATTERN/p'_ 。这一段基本分为四个部分。第一部分的 “s” 指定执行替换，或搜索和替换。
|
||||
|
||||
剩下的第二部分和第三部分是模式。第一个是要搜索的模式，最后一个是替换流中匹配字符串的模式。此例中，我们找到字符串 “PATTERN”，并用 “PATTERN” 替换。通过查找和替换相同的字符串，我们完全不会更改文件，甚至在屏幕上也一样。
|
||||
|
||||
最后一个命令是 “p”。 “p” 指定在替换后打印新行。当然，因为替换的是相同的字符串，所以没有改变。由于我们使用 “-n” 参数抑制打印行，所以更改的行将使用 “p” 命令打印。
|
||||
|
||||
完整的命令允许我们执行搜索并查看匹配的结果。
|
||||
|
||||
**替换**
|
||||
|
||||
当搜索特定字符串时，你可能希望用匹配的字符串替换新字符串。用另一个字符串替换是很常见的。
|
||||
|
||||
我们可以使用以下命令执行相同的搜索，结果如图 2 所示：
|
||||
|
||||
_sed -n 's/PATTERN/Pattern/p' grephelp.txt_
|
||||
|
||||
|
||||
|
||||
**图 2**
|
||||
|
||||
在这时，字符串 “PATTERN” 变为 “Pattern” 并显示。如果你使用命令 “cat grephelp.txt” 查看文件，你会看到该文件没有更改。该更改仅对屏幕上的输出进行。你可以使用以下命令将输出通过管道传输到另一个文件：
|
||||
|
||||
_sed 's/PATTERN/Pattern/' grephelp.txt > grephelp1.txt_
|
||||
|
||||
现在将存在一个名为 “grephelp1.txt” 的新文件，其中保存了更改的文件。如果 “p” 作为第四个选项留下，那么有个问题是每一行被替换的字符串将在文件中重复两次。我们也删除 “-n” 参数以允许所有的行打印。
|
||||
|
||||
使用相同字符串替换字符串的另一种方法是使用 “&” 符号来表示搜索字符串。例如，命令 “s/PATTERN/&/p” 效果是一样的。我们可以添加字符串，例如添加 “S” , 可以使用命令 “s/PATTERN/&S/p”。
|
||||
|
||||
如果我们希望在每一行中只替换某种模式呢？可以指定要替换的匹配项的特定出现。当然，每一行的替换都是一个特定的编号。例如，示例文件上有很多破折号。一些行至少有两条破折号，所以我们可以用另一个字符代替每一行的第二个破折号。每行用星号（*）替换第二个破折号（ - ）的命令将是：
|
||||
|
||||
sed 's/-/*/2' grephelp.txt
|
||||
|
||||
在这里，我们用最初的 “s” 来执行替换。字符 “-” 被替换为 “*”。“2” 表示我们想要替换每行上的第二个 “-”（如果存在）。示例结果如图 3 所示。如果我们忽略了命令 “2”，则替换第一次出现的破折号。只有第一个破折号而不是每行的破折号都被替换。
|
||||
|
||||
|
||||
|
||||
**图 3**
|
||||
|
||||
如果要搜索并替换带有星号的行上的所有破折号，请使用 “g” 命令：
|
||||
|
||||
_sed 's/-/*/g' grephelp.txt_
|
||||
|
||||
命令也可以组合。假设你想要替换从第二次开始出现的破折号，命令将是：
|
||||
|
||||
_sed 's/-/*/2g' grephelp.txt_
|
||||
|
||||
现在从第二个开始出现的破折号将被星号取代。
|
||||
|
||||
**删除**
|
||||
|
||||
搜索过程中有很多时候你可能想要完全删除搜索字符串。
|
||||
|
||||
例如，如果要从文件中删除所有破折号，你可以使用以下命令：
|
||||
|
||||
_sed ‘s/-//g’ grephelp.txt_
|
||||
|
||||
替换字符串为空白，因此匹配的字符串将被删除。
|
||||
|
||||
**添加**
|
||||
|
||||
当找到匹配时，你可以添加一行特定的文本，来使这行在浏览或打印中突出。
|
||||
|
||||
如果要在匹配后插入新行，那么使用 “a” 命令，后面跟上新行的字符串。还包括要匹配的字符串。例如，我们可以找到一个 “--”，并在匹配的行之后添加一行。新行的字符串将是 “double dash before this line”。
|
||||
|
||||
_sed '/--/ a "double dash before this line"' grephelp.txt_
|
||||
|
||||
如果要在包含匹配字符串的行之前加上这行，请使用 “i” 命令，如下所示：
|
||||
|
||||
_sed '/--/ i "double dash after this line"' grephelp.txt_
|
||||
|
||||
**改变/变换**
|
||||
|
||||
如果需要改变/变换一行，则可以使用命令 “c”。
|
||||
|
||||
假设我们有个有一些私人信息的文档，我们需要更改包含特定字符串的行。“c” 命令将改变整行，而不仅仅是搜索字符串。
|
||||
|
||||
假设我们想要阻止示例文件中包含单词 “PATTERN” 的每一行。更改的行将显示为 “This line is Top Secret”。命令是：
|
||||
|
||||
_sed ‘/PATTERN/ c This line is Top Secret’ grephelp.txt_
|
||||
|
||||
可以进行更改特定字母的大小写的转换。例如，我们可以使用命令 “y” 将所有小写 “a” 更改为大写 “A”，如下所示：
|
||||
|
||||
_sed ‘y/a/A/’ grephelp.txt_
|
||||
|
||||
可以指定多个字母，如 “abdg”，如下命令所示：
|
||||
|
||||
_sed ‘y/abdg/ABDG/’ grephelp.txt_
|
||||
|
||||
确保第二组字母与第一组字母的顺序相同，否则会被替换和转换。例如，字符串 ‘y/a/D/’ 将用大写 “D” 替换所有小写的 “a”。
|
||||
|
||||
**就地更改**
|
||||
|
||||
如果你确实要更改所使用的文件，请使用 “-i” 选项。
|
||||
|
||||
例如，要将 “PATTERN” 改为 “Pattern”，并对文件进行更改，则命令为：
|
||||
|
||||
sed -i 's/PATTERN/Pattern/' grephelp.txt
|
||||
|
||||
现在文件 “grephelp.txt” 将被更改。“-i” 选项可以与上述任何命令一起使用来更改原始文件的内容。
|
||||
|
||||
练习这些命令，并确保你理解它们。“sed” 命令非常强大。
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://www.linux.org/threads/lfcs-sed-command.4561/
|
||||
|
||||
作者：[Jarret B ][a]
|
||||
译者：[geekpi](https://github.com/geekpi)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://www.linux.org/threads/lfcs-sed-command.4561/
|
||||
@@ -0,0 +1,110 @@
|
||||
# [开源社交机器人套件运行在 Raspberry Pi 和 Arduino 上][22]
|
||||
|
||||
|
||||
|
||||
|
||||
Thecorpora 的发布的 “Q.bo One” 机器人基于 RPi 3 和 Arduino，并提供立体相机、麦克风、扬声器，以及视觉和语言识别。
|
||||
|
||||
2010 年，机器人开发商 Francisco Paz 及其巴塞罗那的 Thecorpora 公司推出了首款 [Qbo][6] “Cue-be-oh” 机器人作为一个开源概念验证和用于探索 AI 在多传感器、交互式机器人的能力的研究项目。目前，在 2 月移动世界大会上的预览之后，Thecorpora 把它放到了 Indiegogo 上，与 Arrow 合作推出了第一个批量生产的社交机器人版本。
|
||||
|
||||
[][7] [][8]
|
||||
**Q.bo One 的左侧和顶部**
|
||||
|
||||
|
||||
像原来一样，新的 Q.bo One 有一个带眼睛的球形头（双立体相机）、耳朵（3 个麦克风）和嘴（扬声器），并由 WiFi 和蓝牙控制。 Q.bo One 也同样有开源 Linux 软件和开放规格硬件。然而，它不是使用基于 Intel Atom 的 Mini-ITX 板，而是在与 Arduino 兼容的主板相连的 Raspberry Pi 3 上运行 Raspbian。
|
||||
|
||||
|
||||
[][9]
|
||||
**Q.bo One side views**
|
||||
|
||||
|
||||
Q.bo One 于 7 月中旬在 Indiegogo 上架，起价为 369 美元（早期买家）或 399 美元，有包括内置的 Raspberry Pi 3 和基于 Arduino 的 “Qboard” 控制器板。它还有售价 $499 的完整套装。目前，Indiegogo 上的弹性目标是 $100,000，现在大概达成了 15%，并且它 12 月出货。
|
||||
|
||||
更专业的机器人工程师和嵌入式开发人员可能会想要使用只有 RPI 和 Qboard PCB 和软件的价值 $99 的版本，或者提供没有电路板的机器人套件的 $249 版本。使用此版本，你可以用自己的 Arduino 控制器替换 Qboard，并将 RPi 3 替换为另一个 Linux SBC。该公司列出了 Banana Pi、BeagleBone、Tinker Board 以及[即将退市的 Intel Edison][10]，作为兼容替代品的示例。
|
||||
|
||||
<center>
|
||||
[][11]
|
||||
**Q.bo One kit**
|
||||
(click image to enlarge)
|
||||
</center>
|
||||
|
||||
与 2010 年的 Qbo 不同，Q.bo One 除了球面头部之外无法移动，它在双重伺服系统的帮助下在底座上旋转，以便跟踪声音和动作。Robotis Dynamixel 舵机，也在开源中找到，Raspberry Pi 基于 [TurtleBot 3][23] 机器人工具包，除了左右之外，还可以上下移动。
|
||||
|
||||
<center>
|
||||
[][12] [][13]
|
||||
**Q.bo One detail view (left) and Qboard detail**
|
||||
(click images to enlarge)
|
||||
</center>
|
||||
|
||||
Q.bo One 也可类似地与基于 Linux 的 [Jibo][24] “社交机器人”相比，它于 2014 年在 Indiegogo 推出，最后达到 360 万美元。然而，Jibo 还没有出货，[最近的推迟][25]迫使它在今年的某个时候发布一个版本。
|
||||
|
||||
|
|
||||
|
||||
|
||||
**Q.bo One** |
|
||||
|
||||
我们大胆预测 Q.bo One 将会在 2017 年接近 12 月出货。核心技术和 AI 软件已被证明，而 Raspberry Pi 和 Arduino 技术也是如此。Qboard 主板已经由 Arrow 制造和认证。
|
||||
|
||||
开源设计表明, 即使是移动版本也不会有问题。这使它更像是滚动的人形生物 [Pepper][14]，一个来自 Softbank 和 Aldeberan 类似的人工智能对话机器人。
|
||||
|
||||
Q.bo One 自原版以来添加了一些技巧，例如由 20 个 LED 组成的“嘴巴”， 它以不同的、可编程的方式在语音中模仿嘴唇移动。如果你想点击机器人获得关注，那么它的头上还有三个触摸传感器。但是，你真正需要做的就是说话，而 Q.bo One 会像一个可卡犬一样转身并凝视着你。
|
||||
|
||||
接口和你在 Raspberry Pi 3 上的一样，它在我们的[2017 黑客电路板调查][15]中消灭了其他对手。为 RPi 3 的 WiFi 和蓝牙安装了天线。
|
||||
|
||||
<center>
|
||||
[][16] [][17]
|
||||
**Q.bo One software architecture (left) and Q.bo One with Scratch screen**
|
||||
(click images to enlarge)
|
||||
</center>
|
||||
|
||||
Qboard（也称为 Q.board）在 Atmel ATSAMD21 MCU 上运行 Arduino 代码，并有三个麦克风、扬声器、触摸传感器、Dynamixel 控制器和用于嘴巴的 LED 矩阵。其他功能包括 GPIO、I2C接口和可连接到台式机的 micro-USB 口。
|
||||
|
||||
Q.bo One 可以识别脸部和追踪移动，机器人甚至可以在镜子中识别自己。在云连接的帮助下，机器人可以识别并与其他 Q.bo One 机器人交谈。机器人可以在自然语言处理的帮助下回答问题，并通过文字转语音朗读。
|
||||
|
||||
可以使用 Scratch 编程，它是机器人的主要功能，可以教孩子关于机器人和编程。机器人也是为教育者和制造者设计的，可以作为老年人的伴侣。
|
||||
|
||||
基于 Raspbian 的软件使用 OpenCV 进行视觉处理，并可以使用各种语言（包括 C++）进行编程。该软件还提供了 IBM Bluemix、NodeRED 和 ROS 的钩子。大概你也可以整合 [Alexa][18] 或 [Google Assistant][19]语音代理，虽然 Thecorpora 没有提及这一点。
|
||||
|
||||
|
||||
|
||||
**更多信息**
|
||||
|
||||
Q.bo One 在 7 月中旬在 Indiegogo 上架，起价为 $369 的完整套件和 $499 的所有组合
|
||||
。出货量预计在 2017 年 12 月。更多信息请参见[ Q.bo One 的 Indiegogo 页面][20] 和[ Thecorpora 网站][21]。
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: http://linuxgizmos.com/open-source-social-robot-kit-runs-on-raspberry-pi-and-arduino/
|
||||
|
||||
作者：[ Eric Brown][a]
|
||||
译者：[geekpi](https://github.com/geekpi)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:http://linuxgizmos.com/open-source-social-robot-kit-runs-on-raspberry-pi-and-arduino/
|
||||
[1]:http://twitter.com/share?url=http://linuxgizmos.com/open-source-social-robot-kit-runs-on-raspberry-pi-and-arduino/&text=Open+source+social+robot+kit+runs+on+Raspberry+Pi+and+Arduino+
|
||||
[2]:https://plus.google.com/share?url=http://linuxgizmos.com/open-source-social-robot-kit-runs-on-raspberry-pi-and-arduino/
|
||||
[3]:http://www.facebook.com/sharer.php?u=http://linuxgizmos.com/open-source-social-robot-kit-runs-on-raspberry-pi-and-arduino/
|
||||
[4]:http://www.linkedin.com/shareArticle?mini=true&url=http://linuxgizmos.com/open-source-social-robot-kit-runs-on-raspberry-pi-and-arduino/
|
||||
[5]:http://reddit.com/submit?url=http://linuxgizmos.com/open-source-social-robot-kit-runs-on-raspberry-pi-and-arduino/&title=Open%20source%20social%20robot%20kit%20runs%20on%20Raspberry%20Pi%20and%20Arduino
|
||||
[6]:http://linuxdevices.linuxgizmos.com/open-source-robot-is-all-eyes/
|
||||
[7]:http://linuxgizmos.com/files/thecorpora_qboone.jpg
|
||||
[8]:http://linuxgizmos.com/files/thecorpora_qboone2.jpg
|
||||
[9]:http://linuxgizmos.com/files/thecorpora_qboone_side.jpg
|
||||
[10]:http://linuxgizmos.com/intel-pulls-the-plug-on-its-joule-edison-and-galileo-boards/
|
||||
[11]:http://linuxgizmos.com/files/thecorpora_qboone_kit.jpg
|
||||
[12]:http://linuxgizmos.com/files/thecorpora_qboone_detail.jpg
|
||||
[13]:http://linuxgizmos.com/files/thecorpora_qboone_qboard.jpg
|
||||
[14]:http://linuxgizmos.com/worlds-first-emotional-robot-runs-linux/
|
||||
[15]:http://linuxgizmos.com/2017-hacker-board-survey-raspberry-pi-still-rules-but-x86-sbcs-make-gains/
|
||||
[16]:http://linuxgizmos.com/files/thecorpora_qboone_arch.jpg
|
||||
[17]:http://linuxgizmos.com/files/thecorpora_qboone_scratch.jpg
|
||||
[18]:http://linuxgizmos.com/how-to-add-alexa-to-your-raspberry-pi-3-gizmo/
|
||||
[19]:http://linuxgizmos.com/free-raspberry-pi-voice-kit-taps-google-assistant-sdk/
|
||||
[20]:https://www.indiegogo.com/projects/q-bo-one-an-open-source-robot-for-everyone#/
|
||||
[21]:http://thecorpora.com/
|
||||
[22]:http://linuxgizmos.com/open-source-social-robot-kit-runs-on-raspberry-pi-and-arduino/
|
||||
[23]:http://linuxgizmos.com/ubuntu-driven-turtlebot-gets-a-major-rev-with-a-pi-or-joule-in-the-drivers-seat/
|
||||
[24]:http://linuxgizmos.com/cheery-social-robot-owes-it-all-to-its-inner-linux/
|
||||
[25]:https://www.slashgear.com/jibo-delayed-to-2017-as-social-robot-hits-more-hurdles-20464725/
|
||||
@@ -0,0 +1,315 @@
|
||||
[GitHub 的 MySql 基础架构自动化测试][31]
|
||||
============================================================
|
||||
|
||||
我们 MySQL 数据库基础架构是 Github 关键组件。 MySQL 提供 Github.com, GitHub 的 API 和验证等等的服务。每一次的 `git` 请求都以某种方式触及 MySQL。即使我们 MySQL 集群是按流量的，但是我们还是需要执行重型清理，即时更新，在线模式迁移，集群拓扑重构，池化和负载平衡等任务。 我们建有基础架构自动化测试这种方式，在这篇文章中，我们分享几个例子，说明我们如何通过连续不间断的来测试建立基础架构的。这样的方式是为了让我们晚上有一个好梦到早晨。
|
||||
|
||||
### 备份[][36]
|
||||
|
||||
备份数据是非常重要的，如果您没有备份数据库，虽然当时没有说明问题，在之后可能就是一个大问题。Percona [Xtrabackup][37] 是我们一直使用的 MySQL 数据库备份工具。如果有需要备份数据库我们就会备份到另一个专门备份数据的服务器上。
|
||||
|
||||
In addition to the full binary backups, we run logical backups several times a day. These backups allow our engineers to get a copy of recent data. There are times that they would like a complete set of data from a table so they can test an index change on a production sized table or see data from a certain point of time. Hubot allows us to restore a backed up table and will ping us when the table is ready to use.
|
||||
|
||||
除了完整的二进制备份外，我们每天还会多次运行逻辑备份。这些备份数据允许我们的工程师获取最新的副本。有时候，他们希望从表中获取一整套数据，以便他们可以测试表上的索引，更改或从特定时间点查看数据。Hubot 允许我们恢复备份的表，并且当表准备使用时会自动检测连接（ ping ）我们。
|
||||
|
||||
|
||||
**tomkrouper**.mysql 备份列表的位置
|
||||
|
||||
**Hubot**
|
||||
```
|
||||
+-----------+------------+---------------+---------------------+---------------------+----------------------------------------------+
|
||||
| Backup ID | Table Name | Donor Host | Backup Start | Backup End | File Name |
|
||||
+-----------+------------+---------------+---------------------+---------------------+----------------------------------------------+
|
||||
| 1699494 | locations | db-mysql-0903 | 2017-07-01 22:09:17 | 2017-07-01 22:09:17 | backup-mycluster-locations-1498593122.sql.gz |
|
||||
| 1699133 | locations | db-mysql-0903 | 2017-07-01 16:11:37 | 2017-07-01 16:11:39 | backup-mycluster-locations-1498571521.sql.gz |
|
||||
| 1698772 | locations | db-mysql-0903 | 2017-07-01 10:09:21 | 2017-07-01 10:09:22 | backup-mycluster-locations-1498549921.sql.gz |
|
||||
| 1698411 | locations | db-mysql-0903 | 2017-07-01 04:12:32 | 2017-07-01 04:12:32 | backup-mycluster-locations-1498528321.sql.gz |
|
||||
| 1698050 | locations | db-mysql-0903 | 2017-06-30 22:18:23 | 2017-06-30 22:18:23 | backup-mycluster-locations-1498506721.sql.gz |
|
||||
| ...
|
||||
| 1262253 | locations | db-mysql-0088 | 2016-08-01 01:58:51 | 2016-08-01 01:58:54 | backup-mycluster-locations-1470034801.sql.gz |
|
||||
| 1064984 | locations | db-mysql-0088 | 2016-04-04 13:07:40 | 2016-04-04 13:07:43 | backup-mycluster-locations-1459494001.sql.gz |
|
||||
+-----------+------------+---------------+---------------------+---------------------+----------------------------------------------+
|
||||
|
||||
```
|
||||
|
||||
|
||||
**tomkrouper**.mysql 恢复 1699133
|
||||
|
||||
**Hubot**已为备份作业 1699133 创建还原作业，还原完成后，将在 ＃database-ops 中收到通知。
|
||||
|
||||
**Hubot**[@tomkrouper][1]: 恢复 db-mysql-0482 上数据库中的 locations 表已恢复为 locations_2017_07_01_16_11
|
||||
|
||||
数据被加载到非生产环境的数据库，该数据库可供请求恢复的工程师访问。
|
||||
|
||||
我们保留数据的“备份”的最后一个方法是使用 [延迟的副本delayed replicas][38]。这不是一个备份，而是更多层次的保护。对于每个生产集群中，我们有一个主机延迟4个小时。如果运行一个不应该有的查询，我们可以在 chatops 中运行 `mysql panic` 。这将导致我们所有的延迟副本立即停止复制。这也将页面呼叫DBA。从那里我们可以使用延迟复制来验证是否有问题，然后将二进制日志快速转发到错误之前的位置。然后，我们可以将此数据恢复到主服务器，从而恢复数据到该点。
|
||||
|
||||
备份是很棒的，但如果一些未知或未捕获的错误发生破坏它们，它们就显得没有价值了。让脚本恢复备份的好处是它允许我们通过 cron 自动执行备份验证。我们为每个集群设置了一个专用的主机，用于运行最新备份的恢复。这样可以确保备份运行正常，并且我们能够从备份中检索数据。
|
||||
|
||||
根据数据集大小，我们每天运行多个恢复。预期恢复的服务器将加入复制流并能够赶上复制。这不仅测试了我们采取可恢复的备份，而且我们正确地确定采取的时间点，并且可以从该时间点进一步应用更改。如果恢复过程中出现问题，我们会收到通知。
|
||||
|
||||
我们还追踪恢复所需的时间，所以我们知道在紧急情况下建立新的副本或还原需要多长时间。
|
||||
|
||||
以下是由 Hubot 在我们的机器人聊天室中编写的自动恢复过程的输出。
|
||||
|
||||
|
||||
**Hubot**
|
||||
gh-mysql-backup-restore: db-mysql-0752: restore_log.id = 4447
|
||||
gh-mysql-backup-restore: db-mysql-0752: 确定要为集群 “prodcluster” 还原备份。
|
||||
gh-mysql-backup-restore: db-mysql-0752: 启用维护模式
|
||||
gh-mysql-backup-restore: db-mysql-0752: 设置协调器停机时间
|
||||
gh-mysql-backup-restore: db-mysql-0752: 禁用 Puppet
|
||||
gh-mysql-backup-restore: db-mysql-0752: 停止 MySQL
|
||||
gh-mysql-backup-restore: db-mysql-0752: 删除 MySQL 文件
|
||||
gh-mysql-backup-restore: db-mysql-0752: 运行中 gh-xtrabackup-restore
|
||||
gh-mysql-backup-restore: db-mysql-0752: 恢复文件: xtrabackup-notify-2017-07-02_0000.xbstream
|
||||
gh-mysql-backup-restore: db-mysql-0752: 运行 gh-xtrabackup-prepare
|
||||
gh-mysql-backup-restore: db-mysql-0752: 启动 MySQL
|
||||
gh-mysql-backup-restore: db-mysql-0752: 更新文件权限
|
||||
gh-mysql-backup-restore: db-mysql-0752: 升级 MySQL
|
||||
gh-mysql-backup-restore: db-mysql-0752: 停止 MySQL
|
||||
gh-mysql-backup-restore: db-mysql-0752: 启动 MySQL
|
||||
gh-mysql-backup-restore: db-mysql-0752: 备份 Host: db-mysql-0034
|
||||
gh-mysql-backup-restore: db-mysql-0752: 设置开启复制
|
||||
gh-mysql-backup-restore: db-mysql-0752: 启动复制
|
||||
gh-mysql-backup-restore: db-mysql-0752: 响应缓存
|
||||
gh-mysql-backup-restore: db-mysql-0752: 还原完成 (响应运行中)
|
||||
gh-mysql-backup-restore: db-mysql-0752: 开启操作
|
||||
gh-mysql-backup-restore: db-mysql-0752: 关闭维护模式
|
||||
gh-mysql-backup-restore: db-mysql-0752: 设置协调器停机时间
|
||||
gh-mysql-backup-restore: db-mysql-0752: 恢复过程完成
|
||||
|
||||
我们使用备份的一件事是向现有的一组 MySQL 服务器添加一个新的副本。我们将启动一个新的服务器的构建，一旦我们通知它已经准备就绪，我们可以开始恢复该特定集群的最新备份。我们有一个脚本，运行我们手动执行的所有恢复命令。我们的自动恢复系统基本上使用相同的脚本。这简化了系统构建过程，并允许我们使用少数几个手动进程的聊天命令来启动和运行主机。下面显示的是在聊天中手动恢复：
|
||||
|
||||
|
||||
**jessbreckenridge**.mysql backup-restore -H db-mysql-0007 -o -r magic_word=daily_rotating_word
|
||||
|
||||
**Hubot**
|
||||
[@jessbreckenridge][2] gh-mysql-backup-restore: db-mysql-0007: 确定要还原群集'mycluster'的备份
|
||||
[@jessbreckenridge][3] gh-mysql-backup-restore: db-mysql-0007: restore_log.id = 4449
|
||||
[@jessbreckenridge][4] gh-mysql-backup-restore: db-mysql-0007: 启用维护模式
|
||||
[@jessbreckenridge][5] gh-mysql-backup-restore: db-mysql-0007: 设置协调器停机时间
|
||||
[@jessbreckenridge][6] gh-mysql-backup-restore: db-mysql-0007: 禁用操作
|
||||
[@jessbreckenridge][7] gh-mysql-backup-restore: db-mysql-0007: 停止 MySQL
|
||||
[@jessbreckenridge][8] gh-mysql-backup-restore: db-mysql-0007: 删除 MySQL 文件
|
||||
[@jessbreckenridge][9] gh-mysql-backup-restore: db-mysql-0007: 运行 gh-xtrabackup-restore
|
||||
[@jessbreckenridge][10] gh-mysql-backup-restore: db-mysql-0007: 恢复文件: xtrabackup-mycluster-2017-07-02_0015.xbstream
|
||||
[@jessbreckenridge][11] gh-mysql-backup-restore: db-mysql-0007: 运行 gh-xtrabackup-prepare
|
||||
[@jessbreckenridge][12] gh-mysql-backup-restore: db-mysql-0007: 更新文件权限
|
||||
[@jessbreckenridge][13] gh-mysql-backup-restore: db-mysql-0007: 启用 MySQL
|
||||
[@jessbreckenridge][14] gh-mysql-backup-restore: db-mysql-0007: 升级 MySQL
|
||||
[@jessbreckenridge][15] gh-mysql-backup-restore: db-mysql-0007: 停止 MySQL
|
||||
[@jessbreckenridge][16] gh-mysql-backup-restore: db-mysql-0007: 开启 MySQL
|
||||
[@jessbreckenridge][17] gh-mysql-backup-restore: db-mysql-0007: 设置开启复制
|
||||
[@jessbreckenridge][18] gh-mysql-backup-restore: db-mysql-0007: 启动复制
|
||||
[@jessbreckenridge][19] gh-mysql-backup-restore: db-mysql-0007: 备份 Host: db-mysql-0201
|
||||
[@jessbreckenridge][20] gh-mysql-backup-restore: db-mysql-0007: 复制缓存
|
||||
[@jessbreckenridge][21] gh-mysql-backup-restore: db-mysql-0007: 复制后 4589 秒，下次检查之前等待 1800 秒
|
||||
[@jessbreckenridge][22] gh-mysql-backup-restore: db-mysql-0007: 还原完成 (响应运行中)
|
||||
[@jessbreckenridge][23] gh-mysql-backup-restore: db-mysql-0007: 禁用操作
|
||||
[@jessbreckenridge][24] gh-mysql-backup-restore: db-mysql-0007: 禁用维护模式
|
||||
|
||||
### 故障转移[][39]
|
||||
|
||||
[我们使用协调器][40] 在主和从中执行自动化故障切换。我们期望 orchestrator 正确检测主故障，指定副本进行升级，在所指定的副本下修复拓扑，进行升级。我们期待 VIP 变化，池变化，客户端重连，`puppet` 运行的基本组件等等。故障转移是一项复杂的任务，涉及到我们基础架构的许多方面。
|
||||
|
||||
为了建立对我们的故障转移的信任，我们建立了一个 _类似生产的测试集群_ ，并且我们不断地崩溃它来观察故障转移。
|
||||
|
||||
_类似生产的测试集群_ 是一个复制设置，在所有方面与我们的生产集群相同：硬件类型，操作系统，MySQL版本，网络环境，VIP，`puppet` 配置，[haproxy 设置][41] 等。与此集群不同的是它不发送/接收生产流量。
|
||||
|
||||
我们在测试集群上模拟写入负载，同时避免复制滞后。写入负载不会太大，但是有一些有意争取写入相同数据集的查询。这在正常的时间并不是太有意思，但是证明在故障转移中是有用的，正如我们将会简要描述的那样。
|
||||
|
||||
我们的测试集群有三个数据中心的代表服务器。 我们希望故障转移能够在同一个数据中心内推广替代副本。 我们希望在这样的限制下尽可能多地复制副本。 我们要求尽可能适用。 协调者对拓扑结构没有先前的假设; 它必须对崩溃时的状态作出反应。
|
||||
|
||||
然而，我们有兴趣为故障切换创建复杂而多变的场景。我们的故障转移测试脚本为故障转移准备了理由：
|
||||
|
||||
* 它识别现有的主人
|
||||
|
||||
* 重构拓扑结构，使所有三个数据中心的代表成为主控。不同的 DC 具有不同的网络延迟，并且预期会在不同的时间对主机崩溃做出反应。
|
||||
|
||||
* 它选择一个崩溃方法。我们选择 master（`kill -9`）或网络划分它: `iptables -j REJECT` (nice-ish) 或 `iptables -j DROP`(unresponsive无响应)。
|
||||
|
||||
脚本继续通过选择的方法使主机崩溃，并等待 `orchestrator` 可靠地检测到崩溃并执行故障转移。虽然我们期望检测和升级在 `30` 几秒钟内 完成，但脚本会打破这一期望，并在查找故障转移结果之前睡觉一段指定的时间。然后：
|
||||
|
||||
* 检查一个新的（不同的）master 是否到位
|
||||
|
||||
* 集群中有很多副本
|
||||
|
||||
* master 是可写的
|
||||
|
||||
* 对 master 的写入在副本上可见
|
||||
|
||||
* 更新内部服务发现条目（新 master 的身份如预期;旧 master 已删除）
|
||||
|
||||
* 其他内部检查
|
||||
|
||||
这些测试证实故障转移是成功的，不仅是 MySQL 明智的，而且在更大的基础设施范围。VIP 被拒绝; 特别服务已经开始; 信息到达应该去的地方。
|
||||
|
||||
该脚本进一步继续恢复失败的服务器:
|
||||
|
||||
* 从备份恢复，从而隐含地测试我们的备份/恢复过程
|
||||
|
||||
* 验证服务器配置是否符合预期（服务器不再相信是主服务器）
|
||||
|
||||
* 返回到复制集群，期望找到在主机上写入的数据
|
||||
|
||||
考虑以下可视化的计划故障转移测试：从运行良好的群集到某些副本上的问题，诊断主机 (`7136`) 是否死机，选择一个服务器来促进 (`a79d`) ，重构该服务器下面的拓扑，推动它（故障切换成功），恢复死主机并将其放回群集。
|
||||
|
||||
|
||||
|
||||
#### 测试失败怎么样?
|
||||
|
||||
我们的测试脚本使用了一种停止世界的方法。任何故障切换组件中的单个故障都将失败，因此在人类解决问题之前，无法进行任何未来的自动化测试。我们得到警报，并继续检查状态和日志。
|
||||
|
||||
脚本将在不可接受的检测或故障转移时间失败; 备份/还原问题; 失去太多服务器; 在故障切换后的意外配置; 等等
|
||||
|
||||
我们需要确保 `orchestrator` 正确连接服务器。这是竞争性写入负载有用的地方：如果设置不正确，复制很容易中断。我们会得到 `DUPLICATE KEY` 或其他错误提示出错。
|
||||
|
||||
这是特别重要的，因为我们改进 `orchestrator` 并引入新的行为，允许我们在安全的环境中测试这些变化。
|
||||
|
||||
#### 来了：chaos 测试
|
||||
|
||||
上面所示的测试程序将捕获（并已经捕获）我们基础设施许多部分的问题。这些够了吗？
|
||||
|
||||
在生产环境中总是有其他的东西。关于不适用于我们的生产集群的特定测试方法。它们不具有相同的流量和流量操纵，也不具有完全相同的服务器集。故障类型可能有所不同。
|
||||
|
||||
我们正在为我们的生产集群设计 chaos 测试。 chaos 测试将会在我们的生产中，但是按照预期的时间表和充分控制的方式来破坏碎片。 chaos 测试在恢复机制中引入更高层次的信任，并影响（因此测试）我们的基础设施和应用程序的较大部分。
|
||||
|
||||
这是微妙的工作：当我们承认 chaos 测试的需要时，我们也希望避免对我们的服务造成不必要的影响。不同的测试将在风险级别和影响方面有所不同，我们将努力确保我们的服务的可用性。
|
||||
|
||||
### 模式迁移[][42]
|
||||
|
||||
[我们使用 gh-ost ][43]来运行实时模式迁移。 `gh-ost` 它是稳定的，但也在迅速的发展，增加或计划的主要新功能。
|
||||
|
||||
`gh-ost` 通过将数据复制到 _gh-ost_ 表来迁移，将进行的二进制日志拦截的更改应用到 _gh-ost_表，即使正在写入原始表。然后它将 _gh-ost_ 表交换代替原始表。迁移完成时，GitHub 继续使用生成和填充的 `gh-ost` 表。
|
||||
|
||||
在这个时候，几乎所有的 GitHub 的 MySQL 数据都被重新创建 `gh-ost`，其中大部分都是重新创建的。我们必须高度信任 `gh-ost`，即使面对快速的发展，也可以一再修改数据。这是我们如何获得这种信任。
|
||||
|
||||
`gh-ost` 提供生产测试能力。它支持在副本上运行迁移，其方式与在主服务器上运行的方式大致相同： `gh-ost` 将连接到副本，并将其视为主服务器。它将解析其二进制日志的方式与实际主机迁移相同。但是，它将复制行并将二进制日志事件应用于副本，并避免对主机进行写入。
|
||||
|
||||
我们经营的 ｀gh-ost｀ 复制品在生产中。这些副本不能提供生产流量。每个这样的副本将检索生产表的当前列表，并以随机顺序对其进行迭代。一个接一个地，它选择一个表并在该表上执行复制迁移。迁移实际上并不修改表结构，而是运行一个微不足道的 ｀ENGINE=InnoDB｀。测试运行迁移，即使在生产中使用表，从而复制实际的生产数据，并将真正的生产流量从二进制日志中应用。
|
||||
|
||||
这些迁移可以被审计。以下是我们如何从聊天中检查运行测试的状态：
|
||||
|
||||
|
||||
**ggunson**.migration test-status
|
||||
|
||||
**Hubot**
|
||||
```
|
||||
# Migrating `prod`.`pull_requests`; Ghost table is `prod`.`_pull_requests_gho`
|
||||
# Migrating ghost-db-mysql-0007:3306; inspecting ghost-db-mysql-0007:3306; executing on ghost-db-mysql-0007
|
||||
# Migration started at Mon Jan 30 02:13:39 -0800 2017
|
||||
# chunk-size: 2500; max-lag-millis: 1500ms; max-load: Threads_running=30; critical-load: Threads_running=1000; nice-ratio: 0.000000
|
||||
# throttle-additional-flag-file: /tmp/gh-ost.throttle
|
||||
# panic-flag-file: /tmp/ghost-test-panic.flag
|
||||
# Serving on unix socket: /tmp/gh-ost.test.sock
|
||||
Copy: 57992500/86684838 66.9%; Applied: 57708; Backlog: 1/100; Time: 3h28m38s(total), 3h28m36s(copy); streamer: mysql-bin.000576:142993938; State: migrating; ETA: 1h43m12s
|
||||
```
|
||||
|
||||
当测试迁移完成表数据的复制时，它将停止复制并执行切换，使用 _gh-ost_ 表替换原始表，然后交换回来。我们对实际更换数据并不感兴趣。相反，我们将留下原始的表和 _gh-ost_ 表，它们都应该是相同的。我们通过校验两个表的整个表数据来验证。
|
||||
|
||||
测试可以完成:
|
||||
|
||||
* _成功_ : 一切顺利，校验和相同。我们期待看到这一点。
|
||||
|
||||
* _失败_ : 执行问题。这可能偶尔发生，因为迁移过程被杀死，复制问题等，并且通常与 `gh-ost` 自身无关。
|
||||
|
||||
* _校验失败_ : 表数据不一致。对于被测试的分支，这个调用修复。对于正在进行的 `master` 分支测试，这意味着立即阻止生产迁移。我们不会得到后者。
|
||||
|
||||
测试结果经过审核，发送到机器人聊天室，作为事件发送到我们的度量系统。下图中的每条垂直线代表成功的迁移测试:
|
||||
|
||||
|
||||
|
||||
这些测试不断运行。如果发生故障，我们会收到通知。当然，我们可以随时访问机器人聊天室，了解发生了什么。
|
||||
|
||||
#### 测试新版本
|
||||
|
||||
我们不断提高 `gh-ost`。我们的开发流程基于 `git` 分支机构，然后我们提供通过[拉请求][44]进行合并。
|
||||
|
||||
提交的 `gh-ost` 拉请求通过持续集成（CI）进行基本的编译和单元测试。一旦过去，公关在技术上有资格合并，但更有趣的是它有 [eligible for deployment via Heaven][45] 。作为我们基础架构中的敏感组件，我们 ｀gh-ost｀ 在合并之前，要小心部署 master 分支机构，进行密集测试。
|
||||
|
||||
|
||||
**shlomi-noach**.deploy gh-ost/fix-reappearing-throttled-reasons to prod/ghost-db-mysql-0007
|
||||
|
||||
**Hubot**[@shlomi-noach][25] 正在部署 gh-ost/fix-reappearing-throttled-reasons (baee4f6) 到生产中 (ghost-db-mysql-0007).
|
||||
[@shlomi-noach][26] gh-ost/fix-reappearing-throttled-reasons (baee4f6) 的生产部署已经完成了！ (2s)
|
||||
[@shlomi-noach][27] 确保你在 haystack 中看到异常
|
||||
|
||||
**jonahberquist**.deploy gh-ost/interactive-command-question to prod/ghost-db-mysql-0012
|
||||
|
||||
**Hubot**[@jonahberquist][28] 正在部署 gh-ost/interactive-command-question (be1ab17) 到生产 (ghost-db-mysql-0012).
|
||||
[@jonahberquist][29] gh-ost/interactive-command-question (be1ab17) 的生产部署已经完成了！(2s)
|
||||
[@jonahberquist][30] 确保你在 haystack 中看到异常
|
||||
|
||||
**shlomi-noach**.wcid gh-ost
|
||||
|
||||
**Hubot**shlomi-noach 测试 fix-reappearing-throttled-reasons 41 秒前: ghost-db-mysql-0007
|
||||
jonahberquist 测试 interactive-command-question 7 秒前: ghost-db-mysql-0012
|
||||
|
||||
没人在排队。
|
||||
|
||||
一些 PR 很小，不影响数据本身。对状态消息，交互式命令等的更改对 `gh-ost` 应用程序的影响较小 。其他对迁移逻辑和操作造成重大变化。我们将严格测试这些，通过我们的生产表车队，直到满足这些变化不会造成数据损坏的威胁。
|
||||
|
||||
### 总结[][46]
|
||||
|
||||
在整个测试过程中，我们建立对系统的信任。通过自动化这些测试，在生产中，我们得到重复的确认，一切都按预期工作。随着我们继续发展我们的基础设施，我们还通过调整测试来覆盖最新的变化。
|
||||
|
||||
生产总是令人惊奇的，不包括测试的场景。我们对生产环境的测试越多，我们对应用程序的期望越多，基础设施的能力就越强。
|
||||
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://githubengineering.com/mysql-testing-automation-at-github/
|
||||
|
||||
作者：[tomkrouper ][a], [Shlomi Noach][b]
|
||||
译者：[MonkeyDEcho](https://github.com/MonkeyDEcho)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://github.com/tomkrouper
|
||||
[b]:https://github.com/shlomi-noach
|
||||
[1]:https://github.com/tomkrouper
|
||||
[2]:https://github.com/jessbreckenridge
|
||||
[3]:https://github.com/jessbreckenridge
|
||||
[4]:https://github.com/jessbreckenridge
|
||||
[5]:https://github.com/jessbreckenridge
|
||||
[6]:https://github.com/jessbreckenridge
|
||||
[7]:https://github.com/jessbreckenridge
|
||||
[8]:https://github.com/jessbreckenridge
|
||||
[9]:https://github.com/jessbreckenridge
|
||||
[10]:https://github.com/jessbreckenridge
|
||||
[11]:https://github.com/jessbreckenridge
|
||||
[12]:https://github.com/jessbreckenridge
|
||||
[13]:https://github.com/jessbreckenridge
|
||||
[14]:https://github.com/jessbreckenridge
|
||||
[15]:https://github.com/jessbreckenridge
|
||||
[16]:https://github.com/jessbreckenridge
|
||||
[17]:https://github.com/jessbreckenridge
|
||||
[18]:https://github.com/jessbreckenridge
|
||||
[19]:https://github.com/jessbreckenridge
|
||||
[20]:https://github.com/jessbreckenridge
|
||||
[21]:https://github.com/jessbreckenridge
|
||||
[22]:https://github.com/jessbreckenridge
|
||||
[23]:https://github.com/jessbreckenridge
|
||||
[24]:https://github.com/jessbreckenridge
|
||||
[25]:https://github.com/shlomi-noach
|
||||
[26]:https://github.com/shlomi-noach
|
||||
[27]:https://github.com/shlomi-noach
|
||||
[28]:https://github.com/jonahberquist
|
||||
[29]:https://github.com/jonahberquist
|
||||
[30]:https://github.com/jonahberquist
|
||||
[31]:https://githubengineering.com/mysql-testing-automation-at-github/
|
||||
[32]:https://github.com/tomkrouper
|
||||
[33]:https://github.com/tomkrouper
|
||||
[34]:https://github.com/shlomi-noach
|
||||
[35]:https://github.com/shlomi-noach
|
||||
[36]:https://githubengineering.com/mysql-testing-automation-at-github/#backups
|
||||
[37]:https://www.percona.com/software/mysql-database/percona-xtrabackup
|
||||
[38]:https://dev.mysql.com/doc/refman/5.6/en/replication-delayed.html
|
||||
[39]:https://githubengineering.com/mysql-testing-automation-at-github/#failovers
|
||||
[40]:http://githubengineering.com/orchestrator-github/
|
||||
[41]:https://githubengineering.com/context-aware-mysql-pools-via-haproxy/
|
||||
[42]:https://githubengineering.com/mysql-testing-automation-at-github/#schema-migrations
|
||||
[43]:http://githubengineering.com/gh-ost-github-s-online-migration-tool-for-mysql/
|
||||
[44]:https://github.com/github/gh-ost/pulls
|
||||
[45]:https://githubengineering.com/deploying-branches-to-github-com/
|
||||
[46]:https://githubengineering.com/mysql-testing-automation-at-github/#summary
|
||||
@@ -1,51 +0,0 @@
|
||||
#IoT边缘计算框架的新进展
|
||||
---
|
||||
|
||||
|
||||
开源项目 EdgeX Foundry 旨在开发一个标准化的互操作物联网边缘计算框架.[使用权限获取](https://www.linux.com/licenses/category/used-permission).
|
||||
|
||||
在4月, Linux 基金组织[启动](http://linuxgizmos.com/open-source-group-focuses-on-industrial-iot-gateway-middleware/)了开发一个标准化互操作物联网边缘计算框架的开源项目[EdgeX Foundry](https://www.edgexfoundry.org/). 就在最近, EdgeX Foundry 又[宣布](https://www.edgexfoundry.org/announcement/2017/07/17/edgex-foundry-builds-momentum-for-a-iot-interoperability-and-a-unified-marketplace-with-eight-new-members/)新增 8 个成员, 其总成员达到 58.
|
||||
|
||||
这些新成员是 Absolute, IoT Impact LABS, inwinStack, Parallel Machines, Queen's University Belfast, RIOT, Toshiba Digital Solutions Corporation, 和 Tulip Interfaces. 其原有成员包括 AMD, Analog Devices, Canonical/Ubuntu, Cloud Foundry, Dell, Linaro, Mocana, NetFoundry, Opto 22, RFMicron 和 VMWare 等其他公司或组织.
|
||||
|
||||
戴尔贡献出其基于 Apache2.0 协议的[FUSE](https://medium.com/@gigastacey/dell-plans-an-open-source-iot-stack-3dde43f24feb)框架源码作为 EdgeX Foundry 项目的种子，其中包括十几个微服务和超过 12.5 万行代码. Linux 基金会和 Dell 将合并 FUSE 和 AllJoyn-compliant IoTX 项目, 后者是由现有 EdgeX Foundry 成员 Two Bulls 和 Beechwood 发起的与 FUSE 相似的一个项目. 待合并完成 Linux 基金组织将正式宣布启动 EdgeX Foundry 项目.
|
||||
|
||||
EdgeX Foundry 将创造一个互操作性的, 即插即用的物联网边缘计算组件生态系统. 开源 EdgeX 栈将协调多样的传感器网络与后台数据处理云平台间的消息协议. 该框架旨在充分挖掘横跨边缘计算, 安全, 系统管理和微服务等模块间的通用代码.
|
||||
|
||||
对于项目成员及其客户来说, 其关注焦点在于借助于 IoT 网关和智能边缘设备,预认证的软件可方便集成的可能性. 在 Linux.com 的一次采访中, [IoT Impact LABS](https://iotimpactlabs.com/) 的首席工程师, Dan Mahoney 说:"现实中, EdgeX Foundry 降低我们在部署囊括多供应商解决方案时所面对的挑战."
|
||||
|
||||
Linux 基金组织,在将 AllSeen Alliance 的 AllJoyn 项目合并到 [IoTivity](https://www.linux.com/news/how-iotivity-and-alljoyn-could-combine) 的情况下, 为什么Linux基金组织发起了另外一个物联网标准化项目 (EdgeX Foundry)? 原因之一, EdgeX Foundry 不同于 IoTivity, IoTivity 主要解决工业物联网问题, 而 EdgeX Foundry 旨在一站解决消费级和工业级物联网全部的问题. 更具体来说, EdgeX Foundry 旨在成为网关和智能终端的通用中间件. EdgeX Foundry 与 IoTivity 的另一个不同在于, 前者希望借助预连接的终端塑造一种新产品, 后者更多解决现存产品之间的互操作性.
|
||||
|
||||
Linux 基金会 IoT 高级总监 Philip DesAutels 说:"IoTivity 提供实现设备之间无缝连接的协议, 而 EdgeX Foundry 提供了一个边缘计算框架. EdgeX Foundry 能够兼容如 IoTivity, BacNet, EtherCat 等任何协议设备, 从而实现集成多协议通信系统的通用边缘计算框架, 该项目的目标是为构建互操作组件的生态系统的过程中, 降低不确定性, 缩短市场化时间, 更好地产生规模效应."
|
||||
|
||||
上个月, 由 [Open Connectivity Foundation](https://openconnectivity.org/developer/specifications/international-standards) (OCF) 和 Linux 基金组织共同发起的 IoTivity项目发布了 [IoTivity 1.3](https://wiki.iotivity.org/release_note_1.3.0), 该版本增加 了与其曾经的对手 AllJoyn spec 的纽带, 也增加了对于 OCF 的 UPnP 设备的接口. 预计在 [IoTivity 2.0](https://www.linux.com/news/iotivity-20-whats-store) 中, IoTivity 和 AllJoyn 将会更进一步深入集成.
|
||||
|
||||
DesAutels 告诉 linux.com, IoTivity 和 EdgeX 是高度互补的, 其原因是 EdgeX 项目和IoTivity 项目有好几个共同成员, 如此更强化了 IoTivity 和 EdgeX 的互补关系.
|
||||
|
||||
尽管 IoTivity 和 EdgeX 都宣称是跨平台,包括 CPU 架构和 OS, 但是二者还是存在一定区别. IoTivity 最初是基于 Linux 平台设计, 兼容 Ubuntu, Tizen 和 Android 等 Linux 系列 OS, 后来逐步扩展到 Windows 和 IOS 操作系统. 与之对应的 EdgeX 设计之初就是基于跨平台的理念, 其完美兼容于各种 CPU 架构, 以及 Linux, Windows 和 Mac OS 等操作系统. 未来还将兼容于实时操作系统(RTOSes).
|
||||
|
||||
EdgeX 的新成员 [RIOT](https://riot-os.org/) 提供了一个开源项目 RIOT RTOS. RIOT 的主要维护者 Thomas Eichinger 在一次重要报告时说:"由于 RIOT 初衷就是致力于解决 linux 不太适应的问题, 故对于 RIOT 社区来说,参加和支持类似于 EdgeX Foundry 等与 Linux 互补性社区的积极性是自然而然的."
|
||||
|
||||
##传感器集成的简化
|
||||
IoT Impact LABS (也叫aka impact LABS 或直接称为 LABS) 是另一个 EdgeX 新成员. 该公司推出了一个独特的业务, 旨在帮助中小企业度过物联网解决方案的试用阶段. 该公司的大部分客户, 其中包括几个 EdgeX Foundry 的项目成员, 是致力于建设智慧城市, 基础设施再利用, 提高食品安全, 以及解决会社面临的自然资源缺乏的挑战.
|
||||
|
||||
Dan Mahoney 说:"在 LABS 我们花费了很多时间来调和试点客户的解决方案之间的差异性. EdgeX Foundry 可以最小化部署边缘软件系统的工作量,从而使我们能够更快更好地部署高质量的解决方案."
|
||||
|
||||
该框架在涉及多个供应商, 多种类型传感器的场景尤其凸显优势. "Edgex Foundry 将为我们提供快速构建网关的能力, 以及快速部署传感器的能力." Mahoney 补充说到. 传感器制造商将借助 EdgeX SDK 烧写应用层协议驱动到边缘设备, 该协议能够兼容多供应商和解决方案.
|
||||
|
||||
##边缘分析能力的构建
|
||||
当我们问到, Mahoney 的公司想要见到 EdgeX Foundry 怎样的发展时, 他说:"我们喜见乐闻的一个目标是有更多有效的工业协议作为设备服务出现, 一个更清晰的边缘计算实现路径."
|
||||
|
||||
在工业物联网和消费级物联网中边缘计算都呈现增长趋势. 在后者, 我们已经看到如 Alexa 的智能声控以及录像分析等几个智能家居系统集成了边缘计算分析技术. 这减轻了云服务平台的计算负荷, 但同时也带来了安全, 隐私, 以及由于政策和供应商中断引起的服务中断问题.
|
||||
|
||||
对于工业物联网网关, 隐私问题成为首要的问题. 因此, 在物联网网关方面出现了一些类似于云服务功能的扩展. 其中一个解决方案是, 为了安全将一些云服务上的安全保障应用借助容器如 [RIOS 与 Ubuntu 内核快照机制](https://www.linux.com/news/future-iot-containers-aim-solve-security-crisis)等方式集成到嵌入式设备. 另一种方案是, 开发 IoT 系统迁移云功能到边缘. 上个月, Amazon 为基于 linux 的网关发布了实现 [AWS Greengrass](http://linuxgizmos.com/amazon-releases-aws-greengrass-for-local-iot-processing-on-linux-devices/) 物联网协议栈的 AWS lambda. 该软件能够使计算, 消息路由, 数据收集和同步能力在边缘设备上完成,如物联网网关.
|
||||
|
||||
分析能力是 EdgeX Foundry 的一个关键功能要点. 发起成员 Cloud Foundry 是旨在集成其主要的工业应用平台到边缘设备. 另一个新成员 [Parallel Machines](https://www.parallelmachines.com/) 计划利用EdgeX将AI带到边缘设备.
|
||||
|
||||
EdgeX Foundry 仍然在项目早期, 软件仍然在 α 阶段, 其成员在上个月才刚刚进行了第一次全体成员大会. 同时项目已经为新开发者准备了一些初始训练课程, 另外从[这里](https://wiki.edgexfoundry.org/)也能获取更多的信息.
|
||||
|
||||
原文连接: [https://www.linux.com/blog/2017/7/iot-framework-edge-computing-gains-ground](https://www.linux.com/blog/2017/7/iot-framework-edge-computing-gains-ground)
|
||||
|
||||
作者： [ERIC BROWN](https://www.linux.com/users/ericstephenbrown) 译者：penghuster 校对：校对者ID
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
@@ -0,0 +1,335 @@
|
||||
开发一个 Linux 调试器（九）：处理变量
|
||||
============================================================
|
||||
|
||||
变量是偷偷摸摸的。有时，它们会很高兴地呆在寄存器中，但是一转头就会跑到堆栈中。为了优化，编译器可能会完全将它们从窗口中抛出。无论变量在内存中的移动频率如何，我们都需要一些方法在调试器中跟踪和操作它们。这篇文章将会教你如何处理调试器中的变量，并使用 `libelfin` 演示一个简单的实现。
|
||||
|
||||
* * *
|
||||
|
||||
### 系列文章索引
|
||||
|
||||
1. [设置][1]
|
||||
|
||||
2. [断点][2]
|
||||
|
||||
3. [寄存器和内存][3]
|
||||
|
||||
4. [ELF 和 DWARF][4]
|
||||
|
||||
5. [源和信号][5]
|
||||
|
||||
6. [源码级单步调试][6]
|
||||
|
||||
7. [源码级断点][7]
|
||||
|
||||
8. [堆栈展开][8]
|
||||
|
||||
9. [处理变量][9]
|
||||
|
||||
10. [高级话题][10]
|
||||
|
||||
* * *
|
||||
|
||||
在开始之前，请确保你使用的 `libelfin` 版本是[我分支上的 `fbreg`][11]。这包含了一些 hack 来支持获取当前堆栈帧的基址并评估位置列表，这些都不是由原生的 `libelfin` 提供的。你可能需要给 GCC 传递 `-gdwarf-2` 参数使其生成兼容的 DWARF 信息。但是在实现之前，我将详细说明 DWARF 5 最新规范中的位置编码方式。如果你想要了解更多信息，那么你可以从[这里][12]获取标准。
|
||||
|
||||
### DWARF 未知
|
||||
|
||||
使用 `DW_AT_location` 属性在 DWARF 信息中编码给定时刻内存中变量的位置。位置描述可以是单个位置描述，复合位置描述或位置列表。
|
||||
|
||||
* 简单的位置描述描述对象的一个连续的部分（通常是所有）的位置。简单位置描述可以描述可寻址存储器或寄存器中的位置，或缺少位置（具有或不具有已知值）。
|
||||
* 比如：
|
||||
* `DW_OP_fbreg -32`
|
||||
|
||||
* 一个完全存储的变量 - 从堆栈帧基址开始的32个字节
|
||||
|
||||
* 复合位置描述根据片段描述对象，每个对象可以包含在寄存器的一部分中或存储在与其他片段无关的存储器位置中。
|
||||
* 比如：
|
||||
* `DW_OP_reg3 DW_OP_piece 4 DW_OP_reg10 DW_OP_piece 2`
|
||||
|
||||
* 前四个字节位于寄存器 3 中，后两个字节位于寄存器 10 中的一个变量。
|
||||
|
||||
* 位置列表描述了具有有限周期或在周期内更改位置的对象。
|
||||
* 比如：
|
||||
* `<loclist with 3 entries follows>`
|
||||
* `[ 0]<lowpc=0x2e00><highpc=0x2e19>DW_OP_reg0`
|
||||
|
||||
* `[ 1]<lowpc=0x2e19><highpc=0x2e3f>DW_OP_reg3`
|
||||
|
||||
* `[ 2]<lowpc=0x2ec4><highpc=0x2ec7>DW_OP_reg2`
|
||||
|
||||
* 根据程序计数器的当前值，位置在寄存器之间移动的变量
|
||||
|
||||
根据位置描述的种类，`DW_AT_location` 以三种不同的方式进行编码。`exprloc` 编码简单和复合的位置描述。它们由一个字节长度组成，后跟一个 DWARF 表达式或位置描述。`loclist` 和 `loclistptr` 的编码位置列表。它们在 `.debug_loclists` 部分中提供索引或偏移量，该部分描述了实际的位置列表。
|
||||
|
||||
### DWARF 表达式
|
||||
|
||||
使用 DWARF 表达式计算变量的实际位置。这包括操作堆栈值的一系列操作。有很多 DWARF 操作可用，所以我不会详细解释它们。相反，我会从每一个表达式中给出一些例子，给你一个可用的东西。另外，不要害怕这些; `libelfin` 将为我们处理所有这些复杂性。
|
||||
|
||||
* 字面编码
|
||||
* `DW_OP_lit0`、`DW_OP_lit1`。。。`DW_OP_lit31`
|
||||
* 将字面值压入堆栈
|
||||
|
||||
* `DW_OP_addr <addr>`
|
||||
* 将地址操作数压入堆栈
|
||||
|
||||
* `DW_OP_constu <unsigned>`
|
||||
* 将无符号值压入堆栈
|
||||
|
||||
* 寄存器值
|
||||
* `DW_OP_fbreg <offset>`
|
||||
* 压入在堆栈帧基址找到的值，偏移给定值
|
||||
|
||||
* `DW_OP_breg0`、`DW_OP_breg1`。。。 `DW_OP_breg31 <offset>`
|
||||
* 将给定寄存器的内容加上给定的偏移量压入堆栈
|
||||
|
||||
* 堆栈操作
|
||||
* `DW_OP_dup`
|
||||
* 复制堆栈顶部的值
|
||||
|
||||
* `DW_OP_deref`
|
||||
* 将堆栈顶部视为内存地址，并将其替换为该地址的内容
|
||||
|
||||
* 算术和逻辑运算
|
||||
* `DW_OP_and`
|
||||
* 弹出堆栈顶部的两个值，并压回它们的逻辑 `AND`
|
||||
|
||||
* `DW_OP_plus`
|
||||
* 与 `DW_OP_and` 相同，但是会添加值
|
||||
|
||||
* 控制流操作
|
||||
* `DW_OP_le`、`DW_OP_eq`、`DW_OP_gt` 等
|
||||
* 弹出前两个值，比较它们，并且如果条件为真，则压入 `1`，否则为 `0`
|
||||
|
||||
* `DW_OP_bra <offset>`
|
||||
* 条件分支：如果堆栈的顶部不是 `0`，则通过 `offset` 在表达式中向后或向后跳过
|
||||
|
||||
* 输入转化
|
||||
* `DW_OP_convert <DIE offset>`
|
||||
* 将堆栈顶部的值转换为不同的类型，它由给定偏移量的 DWARF 信息条目描述
|
||||
|
||||
* 特殊操作
|
||||
* `DW_OP_nop`
|
||||
* 什么都能不做！
|
||||
|
||||
### DWARF 类型
|
||||
|
||||
DWARF 的类型表示需要足够强大来为调试器用户提供有用的变量表示。用户经常希望能够在应用程序级别进行调试，而不是在机器级别进行调试，并且他们需要了解他们的变量正在做什么。
|
||||
|
||||
DWARF 类型与大多数其他调试信息一起编码在 DIE 中。它们可以具有指示其名称、编码、大小、字节等的属性。无数的类型标签可用于表示指针、数组、结构体、typedef 以及 C 或 C++ 程序中可以看到的任何其他内容。
|
||||
|
||||
以这个简单的结构体为例：
|
||||
|
||||
```
|
||||
struct test{
|
||||
int i;
|
||||
float j;
|
||||
int k[42];
|
||||
test* next;
|
||||
};
|
||||
```
|
||||
|
||||
这个结构体的父 DIE 是这样的：
|
||||
|
||||
```
|
||||
< 1><0x0000002a> DW_TAG_structure_type
|
||||
DW_AT_name "test"
|
||||
DW_AT_byte_size 0x000000b8
|
||||
DW_AT_decl_file 0x00000001 test.cpp
|
||||
DW_AT_decl_line 0x00000001
|
||||
|
||||
```
|
||||
|
||||
上面说的是我们有一个叫做 `test` 的结构体，大小为 `0xb8`，在 `test.cpp` 的第 `1` 行声明。接下来有许多描述成员的子 DIE。
|
||||
|
||||
```
|
||||
< 2><0x00000032> DW_TAG_member
|
||||
DW_AT_name "i"
|
||||
DW_AT_type <0x00000063>
|
||||
DW_AT_decl_file 0x00000001 test.cpp
|
||||
DW_AT_decl_line 0x00000002
|
||||
DW_AT_data_member_location 0
|
||||
< 2><0x0000003e> DW_TAG_member
|
||||
DW_AT_name "j"
|
||||
DW_AT_type <0x0000006a>
|
||||
DW_AT_decl_file 0x00000001 test.cpp
|
||||
DW_AT_decl_line 0x00000003
|
||||
DW_AT_data_member_location 4
|
||||
< 2><0x0000004a> DW_TAG_member
|
||||
DW_AT_name "k"
|
||||
DW_AT_type <0x00000071>
|
||||
DW_AT_decl_file 0x00000001 test.cpp
|
||||
DW_AT_decl_line 0x00000004
|
||||
DW_AT_data_member_location 8
|
||||
< 2><0x00000056> DW_TAG_member
|
||||
DW_AT_name "next"
|
||||
DW_AT_type <0x00000084>
|
||||
DW_AT_decl_file 0x00000001 test.cpp
|
||||
DW_AT_decl_line 0x00000005
|
||||
DW_AT_data_member_location 176(as signed = -80)
|
||||
|
||||
```
|
||||
|
||||
每个成员都有一个名称，一个类型（它是一个 DIE 偏移量），一个声明文件和行，以及一个字节偏移到该成员所在的结构体中。类型指向下一个。
|
||||
|
||||
```
|
||||
< 1><0x00000063> DW_TAG_base_type
|
||||
DW_AT_name "int"
|
||||
DW_AT_encoding DW_ATE_signed
|
||||
DW_AT_byte_size 0x00000004
|
||||
< 1><0x0000006a> DW_TAG_base_type
|
||||
DW_AT_name "float"
|
||||
DW_AT_encoding DW_ATE_float
|
||||
DW_AT_byte_size 0x00000004
|
||||
< 1><0x00000071> DW_TAG_array_type
|
||||
DW_AT_type <0x00000063>
|
||||
< 2><0x00000076> DW_TAG_subrange_type
|
||||
DW_AT_type <0x0000007d>
|
||||
DW_AT_count 0x0000002a
|
||||
< 1><0x0000007d> DW_TAG_base_type
|
||||
DW_AT_name "sizetype"
|
||||
DW_AT_byte_size 0x00000008
|
||||
DW_AT_encoding DW_ATE_unsigned
|
||||
< 1><0x00000084> DW_TAG_pointer_type
|
||||
DW_AT_type <0x0000002a>
|
||||
|
||||
```
|
||||
|
||||
如你所见，我笔记本电脑上的 `int` 是一个 4 字节的有符号整数类型，`float`是一个 4 字节的浮点数。整数数组类型通过指向 `int` 类型作为其元素类型，`sizetype`（可以认为是 `size_t`）作为索引类型，它具有 `2a` 个元素。 `test *` 类型是 `DW_TAG_pointer_type`，它引用 `test` DIE。
|
||||
|
||||
* * *
|
||||
|
||||
### 实现简单的变量读取器
|
||||
|
||||
如上所述，`libelfin` 将处理我们大部分的复杂性。但是，它并没有实现用于表示可变位置的所有不同方法，并且在我们的代码中处理这些将变得非常复杂。因此，我现在选择只支持 `exprloc`。请随意添加对更多类型表达式的支持。如果你真的有勇气，请提交补丁到 `libelfin` 中来帮助完成必要的支持！
|
||||
|
||||
处理变量主要是将不同部分定位在存储器或寄存器中，读取或写入与之前一样。为了简单起见，我只会告诉你如何实现读取。
|
||||
|
||||
首先我们需要告诉 `libelfin` 如何从我们的进程中读取寄存器。我们创建一个继承自 `expr_context` 的类并使用 `ptrace` 来处理所有内容：
|
||||
|
||||
```
|
||||
class ptrace_expr_context : public dwarf::expr_context {
|
||||
public:
|
||||
ptrace_expr_context (pid_t pid) : m_pid{pid} {}
|
||||
|
||||
dwarf::taddr reg (unsigned regnum) override {
|
||||
return get_register_value_from_dwarf_register(m_pid, regnum);
|
||||
}
|
||||
|
||||
dwarf::taddr pc() override {
|
||||
struct user_regs_struct regs;
|
||||
ptrace(PTRACE_GETREGS, m_pid, nullptr, ®s);
|
||||
return regs.rip;
|
||||
}
|
||||
|
||||
dwarf::taddr deref_size (dwarf::taddr address, unsigned size) override {
|
||||
//TODO take into account size
|
||||
return ptrace(PTRACE_PEEKDATA, m_pid, address, nullptr);
|
||||
}
|
||||
|
||||
private:
|
||||
pid_t m_pid;
|
||||
};
|
||||
```
|
||||
|
||||
读取将由我们 `debugger` 类中的 `read_variables` 函数处理：
|
||||
|
||||
```
|
||||
void debugger::read_variables() {
|
||||
using namespace dwarf;
|
||||
|
||||
auto func = get_function_from_pc(get_pc());
|
||||
|
||||
//...
|
||||
}
|
||||
```
|
||||
|
||||
我们上面做的第一件事是找到我们目前进入的函数，然后我们需要循环访问该函数中的条目来寻找变量：
|
||||
|
||||
```
|
||||
for (const auto& die : func) {
|
||||
if (die.tag == DW_TAG::variable) {
|
||||
//...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
我们通过查找 DIE 中的 `DW_AT_location` 条目获取位置信息：
|
||||
|
||||
```
|
||||
auto loc_val = die[DW_AT::location];
|
||||
```
|
||||
|
||||
接着我们确保它是一个 `exprloc`，并请求 `libelfin` 来评估我们的表达式：
|
||||
|
||||
```
|
||||
if (loc_val.get_type() == value::type::exprloc) {
|
||||
ptrace_expr_context context {m_pid};
|
||||
auto result = loc_val.as_exprloc().evaluate(&context);
|
||||
```
|
||||
|
||||
现在我们已经评估了表达式，我们需要读取变量的内容。它可以在内存或寄存器中，因此我们将处理这两种情况：
|
||||
|
||||
```
|
||||
switch (result.location_type) {
|
||||
case expr_result::type::address:
|
||||
{
|
||||
auto value = read_memory(result.value);
|
||||
std::cout << at_name(die) << " (0x" << std::hex << result.value << ") = "
|
||||
<< value << std::endl;
|
||||
break;
|
||||
}
|
||||
|
||||
case expr_result::type::reg:
|
||||
{
|
||||
auto value = get_register_value_from_dwarf_register(m_pid, result.value);
|
||||
std::cout << at_name(die) << " (reg " << result.value << ") = "
|
||||
<< value << std::endl;
|
||||
break;
|
||||
}
|
||||
|
||||
default:
|
||||
throw std::runtime_error{"Unhandled variable location"};
|
||||
}
|
||||
```
|
||||
|
||||
你可以看到，我根据变量的类型，打印输出了值而没有解释。希望通过这个代码，你可以看到如何支持编写变量，或者用给定的名字搜索变量。
|
||||
|
||||
最后我们可以将它添加到我们的命令解析器中：
|
||||
|
||||
```
|
||||
else if(is_prefix(command, "variables")) {
|
||||
read_variables();
|
||||
}
|
||||
```
|
||||
|
||||
### 测试一下
|
||||
|
||||
编写一些具有一些变量的小功能，不用优化并带有调试信息编译它，然后查看是否可以读取变量的值。尝试写入存储变量的内存地址，并查看程序改变的行为。
|
||||
|
||||
* * *
|
||||
|
||||
已经有九篇文章了，还剩最后一篇！下一次我会讨论一些你可能会感兴趣的更高级的概念。现在你可以在[这里][13]找到这个帖子的代码。
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://blog.tartanllama.xyz/writing-a-linux-debugger-variables/
|
||||
|
||||
作者：[ Simon Brand][a]
|
||||
译者：[geekpi](https://github.com/geekpi)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://www.twitter.com/TartanLlama
|
||||
[1]:https://blog.tartanllama.xyz/writing-a-linux-debugger-setup/
|
||||
[2]:https://blog.tartanllama.xyz/writing-a-linux-debugger-breakpoints/
|
||||
[3]:https://blog.tartanllama.xyz/writing-a-linux-debugger-registers/
|
||||
[4]:https://blog.tartanllama.xyz/writing-a-linux-debugger-elf-dwarf/
|
||||
[5]:https://blog.tartanllama.xyz/writing-a-linux-debugger-source-signal/
|
||||
[6]:https://blog.tartanllama.xyz/writing-a-linux-debugger-dwarf-step/
|
||||
[7]:https://blog.tartanllama.xyz/writing-a-linux-debugger-source-break/
|
||||
[8]:https://blog.tartanllama.xyz/writing-a-linux-debugger-unwinding/
|
||||
[9]:https://blog.tartanllama.xyz/writing-a-linux-debugger-variables/
|
||||
[10]:https://blog.tartanllama.xyz/writing-a-linux-debugger-advanced-topics/
|
||||
[11]:https://github.com/TartanLlama/libelfin/tree/fbreg
|
||||
[12]:http://dwarfstd.org/
|
||||
[13]:https://github.com/TartanLlama/minidbg/tree/tut_variable
|
||||
@@ -0,0 +1,150 @@
|
||||
开发一个 Linux 调试器（十）：高级主题
|
||||
============================================================
|
||||
|
||||
我们终于来到这个系列的最后一篇文章！这一次，我将对调试中的一些更高级的概念进行高层的概述：远程调试、共享库支持、表达式计算和多线程支持。这些想法实现起来比较复杂，所以我不会详细说明如何做，但是如果有的话，我很乐意回答有关这些概念的问题。
|
||||
|
||||
* * *
|
||||
|
||||
### 系列索引
|
||||
|
||||
1. [准备环境][1]
|
||||
|
||||
2. [断点][2]
|
||||
|
||||
3. [寄存器和内存][3]
|
||||
|
||||
4. [Elves 和 dwarves][4]
|
||||
|
||||
5. [源码和信号][5]
|
||||
|
||||
6. [源码层逐步执行][6]
|
||||
|
||||
7. [源码层断点][7]
|
||||
|
||||
8. [调用栈][8]
|
||||
|
||||
9. [处理变量][9]
|
||||
|
||||
10. [高级主题][10]
|
||||
|
||||
* * *
|
||||
|
||||
### 远程调试
|
||||
|
||||
远程调试对于嵌入式系统或不同环境的调试非常有用。它还在高级调试器操作和与操作系统和硬件的交互之间设置了一个很好的分界线。事实上，像 GDB 和 LLDB 这样的调试器即使在调试本地程序时也可以作为远程调试器运行。一般架构是这样的：
|
||||
|
||||
|
||||
|
||||
调试器是我们通过命令行交互的组件。也许如果你使用的是 IDE，那么顶层中另一个层可以通过_机器接口_与调试器进行通信。在目标机器上（可能与本机一样）有一个 _debug stub_ ，它理论上是一个非常小的操作系统调试库的包装程序，它执行所有的低级调试任务，如在地址上设置断点。我说“在理论上”，因为如今 stub 变得越来越大。例如，我机器上的 LLDB debug stub 大小是 7.6MB。debug stub 通过一些使用特定于操作系统的功能（在我们的例子中是 “ptrace”）和被调试进程以及通过远程协议的调试器通信。
|
||||
|
||||
最常见的远程调试协议是 GDB 远程协议。这是一种基于文本的数据包格式，用于在调试器和 debug
|
||||
stub 之间传递命令和信息。我不会详细介绍它，但你可以在[这里][11]阅读你想知道的。如果你启动 LLDB 并执行命令 `log enable gdb-remote packets`，那么你将获得通过远程协议发送的所有数据包的跟踪。在 GDB 上，你可以用 `set remotelogfile <file>` 做同样的事情。
|
||||
|
||||
作为一个简单的例子，这是数据包设置断点：
|
||||
|
||||
```
|
||||
$Z0,400570,1#43
|
||||
|
||||
```
|
||||
|
||||
`$` 标记数据包的开始。`Z0` 是插入内存断点的命令。`400570` 和 `1` 是参数，其中前者是设置断点的地址，后者是特定目标的断点类型说明符。最后，`#43` 是校验值，以确保数据没有损坏。
|
||||
|
||||
GDB 远程协议非常易于扩展自定义数据包，这对于实现平台或语言特定的功能非常有用。
|
||||
|
||||
* * *
|
||||
|
||||
### 共享库和动态加载支持
|
||||
|
||||
调试器需要知道调试程序加载了哪些共享库，以便它可以设置断点，获取源代码级别的信息和符号等。除查找被动态链接的库之外，调试器还必须跟踪在运行时通过 `dlopen` 加载的库。为了打到这个目的，动态链接器维护一个 _交会结构体_。该结构体维护共享描述符的链表以及指向每当更新链表时调用的函数的指针。这个结构存储在 ELF 文件的 `.dynamic` 段中，在程序执行之前被初始化。
|
||||
|
||||
一个简单的跟踪算法：
|
||||
|
||||
* 追踪程序在 ELF 头中查找程序的入口（或者可以使用存储在 `/proc/<pid>/aux` 中的辅助向量）
|
||||
|
||||
* 追踪程序在程序的入口处设置一个断点，并开始执行。
|
||||
|
||||
* 当到达断点时，通过在 ELF 文件中查找 `.dynamic` 的加载地址找到交汇结构体的地址。
|
||||
|
||||
* 检查交汇结构体以获取当前加载的库的列表。
|
||||
|
||||
* 链接器更新函数上设置断点
|
||||
|
||||
* 每当到达断点时，列表都会更新
|
||||
|
||||
* 追踪程序无限循环，继续执行程序并等待信号，直到追踪程序信号退出。
|
||||
|
||||
我给这些概念写了一个小例子，你可以在[这里][12]找到。如果有人有兴趣，我可以将来写得更详细一点。
|
||||
|
||||
* * *
|
||||
|
||||
### 表达式计算
|
||||
|
||||
表达式计算是程序的一项功能，允许用户在调试程序时对原始源语言中的表达式进行计算。例如，在 LLDB 或 GDB 中，可以执行 `print foo()` 来调用 `foo` 函数并打印结果。
|
||||
|
||||
根据表达的复杂程度，有几种不同的计算方法。如果表达式只是一个简单的标识符，那么调试器可以查看调试信息，找到变量并打印出该值，就像我们在本系列最后一部分中所做的那样。如果表达式有点复杂，则可能将代码编译成中间表达式 （IR） 并解释来获得结果。例如，对于某些表达式，LLDB 将使用 Clang 将表达式编译为 LLVM IR 并将其解释。如果表达式更复杂，或者需要调用某些函数，那么代码可能需要 JIT 到目标并在被调试者的地址空间中执行。这涉及到调用 `mmap` 来分配一些可执行内存，然后将编译的代码复制到该块并执行。LLDB 通过使用 LLVM 的 JIT 功能来实现。
|
||||
|
||||
如果你想更多地了解 JIT 编译，我强烈推荐[ Eli Bendersky 关于这个主题的文章][13]。
|
||||
|
||||
* * *
|
||||
|
||||
### 多线程调试支持
|
||||
|
||||
本系列展示的调试器仅支持单线程应用程序，但是为了调试大多数真实程序，多线程支持是非常需要的。支持这一点的最简单的方法是跟踪线程创建并解析 procfs 以获取所需的信息。
|
||||
|
||||
Linux 线程库称为 `pthreads`。当调用 `pthread_create` 时，库会使用 `clone` 系统调用来创建一个新的线程，我们可以用 `ptrace` 跟踪这个系统调用（假设你的内核早于 2.5.46）。为此，你需要在连接到调试器之后设置一些 `ptrace` 选项：
|
||||
|
||||
```
|
||||
ptrace(PTRACE_SETOPTIONS, m_pid, nullptr, PTRACE_O_TRACECLONE);
|
||||
```
|
||||
|
||||
现在当 `clone` 被调用时，该进程将收到我们的老朋友 `SIGTRAP` 发出信号。对于本系列中的调试器，你可以将一个例子添加到 `handle_sigtrap` 来处理新线程的创建：
|
||||
|
||||
```
|
||||
case (SIGTRAP | (PTRACE_EVENT_CLONE << 8)):
|
||||
//get the new thread ID
|
||||
unsigned long event_message = 0;
|
||||
ptrace(PTRACE_GETEVENTMSG, pid, nullptr, message);
|
||||
|
||||
//handle creation
|
||||
//...
|
||||
```
|
||||
|
||||
一旦收到了，你可以看看 `/proc/<pid>/task/` 并查看内存映射之类来获得所需的所有信息。
|
||||
|
||||
GDB 使用 `libthread_db`，它提供了一堆帮助函数，这样你就不需要自己解析和处理。设置这个库很奇怪，我不会在这展示它如何工作，但如果你想使用它，你可以去阅读[这个教程][14]。
|
||||
|
||||
多线程支持中最复杂的部分是调试器中线程状态的建模，特别是如果你希望支持[不间断模式][15]或当你计算中涉及不止一个 CPU 的某种异构调试。
|
||||
|
||||
* * *
|
||||
|
||||
### 最后！
|
||||
|
||||
呼！这个系列花了很长时间才写完，但是我在这个过程中学到了很多东西，我希望它是有帮助的。如果你聊有关调试或本系列中的任何问题，请在 Twitter [@TartanLlama][16]或评论区联系我。如果你有想看到的其他任何调试主题，让我知道我或许会再发其他的文章。
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://blog.tartanllama.xyz/writing-a-linux-debugger-advanced-topics/
|
||||
|
||||
作者：[Simon Brand ][a]
|
||||
译者：[geekpi](https://github.com/geekpi)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://www.twitter.com/TartanLlama
|
||||
[1]:https://blog.tartanllama.xyz/writing-a-linux-debugger-setup/
|
||||
[2]:https://blog.tartanllama.xyz/writing-a-linux-debugger-breakpoints/
|
||||
[3]:https://blog.tartanllama.xyz/writing-a-linux-debugger-registers/
|
||||
[4]:https://blog.tartanllama.xyz/writing-a-linux-debugger-elf-dwarf/
|
||||
[5]:https://blog.tartanllama.xyz/writing-a-linux-debugger-source-signal/
|
||||
[6]:https://blog.tartanllama.xyz/writing-a-linux-debugger-dwarf-step/
|
||||
[7]:https://blog.tartanllama.xyz/writing-a-linux-debugger-source-break/
|
||||
[8]:https://blog.tartanllama.xyz/writing-a-linux-debugger-unwinding/
|
||||
[9]:https://blog.tartanllama.xyz/writing-a-linux-debugger-variables/
|
||||
[10]:https://blog.tartanllama.xyz/writing-a-linux-debugger-advanced-topics/
|
||||
[11]:https://sourceware.org/gdb/onlinedocs/gdb/Remote-Protocol.html
|
||||
[12]:https://github.com/TartanLlama/dltrace
|
||||
[13]:http://eli.thegreenplace.net/tag/code-generation
|
||||
[14]:http://timetobleed.com/notes-about-an-odd-esoteric-yet-incredibly-useful-library-libthread_db/
|
||||
[15]:https://sourceware.org/gdb/onlinedocs/gdb/Non_002dStop-Mode.html
|
||||
[16]:https://twitter.com/TartanLlama
|
||||
@@ -1,91 +0,0 @@
|
||||
创建更好的灾难恢复计划
|
||||
============================================================
|
||||
|
||||
Tanya Reilly 的五个问题：相互依赖的服务如何使恢复更加困难，为什么有意并预先管理依赖是个好主意。
|
||||
|
||||
我最近询问 Google 的网站可靠性工程师 Tanya Reilly 分享她关于如何制定更好的灾难恢复计划的想法。Tanya 在 10 月 1 日到 4 日在纽约举行的 O'Reilly Velocity Conference 上发表了一个题为[_你有没有尝试把它关闭之后再打开？] [9]的演讲。
|
||||
|
||||
### 1\. 在计划备份系统策略时，人们最常犯的错误是什么？
|
||||
|
||||
经典的一条是“你不需要备份策略，你需要一个恢复策略”。如果你有备份，但你尚未测试恢复它们，那么你没有真正的备份。测试不仅仅意味着知道你可以得到数据，还这意味着知道如何把它放回数据库，如何处理增量更改，如果你需要的话，如何重新安装整个系统。这意味着确保你的恢复路径不依赖于与数据同时丢失的某些系统。
|
||||
|
||||
但测试恢复是枯燥的。这是人们在忙碌时会偷工减料的那种事情。这值得花时间使其尽可能简单、无痛、自动化，永远不要靠任何人的意志力！同时，你必须确保有关人员知道该怎么做，所以定期进行大规模的灾难测试是很好的。恢复练习是找出该过程的文档是否缺失或过期的好方法，或者你没有足够的资源（磁盘、网络等）来传输和重新插入数据。
|
||||
|
||||
### 2\. 创建灾难恢复 （DR） 计划最常见的挑战是什么？
|
||||
|
||||
我认为很多 DR 是一种事后的想法：“我们有这个很棒的系统，我们的业务依赖它。。。我猜我们应该为它做 DR 吗？”而且到那时，系统会非常复杂，充满相互依赖关系，很难复制。
|
||||
|
||||
第一次安装的东西，它通常是由人手动调整并正常工作的，有时是相同的版本。当你构建_第二_个时，很难确定它是完全一样的。即使在具有严格配置管理的站点中，你也可以将某些内容留下，或者让其过期。
|
||||
|
||||
例如，如果你已经失去对解密密钥的访问权限，那么加密备份没有太多用处。而且任何只在灾难中使用的部分都可能会因为上次检查它们而破环。确保你已经涵盖所有东西的唯一方法做故障切换。当你准备好了的，就计划一下你的灾难吧！
|
||||
|
||||
如果你可以设计系统，以使灾难恢复模式成为正常运行的一部分，那么情况会更好。如果你的服务从一开始就被设计为可复制的，添加更多的副本就是一个常规的操作并可能是自动化的。没有新的方法，这只是一个容量问题。但是，系统中仍然存在一些只能在一个或两个地方运行的组件。偶然计划中的假灾难能将它们发现。
|
||||
|
||||
顺便说一句，那些被遗忘的组件可能包括仅在一个人的大脑中的信息，所以如果你自己发现说：“我们不能在 X 休假回来前进行 DR 故障切换测试”，那么那个人是一个危险的单点失败。
|
||||
|
||||
仅在灾难中使用的系统需要最多的测试，否则在需要时会失败。你有的越少越安全，且辛苦的测试工作也越少。
|
||||
|
||||
### 3\. 为什么服务相互依赖使得灾难恢复更加困难？
|
||||
|
||||
如果你只有一个二进制文件，那么恢复它是比较容易的：你开始二进制备份。但是我们越来越多地将通用功能分解成单独的服务。微服务意味着我们有更多的灵活性和更少地重新发明轮子：如果我们需要一个后台做一些事情，并且有一个已经存在，那么很好，我们就可以使用它。但是一些需要保留很大的依赖关系，因为它很快会变得纠缠。
|
||||
|
||||
#### 管理、成长，并推动您的系统
|
||||
|
||||
|
||||
你可能知道你直接使用的后端，但是你可能不会注意到有新的添加到你使用的库中。你可能依赖于它，它也间接依赖于你。在中断之后，你可能会遇到一个死锁：两个系统都不能启动，直到另一个运行并提供一些功能。这是一个困难的恢复情况！
|
||||
|
||||
你甚至可以最终得到间接依赖于自身的内容，例如你需要配置启动网络的设备，但在网络关闭时无法访问该设备。人们通常会提前考虑这些循环依赖，并且有某种后 备计划，但是这些本质上是不太行得通的路：它们只适用于极端情况，并且通过你的系统、进程或代码遵循不同的路径。这意味着，他们很可能有一个不会被发现的问题, 直到你真的, 真的需要他们的工作的时候。
|
||||
|
||||
### 4\. 你建议人们在认为需要之前，开始有意管理其依赖关系，以防止潜在的灾难性系统故障。为什么这很重要，你有什么建议有效地做到这一点？
|
||||
|
||||
管理你的依赖关系对于确保你可以从灾难中恢复至关重要。它使操作系统更容易。如果你的依赖不可靠，你就不可靠，所以你需要知道它们是什么。
|
||||
|
||||
它们变得混乱后也可以开始管理依赖关系，但是如果你早点开始，它会变得更容易一些。你可以设置使用各种服务策略-例如，你必须在堆栈中的这一层依赖于这组系统。你可以通过使其成为设计文件审查的常规部分，引入考虑依赖关系的文化。但请记住，依赖关系列表将很快变得陈旧。如果你有程序化的依赖关系发现，甚至依赖强制执行，这是最好的。 [我的 Velocity 谈话][10]涵盖了我们如何做到这一点。
|
||||
|
||||
早期开始的另一个优点是，你可以将服务拆分为垂直“层”，每个层次中的功能必须能够在下一个启动之前完全在线。所以，例如，你可以说网络必须能够完全启动而不使用任何其他服务。那么说，你的存储系统应该仅仅依赖于网络，程序后端应该仅仅依赖于网络和存储，等等。不同的层次对于不同的架构是有意义的。
|
||||
|
||||
如果你提前计划，新服务更容易选择依赖关系。每个服务应该只依赖堆栈中较低的服务。你仍然可以结束循环，在相同的层次服务上批次依赖 - 但是它们更加紧密地包含，并且在逐个情况基础上更容易处理。
|
||||
|
||||
### 5\. 你对 Velocity NY 的其他部分感兴趣么？
|
||||
|
||||
我整个星期二和星期三的时间表都完成了！正如你可能收集的那样，我非常关心使得巨大的相互依赖的系统可以管理，所以我期待听到[ Carin Meier 关于管理系统复杂性的想法][11]、[Sarah Wells 的微服务][12]和[ Baron 的可观察性][13]。我非常着迷听到[ Jon Moore 关于 Comcast 如何从年度发布到每天发布的故事][14]。作为一个前系统管理员，我很期待听到[ Bryan Liles 对这个职位走向的看法][15]。
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
作者简介：
|
||||
|
||||
Nikki McDonald
|
||||
|
||||
Nikki McDonald 是 O'Reilly Media，Inc.的内容总监。她住在密歇根州的安娜堡市。
|
||||
|
||||
Tanya Reilly
|
||||
|
||||
Tanya Reilly 自 2005 年以来一直是 Google 的系统管理员和站点可靠性工程师，致力于分布式锁、负载均衡和引导等底层基础架构。在加入 Google 之前，她是爱尔兰最大的 ISP eircom.net 的系统管理员，在这之前她是一个小型软件公司的整个 IT 部门。
|
||||
|
||||
----------------------------
|
||||
|
||||
via: https://www.oreilly.com/ideas/creating-better-disaster-recovery-plans
|
||||
|
||||
作者：[ Nikki McDonald][a],[Tanya Reilly][b]
|
||||
译者：[geekpi](https://github.com/geekpi)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://www.oreilly.com/people/nikki-mcdonald
|
||||
[b]:https://www.oreilly.com/people/5c97a-tanya-reilly
|
||||
[1]:https://pixabay.com/en/crane-baukran-load-crane-crane-arm-2436704/
|
||||
[2]:https://conferences.oreilly.com/velocity/vl-ny?intcmp=il-webops-confreg-reg-vlny17_new_site_right_rail_cta
|
||||
[3]:https://www.oreilly.com/people/nikki-mcdonald
|
||||
[4]:https://www.oreilly.com/people/5c97a-tanya-reilly
|
||||
[5]:https://conferences.oreilly.com/velocity/vl-ny?intcmp=il-webops-confreg-reg-vlny17_new_site_creating_better_disaster_recovery_plans_text_cta
|
||||
[6]:https://www.oreilly.com/ideas/creating-better-disaster-recovery-plans
|
||||
[7]:https://conferences.oreilly.com/velocity/vl-ny?intcmp=il-webops-confreg-reg-vlny17_new_site_right_rail_cta
|
||||
[8]:https://conferences.oreilly.com/velocity/vl-ny?intcmp=il-webops-confreg-reg-vlny17_new_site_right_rail_cta
|
||||
[9]:https://conferences.oreilly.com/velocity/vl-ny/public/schedule/detail/61400?intcmp=il-webops-confreg-reg-vlny17_new_site_creating_better_disaster_recovery_plans_body_text_cta
|
||||
[10]:https://conferences.oreilly.com/velocity/vl-ny/public/schedule/detail/61400?intcmp=il-webops-confreg-reg-vlny17_new_site_creating_better_disaster_recovery_plans_body_text_cta
|
||||
[11]:https://conferences.oreilly.com/velocity/vl-ny/public/schedule/detail/62779?intcmp=il-webops-confreg-reg-vlny17_new_site_creating_better_disaster_recovery_plans_body_text_cta
|
||||
[12]:https://conferences.oreilly.com/velocity/vl-ny/public/schedule/detail/61597?intcmp=il-webops-confreg-reg-vlny17_new_site_creating_better_disaster_recovery_plans_body_text_cta
|
||||
[13]:https://conferences.oreilly.com/velocity/vl-ny/public/schedule/detail/61630?intcmp=il-webops-confreg-reg-vlny17_new_site_creating_better_disaster_recovery_plans_body_text_cta
|
||||
[14]:https://conferences.oreilly.com/velocity/vl-ny/public/schedule/detail/62733?intcmp=il-webops-confreg-reg-vlny17_new_site_creating_better_disaster_recovery_plans_body_text_cta
|
||||
[15]:https://conferences.oreilly.com/velocity/vl-ny/public/schedule/detail/62893?intcmp=il-webops-confreg-reg-vlny17_new_site_creating_better_disaster_recovery_plans_body_text_cta
|
||||
@@ -1,343 +0,0 @@
|
||||
# 常用的 GDB 命令中文释义
|
||||
|
||||
## 目录
|
||||
|
||||
- [break](#break) -- 缩写 `b`，在指定的行或函数处设置断点
|
||||
- [info breakpoints](#info-breakpoints) -- 简写 `i b`，打印未删除的所有断点，观察点和捕获点的列表
|
||||
- [disable](#disable) -- 禁用断点，可以缩写为 `dis`
|
||||
- [enable](#enable) -- 启用断点
|
||||
- [clear](#clear) -- 清除指定行或函数处的断点
|
||||
- [delete](#delete) -- 缩写 `d`，删除断点
|
||||
- [tbreak](#tbreak) -- 设置临时断点，参数同 `break`，但在程序第一次停住后会被自动删除
|
||||
- [watch](#watch) -- 为表达式（或变量）设置观察点，当表达式（或变量）的值有变化时，停住程序
|
||||
- [step](#step) -- 缩写 `s`，单步跟踪，如果有函数调用，会进入该函数
|
||||
- [reverse-step](#reverse-step) -- 反向单步跟踪，如果有函数调用，会进入该函数
|
||||
- [next](#next) -- 缩写 `n`，单步跟踪，如果有函数调用，不会进入该函数
|
||||
- [reverse-next](#reverse-next) -- 反向单步跟踪，如果有函数调用，不会进入该函数
|
||||
- [return](#return) -- 使选定的栈帧返回到其调用者
|
||||
- [finish](#finish) -- 缩写 `fin`，执行直到选择的栈帧返回
|
||||
- [until](#until) -- 缩写 `u`，执行直到...（用于跳过循环、递归函数调用）
|
||||
- [continue](#continue) -- 同义词 `c`，恢复程序执行
|
||||
- [print](#print) -- 缩写 `p`，打印表达式 EXP 的值
|
||||
- [x](#x) -- 查看内存
|
||||
- [display](#display) -- 每次程序停止时打印表达式 EXP 的值（自动显示）
|
||||
- [info display](#info-display) -- 打印早先设置为自动显示的表达式列表
|
||||
- [disable display](#disable-display) -- 禁用自动显示
|
||||
- [enable display](#enable-display) -- 启用自动显示
|
||||
- [undisplay](#undisplay) -- 删除自动显示项
|
||||
- [help](#help) -- 缩写 `h`，打印命令列表（带参数时查找命令的帮助）
|
||||
- [attach](#attach) -- 挂接到已在运行的进程来调试
|
||||
- [run](#run) -- 缩写 `r`，启动被调试的程序
|
||||
- [backtrace](#backtrace) -- 缩写 `bt`，查看程序调用栈的信息
|
||||
- [ptype](#ptype) -- 打印类型 TYPE 的定义
|
||||
|
||||
------
|
||||
|
||||
## break
|
||||
|
||||
使用 `break` 命令（缩写 `b`）来设置断点。 参见[官方文档][1]。
|
||||
|
||||
- `break` 当不带参数时，在所选栈帧中执行的下一条指令处设置断点。
|
||||
- `break <function-name>` 在函数体入口处打断点，在 C++ 中可以使用 `class::function` 或 `function(type, ...)` 格式来指定函数名。
|
||||
- `break <line-number>` 在当前源码文件指定行的开始处打断点。
|
||||
- `break -N` `break +N` 在当前源码行前面或后面的 `N` 行开始处打断点，`N` 为正整数。
|
||||
- `break <filename:linenum>` 在源码文件 `filename` 的 `linenum` 行处打断点。
|
||||
- `break <filename:function>` 在源码文件 `filename` 的 `function` 函数入口处打断点。
|
||||
- `break <address>` 在程序指令的地址处打断点。
|
||||
- `break ... if <cond>` 设置条件断点，`...` 代表上述参数之一（或无参数），`cond` 为条件表达式，仅在 `cond` 值非零时停住程序。
|
||||
|
||||
## info breakpoints
|
||||
|
||||
查看断点，观察点和捕获点的列表。用法：
|
||||
`info breakpoints [list…]`
|
||||
`info break [list…]`
|
||||
`list…` 用来指定若干个断点的编号（可省略），可以是 `2`， `1-3`， `2 5` 等。
|
||||
|
||||
## disable
|
||||
|
||||
禁用一些断点。 参见[官方文档][2]。
|
||||
参数是用空格分隔的断点编号。
|
||||
要禁用所有断点，不加参数。
|
||||
禁用的断点不会被忘记，但直到重新启用才有效。
|
||||
用法： `disable [breakpoints] [list…]`
|
||||
`breakpoints` 是 `disable` 的子命令（可省略），`list…` 同 `info breakpoints` 中的描述。
|
||||
|
||||
## enable
|
||||
|
||||
启用一些断点。 参见[官方文档][2]。
|
||||
给出断点编号（以空格分隔）作为参数。
|
||||
没有参数时，所有断点被启用。
|
||||
|
||||
- `enable [breakpoints] [list…]` 启用指定的断点（或所有定义的断点）。
|
||||
- `enable [breakpoints] once list…` 临时启用指定的断点。GDB 在停止您的程序后立即禁用这些断点。
|
||||
- `enable [breakpoints] delete list…` 使指定的断点启用一次，然后删除。一旦您的程序停止，GDB 就会删除这些断点。等效于用 `tbreak` 设置的断点。
|
||||
|
||||
`breakpoints` 同 `disable` 中的描述。
|
||||
|
||||
## clear
|
||||
|
||||
在指定行或函数处清除断点。 参见[官方文档][3]。
|
||||
参数可以是行号，函数名称或 `*` 跟一个地址。
|
||||
|
||||
- `clear` 当不带参数时，清除所选栈帧在执行的源码行中的所有断点。
|
||||
- `clear <function>`, `clear <filename:function>` 删除在命名函数的入口处设置的任何断点。
|
||||
- `clear <linenum>`, `clear <filename:linenum>` 删除在指定的文件指定的行号的代码中设置的任何断点。
|
||||
- `clear <address>` 清除指定程序指令的地址处的断点。
|
||||
|
||||
## delete
|
||||
|
||||
删除一些断点或自动显示表达式。 参见[官方文档][3]。
|
||||
参数是用空格分隔的断点编号。
|
||||
要删除所有断点，不加参数。
|
||||
用法： `delete [breakpoints] [list…]`
|
||||
|
||||
## tbreak
|
||||
|
||||
设置临时断点。参数形式同 `break` 一样。 参见[官方文档][1]。
|
||||
除了断点是临时的之外像 `break` 一样，所以在命中时会被删除。
|
||||
|
||||
## watch
|
||||
|
||||
为表达式设置观察点。 参见[官方文档][4]。
|
||||
用法： `watch [-l|-location] <expr>`
|
||||
每当一个表达式的值改变时，观察点就会停止执行您的程序。
|
||||
如果给出了 `-l` 或者 `-location`，则它会对 `expr` 求值并观察它所指向的内存。
|
||||
例如，`watch *(int *)0x12345678` 将在指定的地址处观察一个 4 字节的区域（假设 int 占用 4 个字节）。
|
||||
|
||||
## step
|
||||
|
||||
单步执行程序，直到到达不同的源码行。 参见[官方文档][5]。
|
||||
用法： `step [N]`
|
||||
参数 `N` 表示执行 N 次（或由于另一个原因直到程序停止）。
|
||||
警告：如果当控制在没有调试信息的情况下编译的函数中使用 `step` 命令，则执行将继续进行，
|
||||
直到控制到达具有调试信息的函数。 同样，它不会进入没有调试信息编译的函数。
|
||||
要执行没有调试信息的函数，请使用 `stepi` 命令，后文再述。
|
||||
|
||||
## reverse-step
|
||||
|
||||
反向步进程序，直到到达另一个源码行的开头。 参见[官方文档][6]。
|
||||
用法： `reverse-step [N]`
|
||||
参数 `N` 表示执行 N 次（或由于另一个原因直到程序停止）。
|
||||
|
||||
## next
|
||||
|
||||
单步执行程序，执行完子程序调用。 参见[官方文档][5]。
|
||||
用法： `next [N]`
|
||||
与 `step` 不同，如果当前的源代码行调用子程序，则此命令不会进入子程序，而是继续执行，将其视为单个源代码行。
|
||||
|
||||
## reverse-next
|
||||
|
||||
反向步进程序，执行完子程序调用。 参见[官方文档][6]。
|
||||
用法： `reverse-next [N]`
|
||||
如果要执行的源代码行调用子程序，则此命令不会进入子程序，调用被视为一个指令。
|
||||
参数 `N` 表示执行 N 次（或由于另一个原因直到程序停止）。
|
||||
|
||||
## return
|
||||
|
||||
您可以使用 `return` 命令取消函数调用的执行。 参见[官方文档][7]。
|
||||
如果你给出一个表达式参数，它的值被用作函数的返回值。
|
||||
`return <expression>` 将 `expression` 的值作为函数的返回值并使函数直接返回。
|
||||
|
||||
## finish
|
||||
|
||||
执行直到选定的栈帧返回。 参见[官方文档][5]。
|
||||
用法： `finish`
|
||||
返回后，返回的值将被打印并放入到值历史记录中。
|
||||
|
||||
## until
|
||||
|
||||
执行直到程序到达大于当前栈帧或当前栈帧中的指定位置（与 [break](#break) 命令相同的参数）的源码行。 参见[官方文档][5]。
|
||||
此命令用于通过一个多次的循环，以避免单步执行。
|
||||
`until <location>` 或 `u <location>` 继续运行程序，直到达到指定的位置，或者当前栈帧返回。
|
||||
|
||||
## continue
|
||||
|
||||
在信号或断点之后，继续运行被调试的程序。 参见[官方文档][5]。
|
||||
用法： `continue [N]`
|
||||
如果从断点开始，可以使用数字 `N` 作为参数，这意味着将该断点的忽略计数设置为 `N - 1`(以便断点在第 N 次到达之前不会中断)。
|
||||
如果启用了非停止模式（使用 `show non-stop` 查看），则仅继续当前线程，否则程序中的所有线程都将继续。
|
||||
|
||||
## print
|
||||
|
||||
求值并打印表达式 EXP 的值。 参见[官方文档][8]。
|
||||
可访问的变量是所选栈帧的词法环境，以及范围为全局或整个文件的所有变量。
|
||||
用法： `print [expr]` 或 `print /f [expr]`
|
||||
`expr` 是一个（在源代码语言中的）表达式。
|
||||
默认情况下，`expr` 的值以适合其数据类型的格式打印；您可以通过指定 `/f` 来选择不同的格式，其中 `f` 是一个指定格式的字母；参见[输出格式][9]。
|
||||
如果省略 `expr`，GDB 再次显示最后一个值。
|
||||
要以每行一个成员带缩进的格式打印结构体变量请使用命令 `set print pretty on`，取消则使用命令 `set print pretty off`。
|
||||
可使用命令 `show print` 查看所有打印的设置。
|
||||
|
||||
## x
|
||||
|
||||
检查内存。 参见[官方文档][10]。
|
||||
用法： `x/nfu <addr>` 或 `x <addr>`
|
||||
`n`, `f`, 和 `u` 都是可选参数，用于指定要显示的内存以及如何格式化。
|
||||
`addr` 是要开始显示内存的地址的表达式。
|
||||
`n` 重复次数（默认值是 1），指定要显示多少个单位（由 `u` 指定）的内存值。
|
||||
`f` 显示格式（初始默认值是 `x`），显示格式是 `print('x'，'d'，'u'，'o'，'t'，'a'，'c'，'f'，'s')` 使用的格式之一，再加 `i`（机器指令）。
|
||||
`u` 单位大小，`b` 表示单字节，`h` 表示双字节，`w` 表示四字节，`g` 表示八字节。
|
||||
例如：
|
||||
`x/3uh 0x54320` 表示从地址 0x54320 开始以无符号十进制整数的方式，双字节为单位显示 3 个内存值。
|
||||
`x/16xb 0x7f95b7d18870` 表示从地址 0x7f95b7d18870 开始以十六进制整数的方式，单字节为单位显示 16 个内存值。
|
||||
|
||||
## display
|
||||
|
||||
每次程序停止时打印表达式 EXP 的值。 参见[官方文档][11]。
|
||||
用法： `display <expr>`, `display/fmt <expr>` 或 `display/fmt <addr>`
|
||||
`fmt` 用于指定显示格式。像 [print](#print) 命令里的 `/f` 一样。
|
||||
对于格式 `i` 或 `s`，或者包括单位大小或单位数量，将表达式 `addr` 添加为每次程序停止时要检查的内存地址。
|
||||
|
||||
## info display
|
||||
|
||||
打印自动显示的表达式列表，每个表达式都带有项目编号，但不显示其值。
|
||||
包括被禁用的表达式和不能立即显示的表达式（当前不可用的自动变量）。
|
||||
|
||||
## undisplay
|
||||
|
||||
取消某些表达式在程序停止时自动显示。
|
||||
参数是表达式的编号（使用 `info display` 查询编号）。
|
||||
不带参数表示取消所有自动显示表达式。
|
||||
`delete display` 具有与此命令相同的效果。
|
||||
|
||||
## disable display
|
||||
|
||||
禁用某些表达式在程序停止时自动显示。
|
||||
禁用的显示项目不会被自动打印，但不会被忘记。 它可能稍后再次被启用。
|
||||
参数是表达式的编号（使用 `info display` 查询编号）。
|
||||
不带参数表示禁用所有自动显示表达式。
|
||||
|
||||
## enable display
|
||||
|
||||
启用某些表达式在程序停止时自动显示。
|
||||
参数是重新显示的表达式的编号（使用 `info display` 查询编号）。
|
||||
不带参数表示启用所有自动显示表达式。
|
||||
|
||||
## help
|
||||
|
||||
打印命令列表。 参见[官方文档][12]。
|
||||
您可以使用不带参数的 `help`（缩写为 `h`）来显示命令的类别名的简短列表。
|
||||
使用 `help <class>` 您可以获取该类中各个命令的列表。
|
||||
使用 `help <command>` 显示如何使用该命令的简述。
|
||||
|
||||
## attach
|
||||
|
||||
挂接到 GDB 之外的进程或文件。 参见[官方文档][13]。
|
||||
该命令可以将进程 ID 或设备文件作为参数。
|
||||
对于进程 ID，您必须具有向进程发送信号的权限，并且必须具有与调试器相同的有效的 uid。
|
||||
用法： `attach <process-id>`
|
||||
GDB 在安排调试指定的进程之后做的第一件事是停住它。
|
||||
您可以使用所有通过 `run` 命令启动进程时可以使用的 GDB 命令来检查和修改挂接的进程。
|
||||
|
||||
## run
|
||||
|
||||
启动被调试的程序。 参见[官方文档][14]。
|
||||
可以直接指定参数，也可以用 [set args][15] 设置（启动所需的）参数。
|
||||
例如： `run arg1 arg2 ...` 等效于
|
||||
|
||||
```
|
||||
set args arg1 arg2 ...
|
||||
run
|
||||
```
|
||||
|
||||
还允许使用 `>`, `<`, 或 `>>` 进行输入和输出重定向。
|
||||
|
||||
## backtrace
|
||||
|
||||
打印整个栈的回溯。 参见[官方文档][16]。
|
||||
|
||||
- `bt` 打印整个栈的回溯，每个栈帧一行。
|
||||
- `bt n` 类似于上，但只打印最内层的 n 个栈帧。
|
||||
- `bt -n` 类似于上，但只打印最外层的 n 个栈帧。
|
||||
- `bt full n` 类似于 `bt n`，还打印局部变量的值。
|
||||
|
||||
`where` 和 `info stack`(缩写 `info s`) 是 `backtrace` 的别名。调用栈信息类似如下：
|
||||
|
||||
```
|
||||
(gdb) where
|
||||
#0 vconn_stream_run (vconn=0x99e5e38) at lib/vconn-stream.c:232
|
||||
#1 0x080ed68a in vconn_run (vconn=0x99e5e38) at lib/vconn.c:276
|
||||
#2 0x080dc6c8 in rconn_run (rc=0x99dbbe0) at lib/rconn.c:513
|
||||
#3 0x08077b83 in ofconn_run (ofconn=0x99e8070, handle_openflow=0x805e274 <handle_openflow>) at ofproto/connmgr.c:1234
|
||||
#4 0x08075f92 in connmgr_run (mgr=0x99dc878, handle_openflow=0x805e274 <handle_openflow>) at ofproto/connmgr.c:286
|
||||
#5 0x08057d58 in ofproto_run (p=0x99d9ba0) at ofproto/ofproto.c:1159
|
||||
#6 0x0804f96b in bridge_run () at vswitchd/bridge.c:2248
|
||||
#7 0x08054168 in main (argc=4, argv=0xbf8333e4) at vswitchd/ovs-vswitchd.c:125
|
||||
```
|
||||
|
||||
## ptype
|
||||
|
||||
打印类型 TYPE 的定义。 参见[官方文档][17]。
|
||||
用法： `ptype[/FLAGS] TYPE-NAME | EXPRESSION`
|
||||
参数可以是由 `typedef` 定义的类型名， 或者 `struct STRUCT-TAG` 或者 `class CLASS-NAME` 或者 `union UNION-TAG` 或者 `enum ENUM-TAG`。
|
||||
所选的栈帧的词法上下文用于查找该名字。
|
||||
类似的命令是 `whatis`，区别在于 `whatis` 不展开由 `typedef` 定义的数据类型，而 `ptype` 会展开，举例如下：
|
||||
|
||||
```
|
||||
/* 类型声明与变量定义 */
|
||||
typedef double real_t;
|
||||
struct complex {
|
||||
real_t real;
|
||||
double imag;
|
||||
};
|
||||
typedef struct complex complex_t;
|
||||
|
||||
complex_t var;
|
||||
real_t *real_pointer_var;
|
||||
```
|
||||
|
||||
这两个命令给出了如下输出：
|
||||
|
||||
```
|
||||
(gdb) whatis var
|
||||
type = complex_t
|
||||
(gdb) ptype var
|
||||
type = struct complex {
|
||||
real_t real;
|
||||
double imag;
|
||||
}
|
||||
(gdb) whatis complex_t
|
||||
type = struct complex
|
||||
(gdb) whatis struct complex
|
||||
type = struct complex
|
||||
(gdb) ptype struct complex
|
||||
type = struct complex {
|
||||
real_t real;
|
||||
double imag;
|
||||
}
|
||||
(gdb) whatis real_pointer_var
|
||||
type = real_t *
|
||||
(gdb) ptype real_pointer_var
|
||||
type = double *
|
||||
```
|
||||
|
||||
|
||||
|
||||
------
|
||||
|
||||
## 参考资料
|
||||
|
||||
- [Debugging with GDB](https://sourceware.org/gdb/current/onlinedocs/gdb/)
|
||||
|
||||
------
|
||||
|
||||
编译者：[robot527](https://github.com/robot527)
|
||||
|
||||
[1]: https://sourceware.org/gdb/current/onlinedocs/gdb/Set-Breaks.html
|
||||
[2]: https://sourceware.org/gdb/current/onlinedocs/gdb/Disabling.html
|
||||
[3]: https://sourceware.org/gdb/current/onlinedocs/gdb/Delete-Breaks.html
|
||||
[4]: https://sourceware.org/gdb/current/onlinedocs/gdb/Set-Watchpoints.html
|
||||
[5]: https://sourceware.org/gdb/current/onlinedocs/gdb/Continuing-and-Stepping.html
|
||||
[6]: https://sourceware.org/gdb/current/onlinedocs/gdb/Reverse-Execution.html
|
||||
[7]: https://sourceware.org/gdb/current/onlinedocs/gdb/Returning.html
|
||||
[8]: https://sourceware.org/gdb/current/onlinedocs/gdb/Data.html
|
||||
[9]: https://sourceware.org/gdb/current/onlinedocs/gdb/Output-Formats.html
|
||||
[10]: https://sourceware.org/gdb/current/onlinedocs/gdb/Memory.html
|
||||
[11]: https://sourceware.org/gdb/current/onlinedocs/gdb/Auto-Display.html
|
||||
[12]: https://sourceware.org/gdb/current/onlinedocs/gdb/Help.html
|
||||
[13]: https://sourceware.org/gdb/current/onlinedocs/gdb/Attach.html
|
||||
[14]: https://sourceware.org/gdb/current/onlinedocs/gdb/Starting.html
|
||||
[15]: https://sourceware.org/gdb/current/onlinedocs/gdb/Arguments.html
|
||||
[16]: https://sourceware.org/gdb/current/onlinedocs/gdb/Backtrace.html
|
||||
[17]: https://sourceware.org/gdb/current/onlinedocs/gdb/Symbols.html
|
||||
|
||||
Reference in New Issue
Block a user