document/TranslateProject
2
0
Fork 0
mirror of https://github.com/LCTT/TranslateProject.git synced 2026-09-01 04:27:55 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'LCTT/master'

Browse Source
This commit is contained in:
Xingyu Wang
2019-07-10 09:52:50 +08:00
parent 05750a7c0f 556958fc98
commit 9616814689
19 changed files with 3200 additions and 106 deletions
Download Patch File Download Diff File Expand all files Collapse all files

45
translated/tech/20190626 Tracking down library injections on Linux.md → published/20190626 Tracking down library injections on Linux.md
View File

@@ -1,26 +1,27 @@
[#]: collector: (lujun9972)
[#]: translator: (LuuMing)
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: reviewer: (wxy)
[#]: publisher: (wxy)
[#]: url: (https://linux.cn/article-11079-1.html)
[#]: subject: (Tracking down library injections on Linux)
[#]: via: (https://www.networkworld.com/article/3404621/tracking-down-library-injections-on-linux.html)
[#]: author: (Sandra Henry-Stocker https://www.networkworld.com/author/Sandra-Henry_Stocker/)
追溯 Linux 上的库注入
======
<ruby>库注入<rt>Library injections</rt></ruby>在 Linux 上不如 Windows 上常见，但它仍然是一个问题。下来看看它们如何工作的，以及如何鉴别它们。
![Sandra Henry-Stocker][1]
> <ruby>库注入<rt>Library injections</rt></ruby>在 Linux 上不如 Windows 上常见，但它仍然是一个问题。下来看看它们如何工作的，以及如何鉴别它们。
尽管在 Linux 系统上几乎见不到，但库（Linux 上的共享目标文件）注入仍是一个严峻的威胁。在采访了来自 AT&amp;T 公司 Alien 实验室的 Jaime Blasco 后，我更加意识到了其中一些攻击是多么的易实施。
![](https://img.linux.net.cn/data/attachment/album/201907/09/223222gje7l6a2hjhk364a.jpg)
在这篇文章中，我会介绍一种攻击方法和它的几种检测手段。我也会提供一些展示攻击细节的链接和一些检测工具。首先，引入一个小小的背景。
尽管在 Linux 系统上几乎见不到，但库（Linux 上的共享目标文件）注入仍是一个严峻的威胁。在采访了来自 AT&T 公司 Alien 实验室的 Jaime Blasco 后，我更加意识到了其中一些攻击是多么的易实施。
在这篇文章中，我会介绍一种攻击方法和它的几种检测手段。我也会提供一些展示攻击细节的链接和一些检测工具。首先，引入一个小小的背景信息。
### 共享库漏洞
DLL 和 .so 文件都是允许代码（有时候是数据）被不同的进程共享的共享库文件。公用的代码可以放进一个文件中使得每个需要它的进程可以重新使用而不是多次被重写。这也促进了对公用代码的管理。
Linux 进程经常使用这些共享库。`ldd`（显示共享对象依赖）命令可以为任何程序显示共享库。这里有一些例子：
Linux 进程经常使用这些共享库。（显示共享对象依赖的）`ldd` 命令可以对任何程序文件显示其共享库。这里有一些例子：
```
$ ldd /bin/date
@@ -39,7 +40,7 @@ $ ldd /bin/netstat
`linux-vdso.so.1` （在一些系统上也许会有不同的名字）是内核自动映射到每个进程地址空间的文件。它的工作是找到并定位进程所需的其他共享库。
利用这种库加载机制的一种方法是通过使用 `LD_PRELOAD` 环境变量。正如 Jaime Blasco 在他的研究中所解释的那样，“`LD_PRELOAD` 是最简单且最受欢迎的方法来在进程启动时加载共享库。可以使用共享库的路径配置环境变量，以便在加载其他共享对象之前加载该共享库。”
对库加载机制加以利用的一种方法是通过使用 `LD_PRELOAD` 环境变量。正如 Jaime Blasco 在他的研究中所解释的那样，“`LD_PRELOAD` 是在进程启动时加载共享库的最简单且最受欢迎的方法。可以将此环境变量配置到共享库的路径，以便在加载其他共享对象之前加载该共享库。”
为了展示有多简单，我创建了一个极其简单的共享库并且赋值给我的（之前不存在） `LD_PRELOAD` 环境变量。之后我使用 `ldd` 命令查看它对于常用 Linux 命令的影响。
@@ -47,7 +48,7 @@ $ ldd /bin/netstat
$ export LD_PRELOAD=/home/shs/shownum.so
$ ldd /bin/date
linux-vdso.so.1 (0x00007ffe005ce000)
/home/shs/shownum.so (0x00007f1e6b65f000) <== there it is
/home/shs/shownum.so (0x00007f1e6b65f000) <== 它在这里
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1e6b458000)
/lib64/ld-linux-x86-64.so.2 (0x00007f1e6b682000)
```
@@ -58,13 +59,13 @@ $ ldd /bin/date
### osquery 工具可以检测库注入
`osquery` 工具（可以在 [osquery.io][4]下载）提供了一个非常独特的方式来照看 Linux 系统。它基本上将操作系统表示为高性能关系数据库。然后，也许你会猜到，这就意味着它可以用来查询并且生成 SQL 表，该表提供了诸如以下的详细信息：
`osquery` 工具（可以在 [osquery.io][4]下载）提供了一个非常独特的查看 Linux 系统的方式。它基本上将操作系统视作一个高性能的关系数据库。然后，也许你会猜到，这就意味着它可以用来查询并且生成 SQL 表，该表提供了诸如以下的详细信息：
* 运行中的进程
* 加载的内核模块
* 进行的网络链接
* 打开的网络链接
一个提供了进程信息的内核表叫做 `process_envs`。它提供了各种进程使用环境变量的详细信息。Jaime Blasco 提供了一个相当复杂的查询，可以使用 `osquery` 标识使用 `LD_PRELOAD` 的进程。
一个提供了进程信息的内核表叫做 `process_envs`。它提供了各种进程使用环境变量的详细信息。Jaime Blasco 提供了一个相当复杂的查询，可以使用 `osquery` 识别出使用 `LD_PRELOAD` 的进程。
注意，这个查询是从 `process_envs` 表中获取数据。攻击 ID（T1055）参考 [Mitre 对攻击方法的解释][5]。
@@ -74,16 +75,16 @@ SELECT process_envs.pid as source_process_id, process_envs.key as environment_va
注意 `LD_PRELOAD` 环境变量有时是合法使用的。例如，各种安全监控工具可能会使用到它，因为开发人员需要进行故障排除、调试或性能分析。然而，它的使用仍然很少见，应当加以防范。
同样值得注意的是 osquery 可以交互使用或是作为定期查询的守护进程去运行。了解更多请查阅文章末尾给出的参考。
同样值得注意的是 `osquery` 可以交互使用或是作为定期查询的守护进程去运行。了解更多请查阅文章末尾给出的参考。
你也能够通过查看用户的环境设置定位到 `LD_PRELOAD` 的使用。如果 `LD_PRELOAD` 使用用户账户配置，你可以使用这样的命令来查看（在认证了个人身法之后）：
你也能够通过查看用户的环境设置来定位 `LD_PRELOAD` 的使用。如果在用户账户中使用了 `LD_PRELOAD`，你可以使用这样的命令来查看（假定以个人身份登录后）：
```
$ env | grep PRELOAD
LD_PRELOAD=/home/username/userlib.so
```
如果你之前没有听说过 osquery，别太在意。它正在成为一个更受欢迎的工具。事实上就在上周，Linux 基金会宣布用新的 [osquery 基金会][6]支持 osquery 社区。
如果你之前没有听说过 `osquery`，也别太在意。它正在成为一个更受欢迎的工具。事实上就在上周，Linux 基金会宣布打造了新的 [osquery 基金会][6]以支持 osquery 社区。
#### 总结
@@ -93,15 +94,13 @@ LD_PRELOAD=/home/username/userlib.so
重要的参考和工具的链接：
* [用 osquery 追寻 Linux 库注入][7]，AT&amp;T 网络安全
* [用 osquery 追寻 Linux 库注入][7]，AT&T Cybersecurity
* [Linux：我的内存怎么了？][8]，TrustedSec
* [osquery 下载网站][4]
* [osquery 关系模式][9]
* [下载 osquery][4]
* [osquery 模式][9]
* [osqueryd（osquery 守护进程）][10]
* [Mitre 的攻击框架][11]
* [新的 osquery 基金会宣布][6]
在 [Facebook][12] 和 [LinkedIn][13] 上加入网络会议参与讨论。
* [新的 osquery 基金会成立][6]
--------------------------------------------------------------------------------
@@ -110,7 +109,7 @@ via: https://www.networkworld.com/article/3404621/tracking-down-library-injectio
作者：[Sandra Henry-Stocker][a]
选题：[lujun9972][b]
译者：[LuuMing](https://github.com/LuuMing)
校对：[校对者ID](https://github.com/校对者ID)
校对：[wxy](https://github.com/wxy)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出

235
sources/talk/20180813 Using D Features to Reimplement Inheritance and Polymorphism.md Normal file
View File

@@ -0,0 +1,235 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Using D Features to Reimplement Inheritance and Polymorphism)
[#]: via: (https://theartofmachinery.com/2018/08/13/inheritance_and_polymorphism_2.html)
[#]: author: (Simon Arneaud https://theartofmachinery.com)
Using D Features to Reimplement Inheritance and Polymorphism
======
Some months ago I showed [how inheritance and polymorphism work in compiled languages][1] by reimplementing them with basic structs and function pointers. I wrote that code in D, but it could be translated directly to plain old C. In this post I’ll show how to take advantage of D’s features to make DIY inheritance a bit more ergonomic to use.
Although [I have used these tricks in real code][2], I’m honestly just writing this because I think it’s neat what D can do, and because it helps explain how high-level features of D can be implemented — using the language itself.
### `alias this`
In the original version of the code, the `Run` command inherited from the `Commmand` base class by including a `Command` instance as its first member. `Run` and `Command` were still considered completely different types, so this meant explicit typecasting was needed every time a `Run` instance was polymorphically used as a `Command`.
The D type system actually allows declaring a struct to be a subtype of another struct (or even of a primitive type) using a feature called “[`alias this`][3]”. Here’s a simple example of how it works:
```
struct Base
{
int x;
}
struct Derived
{
// Add an instance of Base as a member like before...
Base _base;
// ...but this time we declare that the member is used for subtyping
alias _base this;
}
void foo(Base b)
{
// ...
}
void main()
{
Derived d;
// Derived "inherits" members from Base
d.x = 42;
// Derived instances can be used where a Base instance is expected
foo(d);
}
```
The code above works in the same way as the code in the previous blog post, but `alias this` tells the type system what we’re doing. This allows us to work _with_ the type system more, and do less typecasting. The example showed a `Derived` instance being passed by value as a `Base` instance, but passing by `ref` also works. Unfortunately, D version 2.081 won’t implicitly convert a `Derived*` to a `Base*`, but maybe it’ll be implemented in future.
Here’s an example of `alias this` being used to implement some slightly more realistic inheritance:
```
import io = std.stdio;
struct Animal
{
struct VTable
{
void function(Animal* instance) greet;
}
immutable(VTable)* vtable;
void greet()
{
vtable.greet(&this);
}
}
struct Penguin
{
private:
static immutable Animal.VTable vtable = {greet: &greetImpl};
auto _base = Animal(&vtable);
alias _base this;
public:
string name;
this(string name) pure
{
this.name = name;
}
static void greetImpl(Animal* instance)
{
// We still need one typecast here because the type system can't guarantee this is okay
auto penguin = cast(Penguin*) instance;
io.writef("I'm %s the penguin and I can swim.\n", penguin.name);
}
}
void main()
{
auto p = Penguin("Paul");
// p inherits members from Animal
p.greet();
// and can be passed to functions that work with Animal instances
doThings(p);
}
void doThings(ref Animal a)
{
a.greet();
}
```
Unlike the code in the previous blog post, this version uses a vtable, just like the polymorphic inheritance in normal compiled languages. As explained in the previous post, every `Penguin` instance will use the same list of function pointers for its virtual functions. So instead of repeating the function pointers in every instance, we can have one list of function pointers that’s shared across all `Penguin` instances (i.e., a list that’s a `static` member). That’s all the vtable is, but it’s how real-world compiled OOP languages work.
### Template Mixins
If we implemented another `Animal` subtype, we’d have to add exactly the same vtable and base member boilerplate as in `Penguin`:
```
struct Snake
{
// This bit is exactly the same as before
private:
static immutable Animal.VTable vtable = {greet: &greetImpl};
auto _base = Animal(&vtable);
alias _base this;
public:
static void greetImpl(Animal* instance)
{
io.writeln("I'm an unfriendly snake. Go away.");
}
}
```
D has another feature for dumping this kind of boilerplate code into things: [template mixins][4].
```
mixin template DeriveAnimal()
{
private:
static immutable Animal.VTable vtable = {greet: &greetImpl};
auto _base = Animal(&vtable);
alias _base this;
}
struct Snake
{
mixin DeriveAnimal;
static void greetImpl(Animal* instance)
{
io.writeln("I'm an unfriendly snake. Go away.");
}
}
```
Actually, template mixins can take parameters, so it’s possible to create a generic `Derive` mixin that inherits from any struct that defines a `VTable` struct. Because template mixins can inject any kind of declaration, including template functions, the `Derive` mixin can even handle more complex things, like the typecast from `Animal*` to the subtype.
By the way, [the `mixin` statement can also be used to “paste” code into places][5]. It’s like a hygienic version of the C preprocessor, and it’s used below (and also in this [compile-time Brainfuck compiler][6]).
### `opDispatch()`
There’s some highly redundant wrapper code inside the definition of `Animal`:
```
void greet()
{
vtable.greet(&this);
}
```
If we added another virtual method, we’d have to add another wrapper:
```
void eat(Food food)
{
vtable.eat(&this, food);
}
```
But D has `opDispatch()`, which provides a way to automatically add members to a struct. When an `opDispatch()` is defined in a struct, any time the compiler fails to find a member, it tries the `opDispatch()` template function. In other words, it’s a fallback for member lookup. A fallback to a fully generic `return vtable.MEMBER(&this, args)` will effectively fill in all the virtual function dispatchers for us:
```
auto opDispatch(string member_name, Args...)(auto ref Args args)
{
mixin("return vtable." ~ member_name ~ "(&this, args);");
}
```
The downside is that if the `opDispatch()` fails for any reason, the compiler gives up on the member lookup and we get a generic “Error: no property foo for type Animal”. This is confusing if `foo` is actually a valid virtual member but was called with arguments of the wrong type, or something, so `opDispatch()` needs some good error handling (e.g., with [`static assert`][7]).
### `static foreach`
An alternative is to use a newer feature of D: [`static foreach`][8]. This is a powerful tool that can create declarations inside a struct (and other places) using a loop. We can directly read a list of members from the `VTable` definition by using some compile-time reflection:
```
import std.traits : FieldNameTuple;
static foreach (member; FieldNameTuple!VTable)
{
mixin("auto " ~ member ~ "(Args...)(auto ref Args args) { return vtable." ~ member ~ "(&this, args); }");
}
```
The advantage in this case is that we’re explicitly creating struct members. Now the compiler can distinguish between a member that shouldn’t exist at all, and a member that exists but isn’t used properly.
### It’s all just like the C equivalent
As I said, this is basically just a tour-de-force of ways that D can improve the code from the previous post. However, the original motivation for this blog post was people asking me about tricks I used to implement polymorphic inheritance in bare metal D code, so I’ll finish up by saying this: All this stuff works in [`-betterC`][9] code, and none of it requires extra runtime support. The code in this post implements the same kind of thing as in the [previous post][1]. It’s just expressed in a more compact and less error-prone way.
--------------------------------------------------------------------------------
via: https://theartofmachinery.com/2018/08/13/inheritance_and_polymorphism_2.html
作者：[Simon Arneaud][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://theartofmachinery.com
[b]: https://github.com/lujun9972
[1]: /2018/04/02/inheritance_and_polymorphism.html
[2]: https://gitlab.com/sarneaud/xanthe/blob/master/src/game/rigid_body.d#L15
[3]: https://dlang.org/spec/class.html#alias-this
[4]: https://dlang.org/spec/template-mixin.html
[5]: https://dlang.org/articles/mixin.html
[6]: /2017/12/31/compile_time_brainfuck.html
[7]: https://dlang.org/spec/version.html#StaticAssert
[8]: https://dlang.org/spec/version.html#staticforeach
[9]: https://dlang.org/blog/2018/06/11/dasbetterc-converting-make-c-to-d/

99
sources/talk/20181007 Why it-s Easier to Get a Payrise by Switching Jobs.md Normal file
View File

@@ -0,0 +1,99 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Why it's Easier to Get a Payrise by Switching Jobs)
[#]: via: (https://theartofmachinery.com/2018/10/07/payrise_by_switching_jobs.html)
[#]: author: (Simon Arneaud https://theartofmachinery.com)
Why it's Easier to Get a Payrise by Switching Jobs
======
It’s an empirical fact that it’s easier to get a payrise if you’re negotiating a new job than if you’re negotiating within your current job. When I look back over my own career, every time I’ve worked somewhere longer term (over a year), payrises have been a hard struggle. But eventually I’d leave for a new position, and my new pay made all payrises at the previous job irrelevant. These days I make job switching upfront and official: I run my own business and most of my money comes from short contracts. Getting rewarded for new skills or extra work is nowhere near as difficult as before.
I know I’m not the only one to notice this effect, but I’ve never heard anyone explain why things might be this way.
Before I give my explanation, let me make a couple of things clear from the start. I’m not going to argue that everyone should quit their jobs. I don’t know your situation, and maybe you’re getting a good deal already. Also, I apply game theory here, but, no, I don’t assume that humans are slaves to simplistic, mechanical laws of behaviour. However, just like music composition, even if humans are free, there are still patterns that matter. If you understand this stuff, you’ll have a career advantage.
But first, some background.
### BATNA
Many geeks think negotiation is like a role-playing game: roll the die, add your charisma score, and if the result is high enough you’re convincing. Geeks who think that way usually have low confidence in their “charisma score”, and they blame that for their struggle with things like asking for payrises.
Charisma isn’t totally irrelevant, but the good news for geeks is that there’s a nerdy thing that’s much more important for negotiation: BATNA, or Best Alternative To Negotiated Agreement. Despite the jargony name, it’s a very simple idea: it’s about analysing the best outcome for both sides in a negotiation, assuming that at least one side says no to the other. Although most people don’t know it’s called “BATNA”, it’s the core of how any agreement works (or doesn’t work).
It’s easy to explain with an example. Imagine you buy a couch for $500, but when you take it home, you discover that it doesn’t fit the place you wanted to put it. A silly mistake, but thankfully the shop offers you a full refund if you return it. Just as you’re taking it back to the shop, you meet a stranger who says they want a couch like that, and they offer to buy it. What’s the price? If you ask for $1000,000, the deal won’t happen because their BATNA is that they go to the shop and buy one themselves for $500. If they offer $1 to buy, your BATNA is that you go to the shop and get the $500 refund. You’ll only come to an agreement if the price is something like $500. If transporting the couch to the shop costs significant time and money, you’ll accept less than $500 because your BATNA is worth $500 minus the cost of transport. On the other hand, if the stranger needs to cover up a stained carpet before the landlord does an inspection in half an hour, they’ll be willing to pay a heavy premium because their BATNA is so bad.
You can’t expect a negotiation to go well unless you’ve considered the BATNA of both sides.
### Employment and Self-Employment
Most people of a certain socioeconomic class believe that the ideal, “proper” career is salaried, full-time employment at someone else’s business. Many people in this class never even imagine any other way to make a living, but there are alternatives. In Australia, like other countries, you’re free to register your own business number and then do whatever it is that people will pay for. That includes sitting at a desk and working on software and computer systems, or other work that’s more commonly done as an employee.
So why is salaried employment so popular? As someone who’s done both kinds of employment, one answer is obvious: stability. You can be (mostly) sure about exactly how much money you’ll make in the next six months when you have a salary. The next obvious answer is simplicity: as long as you meet the minimum bar of “work” done ([whatever “work” means][1]), the company promises to look after you. You don’t have to think about where your next dollar comes from, or about marketing, or insurances, or accounting, or even how to find people to socialise with.
That sums up the main reasons to like salaried employment (not that they’re bad reasons). I sometimes hear claims about other benefits of salaried employment, but they’re typically things that you can buy. If you’re self-employed and your work isn’t paying you enough to have the same lifestyle as you could under a salary (doing the same work) that means you’re not billing high enough. A lot of people make that mistake when they quit a salaried job for self-employment, but it’s still just a mistake.
### Asking for that Payrise
Let’s say you’ve been working as a salaried employee at a company for a while. As a curious, self-motivated person who regularly reads essays by nerds on the internet, you’ve learned a lot in that time. You’ve applied your new skills to your work, and proven yourself to be a much more valuable employee than when you were first hired. Is it time to ask for a payrise? You practise your most charismatic phrasing, and approach your manager with your d20 in hand. The response is that you’re doing great, and they’d love to give you a payrise, but the rules say
1. You can’t get a payrise unless you’ve been working for more than N years
2. You can’t get more than one payrise in N years
3. That inflation adjustment on your salary counted as a payrise, so you can’t ask for a payrise now
4. You can’t be paid more than [Peter][2]
5. We need more time to see if you’re ready, so keep up the great work for another year or so and we’ll consider it then
The thing to realise is that all these rules are completely arbitrary. If the company had a genuine motivation to give you a payrise, the rules would vanish. To see that, try replacing “payrise” with “workload increase”. Software projects are extremely expensive, require skill, and have a high failure rate. Software work therefore carries a non-trivial amount of responsibility, so you might argue that employers should be very conservative about increasing how much involvement someone has in a project. But I’ve never heard an employer say anything like, “Great job on getting that last task completed ahead of schedule, but we need more time to see if you’re ready to increase your workload. Just take a break until the next scheduled task, and if you do well at that one, too, maybe we can start giving you more work to do.”
If you’re hearing feedback that you’re doing well, but there are various arbitrary reasons you can’t get rewarded for it, that’s a strong sign you’re being paid below market rates. Now, the term “market rates” gets used pretty loosely, so let me be super clear: that means someone else would agree to pay you more if you asked.
Note that I’m not claiming that your manager is evil. At most larger companies, your manager really can’t do much against the company rules. I’m not writing this to call companies evil, either, because that won’t help you or me to get any payrises. What _will_ help is understanding why companies can afford to make payrises difficult.
### Getting that Payrise
You’ve probably seen this coming: it’s all about BATNA, and how you can’t expect your employer to agree to something that’s worse than their BATNA. So, what’s their BATNA? What happens if you ask for a payrise, and they say no?
Sometimes you see a story online about someone who was burning themselves out working super hard as an obviously vital member of a team. This person asks for a small payrise and gets rejected for some silly reason. Shortly after that, they tell their employer that they have a much bigger offer from another company. Suddenly the reason for rejecting the payrise evaporates, and the employer comes up with a counteroffer, but it’s too late: the worker leaves for a better job. The original employer is left wailing and gnashing their teeth. If only companies appreciated their employees more!
These stories are like hero stories in the movies. They tickle our sense of justice, but aren’t exactly representative of normal life. The reality is that most employees would just go back to their desks if they’re told, “No.” Sure, they’ll grumble, and they’ll upvote the next “Tech workers are underappreciated!” post on Reddit, but to many companies this is a completely acceptable BATNA.
In short, the main bargaining chip a salaried employee has is quitting, but that negates the reasons to be a salaried employee in the first place.
When you’re negotiating a contract with a new potential employer, however, the situation is totally different. Whatever conditions you ask for will be compared against the BATNA of searching for someone else who has your skills. Any reasonable request has a much higher chance of being accepted.
### The Job Security Tax
Now, something might be bothering you: despite what I’ve said, people _do_ get payrises. But all I’ve argued is that companies can make payrises difficult, not impossible. Sure, salaried employees might not quit when they’re a little underpaid. (They might not even realise they’re underpaid.) But if the underpayment gets big and obvious enough, maybe they will, so employers have to give out payrises eventually. Occasional payrises also make a good carrot for encouraging employees to keep working harder.
At the scale of a large company, it’s just a matter of tuning. Payrises can be delayed a little here, and made a bit smaller there, and the company saves money. Go too far, and the employee attrition rate goes up, which is a sign to back off and start paying more again.
Sure, the employee’s salary will tend to grow as their skills grow, but that growth will be slowed down. How much it is slowed down will depend (long term) on how strongly the employee values job security. It’s a job security tax.
### What Should You Do?
As I said before, I’m not going to tell you to quit (or not quit) without knowing what your situation is.
Perhaps you read this thinking that it sounds nothing like your workplace. If so, you’re lucky to be in one of the better places. You now have solid reasons to appreciate your employer as much as they appreciate you.
For the rest of you, I guess there are two broad options. Obviously, there’s the one I’m taking: not being a salaried employee. The other option is to understand the job security tax and try to optimise it. If you’re young and single, maybe you don’t need job security so much (at least for now). Even if you have good reasons to want job security (and there are plenty), maybe you can reduce your dependence on it by saving money in an emergency fund, and making sure your friendship group includes people who aren’t your current colleagues. That’s a good idea even if you aren’t planning to quit today — you never know what the future will be like.
--------------------------------------------------------------------------------
via: https://theartofmachinery.com/2018/10/07/payrise_by_switching_jobs.html
作者：[Simon Arneaud][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://theartofmachinery.com
[b]: https://github.com/lujun9972
[1]: /2017/09/14/busywork.html
[2]: https://www.youtube.com/watch?v=zBfTrjPSShs

412
sources/talk/20181107 Understanding a -nix Shell by Writing One.md Normal file
View File

@@ -0,0 +1,412 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Understanding a *nix Shell by Writing One)
[#]: via: (https://theartofmachinery.com/2018/11/07/writing_a_nix_shell.html)
[#]: author: (Simon Arneaud https://theartofmachinery.com)
Understanding a *nix Shell by Writing One
======
A typical *nix shell has a lot of programming-like features, but works quite differently from languages like Python or C++. This can make a lot of shell features — like process management, argument quoting and the `export` keyword — seem like mysterious voodoo.
But a shell is just a program, so a good way to learn how a shell works is to write one. I’ve written [a simple shell that fits in a few hundred lines of commented D source][1]. Here’s a post that walks through how it works and how you could write one yourself.
### First (Cheating) Steps
A shell is a kind of REPL (Read Evaluate Print Loop). At its heart is just a simple loop that reads commands from the input, processes them, and returns a result:
```
import std.process;
import io = std.stdio;
enum kPrompt = "> ";
void main()
{
io.write(kPrompt);
foreach (line; io.stdin.byLineCopy())
{
// "Cheating" by using the existing shell for now
auto result = executeShell(line);
io.write(result.output);
io.write(kPrompt);
}
}
$ dmd shell.d
$ ./shell
> head /usr/share/dict/words
A
a
aa
aal
aalii
aam
Aani
aardvark
aardwolf
Aaron
> # Press Ctrl+D to quit
>
$
```
If you try out this code out for yourself, you’ll soon notice that you don’t have any nice editing features like tab completion or command history. The popular Bash shell uses a library called [GNU Readline][2] for that. You can get most of the features of Readline when playing with these toy examples just by running them under [rlwrap][3] (probably already in your system’s package manager).
### DIY Command Execution (First Attempt)
That first example demonstrated the absolute basic structure of a shell, but it cheated by passing commands directly to the shell already running on the system. Obviously, that doesn’t explain anything about how a real shell processes commands.
The basic idea, though, is very simple. Nearly everything that gets called a “shell command” (e.g., `ls` or `head` or `grep`) is really just a program on the filesystem. The shell just has to run it. At the operating system level, running a program is done using the `execve` system call (or one of its alternatives). For portability and convenience, the normal way to make a system call is to use one of the wrapper functions in the C library. Let’s try using `execv()`:
```
import core.sys.posix.stdio;
import core.sys.posix.unistd;
import io = std.stdio;
import std.string;
enum kPrompt = "> ";
void main()
{
io.write(kPrompt);
foreach (line; io.stdin.byLineCopy())
{
runCommand(line);
io.write(kPrompt);
}
}
void runCommand(string cmd)
{
// Need to convert D string to null-terminated C string
auto cmdz = cmd.toStringz();
// We need to pass execv an array of program arguments
// By convention, the first element is the name of the program
// C arrays don't carry a length, just the address of the first element.
// execv starts reading memory from the first element, and needs a way to
// know when to stop. Instead of taking a length value as an argument,
// execv expects the array to end with a null as a stopping marker.
auto argsz = [cmdz, null];
auto error = execv(cmdz, argsz.ptr);
if (error)
{
perror(cmdz);
}
}
```
Here’s a sample run:
```
> ls
ls: No such file or directory
> head
head: No such file or directory
> grep
grep: No such file or directory
> ಠ_ಠ
ಠ_ಠ: No such file or directory
>
```
Okay, so that’s not working so well. The problem is that that the `execve` call isn’t as smart as a shell: it just literally executes the program it’s told to. In particular, it has no smarts for finding the programs that implement `ls` or `head`. For now, let’s do the finding ourselves, and then give `execve` the full path to the command:
```
$ which ls
/bin/ls
$ ./shell
> /bin/ls
shell shell.d shell.o
$
```
This time the `ls` command worked, but our shell quit and we dropped straight back into the system’s shell. What’s going on? Well, `execve` really is a single-purpose call: it doesn’t spawn a new process for running the program separately from the current program, it _replaces_ the current program. (The toy shell actually quit when `ls` started, not when it finished.) Creating a new process is done with a different system call: traditionally `fork`. This isn’t how programming languages normally work, so it might seem like weird and annoying behaviour, but it’s actually really useful. Decoupling process creation from program execution allows a lot of flexibility, as will become clearer later.
### Fork and Exec
To keep the shell running, we’ll use the `fork()` C function to create a new process, and then make that new process `execv()` the program that implements the command. (On modern GNU/Linux systems, `fork()` is actually a wrapper around a system call called `clone`, but it still behaves like the classic `fork` system call.)
`fork()` duplicates the current process. We get a second process that’s running the same program, at the same point, with a copy of everything in memory and all the same open files. Both the original process (parent) and the duplicate (child) keep running normally. Of course, we want the parent process to keep running the shell, and the child to `execv()` the command. The `fork()` function helps us differentiate them by returning zero in the child and a non-zero value in the parent. (This non-zero value is the process ID of the child.)
Let’s try it out in a new version of the `runCommand()` function:
```
int runCommand(string cmd)
{
// fork() duplicates the process
auto pid = fork();
// Both the parent and child keep running from here as if nothing happened
// pid will be < 0 if forking failed for some reason
// Otherwise pid == 0 for the child and != 0 for the parent
if (pid < 0)
{
perror("Can't create a new process");
exit(1);
}
if (pid == 0)
{
// Child process
auto cmdz = cmd.toStringz();
auto argsz = [cmdz, null];
execv(cmdz, argsz.ptr);
// Only get here if exec failed
perror(cmdz);
exit(1);
}
// Parent process
// This toy shell can only run one command at a time
// All the parent does is wait for the child to finish
int status;
wait(&status);
// This is the exit code of the child
// (Conventially zero means okay, non-zero means error)
return WEXITSTATUS(status);
}
```
Here it is in action:
```
> /bin/ls
shell shell.d shell.o
> /bin/uname
Linux
>
```
Progress! But it still doesn’t feel like a real shell if we have to tell it exactly where to find each command.
### PATH
If you try using `which` to find the implementations of various commands, you might notice they’re all in the same small set of directories. The list of directories that contains commands is stored in an environment variable called `PATH`. It looks something like this:
```
$ echo $PATH
/home/user/bin:/home/user/local/bin:/home/user/.local/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/games/bin
```
As you can see, it’s a list of directories separated by colons. If you ask a shell to run `ls`, it’s supposed to search each directory in this list for a program called `ls`. The search should be done in order starting from the first directory, so a personal implementation of `ls` in `/home/user/bin` could override the one in `/bin`. Production-ready shells cache this lookup.
`PATH` is only used by default. If we type in a path to a program, that program will be used directly.
Here’s a simple implemention of a smarter conversion of a command name to a C string that points to the executable. It returns a null if the command can’t be found.
```
const(char*) findExecutable(string cmd)
{
if (cmd.canFind('/'))
{
if (exists(cmd)) return cmd.toStringz();
return null;
}
foreach (dir; environment["PATH"].splitter(":"))
{
import std.path : buildPath;
auto candidate = buildPath(dir, cmd);
if (exists(candidate)) return candidate.toStringz();
}
return null;
}
```
Here’s what the shell looks like now:
```
> ls
shell shell.d shell.o
> uname
Linux
> head shell.d
head shell.d: No such file or directory
>
```
### Complex Commands
That last command failed because the toy shell doesn’t handle program arguments yet, so it tries to find a command literally called “head shell.d”.
If you look back at the implementation of `runCommand()`, you’ll see that `execv()` takes a C array of arguments, as well as the path to the program to run. All we have to do is process the command to make the array `["head", "shell.d", null]`. Something like this would do it:
```
// Key difference: split the command into pieces
auto args = cmd.split();
auto cmdz = findExecutable(args[0]);
if (cmdz is null)
{
io.stderr.writef("%s: No such file or directory\n", args[0]);
// 127 means "Command not found"
// http://tldp.org/LDP/abs/html/exitcodes.html
exit(127);
}
auto argsz = args.map!(toStringz).array;
argsz ~= null;
auto error = execv(cmdz, argsz.ptr);
```
That makes simple arguments work, but we quickly get into problems:
```
> head -n 5 shell.d
import core.sys.posix.fcntl;
import core.sys.posix.stdio;
import core.sys.posix.stdlib;
import core.sys.posix.sys.wait;
import core.sys.posix.unistd;
> echo asdf
asdf
> echo $HOME
$HOME
> ls *.d
ls: cannot access '*.d': No such file or directory
> ls '/home/user/file with spaces.txt'
ls: cannot access "'/home/user/file": No such file or directory
ls: cannot access 'with': No such file or directory
ls: cannot access "spaces.txt'": No such file or directory
>
```
As you might guess by looking at the above, shells like a POSIX Bourne shell (or Bash) do a _lot_ more than just `split()`. Take the `echo $HOME` example. It’s a common idiom to use `echo` for viewing environment variables (like `HOME`), but `echo` itself doesn’t actually do any environment variable handling. A POSIX shell processes a command like `echo $HOME` into an array like `["echo", "/home/user", null]` and passes it to `echo`, which does nothing but reflect its arguments back to the terminal.
A POSIX shell also handles glob patterns like `*.d`. That’s why glob patterns work with _any_ command in *nix (unlike MS-DOS, for example): the commands don’t even see the globs.
The command `ls '/home/user/file with spaces.txt'` got split into `["ls", "'/home/user/file", "with", "spaces.txt'", null]`. Any useful shell lets you use quoting and escaping to prevent any processing (like splitting into arguments) that you don’t want. Once again, quotes are completely handled by the shell; commands don’t even see them. Also, unlike most programming languages, everything is a string in shell, so there’s no difference between `head -n 5 shell.d` and `head -n '5' shell.d` — both turn into `["head", "-n", "5", "shell.d", null]`.
There’s something you might notice from that last example: the shell can’t treat flags like `-n 5` differently from positional arguments like `shell.d` because `execve` only takes a single array of all arguments. So that means argument types are one thing that programs _do_ have to figure out for themselves, which explains [the clichéd inteview question about why quotes won’t help you delete a file called `-`][4] (i.e., the quotes are processed before the `rm` command sees them).
A POSIX shell supports quite complex constructs like `while` loops and pipelines, but the toy shell only supports simple commands.
### Tweaking the Child Process
I said earlier that decoupling `fork` from `exec` allows extra flexibility. Let me give a couple of examples.
#### I/O Redirection
A key design principle of Unix is that commands should be agnostic about where their input and output are from, so that user input/output can be replaced with file input/output, or even input/output of other commands. E.g.:
```
sort events.txt | head -n 10 > /tmp/top_ten_events.txt
```
How does it work? Take the `head` command. The shell forks off a new child process. The child is a duplicate of the parent, so it inherits the same standard input and output. However, the child can replace its own standard input with a pipe shared with the process for `sort`, and replace its own standard output with a file handle for `/tmp/top_ten_events.txt`. After calling `execv()`, the process will become a `head` process that blindly reads/writes to/from whatever standard I/O it has.
Getting down to the low-level details, *nix systems represent all file handles with so-called “file descriptors”, which are just integers as far as user programs are concerned, but point to data structures inside the operating system kernel. Standard input is file descriptor 0, and standard output is file descriptor 1. Replacing standard output for `head` looks something like this (minus error handling):
```
// The fork happens somewhere back here
// Now running in the child process
// Open the new file (no control over the file descriptor)
auto new_fd = open("/tmp/top_ten_events.txt", O_WRONLY | O_CREAT, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH);
// Copy the open file into file #1 (standard output)
dup2(new_fd, 1);
// Close the spare file descriptor
close(new_fd);
// The exec happens somewhere down here
```
The pipeline works in the same kind of way, except instead of using `open()` to open a file, we use `pipe()` to create _two_ connected file descriptors, and then let `sort` use one, and `head` use the other.
#### Environment Variables
If you’ve ever had to deploy something using a command line, there’s a good chance you’ve had to set some of these configuration variables. Each process carries its own set of environment variables, so you can override, say, `AUDIODEV` for one running program without affecting others. The C standard library provides functions for manipulating environment variables, but they’re not actually managed by the operating system kernel — the [C runtime][5] manages them using the same user-space memory that other program variables use. That means they also get copied to child processes on a `fork`. The runtime and the kernel co-operate to preserve them on `execve`.
There’s no reason we can’t manipulate the environment variables the child process ends up using. POSIX shells support this: just put any variable assignments you want directly in front of the command.
```
$ uname
Linux
$ # LD_DEBUG is an environment variable for enabling linker debugging
$ # (Doesn't work on all systems.)
$ LD_DEBUG=statistics uname
12128:
12128: runtime linker statistics:
12128: total startup time in dynamic loader: 2591152 cycles
12128: time needed for relocation: 816752 cycles (31.5%)
12128: number of relocations: 153
12128: number of relocations from cache: 3
12128: number of relative relocations: 1304
12128: time needed to load objects: 1196148 cycles (46.1%)
Linux
$ # LD_DEBUG was only set for uname
$ echo $LD_DEBUG
$ # Pop quiz: why doesn't this print "bar"?
$ FOO=bar echo $FOO
$
```
These temporary environment variables are useful and easy to implement.
### Builtins
It’s great that the fork/exec pattern lets us reconfigure the child process as much as we like without affecting the parent shell. But some commands _need_ to affect the shell. A good example is the `cd` command for changing the current working directory. It would be pointless if it ran in a child process, changed its own working directory, then just quit, leaving the shell unchanged.
The simple solution to this problem is builtins. I said that most shell commands are implemented as external programs on the filesystem. Well, some aren’t — they’re handled directly by the shell itself. Before searching PATH for a command implementation, the shell just checks if it has it’s own built-in implementation. A neat way to code this is [the function pointer approach I described in a previous post][6].
You can read [a list of Bash builtins in the Advanced Bash-Scripting Guide][7]. Some, like `cd`, are builtins because they’re highly coupled to the shell. Others, like `echo`, have built-in implementations for performance reasons (most systems also have a standalone `echo` program).
There’s one builtin that confuses a lot of people: `export`. It makes sense if you realise that the POSIX shell scripting language has its own variables that are totally separate from environment variables. A variable assignment is just a shell variable by default, and `export` makes it into an environment variable (when spawning child processes, at least). The difference is that the C runtime doesn’t know anything about shell variables, so they get lost on `execve`.
```
$ uname
Linux
$ # Let's try setting LD_DEBUG
$ LD_DEBUG=statistics
$ # It has no effect because that's actually just a shell variable
$ uname
Linux
$ # Let's try making into an environment variable:
$ export LD_DEBUG
$ uname
12128:
12128: runtime linker statistics:
12128: total startup time in dynamic loader: 2591152 cycles
12128: time needed for relocation: 816752 cycles (31.5%)
12128: number of relocations: 153
12128: number of relocations from cache: 3
12128: number of relative relocations: 1304
12128: time needed to load objects: 1196148 cycles (46.1%)
Linux
$ # Now every non-builtin will dump debugging info
$ # Let's stop that for sanity's sake
$ unset LD_DEBUG
$
```
### Putting it Together
A POSIX-compliant shell does a lot more stuff (like signal handling and job management) but that’s enough to understand how to write an MVP *nix shell. You can see all the pieces together by checking out [the complete working example in my repository][1].
--------------------------------------------------------------------------------
via: https://theartofmachinery.com/2018/11/07/writing_a_nix_shell.html
作者：[Simon Arneaud][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://theartofmachinery.com
[b]: https://github.com/lujun9972
[1]: https://gitlab.com/sarneaud/toyshell
[2]: https://tiswww.case.edu/php/chet/readline/rltop.html
[3]: https://github.com/hanslub42/rlwrap
[4]: https://unix.stackexchange.com/questions/1519/how-do-i-delete-a-file-whose-name-begins-with-hyphen-a-k-a-dash-or-minus
[5]: /2017/06/04/what_is_the_d_runtime.html#what-about-c--does-c-really-have-a-runtime-too
[6]: /2018/04/02/inheritance_and_polymorphism.html
[7]: https://www.tldp.org/LDP/abs/html/internal.html

44
sources/talk/20181205 Unfortunately, Garbage Collection isn-t Enough.md Normal file
View File

@@ -0,0 +1,44 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Unfortunately, Garbage Collection isn't Enough)
[#]: via: (https://theartofmachinery.com/2018/12/05/gc_not_enough.html)
[#]: author: (Simon Arneaud https://theartofmachinery.com)
Unfortunately, Garbage Collection isn't Enough
======
Here’s a little story of some mysterious server failures I had to debug a year ago. The servers would run okay for a while, then eventually start crashing. After that, trying to run practically anything on the machines failed with “No space left on device” errors, but the filesystem only reported a few gigabytes of files on the ~20GB disks.
The problem turned out to be caused by a log shipper. This was a Ruby app that read in log files, sent the data to a remote server, and deleted the old files. The bug was that the open log files weren’t being explicitly closed. The app was letting Ruby’s automatic garbage collector clean up the `File` objects, instead. Trouble is, `File` objects don’t use much memory, so the log shipper could theoretically keep millions of log files open before a collection was needed.
*nix filesystems decouple filenames from file data. File data on disk can have multiple filenames pointing to it (i.e., hard links), and the data is only deleted when the last reference is removed. An open file descriptor counts as a reference, so if you delete a file while a program is reading it, the filename disappears from the directory listing, but the file data stays until the program closes it. That’s what was happening with the log shipper. The `du` (“disk usage”) command finds files using directory listings, so it didn’t see the gigabytes of file data for the thousands of log files the shipper had open. Those files only appeared after running `lsof` (“list open files”).
Of course, the same kind of bug happens with other things. A couple of months ago I had to deal with a Java app that was breaking in production after a few days because it leaked network connections.
Once upon a time, I wrote most of my code in C and then C++. In those days, I thought manual resource management was enough. How hard could it be? Every `malloc()` needs a `free()`, and every `open()` needs a `close()`. Simple. Except not all programs are simple, so manual resource management became a straitjacket. Then one day I discovered reference counting and garbage collection. I thought that solved all my problems, and I stopped caring about resource management completely. Once again, that was okay for simple programs, but not all programs are simple.
Relying on garbage collection doesn’t work because it only solves the _memory_ management problem, and complex programs have to deal with a lot more than just memory. There’s a popular meme that responds to that by saying that [memory is 95% of your resource problems][1]. Well, you could say that all resources are 0% of your problems — until you run out of one of them. Then that resource becomes 100% of your problems.
But that kind of thinking still treats resources as a special case. The deeper problem is that as programs get more complex, everything tends to become a resource. For example, take a calendar program. A complex calendar program allows multiple users to manage multiple, shareable calendars, with events that can be shared across calendars. Any piece of data will eventually have multiple parts of the program depending on it being up-to-date and accurate. So all dynamic data needs an owner, and not just for memory management. As more features are added, more parts of the program will need to update data. If you’re sane, you’ll only allow one part of the program to update data at a time, so the right and responsibility to update data becomes a limited resource, itself. Modelling mutable data with immutable datastructures doesn’t make these problems disappear; it just translates them into a different paradigm.
Planning the ownership and lifespan of resources is an inescapable part of designing complex software. It’s easier if you exploit some common patterns. One pattern is fungible resources. An example is an immutable string “foo”, which is semantically the same as any other immutable string “foo”. This kind of resource doesn’t need a pre-determined lifespan or ownership. In fact, to keep the system as simple as possible, it’s better to have _no_ pre-determined lifespan or ownership. Another pattern is resources that are non-fungible, but have a deterministic lifespan. This includes network connections, as well as more abstract things like the ownership of a piece of data. It’s sanest to explicitly enforce the lifespan of these things in code.
Notice that automatic garbage collection is really good for implementing the first pattern, but not the second, while manual resource management techniques (like RAII) are great for implementing the second pattern, but terrible for the first. The two approaches become complements in complex programs.
--------------------------------------------------------------------------------
via: https://theartofmachinery.com/2018/12/05/gc_not_enough.html
作者：[Simon Arneaud][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://theartofmachinery.com
[b]: https://github.com/lujun9972
[1]: https://yosefk.com/c++fqa/dtor.html#fqa-11.1

276
sources/talk/20181220 D in the Browser with Emscripten, LDC and bindbc-sdl (translation).md Normal file
View File

@@ -0,0 +1,276 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (D in the Browser with Emscripten, LDC and bindbc-sdl (translation))
[#]: via: (https://theartofmachinery.com/2018/12/20/emscripten_d.html)
[#]: author: (Simon Arneaud https://theartofmachinery.com)
D in the Browser with Emscripten, LDC and bindbc-sdl (translation)
======
Here’s a tutorial about using Emscripten to run D code in a normal web browser. It’s uses a different approach from the [Dscripten game demo][1] and the [dscripten-tools][2] toolchain that’s based on it.
* Instead of porting the D runtime, it uses a lightweight, runtimeless `-betterC` build.
* It uses Docker to manage the Emscripten installation.
LDC has recently gained support for [compiling directly to WebAssembly][3], but (unlike the Emscripten approach) that doesn’t automatically get you libraries.
You can find [the complete working code on Github][4]. `./run.sh` starts a shell in a Docker image that contains the development environment. `dub build --build=release` generates the HTML and JavaScript assets and puts them into the `dist/` directory.
[This tutorial is translated from a Japanese post by outlandkarasu][5], who deserves all the credit for figuring this stuff out.
### Background
#### What’s Emscripten?
[Emscripten][6] is a compiler toolchain for asm.js and WebAssembly that comes with ported versions of the libc and SDL2 C libraries. It can compile regular Linux-based applications in languages like C to code that can run in a browser.
### How do you use Emscripten with D?
Emscripten is a toolchain designed for C/C++, but the C/C++ part is just a frontend. The toolchain actually compiles LLVM intermediate representation (IR). You can generate LLVM IR bitcode from D using [LDC][7], so it should be possible to feed that through Emscripten and run D in a browser, just like C/C++.
#### Gotchas using Emscripten
Ideally that’s all it would take, but there are some things that require special attention (or trial and error).
1. D runtime library features like GC and Phobos can’t be used without an Emscripten port.
2. It’s not enough to just produce LLVM IR. The code needs to meet Emscripten’s requirements.
* It needs to use ported libraries.
* Pointer sizes and data structure binary layouts need to match.
3. Emscripten bugs need to be worked around.
* Debug information is particularly problematic.
### Implementation
#### Plan of attack
Here’s the plan for making D+Emscripten development work:
1. Use `-betterC` and the `@nogc` and `nothrow` attributes to avoid D runtime features.
2. Use SDL2 functions directly by statically compiling with [`bindbc-sdl`][8].
3. Keep on trying.
#### Environment setup
Emscripten is based on LLVM, clang and various other libraries, and is hard to set up, so I decided to [do the job with Docker][9]. I wrote a Dockerfile that would also add LDC and other tools at `docker build` time:
```
FROM trzeci/emscripten-slim:sdk-tag-1.38.21-64bit
# Install D and tools, and enable them in the shell by adding them to .bashrc
RUN apt-get -y update && \
apt-get -y install vim sudo curl && \
sudo -u emscripten /bin/sh -c "curl -fsS https://dlang.org/install.sh | bash -s ldc-1.12.0" && \
(echo 'source $(~/dlang/install.sh ldc -a)' >> /home/emscripten/.bashrc)
# dub settings (explained later)
ADD settings.json /var/lib/dub/settings.json
```
Docker makes these big toolchains pretty easy :)
#### Coding
Here’s a basic demo that displays an image:
```
// Import SDL2 and SDL_image
// Both work with Emscripten
import bindbc.sdl;
import bindbc.sdl.image;
import core.stdc.stdio : printf; // printf works in Emscripten, too
// Function declarations for the main loop
alias em_arg_callback_func = extern(C) void function(void*) @nogc nothrow;
extern(C) void emscripten_set_main_loop_arg(em_arg_callback_func func, void *arg, int fps, int simulate_infinite_loop) @nogc nothrow;
extern(C) void emscripten_cancel_main_loop() @nogc nothrow;
// Log output
void logError(size_t line = __LINE__)() @nogc nothrow {
printf("%d:%s\n", line, SDL_GetError());
}
struct MainLoopArguments {
SDL_Renderer* renderer;
SDL_Texture* texture;
}
// Language features restricted with @nogc and nothrow
extern(C) int main(int argc, const char** argv) @nogc nothrow {
// Initialise SDL
if(SDL_Init(SDL_INIT_VIDEO) != 0) {
logError();
return -1;
}
scope(exit) SDL_Quit();
// Initialise SDL_image (with PNG support)
if(IMG_Init(IMG_INIT_PNG) != IMG_INIT_PNG) {
logError();
return -1;
}
scope(exit) IMG_Quit();
// Make the window and its renderer
SDL_Window* window;
SDL_Renderer* renderer;
if(SDL_CreateWindowAndRenderer(640, 480, SDL_WINDOW_SHOWN, &window, &renderer) != 0) {
logError();
return -1;
}
scope(exit) {
SDL_DestroyRenderer(renderer);
SDL_DestroyWindow(window);
}
// Load image file
auto dman = IMG_Load("images/dman.png");
if(!dman) {
logError();
return -1;
}
scope(exit) SDL_FreeSurface(dman);
// Make a texture from the image
auto texture = SDL_CreateTextureFromSurface(renderer, dman);
if(!texture) {
logError();
return -1;
}
scope(exit) SDL_DestroyTexture(texture);
// Start the image main loop
auto arguments = MainLoopArguments(renderer, texture);
emscripten_set_main_loop_arg(&mainLoop, &arguments, 60, 1);
return 0;
}
extern(C) void mainLoop(void* p) @nogc nothrow {
// Get arguments
auto arguments = cast(MainLoopArguments*) p;
auto renderer = arguments.renderer;
auto texture = arguments.texture;
// Clear background
SDL_SetRenderDrawColor(renderer, 0x00, 0x00, 0x00, 0x00);
SDL_RenderClear(renderer);
// Texture image
SDL_RenderCopy(renderer, texture, null, null);
SDL_RenderPresent(renderer);
// End of loop iteration
emscripten_cancel_main_loop();
}
```
#### Building
Now building is the tricky bit.
##### `dub.json`
Here’s the `dub.json` I made through trial and error. It runs the whole build from D to WebAssembly.
```
{
"name": "emdman",
"authors": [
"outland.karasu@gmail.com"
],
"description": "A minimal emscripten D man demo.",
"copyright": "Copyright © 2018, outland.karasu@gmail.com",
"license": "BSL-1.0",
"dflags-ldc": ["--output-bc", "-betterC"], // Settings for bitcode output
"targetName": "app.bc",
"dependencies": {
"bindbc-sdl": "~>0.4.1"
},
"subConfigurations": {
"bindbc-sdl": "staticBC" // Statically-linked, betterC build
},
"versions": ["BindSDL_Image"], // Use SDL_image
// Run the Emscripten compiler after generating bitcode
// * Disable optimisations
// * Enable WebAssembly
// * Use SDL+SDL_image (with PNG)
// * Set web-only as the environment
// * Embed image file(s)
// * Generate HTML for running in browser
"postBuildCommands": ["emcc -v -O0 -s WASM=1 -s USE_SDL=2 -s USE_SDL_IMAGE=2 -s SDL2_IMAGE_FORMATS='[\"png\"]' -s ENVIRONMENT=web --embed-file images -o dist/index.html app.bc"]
}
```
##### Switch to 32b (x86) code generation
Compiling with 64b “worked” but I got a warning about different data layouts:
```
warning: Linking two modules of different data layouts: '/tmp/emscripten_temp_WwvmL5_archive_contents/mulsc3_20989819.c.o' is 'e-p:32:32-i64:64-v128:32:128-n32-S128' whereas '/src/app.bc' is 'e-m:e-i64:64-f80:128-n8:16:32:64-S128'
warning: Linking two modules of different target triples: /tmp/emscripten_temp_WwvmL5_archive_contents/mulsc3_20989819.c.o' is 'asmjs-unknown-emscripten' whereas '/src/app.bc' is 'x86_64-unknown-linux-gnu'
```
Apparently Emscripten is basically for 32b code. Using mismatched pointer sizes sounds like a pretty bad idea, so I added this `/var/lib/dub/settings.json` to the Dockerfile:
```
{
"defaultArchitecture": "x86", // Set code generation to 32b
"defaultCompiler": "ldc" // Use LDC by default
}
```
There’s an [open issue for documenting `dub`’s `settings.json`][10].
##### Remove debug information
Emscripten gave the following error when I ran a normal build with `dub`:
```
shared:ERROR: Failed to run llvm optimizations:
```
It looks like there’s [an issue related to debugging information][11]. I worked around it by using `dub --build=release`.
### Results
After lots of trial and error, I finally succeeded in getting my demo to run in a browser. Here’s how it looks:
![Meet D-Man \(Demo\)][12]
The Emscripten+D dev environment isn’t as stable as a normal dev environment. For example, rendering didn’t work if I used `SDL_LowerBlit` instead. But here’s D-Man in a browser.
--------------------------------------------------------------------------------
via: https://theartofmachinery.com/2018/12/20/emscripten_d.html
作者：[Simon Arneaud][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://theartofmachinery.com
[b]: https://github.com/lujun9972
[1]: https://github.com/Ace17/dscripten
[2]: https://github.com/CyberShadow/dscripten-tools
[3]: https://wiki.dlang.org/Generating_WebAssembly_with_LDC
[4]: https://github.com/outlandkarasu-sandbox/emdman
[5]: https://qiita.com/outlandkarasu@github/items/15e0f4b6d1b2a0eab846
[6]: http://kripken.github.io/emscripten-site/
[7]: https://wiki.dlang.org/LDC
[8]: https://github.com/BindBC/bindbc-sdl
[9]: https://hub.docker.com/r/trzeci/emscripten/
[10]: https://github.com/dlang/dub/issues/1463
[11]: https://github.com/kripken/emscripten/issues/4078
[12]: /images/emscripten_d/d-man-browser.png

115
sources/talk/20190105 Why Sorting is O(N log N).md Normal file
View File

@@ -0,0 +1,115 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Why Sorting is O(N log N))
[#]: via: (https://theartofmachinery.com/2019/01/05/sorting_is_nlogn.html)
[#]: author: (Simon Arneaud https://theartofmachinery.com)
Why Sorting is O(N log N)
======
Any decent algorithms textbook will explain how fast sorting algorithms like quicksort and heapsort are, but it doesn’t take crazy maths to prove that they’re as asymptotically fast as you can possibly get.
### A pedantic note about notation
Most computer scientists use big-O notation to mean “asymptotically equal, up to a constant scaling factor”, which isn’t quite what it means to other mathematicians. Sorry, I’ll use big-O like in CS textbooks, but at least I won’t mix it with other mathematical notation.
## Comparison-based sorting
Let’s look at the special case of algorithms that compare values two at a time (like quicksort and heapsort, and most other popular algorithms). The ideas can be extended to all sorting algorithms later.
### A simple counting argument for the worst case
Suppose you have an array of four elements, all different, in random order. Can you sort it by comparing just one pair of elements? Obviously not, but here’s one good reason that proves you can’t: By definition, to sort the array, you need to how to rearrange the elements to put them in order. In other words, you need to know which permutation is needed. How many possible permutations are there? The first element could be moved to one of four places, the second one could go to one of the remaining three, the third element has two options, and the last element has to take the one remaining place. So there are (4 \times 3 \times 2 \times 1 = 4! = 24) possible permutations to choose from, but there are only two possible results from comparing two different things: “BIGGER” and “SMALLER”. If you made a list of all the possible permutations, you might decide that “BIGGER” means you need permutation #8 and “SMALLER” means you need permutation #24, but there’s no way you could know when you need the other 22 permutations.
With two comparisons, you have (2 \times 2 = 4) possible outputs, which still isn’t enough. You can’t sort every possible shuffled array unless you do at least five comparisons ((2^{5} = 32)). If (W(N)) is the worst-case number of comparisons needed to sort (N) different elements using some algorithm, we can say
[2^{W(N)} \geq N!]
Taking a logarithm base 2,
[W(N) \geq \log_{2}{N!}]
Asymptotically, (N!) grows like (N^{N}) (see also [Stirling’s formula][1]), so
[W(N) \succeq \log N^{N} = N\log N]
And that’s an (O(N\log N)) limit on the worst case just from counting outputs.
### Average case from information theory
We can get a stronger result if we extend that counting argument with a little information theory. Here’s how we could use a sorting algorithm as a code for transmitting information:
1. I think of a number — say, 15
2. I look up permutation #15 from the list of permutations of four elements
3. I run the sorting algorithm on this permutation and record all the “BIGGER” and “SMALLER” comparison results
4. I transmit the comparison results to you in binary code
5. You re-enact my sorting algorithm run, step by step, referring to my list of comparison results as needed
6. Now that you know how I rearranged my array to make it sorted, you can reverse the permutation to figure out my original array
7. You look up my original array in the permutation list to figure out I transmitted the number 15
Okay, it’s a bit strange, but it could be done. That means that sorting algorithms are bound by the same laws as normal encoding schemes, including the theorem proving there’s no universal data compressor. I transmitted one bit per comparison the algorithm does, so, on average, the number of comparisons must be at least the number of bits needed to represent my data, according to information theory. More technically, [the average number of comparisons must be at least the Shannon entropy of my input data, measured in bits][2]. Entropy is a mathematical measure of the information content, or unpredictability, of something.
If I have an array of (N) elements that could be in any possible order without bias, then entropy is maximised and is (\log_{2}{N!}) bits. That proves that (O(N\log N)) is an optimal average for a comparison-based sort with arbitrary input.
That’s the theory, but how do real sorting algorithms compare? Below is a plot of the average number of comparisons needed to sort an array. I’ve compared the theoretical optimum against naïve quicksort and the [Ford-Johnson merge-insertion sort][3], which was designed to minimise comparisons (though it’s rarely faster than quicksort overall because there’s more to life than minimising comparisons). Since it was developed in 1959, merge-insertion sort has been tweaked to squeeze a few more comparisons out, but the plot shows it’s already almost optimal.
![Plot of average number of comparisons needed to sort randomly shuffled arrays of length up to 100. Bottom line is theoretical optimum. Within about 1% is merge-insertion sort. Naïve quicksort is within about 25% of optimum.][4]
It’s nice when a little theory gives such a tight practical result.
### Summary so far
Here’s what’s been proven so far:
1. If the array could start in any order, at least (O(N\log N)) comparisons are needed in the worst case
2. The average number of comparisons must be at least the entropy of the array, which is (O(N\log N)) for random input
Note that #2 allows comparison-based sorting algorithms to be faster than (O(N\log N)) if the input is low entropy (in other words, more predictable). Merge sort is close to (O(N)) if the input contains many sorted subarrays. Insertion sort is close to (O(N)) if the input is an array that was sorted before being perturbed a bit. None of them beat (O(N\log N)) in the worst case unless some array orderings are impossible as inputs.
## General sorting algorithms
Comparison-based sorts are an interesting special case in practice, but there’s nothing theoretically special about [`CMP`][5] as opposed to any other instruction on a computer. Both arguments above can be generalised to any sorting algorithm if you note a couple of things:
1. Most computer instructions have more than two possible outputs, but still have a limited number
2. The limited number of outputs means that one instruction can only process a limited amount of entropy
That gives us the same (O(N\log N)) lower bound on the number of instructions. Any physically realisable computer can only process a limited number of instructions at a time, so that’s an (O(N\log N)) lower bound on the time required, as well.
### But what about “faster” algorithms?
The most useful practical implication of the general (O(N\log N)) bound is that if you hear about any asymptotically faster algorithm, you know it must be “cheating” somehow. There must be some catch that means it isn’t a general purpose sorting algorithm that scales to arbitrarily large arrays. It might still be a useful algorithm, but it’s a good idea to read the fine print closely.
A well-known example is radix sort. It’s often called an (O(N)) sorting algorithm, but the catch is that it only works if all the numbers fit into (k) bits, and it’s really (O({kN})).
What does that mean in practice? Suppose you have an 8-bit machine. You can represent (2^{8} = 256) different numbers in 8 bits, so if you have an array of thousands of numbers, you’re going to have duplicates. That might be okay for some applications, but for others you need to upgrade to at least 16 bits, which can represent (2^{16} = 65,536) numbers distinctly. 32 bits will support (2^{32} = 4,294,967,296) different numbers. As the size of the array goes up, the number of bits needed will tend to go up, too. To represent (N) different numbers distinctly, you’ll need (k \geq \log_{2}N). So, unless you’re okay with lots of duplicates in your array, (O({kN})) is effectively (O(N\log N)).
The need for (O(N\log N)) of input data in the general case actually proves the overall result by itself. That argument isn’t so interesting in practice because we rarely need to sort billions of integers on a 32-bit machine, and [if anyone’s hit the limits of a 64-bit machine, they haven’t told the rest of us][6].
--------------------------------------------------------------------------------
via: https://theartofmachinery.com/2019/01/05/sorting_is_nlogn.html
作者：[Simon Arneaud][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://theartofmachinery.com
[b]: https://github.com/lujun9972
[1]: http://hyperphysics.phy-astr.gsu.edu/hbase/Math/stirling.html
[2]: https://en.wikipedia.org/wiki/Shannon%27s_source_coding_theorem
[3]: https://en.wikipedia.org/wiki/Merge-insertion_sort
[4]: /images/sorting_is_nlogn/sorting_algorithms_num_comparisons.svg
[5]: https://c9x.me/x86/html/file_module_x86_id_35.html
[6]: https://sortbenchmark.org/

100
sources/talk/20190319 Hello World Marketing (or, How I Find Good, Boring Software).md Normal file
View File

@@ -0,0 +1,100 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Hello World Marketing (or, How I Find Good, Boring Software))
[#]: via: (https://theartofmachinery.com/2019/03/19/hello_world_marketing.html)
[#]: author: (Simon Arneaud https://theartofmachinery.com)
Hello World Marketing (or, How I Find Good, Boring Software)
======
Back in 2001 Joel Spolsky wrote his classic essay [“Good Software Takes Ten Years. Get Used To it”][1]. Nothing much has changed since then: software is still taking around a decade of development to get good, and the industry is still getting used to that fact. Unfortunately, the industry has investors who want to see hockey stick growth rates on software that’s a year old or less. The result is an antipattern I like to call “Hello World Marketing”. Once you start to notice it, you see it everywhere, and it’s a huge red flag when choosing software tools.
Of course, by “Hello World”, I’m referring to the programmer’s traditional first program: the one that just displays the message “Hello World”. The aim isn’t to make a useful program; it’s to make a minimal starting point.
Hello World Marketing is about doing the same thing, but pretending that it’s useful. You’re supposed to be distracted into admiring how neatly a tool solves trivial problems, and forget about features you’ll need in real applications. HWM emphasises what can be done in the first five minutes, and downplays what you might need after several months. HWMed software is optimised for looking good in demos, and sounding exciting in blog posts and presentations.
For a good example, see Nemil Dalal’s [great series of articles about the early marketing for MongoDB][2]. Notice the heavy use of hackathons, and that a lot of the marketing was about how “SQL looks like COBOL”. Now, I can criticise SQL, too, but if `SELECT` and `WHERE` are serious problems for an application, there are already hundreds of solutions like [SQLAlchemy][3] and [LINQ][4] — solutions that don’t compromise on more advanced features of traditional databases. On the other hand, if you were wondering about those advanced features, you could read vomity-worthy, hand-wavey pieces like “[Living in the post-transactional database future][5]”.
### How I Find Good, Boring Software
Obviously, one way to avoid HWM is to stick to software that’s much more than ten years old, and has a good reputation. But sometimes that’s not possible because the tools for a problem only came out during the last decade. Also, sometimes newer tools really do bring new benefits.
However, it’s much harder to rely on reputation for newer software because “good reputation” often just means “popular”, which often just means “current fad”. Thankfully, there’s a simple and effective trick to avoid being dazzled by hype: just look elsewhere. Instead of looking at the marketing for the core features, look at the things that are usually forgotten. Here are the kinds of things I look at:
#### Backups and Disaster Recovery
Backup support is both super important and regularly an afterthought.
The minimum viable product is full data dump/import functionality, but longer term it’s nice to have things like incremental backups. Some vendors will try to tell you to just copy the data files from disk, but this isn’t guaranteed to give you a consistent snapshot if the software is running live.
There’s no point backing up data if you can’t restore it, and restoration is the difficult part. Yet many people never test the restoration (until they actually need it). About five years ago I was working with a team that had started using a new, cutting-edge, big-data database. The database looked pretty good, but I suggested we do an end-to-end test of the backup support. We loaded a cluster with one of the multi-terabyte datasets we had, did a backup, wiped the data in the cluster and then tried to restore it. Turns out we were the first people to actually try to restore a dataset of that size — the backup “worked”, but the restoration caused the cluster to crash and burn. We filed a bug report with the original database developers and they fixed it.
Backup processes that work on small test datasets but fail on large production datasets is a recurring theme. I always recommend testing on production-sized datasets, and testing again as production data grows.
For batch jobs, a related concept is restartability. If you’re copying large amounts of data from one place to another, and the job gets interrupted in the middle, what happens? Can you keep going from the middle? Alternatively, can you safely retry by starting from the beginning?
#### Configuration
A lot of HWMed software can only be configured using a GUI or web UI because that’s what’s obvious and looks good in demos and docs. For one thing, this usually means there’s no good way to back up or restore the configuration. So if a team of people use a shared instance over a year or so, forget about trying to restore it if (or when) it breaks. It’s also much more work to keep multiple deployments consistent (e.g., for dev, testing and prod environments) using separate GUIs. In practice, it just doesn’t happen.
I prefer a well-commented config file for software I deploy, if nothing else because it can be checked into source control, and I know I can reproduce the deployment using nothing but what’s checked into source control. If something is configured using a UI, I look for a config export/import function. Even then, that feature is often an afterthought, and often imcomplete, so it’s worth testing if it’s possible to deploy the software without ever needing to manually tweak something in the UI.
There seems to be a recent trend for software to be configured using a REST API instead. Honestly, this is the worst of both config files and GUI-based config, and most of the time people end up using [hacky ways to put the config into a file instead][6].
#### Upgrades
Life would be much easier if everything were static; software upgrade support makes everything more complicated. It’s also not usually shown in demos, so the first upgrade often ends the honeymoon with shiny, new software.
For HA distributed systems, you’ll need support for graceful shutdown and a certain amount of forward _and_ backwards compatibility (because you’ll have multiple versions running during upgrades). It’s a common mistake to forget about downgrade support.
Distributed systems are simpler when components have independent replicas that don’t communicate with each other. Anything with clustering (or, worse, consensus algorithms) is often extra tricky to upgrade, and worth testing.
Things that support horizontal scaling don’t necessarily support rescaling without downtime. This is especially true whenever sharding is involved because live resharding isn’t trivial.
Here’s a story from a certain popular container app platform. Demos showed how easy it was to launch an app on the platform, and then showed how easy it was to scale it to multiple replicas. What they didn’t show was the upgrade process: When you pushed a new version of your app, the first thing the platform did was _shut down all running instances of it_. Then it would upload the code to a build server and start building it — meaning downtime for however long the build took, plus the time needed to roll out the new version (if it worked). This problem has been fixed in newer releases of the platform.
#### Security
Even if software has no built-in access control, all-or-nothing access control is easy to implement (e.g., using a reverse proxy with HTTP basic auth). The harder problem is fine-grained access control. Sometimes you don’t care, but in some environments it makes a big difference to what features you can even use.
Some immature software has a quick-and-dirty implementation of user-based access control, typically with a GUI for user management. For everything except the core business tool, this isn’t very useful. For human users, every project I’ve worked on has either been with a small team that just shared a single username/password, or with a large team that wanted integration with OpenID Connect, or LDAP, or whatever centralised single-sign-on (SSO) system was used by the organisation. No one wants to manually manage credentials for every tool, every time someone joins or leaves. Similarly, credentials for applications or other non-human users are better generated using an automatable approach — like a config file or API.
Immature implementations of access control are often missing anything like user groups, but managing permissions at the user level is a time waster. Some SSO integrations only integrate users, not groups, which is a “so close yet so far” when it comes to avoiding permissions busywork.
#### Others
I talked about ignoring the hype, but there’s one good signal you can get from the marketing: whether the software is branded as “enterprise” software. Enterprise software is normally bought by someone other than the end user, so it’s usually pleasant to buy but horrible to use. The only exceptions I know of are enterprise versions of normal consumer software, and enterprise software that the buyer will also have to use. Be warned: even if a company sells enterprise software alongside consumer software, there’s no guarantee that they’re just different versions of the same product. Often they’ll be developed by separate teams with different priorities.
A lot of the stuff in this post can be checked just by skimming through the documentation. If a tool stores data, but the documentation doesn’t mention backups, there probably isn’t any backup suppport. Even if there is and it’s just not documented, that’s not exactly a good sign either. So, sure, documentation quality is worth evaluating by itself. On the other hand, sometimes the documentation is better than the product, so I never trust a tool until I’ve actually tried it out.
When I first saw Python, I knew that it was a terrible programming language because of the way it used whitespace indentation. Yeah, that was stupid. Later on I learned that 1) the syntax wasn’t a big deal, especially when I’m already indenting C-like languages in the same way, and 2) a lot of practical problems can be solved just by gluing libraries together with a few dozen lines of Python, and that was really useful. We often have strong opinions about syntax that are just prejudice. Syntax can matter, but it’s less important than how the tool integrates with the rest of the system.
### Weighing Pros and Cons
You never need to do deep analysis to detect the most overhyped products. Just check a few of these things and they’ll fail spectacularly.
Even with software that looks solid, I still like to do more tests before entrusting a serious project with it. That’s not because I’m looking for excuses to nitpick and use my favourite tool instead. New tools often really do bring new benefits. But it’s much better to understand the pros and cons of new software, and to use it because the pros outweigh the cons, not because of how slick the Hello World demo is.
--------------------------------------------------------------------------------
via: https://theartofmachinery.com/2019/03/19/hello_world_marketing.html
作者：[Simon Arneaud][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://theartofmachinery.com
[b]: https://github.com/lujun9972
[1]: https://www.joelonsoftware.com/2001/07/21/good-software-takes-ten-years-get-used-to-it/
[2]: https://www.nemil.com/mongo/
[3]: https://www.sqlalchemy.org/
[4]: https://msdn.microsoft.com/en-us/library/bb308959.aspx
[5]: https://www.mongodb.com/post/36151042528/post-transactional-future
[6]: /2017/07/15/use_terraform_with_vault.html

247
sources/talk/20190405 D as a C Replacement.md Normal file
View File

@@ -0,0 +1,247 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (D as a C Replacement)
[#]: via: (https://theartofmachinery.com/2019/04/05/d_as_c_replacement.html)
[#]: author: (Simon Arneaud https://theartofmachinery.com)
D as a C Replacement
======
Sircmpwn (the main developer behind the [Sway Wayland compositor][1]) recently wrote a blog post about how he thinks [Rust is not a good C replacement][2]. I don’t know if he’d like the [D programming language][3] either, but it’s become a C replacement for me.
### My C to D Story
My story is like a lot of systems programmers’ stories. At one time, C was my go-to language for most programming. One day I realised that most of my C programs kept reimplementing things from C++: dynamic arrays, better strings, polymorphic classes, etc. So I tried using C++ instead, and at first I loved it. RAII and classes and generics made programming fun again. Even better was the promise that if I read all these books on C++, and learned to master things like template metaprogramming, I’d become an almighty god of systems programming and my code would be amazing. But learning more eventually had the opposite effect: (in hindsight) my code actually got worse, and I fell out of love. I remember reading Scott Meyer’s Effective C++ and realising it was really more about _ineffective_ C++ — and that most of my C++ code until then was broken. Let’s face it: C might be fiddly to use, but it has a kind of elegance, and “elegant” is rarely a word you hear when C++ is involved.
Apparently, a lot of ex-C C++ programmers end up going back to C. In my case, I discovered D. It’s also not perfect, but I use it because it feels to me a lot more like the `C += 1` that C++ was meant to be. Here’s an example that’s very superficial, but I think is representative. Take this simple C program:
```
#include <stdio.h>
int main()
{
printf("1 + 1 = %d!\n", 1 + 1);
return 0;
}
```
Here’s a version using the C++ standard library:
```
#include <iostream>
int main()
{
std::cout << "1 + 1 = " << 1 + 1 << "!" << std::endl;
return 0;
}
```
Here’s an idiomatic D version:
```
import std.stdio;
void main()
{
writef("1 + 1 = %d!\n", 1 + 1);
}
```
As I said, it’s a superficial example, but I think it shows a general difference in philosophy between C++ and D. (If I wanted to make the difference even clearer, I’d use an example that needed `iomanip` in C++.)
Update: Unlike in C, [D’s format strings can work with custom types][4]. Stefan Rohe has also pointed out that [D supports compile-time checking of format strings][5] using its metaprogramming features — unlike C which does it through built-in compiler special casing that can’t be used with custom code.
This [article about C++ member function pointers][6] happens to also be a good explanation of the origins of D. It’s a good read if you’re a programming language nerd like me, but here’s my TL;DR for everyone else: C++ member function pointers are supposed to feel like a low-level feature (like normal function pointers are), but the complexity and diversity of implementations means they’re really high level. The complexity of the implementations is because of the subtleties of the rules about what you can do with them. The author explains the implementations from several C++ compilers, including what’s “easily [his] favorite implementation”: the elegantly simple Digital Mars C++ implementation. (“Why doesn’t everyone else do it this way?”) The DMC compiler was written by Walter Bright, who invented D.
D has classes and templates and other core features of C++, but designed by someone who has spent a heck of a lot of time thinking about the C++ spec and how things could be simpler. Walter once said that his experiences implementing C++ templates made him consider not including them in D at all, until he realised they didn’t need to be so complex.
Here’s a quick tour of D from the point of view of incrementally improving C.
### `-betterC`
D compilers support a `-betterC` switch that disables [the D runtime][7] and all high-level features that depend on it. The example C code above can be translated directly into betterC:
```
import core.stdc.stdio;
extern(C):
int main()
{
printf("1 + 1 = %d!\n", 1 + 1);
return 0;
}
$ dmd -betterC example.d
$ ./example
1 + 1 = 2!
```
The resulting binary looks a lot like the equivalent C binary. In fact, if you rewrote a C library in betterC, it could still link to code that had been compiled against the C version, and work without changes. Walter Bright wrote a good article walking through all [the changes needed to convert a real C program to betterC][8].
You don’t actually need the `-betterC` switch just to write C-like code in D. It’s only needed in special cases where you simply can’t have the D runtime. But let me point out some of my favourite D features that still work with `-betterC`.
#### `static assert()`
This allows verifying some assumption at compile time.
```
static assert(kNumInducers < 16);
```
Systems code often makes assumptions about alignment or structure size or other things. With `static assert`, it’s possible to not only document these assumptions, but trigger a compilation error if someone breaks them by adding a struct member or something.
#### Slices
Typical C code is full of pointer/length pairs, and it’s a common bug for them to go out of sync. Slices are a simple and super-useful abstraction for a range of memory defined by a pointer and length. Instead of code like this:
```
buffer_p += offset;
buffer_len -= offset; // Got to update both
```
You can use this much-less-bug-prone alternative:
```
buffer = buffer[offset..$];
```
A slice is nothing but a pointer/length pair with first-class syntactic support.
Update: [Walter Bright has written more about pointer/length pair problem in C][9].
#### Compile Time Function Evaluation (CTFE)
[Many functions can be evaluated at compile time.][10]
```
long factorial(int n) pure
{
assert (n >= 0 && n <= 20);
long ret = 1;
foreach (j; 2..n+1) ret *= j;
return ret;
}
// Statically allocated array
// Size is calculated at compile time
Permutation[factorial(kNumThings)] permutation_table;
```
#### `scope` Guards
Code in one part of a function is often coupled to cleanup code in a later part. Failing to match this code up correctly is another common source of bugs (especially when multiple control flow paths are involved). D’s scope guards make it simple to get this stuff right:
```
p = malloc(128);
// free() will be called when the current scope exits
scope (exit) free(p);
// Put whatever if statements, or loops, or early returns you like here
```
You can even have multiple scope guards in a scope, or have nested scopes. The cleanup routines will be called when needed, in the right order.
D also supports RAII using struct destructors.
#### `const` and `immutable`
It’s a popular myth that `const` in C and C++ is useful for compiler optimisations. Walter Bright has complained that every time he thought of a new `const`-based optimisation for C++, he eventually discovered it didn’t work in real code. So he made some changes to `const` semantics for D, and added `immutable`. You can read more in the [D `const` FAQ][11].
#### `pure`
Functional purity can be enforced. I’ve written about [some of the benefits of the `pure` keyword before][12].
#### `@safe`
SafeD is a subset of D that forbids risky language features like pointer typecasts and inline assembly. Code marked `@safe` is enforced by the compiler to not use these features, so that risky code can be limited to the small percentage of the application that needs it. You can [read more about SafeD in this article][13].
#### Metaprogramming
Like I hinted earlier, metaprogramming has got a bad reputation among some C++ programmers. But [D has the advantage of making metaprogramming less interesting][14], so D programmers tend to just do it when it’s useful, and not as a fun puzzle.
D has great support for [compile-time reflection][15]. In most cases, compile-time reflection can solve the same problems as run-time reflection, but with compile-time guarantees. Compile-time reflection can also be used to implement run-time reflection where it’s truly needed.
Need the names of an enumerated type as an array?
```
enum State
{
stopped,
starting,
running,
stopping,
}
string[] state_names = [__traits(allMembers, State)];
```
Thanks to D’s metaprogramming, the standard library has many nice, type-safe tools, like this [compile-time checked bit flag enum][16].
I’ve written more about [using metaprogramming in `-betterC` code][17].
#### No Preprocessor
Okay, this a non-feature as a feature, but D has no equivalent to C’s preprocessor. All its sane use-cases are replaced with native language features, like [manifest constants][18] and [templates][19]. That includes proper [modules][20] support, which means D can break free of the limitations of that old `#include` hack.
### Normal D
C-like D code can be written and compiled as normal D code without the `-betterC` switch. The difference is that normal D code is linked to the D runtime, which supports higher-level features, the most obvious ones being garbage collection and object-oriented classes. Some people have confused the D runtime with something like the Java virtual machine, so I once wrote [an article explaining exactly what it is][7] (spoiler: it’s like the C and C++ runtimes, but with more features).
Even with the runtime, compiled D is not much different from compiled C++. Sometimes I like to write throwaway code to, say, experiment with a new Linux system call or something. I used to think the best language for that is plain old C, but now I always use D.
D doesn’t natively support `#include`ing C code, but for nice APIs that don’t have a lot of preprocessor craziness (like most of Linux) I usually just write [ad-hoc bindings][21]. Many popular C libraries have maintained D bindings, which can be found in the [Dub registry][22], or in [the Derelict project][23], or in the newer [BindBC project][24]. There are also tools for automated bindings, including the awesome [dpp tool][25] that brings `#include` support directly to D code.
Update: This post has got a lot of attention from people who’ve never heard of D before. If you’re interested in learning D, I recommend
* [The DLang Tour][26] for a quick dive into the language
* [Ali Çehreli’s Programming in D book][27] if you prefer something in-depth
* [The D forum Learn group][28] or [IRC channel][29] to get answers to your questions
--------------------------------------------------------------------------------
via: https://theartofmachinery.com/2019/04/05/d_as_c_replacement.html
作者：[Simon Arneaud][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://theartofmachinery.com
[b]: https://github.com/lujun9972
[1]: https://swaywm.org/
[2]: https://drewdevault.com/2019/03/25/Rust-is-not-a-good-C-replacement.html
[3]: https://dlang.org
[4]: https://wiki.dlang.org/Defining_custom_print_format_specifiers
[5]: https://dlang.org/phobos/std_format.html#format
[6]: https://www.codeproject.com/Articles/7150/Member-Function-Pointers-and-the-Fastest-Possible
[7]: /2017/06/04/what_is_the_d_runtime.html
[8]: https://dlang.org/blog/2018/06/11/dasbetterc-converting-make-c-to-d/
[9]: https://www.digitalmars.com/articles/b44.html
[10]: https://dlang.org/spec/function.html#interpretation
[11]: https://dlang.org/articles/const-faq.html
[12]: /2016/03/28/dirtying_pure_functions_can_be_useful.html
[13]: https://dlang.org/blog/2016/09/28/how-to-write-trusted-code-in-d/
[14]: https://epi.github.io/2017/03/18/less_fun.html
[15]: https://dlang.org/spec/traits.html
[16]: https://dlang.org/phobos/std_typecons.html#BitFlags
[17]: /2018/08/13/inheritance_and_polymorphism_2.html
[18]: https://dlang.org/spec/enum.html#manifest_constants
[19]: https://tour.dlang.org/tour/en/basics/templates
[20]: https://ddili.org/ders/d.en/modules.html
[21]: https://wiki.dlang.org/Bind_D_to_C
[22]: https://code.dlang.org/
[23]: https://github.com/DerelictOrg
[24]: https://github.com/BindBC
[25]: https://github.com/atilaneves/dpp
[26]: https://tour.dlang.org/
[27]: https://ddili.org/ders/d.en/index.html
[28]: https://forum.dlang.org/group/learn
[29]: irc://irc.freenode.net/d

412
sources/talk/20190426 Profiling D-s Garbage Collection with Bpftrace.md Normal file
View File

@@ -0,0 +1,412 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Profiling D's Garbage Collection with Bpftrace)
[#]: via: (https://theartofmachinery.com/2019/04/26/bpftrace_d_gc.html)
[#]: author: (Simon Arneaud https://theartofmachinery.com)
Profiling D's Garbage Collection with Bpftrace
======
Recently I’ve been playing around with using [`bpftrace`][1] to trace and profile D’s garbage collector. Here are some examples of the cool stuff that’s possible.
### What is `bpftrace`?
It’s a high-level debugging tool based on Linux’s eBPF. “eBPF” stands for “extended Berkely packet filter”, but that’s just a historical name and doesn’t mean much today. It’s really a virtual machine (like the [JVM][2]) that sits inside the Linux kernel and runs code in a special eBPF instruction set similar to normal machine code. Users are expected to write short programs in high-level languages (including C and others) that get compiled to eBPF and loaded into the kernel on the fly to do interesting things.
As you might guess, eBPF is powerful for instrumenting a running kernel, but it also supports instrumenting user-space programs.
### What you need
First you need a Linux kernel. Sorry BSD, Mac OS and Windows users. (But some of you can use [DTrace][3].)
Also, not just any Linux kernel will work. This stuff is relatively new, so you’ll need a modern kernel with BPF-related features enabled. You might need to use the newest (or even testing) version of a distro. Here’s how to check if your kernel meets the requirements:
```
$ uname -r
4.19.27-gentoo-r1sub
$ # 4.9+ recommended by bpftrace
$ zgrep CONFIG_UPROBES /proc/config.gz
CONFIG_UPROBES=y
$ # Also need
$ # CONFIG_BPF=y
$ # CONFIG_BPF_SYSCALL=y
$ # CONFIG_BPF_JIT=y
$ # CONFIG_HAVE_EBPF_JIT=y
$ # CONFIG_BPF_EVENTS=y
```
Of course, [you also need to install the `bpftrace` tool itself][4].
### `bpftrace` D “Hello World”
Here’s a quick test you can do to make sure you’ve got everything working. First, let’s make a Hello World D binary:
```
$ pwd
/tmp/
$ cat hello.d
import std.stdio;
void main()
{
writeln("Hello World");
}
$ dmd hello.d
$ ./hello
Hello World
$
```
Now let’s `bpftrace` it. `bpftrace` uses a high-level language that’s obviously inspired by AWK. I’ll explain enough to understand the post, but you can also check out the [`bpftrace` reference guide][5] and [one-liner tutorial][6]. The minimum you need to know is that a bpftrace program is a list of `event:name /filter predicate/ { program(); code(); }` blocks that define code snippets to be run on events.
This time I’m only using Linux uprobes, which trigger on functions in user-space programs. The syntax is `uprobe:/path/to/binary:functionName`. One gotcha is that D “[mangles][7]” (encodes) function names before inserting them into the binary. If we want to trigger on the D code’s `main()` function, we need to use the mangled name: `_Dmain`. (By the way, `nm program | grep ' _D.*functionName'` is one quick trick for finding mangled names.)
Run this `bpftrace` invocation in a terminal as root user:
```
# bpftrace -e 'uprobe:/tmp/hello:_Dmain { printf("D Hello World run with process ID %d\n", pid); }'
Attaching 1 probe...
```
While this is running, it’ll print a message every time the D Hello World program is executed by any user in any terminal. Press `Ctrl+C` to quit.
All `bpftrace` code can be run directly from the command line like in the example above. But to make things easier to read from now on, I’ll make neatly formatted scripts.
### Tracing some real code
I’m using [D-Scanner][8], the D code analyser, as an example of a simple but non-trivial D workload. One nice thing about `bpftrace` and uprobes is that no modification of the program is needed. I’m just using a normal build of the `dscanner` tool, and using the [D runtime source code][9] as a codebase to analyse.
Before using `bpftrace`, let’s try using [the profiling that’s built into the D GC implementation itself][10]:
```
$ dscanner --DRT-gcopt=profile:1 --etags
...
Number of collections: 85
Total GC prep time: 0 milliseconds
Total mark time: 17 milliseconds
Total sweep time: 6 milliseconds
Total page recovery time: 3 milliseconds
Max Pause Time: 1 milliseconds
Grand total GC time: 28 milliseconds
GC summary: 35 MB, 85 GC 28 ms, Pauses 17 ms < 1 ms
```
(If you can make a custom build, you can also use [the D runtime GC API to get stats][11].)
There’s one more gotcha when using `bpftrace` on `dscanner` to trace GC functions: the binary file we specify for the uprobe needs to be the binary file that actually contains the GC functions. That could be the D binary itself, or it could be a shared D runtime library. Try running `ldd /path/to/d_program` to list any linked shared libraries, and if the output contains `druntime`, use that full path when specifying uprobes. My `dscanner` binary doesn’t link to a shared D runtime, so I just use the full path to `dscanner`. (Running `which dscanner` gives `/usr/local/bin/dscanner` for me.)
Anyway, all the GC functions live in a `gc` module, so their mangled names start with `_D2gc`. Here’s a `bpftrace` invocation that tallies GC function calls. For convenience, it also includes a uretprobe to automatically exit when `main()` returns. The output is sorted to make it a little easier to read.
```
# cat dumpgcfuncs.bt
uprobe:/usr/local/bin/dscanner:_D2gc*
{
@[probe] = count();
}
uretprobe:/usr/local/bin/dscanner:_Dmain
{
exit();
}
# bpftrace dumpgcfuncs.bt | sort
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC10freeNoSyncMFNbNiPvZv]: 31
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC10initializeFKCQCd11gcinterface2GCZv]: 1
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC11queryNoSyncMFNbPvZS4core6memory8BlkInfo_]: 44041
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC11removeRangeMFNbNiPvZv]: 2
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC12extendNoSyncMFNbPvmmxC8TypeInfoZm]: 251946
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC14collectNoStackMFNbZv]: 1
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC18fullCollectNoStackMFNbZv]: 1
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC4freeMFNbNiPvZv]: 31
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC5queryMFNbPvZS4core6memory8BlkInfo_]: 47704
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC6__ctorMFZCQBzQBzQBxQCiQBn]: 1
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC6callocMFNbmkxC8TypeInfoZPv]: 80
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC6extendMFNbPvmmxC8TypeInfoZm]: 251946
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC6mallocMFNbmkxC8TypeInfoZPv]: 12423
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC6qallocMFNbmkxC8TypeInfoZS4core6memory8BlkInfo_]: 948995
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC7getAttrMFNbPvZ2goFNbPSQClQClQCjQCu3GcxQBbZk]: 5615
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC7getAttrMFNbPvZk]: 5615
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC8addRangeMFNbNiPvmxC8TypeInfoZv]: 2
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC__T9runLockedS_DQCeQCeQCcQCnQBs10freeNoSyncMFNbNiPvZvS_DQDsQDsQDqQEb8freeTimelS_DQErQErQEpQFa8numFreeslTQCdZQEbMFNbNiKQCrZv]: 31
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC__T9runLockedS_DQCeQCeQCcQCnQBs11queryNoSyncMFNbPvZS4core6memory8BlkInfo_S_DQEmQEmQEkQEv9otherTimelS_DQFmQFmQFkQFv9numOtherslTQDaZQExMFNbKQDmZQDn]: 44041
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC__T9runLockedS_DQCeQCeQCcQCnQBs12extendNoSyncMFNbPvmmxC8TypeInfoZmS_DQEfQEfQEdQEo10extendTimelS_DQFhQFhQFfQFq10numExtendslTQCwTmTmTxQDaZQFdMFNbKQDrKmKmKxQDvZm]: 251946
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC__T9runLockedS_DQCeQCeQCcQCnQBs12mallocNoSyncMFNbmkKmxC8TypeInfoZPvS_DQEgQEgQEeQEp10mallocTimelS_DQFiQFiQFgQFr10numMallocslTmTkTmTxQCzZQFcMFNbKmKkKmKxQDsZQDl]: 961498
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC__T9runLockedS_DQCeQCeQCcQCnQBs18fullCollectNoStackMFNbZ2goFNbPSQEaQEaQDyQEj3GcxZmTQvZQDfMFNbKQBgZm]: 1
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC__T9runLockedS_DQCeQCeQCcQCnQBs7getAttrMFNbPvZ2goFNbPSQDqQDqQDoQDz3GcxQBbZkS_DQEoQEoQEmQEx9otherTimelS_DQFoQFoQFmQFx9numOtherslTQCyTQDlZQFdMFNbKQDoKQEbZk]: 5615
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw15LargeObjectPool10allocPagesMFNbmZm]: 5597
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw15LargeObjectPool13updateOffsetsMFNbmZv]: 10745
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw15LargeObjectPool7getInfoMFNbPvZS4core6memory8BlkInfo_]: 3844
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw15SmallObjectPool7getInfoMFNbPvZS4core6memory8BlkInfo_]: 40197
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw15SmallObjectPool9allocPageMFNbhZPSQChQChQCfQCq4List]: 15022
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw3Gcx10smallAllocMFNbhKmkZ8tryAllocMFNbZb]: 955967
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw3Gcx10smallAllocMFNbhKmkZPv]: 955912
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw3Gcx11ToScanStack4growMFNbZv]: 1
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw3Gcx11fullcollectMFNbbZm]: 85
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw3Gcx11removeRangeMFNbNiPvZv]: 1
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw3Gcx23updateCollectThresholdsMFNbZv]: 84
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw3Gcx4markMFNbNlPvQcZv]: 253
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw3Gcx5sweepMFNbZm]: 84
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw3Gcx7markAllMFNbbZ14__foreachbody3MFNbKSQCm11gcinterface5RangeZi]: 85
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw3Gcx7markAllMFNbbZv]: 85
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw3Gcx7newPoolMFNbmbZPSQBtQBtQBrQCc4Pool]: 6
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw3Gcx7recoverMFNbZm]: 84
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw3Gcx8addRangeMFNbNiPvQcxC8TypeInfoZv]: 2
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw3Gcx8bigAllocMFNbmKmkxC8TypeInfoZ15tryAllocNewPoolMFNbZb]: 5
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw3Gcx8bigAllocMFNbmKmkxC8TypeInfoZ8tryAllocMFNbZb]: 5616
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw3Gcx8bigAllocMFNbmKmkxC8TypeInfoZPv]: 5586
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw3Gcx8isMarkedMFNbNlPvZi]: 635
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw3Gcx9allocPageMFNbhZPSQBuQBuQBsQCd4List]: 15024
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw4Pool10initializeMFNbmbZv]: 6
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw4Pool12freePageBitsMFNbmKxG4mZv]: 16439
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl5protoQo7ProtoGC4termMFZv]: 1
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl5protoQo7ProtoGC6qallocMFNbmkxC8TypeInfoZS4core6memory8BlkInfo_]: 1
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl5protoQo7ProtoGC8addRangeMFNbNiPvmxC8TypeInfoZv]: 1
@[uprobe:/usr/local/bin/dscanner:_D2gc4impl6manualQp8ManualGC10initializeFKCQBp11gcinterface2GCZv]: 1
@[uprobe:/usr/local/bin/dscanner:_D2gc9pooltable__T9PoolTableTSQBc4impl12conservativeQBy4PoolZQBr6insertMFNbNiPQBxZb]: 6
@[uprobe:/usr/local/bin/dscanner:_D2gc9pooltable__T9PoolTableTSQBc4impl12conservativeQBy4PoolZQBr8findPoolMFNaNbNiPvZPQCe]: 302268
@[uprobe:/usr/local/bin/dscanner:_D2gc9pooltable__T9PoolTableTSQBc4impl12conservativeQBy4PoolZQBr8minimizeMFNaNbNjZAPQCd]: 30
Attaching 231 probes...
```
All these functions are in [`src/gc/`][12], and most of the interesting ones here are in [`src/gc/impl/conservative/`][13]. There are 85 calls to `_D2gc4impl12conservativeQw3Gcx11fullcollectMFNbbZm`, which [`ddemangle`][14] translates to `nothrow ulong gc.impl.conservative.gc.Gcx.fullcollect(bool)`. That matches up with the report from `--DRT-gcopt=profile:1`.
The heart of the `bpftrace` program is `@[probe] = count();`. `@` prefixes a global variable, in this case a variable with an empty name (allowed by `bpftrace`). We’re using the variable as a map (like an associative array in D), and indexing it with `probe`, a built-in variable containing the name of the uprobe that was triggered. The tally is kept using the magic `count()` function.
### Garbage collection timings
How about something more interesting, like generating a profile of collection timings? This time, to get more data, I won’t make `bpftrace` exit as soon as the `dscanner` exits. I’ll keep it running and run `dscanner` 100 times before quitting `bpftrace` with `Ctrl+C`:
```
# cat gcprofile.bt
uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw3Gcx11fullcollectMFNbbZm
{
@t = nsecs;
}
uretprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw3Gcx11fullcollectMFNbbZm / @t /
{
@gc_times = hist(nsecs - @t);
}
# bpftrace gcprofile.bt
Attaching 2 probes...
^C
@gc_times:
[64K, 128K) 138 |@ |
[128K, 256K) 1367 |@@@@@@@@@@ |
[256K, 512K) 6687 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[512K, 1M) 7 | |
[1M, 2M) 301 |@@ |
```
Et voila! A log-scale histogram of the `nsecs` timestamp difference between entering and exiting `fullcollect()`. The times are in nanoseconds, so we see that most collections are taking less than half a millisecond, but we have tail cases that take 1-2ms.
### Function arguments
`bpftrace` provides `arg0`, `arg1`, `arg2`, etc. built-in variables for accessing the arguments to a traced function. There are a couple of complications with using them with D code, however.
The first is that (at the binary level) `dmd` makes `extern(D)` functions (i.e., normal D functions) take arguments in the reverse order of `extern(C)` functions (that `bpftrace` is expecting). Suppose you have a simple three-argument function. If it’s using the C calling convention, `bpftrace` will recognise the first argument as `arg0`. If it’s using the D calling convention, however, it’ll be picked up as `arg2`.
```
extern(C) void cfunc(int arg0, int arg1, int arg2)
{
// ...
}
// (extern(D) is the default)
extern(D) void dfunc(int arg2, int arg1, int arg0)
{
// ...
}
```
If you look at [the D ABI spec][15], you’ll notice that (just like in C++) there can be a couple of hidden arguments if the function is more complex. If `dfunc` above returned a large struct, there can be an extra hidden argument for implementing [copy elision][16], which means the first argument would actually be `arg3`, and `arg0` would be the hidden argument. If `dfunc` were also a member function, it would have a hidden `this` argument, which would bump up the first argument to `arg4`.
To get the hang of this, you might need to experiment with tracing function calls with known arguments.
### Allocation sizes
Let’s get a histogram of the memory allocation request sizes. Looking at the list of GC functions traced earlier, and comparing it with the GC source code, it looks like we need to trace these functions and grab the `size` argument:
```
class ConservativeGC : GC
{
// ...
void *malloc(size_t size, uint bits, const TypeInfo ti) nothrow;
void *calloc(size_t size, uint bits, const TypeInfo ti) nothrow;
BlkInfo qalloc( size_t size, uint bits, const TypeInfo ti) nothrow;
// ...
}
```
As class member functions, they have a hidden `this` argument as well. The last one, `qalloc()`, returns a struct, so it also has a hidden argument for copy elision. So `size` is `arg3` for the first two functions, and `arg4` for `qalloc()`. Time to run a trace:
```
# cat allocsizeprofile.bt
uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC6mallocMFNbmkxC8TypeInfoZPv,
uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC6callocMFNbmkxC8TypeInfoZPv
{
@ = hist(arg3);
}
uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC6qallocMFNbmkxC8TypeInfoZS4core6memory8BlkInfo_
{
@ = hist(arg4);
}
uretprobe:/usr/local/bin/dscanner:_Dmain
{
exit();
}
# bpftrace allocsizeprofile.bt
Attaching 4 probes...
@:
[2, 4) 2489 | |
[4, 8) 9324 |@ |
[8, 16) 46527 |@@@@@ |
[16, 32) 206324 |@@@@@@@@@@@@@@@@@@@@@@@ |
[32, 64) 448020 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[64, 128) 147053 |@@@@@@@@@@@@@@@@@ |
[128, 256) 88072 |@@@@@@@@@@ |
[256, 512) 2519 | |
[512, 1K) 1830 | |
[1K, 2K) 3749 | |
[2K, 4K) 1668 | |
[4K, 8K) 256 | |
[8K, 16K) 2533 | |
[16K, 32K) 312 | |
[32K, 64K) 239 | |
[64K, 128K) 209 | |
[128K, 256K) 164 | |
[256K, 512K) 124 | |
[512K, 1M) 48 | |
[1M, 2M) 30 | |
[2M, 4M) 7 | |
[4M, 8M) 1 | |
[8M, 16M) 2 | |
```
So, we have a lot of small allocations, with a very long tail of larger allocations. Remember, size is on a log scale, so that long tail represents a very skewed distribution.
### Small allocation hotspots
Now for something more complex. Suppose we’re profiling our code and looking for low-hanging fruit for reducing the number of memory allocations. Code that makes a lot of small allocations tends to be a good candidate for this kind of refactoring. `bpftrace` lets us grab stack traces, which can be used to see what part of the main program caused an allocation.
As of writing, there’s one little complication because of a limitation of `bpftrace`’s stack trace handling: it can only show meaningful function symbol names (as opposed to raw memory addresses) if `bpftrace` quits while the target program is still running. There’s [an open bug report for improving this behaviour][17], but in the meantime I just made sure `dscanner` took a long time, and that I shut down `bpftrace` first.
Here’s how to grab the top three stack traces that lead to small (&lt;16B) memory allocations with `qalloc()`:
```
# cat smallallocs.bt
uprobe:/usr/local/bin/dscanner:_D2gc4impl12conservativeQw14ConservativeGC6qallocMFNbmkxC8TypeInfoZS4core6memory8BlkInfo_
{
if (arg4 < 16)
{
@[ustack] = count();
}
}
END
{
print(@, 3);
clear(@);
}
# bpftrace smallallocs.bt
Attaching 2 probes...
^C@[
_D2gc4impl12conservativeQw14ConservativeGC6qallocMFNbmkxC8TypeInfoZS4core6memory8BlkInfo_+0
_D2rt8lifetime12__arrayAllocFNaNbmxC8TypeInfoxQlZS4core6memory8BlkInfo_+236
_d_arraysetlengthT+248
_D8dscanner8analysis25label_var_same_name_check17LabelVarNameCheck9pushScopeMFZv+29
_D8dscanner8analysis25label_var_same_name_check17LabelVarNameCheck9__mixin175visitMFxC6dparse3ast6ModuleZv+21
_D8dscanner8analysis3run7analyzeFAyaxC6dparse3ast6ModulexSQCeQBy6config20StaticAnalysisConfigKS7dsymbol11modulecache11ModuleCacheAxS3std12experimental5lexer__T14TokenStructureThVQFpa305_0a20202020737472696e6720636f6d6d656e743b0a20202020737472696e6720747261696c696e67436f6d6d656e743b0a0a20202020696e74206f70436d702873697a655f7420692920636f6e73742070757265206e6f7468726f77204073616665207b0a202020202020202069662028696e646578203c2069292072657475726e202d313b0a202020202020202069662028696e646578203e2069292072657475726e20313b0a202020202020202072657475726e20303b0a202020207d0a0a20202020696e74206f70436d702872656620636f6e737420747970656f66287468697329206f746865722920636f6e73742070757265206e6f7468726f77204073616665207b0a202020202020202072657475726e206f70436d70286f746865722e696e646578293b0a202020207d0aZQYobZCQZv9container6rbtree__T12RedBlackTreeTSQBGiQBGd4base7MessageVQBFza62_20612e6c696e65203c20622e6c696e65207c7c2028612e6c696e65203d3d20622e6c696e6520262620612e636f6c756d6e203c20622e636f6c756d6e2920Vbi1ZQGt+11343
_D8dscanner8analysis3run7analyzeFAAyaxSQBlQBf6config20StaticAnalysisConfigQBoKS6dparse5lexer11StringCacheKS7dsymbol11modulecache11ModuleCachebZb+337
_Dmain+3618
_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFZv+40
_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ7tryExecMFMDFZvZv+32
_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZv+139
_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ7tryExecMFMDFZvZv+32
_d_run_main+463
main+16
__libc_start_main+235
0x41fd89415541f689
]: 450
@[
_D2gc4impl12conservativeQw14ConservativeGC6qallocMFNbmkxC8TypeInfoZS4core6memory8BlkInfo_+0
_D2rt8lifetime12__arrayAllocFNaNbmxC8TypeInfoxQlZS4core6memory8BlkInfo_+236
_d_arrayappendcTX+1944
_D8dscanner8analysis10unmodified16UnmodifiedFinder9pushScopeMFZv+61
_D8dscanner8analysis10unmodified16UnmodifiedFinder5visitMFxC6dparse3ast6ModuleZv+21
_D8dscanner8analysis3run7analyzeFAyaxC6dparse3ast6ModulexSQCeQBy6config20StaticAnalysisConfigKS7dsymbol11modulecache11ModuleCacheAxS3std12experimental5lexer__T14TokenStructureThVQFpa305_0a20202020737472696e6720636f6d6d656e743b0a20202020737472696e6720747261696c696e67436f6d6d656e743b0a0a20202020696e74206f70436d702873697a655f7420692920636f6e73742070757265206e6f7468726f77204073616665207b0a202020202020202069662028696e646578203c2069292072657475726e202d313b0a202020202020202069662028696e646578203e2069292072657475726e20313b0a202020202020202072657475726e20303b0a202020207d0a0a20202020696e74206f70436d702872656620636f6e737420747970656f66287468697329206f746865722920636f6e73742070757265206e6f7468726f77204073616665207b0a202020202020202072657475726e206f70436d70286f746865722e696e646578293b0a202020207d0aZQYobZCQZv9container6rbtree__T12RedBlackTreeTSQBGiQBGd4base7MessageVQBFza62_20612e6c696e65203c20622e6c696e65207c7c2028612e6c696e65203d3d20622e6c696e6520262620612e636f6c756d6e203c20622e636f6c756d6e2920Vbi1ZQGt+11343
_D8dscanner8analysis3run7analyzeFAAyaxSQBlQBf6config20StaticAnalysisConfigQBoKS6dparse5lexer11StringCacheKS7dsymbol11modulecache11ModuleCachebZb+337
_Dmain+3618
_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFZv+40
_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ7tryExecMFMDFZvZv+32
_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZv+139
_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ7tryExecMFMDFZvZv+32
_d_run_main+463
main+16
__libc_start_main+235
0x41fd89415541f689
]: 450
@[
_D2gc4impl12conservativeQw14ConservativeGC6qallocMFNbmkxC8TypeInfoZS4core6memory8BlkInfo_+0
_D2rt8lifetime12__arrayAllocFNaNbmxC8TypeInfoxQlZS4core6memory8BlkInfo_+236
_d_arrayappendcTX+1944
_D8dscanner8analysis3run7analyzeFAyaxC6dparse3ast6ModulexSQCeQBy6config20StaticAnalysisConfigKS7dsymbol11modulecache11ModuleCacheAxS3std12experimental5lexer__T14TokenStructureThVQFpa305_0a20202020737472696e6720636f6d6d656e743b0a20202020737472696e6720747261696c696e67436f6d6d656e743b0a0a20202020696e74206f70436d702873697a655f7420692920636f6e73742070757265206e6f7468726f77204073616665207b0a202020202020202069662028696e646578203c2069292072657475726e202d313b0a202020202020202069662028696e646578203e2069292072657475726e20313b0a202020202020202072657475726e20303b0a202020207d0a0a20202020696e74206f70436d702872656620636f6e737420747970656f66287468697329206f746865722920636f6e73742070757265206e6f7468726f77204073616665207b0a202020202020202072657475726e206f70436d70286f746865722e696e646578293b0a202020207d0aZQYobZCQZv9container6rbtree__T12RedBlackTreeTSQBGiQBGd4base7MessageVQBFza62_20612e6c696e65203c20622e6c696e65207c7c2028612e6c696e65203d3d20622e6c696e6520262620612e636f6c756d6e203c20622e636f6c756d6e2920Vbi1ZQGt+680
_D8dscanner8analysis3run7analyzeFAAyaxSQBlQBf6config20StaticAnalysisConfigQBoKS6dparse5lexer11StringCacheKS7dsymbol11modulecache11ModuleCachebZb+337
_Dmain+3618
_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFZv+40
_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ7tryExecMFMDFZvZv+32
_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZv+139
_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ7tryExecMFMDFZvZv+32
_d_run_main+463
main+16
__libc_start_main+235
0x41fd89415541f689
]: 450
```
It looks like a lot of the small allocations are due to a red-black tree in `ModuleCache`.
### What’s next?
I think these examples already show that `bpftrace` is a pretty powerful tool. There’s a lot more that can done, and I highly recommended reading [Brendan Gregg’s eBPF tutorials][18].
I used uprobes to trace arbitrary functions in the D runtime. The pro of this is the freedom to do anything, but the cons are that I had to refer to the D runtime source code and manually deal with the D ABI. There’s also no guarantee that a script I write today will work with future versions of the runtime. Linux also supports making well-defined tracepoints in user code using a feature called [USDT][19]. That should let D code export stable tracepoints that can be used without worrying about the D ABI. I might do more experiments in future.
--------------------------------------------------------------------------------
via: https://theartofmachinery.com/2019/04/26/bpftrace_d_gc.html
作者：[Simon Arneaud][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://theartofmachinery.com
[b]: https://github.com/lujun9972
[1]: https://github.com/iovisor/bpftrace
[2]: https://en.wikipedia.org/wiki/Java_virtual_machine
[3]: http://dtrace.org/blogs/about/
[4]: https://github.com/iovisor/bpftrace/blob/master/INSTALL.md
[5]: https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md
[6]: https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md
[7]: https://dlang.org/spec/abi.html#name_mangling
[8]: https://github.com/dlang-community/D-Scanner
[9]: https://github.com/dlang/druntime/
[10]: https://dlang.org/spec/garbage.html#gc_config
[11]: https://dlang.org/phobos/core_memory.html#.GC.stats
[12]: https://github.com/dlang/druntime/tree/v2.081.1/src/gc
[13]: https://github.com/dlang/druntime/tree/v2.081.1/src/gc/impl/conservative
[14]: https://github.com/dlang/tools
[15]: https://dlang.org/spec/abi.html#parameters
[16]: https://en.wikipedia.org/wiki/Copy_elision
[17]: https://github.com/iovisor/bpftrace/issues/246
[18]: http://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.html
[19]: https://lwn.net/Articles/753601/

680
sources/talk/20190528 Analysing D Code with KLEE.md Normal file
View File

@@ -0,0 +1,680 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Analysing D Code with KLEE)
[#]: via: (https://theartofmachinery.com/2019/05/28/d_and_klee.html)
[#]: author: (Simon Arneaud https://theartofmachinery.com)
Analysing D Code with KLEE
======
[KLEE][1] is symbolic execution engine that can rigorously verify or find bugs in software. It’s designed for C and C++, but it’s just an interpreter for LLVM bitcode combined with theorem prover backends, so it can work with bitcode generated by `ldc2`. One catch is that it needs a compatible bitcode port of the D runtime to run normal D code. I’m still interested in getting KLEE to work with normal D code, but for now I’ve done some experiments with `-betterC` D.
### How KLEE works
What makes KLEE special is its support for two kinds of variables: concrete and symbolic. Concrete variables are just like the normal variables in normal code: they have a deterministic value at any given point in the program. On the other hand, symbolic variables contain a bundle of logical constraints instead of values. Take this code:
```
int x = klee_int("x");
klee_assume(x >= 0);
if (x > 42)
{
doA(x);
}
else
{
doB(x);
assert (3 * x != 21);
}
```
`klee_int("x")` creates a symbolic integer that will be called “`x`” in output reports. Initially it has no contraints and can imply any value that a 32b signed integer can have. `klee_assume(x >= 0)` tells KLEE to add `x >= 0` as a constraint, so now we’re only analysing the code for non-negative 32b signed integers. On hitting the `if`, KLEE checks if both branches are possible. Sure enough, `x > 42` can be true or false even with the constraint `x >= 0`, so KLEE has to _fork_. We now have two processes being interpreted on the VM: one executing `doA()` while `x` holds the constraints `x >= 0, x > 42`, and another executing `doB()` while `x` holds the contraints `x >= 0, x <= 42`. The second process will hit the `assert` statement, and KLEE will try to prove or disprove `3 * x != 21` using the assumptions `x >= 0, x <= 42` — in this case it will disprove it and report a bug with `x = 7` as a crashing example.
### First steps
Here’s a toy example just to get things working. Suppose we have a function that makes an assumption for a performance optimisation. Thankfully the assumption is made explicit with `assert` and is documented with a comment. Is the assumption valid?
```
int foo(int x)
{
// 17 is a prime number, so let's use it as a sentinel value for an awesome optimisation
assert (x * x != 17);
// ...
return x;
}
```
Here’s a KLEE test rig. The KLEE function declarations and the `main()` entry point need to have `extern(C)` linkage, but anything else can be normal D code as long as it compiles under `-betterC`:
```
extern(C):
int klee_int(const(char*) name);
int main()
{
int x = klee_int("x");
foo(x);
return 0;
}
```
It turns out there’s just one (frustrating) complication with running `-betterC` D under KLEE. In D, `assert` is handled specially by the compiler. By default, it throws an `Error`, but for compatibility with KLEE, I’m using the `-checkaction=C` flag. In C, `assert` is usually a macro that translates to code that calls some backend implementation. That implementation isn’t standardised, so of course various C libraries work differently. `ldc2` actually has built-in logic for implementing `-checkaction=C` correctly depending on the C library used.
KLEE uses a port of [uClibc][2], which translates `assert()` to a four-parameter `__assert()` function, which conflicts with the three-parameter `__assert()` function in other implementations. `ldc2` uses LLVM’s (target) `Triple` type for choosing an `assert()` implementation configuration, but that doesn’t recognise uClibc. As a hacky workaround, I’m telling `ldc2` to compile for Musl, which “tricks” it into using an `__assert_fail()` implementation that KLEE happens to support as well. I’ve opened [an issue report][3].
Anyway, if we put all that code above into a file, we can compile it to KLEE-ready bitcode like this:
```
ldc2 -g -checkaction=C -mtriple=x86_64-linux-musl -output-bc -betterC -c first.d
```
`-g` is optional, but adds debug information that can be useful for later analysis. The KLEE developers recommend disabling compiler optimisations and letting KLEE do its own optimisations instead.
Now to run KLEE:
```
$ klee first.bc
KLEE: output directory is "/tmp/klee-out-1"
KLEE: Using Z3 solver backend
warning: Linking two modules of different target triples: klee_int.bc' is 'x86_64-pc-linux-gnu' whereas 'first.bc' is 'x86_64--linux-musl'
KLEE: ERROR: first.d:4: ASSERTION FAIL: x * x != 17
KLEE: NOTE: now ignoring this error at this location
KLEE: done: total instructions = 35
KLEE: done: completed paths = 2
KLEE: done: generated tests = 2
```
Straight away, KLEE has found two execution paths through the program: a happy path, and a path that fails the assertion. Let’s see the results:
```
$ ls klee-last/
assembly.ll
info
messages.txt
run.istats
run.stats
run.stats-journal
test000001.assert.err
test000001.kquery
test000001.ktest
test000002.ktest
warnings.txt
```
Here’s the example that triggers the happy path:
```
$ ktest-tool klee-last/test000002.ktest
ktest file : 'klee-last/test000002.ktest'
args : ['first.bc']
num objects: 1
object 0: name: 'x'
object 0: size: 4
object 0: data: b'\x00\x00\x00\x00'
object 0: hex : 0x00000000
object 0: int : 0
object 0: uint: 0
object 0: text: ....
```
Here’s the example that causes an assertion error:
```
$ cat klee-last/test000001.assert.err
Error: ASSERTION FAIL: x * x != 17
File: first.d
Line: 4
assembly.ll line: 32
Stack:
#000000032 in _D5first3fooFiZi () at first.d:4
#100000055 in main (=1, =94262044506880) at first.d:16
$ ktest-tool klee-last/test000001.ktest
ktest file : 'klee-last/test000001.ktest'
args : ['first.bc']
num objects: 1
object 0: name: 'x'
object 0: size: 4
object 0: data: b'\xe9&\xd33'
object 0: hex : 0xe926d333
object 0: int : 869476073
object 0: uint: 869476073
object 0: text: .&.3
```
So, KLEE has deduced that when `x` is 869476073, `x * x` does a 32b overflow to 17 and breaks the code.
It’s overkill for this simple example, but `run.istats` can be opened with [KCachegrind][4] to view things like call graphs and source code coverage. (Unfortunately, coverage stats can be misleading because correct code won’t ever hit boundary check code inserted by the compiler.)
### MurmurHash preimage
Here’s a slightly more useful example. D currently uses 32b MurmurHash3 as its standard non-cryptographic hash function. What if we want to find strings that hash to a given special value? In general, we can solve problems like this by asserting that something doesn’t exist (i.e., a string that hashes to a given value) and then challenging the theorem prover to prove us wrong with a counterexample.
Unfortunately, we can’t just use `hashOf()` directly without the runtime, but we can copy [the hash code from the runtime source][5] into its own module, and then import it into a test rig like this:
```
import dhash;
extern(C):
void klee_make_symbolic(void* addr, size_t nbytes, const(char*) name);
int klee_assume(ulong condition);
int main()
{
// Create a buffer for 8-letter strings and let KLEE manage it symbolically
char[8] s;
klee_make_symbolic(s.ptr, s.sizeof, "s");
// Constrain the string to be letters from a to z for convenience
foreach (j; 0..s.length)
{
klee_assume(s[j] > 'a' && s[j] <= 'z');
}
assert (dHash(cast(ubyte[])s) != 0xdeadbeef);
return 0;
}
```
Here’s how to compile and run it. Because we’re not checking correctness, we can use `-boundscheck=off` for a slight performance boost. It’s also worth enabling KLEE’s optimiser.
```
$ ldc2 -g -boundscheck=off -checkaction=C -mtriple=x86_64-linux-musl -output-bc -betterC -c dhash.d dhash_klee.d
$ llvm-link -o dhash_test.bc dhash.bc dhash_klee.bc
$ klee -optimize dhash_test.bc
```
It takes just over 4s:
```
$ klee-stats klee-last/
-------------------------------------------------------------------------
| Path | Instrs| Time(s)| ICov(%)| BCov(%)| ICount| TSolver(%)|
-------------------------------------------------------------------------
|klee-last/| 168| 4.37| 87.50| 50.00| 160| 99.95|
-------------------------------------------------------------------------
```
And it actually works:
```
$ ktest-tool klee-last/test000001.ktest
ktest file : 'klee-last/test000001.ktest'
args : ['dhash_test.bc']
num objects: 1
object 0: name: 's'
object 0: size: 8
object 0: data: b'psgmdxvq'
object 0: hex : 0x7073676d64787671
object 0: int : 8175854546265273200
object 0: uint: 8175854546265273200
object 0: text: psgmdxvq
$ rdmd --eval 'writef("%x\n", hashOf("psgmdxvq"));'
deadbeef
```
For comparison, here’s a simple brute force version in plain D:
```
import std.stdio;
void main()
{
char[8] buffer;
bool find(size_t idx)
{
if (idx == buffer.length)
{
auto hash = hashOf(buffer[]);
if (hash == 0xdeadbeef)
{
writeln(buffer[]);
return true;
}
return false;
}
foreach (char c; 'a'..'z')
{
buffer[idx] = c;
auto is_found = find(idx + 1);
if (is_found) return true;
}
return false;
}
find(0);
}
```
This takes ~17s:
```
$ ldc2 -O3 -boundscheck=off hash_brute.d
$ time ./hash_brute
aexkaydh
real 0m17.398s
user 0m17.397s
sys 0m0.001s
$ rdmd --eval 'writef("%x\n", hashOf("aexkaydh"));'
deadbeef
```
The constraint solver implementation is simpler to write, but is still faster because it can automatically do smarter things than calculating hashes of strings from scratch every iteration.
### Binary search
Now for an example of testing and debugging. Here’s an implementation of [binary search][6]:
```
bool bsearch(const(int)[] haystack, int needle)
{
while (haystack.length)
{
auto mid_idx = haystack.length / 2;
if (haystack[mid_idx] == needle) return true;
if (haystack[mid_idx] < needle)
{
haystack = haystack[mid_idx..$];
}
else
{
haystack = haystack[0..mid_idx];
}
}
return false;
}
```
Does it work? Here’s a test rig:
```
extern(C):
void klee_make_symbolic(void* addr, size_t nbytes, const(char*) name);
int klee_range(int begin, int end, const(char*) name);
int klee_assume(ulong condition);
int main()
{
// Making an array arr and an x to find in it.
// This time we'll also parameterise the array length.
// We have to apply klee_make_symbolic() to the whole buffer because of limitations in KLEE.
int[8] arr_buffer;
klee_make_symbolic(arr_buffer.ptr, arr_buffer.sizeof, "a");
int len = klee_range(0, arr_buffer.length+1, "len");
auto arr = arr_buffer[0..len];
// Keeping the values in [0, 32) makes the output easier to read.
// (The binary-friendly limit 32 is slightly more efficient than 30.)
int x = klee_range(0, 32, "x");
foreach (j; 0..arr.length)
{
klee_assume(arr[j] >= 0);
klee_assume(arr[j] < 32);
}
// Make the array sorted.
// We don't have to actually sort the array.
// We can just tell KLEE to constrain it to be sorted.
foreach (j; 1..arr.length)
{
klee_assume(arr[j - 1] <= arr[j]);
}
// Test against simple linear search
bool has_x = false;
foreach (a; arr[])
{
has_x |= a == x;
}
assert (bsearch(arr, x) == has_x);
return 0;
}
```
When run in KLEE, it keeps running for a long, long time. How do we know it’s doing anything? By default KLEE writes stats every 1s, so we can watch the live progress in another terminal:
```
$ watch klee-stats --print-more klee-last/
Every 2.0s: klee-stats --print-more klee-last/
---------------------------------------------------------------------------------------------------------------------
| Path | Instrs| Time(s)| ICov(%)| BCov(%)| ICount| TSolver(%)| States| maxStates| Mem(MB)| maxMem(MB)|
---------------------------------------------------------------------------------------------------------------------
|klee-last/| 5834| 637.27| 79.07| 68.75| 172| 100.00| 22| 22| 24.51| 24|
---------------------------------------------------------------------------------------------------------------------
```
`bsearch()` should be pretty fast, so we should see KLEE discovering new states rapidly. But instead it seems to be stuck. [At least one fork of KLEE has heuristics for detecting infinite loops][7], but plain KLEE doesn’t. There are timeout and batching options for making KLEE work better with code that might have infinite loops, but let’s just take another look at the code. In particular, the loop condition:
```
while (haystack.length)
{
// ...
}
```
Binary search is supposed to reduce the search space by about half each iteration. `haystack.length` is an unsigned integer, so the loop must terminate as long as it goes down every iteration. Let’s rewrite the code slightly so we can verify if that’s true:
```
bool bsearch(const(int)[] haystack, int needle)
{
while (haystack.length)
{
auto mid_idx = haystack.length / 2;
if (haystack[mid_idx] == needle) return true;
const(int)[] next_haystack;
if (haystack[mid_idx] < needle)
{
next_haystack = haystack[mid_idx..$];
}
else
{
next_haystack = haystack[0..mid_idx];
}
// This lets us verify that the search terminates
assert (next_haystack.length < haystack.length);
haystack = next_haystack;
}
return false;
}
```
Now KLEE can find the bug!
```
$ klee -optimize bsearch.bc
KLEE: output directory is "/tmp/klee-out-2"
KLEE: Using Z3 solver backend
warning: Linking two modules of different target triples: klee_range.bc' is 'x86_64-pc-linux-gnu' whereas 'bsearch.bc' is 'x86_64--linux-musl'
warning: Linking two modules of different target triples: memset.bc' is 'x86_64-pc-linux-gnu' whereas 'bsearch.bc' is 'x86_64--linux-musl'
KLEE: ERROR: bsearch.d:18: ASSERTION FAIL: next_haystack.length < haystack.length
KLEE: NOTE: now ignoring this error at this location
KLEE: done: total instructions = 2281
KLEE: done: completed paths = 42
KLEE: done: generated tests = 31
```
Using the failing example as input and stepping through the code, it’s easy to find the problem:
```
/// ...
if (haystack[mid_idx] < needle)
{
// If mid_idx == 0, next_haystack is the same as haystack
// Nothing changes, so the loop keeps repeating
next_haystack = haystack[mid_idx..$];
}
/// ...
```
Thinking about it, the `if` statement already excludes `haystack[mid_idx]` from being `needle`, so there’s no reason to include it in `next_haystack`. Here’s the fix:
```
// The +1 matters
next_haystack = haystack[mid_idx+1..$];
```
But is the code correct now? Terminating isn’t enough; it needs to get the right answer, of course.
```
$ klee -optimize bsearch.bc
KLEE: output directory is "/tmp/kee-out-3"
KLEE: Using Z3 solver backend
warning: Linking two modules of different target triples: klee_range.bc' is 'x86_64-pc-linux-gnu' whereas 'bsearch.bc' is 'x86_64--linux-musl'
warning: Linking two modules of different target triples: memset.bc' is 'x86_64-pc-linux-gnu' whereas 'bsearch.bc' is 'x86_64--linux-musl'
KLEE: done: total instructions = 3152
KLEE: done: completed paths = 81
KLEE: done: generated tests = 81
```
In just under 7s, KLEE has verified every possible execution path reachable with arrays of length from 0 to 8. Note, that’s not just coverage of individual code lines, but coverage of full pathways through the code. KLEE hasn’t ruled out stack corruption or integer overflows with large arrays, but I’m pretty confident the code is correct now.
KLEE has generated test cases that trigger each path, which we can keep and use as a faster-than-7s regression test suite. Trouble is, the output from KLEE loses all type information and isn’t in a convenient format:
```
$ ktest-tool klee-last/test000042.ktest
ktest file : 'klee-last/test000042.ktest'
args : ['bsearch.bc']
num objects: 3
object 0: name: 'a'
object 0: size: 32
object 0: data: b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
object 0: hex : 0x0000000000000000000000000000000001000000100000000000000000000000
object 0: text: ................................
object 1: name: 'x'
object 1: size: 4
object 1: data: b'\x01\x00\x00\x00'
object 1: hex : 0x01000000
object 1: int : 1
object 1: uint: 1
object 1: text: ....
object 2: name: 'len'
object 2: size: 4
object 2: data: b'\x06\x00\x00\x00'
object 2: hex : 0x06000000
object 2: int : 6
object 2: uint: 6
object 2: text: ....
```
But we can write our own pretty-printing code and put it at the end of the test rig:
```
char[256] buffer;
char* output = buffer.ptr;
output += sprintf(output, "TestCase([");
foreach (a; arr[])
{
output += sprintf(output, "%d, ", klee_get_value_i32(a));
}
sprintf(output, "], %d, %s),\n", klee_get_value_i32(x), klee_get_value_i32(has_x) ? "true".ptr : "false".ptr);
fputs(buffer.ptr, stdout);
```
Ugh, that would be just one format call with D’s `%(` array formatting specs. The output needs to be buffered up and printed all at once to stop output from different parallel executions getting mixed up. `klee_get_value_i32()` is needed to get a concrete example from a symbolic variable (remember that a symbolic variable is just a bundle of constraints).
```
$ klee -optimize bsearch.bc > tests.d
...
$ # Sure enough, 81 test cases
$ wc -l tests.d
81 tests.d
$ # Look at the first 10
$ head tests.d
TestCase([], 0, false),
TestCase([0, ], 0, true),
TestCase([16, ], 1, false),
TestCase([0, ], 1, false),
TestCase([0, 0, ], 0, true),
TestCase([0, 0, ], 1, false),
TestCase([1, 16, ], 1, true),
TestCase([0, 0, 0, ], 0, true),
TestCase([16, 16, ], 1, false),
TestCase([1, 16, ], 3, false),
```
Nice! An autogenerated regression test suite that’s better than anything I would write by hand. This is my favourite use case for KLEE.
### Change counting
One last example:
In Australia, coins come in 5c, 10c, 20c, 50c, $1 (100c) and $2 (200c) denominations. So you can make 70c using 14 5c coins, or using a 50c coin and a 20c coin. Obviously, fewer coins is usually more convenient. There’s a simple [greedy algorithm][8] to make a small pile of coins that adds up to a given value: just keep adding the biggest coin you can to the pile until you’ve reached the target value. It turns out this trick is optimal — at least for Australian coins. Is it always optimal for any set of coin denominations?
The hard thing about testing optimality is that you don’t know what the correct optimal values are without a known-good algorithm. Without a constraints solver, I’d compare the output of the greedy algorithm with some obviously correct brute force optimiser, run over all possible cases within some small-enough limit. But with KLEE, we can use a different approach: comparing the greedy solution to a non-deterministic solution.
The greedy algorithm takes the list of coin denominations and the target value as input, so (like in the previous examples) we make those symbolic. Then we make another symbolic array that represents an assignment of coin counts to each coin denomination. We don’t specify anything about how to generate this assignment, but we constrain it to be a valid assignment that adds up to the target value. It’s [non-deterministic][9]. Then we just assert that the total number of coins in the non-deterministic assignment is at least the number of coins needed by the greedy algorithm, which would be true if the greedy algorithm were universally optimal. Finally we ask KLEE to prove the program correct or incorrect.
Here’s the code:
```
// Greedily break value into coins of values in denominations
// denominations must be in strictly decreasing order
int greedy(const(int[]) denominations, int value, int[] coins_used_output)
{
int num_coins = 0;
foreach (j; 0..denominations.length)
{
int num_to_use = value / denominations[j];
coins_used_output[j] = num_to_use;
num_coins += num_to_use;
value = value % denominations[j];
}
return num_coins;
}
extern(C):
void klee_make_symbolic(void* addr, size_t nbytes, const(char*) name);
int klee_int(const(char*) name);
int klee_assume(ulong condition);
int klee_get_value_i32(int expr);
int main(int argc, char** argv)
{
enum kNumDenominations = 6;
int[kNumDenominations] denominations, coins_used;
klee_make_symbolic(denominations.ptr, denominations.sizeof, "denominations");
// We're testing the algorithm itself, not implementation issues like integer overflow
// Keep values small
foreach (d; denominations)
{
klee_assume(d >= 1);
klee_assume(d <= 1024);
}
// Make the smallest denomination 1 so that all values can be represented
// This is just for simplicity so we can focus on optimality
klee_assume(denominations[$-1] == 1);
// Greedy algorithm expects values in descending order
foreach (j; 1..denominations.length)
{
klee_assume(denominations[j-1] > denominations[j]);
}
// What we're going to represent
auto value = klee_int("value");
auto num_coins = greedy(denominations[], value, coins_used[]);
// The non-deterministic assignment
int[kNumDenominations] nd_coins_used;
klee_make_symbolic(nd_coins_used.ptr, nd_coins_used.sizeof, "nd_coins_used");
int nd_num_coins = 0, nd_value = 0;
foreach (j; 0..kNumDenominations)
{
klee_assume(nd_coins_used[j] >= 0);
klee_assume(nd_coins_used[j] <= 1024);
nd_num_coins += nd_coins_used[j];
nd_value += nd_coins_used[j] * denominations[j];
}
// Making the assignment valid is 100% up to KLEE
klee_assume(nd_value == value);
// If we find a counterexample, dump it and fail
if (nd_num_coins < num_coins)
{
import core.stdc.stdio;
puts("Counterexample found.");
puts("Denominations:");
foreach (ref d; denominations)
{
printf("%d ", klee_get_value_i32(d));
}
printf("\nValue: %d\n", klee_get_value_i32(value));
void printAssignment(const ref int[kNumDenominations] coins)
{
foreach (j; 0..kNumDenominations)
{
printf("%d * %dc\n", klee_get_value_i32(coins[j]), klee_get_value_i32(denominations[j]));
}
}
printf("Greedy \"optimum\": %d\n", klee_get_value_i32(num_coins));
printAssignment(coins_used);
printf("Better assignment for %d total coins:\n", klee_get_value_i32(nd_num_coins));
printAssignment(nd_coins_used);
assert (false);
}
return 0;
}
```
And here’s the counterexample it found after 14s:
```
Counterexample found.
Denominations:
129 12 10 3 2 1
Value: 80
Greedy "optimum": 9
0 * 129c
6 * 12c
0 * 10c
2 * 3c
1 * 2c
0 * 1c
Better assignment for 8 total coins:
0 * 129c
0 * 12c
8 * 10c
0 * 3c
0 * 2c
0 * 1c
```
Note that this isn’t proven to be the new optimum; it’s just a witness that the greedy algorithm isn’t always optimal. There’s a well-known [dynamic programming][10] [solution][11] that always works.
### What’s next?
As I said, I’m interesting in getting this to work with full D code. I’m also interested in using [one of the floating point forks of KLEE][12] on some D because floating point is much harder to test thoroughly than integer and string code.
--------------------------------------------------------------------------------
via: https://theartofmachinery.com/2019/05/28/d_and_klee.html
作者：[Simon Arneaud][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://theartofmachinery.com
[b]: https://github.com/lujun9972
[1]: https://klee.github.io/
[2]: https://www.uclibc.org/
[3]: https://github.com/ldc-developers/ldc/issues/3078
[4]: https://kcachegrind.github.io/html/Home.html
[5]: https://github.com/dlang/druntime/blob/4ad638f61a9b4a98d8ed6eb9f9429c0ef6afc8e3/src/core/internal/hash.d#L670
[6]: https://www.calhoun.io/lets-learn-algorithms-an-intro-to-binary-search/
[7]: https://github.com/COMSYS/SymbolicLivenessAnalysis
[8]: https://en.wikipedia.org/wiki/Greedy_algorithm
[9]: http://people.clarkson.edu/~alexis/PCMI/Notes/lectureB03.pdf
[10]: https://www.algorithmist.com/index.php/Dynamic_Programming
[11]: https://www.topcoder.com/community/competitive-programming/tutorials/dynamic-programming-from-novice-to-advanced/
[12]: https://github.com/srg-imperial/klee-float

96
sources/talk/20190708 Colocation facilities buck the cloud-data-center trend.md Normal file
View File

@@ -0,0 +1,96 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Colocation facilities buck the cloud-data-center trend)
[#]: via: (https://www.networkworld.com/article/3407756/colocation-facilities-buck-the-cloud-data-center-trend.html)
[#]: author: (Andy Patrizio https://www.networkworld.com/author/Andy-Patrizio/)
Colocation facilities buck the cloud-data-center trend
======
Lower prices and latency plus easy access to multiple cloud providers make colocation facilities an attractive option compared to building on-site data centers.
![gorodenkoff / Getty Images][1]
[Data center][2] workloads are moving but not only to the cloud. Increasingly, they are shifting to colocation facilities as an alternative to privately owned data centers.
### What is colocation?
A colocation facility or colo is a data center in which a business can rent space for servers and other computing hardware that they purchase but that the colo provider manages.
[Read about IPv6 and cloud-access security brokers][3]
The colo company provides the building, cooling, power, bandwidth and physical security. Space is leased by the rack, cabinet, cage or room. Many colos started out as managed services and continue  to offer those specialized services.
Some prominent providers include Equinix, Digital Reality Trust, CenturyLink, and NTT Communications, and there are several Chinese providers that only serve the China market. Unlike the data centers of cloud vendors like Amazon and Microsoft, these colo facilities are generally in large metropolitan areas.
“Colos have been around a long time, but their initial use case was Web servers,” said Rick Villars, vice president of data centers and cloud research at IDC. “What’s changed now is the ratio of what’s customer-facing is much greater than in 2000, [with the]  expansion of companies needing to have more assets that are network-facing.”
### Advantages of colos: Cost, cloud interconnect
Homegrown data centers are often sized correctly, with either too much capacity or too little, said Jim Poole, vice president of business development at Equinix. “Customers come to us all the time and say, ‘Would you buy my data center? Because I only use 25 percent of it,’” he said.
Poole said the average capital expenditure for a stand-alone enterprise data center that is not a part of the corporate campus is $9 million. Companies are increasingly realizing that it makes sense to buy the racks of hardware but place it in someone else’s secure facility that handles the power and cooling. “It’s the same argument for doing cloud computing but at the physical-infrastructure level,” he said.
Mike Satter, vice president for OceanTech, a data-center-decommissioning service provider, says enterprises should absolutely outsource data-center construction or go the colo route. Just as there are contractors who specialize in building houses, there are experts who specialize in data-center design, he said.
He added that with many data-center closures there is subsequent consolidation. “For every decommissioning we do, that same company is adding to another environment somewhere else. With the new hardware out there now, the servers can do the same work in 20 racks as they did in 80 racks five years ago. That means a reduced footprint and energy cost,” he said.
Often these closures mean moving to a colo. OceanTech recently decommissioned a private data center for a major media outlet he declined to identify that involved shutting down a data center in New Jersey that held 70 racks of gear. The firm was going to move its apps to the cloud but ended up expanding to a colo facility in New York City.
### Cloud isn't cheaper than private data centers
Satter said he’s had conversations with companies that planned to go to the cloud but changed their minds when they saw what it would cost if they later decided to move workloads out. Cloud providers can “kill you with guidelines and costs” because your data is in their infrastructure, and they can set fees that make it expensive to move it to another provider, he said. “The cloud not a money saver.”
That can drive decisions to keep data in-house or in a colo in order to keep tighter possession of their data. “Early on, when people weren’t hip to the game for how much it cost to move to the cloud, you had decision makers with influence say the cloud sounded good. Now they are realizing it costs a lot more dollars to do that vs. doing something on-prem, on your own,” said Satter.
Guy Churchward, CEO of Datera, developer of software designed storage platforms for enterprises, has noticed a new trend among CIOs making a cloud vs. private decision for apps based on the lifespan of the app.
“Organizations don’t know how much resource they need to throw at a task. The cloud makes more sense for [short-term apps],” he said. For applications that will be used for five years or more, it makes more sense to place them in company-controlled facilities, he said. That's because with three-to-five-year hardware-refresh cycles, the hardware lasts the entire lifespan of the app, and the hardware and app can be retired at the same time.
Another force driving the decision of private data center vs. the cloud is machine learning. Churchward said that’s because machine learning is often done using large amounts of highly sensitive data, so customers wanted data kept securely in house. They also wanted a low-latency loop between their ML apps and the data lake from which they draw.
### Colos connect to mulitple cloud providers
Another allure of colocation providers is that they can act as a pipeline between enterprises and multiple cloud providers. So rather than directly connecting to AWS, Azure, etc., businesses can connect to a colo, and that colo acts like a giant switch, connecting them to cloud providers through dedicated, high-speed networks.
Villars notes the typical corporate data center is either inside corporate HQ or someplace remote, like South Dakota where land was cheap. But the trade-off is that network connectivity to remote locations is often slower and more expensive.
That’s where a data-center colo providers with a large footprints come in, since they have points of presence in major cities. No one would fault a New York City-based firm for putting its data center in upstate New York or even further away. But when Equinix, DTR, and others all have data centers right in New York City, customers might get faster and sometimes cheaper connections plus lower latency.
Steve Cretney, vice president and CIO for food distributor Colony Brands, is in the midst of migrating the company to the cloud and moving everything he can from his data center to AWS. Rather than connect directly to AWS, Colony’s Wisconsin headquarters is connected to an Equinix data center in Chicago.
Going with Equinix provides more and cheaper bandwidth to the cloud than buying direct connectivity on his own. “I effectively moved my data center into Chicago. Now I can compete with a better price on data communication and networks,” he said.
Cretney estimates that by moving Colony’s networking from a smaller, local provider to Chicago, the company is seeing an annual cost savings of 50 percent for network connectivity that includes telecommunications.
Also, Colony wants to adopt a mult-cloud-provider strategy to avoid vendor lock-in, and he gets that by using Equinix as his network connection. As the company eventually uses Microsoft Azure and Google Cloud and other providers, Equinex can provide flexible and economic interconnections, he said.
### **Colos reduce the need for enterprise data-center real estate**
In 2014, 80 percent of data-centers were owned by enterprises, while colos and the early cloud accounted for 20 percent, said Villars. Today that’s a 50-50 split, and by 2022-2023, IDC projects service providers will own 70 percent of the large-data-center space.
For the past five years, the amount of new data-center construction by enterprises has been falling steadily at  5 to 10 percent per year, said Villars. “They are not building new ones because they are coming to the realization that being an expert at data-center construction is not something a company has.”
Enterprises across many sectors are looking at their data-center environment and leveraging things like virtual machines and SSD, thereby compressing the size of their data centers and getting more work done within smaller physical footprints. “So at some point they ask if they are spending appropriately for this space. That’s when they look at colo,” said Villars.
Join the Network World communities on [Facebook][4] and [LinkedIn][5] to comment on topics that are top of mind.
--------------------------------------------------------------------------------
via: https://www.networkworld.com/article/3407756/colocation-facilities-buck-the-cloud-data-center-trend.html
作者：[Andy Patrizio][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://www.networkworld.com/author/Andy-Patrizio/
[b]: https://github.com/lujun9972
[1]: https://images.idgesg.net/images/article/2019/05/cso_cloud_computing_backups_it_engineer_data_center_server_racks_connections_by_gorodenkoff_gettyimages-943065400_3x2_2400x1600-100796535-large.jpg
[2]: https://www.networkworld.com/article/3223692/what-is-a-data-centerhow-its-changed-and-what-you-need-to-know.html
[3]: https://www.networkworld.com/article/3391380/does-your-cloud-access-security-broker-support-ipv6-it-should.html
[4]: https://www.facebook.com/NetworkWorld/
[5]: https://www.linkedin.com/company/network-world

2
sources/tech/20190705 Manage your shell environment.md
View File

@@ -1,5 +1,5 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: translator: (geekpi)
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )

82
sources/tech/20190708 Linux Games Get A Performance Boost for AMD GPUs Thanks to Valve-s New Compiler.md
View File

@@ -1,82 +0,0 @@
[#]: collector: (lujun9972)
[#]: translator: (geekpi)
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Linux Games Get A Performance Boost for AMD GPUs Thanks to Valve’s New Compiler)
[#]: via: (https://itsfoss.com/linux-games-performance-boost-amd-gpu/)
[#]: author: (Ankush Das https://itsfoss.com/author/ankush/)
Linux Games Get A Performance Boost for AMD GPUs Thanks to Valve’s New Compiler
======
It has been a few days since Steam asked for the public feedback in order to test ACO (a new Mesa [shader][1] compiler) for AMD GPUs.
Currently, the AMD drivers use a shader compiler utilizing LLVM. However, [Mesa][2] is an open source alternative to [LLVM][3].
So, in this case, Valve wants to support AMD graphics to improve the performance of Linux games on various Linux distros.
![][4]
For gaming performance improvement, the compile time is critical and with the new ACO compiler, it reduces the time by almost 50%. Valve explained more about it in its [Steam community][5] post:
> The AMD OpenGL and Vulkan drivers currently use a shader compiler that is part of the upstream LLVM project. That project is massive, and has many different goals, with online compilation of game shaders only being one of them. That can result in development tradeoffs, where improving gaming-specific functionality is harder than it otherwise would, or where gaming-specific features would often accidentally get broken by LLVM developers working on other things. In particular, shader compilation speed is one such example: it’s not really a critical factor in most other scenarios, just a nice-to-have. But for gaming, compile time is critical, and slow shader compilation can result in near-unplayable stutter.
### Is there really a performance boost for Linux games?
Yes, there is.
The primary highlight here is the compile time. If the shader compilation time reduces dramatically, it should theoretically improve the performance of the game.
And, as per the [initial benchmark reports][6], we really do see some significant improvements.
![][7]
Of course, the in-game FPS improvement isn’t huge. But, it is still a good progress in its early stage.
If you’re curious about the compile time improvement, then here it is:
![][8]
Yes, even a big compile time reduction did not affect the in-game FPS by a large margin. But, it is still a big deal because currently, it is a work in progress. So, we can expect even more.
[][9]
Suggested read  Chrome OS Look-Alike Linux Distro Chromixium Released
But, what more can be done?
Well, the ACO compiler isn’t complete yet. Here’s why (as Valve mentioned):
> Right now, ACO only handles pixel and compute shader stages. When the rest of the stages are implemented, we expect the compile times will be reduced even further.
#### Wrapping Up
Even though I don’t have an AMD GPU on board, it is actually interesting to see improvements for the Linux gaming scene in general.
Also, we shall be expecting more benchmarks and reports as this progresses.
What do you think? Let us know your thoughts in the comments below. If you have a benchmark report to share, do let us know about it.
--------------------------------------------------------------------------------
via: https://itsfoss.com/linux-games-performance-boost-amd-gpu/
作者：[Ankush Das][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://itsfoss.com/author/ankush/
[b]: https://github.com/lujun9972
[1]: https://en.wikipedia.org/wiki/Shader
[2]: https://en.wikipedia.org/wiki/Mesa_(computer_graphics)
[3]: https://en.wikipedia.org/wiki/LLVM
[4]: https://i1.wp.com/itsfoss.com/wp-content/uploads/2019/07/Improved-Linux-Gaming.png?resize=800%2C450&ssl=1
[5]: https://steamcommunity.com/games/221410/announcements/detail/1602634609636894200
[6]: https://gist.github.com/pendingchaos/aba1e4c238cf039d17089f29a8c6aa63
[7]: https://i0.wp.com/itsfoss.com/wp-content/uploads/2019/07/fps-improvement-amd.png?fit=800%2C412&ssl=1
[8]: https://i1.wp.com/itsfoss.com/wp-content/uploads/2019/07/compile-time-amd-gpu-linux.png?ssl=1
[9]: https://itsfoss.com/chromixiumos-released/

56
sources/tech/20190709 From BASIC to Ruby- Life lessons from first programming languages on Command Line Heroes.md Normal file
View File

@@ -0,0 +1,56 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (From BASIC to Ruby: Life lessons from first programming languages on Command Line Heroes)
[#]: via: (https://opensource.com/19/7/command-line-heroes-ruby-basic)
[#]: author: (Matthew Broberg https://opensource.com/users/mbbroberg)
From BASIC to Ruby: Life lessons from first programming languages on Command Line Heroes
======
Find out more about why BASIC is a beloved first language and how the
next generation will learn to code.
![Listen to the Command Line Heroes Podcast][1]
The second episode of this [Command Line Heroes][2] season 3 drops today and it sent me back through a nostalgic look at the idea of first programming languages.
### Languages affect accessibility
This episode taught me that BASIC was a huge leap in the democratization of computer comprehension. It's hard for me to imagine a time when computers were scarce, but that not-so-distant past was when BASIC changed the world. As [Saron Yitbarek][3] mentions, "In the early days of programming, you pretty much needed a Ph.D. to do anything." BASIC was such a monumental leap with its focus on usability (beginner-friendly commands) and resource sharing (timesharing of a single computer). It helped programming get beyond the "computer jocks" of the time (I love that phrase from the episode) and helped a new generation of people participate. The barrier of entry dropped.
### First programming languages
The heart of this episode rests on the topic of learning the first language. There is so much advice out there about what to learn and how to learn it. Quite a lot has been written on the subject [on here][4]. I love hearing Saron's story of Ruby being her introduction, and how it was fun in an almost unexpected way. I had a similar experience as I dug into Ruby for a few projects. It's wildly flexible in a way that makes me happy. It's that happiness that keeps me coming back to it when I'm in a pinch, and there's something powerful about how languages can be so emotionally charged.
I first experienced programming with HTML and CSS, but the first heavy-duty language was Java. I will never forget being told on day one of class to memorize **public static void main** without any context on what it meant. We took a good bit of that semester to explore what it in the context of object-oriented programming, but it never made me feel as excited as when I iterate over a list using **.each** in Ruby or **import numpy** and do some mathematical magic in Python. Then I hear about how kids are learning to program with Python for [Minecraft][5] or visual programming languages like [Scratch][6] and I am inspired. The legacy of BASIC lives on in new ways. 
Which leads to my takeaways from this episode: 
* Remember that no one is born a programmer. Everyone starts with no background. You're not alone there.
* Learn a language. Any of them. Choose the one that brings you the most joy if you have the luxury of choosing.
* Don't forget that all languages are there to build something. Create meaningful things for humans.
Command Line Heroes will cover programming languages for all of season 3. [Subscribe here to learn everything you want to know about the origin of programming languages][2], and I would love to hear your thoughts in the comments below.
--------------------------------------------------------------------------------
via: https://opensource.com/19/7/command-line-heroes-ruby-basic
作者：[Matthew Broberg][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://opensource.com/users/mbbroberg
[b]: https://github.com/lujun9972
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/lead-images/ep1_blog-header-520x292_lgr.png?itok=I8IS1hkt (Listen to the Command Line Heroes Podcast)
[2]: https://www.redhat.com/en/command-line-heroes
[3]: https://twitter.com/saronyitbarek
[4]: /article/17/1/choosing-your-first-programming-language
[5]: /life/15/5/getting-started-minecraft-pi
[6]: /education/11/6/how-teach-next-generation-open-source-scratch

185
sources/tech/20190709 How to install Elasticsearch and Kibana on Linux.md Normal file
View File

@@ -0,0 +1,185 @@
[#]: collector: (lujun9972)
[#]: translator: (runningwater)
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (How to install Elasticsearch and Kibana on Linux)
[#]: via: (https://opensource.com/article/19/7/install-elasticsearch-and-kibana-linux)
[#]: author: (Seth Kenlon https://opensource.com/users/seth)
How to install Elasticsearch and Kibana on Linux
======
Get our simplified instructions for installing both.
![5 pengiuns floating on iceburg][1]
If you're keen to learn Elasticsearch, the famous open source search engine based on the open source Lucene library, then there's no better way than to install it locally. The process is outlined in detail on the [Elasticsearch website][2], but the official instructions have a lot more detail than necessary if you're a beginner. This article takes a simplified approach.
### Add the Elasticsearch repository
First, add the Elasticsearch software repository to your system, so you can install it and receive updates as needed. How you do so depends on your distribution. On an RPM-based system, such as [Fedora][3], [CentOS][4], [Red Hat Enterprise Linux (RHEL)][5], or [openSUSE][6], (anywhere in this article that references Fedora or RHEL applies to CentOS and openSUSE as well) create a repository description file in **/etc/yum.repos.d/** called **elasticsearch.repo**:
```
$ cat &lt;&lt; EOF | sudo tee /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-7.x]
name=Elasticsearch repository for 7.x packages
baseurl=<https://artifacts.elastic.co/packages/oss-7.x/yum>
gpgcheck=1
gpgkey=<https://artifacts.elastic.co/GPG-KEY-elasticsearch>
enabled=1
autorefresh=1
type=rpm-md
EOF
```
On Ubuntu or Debian, do not use the **add-apt-repository** utility. It causes errors due to a mismatch in its defaults and what Elasticsearch’s repository provides. Instead, set up this one:
```
$ echo "deb <https://artifacts.elastic.co/packages/oss-7.x/apt> stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
```
This repository contains only Elasticsearch’s open source features, under an [Apache License][7], with none of the extra features provided by a subscription. If you need subscription-only features (these features are _not_ open source), the **baseurl** must be set to:
```
`baseurl=https://artifacts.elastic.co/packages/7.x/yum`
```
 
### Install Elasticsearch
The name of the package you need to install depends on whether you use the open source version or the subscription version. This article uses the open source version, which appends **-oss** to the end of the package name. Without **-oss** appended to the package name, you are requesting the subscription-only version.
If you create a repository pointing to the subscription version but try to install the open source version, you will get a fairly non-specific error in return. If you create a repository for the open source version and fail to append **-oss** to the package name, you will also get an error.
Install Elasticsearch with your package manager. For instance, on Fedora, CentOS, or RHEL, run the following:
```
$ sudo dnf install elasticsearch-oss
```
On Ubuntu or Debian, run:
```
$ sudo apt install elasticsearch-oss
```
If you get errors while installing Elasticsearch, then you may be attempting to install the wrong package. If your intention is to use the open source package, as this article does, then make sure you are using the correct **apt** repository or baseurl in your Yum configuration.
### Start and enable Elasticsearch
Once Elasticsearch has been installed, you must start and enable it:
```
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now elasticsearch.service
```
Then, to confirm that Elasticsearch is running on its default port of 9200, point a web browser to **localhost:9200**. You can use a GUI browser or you can do it in the terminal:
```
$ curl localhost:9200
{
  "name" : "fedora30",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "OqSbb16NQB2M0ysynnX1hA",
  "version" : {
    "number" : "7.2.0",
    "build_flavor" : "oss",
    "build_type" : "rpm",
    "build_hash" : "508c38a",
    "build_date" : "2019-06-20T15:54:18.811730Z",
    "build_snapshot" : false,
    "lucene_version" : "8.0.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
```
### Install Kibana
Kibana is a graphical interface for Elasticsearch data visualization. It’s included in the Elasticsearch repository, so you can install it with your package manager. Just as with Elasticsearch itself, you must append **-oss** to the end of the package name if you are using the open source version of Elasticsearch, and not the subscription version (the two installations need to match):
```
$ sudo dnf install kibana-oss
```
On Ubuntu or Debian:
```
$ sudo apt install kibana-oss
```
Kibana runs on port 5601, so launch a graphical web browser and navigate to **localhost:5601** to start using the Kibana interface, which is shown below:
![Kibana running in Firefox.][8]
### Troubleshoot
If you get errors while installing Elasticsearch, try installing a Java environment manually. On Fedora, CentOS, and RHEL:
```
$ sudo dnf install java-openjdk-devel java-openjdk
```
On Ubuntu:
```
`$ sudo apt install default-jdk`
```
If all else fails, try installing the Elasticsearch RPM directly from the Elasticsearch servers:
```
$ wget <https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-7.2.0-x86\_64.rpm{,.sha512}>
$ shasum -a 512 -c elasticsearch-oss-7.2.0-x86_64.rpm.sha512 &amp;&amp; sudo rpm --install elasticsearch-oss-7.2.0-x86_64.rpm
```
On Ubuntu or Debian, use the DEB package instead.
If you cannot access either Elasticsearch or Kibana with a web browser, then your firewall may be blocking those ports. You can allow traffic on those ports by adjusting your firewall settings. For instance, if you are running **firewalld** (the default on Fedora and RHEL, and installable on Debian and Ubuntu), then you can use **firewall-cmd**:
```
$ sudo firewall-cmd --add-port=9200/tcp --permanent
$ sudo firewall-cmd --add-port=5601/tcp --permanent
$ sudo firewall-cmd --reload
```
You’re now set up and can follow along with our upcoming installation articles for Elasticsearch and Kibana.
--------------------------------------------------------------------------------
via: https://opensource.com/article/19/7/install-elasticsearch-and-kibana-linux
作者：[Seth Kenlon][a]
选题：[lujun9972][b]
译者：[runningwater](https://github.com/runningwater)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://opensource.com/users/seth
[b]: https://github.com/lujun9972
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/lead-images/rh_003499_01_linux31x_cc.png?itok=Pvim4U-B (5 pengiuns floating on iceburg)
[2]: https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html
[3]: https://getfedora.org
[4]: https://www.centos.org
[5]: https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux
[6]: https://www.opensuse.org
[7]: http://www.apache.org/licenses/
[8]: https://opensource.com/sites/default/files/uploads/kibana.jpg (Kibana running in Firefox.)

81
sources/tech/20190709 Open education- There isn-t an app for that.md Normal file
View File

@@ -0,0 +1,81 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Open education: There isn't an app for that)
[#]: via: (https://opensource.com/open-organization/19/7/open-ed-app)
[#]: author: (Charlie Reisinger https://opensource.com/users/charlie/users/mchua)
Open education: There isn't an app for that
======
Building classrooms that function as open organizations requires more
than software.
![Classroom chairs][1]
In 2010, I confronted a problem common to all public school leaders: _How do we optimize our limited funding to bring powerful learning technology to thousands of students?_ Faced with an end-of-life fleet of student laptops, district-wide budget cuts, and teachers pleading for more technology, I made a small bet that open source software could be an affordable path forward. Fast forward to 2019: What started with a few elementary school laptop carts running a Linux operating system and open source applications grew into an award-winning, district-wide, one-to-one laptop learning program and student technology help desk—all built with open source principles and software.
Open source software has saved my district—Penn Manor School District in Lancaster County, Pennsylvania—more than a million dollars on its technology budget. But more importantly, making a deliberate and concerted effort to infuse open principles and practices into our learning environments has cultivated a vibrant and inclusive learning community that cuts across the school. And as a result, student success has exceeded our expectations.
But how do schools put open ideas into practice to foster future innovators and leaders? It's not as simple as installing Linux on 4,000 student laptops, holding hands, and singing the alma mater in the high school cafeteria.
An open schoolhouse values all learners' unique strengths and passions to help them reach their potential. This work does not begin and end with curricula, worksheets, and test scores. It starts with building connections, relationships, and trust with students. In this article, I'll explain how we put these ideas into practice.
### Building the open schoolhouse
First, school leaders must recognize that traditional school board computer policies and device decisions are retrograde to learning. Tablets do little to help students explore an operating system. Worse, repressive school device management policies lock access to the command line and block students from installing applications. Sealed tablets and locked-down laptops are like kryptonite for classrooms—they weaken critical thinking and crush a student's ability to create, explore, and learn.
Penn Manor designed district technology policies to amplify student curiosity and learning freedom. Each student has root access on their school-issued Linux laptop. Students are trusted—and encouraged—to tinker and experiment with their school laptops. And our students haven't let us down. Five years into our program, we've experienced zero discipline issues resulting from students' being trusted with admin rights.
But access to the terminal isn't enough to turn a school into an open organization. We must elevate student privileges, write a new script, and empower students to be equal partners in their education. What if our classrooms pushed aside lecture and standard curriculum and reorganized as a community of practitioners working toward a common goal?
When Penn Manor High School launched the Linux laptop learning initiative, our team designed apprenticeship opportunities for students to provide technology support to their peers. What better way to help budding technologists learn the craft than through authentic practice? What better way to encourage a culture of collaboration?
Access to the terminal isn't enough to turn a school into an open organization. We must elevate student privileges, write a new script, and empower students to be equal partners in their education. What if our classrooms pushed aside lecture and standard curriculum and reorganized as a community of practitioners working toward a common goal?
Penn Manor School District's Student Help Desk program is an honors-level, independent study apprenticeship course. Students report for the class like they would for math, science, or art. But any similarity to a traditional high school courses ends there. Apprentices work alongside district IT staff on hardware repairs, software tutorials, system imaging, peer training, and any number of tasks related to the one-to-one program. Daily work assignments are guided by the shifting needs of fellow students and classroom teachers. Visitors observe help desk apprentices fielding questions from students or staff, replacing a damaged laptop screen, or diving into Linux configuration files. Past student apprentices even wrote code for laptop imaging and device inventory. Motivated by authentic use cases, the young programmers developed the very software their peers use today.
The student help desk has no curriculum and no textbook; students search the Internet to discover solutions to problems, or borrow code and ideas that open source communities have freely shared by. Students learn and experiment with the same open source software and techniques that industry professionals use. And assessment? How can a pop-quiz measure a student's elation when their logic board repair is successful, or the joy they feel when the entire school starts using software they've designed?
In this participatory and inclusive classroom culture, traditional power structures dissolve and students are empowered to act, contribute, iterate—and solve real problems.
### Teaching in the open schoolhouse
We educators in the open schoolhouse don't lecture and test. We clear obstacles, provide prompts, and create a culture where trial and risk receive encouragement and praise. Together, as a team, students and staff shape the world around them. When we honor learning by doing, students become active agents in their education and they contribute to the school community in innovative new ways.
Beyond the obvious career preparation and technical skill-building experiences, the Penn Manor Student Help Desk provides students an opportunity to explore individual passions via independent study. As part of this program, every student is challenged to create a unique and compelling personal project that breaks new ground. Choice of the project is entirely up to each student. In the past, they've explored software defined radio, built virtual reality tours, developed a software system for Roll20, produced a podcast series, and even programmed a TurtleBot robot to self-navigate across the school's hallways.
One of our goals is to help technology apprentices discover they can build and command technology—not be content with someone else's technical or marketing decisions. Hacking isn't a concept or skill we teach. It is an ethos we embrace.
A few years ago, I was struck by the wisdom of student one help desk apprentice and hacker, Aytekin Oldac. I asked for his thoughts about the program. The pensive young man paused for the briefest moment and said, "There is a quote from Aristotle, 'Men become builders by building.' I think that applies to the help desk."
Aytekin was indeed becoming a builder. The student help desk needed a visitor registration system, so Aytekin built one. He set to work on a check-in system built atop a decommissioned point-of-sale terminal. The once-obsolete cafeteria terminal was a laughable gray box of thick industrial plastic topped with a fry-grease resistant touchscreen. But Aytekin gave it a new life beyond the lunch line.
One of our goals is to help technology apprentices discover they can build and command technology—not be content with someone else's technical or marketing decisions. Hacking isn't a concept or skill we teach. It is an ethos we embrace.
Using LibreOffice, he programmed a data-entry form with a large on-screen number pad. When a visiting student entered their student ID, the on-screen form would add a timestamp and log the visit into our database. With no explicit curriculum, he relied on his Linux laptop, the Internet, and his intellect to build a new contraption for the help desk team.
Of course, we could have rushed to an app store for a proprietary registration application for a tablet. But what would our young builder have learned? If there were an "app for that," Aytekin would never have spent weeks prototyping a solution, parsing sample code, debugging, or iterating designs from peer feedback. Empowered with freedom and trust, Aytekin became lost in the flow of discovery, hacking, and problem-solving.
But there's a deeper spirit in the open schoolhouse. The thoughts of former student apprentice, Susan Black, transcend education and hardware. "I cannot imagine a more perfect day than one spent repairing laptops and solving software issues at the help desk. I think of our help desk room not as a class, but as a family. We motivate and teach each other, but we also have a few good laughs. We make memories daily, and I don't have to hide who I am in this class. Nobody dares to judge one another, and we become closer by our differences."
Susan's voice resonates a sense of place, a safe and inviting space untangled from the curriculum assembly line and insulated from high school angst and drama. In this place, she is free to be herself and empowered to learn and create. Shouldn't all students be afforded the same opportunity to build self-esteem and leadership skills? To follow their passions? To find their tribes? When the classroom hierarchy is flattened, when students are exalted, when the roles of student and teacher are blurred, the open schoolhouse emerges.
### Read this next
One of the hardest things about trying to bridge two worlds--for instance, open source communities...
--------------------------------------------------------------------------------
via: https://opensource.com/open-organization/19/7/open-ed-app
作者：[Charlie Reisinger][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://opensource.com/users/charlie/users/mchua
[b]: https://github.com/lujun9972
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/lead-images/edu_rcos.png?itok=ylXuvWqA (Classroom chairs)

61
sources/tech/20190709 Sysadmin vs SRE- What-s the difference.md Normal file
View File

@@ -0,0 +1,61 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Sysadmin vs SRE: What's the difference?)
[#]: via: (https://opensource.com/article/19/7/sysadmins-vs-sres)
[#]: author: (Vince Power https://opensource.com/users/vincepower/users/craig5/users/dawnparzych/users/penglish)
Sysadmin vs SRE: What's the difference?
======
Both sysadmins and site reliability engineers are valuable parts of any
organization. Here's a look at each role differs.
![People work on a computer server with devices][1]
In the IT world, there has always been a pull between generalist and specialist. The stereotypical sysadmin falls in the generalist category 99 times out of 100. The [site reliability engineer (SRE)][2] role is specialized, however, and grew out of the needs of one of the first companies to know real scale: Google. Ultimately, these two roles have the same goal for the applications whose infrastructure they operate: providing a good experience for the applications’ consumers. Yet, these roles have drastically different starting points.
### Sysadmins: Neutral good incarnate
Sysadmins typically grow into their position by starting as entry-level desktop and network support, and then over time acquiring the broad set of skills most sysadmins have in common. By that point, these sysadmins know all of the systems and applications they are responsible for. They know the app on server one needs to be restarted every other Tuesday, or the service on server nine will crash on Wednesday with no errors. They have fine-tuned their monitoring so it ignores what doesn’t matter, even that error that happens on the third Sunday of the month, despite the fact that it’s marked as fatal.
In short, sysadmins know how to feed and care for the servers that run the core of your business. These sysadmins have grown to use automation to handle routine tasks across all the servers they manage. They love templates, golden images, and standards, but are flexible enough to make a parameter change on just the one server that has an error, and then make a note regarding why that server is now uniquely configured.
Sysadmins are great, but they have a couple of quirks. The first being that you just do not get root access without divine intervention, and that any changes they make which were not their idea have to be documented as required by the application they are working with’s vendor, and then will still be double-checked.
The servers are their domain, and no one messes with their stuff.
### SREs: Thanos would be proud
As opposed to the path to becoming a sysadmin, SREs are as likely to come from a development background as a sysadmin background. The SRE position is closer to the lifecycle you find in an application development environment.
As an organization grows and introduces [DevOps][3] concepts like [continuous integration][4] and [continuous delivery][5] (CI/CD), there will often be a skills gap on how to run those immutable applications across multiple environments while having them scale to meet the business’s needs. This is the world of an SRE. Yes, a sysadmin can learn the additional tools, but at scale, this easily becomes a full-time position to keep up. A specialist makes sense.
SREs use concepts like [infrastructure-as-code][6] to produce templates, which are called to deploy the environment an application will run in, with the goal of every application and its environment being completely reproducible with the push of a button. So, app one on server one in system testing will have the exact same binaries that will be used on server fifteen in production, with the exception of environment-specific variables like passwords and database connection strings.
An SRE will also completely destroy an environment and rebuild it based on a configuration change. There is no emotional attachment to any system. Each system is just a number and is tagged and lifecycled accordingly, even to the point that routine server patching is done by redeploying the entire application stack.
### Conclusion
In certain situations, especially when operating in large DevOps-based environments, the specialized skills an SRE provides regarding how to handle any level of scale definitely offer an advantage. And every time they get stuck, they will seek out the help of their friendly neighborhood sysadmin—or [(BOFH)][7] on a bad day—for those well-honed troubleshooting skills, and the breadth of experiences which sysadmins rely on to provide value to any organization they are a part of.
--------------------------------------------------------------------------------
via: https://opensource.com/article/19/7/sysadmins-vs-sres
作者：[Vince Power][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://opensource.com/users/vincepower/users/craig5/users/dawnparzych/users/penglish
[b]: https://github.com/lujun9972
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/lead-images/rh_003499_01_linux11x_cc.png?itok=XMDOouJR (People work on a computer server with devices)
[2]: https://en.wikipedia.org/wiki/Site_Reliability_Engineering
[3]: https://opensource.com/resources/devops
[4]: https://en.wikipedia.org/wiki/Continuous_integration
[5]: https://en.wikipedia.org/wiki/Continuous_delivery
[6]: https://en.wikipedia.org/wiki/Infrastructure_as_code
[7]: http://www.bofharchive.com/BOFH.html

78
translated/tech/20190708 Linux Games Get A Performance Boost for AMD GPUs Thanks to Valve-s New Compiler.md Normal file
View File

@@ -0,0 +1,78 @@
[#]: collector: (lujun9972)
[#]: translator: (geekpi)
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Linux Games Get A Performance Boost for AMD GPUs Thanks to Valve’s New Compiler)
[#]: via: (https://itsfoss.com/linux-games-performance-boost-amd-gpu/)
[#]: author: (Ankush Das https://itsfoss.com/author/ankush/)
借助 Valve 的新编译器，Linux 游戏在 AMD GPU 中获得了性能提升
======
Steam 寻求公众反馈以便为 AMD GPU 测试 ACO（一个新的 Mesa [着色器][1]编译器）已经有几天了。
目前，AMD 驱动使用 LLVM 作为着色器编译器。然而，[Mesa][2] 是 [LLVM][3] 的开源替代品。
因此，在这种情况下，Valve 希望支持 AMD 显卡以提高 Linux 游戏在各种 Linux 发行版上的性能。
![][4]
为了提高游戏性能，编译时间至关重要，使用新的 ACO 编译器，它将时间缩短了近 50％。 Valve 在其 [Steam 社区][5]的帖子中解释了更多关于它的信息：
> AMD OpenGL 和 Vulkan 驱动目前使用的着色器编译器是上游 LLVM 项目的一部分。该项目规模庞大，并且有许多不同的目标，在线编辑游戏着色器只是其中之一。这可能会导致开发权衡，其中改进游戏特定功能比其他情况更难，特定于游戏的功能也经常被 LLVM 的开发人员因其他事情破坏。特别是，着色器编译速度就是这样一个例子：它在大多数其他场景中并不是一个关键因素，只能锦上添花。但是对于游戏来说，编译时间是至关重要的，而缓慢的着色器编译可能导致几乎无法播放的口吃。
### Linux 游戏真的有性能提升吗？
是的，没错。
这里的主要亮点是编译时间。如果着色器编译时间急剧减少，理论上应该会改善游戏的性能。
而且，根据[最初的基准报告][6]，我们确实看到了一些重大改进。
![][7]
当然，游戏中的 FPS 改进并不是很大。但是，它在早期阶段仍然是一个很好的进步。
如果你对编译时间的改进感到好奇，下面是结果：
![][8]
是的，即使大幅的编译时间减少也不会大幅影响游戏中的FPS。但是，它仍然是一件大事，因为目前，这是一项正在进行中的工作。所以，我们可以有更多期待。
但是，还能做些什么呢？
好吧，ACO 还没完成。下面是为什么（在 Valve 中提到）：
> 现在，ACO 只处理像素和计算着色器阶段。当其余的阶段实现时，我们预计编译时间将进一步减少。
#### 总结
尽管我没有配备 AMD GPU，但我很有兴趣看到对 Linux 游戏场景的改进。
此外，随着事情进展，我们将期待更多的基准和报告。
你怎么看待？请在下面的评论中告诉我们你的想法。如果你有基本报告要分享，请告诉我们。
--------------------------------------------------------------------------------
via: https://itsfoss.com/linux-games-performance-boost-amd-gpu/
作者：[Ankush Das][a]
选题：[lujun9972][b]
译者：[geekpi](https://github.com/geekpi)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://itsfoss.com/author/ankush/
[b]: https://github.com/lujun9972
[1]: https://en.wikipedia.org/wiki/Shader
[2]: https://en.wikipedia.org/wiki/Mesa_(computer_graphics)
[3]: https://en.wikipedia.org/wiki/LLVM
[4]: https://i1.wp.com/itsfoss.com/wp-content/uploads/2019/07/Improved-Linux-Gaming.png?resize=800%2C450&ssl=1
[5]: https://steamcommunity.com/games/221410/announcements/detail/1602634609636894200
[6]: https://gist.github.com/pendingchaos/aba1e4c238cf039d17089f29a8c6aa63
[7]: https://i0.wp.com/itsfoss.com/wp-content/uploads/2019/07/fps-improvement-amd.png?fit=800%2C412&ssl=1
[8]: https://i1.wp.com/itsfoss.com/wp-content/uploads/2019/07/compile-time-amd-gpu-linux.png?ssl=1