From 0abe234e4f6beb2a26bb1b01b9920b4981aa5774 Mon Sep 17 00:00:00 2001 From: DarkSun Date: Tue, 24 Nov 2020 18:42:23 +0800 Subject: [PATCH 1/3] =?UTF-8?q?=E9=80=89=E9=A2=98[tech]:=2020201115=20How?= =?UTF-8?q?=20do=20you=20write=20simple=20explanations=20without=20soundin?= =?UTF-8?q?g=20condescending=3F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit sources/tech/20201115 How do you write simple explanations without sounding condescending.md --- ...anations without sounding condescending.md | 105 ++++++++++++++++++ 1 file changed, 105 insertions(+) create mode 100644 sources/tech/20201115 How do you write simple explanations without sounding condescending.md diff --git a/sources/tech/20201115 How do you write simple explanations without sounding condescending.md b/sources/tech/20201115 How do you write simple explanations without sounding condescending.md new file mode 100644 index 0000000000..108440cabd --- /dev/null +++ b/sources/tech/20201115 How do you write simple explanations without sounding condescending.md @@ -0,0 +1,105 @@ +[#]: collector: (lujun9972) +[#]: translator: ( ) +[#]: reviewer: ( ) +[#]: publisher: ( ) +[#]: url: ( ) +[#]: subject: (How do you write simple explanations without sounding condescending?) +[#]: via: (https://jvns.ca/blog/2020/11/15/simple-explanations-without-sounding-condescending/) +[#]: author: (Julia Evans https://jvns.ca/) + +How do you write simple explanations without sounding condescending? +====== + +Sumana Harihareswara wrote an interesting blog post [Plain Language Choices][1] recently, about writing about complicated topics using simple language and how it can sometimes come off as condescending. + +I really like explaining complicated topics while trying to avoid unnecessary jargon, and I realized that I’ve thought a lot about how to do it well. So here are a bunch of things I try to do when I use simple language to avoid coming off as condescending. + +### use some jargon to give the reader search terms + +Sometimes I see writing that completely avoids all jargon and instead substitutes simple language for all “jargon”-y words. + +I like to include some jargon in my explanations because otherwise it’s impossible for the reader to search & learn more about the concept they’re trying to learn about. + +### write (mostly) true explanations + +Something else I see sometimes in ELI5-type explanations is an explanation in plain language that’s not actually true in a useful way. I’m pretty sympathetic to why people do this – it’s super hard to write simple explanations that are also true! + +And actually sometimes when I’m trying to write down a simple/clear explanation for a concept, I realize that I don’t actually understand the concept as well as I thought and that I’m not able to explain it. That’s okay! + +I think there are a few options here: + + * try to say only things that are true (or at least which are a useful model for how the world works even if they’re not 100% true) + * write things that are not really true / that you’re not sure of, but point out that they may not be true (“I think it works like X, but I realize now that it might be Y instead, I’m not sure!“) + + + +### only use “fun” visual elements on explanations that are actually well written & easy to understand + +This happens more with visual aids than with simple language but I’ll include it anyway. Sometimes I see explanations which have “fun” elements to it to make it them seem more approachable where the explanation itself is still pretty unclear. + +I try to be careful about this in my own work – I try to only attach “fun” elements (like a fun illustrated cover) to explanations that I’ve spent a lot of time on making really clear. Basically to me “fun” things are a signal that the content itself is really clear/accessible, and I try to not misuse that signal. + +I think [why’s poignant guide to ruby][2] is a nice example of something that’s fun and clear and which has helped a lot of people learn Ruby. + +Another nice example of this is: I know someone who got her master’s thesis printed as a paperback book and illustrated with some great drawings related to the topic of her thesis (trans represensentation in media). It’s called “I’m supposed to relate to this?”, here’s the [paperback][3]. + +I ended up reading the whole thing because, in addition to having the fun illustrations, her master’s thesis was really well written and interesting! The fact that she did the work to print a paperback book of her thesis and get it illustrated was a sign that she’d worked on making the writing accessible to a non-academic audience, and it was true! + +### tell a relevant story + +Stories can really help people learn! For example, something I’ve done a lot on this blog is talk about a problem I ran into in the course of my job and what I did to solve that problem. + +Some kinds of stories that I think work well: + + * a real problem that someone ran into, to motivate why the concept is interesting / important to learn + * something that’s happening on a computer, framed as a “story” (for example tells a story about how DNS works. Everything in the story literally corresponds to exactly what happens when you make a DNS query) + + + +Sometimes I see stories used to explain concepts that don’t fit into either of these and feel kind of pasted on, like they’re there to help the concept seem “fun” but don’t actually illustrate the concept or motivate why it might be useful to learn it. + +### have a specific audience in mind + +I try to write relatively simple explanations, but when I write I also generally assume a lot of knowledge on the part of my audience. + +Sometimes I see explanations of complicated concepts that start with explaining the very basics of the topic. This usually isn’t that effective: if someone is trying to understand some super technical aspect of containers, they probably understand the basics of containers already! + +“Have an audience” is more of a general writing tip so I’ll leave it at that. + +### on using simple language as a joke for people who already understand the idea + +Here’s a very fun explanation of a complicated thing using simple language: [Gödel’s Second Incompleteness Theorem Explained in Words of One Syllable][4]. + +On one hand, this is fun! I enjoyed reading it. On the other hand, I think the main audience for this is probably people who already more or less understand Gödel’s Second Incompleteness Theorem. + +For example, someone pointed out that “if math is a not a load of bunk” in this text is code for “Peano arithmetic is consistent” with (“math” being “Peano arithmetic” and “not a load of bunk” meaning “consistent”). Which I find very charming, but also I found it a little hard to decode when reading it. + +And (as we talked about before about jargon), if you know that “Peano arithmetic is consistent” is the relevant bit of jargon, you can find all kind of fascinating things, like [a blog post by John Baez from 2011 discussing an attempted proof that Peano arithmetic was inconsistent][5]) + +(I’m also reminded here of the XKCD [up goer five][6], which is very delightful, but I don’t think I learned anything about spaceships from reading it) + +### that’s all! + +I’d love to hear more thoughts on this – I think there are probably more ways that simple explanations can feel condescending that I’ve missed! + +I really don’t think they _need_ to feel condescending though – to me the point of writing a clear/simple explanation is usually that I think the idea is _not actually fundamentally that complicated_ and so I’m just explaining it in a way that’s exactly as complicated as it needs to be. + +-------------------------------------------------------------------------------- + +via: https://jvns.ca/blog/2020/11/15/simple-explanations-without-sounding-condescending/ + +作者：[Julia Evans][a] +选题：[lujun9972][b] +译者：[译者ID](https://github.com/译者ID) +校对：[校对者ID](https://github.com/校对者ID) + +本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出 + +[a]: https://jvns.ca/ +[b]: https://github.com/lujun9972 +[1]: https://www.harihareswara.net/sumana/2020/11/12/0 +[2]: https://poignant.guide/ +[3]: https://untrans.bigcartel.com/ +[4]: https://www2.kenyon.edu/Depts/Math/Milnikel/boolos-godel.pdf +[5]: https://golem.ph.utexas.edu/category/2011/09/the_inconsistency_of_arithmeti.html +[6]: https://xkcd.com/1133/ From 18345d1d8925467ab0dcbd468d50f079c9c4579d Mon Sep 17 00:00:00 2001 From: DarkSun Date: Tue, 24 Nov 2020 18:42:43 +0800 Subject: [PATCH 2/3] add done: 20201115 How do you write simple explanations without sounding condescending.md --- ...eate a machine learning model with Bash.md | 638 ++++++++++++++++++ 1 file changed, 638 insertions(+) create mode 100644 sources/tech/20201124 Create a machine learning model with Bash.md diff --git a/sources/tech/20201124 Create a machine learning model with Bash.md b/sources/tech/20201124 Create a machine learning model with Bash.md new file mode 100644 index 0000000000..f6abb3cd5c --- /dev/null +++ b/sources/tech/20201124 Create a machine learning model with Bash.md @@ -0,0 +1,638 @@ +[#]: collector: (lujun9972) +[#]: translator: ( ) +[#]: reviewer: ( ) +[#]: publisher: ( ) +[#]: url: ( ) +[#]: subject: (Create a machine learning model with Bash) +[#]: via: (https://opensource.com/article/20/11/machine-learning-bash) +[#]: author: (Girish Managoli https://opensource.com/users/gammay) + +Create a machine learning model with Bash +====== +Bash, Tcsh, or Zsh can help you get ready for machine learning. +![bash logo on green background][1] + +[Machine learning][2] is a powerful computing capability for predicting or forecasting things that conventional algorithms find challenging. The machine learning journey begins with collecting and preparing data—a _lot_ of it—then it builds mathematical models based on that data. While multiple tools can be used for these tasks, I like to use the [shell][3]. + +A shell is an interface for performing operations using a defined language. This language can be invoked interactively or scripted. The concept of the shell was introduced in [Unix][4] operating systems in the 1970s. Some of the most popular shells include [Bash][5], [tcsh][6], and [Zsh][7]. They are available for all operating systems, including Linux, macOS, and Windows, which gives them high portability. For this exercise, I'll use Bash. + +This article is an introduction to using a shell for data collection and data preparation. Whether you are a data scientist looking for efficient tools or a shell expert looking at using your skills for machine learning, I hope you will find valuable information here. + +The example problem in this article is creating a machine learning model to forecast temperatures for US states. It uses shell commands and scripts to do the following data collection and data preparation steps: + + 1. Download data + 2. Extract the necessary fields + 3. Aggregate data + 4. Make time series + 5. Create the train, test, and validate data sets + + + +You may be asking why you should do this with shell, when you can do all of it in a machine learning programming language such as [Python][8]. This is a good question. If data processing is performed with an easy, friendly, rich technology like a shell, a data scientist focuses only on machine learning modeling and not the details of a language. + +## Prerequisites + +First, you need to have a shell interpreter installed. If you use Linux or macOS, it will already be installed, and you may already be familiar with it. If you use Windows, try [MinGW][9] or [Cygwin][10]. + +For more information, see: + + * [Bash tutorials][11] here on opensource.com + * The official [shell scripting tutorial][12] by Steve Parker, the creator of Bourne shell + * The [Bash Guide for Beginners][13] by the Linux Documentation Project + * If you need help with a specific command, type ` --help` in the shell for help; for example: `ls --help`. + + + +## Get started + +Now that your shell is set up, you can start preparing data for the machine learning temperature-prediction problem. + +### 1\. Download data + +The data for this tutorial comes from the US National Oceanic and Atmospheric Administration (NOAA). You will train your model using the last 10 complete years of data. The data source is at , and the data is in .csv format and gzipped. + +Download and unzip the data using a [shell script][14]. Use your favorite text editor to create a file named `download.sh` and paste in the code below. The comments in the code explain what the commands do: + + +``` +#!/bin/sh +# This is called hashbang. It identifies the executor used to run this file. +# In this case, the script is executed by shell itself. +# If not specified, a program to execute the script must be specified. +# With hashbang: ./download.sh;  Without hashbang: sh ./download.sh; + +FROM_YEAR=2010 +TO_YEAR=2019 + +year=$FROM_YEAR +# For all years one by one starting from FROM_YEAR=2010 upto TO_YEAR=2019 +while [ $year -le $TO_YEAR ] +do +    # show the year being downloaded now +    echo $year +    # Download +    wget +    # Unzip +    gzip -d ${year}.csv.gz +    # Move to next year by incrementing +    year=$(($year+1)) +done +``` + +Notes: + + * If you are behind a proxy server, consult Mark Grennan's [how-to][15], and use: [code] export http_proxy= +export https_proxy= +``` + * Make sure all standard commands are already in your PATH (such as `/bin` or `/usr/bin`). If not, [set your PATH][16]. + * [Wget][17] is a utility for connecting to web servers from the command line. If Wget is not installed on your system, [download it][18]. + * Make sure you have [gzip][19], a utility used for compression and decompression. + + + +Run this script to download, extract, and make 10 years' worth of data available as CSVs: +``` + + +$ ./download.sh +2010 +\--2020-10-30 19:10:47--   +Resolving www1.ncdc.noaa.gov (www1.ncdc.noaa.gov)... 205.167.25.171, 205.167.25.172, 205.167.25.178, ... +Connecting to www1.ncdc.noaa.gov (www1.ncdc.noaa.gov)|205.167.25.171|:443... connected. +HTTP request sent, awaiting response... 200 OK +Length: 170466817 (163M) [application/gzip] +Saving to: '2010.csv.gz' + +     0K .......... .......... .......... .......... ..........  0% 69.4K 39m57s +    50K .......... .......... .......... .......... ..........  0%  202K 26m49s +   100K .......... .......... .......... .......... ..........  0% 1.08M 18m42s + +... + +``` +The [ls][20] command lists the contents of a folder. Use `ls 20*.csv` to list all your files with names beginning with 20 and ending with .csv. +``` + + +$ ls 20*.csv +2010.csv  2011.csv  2012.csv  2013.csv  2014.csv  2015.csv  2016.csv  2017.csv  2018.csv  2019.csv + +``` +### 2\. Extract average temperatures + +Extract the TAVG (average temperature) data from the CSVs for US regions: + +**extract_tavg_us.sh** +``` + + +#!/bin/sh + +# For each file with name that starts with "20" and ens with ".csv" +for csv_file in `ls 20*.csv` +do +    # Message that says file name $csv_file is extracted to file TAVG_US_$csv_file +    # Example: 2010.csv extracted to TAVG_US_2010.csv +    echo "$csv_file -> TAVG_US_$csv_file" +    # grep "TAVG" $csv_file: Extract lines in file with text "TAVG" +    # |: pipe +    # grep "^US": From those extract lines that begin with text "US" +    # > TAVG_US_$csv_file: Save xtracted lines to file TAVG_US_$csv_file +    grep "TAVG" $csv_file | grep "^US" > TAVG_US_$csv_file +done + +``` +This script: +``` + + +$ ./extract_tavg_us.sh +2010.csv -> TAVG_US_2010.csv +... +2019.csv -> TAVG_US_2019.csv + +``` +creates these files: +``` + + +$ ls TAVG_US*.csv +TAVG_US_2010.csv  TAVG_US_2011.csv  TAVG_US_2012.csv  TAVG_US_2013.csv +TAVG_US_2014.csv  TAVG_US_2015.csv  TAVG_US_2016.csv  TAVG_US_2017.csv +TAVG_US_2018.csv  TAVG_US_2019.csv + +``` +Here are the first few lines for `TAVG_US_2010.csv`: +``` + + +$ head TAVG_US_2010.csv +USR0000AALC,20100101,TAVG,-220,,,U, +USR0000AALP,20100101,TAVG,-9,,,U, +USR0000ABAN,20100101,TAVG,12,,,U, +USR0000ABCA,20100101,TAVG,16,,,U, +USR0000ABCK,20100101,TAVG,-309,,,U, +USR0000ABER,20100101,TAVG,-81,,,U, +USR0000ABEV,20100101,TAVG,-360,,,U, +USR0000ABEN,20100101,TAVG,-224,,,U, +USR0000ABNS,20100101,TAVG,89,,,U, +USR0000ABLA,20100101,TAVG,59,,,U, + +``` +The [head][21] command is a utility for displaying the first several lines (by default, 10 lines) of a file. + +The data has more information than you need. Limit the number of columns by eliminating column 3 (since all the data is average temperature) and column 5 onward. In other words, keep columns 1 (climate station), 2 (date), and 4 (temperature recorded). + +**key_columns.sh** +``` + + +#!/bin/sh + +# For each file with name that starts with "TAVG_US_" and ens with ".csv" +for csv_file in `ls TAVG_US_*.csv` +do +    echo "Exractiing columns $csv_file" +    # cat $csv_file: 'cat' is to con'cat'enate files - here used to show one year csv file +    # |: pipe +    # cut -d',' -f1,2,4: Cut columns 1,2,4 with , delimitor +    # > $csv_file.cut: Save to temporary file +    | > $csv_file.cut: +    cat $csv_file | cut -d',' -f1,2,4 > $csv_file.cut +    # mv $csv_file.cut $csv_file: Rename temporary file to original file +    mv $csv_file.cut $csv_file +    # File is processed and saved back into the same +    # There are other ways to do this +    # Using intermediate file is the most reliable method. +done + +``` +Run the script: +``` + + +$ ./key_columns.sh +Extracting columns TAVG_US_2010.csv +... +Extracting columns TAVG_US_2019.csv + +``` +The first few lines of `TAVG_US_2010.csv` with the unneeded data removed are: +``` + + +$ head TAVG_US_2010.csv +USR0000AALC,20100101,-220 +USR0000AALP,20100101,-9 +USR0000ABAN,20100101,12 +USR0000ABCA,20100101,16 +USR0000ABCK,20100101,-309 +USR0000ABER,20100101,-81 +USR0000ABEV,20100101,-360 +USR0000ABEN,20100101,-224 +USR0000ABNS,20100101,89 +USR0000ABLA,20100101,59 + +``` +Dates are in string form (YMD). To train your model correctly, your algorithms need to recognize date fields in the comma-separated Y,M,D form (For example, `20100101` becomes `2010,01,01`). You can convert them with the [sed][22] utility. + +**date_format.sh** +``` + + +for csv_file in `ls TAVG_*.csv` +do +    echo Date formatting $csv_file +    # This inserts , after year +    sed -i 's/,..../&,/' $csv_file +    # This inserts , after month +    sed -i 's/,....,../&,/' $csv_file +done + +``` +Run the script: +``` + + +$ ./date_format.sh +Date formatting TAVG_US_2010.csv +... +Date formatting TAVG_US_2019.csv + +``` +The first few lines of `TAVG_US_2010.csv` with the comma-separated date format are: +``` + + +$ head TAVG_US_2010.csv +USR0000AALC,2010,01,01,-220 +USR0000AALP,2010,01,01,-9 +USR0000ABAN,2010,01,01,12 +USR0000ABCA,2010,01,01,16 +USR0000ABCK,2010,01,01,-309 +USR0000ABER,2010,01,01,-81 +USR0000ABEV,2010,01,01,-360 +USR0000ABEN,2010,01,01,-224 +USR0000ABNS,2010,01,01,89 +USR0000ABLA,2010,01,01,59 + +``` +### 3\. Aggregate states' average temperature data + +The weather data comes from climate stations located in US cities, but you want to forecast whole states' temperatures. To convert the climate-station data to state data, first, map climate stations to their states. + +Download the list of climate stations using wget: +``` +`$ wget ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/ghcnd-stations.txt` +``` +Extract the US stations with the [grep][23] utility to find US listings. The following command searches for lines that begin with the text `"US`." The `>` is a [redirection][24] that writes output to a file—in this case, to a file named `us_stations.txt`: +``` +`$ grep "^US" ghcnd-stations.txt > us_stations.txt` +``` +This file was created for pretty print, so the column separators are inconsistent: +``` + + +$ head us_stations.txt +US009052008  43.7333  -96.6333  482.0 SD SIOUX FALLS (ENVIRON. CANADA) +US10RMHS145  40.5268 -105.1113 1569.1 CO RMHS 1.6 SSW +US10adam001  40.5680  -98.5069  598.0 NE JUNIATA 1.5 S +... + +``` +Make them consistent by using [cat][25] to print the file, using [tr][26] to squeeze repeats and output to a temp file, and renaming the temp file back to the original—all in one line: +``` +`$ cat us_stations.txt | tr -s ' ' > us_stations.txt.tmp; cp us_stations.txt.tmp us_stations.txt;` +``` +The first lines of the command's output: +``` + + +$ head us_stations.txt +US009052008 43.7333 -96.6333 482.0 SD SIOUX FALLS (ENVIRON. CANADA) +US10RMHS145 40.5268 -105.1113 1569.1 CO RMHS 1.6 SSW +US10adam001 40.5680 -98.5069 598.0 NE JUNIATA 1.5 S +... + +``` +This contains a lot of info—GPS coordinates and such—but you only need the station code and state. Use [cut][27]: +``` +`$ cut -d' ' -f1,5 us_stations.txt > us_stations.txt.tmp; mv us_stations.txt.tmp us_stations.txt;` +``` +The first lines of the command's output: +``` + + +$ head us_stations.txt +US009052008 SD +US10RMHS145 CO +US10adam001 NE +US10adam002 NE +... + +``` +Make this a CSV and change the spaces to comma separators using sed: +``` +`$ sed -i s/' '/,/g us_stations.txt` +``` +The first lines of the command's output: +``` + + +$ head us_stations.txt +US009052008,SD +US10RMHS145,CO +US10adam001,NE +US10adam002,NE +... + +``` +Although you used several commands for these tasks, it is possible to perform all the steps in one run. Try it yourself. + +Now, replace the station codes with their state locations by using [AWK][28], which is functionally high performant for large data processing. + +**station_to_state_data.sh** +``` + + +PATTERN_FILE=us_stations.txt + +for DATA_FILE in `ls TAVG_US_*.csv` +do +    echo ${DATA_FILE} + +    awk -F, \ +        'FNR==NR { x[$1]=$2; next; } { $1=x[$1]; print $0 }' \ +        OFS=, \ +        ${PATTERN_FILE} ${DATA_FILE} > ${DATA_FILE}.tmp + +   mv ${DATA_FILE}.tmp ${DATA_FILE} +done + +``` +Here is what these parameters mean: + +`-F,` | Field separator is `,` +---|--- +`FNR` | Line number in each file +`NR` | Line number in both files together +`FNR==NR` | Is TRUE only in the first file `${PATTERN_FILE}` +`{ x[$1]=$2; next; }` | If `FNR==NR` is TRUE (for all lines in `$PATTERN_FILE` only) +`- x` | Variable to store `station=state` map +`- x[$1]=$2` | Adds data of `station=state` to map +`- $1` | First column in first file (station codes) +`- $2` | Second column in first file (state codes) +`- x` | Map of all stations e.g., `x[US009052008]=SD`, `x[US10RMHS145]=CO`, ..., `x[USW00096409]=AK` +`- next` | Go to next line matching `FNR==NR` (essentially, this creates a map of all stations-states from the `${PATTERN_FILE}` +`{ $1=x[$1]; print $0 }` | If `FNR==NR` is FALSE (for all lines `in $DATA_FILE` only) +`- $1=x[$1]` | Replace first field with `x[$1]`; essentially, replace station code with state code +`- print $0` | Print all columns (including replaced `$1`) +`OFS=,` | Output fields separator is `,` + +The CSV with station codes: +``` + + +$ head TAVG_US_2010.csv +USR0000AALC,2010,01,01,-220 +USR0000AALP,2010,01,01,-9 +USR0000ABAN,2010,01,01,12 +USR0000ABCA,2010,01,01,16 +USR0000ABCK,2010,01,01,-309 +USR0000ABER,2010,01,01,-81 +USR0000ABEV,2010,01,01,-360 +USR0000ABEN,2010,01,01,-224 +USR0000ABNS,2010,01,01,89 +USR0000ABLA,2010,01,01,59 + +``` +Run the command: +``` + + +$ ./station_to_state_data.sh +TAVG_US_2010.csv +... +TAVG_US_2019.csv + +``` +Stations are now mapped to states: +``` + + +$ head TAVG_US_2010.csv +AK,2010,01,01,-220 +AZ,2010,01,01,-9 +AL,2010,01,01,12 +AK,2010,01,01,16 +AK,2010,01,01,-309 +AK,2010,01,01,-81 +AK,2010,01,01,-360 +AK,2010,01,01,-224 +AZ,2010,01,01,59 +AK,2010,01,01,-68 + +``` +Every state has several temperature readings for each day, so you need to calculate the average of each state's readings for a day. Use AWK for text processing, [sort][29] to ensure the final results are in a logical order, and [rm][30] to delete the temporary file after processing. + +**station_to_state_data.sh** +``` + + +PATTERN_FILE=us_stations.txt + +for DATA_FILE in `ls TAVG_US_*.csv` +do +    echo ${DATA_FILE} + +    awk -F, \ +        'FNR==NR { x[$1]=$2; next; } { $1=x[$1]; print $0 }' \ +        OFS=, \ +        ${PATTERN_FILE} ${DATA_FILE} > ${DATA_FILE}.tmp + +   mv ${DATA_FILE}.tmp ${DATA_FILE} +done + +``` +Here is what the AWK parameters mean: + +`FILE=$DATA_FILE` | CSV file processed as `FILE` +---|--- +`-F,` | Field separator is `,` +`state_day_sum[$1 "," $2 "," $3 "," $4] = $5 state_day_sum[$1 "," $2 "," $3 "," $4] + $5` | Sum of temperature (`$5`) for the state `($1`) on year (`$2`), month (`$3`), day (`$4`) +`state_day_num[$1 "," $2 "," $3 "," $4] = $5 state_day_num[$1 "," $2 "," $3 "," $4] + 1` | Number of temperature readings for the state (`$1`) on year (`$2`), month (`$3`), day (`$4`) +`END` | In the end, after collecting sum and number of readings for all states, years, months, days, calculate averages +`for (state_day_key in state_day_sum)` | For each state-year-month-day +`print state_day_key "," state_day_sum[state_day_key]/state_day_num[state_day_key]` | Print state,year,month,day,average +`OFS=,` | Output fields separator is `,` +`$DATA_FILE` | Input file (all files with name starting with `TAVG_US_` and ending with `.csv`, one by one) +`> STATE_DAY_${DATA_FILE}.tmp` | Save result to a temporary file + +Run the script: +``` + + +$ ./TAVG_avg.sh +TAVG_US_2010.csv +TAVG_US_2011.csv +TAVG_US_2012.csv +TAVG_US_2013.csv +TAVG_US_2014.csv +TAVG_US_2015.csv +TAVG_US_2016.csv +TAVG_US_2017.csv +TAVG_US_2018.csv +TAVG_US_2019.csv + +``` +These files are created: +``` + + +$ ls STATE_DAY_TAVG_US_20*.csv +STATE_DAY_TAVG_US_2010.csv  STATE_DAY_TAVG_US_2015.csv +STATE_DAY_TAVG_US_2011.csv  STATE_DAY_TAVG_US_2016.csv +STATE_DAY_TAVG_US_2012.csv  STATE_DAY_TAVG_US_2017.csv +STATE_DAY_TAVG_US_2013.csv  STATE_DAY_TAVG_US_2018.csv +STATE_DAY_TAVG_US_2014.csv  STATE_DAY_TAVG_US_2019.csv + +``` +See one year of data for all states ([less][31] is a utility to see output a page at a time): +``` + + +$ less STATE_DAY_TAVG_US_2010.csv +AK,2010,01,01,-181.934 +... +AK,2010,01,31,-101.068 +AK,2010,02,01,-107.11 +... +AK,2010,02,28,-138.834 +... +WY,2010,01,01,-43.5625 +... +WY,2010,12,31,-215.583 + +``` +Merge all the data files into one: +``` +`$ cat STATE_DAY_TAVG_US_20*.csv > TAVG_US_2010-2019.csv` +``` +You now have one file, with all states, for all years: +``` + + +$ cat TAVG_US_2010-2019.csv +AK,2010,01,01,-181.934 +... +WY,2018,12,31,-167.421 +AK,2019,01,01,-32.3386 +... +WY,2019,12,30,-131.028 +WY,2019,12,31,-79.8704 + +``` +## 4\. Make time-series data + +A problem like this is fittingly addressed with a time-series model such as long short-term memory ([LSTM][32]), which is a recurring neural network ([RNN][33]). This input data is organized into time slices; consider 20 days to be one slice. + +This is a one-time slice (as in `STATE_DAY_TAVG_US_2010.csv`): +``` + + +X (input – 20 weeks): +AK,2010,01,01,-181.934 +AK,2010,01,02,-199.531 +... +AK,2010,01,20,-157.273 + +y (21st week, prediction for these 20 weeks): +AK,2010,01,21,-165.31 + +``` +This time slice is represented as (temperature values where the first 20 weeks are X, and 21 is y): +``` + + +AK, -181.934,-199.531, ... , +-157.273,-165.3 + +``` +The slices are time-contiguous. For example, the end of 2010 continues into 2011: +``` + + +AK,2010,12,22,-209.92 +... +AK,2010,12,31,-79.8523 +AK,2011,01,01,-59.5658 +... +AK,2011,01,10,-100.623 + +``` +Which results in the prediction:  +``` +`AK,2011,01,11,-106.851` +``` +This time slice is taken as: +``` +`AK, -209.92, ... ,-79.8523,-59.5658, ... ,-100.623,-106.851` +``` +and so on, for all states, years, months, and dates. For more explanation, see this tutorial on [time-series forecasting][34]. + +Write a script to create time slices: + +**timeslices.sh** +``` + + +#!/bin/sh + +TIME_SLICE_PERIOD=20 + +file=TAVG_US_2010-2019.csv + +# For each state in file +for state in `cut -d',' -f1 $file | sort | uniq` +do +    # Get all temperature values for the state +    state_tavgs=`grep $state $file | cut -d',' -f5` +    # How many time slices will this result in? +    # mber of temperatures recorded minus size of one timeslice +    num_slices=`echo $state_tavgs | wc -w` +    num_slices=$((${num_slices} - ${TIME_SLICE_PERIOD})) +    # Initialize +    slice_start=1; num_slice=0; +    # For each timeslice +    while [ $num_slice -lt $num_slices ] +    do +        # One timeslice is from slice_start to slice_end +        slice_end=$(($slice_start + $TIME_SLICE_PERIOD - 1)) +        # X (1-20) +        sliceX="$slice_start-$slice_end" +        # y (21) +        slicey=$(($slice_end + 1)) +        # Print state and timeslice temperature values (column 1-20 and 21) +        echo $state `echo $state_tavgs | cut -d' ' -f$sliceX,$slicey` +        # Increment +        slice_start=$(($slice_start + 1)); num_slice=$(($num_slice + 1)); +    done +done + +``` +Run the script. It uses spaces as column separators; make them commas with sed: +``` +`$ ./timeslices.sh > TIMESLICE_TAVG_US_2010-2019.csv; sed -i s/' '/,/g TIME_VARIANT_TAVG_US_2010-2019.csv` +``` +Here are the first few lines and the last few lines of the output .csv: +``` + + +$ head -3 TIME_VARIANT_TAVG_US_2009-2019.csv +AK,-271.271,-290.057,-300.324,-277.603,-270.36,-293.152,-292.829,-270.413,-256.674,-241.546,-217.757,-158.379,-102.585,-24.9517,-1.7973,15.9597,-5.78231,-33.932,-44.7655,-92.5694,-123.338 +AK,-290.057,-300.324,-277.603,-270.36,-293.152,-292.829,-270.413,-256.674,-241.546,-217.757,-158.379,-102.585,-24.9517,-1.7973,15.9597,-5.78231,-33.932,-44.7655,-92.5694,-123.338,-130.829 +AK,-300.324,-277.603,-270.36,-293.152,-292.829,-270.413,-256.674,-241.546,-217.757,-158.379,-102.585,-24.9517,-1.7973,15.9597,-5.78231,-33.932,-44.7655,-92.5694,-123.338,-130.829,-123.979 + +$ tail -3 TIME_VARIANT_TAVG_US_2009-2019.csv +WY,-76.9167,-66.2315,-45.1944,-27.75,-55.3426,-81.5556,-124.769,-137.556,-90.213,-54.1389,-55.9907,-30.9167,-9.59813,7.86916,-1.09259,-13.9722,-47.5648,-83.5234,-98.2963,-124.694,-142.898 +WY,-66.2315,-45.1944,-27.75,-55.3426,-81.5556,-124.769,-137.556,-90.213,-54.1389,-55.9907,-30.91 \ No newline at end of file From e7e5d6dc4ffbce4455214f0d1adced9a7b65ad14 Mon Sep 17 00:00:00 2001 From: "Xingyu.Wang" Date: Tue, 24 Nov 2020 23:16:43 +0800 Subject: [PATCH 3/3] Delete 20201115 How do you write simple explanations without sounding condescending.md --- ...anations without sounding condescending.md | 105 ------------------ 1 file changed, 105 deletions(-) delete mode 100644 sources/tech/20201115 How do you write simple explanations without sounding condescending.md diff --git a/sources/tech/20201115 How do you write simple explanations without sounding condescending.md b/sources/tech/20201115 How do you write simple explanations without sounding condescending.md deleted file mode 100644 index 108440cabd..0000000000 --- a/sources/tech/20201115 How do you write simple explanations without sounding condescending.md +++ /dev/null @@ -1,105 +0,0 @@ -[#]: collector: (lujun9972) -[#]: translator: ( ) -[#]: reviewer: ( ) -[#]: publisher: ( ) -[#]: url: ( ) -[#]: subject: (How do you write simple explanations without sounding condescending?) -[#]: via: (https://jvns.ca/blog/2020/11/15/simple-explanations-without-sounding-condescending/) -[#]: author: (Julia Evans https://jvns.ca/) - -How do you write simple explanations without sounding condescending? -====== - -Sumana Harihareswara wrote an interesting blog post [Plain Language Choices][1] recently, about writing about complicated topics using simple language and how it can sometimes come off as condescending. - -I really like explaining complicated topics while trying to avoid unnecessary jargon, and I realized that I’ve thought a lot about how to do it well. So here are a bunch of things I try to do when I use simple language to avoid coming off as condescending. - -### use some jargon to give the reader search terms - -Sometimes I see writing that completely avoids all jargon and instead substitutes simple language for all “jargon”-y words. - -I like to include some jargon in my explanations because otherwise it’s impossible for the reader to search & learn more about the concept they’re trying to learn about. - -### write (mostly) true explanations - -Something else I see sometimes in ELI5-type explanations is an explanation in plain language that’s not actually true in a useful way. I’m pretty sympathetic to why people do this – it’s super hard to write simple explanations that are also true! - -And actually sometimes when I’m trying to write down a simple/clear explanation for a concept, I realize that I don’t actually understand the concept as well as I thought and that I’m not able to explain it. That’s okay! - -I think there are a few options here: - - * try to say only things that are true (or at least which are a useful model for how the world works even if they’re not 100% true) - * write things that are not really true / that you’re not sure of, but point out that they may not be true (“I think it works like X, but I realize now that it might be Y instead, I’m not sure!“) - - - -### only use “fun” visual elements on explanations that are actually well written & easy to understand - -This happens more with visual aids than with simple language but I’ll include it anyway. Sometimes I see explanations which have “fun” elements to it to make it them seem more approachable where the explanation itself is still pretty unclear. - -I try to be careful about this in my own work – I try to only attach “fun” elements (like a fun illustrated cover) to explanations that I’ve spent a lot of time on making really clear. Basically to me “fun” things are a signal that the content itself is really clear/accessible, and I try to not misuse that signal. - -I think [why’s poignant guide to ruby][2] is a nice example of something that’s fun and clear and which has helped a lot of people learn Ruby. - -Another nice example of this is: I know someone who got her master’s thesis printed as a paperback book and illustrated with some great drawings related to the topic of her thesis (trans represensentation in media). It’s called “I’m supposed to relate to this?”, here’s the [paperback][3]. - -I ended up reading the whole thing because, in addition to having the fun illustrations, her master’s thesis was really well written and interesting! The fact that she did the work to print a paperback book of her thesis and get it illustrated was a sign that she’d worked on making the writing accessible to a non-academic audience, and it was true! - -### tell a relevant story - -Stories can really help people learn! For example, something I’ve done a lot on this blog is talk about a problem I ran into in the course of my job and what I did to solve that problem. - -Some kinds of stories that I think work well: - - * a real problem that someone ran into, to motivate why the concept is interesting / important to learn - * something that’s happening on a computer, framed as a “story” (for example tells a story about how DNS works. Everything in the story literally corresponds to exactly what happens when you make a DNS query) - - - -Sometimes I see stories used to explain concepts that don’t fit into either of these and feel kind of pasted on, like they’re there to help the concept seem “fun” but don’t actually illustrate the concept or motivate why it might be useful to learn it. - -### have a specific audience in mind - -I try to write relatively simple explanations, but when I write I also generally assume a lot of knowledge on the part of my audience. - -Sometimes I see explanations of complicated concepts that start with explaining the very basics of the topic. This usually isn’t that effective: if someone is trying to understand some super technical aspect of containers, they probably understand the basics of containers already! - -“Have an audience” is more of a general writing tip so I’ll leave it at that. - -### on using simple language as a joke for people who already understand the idea - -Here’s a very fun explanation of a complicated thing using simple language: [Gödel’s Second Incompleteness Theorem Explained in Words of One Syllable][4]. - -On one hand, this is fun! I enjoyed reading it. On the other hand, I think the main audience for this is probably people who already more or less understand Gödel’s Second Incompleteness Theorem. - -For example, someone pointed out that “if math is a not a load of bunk” in this text is code for “Peano arithmetic is consistent” with (“math” being “Peano arithmetic” and “not a load of bunk” meaning “consistent”). Which I find very charming, but also I found it a little hard to decode when reading it. - -And (as we talked about before about jargon), if you know that “Peano arithmetic is consistent” is the relevant bit of jargon, you can find all kind of fascinating things, like [a blog post by John Baez from 2011 discussing an attempted proof that Peano arithmetic was inconsistent][5]) - -(I’m also reminded here of the XKCD [up goer five][6], which is very delightful, but I don’t think I learned anything about spaceships from reading it) - -### that’s all! - -I’d love to hear more thoughts on this – I think there are probably more ways that simple explanations can feel condescending that I’ve missed! - -I really don’t think they _need_ to feel condescending though – to me the point of writing a clear/simple explanation is usually that I think the idea is _not actually fundamentally that complicated_ and so I’m just explaining it in a way that’s exactly as complicated as it needs to be. - --------------------------------------------------------------------------------- - -via: https://jvns.ca/blog/2020/11/15/simple-explanations-without-sounding-condescending/ - -作者：[Julia Evans][a] -选题：[lujun9972][b] -译者：[译者ID](https://github.com/译者ID) -校对：[校对者ID](https://github.com/校对者ID) - -本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出 - -[a]: https://jvns.ca/ -[b]: https://github.com/lujun9972 -[1]: https://www.harihareswara.net/sumana/2020/11/12/0 -[2]: https://poignant.guide/ -[3]: https://untrans.bigcartel.com/ -[4]: https://www2.kenyon.edu/Depts/Math/Milnikel/boolos-godel.pdf -[5]: https://golem.ph.utexas.edu/category/2011/09/the_inconsistency_of_arithmeti.html -[6]: https://xkcd.com/1133/