2021/07/24

How to stream large responses using the Javascript Fetch API:

const response = await fetch(url);
const reader = response.body.getReader();

while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  console.log('Received', value);
}

console.log('Response fully received');

Also: the circumstances under which ~/.*profile files are run: only if you log in.

2021/07/31

That you can use git grep to search for a string in a git repo.

2021/09/15

That you can, technically, write inline comments in bash. Specifically, you have to put the comments in a subshell:

echo abc `#put your comment here` \
     def `#another chance for a comment` \
     xyz etc

^ from https://stackoverflow.com/a/23872003/6571327

2021/10/04

that different versions of sort(1) have different sorting presets. For example, on ubuntu 20.04 vs debian buster, sort will swap the order of how it sorts _s. These differences can be resolved by passing --unique --dictionary-order --stable.

Bonus: -h/--human-numeric-sort can sort SI suffixes.

2021/10/13

about reset, which resets your terminal after a series of undesired ANSI escape codes. I also learned about echo -e "\e" meaning “interpret backslash escape codes” (-e) and "\e" == "\033".

2021/10/19

that you can write multiple lines to a file using a cat > path/to/file with argument, like

cat > /tmp/file
subsequent typed input
is still stdin, which is being read by `cat`
and so will be written to the file 
^C # cancels `cat` writing to the file

2021/10/30

about terminfo, the terminal information database: it can be queried to find all kinds of fun facts about your current terminal. For instance,

• tput cols => number of columns in the current terminal
• tput colors => number of colors the terminal supports
• tput setaf => generate ansi codes to set the foregrount color in the terminal

Also, infocmp screen dumps a bunch of other related information drawn from terminfo.

2021/11/23

locate searches all filenames for a keyword using a database which you can update updatedb. whereis searches for binaries using the same mechanism.

https://www.madebygps.com/an-intro-to-finding-things-in-linux/

2021/12/01

that you can write any file in your git history using git show <ref>:<path>.

See https://juplo.de/cat-any-file-in-any-commit-with-git/

2022/01/31

that the fmt utility exists. It seems to fold or center text according to a target width.

2022/02/08

In zsh, ls *(n) will “human-sort” mostly-numeric filenames.

https://zsh.sourceforge.io/Doc/Release/Expansion.html#Glob-Qualifiers

https://twitter.com/nedbat/status/1491032300800421892

2022/02/25

that you can use variables in make targets, e.g.

path/${TO}/file: path/${TO}/other/file

2022/03/06

About git-sparse-checkout:

• https://linuxhint.com/what-is-git-sparse-checkout/
• https://git-scm.com/docs/git-sparse-checkout

The general script for doing a sparse checkout is:

#!/usr/bin/env bash
git_url="${git_url:?required}"
target_dir="${target_dir:?required}"

is_git_dir() { git rev-parse; }

init() {
  mkdir -p "$target_dir" && cd "$target_dir"
  if ! is_git_dir; then
    git init && git remote add -f origin "$git_url"
  fi

  git config core.sparseCheckout true &&
    git sparse-checkout init &&
    git sparse-checkout set test &&
    git pull origin master
}

see also: https://about.gitlab.com/blog/2020/03/13/partial-clone-for-massive-repositories/

2022/03/18

How to check if a debian package is installed:

deb_pkg_is_installed() {
  sudo dpkg --get-selections "$1" | grep -q "$1";
}

2022/03/19

How to pass a multi-character field separator to awk:

awk -F '[ :]+' '{print $1, $3}'
# character set: [ :]
# repetition:        +

2022/03/22

You can replace the nth instance of a pattern on a line with sed:

; printf "a\tb\tc\n"  | sed 's/\t/\t~/2' 
# a       b       ~c

2022/04/04

to run terraform output $output_name to print a named output

2022/04/20

tar -ztvf file.tar.gz will list the files in tar.gz

2022/04/29

how to check the last apt update time:

last_update="$(stat --format=%Y /var/cache/apt/pkgcache.bin)"

https://askubuntu.com/a/410259/505362

2022/05/09

• that sudo can be configured with the visudo command and that sudo can produce audit logs. See https://www.sudo.ws/posts/2022/05/sudo-for-blue-teams-how-to-control-and-log-better/

• that there’s an Element.closest(selector) method, which means you can run this function to set the url to the closest link to an element:

  ((el) => {
    let id = el.closest("[id]")?.id;
    if (!id) return;
    else location.hash = id;
  })($0);
  

• that you can use

    SELECT /*...*/
    FROM one_table
    JOIN other_table
      USING (common_column)
  

  as shorthand for one_table.common_column = other_table.common_column. See https://www.postgresql.org/docs/current/sql-select.html#SQL-FROM

2022/06/01

That pg_dumpall exists. That you need to use pg_dumpall to dump tablespaces, roles, and subscriptions. That pg_dumpall will dump the postgres role by default, causing resultant dump to fail to restore. Fortunately, I’m able to grep -ve '^CREATE ROLE postgres

2022/06/07

That 1Password has a separate SSH key type! https://developer.1password.com/docs/ssh/manage-keys/

Also, that make has a MAKEFLAGS variable that can be used in recursive calls to make: see https://earthly.dev/blog/make-flags/. This also means you can set -r, which eliminates default rules and file actions.

2022/07/02

That you can unescape newlines with printf "%b" 'escaped\nstring':

escaped='a\nb'
printf "%b" "$escaped"
# a
# b

2022/08/24

that you can lsblk to list the block devices that the OS knows about without sudo.

that you can sudo growpart to increase a partition size without restarting the machine.

2022/10/03

That you can write >& 1 and it still redirects to stdout! example:

f() { echo "ok" >&2; }
f 2>& 1 | grep ok

2022/10/23

That ARIA stands for Accessible Rich Internet Applications! See https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA

That in linux ls(1) uses the environment variable LS_COLORS to determine the colors in which the filenames are to be displayed. Also, that linux specifies a dir_colors(5) configuration file format for dircolors(1), which exports LS_COLORS

See https://man7.org/linux/man-pages/man5/dir_colors.5.html

That nix is complicated, which is exactly what my prior research told me.

On one hand, it is AWESOME to cd into a project and watch direnv automatically spawn a development shell with all the fixings. On the other, the nix ecosystem is bewildering. I read through a few articles:

• https://learnxinyminutes.com/docs/nix/ : covered the expression language

• https://xeiaso.net/blog/nix-flakes-1-2022-02-21, part of the https://xeiaso.net/blog/series/nix-flakes series : made a great pitch for using flakes:

  One of the big annoying parts about getting into Nix is that setting up projects isn’t totally a defined science…Nix flakes helps fix this by doing a few things:

  1. Defining a flake.nix as the central “hub” for your project’s dependencies, exposed packages, NixOS configuration modules and more.
  2. Shipping a set of templates so that you can get projects started easily. Think something like Yeoman but built directly into Nix. You can also define your own templates in your flake.nix.

• https://nixos.org/guides/declarative-and-reproducible-developer-environments.html : relied on shell.nix

• https://nixos.org/guides/towards-reproducibility-pinning-nixpkgs.html introduced niv, a nix package-management cli that differs from nix flake

• https://github.com/nix-community/awesome-nix#rust : there are many nix+rust tools, and it’s not at all clear to me which (if any) to use

• https://xeiaso.net/talks/nixos-pain-2021-11-10 : talks about some pain I’ve encountered (e.g. the ambiguity of the term “nix”, flakes splitting the ecosystem) and some I haven’t (NixOs, NixOps troubles – I don’t use those yet.)

• https://o.librepush.net/solutions/nix/developing-python-rust-projects-on-nixos/ eschewed flakes since the author wanted to install editable python packages (reasonable)

Then there are the reasonable-yet-still-more confusing tools like

• https://github.com/jetpack-io/devbox : provide development shells
• https://www.riff.sh/ : provide external linkable libraries for rust projects

both of which expose tools to use nix (package manager) to provide development environments without requiring devs to write any nix (the language). Since I’m still struggling with the configuration language, I’m sympathetic to anyone wanting to avoid writing nix.

By dint of procrastinating, I now know that

• someone’s rewriting nix-the-language? https://tvl.fyi/blog/tvix-status-september-22
• someone bolted nix-the-build-system onto Bazel? https://www.tweag.io/blog/2018-03-15-bazel-nix/
• pants build currently uses absolute paths for linking some tools into its hermetic build environment? https://github.com/pantsbuild/pants/issues/14492
• someone’s introducing a new concept, nix cells to make nix flakes easier? https://github.com/divnix/std

I’m still putting off reading up on nix overlays at the time of writing.

I could see someone writing SDKs for nix-the-package-manager in non-nix languages that serialize and deserialize to JSON, just like is happening to infrastructure-as-code languages (pulumi, helm, terraform cloud development kit), etc.

that you can search for nix package names on https://search.nixos.org/packages

2022/10/24

That you can list all addresses in terraform state by running

terraform state list

Docs at https://developer.hashicorp.com/terraform/cli/commands/state/list

Also, that compgen -v is a function, at least on Mac zsh. Apparently in zsh compgen -v calls something like

for var_name in "${(k)parameters[@]}"; do
  printf '%s\n' "$var_name"
done

which is the first time I’ve seen that kind of shell syntax. It generates a bad substitution message in bash 3.2, so I can only assume it’s zsh-specific.

2023/01/28

That you need ato git add a flake.nix for the flake’s contents to start to work with nix develop

This is needed because Nix flakes respects gitignores. If you don’t add things to the git staging area, git doesn’t know about the files at all, and Nix flakes can’t know if it should ignore them. – https://xeiaso.net/blog/nix-flakes-go-programs

that “truecolor” support in shells is detected with the $COLORTERM env var

2023/05/30

That there’s a Linux command to do sql-like JOINs of lines in files, possibly based on a field in each line: join(1)

2024/04/17

That shasum(1) is implemented in perl: https://linux.die.net/man/1/shasum, though shaXXXsum et al. seem to be implemented in C: https://github.com/coreutils/coreutils/blob/a9b78541fa7c06567c5b82fb4e89d85a1dc0c611/src/digest.c#L103